Java, Multi Threading, ExecutorService

Question

Below are some parts of my code that uses Threading. The goal is to retrieve all the records from the database (about 5,000,000) and send them messages with warning messages. The problem I am facing is that the emailRecords variable gets very heavy and it takes too much time to send an email message. How can I do this quickly using multithreading to process 5,000 records in parallel? I tried to use ExecutorService, but got confused in its implementation. I was confused in the checkName (), getRecords () and sendAlert () methods. All these 3 methods are used appropriately. So where to use executorService?

Please provide me a suggestion on how to proceed with the following code and which part needs editing? Thank you in advance!!

public class sampledaemon implements Runnable { private static List<String[]> emailRecords = new ArrayList<String[]>(); public static void main(String[] args) { if (args.length != 1) { return; } countryName = args[0]; try { Thread t = null; sampledaemon daemon = new sampledaemon(); t = new Thread(daemon); t.start(); } catch (Exception e) { e.printStackTrace() } } public void run() { Thread thisThread = Thread.currentThread(); try { while (true) { checkName(countryName); Thread.sleep(TimeUnit.SECONDS.toMillis(10)); } } catch (Exception e) { e.printStackTrace(); } } public void checkName(String countryName) throws Exception { Country country = CountryPojo.getDetails(countryName) if (country != null) { getRecords(countryconnection); } } private void getRecords(Country country, Connection con) { String users[] = null; while (rs.next()) { users = new String[2]; users[0] = rs.getString("userid"); users[1] = rs.getString("emailAddress"); emailRecords.add(props); if (emailRecords.size() > 0) { sendAlert(date, con); } } } void sendAlert(String date, Connection con) { for (int k = 0; k < emailRecords.size(); k++) { //check the emailRecords and send email } } } 

Answer 1

From what I can say, is that you are likely to receive single-threaded data and multi-threaded for sending via email. Roughly speaking, you will ride a bike through your result set and make a list of entries. When this list reaches a certain size, you make a copy and send this copy for processing in the stream and clear the original list. At the end of the result set, check to see if you have raw records in your list and submit them to the pool.

Finally, wait until threadpool finishes processing all entries.

Something like that:

 protected void processRecords(String countryName) { ThreadPoolExecutor executor = new ThreadPoolExecutor(10, 10, 10, TimeUnit.SECONDS, new ArrayBlockingQueue<Runnable>(5), new ThreadPoolExecutor.CallerRunsPolicy()); List<String[]> emaillist = new ArrayList<String>(1000); ResultSet rs = .... try { while (rs.next()) { String user[] = new String[2]; users[0] = rs.getString("userid"); users[1] = rs.getString("emailAddress"); emaillist.add(user); if (emaillist.size() == 1000) { final List<String[]> elist = new ArrayList<String[]>(emaillist); executor.execute(new Runnable() { public void run() { sendMail(elist); } } emaillist.clear(); } } } finally { DbUtils.close(rs); } if (! emaillist.isEmpty()) { final List<String[]> elist = emaillist; executor.execute(new Runnable() { public void run() { sendMail(elist); } } emaillist.clear(); } // wait for all the e-mails to finish. while (! executor.isTerminated()) { executor.shutdown(); executor.awaitTermination(10, TimeUnit.DAYS); } } 

Answer 2

The advantage of using FixedThreadPool is that you don’t have to do the expensive process of creating threads over and over , it runs at the beginning ... see below ..

 ExecutorService executor = Executors.newFixedThreadPool(100); ArrayList<String> arList = Here your Email addresses from DB will go in ; for(String s : arList){ executor.execute(new EmailAlert(s)); } public class EmailAlert implements Runnable{ String addr; public EmailAlert(String eAddr){ this.addr = eAddr; } public void run(){ // Do the process of sending the email here.. } } 

Answer 3

Creating a second thread to do all the work instead of doing the same job in the main thread will not help you avoid the problem of populating the emailRecords list emailRecords 5 million records before processing any of them.

It seems your goal is to be able to read from the database and send emails in parallel. Instead of worrying about code, first think about the work algorithm you want to perform. Something like that:

• In one thread, the query is for records from the database and for each result add one job to the ExecutorService
• This work sends an email to one person / address / entry.

or alternatively

• Reading records from the database of batches N (50, 100, 1000, etc.)
• Send each batch to the performer.