Sending a large volume of emails with ColdFusion

Question

This question can probably be related to the execution of any large volumes, but in this case I am trying to send emails.

I set up the sending process in a new thread so that the user does not wait and reset the request timeout for an hour.

The problem is that after the process receives up to 2000 sent messages (loops over a code below 2000 times), the server runs out of memory, stops responding and needs to be rebooted.

Reading other topics on this subject, CF should be able to handle this volume of email messages.

One thing that I considered is changing all calls to objects to direct database queries and using the cfmail tag to (I think) remove all objects from the created one and create an access request (which, I think, is what happens), but I’m not sure that this will change the situation and really want to avoid this approach, if possible. Something else that I was considering was dividing it into 3 or 4 separate threads, but again, not sure if this would solve the problem.

Has anyone encountered this problem, and what did you find while working to continue processing without slowly populating and destroying the server?

thread name="sendBroadcastEmail" rc="#rc#" prc="#prc#" filters="#rc.filters#" email="#email#" emailSignature="#emailSignature#"{ createObject( "java", "coldfusion.tagext.lang.SettingTag" ).setRequestTimeout(javaCast( "double", 3600 )); //get profiles that it will be sent to var sendToProfiles = profileService.getWithFilters(rc.filters); var mailService = getPlugin("MailService"); var emailSent = false; var sentCount = 0; var failedCount = 0; //send the email (and log in profile events) if (listFind(attributes.rc.email.action,'send')){ for ( i=1; i<=arrayLen(sendToProfiles);i++){ var profile = sendToProfiles[i]; try{ if (len(trim(profile.getPrimaryEmail()))){ var emailBody = profile.processDynamicPlaceholders(attributes.rc.email.body); var emailBody = attributes.emailSignature.getHeader() & emailBody & attributes.emailSignature.getFooter(); var sendEmail = mailService.newMail( from = attributes.emailSignature.getEmailAddress(), //to = profile.getPrimaryEmail(), to = Application.settings.testemail, subject = attributes.rc.email.subject, body = emailBody, type="html"); sendEmail.addMailParam(disposition='attachment', file=attributes.email.getAttachmentWithPath()); mailService.send(sendEmail); //log profile event profile.saveEvent(eventType = 3, title="Broadcast Email: #attributes.rc.email.subject#", description="Broadcast Email Sent: Subject: <br> #attributes.rc.email.subject#", sentContent=emailBody, ref2=1); sentCount++; } } catch (any exception){ //log profile event profile.saveEvent(eventType = 3, title="FAILED Broadcast Email", description="<br>Subject: #attributes.email.subject#<br>This email should have been sent to this profile, but the attempted send failed. The likely cause is a malformed email address.", sentContent=emailBody, ref2=0); failedCount++; } } emailSent = true; } //persist email object if (listFind(attributes.rc.email.action,'save')){ email.setTstamp(attributes.prc.now); email.setSent(emailSent); email.setStatsSent(sentCount); email.save(); } }//end thread 

Answer 1

One approach would be to generate emails in time series to evenly distribute the load. Batch size and delay between batches can be customized to suit your environment.

  thread name="sendBroadcastEmail" rc="#rc#" prc="#prc#" filters="#rc.filters#" email="#email#" emailSignature="#emailSignature#"{ createObject( "java", "coldfusion.tagext.lang.SettingTag" ).setRequestTimeout(javaCast( "double", 3600 )); // set thread to a lowish prority var currentThread = CreateObject( "java","java.lang.Thread" ).currentThread(); var priority = currentThread.getPriority(); currentThread.setPriority( 3 ); //get profiles that it will be sent to var sendToProfiles = profileService.getWithFilters(rc.filters); var mailService = getPlugin("MailService"); var emailSent = false; var sentCount = 0; var failedCount = 0; //send the email (and log in profile events) if (listFind(attributes.rc.email.action,'send')){ var emailsPerBatch = 1000; // divide into batches, set size here var batchcount = Ceiling( ArrayLen( sendToProfiles ) / emailsPerBatch ); // number of batches var batchdelay = 120000; // set delay between batches (ms) // initialise first batch var firstitem = 1; var lastitem = emailsPerBatch; for( var batch=1; batch<=batchcount; batch++ ) { if( batch > 1 ){ // delay sending next batch and give way to other threads currentThread.yield(); currentThread.sleep( batchdelay ); } for ( var i=firstitem; i<=lastitem;i++ ){ var profile = sendToProfiles[i]; // generate emails ... } // initialise next batch firstitem = lastitem++; lastitem += emailsPerBatch; if( lastitem > ArrayLen( sendToProfiles ) ) { // last batch lastitem = ArrayLen( sendToProfiles ); } } emailSent = true; } currentThread.setPriority( priority ); // reset thread priority }//end thread