Good design for transparent SMTP proxy in C #

Question

If you were to create a transparent SMTP proxy in C # (.net 4) to meet the following initial requirements

• Scales well
• Writes all traffic to databases
• Can be easily extended for antivirus scanning applications.

Given these factors in a broad sense, what will your design look like? Are you creating specific Listener, Sender, and logger classes, or something more abstract? And would you use callbacks, threads or processes and why?

Answer 1

This is a non-trivial application. Some ideas that should help:

SMTP scalability

In general, scaling a network application means being able to scale (as with most machines) rather than up (a more expensive machine). This means the ability to have multiple servers to handle SMTP requests. Please note that this will probably require network-level support (routers that can distribute messages in the SMTP farm).

Yes, in order to scale and shape the SMTP, you probably want to use multiple threads (most likely from some thread pool). Note that implementing multi-threaded sockets is not trivial.

As for the processes, I think that one process (probably a Windows service) with multiple threads for each SMTP server is a good way.

Database scalability

Keep in mind that a database can also be a scalability bottleneck. To design large loads, you would like to be able to scale your data level horizontally. This means being able to write to more than one db server. This leads to the fact that they can report on many database servers (which is much more complicated than reporting from one).

SMTP Reliability

Is this a problem / requirement? If so, this is another reason to support the farm (well, if we have several servers for reliability, which we could call a cluster) of servers, and not just one. Note that the farm will have to have a way to tell the clan that it failed (perhaps through some kind of heart rate mechanism).

Database Reliability

To make the database reliable, you also have to do some clustering. This is not cheap or trivial (but has been done several times with multiple database platforms).

Queuing

One way to handle surge during server startup is through a message queue. Thus, the server can continue to send messages, but you do not wait for the chain of extensible modules to complete their processing. Note that this adds another level of complexity and a point of failure for the system.

extensibility

One way to add features such as database logging and scanning attachments is to add a chain of MessageInsepctors or MessageHandlers. You probably want to allow them to be configured in a specific order (for example, checking for viruses before logging so that you do not register infected items).

Another aspect to consider is which plug-ins can block the transmission of a message (such as a virus scanner) and the plug-in that can run after the message has passed (logging).

In terms of adding plug-in support, you can use something like the MEF (Managed Extensibility Framework).

Rethink the wheel

Enabling all this functionality will require considerable development time. It can be cheaper / faster / easier to just buy a solution from the shelf, which does all this for you (this problem has already been solved several times).

Answer 2

It seems to me that you want to create something that is future proof, but without immediately performing a complete monty data separation, clustering, etc.

Partitioning It is advisable to consider ways to split the data load in advance. Initially, you can apply partitioning logic, which simply redirects everything to the same destination, but this will allow you to easily share the load when necessary - and the ability to verify that it works in advance.

Queuing I would highly recommend a queuing solution because it allows you to separate the workload of receiving messages and actually sending them. The queue is also great for reliable messaging, since you want to guarantee delivery only once. Take a look at some of the questions comparing MSMQ with Service Broker, as they serve different audiences, and both have different caveats.

SMTP Most email servers allow you to send more than one email per connection (and I mean more than the same email with multiple recipients). This can significantly increase the number of emails that you can click on the remote mail server, but it depends on how it was configured. If you did not configure them or do not know the allow values, I would recommend a testing strategy in which you start by trying to deliver two mails and register the result for a remote server. Try doubling the next time if it works, and reduce to half if it doesn't work. It looks like the TCP window is expanding when you reliably transmit.

extensibility I would not have thought. Achieving scalability and reliability is much more difficult and therefore much more important to obtain the right. Extensibility is simply the ability to connect to additional steps along the way, and you can add seams for this when the main system is in place or when you start adding features that you think should be optional but built-in.