Understanding Python Web Application Deployment

Question

I think I do not quite understand the deployment process. Here is what I know:

• when we need to do a hot deployment - that means we need to change the code that is live - we can do this by reloading the modules, but
• imp.reload is a bad idea, and we must restart the application instead of reloading the modified modules.
• Ideally, the working code should be a clone of your code repository, and anytime you need to deploy it, you just pull the changes

Now let's say that I have several instances of the wsgi application working behind a reverse proxy, for example nginx (on ports such as 8011, 8012, ...). Suppose I get 5 requests per second.

Now in this case, how can I update my code in all running instances of the application.

• If I stop all instances, update them all, and then restart them all - I will definitely lose some queries.
• If I update each instance one by one, then the instances will be in conflicting states (some of them will run the old code and some new ones) until all of them are updated. Now, if the request falls into the updated instance, and then the subsequent (and related) request falls into the older instance (not yet updated), then I get the wrong results.

Can someone explain in detail how busy applications like this are hot deployed?

Answer 1

To deploy across multiple hot instances that are behind a load balancer, such as nginx I like to deploy the deployment using a tool like Fabric .

• Cloth connects you to the server 1
• Web server shutdown
• Deploy the changes either using VCS or using tarball migration with a new application
• Start the web server
• GOTO1 and connect to the next server.

This way you will never be offline, and that will be unimpeded, since nginx knows when the web server is reset, when it tries to round it, and moves on to the next, and as soon as the node / instance comes back, it will go back in production.

EDIT:

You can use the ip_hash module in nginx to ensure that all requests from the same IP address go to the same server for the session length

This directive forces requests to be distributed between upstream streams based on the client IP address. The key for the hash is the network address of the C-client class. This method ensures that a client request is always sent to the same server. But if this server is considered inoperative, the request of this client will be transferred to another server. This makes it highly likely that clients will always connect to the same server.

This means that after updating your web server and connecting the client to a new instance, all connections for this session will continue to be redirected to the same server.

It will leave you in a situation

• The client connects to the site, gets access from the server 1
• Server 1 is updated before the client completes everything they do.
• Is the client potentially left in a state of uncertainty?

This scenario raises the question: are you removing things from your API / site that could potentially leave the client in a state of uncertainty? If everything you do, for example, updates user interface elements or adds pages, etc., but does not change any front-end APIs, you should not have any problems. If you remove API functions, you may have problems.

Answer 2

Could you disconnect half of your servers (say, pulling them out of the load balancing pool), and then upgrade them. Then return them online while knocking the other half down. Then update them and bring them back online.

This ensures that you stay online and make sure that you have never had old and new versions of your application online at the same time. Yes, this will mean that your site will operate at half its capacity over time. But can this be normal?