What is the easiest way to run python scripts on a cloud server?

Question

I have a web crawl of a python script that takes many hours, and it cannot be run completely on my local machine. Is there any convenient way to deploy this to a simple web server? The script basically loads web pages into text files. What is the best way to do this? Thanks!

Answer 1

Since you said that performance is a problem and you are making a web scraper, the first thing to try is the Scrapy framework - it is a very fast and easy to use web scraper. scrapyd tool will allow you to distribute the workaround - you can have several scrapyd services running on different servers and share the load between them. Cm:

• Common workarounds
• Launch Scrapy on Amazon EC2

There is also a Scrapy Cloud :

Scrapy Cloud combines the highly efficient development of Scrapy environments with a robust, fully featured production environment for deploying and running your workarounds. This is similar to Heroku for Scrapy, although other technologies will be supported in the near future. It runs at the top of the Scrapinghub platform, which means your project can scale if necessary.

Answer 2

There are several places where you can do this. Just google for "python in the cloud", you'll come up with a few, for example https://www.pythonanywhere.com/ .

In addition, there are also several cloud-based IDEs that essentially give you a small virtual machine for free, where you can develop your code in an IDE web environment and also run it in a virtual machine, one example is http: //www.c9 .io .

Answer 3

As an alternative to the solutions already provided, I would suggest Heroku . You can not just easily deploy a website, but scripts for launching bots.

The basic account is free and quite flexible.

This blog entry , this and this video contains practical examples of how to make it work.

Answer 4

As another alternative, you can copy and paste your script into WayScript (no need to change the code or configure anything).

Here is a tutorial.