Quick Response Command Line Scripts

Question

I have written Python scripts with the command line, but lately I have really been disappointed in speed.

I'm not necessarily talking about processing speed, task dispatching, or other processes that depend on the command line (usually this is a design / implementation problem), but I'm just talking about launching a tool to get the help menu, or display minimal information.

As an example, Mercurial has a value of about 0.080scs, and GIT has a value of 0.030scs.

I have studied the Mercurial source code (it's still Python), but the response to the quick response script still eludes me.

I think imports and the way you manage them is a big reason for the initial slowdown. But is there any best practice for fast, responsive command line scripts in Python?

One Python script that imports os and optparse and executes main () to parse some argument parameters takes 0.160scs on my machine just to display the help menu ...

This is 5 times slower than just starting git!

Edit:

I shouldn't have mentioned GIT as it is written in C. But the Mercurial part is still standing, and no, pyc don't feel much improvement (at least for me).

Edit 2:

Although lazy imports are the key to speeding up in Mercurial, they reduced to regular Python scripts do not have automatically generated scripts with pkg_resources in them, for example:

 from pkg_resources import load_entry_point 

If you manually created scripts that do not use pkg_resources, you should see at least a 2-speed increase.

But! Be careful that pkg_resources provides a great way to version dependencies, so make sure you know that not using it basically means possible version conflicts.

Answer 1

In addition to compiling Python files, Mercurial modifies import on demand, which really shortens startup time. It installs __builtin__.__import__ into its own import function in the demandimport module.

If you look at the hg script in / usr / lib / (or wherever it is on your computer), you can see this for yourself in the following lines:

 try: from mercurial import demandimport; demandimport.enable() except ImportError: import sys sys.stderr.write("abort: couldn't find mercurial libraries in [%s]\n" % ' '.join(sys.path)) sys.stderr.write("(check your install and PYTHONPATH)\n") sys.exit(-1) 

If you change the demandimport line to pass , you will find that the startup time increases significantly. On my car, it nearly doubles.

I recommend that you study requestimport.py to learn how to apply a similar method in your own projects.

PS Git, as I'm sure you know, is written in C, so I'm not surprised that it has a quick start.

Answer 2

Sorry, but certainly not 0.08 seconds that bother you. Although you don’t say that it looks like you are using an “outter” shell (or other language) scritp that calls several hundred Python scripts inside a loop - This is the only way these startup times start to make any difference . Thus, either withhold this important information in your question, or your father is this guy .

So, suppose you have external scripts that call the python process extraction order: write this external script in Python and import the desired python material that you need in the same process and run it from there. Therefore, you will click on the launch of the interpreter and import of the module for each execution of the script.

This applies even to mercury, for example. You can import "mercurial" and the corresponding submodules and call functions inside it that perform the same actions as the equivalent command line arguments