Can I show the build progress using SCons?

Question

I have a simple build system for a project that I built using SCons . The project has many source files, and I thought it would be more convenient to display some information about the progress as the assembly develops. SCons provides constructive variables such as CXXCOMSTR , which I can override to control what is displayed on the terminal during each build step. For example, instead of seeing something like:

 gcc file.c -o file.o 

It would be nice to have something like:

 [1/10] Compiling file.c: `gcc file.c -o file.o` 

Where [1/10] indicates that this is the first of ten goals that are updated during this build. Is there a way to access this information so that I can generate such messages? It seems like I need to know the total number of goals to be updated (which is based on a dependency scan that SCons does) and some way to list each of them. I know that this behavior is similar to other build systems like CMake and waf, but have nothing to do with SCons (expansive) documentation.

Answer 1

This can be done using the Progress () SCons function.

There are more examples on the SCons task page, just search for "Progress."

You can either provide the called function, a string that will be displayed periodically, or a list of rows that will be displayed in a rotating manner. You can access the current goal being created, but I don’t think it can be found out that the percentage is complete.

Answer 2

I was able to come up with a pretty decent workaround that allows me to display build progress in percent. Basically how the following code works:

• If the goal has never been built before, it displays [??%] next to each goal
• The first time it is built, it will cache shared nodes processed for this purpose in .scons_progress
• The next time you run the assembly, it will be read from the file at startup and for the last time will receive shared nodes from this target and use it to display the percentage (ie [50%] ).
• If the number of nodes exceeds the total number of caches, it will revert to [??%] and update .scons_progress
• Over time, when your SConstruct "stabilizes" (that is, you do not change all the time), these cache values ​​should become more and more accurate.
• Updating the cache will slow down your build (a bit per node), so you can set interval to 5 so that it only updates the cache every 5 nodes

Here is the code:

In custom_utils.py in the site_scons/ section:

 PROGRESS = { "env": None, "nodes": 0, "interval": 1, "total": 0, "cache": {}, "cache_key": "_", "cache_name": ".scons_progress", "cache_update": False } def get_progress_total(): return PROGRESS["cache"].get(PROGRESS["cache_key"], 0) def update_progress_cache(): PROGRESS["cache"][PROGRESS["cache_key"]] = PROGRESS["nodes"] with open(PROGRESS["cache_name"], "w") as f: f.write(json.dumps(PROGRESS["cache"])) def load_progress_cache(): try: with open(PROGRESS["cache_name"]) as f: PROGRESS["cache"] = json.load(f) cache_key = "_".join(COMMAND_LINE_TARGETS) if cache_key: PROGRESS["cache_key"] = cache_key except IOError as e: pass def progress_function(node): PROGRESS["nodes"] += PROGRESS["interval"] #Only update cached environment if it is not None if node.env: PROGRESS["env"] = node.env PROGRESS["env"].Replace(PROGRESS = "[??%]") #Update display message if PROGRESS["nodes"] <= PROGRESS["total"]: PROGRESS["env"].Replace(PROGRESS = "[%d%%]" % (100*PROGRESS["nodes"]/PROGRESS["total"])) #If current node is more than total, we need to update the cache elif not PROGRESS["cache_update"]: PROGRESS["cache_update"] = True PROGRESS["env"].Replace(PROGRESS = "[??%]") #If cache flag is on, update cache file else: update_progress_cache() def progress_settings(env, interval): load_progress_cache() PROGRESS["interval"] = interval PROGRESS["env"] = env PROGRESS["total"] = get_progress_total() Progress(progress_function, interval=PROGRESS["interval"]) 

In SConstruct :

 if GetOption("progress"): custom_utils.progress_settings(env, 5) 

Output Example:

 user@linuxbox :~/project1$ scons -j4 --progress scons: Reading SConscript files ... scons: done reading SConscript files. scons: Building targets ... [??%] Compiling (shared): src/ExpressRuntimeException.os [??%] Compiling (shared): src/Application.os [??%] Compiling (shared): src/Callback.os [??%] Compiling (shared): src/Response.os [??%] Compiling (shared): src/Router.os [??%] Compiling (shared): src/Route.os [??%] Compiling (shared): src/Regex.os [??%] Compiling (shared): src/tools.os [??%] Linking (shared): libexpresscpp.so scons: done building targets. user@linuxbox :~/project1$ scons -c scons: Reading SConscript files ... scons: done reading SConscript files. scons: Cleaning targets ... Removed src/ExpressRuntimeException.os Removed src/Application.os Removed src/Callback.os Removed src/Response.os Removed src/Router.os Removed src/Route.os Removed src/Regex.os Removed src/tools.os Removed libexpresscpp.so scons: done cleaning targets. user@linuxbox :~/project1$ scons -j4 --progress scons: Reading SConscript files ... scons: done reading SConscript files. scons: Building targets ... [14%] Compiling (shared): src/ExpressRuntimeException.os [51%] Compiling (shared): src/Application.os [59%] Compiling (shared): src/Callback.os [66%] Compiling (shared): src/Response.os [74%] Compiling (shared): src/Router.os [81%] Compiling (shared): src/Route.os [88%] Compiling (shared): src/Regex.os [96%] Compiling (shared): src/tools.os [100%] Linking (shared): libexpresscpp.so scons: done building targets. 

Answer 3

I liked that RPGilespie answered a lot and played with it for our software package with ~ 4500 nodes. I made friends a little and thought I would send it if this helps anyone. The main differences:

1.) I added a command to run at the end of the job to write the total number of node to a file based on this message . This saves time to constantly hit the file system as a function of progress.

2.) Because of 1.) it updates the file every time scons is started (not only when it considers that the number of nodes has exceeded the previous maximum). This is useful if you are working with one part of the tree (i.e., with fewer nodes) and constantly rebuilding that particular part.

3.) I put the carriage back before updating the screen so that he could write it directly from progress_function, and not to set a variable for a later one. This has the added benefit of showing a counter update while it scans the mostly constructed tree at the beginning.

As a side note, I set the refresh interval to only 1 here. The reason is that sometimes I noticed that the number of nodes associated with one build command was less than the interval, causing it not to print a counter for this line. With the above change in how the counter is written, I did not notice a noticeable slowdown due to this. YMMV.

At the end of my SConstruct file, I put the following:

 screen = open('/dev/tty', 'w') node_count = 0 node_count_max = 0 node_count_interval = 1 node_count_fname = str(env.Dir('#')) + '/.scons_node_count' def progress_function(node): global node_count, node_count_max, node_count_interval, node_count_fname node_count += node_count_interval if node_count > node_count_max: node_count_max = 0 if node_count_max>0 : screen.write('\r[%3d%%] ' % (node_count*100/node_count_max)) screen.flush() def progress_finish(target, source, env): global node_count with open(node_count_fname, 'w') as f: f.write('%d\n' % node_count) try: with open(node_count_fname) as f: node_count_max = int(f.readline()) except: pass Progress(progress_function, interval=node_count_interval) progress_finish_command = Command('progress_finish', [], progress_finish) Depends(progress_finish_command, BUILD_TARGETS) if 'progress_finish' not in BUILD_TARGETS: BUILD_TARGETS.append('progress_finish') 

Output Example:

 [ 0%] Installing [/Users/davidl/HallD/builds/sim-recon/Darwin_macosx10.11-x86_64-llvm8.0.0/bin/MakeEventWriterROOT.pl] [ 0%] Installing [/Users/davidl/HallD/builds/sim-recon/Darwin_macosx10.11-x86_64-llvm8.0.0/bin/MakeReactionPlugin.pl] [ 0%] Compiling [programs/Simulation/genr8/genkin.c] [ 0%] Compiling [programs/Simulation/genr8/genr8.c] [ 3%] Compiling [programs/Utilities/hddm/hddm-cpp.cpp] [ 3%] Compiling [programs/Utilities/hddm/XString.cpp] [ 3%] Compiling [programs/Utilities/hddm/XParsers.cpp] [ 3%] Compiling [programs/Utilities/hddm/md5.c] [ 4%] Compiling [external/xstream/src/base64.cpp] [ 4%] Compiling [external/xstream/src/bz.cpp] [ 4%] Compiling [external/xstream/src/common.cpp] [ 4%] Compiling [external/xstream/src/dater.cpp] [ 4%] Linking [.Darwin_macosx10.11-x86_64-llvm8.0.0/programs/Simulation/genr8/genr8] [ 4%] Installing [/Users/davidl/HallD/builds/sim-recon/Darwin_macosx10.11-x86_64-llvm8.0.0/bin/genr8] [ 4%] Compiling [external/xstream/src/debug.cpp] [ 4%] Compiling [external/xstream/src/digest.cpp] ... 

For completeness, here are my COMSTR definitions:

 env.Replace( CCCOMSTR = "Compiling [$SOURCE]", CXXCOMSTR = "Compiling [$SOURCE]", FORTRANPPCOMSTR = "Compiling [$SOURCE]", FORTRANCOMSTR = "Compiling [$SOURCE]", SHCCCOMSTR = "Compiling [$SOURCE]", SHCXXCOMSTR = "Compiling [$SOURCE]", LINKCOMSTR = "Linking [$TARGET]", SHLINKCOMSTR = "Linking [$TARGET]", INSTALLSTR = "Installing [$TARGET]", ARCOMSTR = "Archiving [$TARGET]", RANLIBCOMSTR = "Ranlib [$TARGET]")