What is the best way to allow overriding configuration options on the command line in Python?

Question

I have a Python application that needs quite a few (~ 30) configuration options. So far, I have used the OptionParser class to define default values ​​in the application itself, with the ability to change individual parameters on the command line when the application is called.

Now I would like to use the “correct” configuration files, for example, from the ConfigParser class. At the same time, users should still be able to change individual options on the command line.

I was wondering if there is a way to combine the two steps, for example, using optparse (or a newer argparse) to process command line parameters, but reading the default values ​​from the configuration file in ConfigParse syntax.

Any ideas how to do this easily? I don’t really like to manually call ConfigParse, and then manually set all the default values ​​for all opto to the corresponding values ​​...

Answer 1

I just discovered that you can do this with argparse.ArgumentParser.parse_known_args() . Start by using parse_known_args() to parse the configuration file from the command line, then read it with ConfigParser and set the default values, and then parse_args() rest of the parameters with parse_args() . This will allow you to have a default value, override it with the configuration file, and then override it with the command line option. For example:.

By default without user input:

 $ ./argparse-partial.py Option is "default" 

By default from the configuration file:

 $ cat argparse-partial.config [Defaults] option=Hello world! $ ./argparse-partial.py -c argparse-partial.config Option is "Hello world!" 

By default, from a configuration file overridden by the command line:

 $ ./argparse-partial.py -c argparse-partial.config --option override Option is "override" 

argprase -partial.py follows. It's a little tricky to handle -h for proper handling.

 import argparse import ConfigParser import sys def main(argv=None): # Do argv default this way, as doing it in the functional # declaration sets it at compile time. if argv is None: argv = sys.argv # Parse any conf_file specification # We make this parser with add_help=False so that # it doesn't parse -h and print help. conf_parser = argparse.ArgumentParser( description=__doc__, # printed with -h/--help # Don't mess with format of description formatter_class=argparse.RawDescriptionHelpFormatter, # Turn off help, so we print all options in response to -h add_help=False ) conf_parser.add_argument("-c", "--conf_file", help="Specify config file", metavar="FILE") args, remaining_argv = conf_parser.parse_known_args() defaults = { "option":"default" } if args.conf_file: config = ConfigParser.SafeConfigParser() config.read([args.conf_file]) defaults.update(dict(config.items("Defaults"))) # Parse rest of arguments # Don't suppress add_help here so it will handle -h parser = argparse.ArgumentParser( # Inherit options from config_parser parents=[conf_parser] ) parser.set_defaults(**defaults) parser.add_argument("--option") args = parser.parse_args(remaining_argv) print "Option is \"{}\"".format(args.option) return(0) if __name__ == "__main__": sys.exit(main()) 

Answer 2

Check out ConfigArgParse is the new PyPI ( open source ) package that serves as a non-replacement for argparse with additional support for configuration files and environment variables.

Answer 3

I use ConfigParser and argparse with subcommands to handle such tasks. Important line in the code below:

 subp.set_defaults(**dict(conffile.items(subn))) 

This will set the default values ​​of the subcommand (from argparse) to the values ​​in the configuration file section.

Below is a more complete example:

 ####### content of example.cfg: # [sub1] # verbosity=10 # gggg=3.5 # [sub2] # host=localhost import ConfigParser import argparse parser = argparse.ArgumentParser() subparsers = parser.add_subparsers() parser_sub1 = subparsers.add_parser('sub1') parser_sub1.add_argument('-V','--verbosity', type=int, dest='verbosity') parser_sub1.add_argument('-G', type=float, dest='gggg') parser_sub2 = subparsers.add_parser('sub2') parser_sub2.add_argument('-H','--host', dest='host') conffile = ConfigParser.SafeConfigParser() conffile.read('example.cfg') for subp, subn in ((parser_sub1, "sub1"), (parser_sub2, "sub2")): subp.set_defaults(**dict(conffile.items(subn))) print parser.parse_args(['sub1',]) # Namespace(gggg=3.5, verbosity=10) print parser.parse_args(['sub1', '-V', '20']) # Namespace(gggg=3.5, verbosity=20) print parser.parse_args(['sub1', '-V', '20', '-G','42']) # Namespace(gggg=42.0, verbosity=20) print parser.parse_args(['sub2', '-H', 'www.example.com']) # Namespace(host='www.example.com') print parser.parse_args(['sub2',]) # Namespace(host='localhost') 

Answer 4

I can’t say that this is the best way, but I have the OptionParser class that I did that does just that - acts like optparse.OptionParser with default values ​​coming from the section of the configuration file. You can get it ...

 class OptionParser(optparse.OptionParser): def __init__(self, **kwargs): import sys import os config_file = kwargs.pop('config_file', os.path.splitext(os.path.basename(sys.argv[0]))[0] + '.config') self.config_section = kwargs.pop('config_section', 'OPTIONS') self.configParser = ConfigParser() self.configParser.read(config_file) optparse.OptionParser.__init__(self, **kwargs) def add_option(self, *args, **kwargs): option = optparse.OptionParser.add_option(self, *args, **kwargs) name = option.get_opt_string() if name.startswith('--'): name = name[2:] if self.configParser.has_option(self.config_section, name): self.set_default(name, self.configParser.get(self.config_section, name)) 

Remember to view the source . Tests are in the catalog for sisters.

Answer 5

Try this way

 # encoding: utf-8 import imp import argparse class LoadConfigAction(argparse._StoreAction): NIL = object() def __init__(self, option_strings, dest, **kwargs): super(self.__class__, self).__init__(option_strings, dest) self.help = "Load configuration from file" def __call__(self, parser, namespace, values, option_string=None): super(LoadConfigAction, self).__call__(parser, namespace, values, option_string) config = imp.load_source('config', values) for key in (set(map(lambda x: x.dest, parser._actions)) & set(dir(config))): setattr(namespace, key, getattr(config, key)) 

Use it:

 parser.add_argument("-C", "--config", action=LoadConfigAction) parser.add_argument("-H", "--host", dest="host") 

And create a config example:

 # Example config: /etc/myservice.conf import os host = os.getenv("HOST_NAME", "localhost") 

Answer 6

Update: this answer still has problems; for example, it cannot handle required arguments and requires a clumsy configuration syntax. Instead, ConfigArgParse seems to be exactly what this question asks for, and is a transparent replacement.

One problem with the current one is that it will not throw an error if the arguments in the configuration file are invalid. Here is a version with another drawback: you need to include the -- or - prefix in the keys.

Here is the Python code ( Gist link with MIT license):

 # Filename: main.py import argparse import configparser if __name__ == "__main__": parser = argparse.ArgumentParser() parser.add_argument('--config_file', help='config file') args, left_argv = parser.parse_known_args() if args.config_file: with open(args.config_file, 'r') as f: config = configparser.SafeConfigParser() config.read([args.config_file]) parser.add_argument('--arg1', help='argument 1') parser.add_argument('--arg2', type=int, help='argument 2') for k, v in config.items("Defaults"): parser.parse_args([str(k), str(v)], args) parser.parse_args(left_argv, args) print(args) 

Here is an example configuration file:

 # Filename: config_correct.conf [Defaults] --arg1=Hello! --arg2=3 

Now works

 > python main.py --config_file config_correct.conf --arg1 override Namespace(arg1='override', arg2=3, config_file='test_argparse.conf') 

However, if our configuration file has an error:

 # config_invalid.conf --arg1=Hello! --arg2='not an integer!' 

Running the script will result in an error if desired:

 > python main.py --config_file config_invalid.conf --arg1 override usage: test_argparse_conf.py [-h] [--config_file CONFIG_FILE] [--arg1 ARG1] [--arg2 ARG2] main.py: error: argument --arg2: invalid int value: 'not an integer!' 

The main disadvantage is that it uses parser.parse_args somewhat parser.parse_args to get error checking from ArgumentParser, but I don’t know of any alternatives to this.

Answer 7

fromfile_prefix_chars

It may not be the perfect API, but you should be aware of this. main.py :

 #!/usr/bin/env python3 import argparse parser = argparse.ArgumentParser(fromfile_prefix_chars='@') parser.add_argument('-a', default=13) parser.add_argument('-b', default=42) print(parser.parse_args()) 

Then:

 $ printf -- '-a\n1\nb\n2\n' > opts.txt $ ./main.py Namespace(a=13, b=42) $ ./main.py @opts.txt Namespace(a='1', b='2') $ ./main.py @opts.txt -a 3 -b 4 Namespace(a='3', b='4') $ ./main.py -a 3 -b 4 @opts.txt Namespace(a='1', b='2') 

Documentation: https://docs.python.org/3.6/library/argparse.html#fromfile-prefix-chars

Tested on Python 3.6.5, Ubuntu 18.04.

Answer 8

You can use chainmap

 A ChainMap groups multiple dicts or other mappings together to create a single, updateable view. If no maps are specified, a single empty dictionary is provided so that a new chain always has at least one mapping. 

You can combine values ​​from the command line, environment variables, configuration file, and in case this value is missing, define a default value.

 import os from collections import ChainMap, defaultdict options = ChainMap(command_line_options, os.environ, config_file_options, defaultdict(lambda: 'default-value')) value = options['optname'] value2 = options['other-option'] print(value, value2) 'optvalue', 'default-value'