Using argparse with parameters defined in config file

I understand that and why using eval is generally considered bad practice in most cases (see e.g. here). Related questions on config files and argparse don't use types. (1, 2)

I needed to parse a number of command line arguments and decided to store the arguments, defaults and help strings in a .json file. I read the .json and add every argument to an argparse.ArgumentParser. To specify a type I need to pass a callable like float and not the string "float".

To get from the string to the callable I thought of

• using eval

• using a dict to map from string to callable

• using a if/else or switch

and decided to use to use eval to avoid hard coding all types.

I have no security concerns because the argument file is supplied by the user and the program that uses this code is targeted at scientists that will realistically want to change the file to add parameters or change defaults. (Also, it is a university project which will never be run except for grading. I handed in a version using eval.)

Is there a smart solution avoiding hardcoding all types and avoiding eval, or did I find a place where eval is a sensible choice? I was only allowed to use the standard library.

Minimal args.json:

{
"dt": {
"type": "float",
"help": "Time step size",
"default": 0.4},
"T": {
"type": "int",
"help": "Time steps between outputs",
"default": 50},
"f": {
"type": "str",
"help": "Landscape file name",
"default": "small.dat"}
}


Runnable code, put args.json above in same directory to run:

`import json
import argparse
import pprint

def setup_parser(arguments, title):

parser = argparse.ArgumentParser(description=title,
formatter_class=argparse.ArgumentDefaultsHelpFormatter)

for key, val in arguments.items():
parser.add_argument('-%s' % key,
type=eval(val["type"]),
h

The use of a json file/expression as a source for defining argparse arguments is clever, and I think sound. ipython defines many of its commandline arguments from values in a config file.

I commented that we need to be careful about what type does. It's a frequent point of confusion. It is a callable, usually a function, that takes a string, and converts it to the kind of value you want in the args namespace. Its relation to builtin types and the the Python type function is only tangential.

int and float work because those functions take a string and return a desired value (or raise an error). bool does not because it does not take string and return a True/False (try bool('False').

argparse has an undocumentated registries mechanism that maps from strings to callables. Mostly it is used for the action parameter, but can also be used with type. Basically it's a dictionary.

Here's a quick session to illustrate its use.

In [2]: parser=argparse.ArgumentParser()

parser.register is an undocumented method, used internally, but available to the user (no _):

In [3]: parser.register?
Type:       instancemethod
...
File:       /usr/local/lib/python2.7/argparse.py
Definition: parser.register(self, registry_name, value, object)
Docstring:  

It sets an entry in a dictionary that the parser owns. Notice all the familiar action values. This maps from those strings to the Action subclasses. There is only one type entry, the default case that does nothing (leaves a string unchanged).

In [4]: parser._registries
Out[4]: 
{'action': {None: argparse._StoreAction,
  'append': argparse._AppendAction,
  'append_const': argparse._AppendConstAction,
  'count': argparse._CountAction,
  'help': argparse._HelpAction,
  ...
 'type': {None: }}

So let's add some entries to the register:

In [5]: parser.register('type','float',float)
In [6]: parser.register('type','int',int)
In [7]: parser.register('type','str',None)

and create some Actions to use them:

In [8]: parser.add_argument('-i',type='int')
....
In [9]: parser.add_argument('-f',type='float')
...
In [10]: parser.add_argument('-s',type='str')
... 
ValueError: None is not callable

Oops, it didn't like my None. I was intending the None to map on to the existing None mapping in the registries, but it does not do that kind of double mapping.

I think the str() function will work. It takes a string and returns it unchanged. The argparse identity is more like a do nothing lambda x:x.

In [12]: str('astirng')
Out[12]: 'astirng'

In [13]: parser.register('type','str',str)

In [14]: parser.add_argument('-s',type='str')
Out[14]: _StoreAction(option_strings=['-s'], dest='s', nargs=None, const=None, default=None, type='str', choices=None, help=None, metavar=None)

Now let's test:

In [15]: parser.parse_args('-i 1 -f 23 -s 23'.split())
Out[15]: Namespace(f=23.0, i=1, s='23')

If you don't want to use register, I'd recommend a dictionary over eval. As another answer notes there aren't that many types, and you have more control over the translation.

A recent SO question suggests on potential difficulty with your json-argparse mapping. Not all Actions take a type parameter. Your set of parameters is fine for the defaultstoreaction. But astore_trueaction raises an error if you give it atype parameter.

https://stackoverflow.com/questions/33574270/typeerror-init-got-an-unexpected-keyword-argument-type-in-argparse

I'd prefer calling add_argument with:

parser.add_argument(*args, **kwargs)

and build up args as a list, and kwargs as a dictionary. For example instead of :

for key, val in arguments.items():
    parser.add_argument('-%s' % key,
                        type=eval(val["type"]),
                        help=val["help"],
                        default=val["default"])

use something like:

for key, val in arguments.items():
    args = ['-%s'%key]
    kwargs = {}
    kwargs.update(val)
    parser.add_argument(*args, **kwargs)

This would allow me to use json strings like

'foo': {'action': 'store_true',
'help': 'True/False values'},
'bar': {'type': 'int',
'default': 2},
'baz': {'nargs': '+',
'help': 'multiple values'}

I haven't tested this yet, but if I've got it right, it would give more flexibility. Internally argparse uses **kwargs` a lot to handle the large numbers of parameters that its methods take.

StackExchange Code Review Q#110108, answer score: 6