How do argparse (and legacy optparse) react to pressing the tab key after the python program name in bash?

Question

I tested optcomplete while working with the optparse module. His example is a simple file, so I could make it work. I also tested it with the argparse module, as the previous one is deprecated. But I really don't understand how and by whom the python program is called on tabs. I suspect that bash along with the shebang line and the argparse (or optparse ) module are somehow involved. I tried to figure this out (now I will read the source code).

I have a slightly more complex program structure that includes a wrapper around a piece of code that processes arguments. Its an argparse.ArgumentParser() instance and calls add_argument() - which are reclassified to another intermediate module to avoid code duplication, and the wrapper around the called one is inside the function.

I want to understand how this tab completion works between bash and python (or, in any case, any other interpreter like perl ).

NOTE. I have a fair understanding of bash completion (which I only found out now), and I think I understand bash (only) user-defined completion.

NOTE. I read other similar SO questions and no one answered this question.

Edit: Here is the bash function.
I already understood how the python module learns about the words entered on the command line by reading os.environ variable values

 $COMP_WORDS $COMP_CWORD $COMP_LINE $COMP_POINT $COMPREPLY 

These variables have meaning only when tabs are pressed. My question is how the python module starts?

Answer 1

To understand what is going on here, let's see what this bash function does:

 COMPREPLY=( $( \ COMP_LINE=$COMP_LINE COMP_POINT=$COMP_POINT \ COMP_WORDS="${COMP_WORDS[*]}" COMP_CWORD=$COMP_CWORD \ OPTPARSE_AUTO_COMPLETE=1 $1 ) ) 

See $1 at the end? This means that it actually calls the Python file that we want to execute, with the special environment variables set! To keep track of what is happening, let's prepare a little script to capture what optcomplete.autocomplete does:

 #!/usr/bin/env python2 import os, sys import optparse, optcomplete from cStringIO import StringIO if __name__ == '__main__': parser = optparse.OptionParser() parser.add_option('-s', '--simple', action='store_true', help="Simple really simple option without argument.") parser.add_option('-o', '--output', action='store', help="Option that requires an argument.") opt = parser.add_option('-p', '--script', action='store', help="Option that takes python scripts args only.") opt.completer = optcomplete.RegexCompleter('.*\.py') # debug env variables sys.stderr.write("\ncalled with args: %s\n" % repr(sys.argv)) for k, v in sorted(os.environ.iteritems()): sys.stderr.write(" %s: %s\n" % (k, v)) # setup capturing the actions of `optcomplete.autocomplete` def fake_exit(i): sys.stderr.write("autocomplete tried to exit with status %d\n" % i) sys.stdout = StringIO() sys.exit = fake_exit # Support completion for the command-line of this script. optcomplete.autocomplete(parser, ['.*\.tar.*']) sys.stderr.write("autocomplete tried to write to STDOUT:\n") sys.stderr.write(sys.stdout.getvalue()) sys.stderr.write("\n") opts, args = parser.parse_args() 

This gives us the following when we try to autofill it:

 $ ./test.py [tab] called with args: ['./test.py'] ... COMP_CWORD: 1 COMP_LINE: ./test.py COMP_POINT: 10 COMP_WORDS: ./test.py ... OPTPARSE_AUTO_COMPLETE: 1 ... autocomplete tried to exit with status 1 autocomplete tried to write to STDOUT: -o -h -s -p --script --simple --help --output 

So optcomplete.autocomplete just reads the environment, prepares the matches, writes them to STDOUT and completes. The result -o -h -s -p --script --simple --help --output then placed into the bash array ( COMPREPLY=( ... ) ) and returned to bash to present the choice to the user. There was no magic :)