-m option to run a module as a script

[ncoghlan]

BPO	1035498
Nosy	@arigo, @theller, @rhettinger, @ncoghlan
Files	run_module_6.diff: Should fix Windows compile bug run_module_7.diff: Sets argv[0] correctly run_module_docs.diff: Update tutorial section on Invoking the Interpreter run_module_complete.diff: All code and documentation changes

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2004-10-08.22:53:28.000>
created_at = <Date 2004-09-27.14:22:45.000>
labels = ['interpreter-core']
title = '-m option to run a module as a script'
updated_at = <Date 2004-10-08.22:53:28.000>
user = 'https://github.com/ncoghlan'

bugs.python.org fields:

activity = <Date 2004-10-08.22:53:28.000>
actor = 'ncoghlan'
assignee = 'none'
closed = True
closed_date = None
closer = None
components = ['Interpreter Core']
creation = <Date 2004-09-27.14:22:45.000>
creator = 'ncoghlan'
dependencies = []
files = ['6268', '6269', '6270', '6271']
hgrepos = []
issue_num = 1035498
keywords = ['patch']
message_count = 22.0
messages = ['46957', '46958', '46959', '46960', '46961', '46962', '46963', '46964', '46965', '46966', '46967', '46968', '46969', '46970', '46971', '46972', '46973', '46974', '46975', '46976', '46977', '46978']
nosy_count = 4.0
nosy_names = ['arigo', 'theller', 'rhettinger', 'ncoghlan']
pr_nums = []
priority = 'normal'
resolution = 'accepted'
stage = None
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue1035498'
versions = ['Python 2.4']

[ncoghlan]

Implements the '-m' option recently discussed on
python-dev.

Runs a module 'as if' it had been invoked from the
command line.

E.g., for a build directory, the following two commands
are equivalent:
./python -m pdb
./python Lib/pdb.py

Note that neither of these does the same thing as
"./python -c "import pdb". (In this latter case, any
"if __name__ == "__main__" code would not be executed,
whereas it will be invoked in the first two cases).

Given the vagaries of sys.path, this is quite handy -
if a module can be imported, it can be easily used as a
script too. Not that all modules will necessarily do
anything _interesting_ when used as a script. . .

The current implementation makes changes to main.c
(handle the new argument), pythonrun.[ch] (new flavour
of PyRun_) and import.[ch] (expose a
PyImport_FindModule C API)

Current limitations / to-do if this is pursued:

• print sensible errors for non-Python files (e.g.
  hotshot)
• allow 'dotted' names
• decide what to do with packages (e.g. run __init__.py)
• check usage messages in main.c are all still under
  512 characters (I assume that limit was set for a reason)
• the 1K limit on absolute file paths seems rather
  arbitrary.
• this is code written in the wee hours of the
  morning, so it probably has a few other lurking 'features'.

[ncoghlan]

Logged In: YES
user_id=1038590

New version of patch attached (run_module_2.diff).

This version allows modules within packages to be executed.
It does this *without* loading the packages first (just as
if the script were specified directly on the command line).

It's also a little more responsive when it comes to error
message, and I've confirmed that the usage message
components are now under 512 bytes each.

The FindModule method is now marked as private (i.e. with
leading underscore).

The limit on file paths in pythonrun.c is now taken from
osdefs.h (the same limit that import.c already uses)

The only bug I know of is that it misbehaves if you supply a
name like "a.b.module.d". You will get an ImportError saying
that d was not found, and the file pointer for 'module'
won't get closed properly.

[ncoghlan]

Logged In: YES
user_id=1038590

Barry, kicking this in your direction solely because you
were one of the people to comment on it on py-dev. Feel free
to say 'too busy' :)

[ncoghlan]

Logged In: YES
user_id=1038590

New version (run_module_3.diff)

Takes more account of the file description reported by
PyImport. Explictly restricts itself to PY_SOURCE and
PY_COMPILED scripts.

Behaves correctly when a non-package is encountered while
parsing a dotted name (closes the file descriptor and
reports an error).

I also realised that the '-m' option doesn't currently
comprehend "__path__" variables in packages. So maybe we
*should* be loading the packages as we find them, and then
querying them for their __path__ variable.

[ncoghlan]

Logged In: YES
user_id=1038590

New version (#4)

Trivial fix - tell PyRun_SimpleFile to close the file when
it's done with it.

[ncoghlan]

Logged In: YES
user_id=1038590

Relevant documentation locations that I can find:
Tutorial section 2.1 (Invoking the Interpreter)
Add a very brief description of the '-m' option

Python/C API section 2 (The Very High Level Layer)
Document PyRun_SimpleModule

What's New sections 12 & 13
I'll do this as a separate patch from the previous two,
as I can't recall if Andrew prefers to write these himself.

[ncoghlan]

Logged In: YES
user_id=1038590

5th version of patch attached

• restricts option to top-level modules only. The correct
  semantics for a module inside a package just aren't clear
  enough at this stage. Support for dotted names can be added
  later if this patch gets accepted, and it still seems like a
  good idea.

• adds a missing check from main.c (if we got the module
  option, don't try to open the file).

[ncoghlan]

Logged In: YES
user_id=1038590

Doc patch attached. Take it with a grain of salt, since this
is the first real doc patch I've made.

This doesn't cover anything for 'What's New' - that can wait
until the patch gets accepted (if it does).

[ncoghlan]

Logged In: YES
user_id=1038590

Raymond,

Passing to you to see what you think of the code, rather
than just the feature in the abstract.

Guido settled for not vetoing it, but the discussion with
him allowed me to articulate why I think this is a
worthwhile feature.
(Details at
http://mail.python.org/pipermail/python-dev/2004-October/049236.html)

Extension to allow packages and zip imports can wait until
there is any demand for either feature. I don't really
expect to see any, since I think the main uses will be
"python -m pdb", "python -m profile" and "python -m timeit".

[rhettinger]

Logged In: YES
user_id=80475

Armin, do you have time for this one. I'm buried for a few
days.

[arigo]

Logged In: YES
user_id=4771

Sorry, I didn't follow any of this discussion, neither here
nor in python-dev, so I cannot really comment.

[rhettinger]

Logged In: YES
user_id=80475

The patch does not compile on windows. AFAICT, importdl.h
is the culprit because it loads windows.h for a second time
resulting a double redinition of BYTE deep in the bowls of
include files.

Wish I had time to help you on this one, but I don't.

[ncoghlan]

Logged In: YES
user_id=1038590

New version attached which should fix the Windows compile
bug (pythonrun.c does the right thing by undefining BYTE
before including windows.h - the inclusion of 'importdl.h'
has been moved to be after that 'fixed' include).

Unfortunately, I won't get access to a Windows build machine
until after the beta 1 deadline, or I'd check it myself :(

[rhettinger]

Logged In: YES
user_id=80475

Put the test directory in PYTHONPATH and run the following
to see if you get the same results:

python -tt -m regrtest  -u network
python -tt lib\test\regrtest.py -u network

[ncoghlan]

Logged In: YES
user_id=1038590

Nice catch. . .

I figured out what the problem is - regrtest.py uses
sys.argv[0] to delete it's own directory from sys.path. And
-m currently sets sys.argv[0] to just 'regrtest', instead of
the full path to the module.

Obviously, that is going to have to change. Either
PyRun_SImpleModule will have to do it, or I'll get rid of
that function all together, and put the 'module-finding'
logic directly in main.c.

[ncoghlan]

Logged In: YES
user_id=1038590

New version (7) attached which does the right thing for
"python -tt -m regrtest -u network". (I really like that
test case btw - very concise)

This one sets argv[0] correctly, but was initially a little
ungraceful in some failure modes (such as "python -m
parser"). I had to add a new private API function to
import.h (_PyImport_IsScript) to clean that up.

Examples of the error handling in this version:

[...@localhost src]$ ./python -m foo
./python: module foo not found
[...@localhost src]$ ./python -m sys
./python: module sys has no associated file
[...@localhost src]$ ./python -m parser
./python: module parser not usable as script

(/home/.../dev/python/dist/src/build/lib.linux-i686-2.4/parser.so)

[ncoghlan]

Logged In: YES
user_id=1038590

New version of doc patch which changes the tutorial only
(since PyRun_SimpleModule is no longer part of the patch).

[rhettinger]

Logged In: YES
user_id=80475

Please put in a single, combined documentation patch.

Be sure to spell-check and run: python -m texcheck tut.tex

[ncoghlan]

Logged In: YES
user_id=1038590

texcheck complains about the Unicode in the tutorial even on
a clean checkout. The line numbers (> 900) are well past the
point where I'm editing. (It turns out editing the the tex
files works a lot better with xemacs than with the KDE
editors - no spurious edits this time)

I'm not sure what, if anything, I need to do for "what's
new", but I've uploaded a combined code & documentation
patch that implements version 7 of the -m behaviour, and
makes relevant changes to section 2 of the tutorial.

[rhettinger]

Logged In: YES
user_id=80475

Accepted and applied.

Checking in Doc/tut/tut.tex;
/cvsroot/python/python/dist/src/Doc/tut/tut.tex,v <-- tut.tex
new revision: 1.254; previous revision: 1.253
done
Checking in Include/import.h;
/cvsroot/python/python/dist/src/Include/import.h,v <--
import.h
new revision: 2.30; previous revision: 2.29
done
Checking in Misc/NEWS;
/cvsroot/python/python/dist/src/Misc/NEWS,v <-- NEWS
new revision: 1.1154; previous revision: 1.1153
done
Checking in Modules/main.c;
/cvsroot/python/python/dist/src/Modules/main.c,v <-- main.c
new revision: 1.84; previous revision: 1.83
done
Checking in Python/import.c;
/cvsroot/python/python/dist/src/Python/import.c,v <-- import.c
new revision: 2.240; previous revision: 2.239
cvs diff: [06:46:25] waiting for rhettinger's lock in
/cvsroot/python/python/dis
t/src/Doc/tut
done

[theller]

Logged In: YES
user_id=11105

Regarding support for packages, here's a use case:

python -m pychecker.checker mymodule.py

(plus my own tools which live inside packages)

[ncoghlan]

Logged In: YES
user_id=1038590

I've opened a new tracker item to cover enhancement to
support packages.
http://sourceforge.net/support/tracker.php?aid=1043356