Re: Python bytecode compatibility between interpreter versions

From: Roger Binns (rogerb_at_rogerbinns.com)
Date: 03/22/04 

Date: Sun, 21 Mar 2004 23:50:06 -0800

> Is it really the case that it isn't even recommended to distribute one's
> application as .pyc files?
>
> In commercial apps for which one might not want to distribute the source
> for, it would be really nice if one can count on just requiring the end-user
> to install a version of Python (but not extra extension modules anymore) and
> then distribute the app as a bunch of .pyc's packaged in a .zip file and
> use the PYTHONPATH method you showed. Just remember to provide a set of
> .pyc's for each version of Python you support.

In the Java world, the JVM is a seperate installable component.
Some installers like ZeroG will install them for you. In general
all Java bytecode is expected (but not required) to share the
main system JVM.

In Python, if you want to distribute an application, and you
don't want the user to have to install Python or any extensions,
you use one of the installer generators out there. They gather
up the interpretter, your code and extension libraries and dump
them all into one directory. They then include a stub that makes
them all work together. It especially has no reliance on any
pre-installed interpretter.

If you want to see this in action, try my project at
http://bitpim.sf.net The downloads give you a program that runs
on Windows, Mac and Linux without any prerequisites. If you
examine the install directory, you will find the find Python
interpretter DLL or binary, all needed extension shared libraries
and the program code (often as .pyc files in a zip archive or
appended to the main binary).

Note that the installer generator happens to know what formats
and other quirks the Python interpretter works with which is why
it does include some .pyc's.

> In Java, profiling info may not be written out to disk, but the
> bytecodes certainly are (they are what is in the class files). Which
> is roughly equivalent to what happens with .pyc files, correct?

You are pedantically correct in that they both contain bytecode, but
the level is different. *If* the JVM wrote out HotSpot compilations
and profiling information, the resulting files would be analogous to
.pyc files. They would not be *required* for execution, and they
would be automatically regenerated if necessary (eg if you changed
source files). In theory it could then execute from them without
requiring the original class files, but that would be an implementation
specific thing.

> .pyc
> files don't contain profiling info, nor was I ever aware that the Python
> VM even does run-time behaviour profiling... (Pysco may, though)

I don't know of any implementations that do, but the implementations
are free to do so. As I keep saying, .pyc files are an implementation
specific detail of one version of one implementation of the Python
interpretter. It can make as many intermediate file formats and contents
as it wants, as can any other implementation. The only requirement for
a conforming Python implementation is that it execute Python *source*,
and how it does it under the scenes is not specified.

> In practice though, there is really just one implementation of Python
> (with the standard libraries... excluding Jython) which works using the
> same bytecode mechanism on _all_ platforms, so as far as I can tell,
> it would be quite feasible to distribute applications in .pyc form.

If you distribute the exact interpretter that generated those exact
.pyc files then you will be fine in practise. That is what the various
installer generators out there do. Some examples are py2exe, cx_Freeze
and BundleBuilder which is what my project uses on Windows, Linux and
Mac respectively.

With the addition of zip imports in Python 2.3, I think things will
lead to a unification of the installer generators out there (there have
been several different ones, often limited to a subset of platforms for
historical reasons).

Roger

Relevant Pages

  • Re: Python bytecode compatibility between interpreter versions
    ... Note that I specifically mentioned .pyc files vs. java class files, ... not Java the language/environment/platform vs. Python. ... > your platform. ... Perhaps one day .pyc files can work in such a manner. ...
    (comp.lang.python)
  • Re: Any way to not create .pyc files?
    ... As long as you zipped up the package will all .pyc and .pyo files removed, Python will have no choice but to compile the files every time they are imported - unless I'm grossly mistaken Python won't put the pyc files into the zip archive, or modify any that were there already. ... Instead of the users starting the application directly, they could start a starter application that checks with the server to determine if local files need to be updated, and if so grab them and then start the main app. ...
    (comp.lang.python)
  • Re: Race condition when generating .pyc files
    ... Part of the code lives in one place, ... hosts when the module's .pyc file is generated. ... python source first. ... Inhibiting the generation of .pyc files altogether if that's even ...
    (comp.lang.python)
  • Re: bytecode non-backcompatibility
    ... > So there are currently 7 implementations or variations of the Python ... >The same can't be said for .pyc files ... One of the principles of OOP is separation of interface from ... Python language is interface. ...
    (comp.lang.python)
  • Re: Python bytecode compatibility between interpreter versions
    ... >>details of the particular version of the Python interpreter that it was ... > of .pyc files is an implementation detail of one version of the Python ... files don't contain profiling info, nor was I ever aware that the Python ...
    (comp.lang.python)