pythonic module organization - how to refer to files in root directory?

Question

I have my python code in a folder called "project", so my code files are in project/*.py. I want to have submodules within it, e.g.

project/code.py # where code lives
project/mymodule1  # where more code lives
project/mymodule2

each module directory has its own init.py file, e.g.

project/mymodule1/__init__.py

suppose I have a file "test.py" within mymodule1 (project/mymodule1/test.py) and I'd like to refer to something from "code", e.g. import the function "myfunc"

== project/mymodule1/test.py ==
from code import myfunc

the problem is that "code" will not be found unless the user has placed the "project/" directory in their PYTHONPATH. Is there a way to avoid this and use some kind of "relative path" to import myfunc, e.g.

from ../code import myfunc

basically, I don't want to force users of the code to alter the PYTHONPATH unless I can do it programmatically for them from within my script. I'd like it to work out of the box.

How can this be done? either solution is good: altering PYTHONPATH programmatically, or more ideally, refering to "code" using some kind of relative importing, since even though I don't know where "project/code.py" lives on the user's computer, I know where it is relative to "myfunc".

EDIT: Can someone please show proper example of intra-package import? I tried, from "mymodule1" to do:

from .. import foo

where "foo" is in code.py but it does not work. I have init.py in mymodule1, so:

project/code.py
project/mymodule1/__init__.py
project/mymodule1/module1_code.py

where module1_code.py tries to import foo, a function defined in "code.py".

EDIT: The main confusion I still have is that even after adopting the example given in response to my message, showing the project/sub1/test hierarchy, you still cannot "cd" into sub1 and do "python test.py" and have it work. The user has to be in the directory containing "project" and do "import project.sub1.test". I'd like this to work regardless of what directory the user is in. The user in this case has to execute the file "test.py", which lives in project/sub1/. So the test case is:

$ cd project/sub1
$ python test.py

which yields the error:

ValueError: Attempted relative import in non-package

how can this be fixed?

thanks.

Answer 1

This is possible in Python 2.5 and above. See the docs on Intra-Package References.

A couple of things to note:

If you intend for your users to install your package somewhere, for example using distutils or setuptools, then project will likely already be in the search path, and you can change the relative import to from project.code import ... or similar.

In the case where your users are installing your package to a non-standard directory (e.g. their home directory, someplace else that's not in sys.path, etc.), my opinion is that it leads to less confusion to instruct the user to modify PYTHONPATH rather than programmatically change sys.path.

In the case where you don't intend your users to install your code at all - for example, they will simply be untarring the source, cd'ing to the parent directory of project, and running a script - then intra-package references and relative imports will probably work fine.

EDIT: per request, here's an example:

Suppose package layout is as follows:

project/
    __init__.py (empty)
    code.py
    sub1/
        __init__.py (empty)
        test.py

Now, the contents of project/code.py are:

# code.py (module that resides in main "project" package)

def foo():
    print "this is code.foo"

And, the contents of project/sub1/test.py are:

# test.py (module that resides in "sub1" subpackage of main "project" package)

from ..code import foo
foo()

So, test.py imports the name foo from the relative path ..code, which uses intra-package references to get us back to the code.py module within the parent (that's the .. part) of the sub1.test package where we are executing from.

To test this:

shell$ (cd to directory where "project" package is located)
shell$ python
Python 2.6.1 (r261:67515, Aug  2 2010, 20:10:18) 
[GCC 4.2.1 (Apple Inc. build 5646)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import project.sub1.test
this is code.foo

Note that the double dot in the from .. import X syntax merely gets you to the parent package, but you can specify modules within that package.

In other words, from .. import X in this case is equivalent to from project import X, and thus X must either be a module in project or a class/function/name within project/__init__.py.

Thus, from ..code import X is equivalent to from project.code import X.

Answer 2

best way to avoid this is to keep all your code in a src folder or better name it on your project e.g. myproject and keep a __init__.py there, so you can do this

from myproject import code

so your folder structure would be

project
    main.py
    myproject
        __init__.py
        code.py
        module1
        module2

main.py or whatever name you give it should not have much code, it should get required module from myproject and run it e.g.

from myproject import myapp
myapp.run()

see a good article on how to arrange your python project.