How to load all modules in a folder?
-
21-08-2019 - |
Question
Could someone provide me with a good way of importing a whole directory of modules?
I have a structure like this:
/Foo
bar.py
spam.py
eggs.py
I tried just converting it to a package by adding __init__.py
and doing from Foo import *
but it didn't work the way I had hoped.
Solution
List all python (.py
) files in the current folder and put them as __all__
variable in __init__.py
from os.path import dirname, basename, isfile, join
import glob
modules = glob.glob(join(dirname(__file__), "*.py"))
__all__ = [ basename(f)[:-3] for f in modules if isfile(f) and not f.endswith('__init__.py')]
OTHER TIPS
Add the __all__
Variable to __init__.py
containing:
__all__ = ["bar", "spam", "eggs"]
Update: Today you probably want to use importlib
instead.
Make the Foo directory a package by adding an __init__.py
. In that __init__.py
add:
import bar
import eggs
import spam
Since you want it dynamic (which may or may not be a good idea), list all py-files with list dir and import them with something like this:
import os
for module in os.listdir(os.path.dirname(__file__)):
if module == '__init__.py' or module[-3:] != '.py':
continue
__import__(module[:-3], locals(), globals())
del module
Then, from your code do this:
import Foo
You can now access the modules with
Foo.bar
Foo.eggs
Foo.spam
etc. from Foo import * is not a good idea for several reasons, including name clashes and making it hard to analyze the code.
Expanding on Mihail's answer, I believe the non-hackish way (as in, not handling the file paths directly) is the following:
- create an empty
__init__.py
file underFoo/
- Execute
import pkgutil
import sys
def load_all_modules_from_dir(dirname):
for importer, package_name, _ in pkgutil.iter_modules([dirname]):
full_package_name = '%s.%s' % (dirname, package_name)
if full_package_name not in sys.modules:
module = importer.find_module(package_name
).load_module(full_package_name)
print module
load_all_modules_from_dir('Foo')
You'll get:
<module 'Foo.bar' from '/home/.../Foo/bar.pyc'>
<module 'Foo.spam' from '/home/.../Foo/spam.pyc'>
Python, include all files under a directory:
For newbies who just can't get it to work who need their hands held.
Make a folder /home/el/foo and make a file
main.py
under /home/el/foo Put this code in there:from hellokitty import * spam.spamfunc() ham.hamfunc()
Make a directory
/home/el/foo/hellokitty
Make a file
__init__.py
under/home/el/foo/hellokitty
and put this code in there:__all__ = ["spam", "ham"]
Make two python files:
spam.py
andham.py
under/home/el/foo/hellokitty
Define a function inside spam.py:
def spamfunc(): print "Spammity spam"
Define a function inside ham.py:
def hamfunc(): print "Upgrade from baloney"
Run it:
el@apollo:/home/el/foo$ python main.py spammity spam Upgrade from baloney
I got tired of this problem myself, so I wrote a package called automodinit to fix it. You can get it from http://pypi.python.org/pypi/automodinit/.
Usage is like this:
- Include the
automodinit
package into yoursetup.py
dependencies. - Replace all __init__.py files like this:
__all__ = ["I will get rewritten"] # Don't modify the line above, or this line! import automodinit automodinit.automodinit(__name__, __file__, globals()) del automodinit # Anything else you want can go after here, it won't get modified.
That's it! From now on importing a module will set __all__ to a list of .py[co] files in the module and will also import each of those files as though you had typed:
for x in __all__: import x
Therefore the effect of "from M import *" matches exactly "import M".
automodinit
is happy running from inside ZIP archives and is therefore ZIP safe.
Niall
I know I'm updating a quite old post, and I tried using automodinit
, but found out it's setup process is broken for python3. So, based on Luca's answer, I came up with a simpler answer - which might not work with .zip - to this issue, so I figured I should share it here:
within the __init__.py
module from yourpackage
:
#!/usr/bin/env python
import os, pkgutil
__all__ = list(module for _, module, _ in pkgutil.iter_modules([os.path.dirname(__file__)]))
and within another package below yourpackage
:
from yourpackage import *
Then you'll have all the modules that are placed within the package loaded, and if you write a new module, it'll be automagically imported as well. Of course, use that kind of things with care, with great powers comes great responsibilities.
import pkgutil
__path__ = pkgutil.extend_path(__path__, __name__)
for imp, module, ispackage in pkgutil.walk_packages(path=__path__, prefix=__name__+'.'):
__import__(module)
I have also encountered this problem and this was my solution:
import os
def loadImports(path):
files = os.listdir(path)
imps = []
for i in range(len(files)):
name = files[i].split('.')
if len(name) > 1:
if name[1] == 'py' and name[0] != '__init__':
name = name[0]
imps.append(name)
file = open(path+'__init__.py','w')
toWrite = '__all__ = '+str(imps)
file.write(toWrite)
file.close()
This function creates a file (in the provided folder) named __init__.py
, which contains an __all__
variable that holds every module in the folder.
For example, I have a folder named Test
which contains:
Foo.py
Bar.py
So in the script I want the modules to be imported into I will write:
loadImports('Test/')
from Test import *
This will import everything from Test
and the __init__.py
file in Test
will now contain:
__all__ = ['Foo','Bar']
Anurag's example with a couple of corrections:
import os, glob
modules = glob.glob(os.path.join(os.path.dirname(__file__), "*.py"))
__all__ = [os.path.basename(f)[:-3] for f in modules if not f.endswith("__init__.py")]
Anurag Uniyal answer with suggested improvements!
#!/usr/bin/python
# -*- encoding: utf-8 -*-
import os
import glob
all_list = list()
for f in glob.glob(os.path.dirname(__file__)+"/*.py"):
if os.path.isfile(f) and not os.path.basename(f).startswith('_'):
all_list.append(os.path.basename(f)[:-3])
__all__ = all_list
See that your __init__.py
defines __all__
. The modules - packages doc says
The
__init__.py
files are required to make Python treat the directories as containing packages; this is done to prevent directories with a common name, such as string, from unintentionally hiding valid modules that occur later on the module search path. In the simplest case,__init__.py
can just be an empty file, but it can also execute initialization code for the package or set the__all__
variable, described later....
The only solution is for the package author to provide an explicit index of the package. The import statement uses the following convention: if a package’s
__init__.py
code defines a list named__all__
, it is taken to be the list of module names that should be imported when from package import * is encountered. It is up to the package author to keep this list up-to-date when a new version of the package is released. Package authors may also decide not to support it, if they don’t see a use for importing * from their package. For example, the filesounds/effects/__init__.py
could contain the following code:
__all__ = ["echo", "surround", "reverse"]
This would mean that
from sound.effects import *
would import the three named submodules of the sound package.
This is the best way i've found so far:
from os.path import dirname, join, isdir, abspath, basename
from glob import glob
pwd = dirname(__file__)
for x in glob(join(pwd, '*.py')):
if not x.startswith('__'):
__import__(basename(x)[:-3], globals(), locals())
Look at the pkgutil module from the standard library. It will let you do exactly what you want as long as you have an __init__.py
file in the directory. The __init__.py
file can be empty.
I've created a module for that, which doesn't rely on __init__.py
(or any other auxiliary file) and makes me type only the following two lines:
import importdir
importdir.do("Foo", globals())
Feel free to re-use or contribute: http://gitlab.com/aurelien-lourot/importdir
Just import them by importlib and add them to __all__
(add
action is optional) in recurse in the __init__.py
of package.
/Foo
bar.py
spam.py
eggs.py
__init__.py
# __init__.py
import os
import importlib
pyfile_extes = ['py', ]
__all__ = [importlib.import_module('.%s' % filename, __package__) for filename in [os.path.splitext(i)[0] for i in os.listdir(os.path.dirname(__file__)) if os.path.splitext(i)[1] in pyfile_extes] if not filename.startswith('__')]
del os, importlib, pyfile_extes