Scrapyd: How to specify libs and common folders that deployed projects can use?

Question

Scrapyd is service where we can eggify deploy our projects. However I am facing a problem. I have a Project named MyScrapers whose spider classes uses an import statement as follows:

from mylibs.common.my_base_spider import MyBaseSpider

The path to my_base_spider is /home/myprojectset/mylibs/common/my_base_spider

While setting environment variable PYTHONPATH=$HOME/myprojectset/, I am able to run MyScrapers using scrapy command: scrapy crawl MyScrapers.

But when I use scrapyd for deploying MyScrapers by following command: scrapy deploy scrapyd2 -p MyScrapers, I get the following error:

Server response (200): {"status": "error", "message": "ImportError: No module named mylibs.common.my_base_spider"}

Please tell how to make deployed project to use these libs?

Answer 1

I found the answer by adding mylibs to site-packages of python by using setup.py inside mylib folder. That way I could import everything inside mylib in my projects. Actually mylibs were way outside from the location where setup.py of my deploy-able project is present. setup.py looks for packages on same level and inside the folders where it is located.

Answer 2

You need to edit your setup.py and/or MANIFEST.in to declare that it has a mylibs package dependency.

See the distutils documentation for more information.