Error Using Python's “Scrapy”

Question

I installed the newest version of Scrapy on Python 2.7 (Yes, I still use this version) and am getting an error while running Scrapy's tutorial script. The code I'm running is:

from scrapy.spider import BaseSpider

class DmozSpider(BaseSpider):
    name = "dmoz.org"
    allowed_domains = ["dmoz.org"]
    start_urls = [
        "http://www.dmoz.org/Computers/Programming/Languages/Python/Books/",
        "http://www.dmoz.org/Computers/Programming/Languages/Python/Resources/"
    ]

    def parse(self, response):
        filename = response.url.split("/")[-2]
        open(filename, 'wb').write(response.body)

The error is:

Traceback (most recent call last):
  File "C:/Users/Rainey/Desktop/dmoz/spiders/dmoz_spider.py", line 1, in <module>
    from scrapy.spider import BaseSpider
  File "C:\Python27\lib\site-packages\scrapy\spider.py", line 7, in <module>
    from scrapy import log
  File "C:\Python27\lib\site-packages\scrapy\log.py", line 10, in <module>
    from twisted.python import log
ImportError: No module named twisted.python

The tutorial script is on my desktop. Will this effect and should I have it in a different place?

Answer 1

The error message 'ImportError: No module named twisted.python' indicates that you are missing the twisted module.

Install twisted using a package manager like pip or download a Windows binary.

Answer 2

You did not install scrapy successfully on windows. following steps should work for you to install scrapy 0.14(latest) on windows.

1. Install lxml-2.3 using windows installer here. I tried libxml but not work.
2. Install Twisted-12.0.0 using windows installer here
3. Install zope.interface with Mingw32 or not.
4. Install pyOpenSSL 0.13 using windows installer here
5. run "pip install scrapy", you are ready to go!

Hope it helps.