windows scrapyd-deploy is not recognized

Question

I have install the scrapyd like this

pip install scrapyd

I want to use scrapyd-deploy

when i type scrapyd

i got this exception in cmd:

'scrapyd' is not recognized as an internal or external command, operable program or batch file.

Answer 1

I ran into the same issue, and I also read some opinions that scrapyd isn't available / can't run on windows and nearly gave it up (didn't really need it as I intend on deploying to a linux machine, wanted scrapyd on windows for debug purposes). However, after some research I found a way. As I haven't found any clear instructions on this, I will try to make my answer as detailed as possible, listing all the steps that worked for me.

Assuming you want to run scrapyd on your local machine, you will need two command lines running: The first is used to connect to scrapyd and keeping the connection open, and the second is for deploying and scheduling.

• You already pip installed scrapyd.

• Create a folder C:\scrapyd and an empty .log file named scrapyd.log in this folder (not sure this step is necessary).

• Open your cmd and cd to your Scripts folder inside Python. This is usually something like: C:\Python27\Scripts. Type: python scrapyd

• At this point you should see something like that:

  2014-03-26 13:57:30+0200 [-] Log opened.
  
  2014-03-26 13:57:30+0200 [-] twistd 13.2.0 (C:\Python27\python.exe 2.7.6) starting up.
  
  2014-03-26 13:57:30+0200 [-] reactor class: twisted.internet.selectreactor.SelectReactor.
  
  2014-03-26 13:57:30+0200 [-] Site starting on 6800
  
  2014-03-26 13:57:30+0200 [-] Starting factory <twisted.web.server.Site instance at 0x0000000003F69208>
  
  2014-03-26 13:57:30+0200 [Launcher] Scrapyd 1.0.1 started: max_proc=16, runner='scrapyd.runner'
  

• If you can open your browser and go to http://localhost:6800 then you're ok. The command line window should stay open in the background as connection will be closed if you'll close it.

• On windows explorer navigate to your scrapy project folder and edit the scrapy.cfg file found there: write your deploy target name: [deploy:scrapyd] for instance, and uncomment the url line.

• Open a second command line and cd to your scrapy project folder. Type: scrapy deploy -l

• Deploy: type: scrapy deploy scrapyd -p project_name (scrapyd is your target). You should get a server response code 200 and an ok status. You can check if deploy was successful also by typing: scrapy deploy -L scrapyd

• Scheduling: you need to install curl for windows. Here's how: questions/9507353/steps-to-setup-curl-in-windows type: curl http://localhost:6800/schedule.json -d project=project_name -d spider=spider_name Again, you should get an ok status, and in your browser at http://localhost:6800 under jobs you can check whether the job was indeed scheduled.

I hope this helps.

Answer 2

If you install scrapyed-client by using pip install scrapyd-client I suggest try this command:

pip install git+https://github.com/scrapy/scrapyd-client

it worked for me.

thanks to scrapyd-client command not found

Answer 3

For me proposed solution above didn't work.

Below what worked for me: (for scrapy in version 0.24).

Go to C:\Python27\Scripts and create two files:

scrapy.bat

scrapyd-deploy.bat

Edit both files with notepad and paste:

In the scrapy.bat file:

@echo off
c:\Python27\python c:\Python27\Scripts\scrapy %*

In the scrapyd-deploy.bat file:

@echo off
c:\Python27\python c:\Python27\Scripts\scrapyd-deploy %*

Then save these files and restart cmd.

If you have C:\Python27\Python and C:\Python27\Scripts in your PATH then both commands scrapy and scrapyd-deploy should work.

Answer 4

Marcin Rapacz's answer work for me. However, I use anaconda to manage my python library. So, the files should be in "C:\Program Files (x86)\Anaconda3\Scripts", and content in files should be changed like:

@echo off
"C:\Program Files (x86)\Anaconda3\python.exe" "C:\Program Files 
(x86)\Anaconda3\Scripts\scrapyd-deploy" %1 %2 %3 %4 %5 %6 %7 %8 %9