How to store scrapyd items in json format

Question

I am trying to store scrapyd items in a JSON file. Actually by default it store items in a json file but like this: File_1:

{item1}
{item2}
....

And if i run my spider by scrapy crawl spidername -o fileName -t json

it will store item like this: File_2:

[{item1},
{item2},
....]

If i try to copy paste contents of File_1 into jsonParser i got error saying expected [ but File_2 works fine. Can anyone tell me how to store items in proper JSON format using scrapyd

Answer 1

Try to save the items directly into the database rather then storing them on the file....then storing the items from file to database... or try to read line by line from that JSON file and then do what ever you want instead of reading the whole file at a time...

Answer 2

Actually its a json lines format , json is not good for very bid data. thats why scrapy separate each item in a comma separated format. you can add [ & ] in the start and end of file it will be json then,

Answer 3

Add to your settings.py:

FEED_FORMAT='json'