Django fixtures. Loading initial data process is being Killed

Question

I have been working on refining and refactoring some 57k+ records from two legacy databases into one Django-compatible entity. Now when I'm done, I dumped it as a fixture and I am trying to load it in a production environment.

My problem is that the process is being "Killed" after a short while. My process is:

./manage.py syncdb --noinput
./manage.py loaddata core/fixtures/auth.json  # just a default user
./manage.py migrate

result:

Running migrations for django_extensions:  # custom apps migrate just fine
 - Migrating forwards to 0001_empty.
 > django_extensions:0001_empty
 - Loading initial data for django_extensions.
Installed 0 object(s) from 0 fixture(s)
Running migrations for myotherapp:
 - Migrating forwards to 0001_initial.
 > myotherapp:0001_initial
 - Loading initial data for myotherapp.
Installed 4 object(s) from 1 fixture(s)  # my other app with a fixture migrates ok
Running migrations for myapp:
 - Migrating forwards to 0001_initial.
 > myapp:0001_initial
 - Loading initial data for myapp.
Killed

I must note that the process is going without problems on my development machine. Another note is that my dev machine is running postgres 9.2, in production there is 9.1 - can this be that much of a problem?

How do I approach debugging this? I don't even know what is wrong from vague "Killed" being printed. Does South store any logs? Any help appreciated.

EDIT: As Paulo Scardine pointed out, the problem was with JSON file being to heavy. First I tried XML dump and it went further but was killed off eventually too. A way to go is SQL dump. For Postgres what worked for me was:

pg_dump dbname | gzip > filename.gz # dump data on dev machine
createdb dbname # create empty db in production
gunzip -c filename.gz | psql dbname # restore the dump in production

Answer 1

Could not find the specific bug regarding loading fixtures. This one is for dumping, but I guess the root cause is related:

• "dumpdata" should stream output one row at a time

There are a couple duplicates of your question:

• Reading rather large json files in Python
• Is there a memory efficient and fast way to load big json files in python?

When I run into this bug I was told to use XML fixtures instead because the XML parser behaves better regarding the memory footprint.

My advice is to not lose much sleep over this issue, resort to plain SQL dumps if you can.