Importing a dict to an App Engine ndb.JsonProperty with bulkloader

Question

I'm using the bulkloader to upload data into my App Engine data storage.

I can't seem to store a dictionary into a JsonProperty and I am getting the following error:

BadValueError: Unsupported type for property nearby_countries: <type 'dict'>

My model defines this property as a JsonProperty:

nearby_countries = ndb.JsonProperty()

The only workaround I found seems to store a json.dumps() of my value instead but I guess this basically storing the string representation of the dictionary rather than the dictionary itself.

My understanding of the JsonProperty is that it takes a python object as the value and that I should not be bothered about the JSON serialization which ndb will take care of. Am I correct?

Value is a Python object (such as a list or a dict or a string) that is serializable using Python's json module; the Datastore stores the JSON serialization as a blob.

Answer 1

After lots of try & error as well as googling around for similar posts, I manage to find the following post which lead me the below solution:

http://blog.thekensta.com/2012/06/google-app-engine-bulk-loader-and-ndb.html

In short, JsonProperties are stored as blobs and we need to pass the bulkloader the correct transform method to generate a blob from the json string. We can use transform.blobproperty_from_base64 (from the google.appengine.ext.bulkload.transform module)

So I convert my list or dict to a string JSON string representation which then gets converted to a blob so that the bulkloader can store it:

import_transform: "lambda x: transform.blobproperty_from_base64(base64.b64encode(bytes(json.dumps(x.strip(' ,').split(',')))))"

The same reasoning fixes the TextProperty saved as Strings (mentioned in my comment above). You need to use db.Text as the transform function:

import_transform: db.Text

And to save a repeated=True TextProperty, I actually had to transform it to a blob as well:

import_transform: "lambda x: transform.blobproperty_from_base64(base64.b64encode(bytes(json.dumps(x.strip(' ,').split(',')))))"

(in the exemple above I actually turn a coma separated string into a list of Text objects to be stored in a TextProperty(repeated=True)

Answer 2

In general you are correct about the JsonProperty. However, the bulk loader is special. Honestly, I don't know much about how it works, but in that context I wouldn't be surprised if it required you to call json.dumps() yourself.