AppEngine MapReduce how to filter StructuredProperty while using datastore input reader?

Question

While using appengine mapreduce lib, how to filter by StructuredProperty?

I tried:

class Tag(ndb.Model):
    # ...
    tag = ndb.StringProperty()
    value = ndb.FloatProperty(indexed=False)

class User(ndb.Model):
    # ...
    tags = ndb.StructuredProperty(Tag, repeated=True)

class SamplePipeline(base_handler.PipelineBase):


    def run(self, tags, start_time, account_type, gsbucketname):

        start_time = datetime.strptime(start_time, "%Y-%m-%d %H:%M:%S")
        filters = []

        for tag in tags:
            filters.append(("tags.tag", "=", tag))

        yield mapreduce_pipeline.MapperPipeline(
            "name",
            "mapper",
            "mapreduce.input_readers.DatastoreInputReader",
            output_writer_spec="mapreduce.output_writers.FileOutputWriter",
            params={
                "input_reader": {
                    "entity_kind": "User",
                    "batch_size": 500,
                    "filters": filters
                },
                "output_writer": {
                    "filesystem": "gs",
                    "gs_bucket_name": gsbucketname,
                },
                "root_pipeline_id": self.root_pipeline_id,
                "account_type": account_type
            },
            shards=255
        )

What I got.

File "/Users/lucemia/vagrant_home/adex2/lib/mapreduce/input_readers.py", line 794, in _validate_filters_ndb
    prop, model_class._get_kind())
BadReaderParamsError: ('Property %s is not defined for entity type %s', u'tags.tag', 'User')

Answer 1

Looks like there is indeed a bug in the _validate_filters_ndb() function of mapreduce/input_readers.py.

As a work around you can add the following lines at line 792 of mapreduce/input_readers.py to skip over the validation of properties that contain a . in them like properties of type StructuredProperty do:

if "." in prop:
    continue

We will work on a fix to do the proper validation of properties of type StructuredProperty and provide the correct code in a future release.