SQLAlchemy filter query by PickleType contents

Question

I have this Meter object model

class Meter(db.Model):
    """
    Model for repair meter.

    """
    __tablename__ = 'meters'

    id = db.Column(db.Integer, primary_key=True, nullable=False)
    meter_no = db.Column(db.String(50), unique=True, nullable=False)
    client = db.Column(db.String(50), unique=True, nullable=False)
    problems = db.Column(db.PickleType)
    functional = db.Column(db.Boolean(), default=False, server_default="false")
    location = db.Column(db.Integer, db.ForeignKey('locations.id'))
    date_in = db.Column(db.Date, default=dt.date.today())
    date_out = db.Column(db.Date)

The problems PickleType column stores a Python list that contains ids of Problem objects

Then I have this function that tries to query the meters database table and get only Meter Objects that have a particular problem id (prob_id) in the list store as PickleType.

def problemrate(prob_id, month):
    """
    Return meters with the problem id for a particular month.

    args:
        prob_id (int): problem id
        month (int): month number e.g. april -> 4

    returns:
        meters (list): list of meter objects

    raises: None

    """
    meters = Meter.query.filter(and_(extract('month', Meter.date_in) == month,
                    prob_id in Meter.problems)).all()

    return meters

However when a call the function I get an error:

NotImplementedError: Operator 'getitem' is not supported on this expression

What is the write way to filter sqlalchemy queries by the contents of a PickleType column? It is possible or sqlalchemy has no support for this?

Answer 1

Short explanation : PickleType does not support any relational functionalities such as queries/filters. Its purpose is storing and retrieving. Use sqlalchemy.orm.relationship instead.

Long explanation : The error message is actually right. Everything in the filter function will compile to an sql query, (print the query to see this), so the 'in' operator does not compile to a valid query, neither do any other getItem operators. What does work is '==', however for this you need to pass the exact same object so it can be transformed to a PickleType object that is identical to the stored one. The comparison is done internally.

Solution : In plain sql you would store each list item separately in a different table, and associate by id. In SQLAlchemy you can do something similar:

from sqlalchemy.orm import relationship
from sqlalchemy.ext.associationproxy import association_proxy

class Meter(db.Model):
    """
    Model for repair meter.

    """
    __tablename__ = 'meters'

    id = db.Column(db.Integer, primary_key=True, nullable=False)
    meter_no = db.Column(db.String(50), unique=True, nullable=False)
    client = db.Column(db.String(50), unique=True, nullable=False)
    meter_problems = relationship('Problem', secondary=lambda: meterproblems_table)
    functional = db.Column(db.Boolean(), default=False, server_default="false")
    location = db.Column(db.Integer, db.ForeignKey('locations.id'))
    date_in = db.Column(db.Date, default=dt.date.today())
    date_out = db.Column(db.Date)

    problems = association_proxy('meter_problems', 'problem')

class Problem(db.Model):
    __tablename__ = 'problem'
    id = db.Column(db.Integer, primary_key=True, nullable=False)
    problem = db.Column('keyword', db.String(64))

    def __init__(self, problem):
        self.problem = problem

meterproblems_table = db.Table(
    'meterproblems',
    db.metadata,
    db.Column(
        'meter_id',
        db.Integer,
        db.ForeignKey("meters.id"),
        primary_key=True
    ),
    db.Column(
        'problem_id',
        db.Integer,
        db.ForeignKey("problem.id"),
        primary_key=True
    )
)

your query than becomes:

meters = Meter.query.filter(
    extract('month', Meter.date_in) == month,
    Meter.meter_problems.has(keyword=prob_id)
).all()

The assocation_proxy is used to easily add items to the meter_problems column:

obj.problems.append('something')

Where obj is a meter object.

Doumentation here