PL/Python Error "could not convert SPI error to Python exception"

Question

I am working on a set of PL/Python stored procedures. I am using PostgreSQL 9.3 ( installed from apt.postgresql.org ) and a Python 2.7 interpreter; running on Ubuntu 13.04.

The error occurs in the middle of a large operation ( creating a materialized view that uses data from a source table with > 300,000 rows and computes some fields using the PL/Python stored procedures ).

The error output that I get is:

ERROR:  could not convert SPI error to Python exception
CONTEXT:  PL/Python function "get_first_level_parent"
********** Error **********

ERROR: could not convert SPI error to Python exception
SQL state: XX000
Context: PL/Python function "get_first_level_parent"

("get_first_level_parent" is the name of one of the stored procedures ).

The PostgreSQL server logs do not provide any further illumination ( I am not familiar with PostgreSQL and PL/Python internals ). When running with verbose error logging, I get this:

2013-10-18 12:56:43 EAT ERROR:  XX000: could not convert SPI error to Python exception
2013-10-18 12:56:43 EAT CONTEXT:  PL/Python function "get_first_level_parent"
2013-10-18 12:56:43 EAT LOCATION:  PLy_spi_exception_set, plpy_spi.c:576

The PostgreSQL error code documentation tells me that XX000 is the error code for internal_error. If I isolate the call to the stored procedure and run it in it's own statement eg SELECT get_first_level_parent(373673007), it does not raise an error.

My question is - what tools / techniques can I use to debug this sort of error? For my current problem, I am likely to "solve" the problem by rewriting the stored procedure ( slowly, testing one small part at a time ). Is that the only way out?

Answer 1

I would sprinkle some debug statements in src/pl/plpython/plpy_spi.c:PLy_spi_exception_set(). This might be a bug in PL/Python in handling some edge condition.

Answer 2

After sprinkling a generous helping of plpy.notice(... statements in the stored procedures, I eventually isolated a single SQL stored procedure invocation that reliably reproduced the error.

The culprit here was an edge case in one of the PL/Python procedures I wrote. I have a recursive algorithm in one of the stored procedures. The use of recursion is a dictated by the problem domain. Under all normal operating conditions, the recursion will not go beyond 10-12 deep ( the input data is well known and relatively static ). There was a bug in my implementation that lead to infinite recursion under some circumstances.

PL/Python does not seem to be set up to handle the RuntimeError that occurs when the maximum recursion depth is exceeded. This is probably a highly infrequent edge case.