Why are my locals not being updated outside rof?

Question

After experimenting with trying to implement C-for loops in Python, the following function was developed:

import sys

def rof(init, cond, post):
    init, cond, post, context = compile(init, '<rof>', 'exec'), \
                                compile(cond, '<rof>', 'eval'), \
                                compile(post, '<rof>', 'exec'), \
                                sys._getframe(1)
    context = context.f_globals, context.f_locals
    exec(init, *context)
    while eval(cond, *context):
        yield None
        exec(post, *context)

As any programmer knows, the new function needed to be tested to make sure it works:

Setup

class Employee:

    def __init__(self, employee_id, category, hired, salary, years):
        vars(self).update(locals())

    def __repr__(self):
        return '{}({})'.format(self.__class__.__name__,
                               ', '.join(map(repr, self)))

    def __iter__(self):
        yield self.employee_id
        yield self.category
        yield self.hired
        yield self.salary
        yield self.years

database = [Employee(123, 'P', 2014, 2000, 0),
            Employee(234, 'F', 2000, 20000, 14),
            Employee(123, 'F', 2010, 10000, 4)]

The code runs without errors in some cases (such as the one below):

Trial 1

for _ in rof('a = 0', 'a < len(database)', 'a += 1'):
    employee_id = database[a].employee_id
    for _ in rof('b = len(database) - 1', 'b > a', 'b -= 1'):
        if database[b].employee_id == employee_id:
            print(database[b], 'is being removed.')
            del database[b]

However, it does not work when the loops are in a separate function.

Trial 2

def remove_duplicates(database):
    a = b = int
    for _ in rof('a = 0', 'a < len(database)', 'a += 1'):
        employee_id = database[a].employee_id
        for _ in rof('b = len(database) - 1', 'b > a', 'b -= 1'):
            if database[b].employee_id == employee_id:
                print(database[b], 'is being removed.')
                del database[b]

remove_duplicates(database)

An error is generatated instead (TypeError: list indices must be integers, not type).

---

We can all agree that this code in not Pythonic, but can anyone identify what is causing the problem and how to fix it?

Answer 1

In Python 3 it is not possible to create new local variables into locals() as the set of local variables is deducted at compile time. Especially if you modify remove_duplicates so that it does not have the a = b = int line, Python does not consider these names to refer to a local but a global variable. With the presence of that line, they are considered to be a local variable, yes.

Also, changing the locals is not possible through the frame object, as in Python 3 the local variables are not stored in a dictionary anymore. Instead, on CPython 3 the frame.f_locals access creates a copy of the variables using PyFrame_FastToLocals, but it is normally a one-way trip. Thus while you can read the values of the variables, no changes will be propagated, and a and b continue to is int. However (module) global variables are still stored in a dictionary that is directly accessible through the frame.f_globals; and the dictionary is open for changes.

However, there is a blog post by the PyDev maintainer on how to achieve this on CPython 3. Thus the following rof implementation seems to do the trick for me:

def apply(frame):
    ctypes.pythonapi.PyFrame_LocalsToFast(ctypes.py_object(frame), ctypes.c_int(0))

def rof(init, cond, post):
    init, cond, post, context = compile(init, '<rof>', 'exec'), \
                                compile(cond, '<rof>', 'eval'), \
                                compile(post, '<rof>', 'exec'), \
                                sys._getframe(1)

    exec(init, context.f_globals, context.f_locals)
    apply(context)
    while eval(cond, context.f_globals, context.f_locals):
        apply(context)
        yield None
        exec(post, context.f_globals, context.f_locals)
        apply(context)

I think this is code is an abomination if anything, and recommend that instead of this, the hypothetical programmers would know how to change a C for loop into a C while loop... and work it into Python from there. And it still cannot work without giving initial value to these variables within the function body anyway.

Thus I propose an alternative rof implementation:

def rof(init, cond, post):
    print(init)
    print('while {}:'.format(cond))
    print('    # code goes here')
    print('    ' + post)

rof('b = len(database) - 1', 'b > a', 'b -= 1')

prints:

b = len(database) - 1
while b > a:
    # code goes here
    b -= 1

which is what ought to be written anyway.

though there is not much wrong in this case with:

for a in range(len(database)):
    for b in range(len(database) - 1, a, -1):
        ...