Environment Variables in Python on Linux

Question

Python's access to environment variables does not accurately reflect the operating system's view of the processes environment.

os.getenv and os.environ do not function as expected in particular cases.

Is there a way to properly get the running process' environment?

---

To demonstrate what I mean, take the two roughly equivalent programs (the first in C, the other in python):

#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
int main(int argc, char *argv[]){
    char *env;
    for(;;){
        env = getenv("SOME_VARIABLE");
        if(env)
            puts(env);
        sleep(5);
    }
}

---

import os
import time
while True:
    env = os.getenv("SOME_VARIABLE")
    if env is not None:
        print env
    time.sleep(5)

---

Now, if we run the C program and attach to the running process with gdb and forcibly change the environment under the hood by doing something like this:

(gdb) print setenv("SOME_VARIABLE", "my value", 1)
[Switching to Thread -1208600896 (LWP 16163)]
$1 = 0
(gdb) print (char *)getenv("SOME_VARIABLE")
$2 = 0x8293126 "my value"

then the aforementioned C program will start spewing out "my value" once every 5 seconds. The aforementioned python program, however, will not.

Is there a way to get the python program to function like the C program in this case?

(Yes, I realize this is a very obscure and potentially damaging action to perform on a running process)

Also, I'm currently using python 2.4, this may have been fixed in a later version of python.

Answer 1

That's a very good question.

It turns out that the os module initializes os.environ to the value of posix.environ, which is set on interpreter start up. In other words, the standard library does not appear to provide access to the getenv function.

That is a case where it would probably be safe to use ctypes on unix. Since you would be calling an ultra-standard libc function.

Answer 2

You can use ctypes to do this pretty simply:

>>> from ctypes import CDLL, c_char_p
>>> getenv = CDLL("libc.so.6").getenv
>>> getenv.restype = c_char_p
>>> getenv("HOME")
'/home/glyph'

Answer 3

Another possibility is to use pdb, or some other python debugger instead, and change os.environ at the python level, rather than the C level. Here's a small recipe I posted to interrupt a running python process and provide access to a python console on receiving a signal. Alternatively, just stick a pdb.set_trace() at some point in your code you want to interrupt. In either case, just run the statement "import os; os.environ['SOME_VARIABLE']='my_value'" and you should be updated as far as python is concerned.

I'm not sure if this will also update the C environment with setenv, so if you have C modules using getenv directly you may have to do some more work to keep this in sync.

Answer 4

I don't believe many programs EVER expect to have their environment externally modified, so loading a copy of the passed environment at startup is equivalent. You have simply stumbled on an implementation choice.

If you are seeing all the set-at-startup values and putenv/setenv from within your program works, I don't think there's anything to be concerned about. There are far cleaner ways to pass updated information to running executables.

Answer 5

Looking at the Python source code (2.4.5):

• Modules/posixmodule.c gets the environ in convertenviron() which gets run at startup (see INITFUNC) and stores the environment in a platform-specific module (nt, os2, or posix)

• Lib/os.py looks at sys.builtin_module_names, and imports all symbols from either posix, nt, or os2

So yes, it gets decided at startup. os.environ is not going to be helpful here.

If you really want to do this, then the most obvious approach that comes to mind is to create your own custom C-based python module, with a getenv that always invokes the system call.