Why are os.system and subprocess.call spawning so many processes?

Question

import os
import subprocess
import sys
import re

## fname_ext=sys.argv[1]
fname_ext=r"C:\mine\.cs\test.cs"
exe=os.path.splitext(fname_ext)[0]+".exe" # Executable
fdir=os.path.split(fname_ext)[0]
fcontent=open(fname_ext).read()

p_using=re.compile("\s*using\s+((\w+[.]*)+)")
p_namespace=re.compile("\s*namespace\s+(\w+)")
usings=p_using.findall(fcontent)
usings=[x[0] for x in usings]
references=[]
for i in os.listdir(fdir):
    path=fdir+"\\"+i
    try:
        if os.path.isdir(path) or (not path.endswith('cs')):continue
        with open(path) as fp:
            content=fp.read()
            namespaces=p_namespace.findall(content)
            for n in namespaces:
                if n in usings and 'System' not in n:
                    references+=[path]
    except:
        pass

command="csc /nologo "+" ".join(references)+" "+fname_ext
## command=" ".join(references)
#~ ---------------------------------------------------------
# Build:
option=1
if option==0:
    # using os.system
    print ">>",command
    if os.system(command)==0:
        os.system(exe)
else:
    #~ Using subprocess module
    ## print type(references)
    command=['csc']
    ## print command,references
    command.extend(["/nologo","/out:"+exe])
    command.extend(references)
    command.append(fname_ext)
    ## print command
    if subprocess.call(command,shell=True)==0:
        ## print "running %s"%exe
        subprocess.call([exe],shell=True)
    else:
        pass
        ## print "Failed to run"
#~ ---------------------------------------------------------

I have this code above that is supposed to run a Csharp program from SciTE. It searches
every .cs file in the directory and finds the file with the namespace that the current
file has included. The command to run the file in SciTE is:
command.go.*.cs=python C:\mine\.py\csc.py $(FilePath)
command.go.subsystem.*.cs=0

That program logic part is okay.
The issue is that when hit F5 with sample Csharp code like this:

using System;
using System.Collections;
using MyNamespace;
class Test{
    public static void Main(String[] args){
        MyObject inst=new MyObject();
        MyObject.self_destruct(inst);
    }
}

it runs ok. But when I uncomment the second fname_ext and comment the first one
and run the csc.py file, a window opens and keeps running, printing command(this happens
using the os.system option). When you use the subprocess.call option, the same thing
happens but this time only when shell=True. It ran for only 15 seconds and there were 800+ cmd.exe and python.exe processes.I had to wait almost 5 minutes after killing cmd.exe
for the mouse to start responding and 2 minutes more for desktop peek to work.
When shell=False, it runs ok, the same way as when you hit the F5 key from the file.
What is happening here?
What is shell=True doing that makes it behave that way?

Answer 1

Okay, I'll take a stab at this. If I understand the situation, this script is called csc.py and you want to call the csc c# compiler. When you run csc /nologo (etc...) through cmd.exe, it starts looking for something called 'csc' with a known extension. It finds csc.py in the current directory and since .py is a registered extension, that's what gets executed.

The solution is to rename your python file or call out 'csc.exe' explicitly.

Answer 2

The problem is that your sys.argv looks something like this:

['python', r'C:\mine\.py\csc.py', 'whatever.cs']

So, with the fname_ext line uncommented, you set fname_ext to r'C:\mine\.py\csc.py'. Which means your script ends up just running itself—which again runs itself, etc., as fast as possible until your system chokes.

The reason it doesn't happen with shell=False is that you can't actually exec a Python script. Ultimately you end up calling CreateProcess with your script, which tries to interpret it as a .exe file, fails, and returns an error. But with shell=True, you pass your script to cmd.exe to run as a program, and it does the same thing an interactive prompt or Explorer would do: finds the right mapping to execute .py files and uses it. (And os.system does effectively the same thing as shell=True, but with a couple extra layers tossed in for good measure.)