Multiple port scans using external tool, subprocess.Popen and threads

Question

I am using a port scanner to scan my subnet. The port scanner unfortunately can scan one port of only one host at a time. Also the scanner has a 1 sec timeout for unreachable hosts. The scanner(being an outside program) has to be run from subprocess.Popen() and to speed it up - so that I may send multiple probes while some previous ones are waiting for replies- I use threads. The problem comes up for a complete /24 subnet scan with large number of threads. SOme of the actually open ports are displayed as closed. I suspect somehow the output gets garbled. Note that this does not occur if I scan fewer hosts or one host at a time

The following code is my attempt to create a pool of threads which take an IP address and run 'sequential' port scan for defined port. Once all specified ports are scanned, it picks up next IP from the list.

        while True:
            if not thread_queue.empty():
                try:
                    hst = ip_iter.next()
                except StopIteration:
                    break
                m=thread_queue.get()
                l=ThreadWork(self,hst,m)
                l.start()
        while open_threads != 0:
            pass      

Where this fragment sets up thread queue

        thread_list = [x for x in range(num_threads)]
        for t in thread_list:
            thread_queue.put(str(t))
        ip_iter=iter(self.final_target)

In the ThreadWork function I keep a tab of open threads(since thread_queue.empty proved to be unreliable, I had to use this crude way)

class ThreadWork(threading.Thread):
    def __init__(self,i,hst,thread_no):
        global open_threads
        threading.Thread.__init__(self)
        self.host = hst
        self.ptr = i
        self.t = thread_no
        lock.acquire()
        open_threads = open_threads + 1
        lock.release() 

    def run(self):
        global thread_queue
        global open_threads
        global lock
        user_log.info("Executing sinfp for IP Address : %s"%self.host)
        self.ptr.result.append(SinFpRes(self.host,self.ptr.init_ports,self.ptr.all_ports,self.ptr.options,self.ptr.cf))
        lock.acquire()
        open_threads = open_threads - 1
        lock.release()
        thread_queue.put(self.t)

The call to SinFpRes creates a result object for one IP and initiate sequential scanning of ports for that IP only. The actual scan per port is as shown

        com_string = '/usr/local/sinfp/bin/sinfp.pl '+self.options+' -ai '+str(self.ip)+' -p '+str(p)
        args = shlex.split(com_string) 
        self.result=subprocess.Popen(args,stdout=subprocess.PIPE).communicate()[0]
        self.parse(p)

The parse function then utilises result stored in self.result to store output for that PORT. An aggregate of all the ports is what constitues a scan result for an IP.

Calling this code using 10 threads gives accurate o/p (when compared to nmap output). On giving 15 threads an occasional open port is missed. On giving 20 threads, more open ports are missed. On giving 50 threads many ports are missed.

P.S. - As a first timer, this code is very convoluted. Apologies to the puritans.

P.P.S. - Even a threaded port scan takes 15 minutes for an entire class C subnet with hardly 20 ports scanned. I was wondering if I should move this code to another language and use Python to only parse the results. Could somebody suggest me a language? Note -I am exploring Shell option as shown by S.Lott but manual processing is required before dumping it into file.

Answer 1

Use the shell

 for h in host1 host2 host3
 do
     scan $h >$h.scan &
 done
 cat *.scan >all.scan

This will scan the entire list of hosts all at the same time, each in a separate process. No threads.

Each scan will produce a .scan file. You can then cat all the .scan files into a massive all.scan file for further processing or whatever it is you're doing.

Answer 2

Why don't you try it?

(Answer: No they will have their own pipe)

Answer 3

Use Perl instead of Python. The program (SinFP) is written in Perl, you can modify the code to suite your needs.