Async ping with multiprocessing.pool

Question

I am trying to ping a few hundred devices in Python using multiprocessing.pool on Windows as part of a larger program. The responses are parsed into success cases and failure cases (i.e request times out, or the host is unreachable).

The code below works fine, however, another part of the program takes the average from the response and calculates a rolling average with previous results fetched from a database.

The rolling average function fails very occasionally on int(new_average) because the new_average passed in is a None type. Note that the rolling average function is only calculated in success cases.

I think the error must be in the parse function (this seems unlikely), or with how I am using multiprocessing.pool.

My question: am I using multiprocessing correctly? More generally, is there a better way to implement this asynchronous ping? I have looked at using Twisted, but I didn't see any ICMP protocol implementation (there is txnettools on GitHub, but I am not sure of the correctness of this, and it doesn't look maintained anymore).

A node object looks like:

class Node(object):
    def __init__(self, ip):
        self.ip = ip
        self.ping_result = None
    # other attributes and methods...

Here is the idea of the async ping code:

import os
from multiprocessing.pool import ThreadPool

def parse_ping_response(response):
    '''Parses a response into a list of the format:
    [ip_address, packets_lost, average, success_or_failure]
    Ex: ['10.10.10.10', '0', '90', 'success']
    Ex: [None, 5, None, 'failure']
    '''
    reply = re.compile("Reply\sfrom\s(.*?)\:")
    lost = re.compile("Lost\s=\s(\d*)\s")
    average = re.compile("Average\s=\s(\d+)m")
    results = [x.findall(response) for x in [reply, lost, average]]

    # Get reply, if it was found. Set [] to None.
    results = [x[0] if len(x) > 0 else None for x in results]

    # Check for host unreachable error.
    # If we cannot find an ip address, the request timed out.
    if results[0] is None:
        return results + ['failure']
    elif 'Destination host unreachable' in response:
        return results + ['failure']
    else:
        return results + ['success']


def ping_ip(node):
    ping = os.popen("ping -n 5 "+node.ip,"r")
    node.ping_result = parse_ping_response(ping.read())
    return


def run_ping_tests(nodelist):
    pool = ThreadPool(processes=100)
    pool.map(ping_ip, nodelist)
    return

if __name__ == "__main__":
    # nodelist is a list of node objects
    run_ping_tests(nodelist)

An example ping response for reference (from the Microsoft docs):

Pinging 131.107.8.1 with 1450 bytes of data:

Reply from 131.107.8.1: bytes=1450 time<10ms TTL=32
Reply from 131.107.8.1: bytes=1450 time<10ms TTL=32

Ping statistics for 131.107.8.1:
    Packets: Sent = 2, Received = 2, Lost = 0 (0% loss),
Approximate roundtrip times in milliseconds:
    Minimum = 0ms, Maximum =  10ms, Average =  2ms

Answer 1

I recommend you to use gevent (http://www.gevent.org, asynchronous I/O library based on libev and greenlet coroutines) instead of multiprocessing.

It turns out there an implementation of ICMP for gevent: https://github.com/mastahyeti/gping