Best way to send many DNS queries with twisted framework (over UDP)?

Question

I am currently learning the twisted framework and I am trying to make a asynchronous DNS resolver using twisted.names.client.Resolver and twisted.names.client.getHostByName.

The script should brute subdomains by querying the authoritative nameservers. 10000-50000 per second concurrent connections is my minimum threshold in order to consider the tool usable for my intentions.

My questions are:

• Is twisted appropriate/suitable for such endeavors?
• How strong is Python/Twisted general performance struggle in comparison to C for such projects? I assume twisted isn't exactly made for such kind of ideas and the internal reactor management has quite a overhead when it comes to many connections...
• Projects like masscan are very fast. The author manages to send 2 Mio. packets/second (with special purpose drivers PF_RING even more). I am currently figuring out how he does so, but I hope I don't need to get down this road because I want to stay with twisted.

Becoming concrete: The below script is my very first attempt, but it doesn't work as fast as hoped.

I strongly assume that my approach is completely wrong. If you call the bottom script like that:

[nikolai@niko-arch subdomains]$ python2 subdomains.py -n50 nytimes.com       
www ==> ``170.149.168.130``
blog ==> ``170.149.168.153``
cs ==> ``199.181.175.242``
my ==> ``170.149.172.130``
blogs ==> ``170.149.168.153``
search ==> ``170.149.168.135``
cn ==> ``61.244.110.199``
feeds ==> ``170.149.172.130``
app ==> ``54.243.156.140``
games ==> ``184.73.175.199``
mail ==> ``170.149.172.135``
up ==> ``107.20.203.136``
tv ==> ``170.149.168.135``
data ==> ``174.129.28.73``
p ==> ``75.101.137.16``
open ==> ``170.149.168.153``
ts ==> ``170.149.97.51``
education ==> ``170.149.168.130``
wap ==> ``170.149.172.163``
m ==> ``170.149.172.163``

Everything works fine with 50 subdomain requests in most cases. But when I specify -n1000 (and thus 1000 upd dns requests) it takes very very long (5 minutes and up) and the reactor is yielding all kinds of strange errors like twisted.internet.error.DNSLookupError and twisted.internet.defer.TimeoutError (Example: Failure: twisted.internet.defer.TimeoutError: [Query('blogger.l.google.com', 255, 1)]). Normally, it just hangs and doesn't finish.

I expect for each non existent subdomain to receive a twisted.names.error.DNSNameError or in the case the subdomain exists, a valid A or AAAA resource record answer, but no a DNSLookupError as above.

Can anybody give me a hint what I am doing wrong? Normally, epoll() should be easily capable of sending more than 1000 requests (years ago I made the same in C and 10000 udp datagrams where sent in a matter of seconds). So which part of twisted did I not figure out correctly?

Is gatherResults() incorrect? I don't know what I am doing wrong..

Best thanks in advance for all answers!

# Looks promising: https://github.com/zhangyuyan/github
# https://github.com/zhangyuyan/github/blob/01dd311a1f07168459b222cb5c59ac1aa4d5d614/scan-dns-e3-1.py

import os
import argparse
import exceptions

from twisted.internet import defer, reactor
import twisted.internet.error as terr
from twisted.names import client, dns, error

def printResults(results, subdomain):
    """
    Print the ip address for the successful query.
    """
    return '%s ==> ``%s``' % (subdomain, results)

def printError(failure, subdomain):
    """
    Lookup failed for some reason, just catch the DNSNameError and DomainError.
    """
    reason = failure.trap(error.DNSNameError, error.DomainError, terr.DNSLookupError, defer.TimeoutError) # Subdomain wasn't found
    print(failure)
    return reason

def printRes(results):
    for i in results:
        if not isinstance(i, type): # Why the heck are Failure objects of type 'type'???
            print(i)
    reactor.stop()
    global res
    res = results

def get_args():
    parser = argparse.ArgumentParser(
        description='Brute force subdomains of a supplied target domain. Fast, using async IO./n')
    parser.add_argument('target_domain', type=str, help='The domain name to squeeze the subdomains from')
    parser.add_argument('-r', '--default-resolver', type=str, help='Add here the ip of your preferred DNS server')
    parser.add_argument('-n', '--number-connections', default=100, type=int, help='The number of file descriptors to acquire')
    parser.add_argument('-f', '--subdomain-file', help='This file should contain the subdomains separated by newlines')
    parser.add_argument('-v', '--verbosity', action='count', help='Increase the verbosity of output', default=0)

    args = parser.parse_args()

    if args.number_connections > 1000:
        # root privs required to acquire more than 1024 fd's
        if os.geteuid() != 0:
            parser.error('You need to be root in order to use {} connections'.format(args.number_connections))
    if not args.default_resolver:
        # Parse /etc/resolv.conf
        args.default_resolver = [line.split(' ')[1].strip() for line in open('/etc/resolv.conf', 'r').readlines() if 'nameserver' in line][0]
    return args

def main(args=None):
    if args:
        args = args
    else:
        args = get_args()

    subs = [sub.strip() for sub in open('subs.txt', 'r').readlines()[:args.number_connections]]

    # use openDNS servers
    r = client.Resolver('/etc/resolv.conf', servers=[('208.67.222.222', 53), ('208.67.220.220', 53)])
    d = defer.gatherResults([r.getHostByName('%s.%s' % (subdomain, args.target_domain)).addCallbacks(printResults, printError, callbackArgs=[subdomain], errbackArgs=[subdomain]) for subdomain in subs])
    d.addCallback(printRes)
    reactor.run()

if __name__ == '__main__':
    main()

Answer 1

The way you're doing this is to buffer up all subdomain requests into one giant list, then make all of the requests, then buffer up the query responses in one giant list, then print that list. Since you probably just want to print the name resolutions as they arrive, you should instead be scheduling timed calls to make the requests, probably on some very short timed interval, in batches of some specified size.

Also, if you're interested in high-performance Python, you should be using PyPy and not CPython. Just making that change alone, even without making your code more scalable, might get you enough of a performance boost to hit your targets.