Banner grabbing HTTP

Question

When a http web server receive a HTTP method from your client for example Server:\r\n and that is meaningless for web server, it may return a response that have both header and content.

4xx Client Error:

The 4xx class of status code is intended for cases in which the client seems to have errored. Except when responding to a HEAD request, the server should include an entity containing an explanation of the error situation, and whether it is a temporary or permanent condition. These status codes are applicable to any request method. User agents should display any included entity to the user.

So if you want only header section for grabbing banner, send a HTTP HEAD request.

Here is an example:

import socket


def http_banner_grabber(ip, port=80, method="HEAD",
                        timeout=60, http_type="HTTP/1.1"):
    assert method in ['GET', 'HEAD']
    # @see: http://stackoverflow.com/q/246859/538284
    assert http_type in ['HTTP/0.9', "HTTP/1.0", 'HTTP/1.1']
    cr_lf = '\r\n'
    lf_lf = '\n\n'
    crlf_crlf = cr_lf + cr_lf
    res_sep = ''
    # how much read from buffer socket in every read
    rec_chunk = 4096
    s = socket.socket()
    s.settimeout(timeout)
    s.connect((ip, port))
    # the req_data is like 'HEAD HTTP/1.1 \r\n'
    req_data = "{} / {}{}".format(method, http_type, cr_lf)
    # if is a HTTP 1.1 protocol request,
    if http_type == "HTTP/1.1":
        # then we need to send Host header (we send ip instead of host here!)
        # adding host header to req_data like 'Host: google.com:80\r\n'
        req_data += 'Host: {}:{}{}'.format(ip, port, cr_lf)
        # set connection header to close for HTTP 1.1
        # adding connection header to req_data like 'Connection: close\r\n'
        req_data += "Connection: close{}".format(cr_lf)
    # headers join together with `\r\n` and ends with `\r\n\r\n`
    # adding '\r\n' to end of req_data
    req_data += cr_lf
    # the s.send() method may send only partial content. 
    # so we used s.sendall()
    s.sendall(req_data.encode())
    res_data = b''
    # default maximum header response is different in web servers: 4k, 8k, 16k
    # @see: http://stackoverflow.com/a/8623061/538284
    # the s.recv(n) method may receive less than n bytes, 
    # so we used it in while.
    while 1:
        try:
            chunk = s.recv(rec_chunk)
            res_data += chunk
        except socket.error:
            break
        if not chunk:
            break
    if res_data:
        # decode `res_data` after reading all content of data buffer
        res_data = res_data.decode()
    else:
        return '', ''
    # detect header and body separated that is '\r\n\r\n' or '\n\n'
    if crlf_crlf in res_data:
        res_sep = crlf_crlf
    elif lf_lf in res_data:
        res_sep = lf_lf
    # for under HTTP/1.0 request type for servers doesn't support it
    #  and servers send just send body without header !
    if res_sep not in [crlf_crlf, lf_lf] or res_data.startswith('<'):
        return '', res_data
    # split header and data section from
    # `HEADER\r\n\r\nBODY` response or `HEADER\n\nBODY` response
    content = res_data.split(res_sep)
    banner, body = "".join(content[:1]), "".join(content[1:])
    return banner, body

Demo:

addresses = {'google.com': '216.239.32.20',
             'msdn.microsoft.com': '157.56.148.19',
}

for domain, ip in addresses.items():
    banner, body = http_banner_grabber(ip)
    print('*' * 24)
    print(domain, ip, 'HEAD HTTP/1.1')
    print(banner)

Also you can try it with GET method and also other options:

for domain, ip in addresses.items():
    banner, body = http_banner_grabber(ip, method="GET", http_type='HTTP/0.9')
    print('*' * 24)
    print(domain, ip, 'GET HTTP/0.9')
    print(banner)

Output (first example):

************************
google.com 216.239.32.20 HEAD HTTP/1.1
HTTP/1.1 200 OK
Date: Mon, 31 Mar 2014 01:25:53 GMT
Expires: -1
Cache-Control: private, max-age=0
Content-Type: text/html; charset=ISO-8859-1
Set-Cookie: **** it was to long line and removed ****
P3P: **** it was to long line and removed ****
Server: gws
X-XSS-Protection: 1; mode=block
X-Frame-Options: SAMEORIGIN
Connection: close
************************
msdn.microsoft.com 157.56.148.19 HEAD HTTP/1.1
HTTP/1.1 301 Moved Permanently
Content-Length: 0
Location: http://157.56.148.19/en-us/default.aspx
Server: Microsoft-IIS/8.0
P3P: **** it was to long line and removed ****
X-Powered-By: ASP.NET
X-Instance: CH104
Date: Mon, 31 Mar 2014 01:25:53 GMT
Connection: close

Output (second example):

msdn.microsoft.com 157.56.148.19 GET HTTP/0.9
HTTP/1.1 400 Bad Request
Content-Type: text/html; charset=us-ascii
Server: Microsoft-HTTPAPI/2.0
Date: Mon, 31 Mar 2014 01:27:13 GMT
Connection: close
Content-Length: 311
************************
google.com 216.239.32.20 GET HTTP/0.9
HTTP/1.0 400 Bad Request
Content-Type: text/html; charset=UTF-8
Content-Length: 1419
Date: Mon, 31 Mar 2014 01:27:14 GMT
Server: GFE/2.0

Now if you look at Server header of msdn.microsoft.com and google.com in two type of our example, through by this tool, we were able to discover a new thing:

For HTTP 1.1 request to google.com, Server is gws and for HTTP 0.9 request, Server is changed to GFE/2.0.
And for HTTP 1.1 request to msdn.microsoft.com, Server is
Microsoft-IIS/8.0 and for HTTP 0.9 request, Server is changed to Microsoft-HTTPAPI/2.0.