Understanding IP . The requested URL was not found on this server

Question 1

The relation between FQDNs (e.g. www.stackoverflow.com) and IP addresses (e.g. 198.252.206.140) is not necessarily a one-to-one relation. For instance, if I do a DNS lookup for www.stackoverflow.com, I get 198.252.206.140. So, the website for www.stackoverflow.com is hosted on a web server with the IP address 198.252.206.140. But, it's possible that there may be other web sites hosted at 198.252.206.140 as well.

That's why we have the Host command in the http protocol. After the browser makes a connection to the web server on port 80, the browser sends the host command to indicate which site on the web server it is attempting to connect to. See http://blog.tonycode.com/tech-stuff/http-notes/making-http-requests-via-telnet for a good tutorial on how this works. Copied below is a telnet session with 198.252.206.140, where an http connection is made, and the host command is issued to select www.stackoverflow.com, and the default response for www.stackoverflow.com is returned (which in this case is a 301 redirect to stackoverflow.com):

$ telnet 198.252.206.140 80
Trying 198.252.206.140...
Connected to 198.252.206.140.
Escape character is '^]'.
GET / HTTP/1.1
Host: www.stackoverflow.com

HTTP/1.1 301 Moved Permanently
Content-Type: text/html; charset=UTF-8
Location: http://stackoverflow.com/
Date: Tue, 04 Mar 2014 10:58:36 GMT
Content-Length: 148

<head><title>Document Moved</title></head>
<body><h1>Object Moved</h1>This document may be found <a   
HREF="http://stackoverflow.com/">here</a></body>

Question 2

The host name is roughly equivalent to the IP address up to the moment when your browser reaches the remote server.

But the HTTP/1.1 protocol specifies that the client (the browser) must pass the host name as an header, as in:

Host: www.howstuffworks.com

This is very useful, as it allows several sites to be hosted on the same server, or anyway to be reachable from the same IP address.

You can find this out using the utility command curl:

curl -v http://54.236.73.243/ -H 'Host: www.howstuffworks.com'

Or using your browser with some plugin that allows to set custom headers.

Question 3

To answer the second part of your question:

The ping command chooses one of the addresses to ping when the hostname has multiple addresses. On Windows systems ping can be used both for IPv4 and IPv6. On Unix-like systems you have to use ping6 for IPv6.

If you want to look what is in the DNS you'll have to use a tool that is made for that purpose such as host or dig (when on Unix-like system) or nslookup (when on a Windows system). For example:

$ dig google.com any

; <<>> DiG 9.8.3-P1 <<>> google.com any
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 42780
;; flags: qr rd ra; QUERY: 1, ANSWER: 19, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;google.com.            IN  ANY

;; ANSWER SECTION:
google.com.     299 IN  A   173.194.65.138
google.com.     299 IN  A   173.194.65.100
google.com.     299 IN  A   173.194.65.101
google.com.     299 IN  A   173.194.65.113
google.com.     299 IN  A   173.194.65.139
google.com.     299 IN  A   173.194.65.102
google.com.     299 IN  AAAA    2a00:1450:4013:c00::8b
google.com.     599 IN  MX  10 aspmx.l.google.com.
google.com.     599 IN  MX  20 alt1.aspmx.l.google.com.
google.com.     599 IN  MX  30 alt2.aspmx.l.google.com.
google.com.     599 IN  MX  40 alt3.aspmx.l.google.com.
google.com.     599 IN  MX  50 alt4.aspmx.l.google.com.
google.com.     21599   IN  NS  ns1.google.com.
google.com.     21599   IN  NS  ns2.google.com.
google.com.     21599   IN  NS  ns3.google.com.
google.com.     21599   IN  NS  ns4.google.com.
google.com.     3599    IN  TXT "v=spf1 include:_spf.google.com ip4:216.73.93.70/31 ip4:216.73.93.72/31 ~all"
google.com.     21599   IN  SOA ns1.google.com. dns-admin.google.com. 2014021800 7200 1800 1209600 300
google.com.     21599   IN  TYPE257 \# 19 0005697373756573796D616E7465632E636F6D

;; Query time: 27 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Fri Feb 28 23:53:29 2014
;; MSG SIZE  rcvd: 497

As you can see there is more in the DNS than A and AAAA records :-)

And to make this answer at least a little on-topic for StackOverflow, here is how you can properly deal with name resolving in a dual-stack environment. Write the code in such a way that it automatically uses whatever protocol is available. This is an example in Python:

#!/usr/bin/python
import sys, socket

# Get the hostname from the command line
if len(sys.argv) == 2:
    host = sys.argv[1]
else:
    print("Usage: {} <hostname>".format(sys.argv[0]))
    sys.exit(1)

# Set the parameters for the getaddrinfo call
service = "http"
family = socket.AF_UNSPEC
socktype = socket.SOCK_STREAM
protocol = socket.SOL_TCP
flags = 0

# Call getaddrinfo, it will give back a list of possible parameter-sets
try:
    resultset = socket.getaddrinfo(host, service, family, socktype, protocol, flags)
except socket.error:
    print("I'm sorry, but {} doesn't seem to exist".format(host))
    sys.exit(1)

# Now try to connect to them one by one
sock = None
for family, socktype, protocol, canonname, sockaddr in resultset:
    print("Trying to connect to {}".format(sockaddr))

    try:
        # Create a socket with the given parameters
        sock = socket.socket(family, socktype, protocol)
    except socket.error:
        # Failed to create a socket, try the next one
        continue

    try:
        # We have a socket, now use it to connect
        sock.connect(sockaddr)
    except socket.error:
        # The connection failed, close the socket
        sock.close()
        sock = None

        # And try the next one
        continue

    # Wonderful, we have a socket, and it is now connected!
    # Stop retrying
    break

if sock is None:
    print("None of the available addresses worked")
    sys.exit(1)

print("That one worked!")
my_addr = sock.getsockname()
remote_addr = sock.getpeername()
print("There is now a connection from {} to {}".format(my_addr, remote_addr))

# And close the connection nicely
sock.close()