Why is a website's response in python's `urllib.request` different to a request sent directly from a web-browser?

Question

I have a program that takes a URL and gets a response from the server using urllib.request. It all works fine, but I tested it a little more and realised that when I put in a URL such as http://google.com into my browser, I got a different page (which had a doodle and a science fair promotion etc.) but with my program it was just plain Google with nothing special on it.

It is probably due to redirection, but if the request from my program goes through the same router and DNS, surely the output should be exactly the same?

Here is the code:

"""
This is a simple browsing widget that handles user requests, with the
added condition that all proxy settings are ignored. It outputs in the
default web browser.
"""

# This imports some necessary libraries.
import tkinter as tk
import webbrowser

from tempfile import NamedTemporaryFile
import urllib.request


def parse(data):
    """
    Removes junk from the data so it can be easily processed.
    :rtype : list
    :param data: A long string of compressed HTML.
    """
    data = data.decode(encoding='UTF-8')  # This makes data workable.
    lines = data.splitlines()  # This clarifies the lines for writing.
    return lines


class Browser(object):
    """This creates an object for getting a direct server response."""
    def __init__(self, master):
        """
        Sets up a direct browsing session and a GUI to manipulate it.
        :param master: Any Tk() window in which the GUI is displayable.
        """
        # This creates a frame within which widgets can be stored.
        frame = tk.Frame(master)
        frame.pack()

        # Here we create a handler that ignores proxies.
        proxy_handler = urllib.request.ProxyHandler(proxies=None)
        self.opener = urllib.request.build_opener(proxy_handler)

        # This sets up components for the GUI.
        tk.Label(frame, text='Full Path').grid(row=0)
        self.url = tk.Entry(frame)  # This takes the specified path.
        self.url.grid(row=0, column=1)
        tk.Button(frame, text='Go', command=self.browse).grid(row=0, column=2)

        # This binds the return key to calling the method self.browse.
        master.bind('<Return>', self.browse)

    def navigate(self, query):
        """
        Gets raw data from the queried server, ready to be processed.
        :rtype : str
        :param query: The request entered into 'self.url'.
        """
        # This contacts the domain and parses it's response.
        response = self.opener.open(query)
        html = response.read()
        return html

    def browse(self, event=None):
        """
        Wraps all functionality together for data reading and writing.
        :param event: The argument from whatever calls the method.
        """
        # This retrieves the input given by the user.
        location = self.url.get()
        print('\nUser inputted:', location)
        # This attempts to access the server and gives any errors.
        try:
            raw_data = self.navigate(location)
        except Exception as e:
            print(e)
        # This executes assuming there are no errors.
        else:
            clean_data = parse(raw_data)
            # This creates and executes a temporary HTML file.
            with NamedTemporaryFile(suffix='.html', delete=False) as cache:
                cache.writelines(line.encode('UTF-8') for line in clean_data)
                webbrowser.open_new_tab(cache.name)
                print('Done.')


def main():
    """Using a main function means not doing everything globally."""
    # This creates a window that is always in the foreground.
    root = tk.Tk()
    root.wm_attributes('-topmost', 1)
    root.title('DirectQuery')

    # This starts the program.
    Browser(root)
    root.mainloop()

# This allows for execution as well as for importing.
if __name__ == '__main__':
    main()

Note: I don't know if it is something to do with the fact that it is instructed to ignore proxies? My computer doesn't have any proxy settings turned on by the way. Also, if there is a way that I can get the same response/output as a web browser such as chrome would, I would love to hear it.

Answer 1

In order to answer your general question you need to understand how the web site in question operates, so this isn't really a Python question. Web sites frequently detect the browser's "make and model" with special detection code, often (as indicated in the comment on your question) starting with the User-Agent: HTTP header.

It would therefor make sense for Google's home page not to include any JavaScript-based functionality if the User-Agent identifies itself as a program.