Python Urllib Urlopen won't return after new line?

Question

I'm trying to get authenticated by an API I'm attempting to access. I'm using urllib.parse.urlencode to encode the parameters which go in my URL. I'm using urllib.request.urlopen to fetch the content.

This should return 3 values from the server, such as:

SID=AAAAAAAAAAA
LSID=BBBBBBBBBBB
AUTH=CCCCCCCCCCC

The problem is it only returns the first value, and the trailing new line character.

import urllib.request
import urllib.parse

Emailparamx = 'Email'
Emailparam = Emailparamx.encode('utf-8')
email = 'myemail@stackoverflow.com'
email = email.encode('utf-8')
Passwdparam = 'Passwd'
Passwdparam = Passwdparam.encode('utf-8')
password = 'hidden'
password = password.encode('utf-8')
Accounttypeparam = 'accountType'
Accounttypeparam = Accounttypeparam.encode('utf-8')
accounttype = 'GOOGLE'
accounttype = accounttype.encode('utf-8')
Serviceparam = 'service'
Serviceparam = Serviceparam.encode('utf-8')
service = 'adwords'
service = service.encode('utf-8')


url = 'https://accounts.google.com/ClientLogin?'
urlen = url.encode('utf-8')
data = [(Emailparamx, email), (Passwdparam, password),
        (Accounttypeparam, accounttype), (Serviceparam, service)]



auth = ''

dataurl = urllib.parse.urlencode(data)


accessurl = (url + "%s" % dataurl)

fh = urllib.request.urlopen(accessurl)

equals = '='
eqenc = equals.encode('utf-8')

try:
  msg = fh.readline().split(eqenc)
  print (msg)

And then msg prints

[b'SID', b'AAAAAAAAAAAAAAAAA\n']

I know that's some seriously ugly code, I'm about a week old in Python. Any help would be greatly appreciated.

Answer 1

The problem is that you're only calling readline once, so it only reads one line. If you want to read the lines one by one, you have to keep calling readline in a loop until done:

while True:
    msg = fh.readline()
    if not msg:
        break
    msg = msg.split(eqenc)
    print(msg)

---

However, there's really no good reason to call readline here, because any file-like object (including a urlopen object) is already an iterable full of lines, so you can just do this:

for msg in fh:
    print(msg)

---

Meanwhile, your original code has a try without an except or a finally, which will just raise a SyntaxError. Presumably you wanted something like this:

try:
    for msg in fh:
        print(msg)
except Exception as e:
    print('Exception: {}'.format(e))

---

While we're at it, we can simplify your code a bit.

If you look at the examples:

Here is an example session that uses the GET method to retrieve a URL containing parameters:

That's exactly what you want to do here (except for the last line). All the extra stuff you're doing with encoding the strings is not only unnecessary, but incorrect. UTF-8 is the wrong encoding is the wrong encoding to use for URLs (you get away with it because all of your strings are pure ASCII); urlopen requires a string rather than an encoded byte string (although, at least in CPython 3.0-3.3, it happens to work if you give it byte strings that happen to be encoded properly); urlencode can take byte strings but may not do the right thing (you want to give it the original Unicode so it can quote things properly); etc.

Also, you probably want to decode the result (which is sent as ASCII—for more complicated examples, you'll have to either parse the fh.getheader('Content-Type'), or read the documentation for the API), and strip the newlines.

You also may want to build a structure you can use in your code instead of just printing it out. For example, if you store the results in login_info, and you need the SID in a later request, it's just login_info['SID'].

So, let's wrap things up in a function, then call that function:

import urllib.request
import urllib.parse

def client_login(email, passwd, account_type, service):
    params = {'Email': email,
              'Passwd': passwd,
              'accountType': account_type,
              'service': service}
    qs = urllib.parse.urlencode(params)
    url = 'https://accounts.google.com/ClientLogin?'
    with urllib.request.urlopen(url + qs) as fh:
        return dict(line.strip().decode('ascii').split('=', 1) for line in fh)

email = 'myemail@stackoverflow.com'
password = 'hidden'
accounttype = 'GOOGLE'
service = 'adwords'
try:
    results = client_login(email, password, accounttype, service)
    for key, value in results.items():
        print('key "{}" is "{}".format(key, value))
except Exception as e:
    print('Exception: {}'.format(e))