Exception shows up in wrong place

Question

In the t.cursor() method, an exception from the Twython library is thrown for a small percentage of ids. However, whenever the exception occurs, the actual line in my code where it's thrown is the for-loop that comes after the try/except block, which prevents the continue from being called.

How is it that an exception can be thrown in the try block, not be caught by the except block, and then show up later during some (mostly) unrelated code?

And yes, it's a 401 error, but that's just the Twitter API returning the wrong code. In reality I'm correctly authenticating. I also know I can just move the except block to after the for-loop, but I just want to know how this can happen at all.

from twython import Twython
t = Twython(...)

# ...

for id in ids:
    try:
        # exception is truly caused by the following line
        followers = t.cursor(t.get_followers_ids, id=id)
    except:
        # this block is never run
        print("Exception with user " + str(id))
        continue

    # this line actually throws the exception, inexplicably
    for follower_id in followers:
        values.append((follower_id, id, scrape_datetime))

    # ...

The traceback:

Traceback (most recent call last):
File "/root/twitter/nightly.py", line 5, in <module>
    t.get_followers(t.current_tweeters)
File "/root/twitter/tweets.py", line 81, in get_followers
    for follower_id in followers:
File "/usr/local/lib/python3.3/dist-packages/twython-3.0.0-py3.3.egg/twython/api.py", line 398, in cursor
content = function(**params)
File "/usr/local/lib/python3.3/dist-packages/twython-3.0.0-py3.3.egg/twython/endpoints.py", line 212, in get_followers_ids
return self.get('followers/ids', params=params)
File "/usr/local/lib/python3.3/dist-packages/twython-3.0.0-py3.3.egg/twython/api.py", line 231, in get
return self.request(endpoint, params=params, version=version)
File "/usr/local/lib/python3.3/dist-packages/twython-3.0.0-py3.3.egg/twython/api.py", line 225, in request
content = self._request(url, method=method, params=params, api_call=url)
File "/usr/local/lib/python3.3/dist-packages/twython-3.0.0-py3.3.egg/twython/api.py", line 195, in _request
retry_after=response.headers.get('retry-after'))
twython.exceptions.TwythonAuthError: Twitter API returned a 401 (Unauthorized), An error occurred processing your request.

Answer 1

Looks like t.cursor(...) returns a generator that doesn't actually execute until you iterate through it. While it might appear that the connection happens at:

followers = t.cursor(t.get_followers_ids, id=id)

It doesn't until you iterate through the generator with your for loop. Sort of mentioned here

If you have a need to defer processing until later but still want to catch the exception, turn the generator into a list. This will exhaust the generator and save the data for later.

followers = t.cursor(t.get_followers_ids, id=id)
followers = list(followers)

Answer 2

What you probably got with

followers = t.cursor(t.get_followers_ids, id=id)

is a cursor to a piece of code that will fetch your list. But because this cursor is lazy no Twyton code is yet executed. The acual fetching code is only executed when it is first used .... at the line that throws your exception. So wrap that in the exception handling also.