IndexError: list index out of range?

Question

I keep getting a list index out of range error when trying to run this code, the code parses a sites tables by going through its pages and inputs the data into an excel sheet.

The error is being given on revenue = cols[0].string:

from urllib.request import urlopen
from bs4 import BeautifulSoup
from openpyxl import Workbook
from openpyxl.cell import get_column_letter
import datetime


now = datetime.datetime.now()

wb = Workbook()
dest_filename = r'iOS Top Grossing Data.xlsx'
ws = wb.active
ws = wb.create_sheet()
ws.title = now.strftime("%m-%d-%y")
sh = wb.get_sheet_by_name('Sheet')
wb.remove_sheet(sh)  

ws['A1'] = "REVENUE"
ws.column_dimensions['A'].width = 11
ws.cell('A1').style.alignment.horizontal = 'center'
ws.cell('A1').style.font.bold = True

ws['B1'] = "FREE"
ws.column_dimensions['B'].width = 7
ws.cell('B1').style.alignment.horizontal = 'center'
ws.cell('B1').style.font.bold = True

ws['C1'] = "PAID"
ws.column_dimensions['C'].width = 7
ws.cell('C1').style.alignment.horizontal = 'center'
ws.cell('C1').style.font.bold = True

ws['D1'] = "GAME"
ws.column_dimensions['D'].width = 27
ws.cell('D1').style.alignment.horizontal = 'center'
ws.cell('D1').style.font.bold = True

ws['E1'] = "PRICE"
ws.column_dimensions['E'].width = 7
ws.cell('E1').style.alignment.horizontal = 'center'
ws.cell('E1').style.font.bold = True

ws['F1'] = "REVENUE"
ws.column_dimensions['F'].width = 11
ws.cell('F1').style.alignment.horizontal = 'center'
ws.cell('F1').style.font.bold = True

ws['G1'] = "ARPU INDEX"
ws.column_dimensions['G'].width = 15
ws.cell('G1').style.alignment.horizontal = 'center'
ws.cell('G1').style.font.bold = True

ws['H1'] = "DAILY NEW USERS"
ws.column_dimensions['H'].width = 17
ws.cell('H1').style.alignment.horizontal = 'center'
ws.cell('H1').style.font.bold = True

ws['I1'] = "DAILY ACTIVE USERS"
ws.column_dimensions['I'].width = 19
ws.cell('I1').style.alignment.horizontal = 'center'
ws.cell('I1').style.font.bold = True

ws['J1'] = "ARPU"
ws.column_dimensions['J'].width = 7
ws.cell('J1').style.alignment.horizontal = 'center'
ws.cell('J1').style.font.bold = True

ws['K1'] = "RANK CHANGE"
ws.column_dimensions['K'].width = 14
ws.cell('K1').style.alignment.horizontal = 'center'
ws.cell('K1').style.font.bold = True

page = 0

while page < 6:
        page += 1
        url = "http://thinkgaming.com/app-sales-data/?page=" + str(page) 
        html = str(urlopen(url).read()) 

        soup = BeautifulSoup(html) 
        table = soup.find("table")

        counter = 0

        while counter < 51:      
                        rows = table.findAll('tr')[counter]
                        cols = rows.findAll('td')

                        revenue = cols[0].string
                        revenue = revenue.replace('\\n', '')
                        revenue = revenue.replace(' ', '') 

                        free = cols[1].string
                        free = free.replace('\\n', '')
                        free = free.replace(' ', '') 

                        paid = cols[2].string
                        paid = paid.replace('\\n', '')
                        paid = paid.replace(' ', '') 

                        game = cols[3].string

                        price = cols[4].string
                        price = price.replace('\\n', '')
                        price = price.replace(' ', '') 

                        revenue2 = cols[5].string
                        revenue2 = revenue2.replace('\\n', '')
                        revenue2 = revenue2.replace(' ', '') 

                        dailynewusers = cols[6].string
                        dailynewusers = dailynewusers.replace('\\n', '')
                        dailynewusers = dailynewusers.replace(' ', '') 

                        cell_location = counter
                        cell_location += 1

                        ws['A'+str(cell_location)] = revenue

                        counter += 1

wb.save(filename = dest_filename)             

Here's the traceback:

Traceback (most recent call last): 
File "C:\Users\shiver_admin\Desktop\script.py", line 89, in <module> revenue = cols[0].string IndexError: list index out of range

Answer 1

Same as the comments, you are not getting any <td> tags simply because they don't exist, especially for index [0]. The first <tr> tag in that table is this:

If you notice, it has headers inside. Basically, you should start your counter at 1 instead of 0.

Another way to be sure that you're getting the correct rows is checking if they have classes. If you noticed, the proper <tr> rows have classes inside them (odd and even). You can employ something like table.find_all("tr", class_=True) to get them.

Sample code (note: coded in Python 2.7, but easy enough to modify to suit Python 3.x):

import requests as rq
from bs4 import BeautifulSoup as bsoup

url = "http://thinkgaming.com/app-sales-data/?page=1"
r = rq.get(url)
soup = bsoup(r.content)

table = soup.find("table", class_="table")

rows = table.find_all("tr", class_=True)
cols = [td.get_text().strip().encode("utf-8") for td in rows[0].find_all("td")]

print cols

Result:

['1', '10', '-', 'Clash of Clans', 'Free', 'n/a', '44,259']
[Finished in 2.8s]

Let us know if this helps.