Transforming a List to a Pivot Table with Python

Question

I have a list of lists in the following format:

listA = [[142L, u'Work Load', [57.35, 19.57]],
[142L, u'Days', [84.0, 44.0]],
[142L, u'Payed', [5684.0, 3944.0]],
[547L, u'Work Load', [87.25, 12.70]],
[547L, u'Days', [98.0, 128.0]],
[547L, u'Payed', [3247.0, 4712.0]],
...]

and I want to transform it to:

listB = [['id', u'Work Load', u'Days', u'Payed'],
[142L, 57.35, 84.0, 5684.0],
[142L, 19.57, 44.0, 3944.0],
[547L, 87.25, 98.0, 3247.0],
[547L, 12.70, 128.0, 4712.0],
...]

How can I group them? The reason I am doing this is because I am trying to export the list to csv. Any ideas? Thanks!

Answer 1

Here is a working solution to my problem:

    import pandas as pd
    dt = [[142L, u'Work Load', [57.35, 19.57]],
          [142L, u'Days', [84.0, 44.0]],
          [142L, u'Payed', [5684.0, 3944.0]],
          [547L, u'Work Load', [87.25, 12.70]],
          [547L, u'Days', [98.0, 128.0]],
          [547L, u'Payed', [3247.0, 4712.0]]]
    dfA = pd.DataFrame(dt)
    dfA.columns = [u'id','field','data']
    dfB = dfA.groupby(u'id').apply(
        lambda grp: pd.DataFrame(zip(*grp['data']), columns=grp['field']))
    dfB.index = dfB.index.droplevel(-1)
    data = StringIO()
    dfB.to_csv(data, sep=';', encoding='utf-8')
    self.response = HttpResponse(data.getvalue(), mimetype='text/csv')
    self.add_response_headers()
    self.response.close()
    self.response.flush()
    return self.response

Thanks for your help.

Answer 2

Since the list you showed is sorted by "id", you can use itertools.groupby directly and zip:

In [189]: lst  #if not sorted, lst.sort(key=lambda x: x[0]) first
Out[189]: 
[[142L, u'Work Load', [57.35, 19.57]],
 [142L, u'Days', [84.0, 44.0]],
 [142L, u'Payed', [5684.0, 3944.0]],
 [547L, u'Work Load', [87.25, 12.7]],
 [547L, u'Days', [98.0, 128.0]],
 [547L, u'Payed', [3247.0, 4712.0]]]

In [190]: lstB=[['id', u'Work Load', u'Days', u'Payed'],]
     ...: for k, g in itertools.groupby(lst, lambda x: x[0]):
     ...:     t=zip(*(i[-1] for i in g))
     ...:     for i in t:
     ...:         lstB.append([k]+list(i))

#outputs:
In [587]: lstB
Out[587]: 
[['id', u'Work Load', u'Days', u'Payed'],
 [142L, 57.35, 84.0, 5684.0],
 [142L, 19.57, 44.0, 3944.0],
 [547L, 87.25, 98.0, 3247.0],
 [547L, 12.7, 128.0, 4712.0]]

Answer 3

Using pandas:

import pandas as pd

listA = [[142L, u'Work Load', [57.35, 19.57]],
[142L, u'Days', [84.0, 44.0]],
[142L, u'Payed', [5684.0, 3944.0]],
[547L, u'Work Load', [87.25, 12.70]],
[547L, u'Days', [98.0, 128.0]],
[547L, u'Payed', [3247.0, 4712.0]]]

dfA = pd.DataFrame(listA)
dfA.columns = ['id','field','data']

dfB = dfA.groupby('id').apply(
    lambda grp: pd.DataFrame(zip(*grp['data']), columns=grp['field']))
dfB.index = dfB.index.droplevel(-1)
print(dfB)

yields the DataFrame

field  Work Load  Days  Payed
id                           
142        57.35    84   5684
142        19.57    44   3944
547        87.25    98   3247
547        12.70   128   4712

Then you could write the DataFrame to a CSV using

dfB.to_csv('/tmp/test.csv', sep=',')

which would look like this:

id,Work Load,Days,Payed
142,57.35,84.0,5684.0
142,19.57,44.0,3944.0
547,87.25,98.0,3247.0
547,12.7,128.0,4712.0