Enable full pandas summary on a huge data frame

Question

I have a large data frame. Usually, when I have a data frame like this I get the summary for that data frame, where I get the info of how many non-NaN values in each column and column names. However for this one I get an even shorter summary:

<class 'pandas.core.frame.DataFrame'>
Index: 138289 entries, 1993-07-23 to 2012-11-26
Columns: 101 entries, AAT to ZZT
dtypes: object(101)

I'd like to get a standard summary, with info about each column.

I'm using ipython notebook and pandas 0.9.1 if that has anything to do with it

Answer 1

DataFrame.info unfortunately is hacked to not display the full summary unless there are fewer than 100 columns (look at the source code). We'll get it fixed for 0.10:

https://github.com/pydata/pandas/issues/2524

Answer 2

Try setting the maximum rows displayed in a DataFrame using set_printoptions:

pd.set_printoptions(max_columns=101)

This should allow you to see all of the columns in the (summarized) DataFrame.

The max_rows and max_columns control how many rows and columns of DataFrame objects are shown by default.

Answer 3

when the dataframe is large it does not display in ipython notebook . I just force it to:

from IPython.display import HTML
HTML(df.head().to_html())

make sure to use head :)