How to generate pandas DataFrame column of Categorical from string column?

Question

I can convert a pandas string column to Categorical, but when I try to insert it as a new DataFrame column it seems to get converted right back to Series of str:

train['LocationNFactor'] = pd.Categorical.from_array(train['LocationNormalized'])

>>> type(pd.Categorical.from_array(train['LocationNormalized']))
<class 'pandas.core.categorical.Categorical'>
# however it got converted back to...
>>> type(train['LocationNFactor'][2])
<type 'str'>
>>> train['LocationNFactor'][2]
'Hampshire'

Guessing this is because Categorical doesn't map to any numpy dtype; so do I have to convert it to some int type, and thus lose the factor labels<->levels association? What's the most elegant workaround to store the levels<->labels association and retain the ability to convert back? (just store as a dict like here, and manually convert when needed?) I think Categorical is still not a first-class datatype for DataFrame, unlike R.

(Using pandas 0.10.1, numpy 1.6.2, python 2.7.3 - the latest macports versions of everything).

Answer 1

The only workaround for pandas pre-0.15 I found is as follows:

• column must be converted to a Categorical for classifier, but numpy will immediately coerce the levels back to int, losing the factor information
• so store the factor in a global variable outside the dataframe

.

train_LocationNFactor = pd.Categorical.from_array(train['LocationNormalized']) # default order: alphabetical

train['LocationNFactor'] = train_LocationNFactor.labels # insert in dataframe

[UPDATE: pandas 0.15+ added decent support for Categorical]

Answer 2

The labels<->levels is stored in the index object.

• To convert an integer array to string array: index[integer_array]
• To convert a string array to integer array: index.get_indexer(string_array)

Here is some exampe:

In [56]:

c = pd.Categorical.from_array(['a', 'b', 'c', 'd', 'e'])

idx = c.levels

In [57]:

idx[[1,2,1,2,3]]

Out[57]:

Index([b, c, b, c, d], dtype=object)

In [58]:

idx.get_indexer(["a","c","d","e","a"])

Out[58]:

array([0, 2, 3, 4, 0])