selecting observation of datetime64[ns] type in particular time range

Question

I have a pandas dataframe(dfnew) in which one column(timestamp) is of datetime64[ns] type. Now I want to see how many observations are in particular time range lets say 10:00:00 to 12:00:00.

    dfnew['timestamp'] = dfnew['timestamp'].astype('datetime64[ns]')
    dfnew['timestamp]
0    2013-12-19 09:03:21.223000
1    2013-12-19 11:34:23.037000
2    2013-12-19 11:34:23.050000
3    2013-12-19 11:34:23.067000
4    2013-12-19 11:34:23.067000
5    2013-12-19 11:34:23.067000
6    2013-12-19 11:34:23.067000
7    2013-12-19 11:34:23.067000
8    2013-12-19 11:34:23.067000
9    2013-12-19 11:34:23.080000
10   2013-12-19 11:34:23.080000
11   2013-12-19 11:34:23.080000
12   2013-12-19 11:34:23.080000
13   2013-12-19 11:34:23.080000
14   2013-12-19 11:34:23.080000
15   2013-12-19 11:34:23.097000
16   2013-12-19 11:34:23.097000
17   2013-12-19 11:34:23.097000
18   2013-12-19 11:34:23.097000
19   2013-12-19 11:34:23.097000
Name: timestamp

    dfnew['Time']=dfnew['timestamp'].map(Timestamp.time)
    t1 = datetime.time(10, 0, 0)
    t2 = datetime.time(12, 0, 0)
    print len(dfnew[t1<dfnew["Time"]<t2])

This produce an error TypeError: can't compare datetime.time to Series. I am new to pandas dataframe. I guess I am making a very silly mistake here.Any help appreciated.

Answer 1

You can use the DatetimeIndex indexer_between_time method, so a trick here to make use of it is to pass the Series / column to the DatetimeIndex constructor:

from datetime import time

# s is your datetime64 column

In [11]: pd.DatetimeIndex(s).indexer_between_time(time(10), time(12))
Out[11]: 
array([ 1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19])

This gets the position of the times between 10 and 12 (inclusive*), so use iloc to filter:

In [12]: s.iloc[pd.DatetimeIndex(s).indexer_between_time(time(10), time(12))]
Out[12]: 
1    2013-12-19 11:34:23.037000
2    2013-12-19 11:34:23.050000
3    2013-12-19 11:34:23.067000
4    2013-12-19 11:34:23.067000
5    2013-12-19 11:34:23.067000
6    2013-12-19 11:34:23.067000
7    2013-12-19 11:34:23.067000
8    2013-12-19 11:34:23.067000
9    2013-12-19 11:34:23.080000
10   2013-12-19 11:34:23.080000
11   2013-12-19 11:34:23.080000
12   2013-12-19 11:34:23.080000
13   2013-12-19 11:34:23.080000
14   2013-12-19 11:34:23.080000
15   2013-12-19 11:34:23.097000
16   2013-12-19 11:34:23.097000
17   2013-12-19 11:34:23.097000
18   2013-12-19 11:34:23.097000
19   2013-12-19 11:34:23.097000
Name: timestamp, dtype: datetime64[ns]

* include_start and include_end are optional boolean arguments of indexer_between_time.