You'll need to define your own metric, which handles "time" in an appropriate way. In the docs for scipy.spatial.distance.pdist you can define your own function
Y = pdist(X, f)
Computes the distance between all pairs of vectors in X using the user supplied 2-arity function f. [...] For example, Euclidean distance between the vectors could be computed as follows:
dm = pdist(X, lambda u, v: np.sqrt(((u-v)**2).sum()))
The metric can be passed to any scipy clustering algorithm, via the metric
keyword. For example, using linkage
:
scipy.cluster.hierarchy.linkage(y, method='single', metric='euclidean')