Multiply number of distances in distance matrix prior to histogram binning

Question

I am using scipy.spatial.distance.pdist to calculate the distances from an array of coordinates followed by numpy.histogram to bin the results. Currently this treats each coordinate as though one object were there, however I have multiple objects at that same coordinate. One option is to change the arrays so that each coordinate occurs multiple times, once for each object at that coordinate, however this would substantially increase the size of the array and the time of calculation for pdist, since it scales as N^2, and this is prohibitively costly and speed is important in this application.

A second approach would be to treat the resulting distance matrix such that each distance is repeated ninj times, where ni is the number of objects at coordinate i and nj the number of objects at coordinate j. This would transform the original MxM distance matrix to be a NxN distance matrix, where M is the total number of coordinates in the array, but N is the total number of objects. But again, this seems to be unnecessarily costly since all I really need to do is somehow tell the histogramming function to multiply the number of events at distance ij by ninj. In other words, is there any way to tell numpy.histogram that there's not just one object at distance ij, but that there's ni*nj objects instead?

Other ideas are obviously welcome.

Edit:

This is an example of the first approach.

import numpy as np
from scipy import spatial
import matplotlib.pyplot as plt

#create array of 5 coordinates in 3D
coords = np.random.random(15).reshape(5,3)
'''array([[ 0.66500534,  0.10145476,  0.92528492],
       [ 0.52677892,  0.07756804,  0.50976737],
       [ 0.50030508,  0.37635556,  0.20828815],
       [ 0.02707651,  0.21878467,  0.55855427],
       [ 0.81564621,  0.82750694,  0.53083443]])'''

#number of objects at each coordinate
objects = np.random.randint(1,10,5)
#array([5, 3, 8, 5, 1])

#create new array with coordinates for each individual object
new_coords = np.zeros((objects.sum(),3))

#there's surely a simpler way to do this
j=0
for coord in range(coords.shape[0]):
    for i in range(objects[coord]):
            new_coords[j] = coords[coord]
            j+=1

'''new_coords
array([[ 0.66500534,  0.10145476,  0.92528492],
       [ 0.66500534,  0.10145476,  0.92528492],
       [ 0.66500534,  0.10145476,  0.92528492],
       [ 0.66500534,  0.10145476,  0.92528492],
       [ 0.66500534,  0.10145476,  0.92528492],
       [ 0.52677892,  0.07756804,  0.50976737],
       [ 0.52677892,  0.07756804,  0.50976737],
       [ 0.52677892,  0.07756804,  0.50976737],
       [ 0.50030508,  0.37635556,  0.20828815],
       [ 0.50030508,  0.37635556,  0.20828815],
       [ 0.50030508,  0.37635556,  0.20828815],
       [ 0.50030508,  0.37635556,  0.20828815],
       [ 0.50030508,  0.37635556,  0.20828815],
       [ 0.50030508,  0.37635556,  0.20828815],
       [ 0.50030508,  0.37635556,  0.20828815],
       [ 0.50030508,  0.37635556,  0.20828815],
       [ 0.02707651,  0.21878467,  0.55855427],
       [ 0.02707651,  0.21878467,  0.55855427],
       [ 0.02707651,  0.21878467,  0.55855427],
       [ 0.02707651,  0.21878467,  0.55855427],
       [ 0.02707651,  0.21878467,  0.55855427],
       [ 0.81564621,  0.82750694,  0.53083443]])''' 

#calculate distance matrix of old and new arrays
distances_old = distance.pdist(coords)
distances_new = distance.pdist(new_coords)

#calculate and plot normalized histograms (typically just use np.histogram without plotting)
plt.hist(distances_old, range=(0,1), alpha=.5, normed=True)
(array([ 0.,  0.,  0.,  0.,  2.,  1.,  2.,  2.,  2.,  1.]), array([ 0. ,  0.1,  0.2,  0.3,  0.4,  0.5,  0.6,  0.7,  0.8,  0.9,  1. ]), <a list of 10 Patch objects>)

plt.hist(distances_new, range=(0,1), alpha=.5, normed=True)
(array([ 2.20779221,  0.        ,  0.        ,  0.        ,  1.68831169,
        0.64935065,  2.07792208,  2.81385281,  0.34632035,  0.21645022]), array([ 0. ,  0.1,  0.2,  0.3,  0.4,  0.5,  0.6,  0.7,  0.8,  0.9,  1. ]), <a list of 10 Patch objects>)

plt.show()

The second approach would instead treat the distance matrix rather than the coordinate matrix, but I haven't figured that code out yet.

Both approaches seem inefficient to me and I think manipulating the binning process of np.histogram is more likely to be efficient, since it's just basic multiplication, but I'm not sure how to tell np.histogram to treat each coordinate as having a variable number of objects to count.

Answer 1

Something like this might work:

from scipy.spatial import distance

positions = np.random.rand(10, 2)
counts = np.random.randint(1, 5, len(positions))

distances = distance.pdist(positions)
i, j = np.triu_indices(len(positions), 1)

bins = np.linspace(0, 1, 10)
h, b = np.histogram(distances, bins=bins, weights=counts[i]*counts[j])

It checks out compared to repeating, excepting the 0-distances:

repeated = np.repeat(positions, counts, 0)
rdistances_r = distance.pdist(repeated)

hr, br = np.histogram(rdistances, bins=bins)

In [83]: h
Out[83]: array([11, 22, 27, 43, 67, 46, 40,  0, 19,  0])

In [84]: hr
Out[84]: array([36, 22, 27, 43, 67, 46, 40,  0, 19,  0])