An IP address has 32 bit, an RGB value only has 24 bit. That means that you have to discard some data. It would make sense to map the bits in a matter that the most significant bits in the IP address map to the most significant bits of the color space.
IP address are organized in a hierarchical manner. Large contiguous blocks are assigned to registrars of countries, which then split these blocks and assign them to ISPs, which then again split these blocks and assign them to companies, or to groups of clients, often based on geographical proximity. That means the first byte is the most significant and the last one is the least significant.
In RGB values, on the other hand, each color channel has technically-speaking equal significance. Technically. The human eye is most perceptive to green and least perceptive to blue, so green is more significant than red which is more significant than blue. In each color-channel, the significance of bits decreases.
That means I would map the 32 bit of the IP address to RGB in this manner:
12345678 90123456 78901234 56789012
GRBGRBGR BGRBGRBG RBGRBGRB --------
That means addresses from different networks will be colored very differently and addresses from the same subnet in similar shades. The smaller the distance between two addresses in network topology, the smaller their color difference. Addresses from the same /24 subnet will have exactly the same color. But when two IPs come from the same /24 subnet, they very likely belong to two machines from the same organization or come from an ISP address pool which is dynamically assigned to clients living in a rather small georgraphical area, so the chance that you are dealing with the same person is pretty high.