how to know if WAN IP has changed and receive massive IP address broadcasts

Question

The system I am developing potentially has a very large number of clients (lets say one million) that need to periodically update a central server with some information. Clients are written in Java.

The specific use-case is that the server backend needs to have an up to date mapping of IP address to clients. But the client IPs are dynamic and subject to (effectively random) change.

The solution I have in mind requires the clients to ping the server to update their IP. The period ideally should be once every minute, but even 1 ping/10 mins is acceptable.

My questions, in sequence:

1. 1M pings per 1 min is over 10k/sec. So first off I want to know the approaches can scale to handle such a load. This is to know the options available.

2. Assuming you have more than one solution in mind, which of these would be the most economical? The cost effectiveness is critically important. I don't have my own data center or static and fat end-point on the net, so the server application will need to run on some sort of provider or ultimately on the cloud.

Notes:

• I considered running the server from home using my own ISP provided connection, but I am neither sure of the performance issues, nor what my ISP will think about a constant stream of pings.

• I can't see how the server can auto-discover these IP changes.

Answer 1

Erik, your problem is much simpler than it seems to have been made to sound.

This problem been around for a decade maybe two. No need to re-invent the wheel here.

Why Polling/Pinging is a Bad Idea

The dynamic IPs provided by ISPs can have a variable lease time, but will often be at least 24-72 hours. Pinging your server every 1-10m will be a horrible waist of resources potentially making over a 4,320 useless HTTP requests PER CLIENT in a 72 hour period. Each request will be say around 300 bytes * 4,320 wasted http requests equals 1.3mb wasted bandwidth multiplied by your target client count of 1 million clients, you are talking about a monthly wasted bandwidth of ~1.2 TB! And that's just the wasted bandwidth, not the other bandwidth you might need to run your app and provide useful info.

The clients need to be smarter than just pinging frequently. Rather they should be able to check if their IP address matches the DNS on startup, then only when the IP changes, send a notification to the server. This will cut down your bandwidth and server processing requirements by thousands of times.

What you are describing is Dynamic DNS

What you are talking about is "Dynamic DNS" (both a descriptive name for the technology and also the name of one company that provides a SaaS solution).

Dynamic DNS is quite simply a DNS server that allows you to very rapidly change the mapping between a name and an IP address. Normally this is useful for devices using an ISP which only provides dynamic IPs. Whenever the IP changes for the router/server on a dynamic IP it will inform the Dynamic DNS server of the change.

• The defacto standard protocol for dynamic DNS is well documented. Start here: DNS Update API, I think the specifics you are looking for are here: DynDNS Perform Update. Most commercial implementations out there are very close to the same protocol due to the fact that router hardware usually has a built in DynDNS client which everyone wants to use.
• Most routers (even cheap ones) already have Dynamic DNS clients built into them. (You can write your own soft client, but the router is likely the most efficient location for this as your clients are likely being NAT'd with a private IP - you can still do it but at a cost of more bandwidth for public IP discovery)
• A quick google search for "dynamic DNS java client" brings up full source projects like this one: Java DynDNS client (untested, just illustrating the power of search)

Other Considerations for your System Architecture

Lets say the IP-client mapping thing gets resolved. You figured it all out and it works perfectly, you always knows the IP for each client. Would you then have a nice reliable system for transferring files to clients from mobile devices? I would say no.

Both mobiles and home computers can have multiple connection types, Wi-Fi, Cellular Data, maybe wired data. Each of these networks may have different security systems in place. So a connection from a cellular data mobile to a wifi laptop behind a home router is going to look very different than a wifi mobile device connecting to laptop on the same wifi network.

You may have physical router firewalls to contend with. Also home computers may have windows firewall enabled, maybe norton internet security, maybe symantec, maybe AVG, maybe zone alarm, etc... Do you know the firewall considerations for all these potential clients?

Answer 2

Maybe you could use SIP as protocol for that purpose ? Probably the java SIP libs already solved your problem.

Nice app by the way.

Answer 3

I would suggest better tweak you java program to know the IP change and then only hit the web service.

You can do it like,

1. on your java program initiation extract the IP of machine and store it in Global variable or better some property file.
2. Run a batch process/scheduler which will check your IP every 30sec/1 minute for change.Java Quartz Scheduler will come very handy for you.
3. Invoke the web service in case of a change of IP.

This way it reduces your server role and thus traffic and connections.

Answer 4

You could create your own protocol on top of UDP, for example XML based. Define 3 messages:

• request - client requests a challenge from server
• challenge - server replies with challenge (basically a random number)
• response - client sends username and hashed password + challenge back to the server

It's lightweight and not too traffic-heavy. You can load-balance it to multiple servers at any layer or using load-balancer.

Any average PC could handle million such hits per minute, provided you do server-side in C/C++ (I don't know about java network performance)

Answer 5

Please have a look at how no-ip works. Your requirement is exactly same as what it does.

Answer 6

Do I have the use case right? A community of users all want to receive pictures from each other? You don't want to host the images on the server but broadcast them directly to all the users?

There are two questions here. The first question is "how to know if my own WAN IP address has changed."

If you are not NATed then:

InetAddress.getLocalHost()

will tell you your IP address.

If you are NATed, then using dynamic DNS and resolving your own host name will work.

The second question is something like "How to share pictures between hosts which come and go on the internet".

The possible solution space includes:

IP Multicast, probably with Forward Error Correction and Carouseling, e.g. FLUTE.

File Swarming - e.g. bittorrent.

A Publish/Subscribe message bus solution using Jabber, AMQP, JMS, STOMP or similar. Suitable implementations include RabbitMQ, ActiveMQ, etc. JMS Topics are a key concept here.

The solution should avoid the massive overheads of doing things at the IP level.