Updating Database every X seconds [closed]

Question

Closed. This question needs to be more focused. It is not currently accepting answers.

---

Want to improve this question? Update the question so it focuses on one problem only by editing this post.

Closed 8 years ago.

Improve this question

First off, I'm building the application in Android, and using PHP as my web-services and database communication, I'm using MySQL database.

Alright, here's the question: I'm building a Near Real-Time application and I need to update a row in my Database every 5 seconds, diffrent row for each user in the same table.. What is the best way to optimize the costs? I mean, It's working fine for 10 users.. but my guess is when I'll get 50-100 the DB will crash. And another important issue is the bandwidth, I really want to use as less as I can.

Any advice? If I'll put all the requests into a queue intead of updating the specific row, will it prevent it from crashing? I mean.. there still will be 50 connections to the DB in the same time..

Answer 1

Ok, so - bandwidth and database usage.

The first is easy. You may want to try and determine what's your optimal values. Some things to keep in mind from the very start:

• Keep your protocol light. Check JSON - it's human-readable and yet compact enough. Or you may want to implement a binary load.
• You may want to try and throw some flow control in the mix:
  • Client-side heuristics: What's my current write latency? Under X milliseconds, keep updating each 5 seconds. Over X, 10 seconds. Over 5x X, 30 seconds.
  • Server-side Quality of Service (QoS): Processor load over X%? Push a message to the clients, tell them to update every 10 seconds instead of 5.

About the database - it can be tricky, but nothing is impossible. Some thoughts:

• Cache your data access; if you're asking for record N, and there was no update for it, no need to hit the database again - just read from a memory cache.
• Use in-memory objects to read/write, commit to database once the processor load is low, a time limit is reached, or N operations where processed.

Answer 2

You may want to use LZ4 (fast data compression) for keeping the bandwidth limited.

There should also be an implementation for Java and maybe even one written in PHP

For the MySQL server you should have some RAM (24-64Gb wild guess) and InnoDB engine (so innodb buffer pool can cache data/index)

If you have MySQL 5.6 you should enable memcache with InnoDB so you can query without the overhead off "complex" SQL parsing..

Oracle claims queries can be handled 7-9 times faster with this "NoSQL" interface

For an high write environment you should really have an RAID 10 Array with 10K/15K disks or SSD disk(s) to handle the write load, but it also depends on the table structure..

Watch out with RAID 5 because you get an write penalty because the parity bits needs to be calculated and written to every disk. RAID 10 almost keeps the write speed of RAID 0 and protects your data with an mirror (RAID 1)

Your MySQL will always handle an incoming connection as an thread so 100 queries at the same time means 100 threads needs to be created to handle that load.. So you should correctly configure your threads in the MySQL configuration.