How to achieve Concurrency in mysql when there are 500+ requests simultaneously to the server

StackOverflow https://stackoverflow.com/questions/22982215

  •  01-07-2023
  •  | 
  •  

Question

I am using Mysql (InnoDb), in this I am facing problem when we checked the curl request upto 500 connections at a time (simultaneously) but when we try more than that then my servers goes to idle condition and then we have to restart it.

In simply manner for more understanding I have a DB table in MYSQL (InnoDB) which contains 'remaining amount' field. Ex. field containing the values of $500 and the $10 will deduct per user.

Now when users request concurrently (at a time), server doesn't know which users has more priority or what to do when request is accepted but no more amount remains.

Here are two questions:

  1. How to manage the deductions when concurrent requests received
  2. How to set maximum requests for server so that will not be Idle after some more requests (500+)

Edit

If I suppose to use LOCK Mechanism of mysql then it would be work perfectly or not ?

I am using php script for communicating with mysql

Was it helpful?

Solution

Modern server environments can operate with a multitude of languages, processes, environments etc. which makes them easy to debug, monitor, change and combine in various ways.

If this request alone is putting big stress on your MySQL server, I would suggest a helper process to handle it in a single-threaded way. Below I present a very simple helper-server in Tcl and the glue code for Php. Other languages and communication protocols can be chosen for performance, but HTTP and Tcl are super-easy to understand, and can easily handle hundreds of requests per second.

This will handle your processing nicely even without locking. If the problem is with MySQL server not being able to process update requests at the incoming speed, then it is a hardware problem, and you need a completely different infrastructure with multiple database servers.

Php code delegates processing to the helper server:

<?php
$result = "FAILURE";
$result = file_get_contents('http://localhost:2222?id=123&amount=10');
echo "result=$result!";
?>

Tcl server is as follows, uncomment and put your actual processing in procedure "Respond":

package require mysqltcl
set db [mysqlconnect -host localhost -port 3306 -user root -db test -encoding utf-8 ]
proc sql {sql} {
    global db
    if [catch {::mysql::exec $db $sql} err] {puts $err}
};#sql
proc select {sql} {::mysql::sel $::db $sql -flatlist}

set listenSocket [socket -server Accept 2222]

proc Accept {sock addr port} {
   puts "Accepted $sock from $addr port $port"
   fconfigure $sock -buffering line
   fileevent $sock readable "Respond $sock"
}

proc Respond {sock} {
   gets $sock request
   puts request=$request!
   set result "ERROR IN THE INVOKING PHP PROGRAM"
   if [regexp {id=(\d+)&amount=(\d+)} $request x id amount] {
    puts "request for amount=$amount id=$id"
    #set balance [select "SELECT balance FROM `accounts` WHERE somefield=$id"]
    #sql "INSERT Table2(name,field1,field2) VALUES('xxx',$balance,$amount)"
    #sql "UPDATE `accounts` SET amount=amount-$amount WHERE id=$id LIMIT 1"
    #set result [select "SELECT * FROM `whatever` WHERE condition"]
    set result "{\"balance\": \"$balance\",\"value\":\"$result\"}";#PHP will get this string, choose convenient format
   }
   puts $sock "HTTP/1.1 200 OK\n\n$result"
   close $sock
   puts "closed $sock"
}

vwait forever

All the "puts" are for debugging and can be deleted. As I said, the communication can be optimized immensely (to Unix pipes, message queues, shared memory), C can be chosen instead of Tcl etc. I just chose what was simple for me.

The request of "?id=123&amount=10" is, of course, completely arbitrary and you can design a different protocol.

OTHER TIPS

  1. Use database sessions. This way you can get row from DB, do calculations in PHP (or any other lang) and return new value to DB without worrying about other users changing value in meantime.

  2. max_connections in file my.cnf (mysql configuration file). You can use this tool https://tools.percona.com/wizard to help you configure your DB. On Percona website you can find also many good texts about DB configuration and optimization.

Michael Blood's answer won't work because the sequence of SQL statements is not "atomic".

If you are only changing one table, it is much simpler. And it is safer than having to do multiple queries:

UPDATE foo SET amt = amt - 15 WHERE amt > 15 AND id = 1001;
if rows_affected > 0, then success

If you need multiple statements (such as take money out of one account, then add to another), you must add BEGIN...COMMIT around them.

BEGIN;
SELECT amt FROM foo WHERE id = 1001 FOR UPDATE;  -- Note the lock on that row
if $amt >= 15 then
    UPDATE foo SET amt = amt - 15 WHERE id = 1001;
    UPDATE foo SET amt = amt + 15 WHERE id = 2222;
end
COMMIT;

Connections should be disconnected after they have done what they need to. Assuming that the updates take only milliseconds (as they will), you can easily handle at least 100 transactions per second.

If you need more than 100/sec, look at innodb_flush_log_at_trx_commit.

You may use one buffer table (Temporary table) in the sense of concurrency control with the help of LOCK'ing mechanism of MySQL. So while one request's to the server on the priority base you can set remains request as in queue or in the sense restrict the table. It means no new entry will execute till the one's request has been complete successfully.

My understanding is that you are attempting to take the following steps

 a. Retrieve the balance from the database
 b. Verify that the balance is sufficient
 c. If the balance is sufficient, update the database with the new balance (less $10)
 d. Return true

THe problem occurs when you have two near simultaneous results and

 a. Request #1 retrieves the balance of $15 from the database
 a. Request #2 retrieves the balance of $15 from the database
 b. Request #1 verifies that the balance is sufficient  - more than $10
 b. Request #2 verifies that the balance is sufficient  - more than $10
 c. Request #1 updates the database with a new balance of $5 ($15 - $10)
 c. Request #2 updates the database with a new balance of $5 ($15 - $10)
 d. Request #1 Returns true
 d. Request #2 Returns true

A more 'thread safe' or 'transaction safe' method may be to update the database and then look at the balance. This method makes it so that you do not have to worry about the time between retrieving the value from the database and updating the database

 a. Update the balance in the database subtracting $10
 b. Retrieve the remaining balance and if it is greater than $0 return true
 c. If the balance is less than $0 put the money back into the database and return false

some very basic pseudo code on some invented tables with the users account.id of 1001:

update account set balance = balance - 10 where id = 1001
newbalance = select balance from account where id = 1001
if  newbalance > 0  
    return true
else
    update account set balance = balance + 10 where id = 1001
    return false
Licensed under: CC-BY-SA with attribution
Not affiliated with StackOverflow
scroll top