Tweaking MySQL configuration to support hundreds of connections locking the same table

Question

I have a resources table with 2000 rows. Engine is innodb. There is a 'free_at' field (indexed). On each request I need to lock the table, get a free resource (ordered by 'free_at' column), update that row to non free, and release the lock.

This is a basic pool implementation that I've been using and had worked fine with 100-200 connections and less than 1000 rows (resources in the pool).

Right now there are about 800 processes that are constantly requesting resources from the table (each every 10-15 seconds, so the average is up to 80\s).

My bottle neck is the lock wait time, which is ranging between 30 to 60 seconds (!) for each request. I'm sure there is some configuration I should change to make it lock and release faster.

I've tried changing the engine type to MEMORY but that didn't improve the lock wait time.

Should I look for another pool solution that is not MySQL based and can dispense resources by priority (which in my case is the 'free_at' field)?

EDIT:

for locking I use LOCK TABLES table_name WRITE

Then selecting SELECT * FROM table_name WHERE (free_at < NOW() OR free_at is null) ORDER BY free_at ASC

Updating the 'free_at' field UPDATE table_name SET free_at = NOW() + INTERVAL 5 MINUTE WHERE id= 1234

Finally unlocking UNLOCK TABLES

Table schema

`resources` (
  `id` int(11) NOT NULL AUTO_INCREMENT,
  `free_at` datetime DEFAULT NULL,
  PRIMARY KEY (`id`),
  KEY `free_at` (`free_at`),
) ENGINE=InnoDB

Answer 1

I managed to solve this issue by internally buffering resources. Instead of fetching 1 resource at a time, I'm fetching 5 of them, thus reducing the amount of calls and locks on that table by a factor of 5.

Answer 2

You need to lock rows in your table, not the whole table. Locking the whole table doesn't scale up, as you have discovered.

Stick with InnoDB. The MEMORY and MyISAM access methods don't do this.

Assuming your resource table has the following columns (it can have others):

resource_id int not null primary key
free        int not null 1 means free, 0 means in use
free_at     timestamp not null 

And assuming you want a transaction which will grab the oldest free resource row (free = 1), here's what you do from each client needing a resource.

 START TRANSACTION;

 SELECT resource_id
   FROM resource
  WHERE free = 1
  ORDER BY free_at ASC
  LIMIT 1
    FOR UPDATE;

At this point your application will get either a single resource_id, which is the one you can allocate, or it will get none. If it gets none, that means it has to wait and try again.

If you got a resource_id, update it to indicate it's in use.

 UPDATE resource
    SET free = 0,
        free_at = NOW()
  WHERE resource_id = 'the resource ID you just got';

Then, as soon as you can, do

 COMMIT;

to complete the transaction you started and release the lock on that row. This way of allocating resources works better than locking the table because each connection only needs to lock its own row.

If you don't get a resource id from the SELECT ... FOR UPDATE query, you need immediately to do

 ROLLBACK;

to cancel the transaction you started on that row. Then you need to wait and try again. Wait for a meaningful amount of time: at least the time it takes your application to use one of your resources and release it. If you wait a shorter amount of time, your many connections will hammer away on the resource table and slow things down. If you need more resources, add some.

When your software is done with the resource it allocated, do this to release it back to the pool. You can just do this operation in auto-commit mode; there's no need for an explicit transaction here. But do make sure your clients have autocommit mode turned on for this.

 UPDATE resource
    SET free = 1
  WHERE resource_id = 'the resource ID you have been using';

Please note; I don't understand exactly how you're using free_at so I may have that part of the logic wrong.

The SELECT operation finding the free resource will very likely be accelerated by adding a composite index, a covering index, on

 (free, free_at, resource_id)