Disable database tables for writes but report writes as successful

Question

I manage a database where thousands of GBs of useless log data is created daily.

I can't fix the software that is performing all of this (unnecessary) logging. Is there a way to lock (or disable writes to) this log table so it's never written but at same time it will report the write as successful so it doesn't cause issues with the software?

I know I can just truncate the tables with a cronjob but I'd like to know if there is a better approach?

Answer 1

MySQL, and friends, i.e. MariaDB and Percona, have a rather unusual storage mechanism called the Black Hole engine - it might help with your particular request (see the fiddle here).

For example, if you do this:

CREATE TABLE foo
(
  bar INT NOT NULL AUTO_INCREMENT PRIMARY KEY,
  baz VARCHAR (30) NOT NULL
) ENGINE = InnoDB;
Query OK, 0 rows affected (0.07 sec)

The default Engine is InnoDB anyway, but I'm just being explicit for the sake of clarity here.

Then, we input a few records:

mysql> INSERT INTO foo (baz) VALUES ('afadfa'), ('sfasdfa'), ('asdfadf');
Query OK, 3 rows affected (0.01 sec)
Records: 3  Duplicates: 0  Warnings: 0

No problems there! Then, to check we:

mysql> SELECT * FROM foo;
+-----+---------+
| bar | baz     |
+-----+---------+
|   1 | afadfa  |
|   2 | sfasdfa |
|   3 | asdfadf |
+-----+---------+
3 rows in set (0.00 sec)

All's rosy in the garden!

Now, for the interesting part!

mysql> ALTER TABLE foo ENGINE = BLACKHOLE;
Query OK, 3 rows affected (0.10 sec)
Records: 3  Duplicates: 0  Warnings: 0

OK. We continue merrily INSERTing data as follows:

mysql> INSERT INTO foo (baz) VALUES ('xxxxx'), ('yyyyy'), ('zzzzzzz');
Query OK, 3 rows affected (0.00 sec)
Records: 3  Duplicates: 0  Warnings: 0

Again, all is fine and dandy. Database is happy, we're happy and, most importantly, third party software is blissfully happy and ignorant of any change in the environment!

BUT!!! Now, when we do this:

mysql> SELECT * FROM foo;
Empty set (0.00 sec)

Nada! Zilch! Rien! Nought...

Our data has met the Black Hole's event horizon from which nothing can escape! You can put as much data in as you like - it's like sending a file to /dev/null in *nix!

You asked:

Is there a way to lock this log table so it's never written but at same time it will report the write as successful so it doesn't cause issues with the software?

and now you have (see above) outwardly normal behaviour on the INSERT:

Query OK, 3 rows affected (0.00 sec)
    Records: 3  Duplicates: 0  Warnings: 0

So, we're golden! To all intents and purposes, the data being sent to the server is being faithfully stored, except it's not!

To the software writing these logs, it appears that all is hunky-dory but nothing gets written - it's really quite nifty.

This looks as if it was made for your requirement. In MariaDB, you need to load a plugin as explained here, but it seems pretty straightforward. From here:

the application will continue to function with no errors. But since the data is never actually written, all the extra I/O is gone.

Just a word of warning! I don't know what your software is that is writing all this log data to the server, but somebody, somewhere, sometime thought it was worthy of saving. Just make sure you're not throwing the baby out with the bathwater!

Also, if you have a support contract for this pesky software, you might want to check to see if you're potentially invalidating that contact by doing this?

As a final note, if you try something like this on the Black Hole table:

INSERT INTO foo (baz) VALUES (23, 'asdfds'), (33), (43, 'adsfd');

Result:

Column count doesn't match value count at row 1

So, even the errors are authentic, and your software will behave as expected if there's an error and will continue to (all outward appearances) be functioning normally if something like this happens.

You will notice that the fiddle is for MySQL 8 and not MariaDB. With MariaDB (on this fiddle) I get a permission denied error if I try to run

ALTER TABLE foo ENGINE = BLACKHOLE;  

I get message:

Unknown storage engine 'BLACKHOLE'

This is to be expected - from the MariaDB link:

The BLACKHOLE storage engine was installed by default until MariaDB 10.0. In MariaDB 10.1 and later, the storage engine's plugin will have to be installed.

Attempts to change the engine type and to install the plugin both failed (hardly suprising on a public-facing server).

INSTALL SONAME 'ha_blackhole';

gives:

INSERT command denied to user
'fiddle_PIIYJABQHVJRTTYOTJLC'@'localhost' for table 'plugin'