The table is full - MariaDB

Question

I am getting the following error whist trying to execute a long running query.

The table 'C:\Windows\SERVIC~2\NETWOR~1\AppData\Local\Temp#sql1664_349_19be' is full

The C drive (NTFS) on the server has 135GB free space. The D drive (NTFS) which holds the data has 365GB free out of 800GB.

The server has 32GB RAM.

The query I am running is reasonably simple but it is run against 61 million rows.

I have 18 indexes on tblinvoice and the table is INNODB.

SELECT
    InvoiceNo,
    NetAmountAbs,
    InvoiceDate
  FROM tblinvoice
  WHERE GrossAmountAbs >= 200
  GROUP BY InvoiceNo,
           NetAmountAbs,
           InvoiceDate
  HAVING COUNT(*) > 1
  AND SUM(CASE WHEN NetAmount_Doc >= 0 THEN 1 ELSE 0 END) > 1
  AND MIN(AccountNumber) != MAX(AccountNumber)
  AND MAX(GrossAmountAbs) != MIN(GrossAmountAbs)
  AND MAX(ImportID) = 2

This is my.ini file

[mysqld] 
log_bin_trust_function_creators=1 
datadir=E:/MariaDB 10.2.8
port=3306 
sql_mode="STRICT_TRANS_TABLES,NO_ENGINE_SUBSTITUTION"
default_storage_engine=innodb 
innodb_buffer_pool_size=25000M
character-set-server=utf8 
innodb_io_capacity=200
innodb_read_io_threads=64 
innodb_thread_concurrency=0
innodb_write_io_threads=64 
max_connections=1000
innodb_lock_wait_timeout=50 
lower_case_table_names=2
event_scheduler=on 
character-set-server=utf8 
max_allowed_packet=100M
connect_timeout=10 
innodb-log-file-size=1G
innodb_log_buffer_size=64M
tmp_table_size=2G
max_heap_table_size=2G 
[client] 
port=3306 
plugin-dir=C:/ProgramFiles/MariaDB 10.2/lib/plugin

Thanks

EDIT: Updated my.ini with suggested changes to innodb-log-file-size, innodb_log_buffer_size, tmp_table_size, max_heap_table_size,

Answer 1

Please do not get fooled by the error message.

Note that the error is table is full. It does not say disk is full.

What would create a table is full condition ?

It has to do with the changes that are pouring into the rollback segments for a transaction.

I have addressed this many times

• May 31, 2013 : How to solve "The table ... is full" with "innodb_file_per_table"?
• Mar 31, 2014 : mysql directory grow to 246G after one query, which failed due to table is full
• Jun 16, 2014 : MySQL Index creation failing on table is full
• Apr 07, 2015 : MySQL query INSERT INTO ... failed - The Table "x" is full

It appears that one of the following occurred

• The SELECT was in the middle of a transaction
• The SELECT attempted to hold too much rollback info in the face of many transactions

In either case, the tblinvoice table probably had so many changes pending that the SELECT just had to give up. This could have caused all write transactions against tblinvoice to rollback.

If you were running this SELECT on a busy Master with lots of writes against tblinvoice, then I could this SELECT having this problem.

What you should do is set up a reporting slave (running MySQL Replication) and run this type of query on it instead on the main server.

UPDATE 2018-02-27 10:10 EDT

Your problem may be the amount of RAM free.

You currently have this

tmp_table_size=2G
max_heap_table_size=2G

Try running the query again so that the temp table goes immediately to disk

set tmp_table_size = 1024 * 1024 * 16;
set max_heap_table_size = 1024 * 1024 * 16;
SELECT
    InvoiceNo,
    NetAmountAbs,
    InvoiceDate
  FROM tblinvoice
  WHERE GrossAmountAbs >= 200
  GROUP BY InvoiceNo,
           NetAmountAbs,
           InvoiceDate
  HAVING COUNT(*) > 1
  AND SUM(CASE WHEN NetAmount_Doc >= 0 THEN 1 ELSE 0 END) > 1
  AND MIN(AccountNumber) != MAX(AccountNumber)
  AND MAX(GrossAmountAbs) != MIN(GrossAmountAbs)
  AND MAX(ImportID) = 2;

Keep watch on the C:\Windows\SERVIC~2\NETWOR~1\AppData\Local folder. Watch the temp file that shows up in that folder. What I am hoping is that the temp table will reach the needed size on disk. Why ?

With tmp_table_size=2G and max_heap_table_size=2G, the temp table has to reach 2G in RAM before transferring to disk. My working theory is that you do not have 2GB of RAM free (We are talking Windows, right ???). This is why I am suggesting lowering these to 16M (which is the default value anyway). Give it a try !!!

Answer 2

tmp_table_size=2G
max_heap_table_size=2G 

Those are dangerously large. Keep them to no more than about 1% of RAM -- about 300M. Those settings are used in tmp tables inside complex SELECTs; your query is such. If several complex queries are running at the same time, several RAM allocations of 2GB could be happening simultaneously, thereby blowing out RAM.

Temp#sql1664_349_19be looks like such a tmp table name.

ibdata1 can be arbitrarily large -- limited only by disk space (and some really big hard limit, in the terabytes, or an OS limit). Did you have autoextend on ibdata1? 34025472K is more than 32G, so I don't see it as being an OS limit.

It might be useful to see SHOW CREATE TABLE.

Please provide EXPLAIN SELECT ...

Answer 3

I don't think locating big amount of RAM to MySQL parameters like,

sort_buffer_size=
join_buffer_size=
read_buffer_size=
read_rnd_buffer_size
tmp_table_size
max_heap_table_size

, is a good idea, what I know is reserving big size of RAM is a slow process, so big amount of values will slow down MySQL, and this is what I found on previous tests, so even if the system has plenty of RAM, I don't think locating high amount of RAM to similar parameters is a good idea.