MySQL Precison Issues in DECIMAL NUMERIC data type

Question

In writing a function for scientific application, I ran into issues. I traced it back to MySQL's lack of precison.

Here is the page from the official documentation which claims that The maximum number of digits for DECIMAL is 65 - http://dev.mysql.com/doc/refman/5.6/en/fixed-point-types.html . It also describes how the value will be rounded if it exceeds the specified precison.

Here is reproducible code (a mysql stored function) to test it -

  DELIMITER $$
  DROP FUNCTION IF EXISTS test$$
  CREATE FUNCTION test
      (xx DECIMAL(30,25)
      )
      RETURNS DECIMAL(30,25)
      DETERMINISTIC
    BEGIN
      DECLARE result DECIMAL(30,25);
      SET result = 0.339946499848118887e-4;
      RETURN(result);
   END$$
   DELIMITER ;

If you save the code above in a file called test.sql, you can run it by executing the following in mysql prompt -

source test.sql;
select test(0);

It produces the output -

+-----------------------------+
| test(0)                     |
+-----------------------------+
| 0.0000339946499848118900000 |
+-----------------------------+
1 row in set (0.00 sec)

As you can see, the number is getting rounded at the 20th digit, and then five zeroes are being added to it to get to the required/specified precison. That is cheating.

Am I mistaken, or is the documentation wrong?

Answer 1

I don't know anything about SQL, but my guess would be this line:

  SET result = 0.339946499848118887e-4;

If MySQL is anything like other languages I know, then this will first evaluate the right-hand side, and then assign the value to result. No matter what type result is declared to be or what precision it's declared to have, it wouldn't matter if the right-hand side has already lost precision when being evaluated. This is almost surely what is happening here.

I can reproduce your results, but If I change that line to

  SET result = cast('0.339946499848118887e-4' as decimal(30, 25));

(casting from a string instead of from a floating-point constant of unspecified precision) then I correctly get

+-----------------------------+
| test(0)                     |
+-----------------------------+
| 0.0000339946499848118887000 |
+-----------------------------+
1 row in set (0.00 sec)

as desired. So that's your fix.

---

BTW, the documentation that scale in DECIMAL(precision, scale) cannot be greater than 30 seems to be in section 12.19.2. DECIMAL Data Type Changes:

The declaration syntax for a DECIMAL column is DECIMAL(M,D). The ranges of values for the arguments in MySQL 5.6 are as follows:

M is the maximum number of digits (the precision). It has a range of 1 to 65. (Older versions of MySQL permitted a range of 1 to 254.)

D is the number of digits to the right of the decimal point (the scale). It has a range of 0 to 30 and must be no larger than M.

Answer 2

This happens because mysql treats 0.339946499848118887e-4 as float and treats 0.0000339946499848118887 as fixed point.

mysql> select cast(  0.339946499848118887e-4 as DECIMAL(30, 25));
+----------------------------------------------------+
| cast(  0.339946499848118887e-4 as DECIMAL(30, 25)) |
+----------------------------------------------------+
|                        0.0000339946499848118900000 |
+----------------------------------------------------+
1 row in set (0.00 sec)

mysql> select cast(  0.0000339946499848118887 as DECIMAL(30, 25));
+-----------------------------------------------------+
| cast(  0.0000339946499848118887 as DECIMAL(30, 25)) |
+-----------------------------------------------------+
|                         0.0000339946499848118887000 |
+-----------------------------------------------------+
1 row in set (0.00 sec)

As described in the mysql documentation on precision math - expression handling -

If any approximate values are present, the expression is approximate and is evaluated using floating-point arithmetic.

Quoting from, the documentation on numerical types,

Two numbers that look similar may be treated differently. For example, 2.34 is an exact-value (fixed-point) number, whereas 2.34E0 is an approximate-value (floating-point) number.