sprintf : fixed point, big numbers and precision loss

Question

I need to read some numbers in a database and write them into a text file using Perl.

In the table where are the numbers, the data format is defined as numeric (25,5) (it reads 25 digits, including 5 decimals).

I format the numbers in my file with a sprintf "%.5f", $myvalue to force 5 decimals and I just noticed that for greats values, there is a precision loss for numbers with more than 17 digits :

db   = 123.12345
file = 123.12345 (OK)

db   = 12345678901234891.12345
file = 12345678901234892.00000 (seems to be rounded to upper integer)

db   = 12345678901234567890.12345
file = 12345678901234567000.00000 (truncation ?)

What is Perl's greatest precision for fixed decimal numbers?

I am aware of the concepts and limitations of floating point arithmetic in general, but I am not a Perl monk and I do not know the internals of Perl so I don't know if it is normal (or if it is related at all to floating point). I am not sure either if it is a internal limitation of Perl, or a problem related to the sprintf processing.

Is there a workaround or a dedicated module that could help with that problem?

Some notable points :

• this is an additional feature of a system that already uses Perl, so using another tool is not an option
• the data being crunched is financial so I need to keep every cent and I cannot cope with a +/- 10 000 units precision :^S

Answer 1

Once again, I am finding a solution right after asking SO. I am putting my solution here, to help a future visitor :

replace

$myout = sprintf "%.5f", $myvalue;

by

use Math::BigFloat;
$myout = Math::BigFloat->new($myvalue)->ffround( -5 )->bstr;

Answer 2

Without modules like Math::BigFloat, everything above 16 digits is pure magic... e.g.

perl -e 'printf "*10^%02d: %-.50g\n", $_, log(42)*(10**$_) for (0..20)'

produces

*10^00: 3.7376696182833684112267746968427672982215881347656
*10^01: 37.376696182833683224089327268302440643310546875
*10^02: 373.76696182833683224089327268302440643310546875
*10^03: 3737.6696182833684360957704484462738037109375
*10^04: 37376.6961828336861799471080303192138671875
*10^05: 373766.96182833681814372539520263671875
*10^06: 3737669.6182833681814372539520263671875
*10^07: 37376696.18283368647098541259765625
*10^08: 373766961.82833683490753173828125
*10^09: 3737669618.283368587493896484375
*10^10: 37376696182.83368682861328125
*10^11: 373766961828.33685302734375
*10^12: 3737669618283.36865234375
*10^13: 37376696182833.6875
*10^14: 373766961828336.8125
*10^15: 3737669618283368.5
*10^16: 37376696182833688
*10^17: 373766961828336832
*10^18: 3737669618283368448
*10^19: 37376696182833684480
*10^20: 373766961828336828416

Answer 3

What is Perl's greatest precision for fixed decimal numbers?

Perl doesn't have fixed point decimal numbers. Very few languages do, actually. You could use a module like Math::FixedPoint, though

Answer 4

Perl is storing your values as floating-point numbers internally.¹ The precision is dependent on how your version of Perl is compiled, but it's probably a 64-bit double.

C:\>perl -MConfig -E "say $Config::Config{doublesize}"
8

A 64-bit double-precision float² has a 53-bit significand (a.k.a. fraction or mantissa) which gives it approximately 16 decimal characters of precision. Your database is defined as storing 25 characters of precision. You'll be fine if you treat the data as a string but if you treat it as a number you'll lose precision.

Perl's bignum pragma provides transparent support for arbitrarily large numbers. It can slow things down considerably so limit its use to the smallest possible scope. If you want big floats only (without making other numeric types "big") use Math::BigFloat instead.

<sub>1. Internally, perl uses a datatype called an SV that can hold floats, ints, and/or strings simultaneously.
2. Assuming IEEE 754 format.</sub>

Answer 5

Alternatively, if you're just transferring the values from the database to a text file and not operating on them as numbers, then have the DB format them as strings. Then read and print them as strings (perhaps using "printf '%s'"). For example:

select Big_fixed_point_col(format '-Z(24)9.9(5)')(CHAR(32))