- Most computer arithmetic is done using either integers or floating-point
awk uses double-precision
- In the early 1990s Barbie mistakenly said, “Math class is tough!”
Although math isn’t tough, floating-point arithmetic isn’t the same
as pencil-and-paper math, and care must be taken:
- Not all numbers can be represented exactly.
- Comparing values should use a delta, instead of being done directly
with ‘==’ and ‘!=’.
- Errors accumulate.
- Operations are not always truly associative or distributive.
- Increasing the accuracy can help, but it is not a panacea.
- Often, increasing the accuracy and then rounding to the desired
number of digits produces reasonable results.
- Use -M (or --bignum) to enable MPFR
PREC to set the precision in bits, and
ROUNDMODE to set the IEEE 754 rounding mode.
- With -M,
arbitrary-precision integer arithmetic using the GMP library.
This is faster and more space-efficient than using MPFR for
the same calculations.
- There are several areas with respect to floating-point
gawk disagrees with the POSIX standard.
It pays to be aware of them.
- Overall, there is no need to be unduly suspicious about the results from
floating-point arithmetic. The lesson to remember is that floating-point
arithmetic is always more complex than arithmetic using pencil and
paper. In order to take advantage of the power of floating-point arithmetic,
you need to know its limitations and work within them. For most casual
use of floating-point arithmetic, you will often get the expected result
if you simply round the display of your final results to the correct number
of significant decimal digits.
- As general advice, avoid presenting numerical data in a manner that
implies better precision than is actually the case.