Next: Arithmetic Tutorial Exercise 2, Previous: Modes Tutorial Exercise 4, Up: Answers to Exercises [Contents][Index]

Dividing two integers that are larger than the current precision may
give a floating-point result that is inaccurate even when rounded
down to an integer. Consider ‘`123456789 / 2`’ when the current
precision is 6 digits. The true answer is ‘`61728394.5`’, but
with a precision of 6 this will be rounded to
‘`12345700. / 2. = 61728500.`’.
The result, when converted to an integer, will be off by 106.

Here are two solutions: Raise the precision enough that the
floating-point round-off error is strictly to the right of the
decimal point. Or, convert to Fraction mode so that ‘`123456789 / 2`’
produces the exact fraction ‘`123456789:2`’, which can be rounded
down by the `F` command without ever switching to floating-point
format.