Next: Inverse and Hyperbolic, Previous: General Mode Commands, Up: Mode Settings

The `p` (`calc-precision`

) command controls the precision to
which floating-point calculations are carried. The precision must be
at least 3 digits and may be arbitrarily high, within the limits of
memory and time. This affects only floats: Integer and rational
calculations are always carried out with as many digits as necessary.

The `p` key prompts for the current precision. If you wish you
can instead give the precision as a numeric prefix argument.

Many internal calculations are carried to one or two digits higher
precision than normal. Results are rounded down afterward to the
current precision. Unless a special display mode has been selected,
floats are always displayed with their full stored precision, i.e.,
what you see is what you get. Reducing the current precision does not
round values already on the stack, but those values will be rounded
down before being used in any calculation. The `c 0` through
`c 9` commands (see Conversions) can be used to round an
existing value to a new precision.

It is important to distinguish the concepts of precision and
accuracy. In the normal usage of these words, the number
123.4567 has a precision of 7 digits but an accuracy of 4 digits.
The precision is the total number of digits not counting leading
or trailing zeros (regardless of the position of the decimal point).
The accuracy is simply the number of digits after the decimal point
(again not counting trailing zeros). In Calc you control the precision,
not the accuracy of computations. If you were to set the accuracy
instead, then calculations like ‘`exp(100)`’ would generate many
more digits than you would typically need, while ‘`exp(-100)`’ would
probably round to zero! In Calc, both these computations give you
exactly 12 (or the requested number of) significant digits.

The only Calc features that deal with accuracy instead of precision
are fixed-point display mode for floats (`d f`; see Float Formats),
and the rounding functions like `floor`

and `round`

(see Integer Truncation). Also, `c 0` through `c 9`
deal with both precision and accuracy depending on the magnitudes
of the numbers involved.

If you need to work with a particular fixed accuracy (say, dollars and
cents with two digits after the decimal point), one solution is to work
with integers and an “implied” decimal point. For example, $8.99
divided by 6 would be entered `899 <RET> 6 /`, yielding 149.833
(actually $1.49833 with our implied decimal point); pressing `R`
would round this to 150 cents, i.e., $1.50.

See Floats, for still more on floating-point precision and related issues.