"

0.5 Precision and Rounding

Morgan Chase and Marilyn Nielson

Applications

Objectives

When you’ve finished this section, you will be able to perform the following:

  • Choose a level of precision for applied math calculations.
  • Round to that level of precision.

 

You’re a wildlife technician working on a behavioral study of Oregon spotted frogs (Rana pretiosa). You are looking at the time required for larvae to metamorphose into adults and the reaction time of adult frogs to prey items under a variety of conditions in the lab. What units of time will you use to measure the development of larvae into adult frogs? What units of time will you use to measure an adult frog’s reaction time? Are these units of measurement different? Why?

You’re measuring stream habitat using a meter tape. Do you record measurements to the nearest meter? The nearest centimeter? The nearest millimeter? If you record your measurements to centimeters, you might measure a riffle section as 5.38 meters long and 1.46 meters wide. Should you report the area of this riffle as 7.8548 square meters?

 

Oregon Spotted Frog in a close up view.
While surveying for Oregon spotted frog egg masses in Deschutes County, OR, field technicians for the USGS caught an adult female using a dip net. This was the first individual observed for the Spring field season.

 

 

 

 

 

 

 

 

 

 

 

 

In the first few chapters, we rarely concerned ourselves with rounding; we assumed that every number we were told was exact and we didn’t have to worry about any measurement error. However, every measurement contains some error. A standard sheet of paper is [latex]8.5[/latex] inches wide and [latex]11[/latex] inches high, but it’s possible that the actual measurements could be closer to [latex]8.4999[/latex] and [latex]11.0001[/latex] inches. Even if we measure something very carefully, with very sensitive instruments, we should assume that there could be some small measurement error.[1]

Exact Values and Approximations

A number is an exact value if it is the result of counting or a definition.

A number is an approximation if it is the result of a measurement or of rounding.

So, many of the numbers we work with as technicians arise from measurements. These will be approximations and their precision will depend on the instruments we use to measure.

Practice Exercises

Identify each number as an exact value or an approximation.

  1. An inch is [latex]\frac{1}{12}[/latex] of a foot.
  2. This board is [latex]78[/latex] inches long.
  3. There are [latex]14[/latex] students in class.
  4. A car’s tachometer reads [latex]3,000[/latex] rpm.
  5. A right angle measures [latex]90^\circ[/latex].
  6. The angle of elevation of a ramp is [latex]4^\circ[/latex].

Accuracy and Significant Figures

African-American women were vital to NASA’s success in the 1960s, as shown in the movie Hidden Figures. Photo by NASA/Kim Shiflett on flickr.

Because measurements are inexact, we need to consider how accurate they are. This requires us to think about significant figures—often abbreviated “sig figs” in conversation—which are the digits in the measurement that we trust to be correct. The accuracy of a number is equal to the number of significant figures. (By the way, the terms “significant digits” and “significant figures” are used interchangeably.) The following rules aren’t particularly difficult to understand but they can take time to absorb and internalize, so we’ll include lots of examples and exercises.

Rules for Significant Figures

  1. All nonzero digits are significant.
    Ex: [latex]12,345[/latex] has five sig figs, and [latex]123.45[/latex] has five sig figs.
  2. All zeros between other nonzero digits are significant.
    Ex: [latex]10,045[/latex] has five sig figs, and [latex]100.45[/latex] has five sig figs.
  3. Any zeros to the right of a decimal number are significant.
    Ex: [latex]123[/latex] has three sig figs, but [latex]123.00[/latex] has five sig figs.
  4. Zeros on the left of a decimal number are NOT significant.
    Ex: [latex]0.123[/latex] has three sig figs, and [latex]0.00123[/latex] has three sig figs.
  5. Zeros on the right of a whole number are NOT significant unless they are marked with an overbar.
    Ex: [latex]12,300[/latex] has three sig figs, but [latex]12,30\overline{0}[/latex] has five sig figs.

Another way to think about #4 and #5 above is that zeros that are merely showing the place value—where the decimal point belongs—are NOT significant.

Practice Exercises

Determine the accuracy (i.e., the number of significant figures) of each number.

  1. [latex]63,400[/latex]
  2. [latex]63,040[/latex]
  3. [latex]63,004[/latex]
  4. [latex]0.085[/latex]
  5. [latex]0.0805[/latex]
  6. [latex]0.08050[/latex]
Mt. Everest, Lohtse, and Nupse in the early morning. Photo by Ralf Kayser on flickr.

In 1856, the first official measurement of the height of Mount Everest—called Sagarmatha in Nepal and Chomolungma in Tibet—was announced. The height was determined to be exactly [latex]29,000[/latex] feet, but there was concern that people would think this was only a rough estimate rounded to the nearest thousand feet. Therefore, the height was announced as [latex]29,002[/latex] feet, so that everyone seeing that number would believe that the measurement was correct to the nearest foot.[2] Yes, to demonstrate the correctness of the measurement, an incorrect measurement was announced.

Instead of fudging a number like [latex]29,000[/latex] to show that it is correct to the nearest foot, we can write it with an an overbar to indicate that the zeros are significant. Putting [latex]29,00\overline{0}[/latex] in a newspaper headline in 1856 would probably have confused people, but you can handle it because you’re in a math class. Writing [latex]29,00\overline{0}[/latex] is our way of saying “Really, to the nearest foot, it’s exactly [latex]29,000[/latex] feet!”

Practice Exercises

Determine the accuracy (i.e., the number of significant figures) of each number.

  1. [latex]29,000[/latex]
  2. [latex]29,\overline{0}00[/latex]
  3. [latex]29,0\overline{0}0[/latex]
  4. [latex]29,00\overline{0}[/latex]

Two things to remember: we don’t put an overbar over a nonzero digit, and we don’t need an overbar for any zeros on the right of a decimal number because those are already understood to be significant.

Accuracy-Based Rounding

As we saw in Chapter 0.4, it is often necessary to round a number. We often round to a certain place value, such as the nearest hundredth, but there is another way to round. Accuracy-based rounding considers the number of significant figures rather than the place value.

Accuracy-based rounding:

  1. Locate the rounding digit to which you are rounding by counting from the left until you have the correct number of significant figures.
  2. Look at the test digit directly to the right of the rounding digit.
  3. If the test digit is 5 or greater, increase the rounding digit by 1 and drop all digits to its right. If the test digit is less than 5, keep the rounding digit the same and drop all digits to its right.

Practice Exercises

Round each number so that it has the indicated number of significant figures.

  1. [latex]51,837[/latex] (three sig figs)
  2. [latex]51,837[/latex] (four sig figs)
  3. [latex]4.2782[/latex] (two sig figs)
  4. [latex]4.2782[/latex] (three sig figs)

When the rounding digit of a whole number is a [latex]9[/latex] that gets rounded up to a [latex]0[/latex], we must write an overbar above that [latex]0[/latex].

Similarly, when the rounding digit of a decimal number is a [latex]9[/latex] that gets rounded up to a [latex]0[/latex], we must include the [latex]0[/latex] in that decimal place.

Practice Exercises

Round each number so that it has the indicated number of significant figures. Be sure to include trailing zeros or an overbar if necessary.

  1. [latex]13,997[/latex] (two sig figs)
  2. [latex]13,997[/latex] (three sig figs)
  3. [latex]2.596[/latex] (two sig figs)
  4. [latex]2.596[/latex] (three sig figs)

The height of Mount Everest has changed over the years due to plate tectonics and earthquakes. In December 2020, it was jointly announced by Nepal and China that the summit of Mount Everest has an elevation of [latex]29,031.69[/latex] ft.[3]

  1. Round [latex]29,031.69[/latex] ft to two sig figs.
  2. Round [latex]29,031.69[/latex] ft to three sig figs.
  3. Round [latex]29,031.69[/latex] ft to four sig figs.
  4. Round [latex]29,031.69[/latex] ft to five sig figs.
  5. Round [latex]29,031.69[/latex] ft to six sig figs.

Accuracy when Multiplying and Dividing

Suppose you needed to square the number [latex]3\frac{1}{3}[/latex]. You could rewrite [latex]3\frac{1}{3}[/latex] as the improper fraction [latex]\frac{10}{3}[/latex] and then figure out that [latex](\frac{10}{3})^2 = \frac{100}{9}[/latex], which equals the repeating decimal [latex]11.111...[/latex]

Because most people prefer decimals to fractions, we might decide to round [latex]3\frac{1}{3}[/latex] to [latex]3.33[/latex] and find that [latex]3.33^2=11.0889[/latex]. The answer [latex]11.0889[/latex] looks very accurate, but it is a false accuracy because there is round-off error involved. Only when we round to three sig figs do we get an accurate result: [latex]11.0889[/latex] rounded to three sig figs is [latex]11.1[/latex], which is accurate because [latex]11.111...[/latex] rounded to three sig figs is also [latex]11.1[/latex]. It turns out that because [latex]3.33[/latex] has only three significant figures, our answer must be rounded to three significant figures.

When multiplying or dividing approximate numbers, the answer must be rounded to the same number of significant figures as the least accurate of the original numbers.

Don’t round off the original numbers; do the necessary calculations first, then round the answer as your last step.

On the Job

How do you know where to round your measurement? Take a look at the instrument you are using to measure. What is its accuracy? For example, a ruler with [latex]\frac{1}{8}[/latex] marks on it is accurate to the nearest one-eighth of an inch. Record your measurements to the closest eighth. If you report your answer as a decimal, it will look like this:

[latex]3\frac{3}{8}[/latex] inches

[latex]3.375[/latex] inches

Report the answer to three decimal places, because that is the accuracy of this ruler.

[latex]3.4[/latex] inches is not the same as [latex]3\frac{3}{8}[/latex] inches

If you multiply two measurements to find the area – for example [latex]3\frac{3}{8}[/latex] inches by [latex]2\frac{1}{8}[/latex] inches – your calculator will read [latex]3.796875[/latex]. Is this the number you should report as the area? How many digits are significant?

(This calculation from your measurements should be reported as [latex]3.797[/latex])

A clinometer allows us to measure angles in the field.

A clinometer showing divisions of 5 degrees.
This Suunto brand clinometer has marks to show you its accuracy. Using this instrument, to what accuracy will your angles be measured? How do you know?

 

When in doubt, ask your supervisor to clarify the accuracy of your instruments and where you should round to.

 

 

Practice Exercises

Use a calculator to multiply or divide as indicated. Then round to the appropriate level of accuracy.

  1. [latex]8.75\cdot12.25[/latex]
  2. [latex]355.12\cdot1.8[/latex]
  3. [latex]77.3\div5.375[/latex]
  4. [latex]53.2\div4.5[/latex]
  5. Suppose you are filling a 5-gallon can of gasoline. The gasoline costs [latex]$4.579[/latex] per gallon, and you estimate that you will buy [latex]5.0[/latex] gallons. How much should you expect to spend?

Bonus material: Here is a comic strip from xkcd.com showing that including a lot of decimal digits can give a false sense of accuracy.

More on latitude and longitude precision in Chapter 8.1!

 

a comic strip showing that it's silly to give latitude and longitude coordinates with lots of decimal digits
From the web comic xkcd.

 

Practice Exercise Answers

  1. Here is an example of a speed camera with gigantic measurement error: https://youtube.com/shorts/cCYy29RuhV8 ↵
  2. Source: https://en.wikipedia.org/wiki/Mount_Everest#cite_ref-tas1982_33-0 ↵
  3. Source: https://www.washingtonpost.com/world/asia_pacific/mount-everest-height-nepal-china/2020/12/08/a7b3ad1e-389a-11eb-aad9-8959227280c4_story.html ↵
definition

License

Icon for the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License

0.5 Precision and Rounding Copyright © by Morgan Chase and Marilyn Nielson is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License, except where otherwise noted.

Share This Book