# [netcdfgroup] Precision an dSignificant digits in netcdf

```Hi All:

This has come up in the R list.  it is referring to  NCEP/NCAR Reanalysis 2
data [1] that the user obtained.  here is what he wrote:

> Thank you very much Jeff.  I think I’m too far to be able to explain myself.
> Perhaps, this is the wrong list for this question but I sent it in hope there
> is someone has deep understanding of netcdf data and use R. Let me tell the
> story simpler. Assume that you read a numeric vector of data from a netcdf
> file:
>
> data <- c(9.1999979, 8.7999979, 7.9999979, 3.0999980, 6.1000018, 10.1000017,
> 10.4000017, 9.2000017)
>
> you know that the values above are a model output and also you know that,
> physically, first and last values must be equal but somehow they are not.
>
> And now, you want to use “periodic” spline for the values above.
>
> spline(1:8, data, method = “periodic”)
>
> Voila! spline method throws a warning message: “spline: first and last y
> values differ - using y[1] for both”. Then I go on digging and discover 2
> attributes in netcdf file: “precision = 2” and “least_significant_digit = 1”.
> And I also found their definitions at [1].
>
> precision -- number of places to right of decimal point that are significant,
> based on packing used. Type is short.
> least_significant_digit -- power of ten of the smallest decimal place in
> unpacked data that is a reliable value. Type is short.
>
> Please, do not condemn me, english is not my main language :). At this point,
> as a scientist, what would you do according to explanations above? I think I
> didn’t exactly understand the difference between precision and
> least_significant_digit. One says “significant” and latter says “reliable”.
> Should I round the numbers to 2 decimal places or 1 decimal place after
> decimal point?

The data comes out that way because of the way R encodes floating points,  and
for this case in R the correct thing would be to do:

> round(data, digits = 2)

But as the user later wrote:

> For instance, If you check the header information of omega.2015.nc file it
> says;
>
> \$ ncdump -h omega.2015.nc
>
> ...
> omega:precision = 3s;
> omega:least_significant_digit = 3s;
> …
>
> So, I need to round values to 3 decimal places after point?
>
> and if you check the output of rhum.2015.nc;
>
> \$ ncdump -h rhum.2015.nc
> ...
> rhum:precision = 2s ;
> rhum:least_significant_digit = 0s ;
> …
>
> Then I need to round values to 2 decimal places after point?
>
> Should I accomplish the rounding operation according to precision or
> least_significant_digit attributes?

the response.

Thanks,

-Roy

1- http://www.esrl.noaa.gov/psd/data/gridded/data.ncep.reanalysis2.html
**********************
"The contents of this message do not reflect any position of the U.S.
Government or NOAA."
**********************
Roy Mendelssohn
Supervisory Operations Research Analyst
NOAA/NMFS
Environmental Research Division
Southwest Fisheries Science Center
Santa Cruz, CA 95060
Phone: (831)-420-3666
Fax: (831) 420-3980
e-mail: Roy.Mendelssohn@xxxxxxxx www: http://www.pfeg.noaa.gov/

"Old age and treachery will overcome youth and skill."
"From those who have been given much, much will be expected"
"the arc of the moral universe is long, but it bends toward justice" -MLK Jr.

```
• 2016 messages navigation, sorted by:
• Search the `netcdfgroup` archives: