[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[IDV #BEV-647620]: Help for 3D plots in IDV



Hi Randy,

Here at Unidata we come across many netCDF files that are problematic
for one reason or another. I took the opportunity to turn this file into
a case study exercise that, with your permission, I would like to
publish on our Unidata developer blog. (If you don't want to, it is no
problem. We also often work with scientists who don't want their data
mentioned in blogs, for whatever reason).

In summary, I found the following problems:

- time dimension, but no time [coordinate variable]
- no units on the geophysical variables

Jumping to the punch line, here is an [OpenDAP link of the repaired
netCDF file]. You can open that in the IDV.

Also, about plotting your data in 3D, the data are 2D, but the IDV gives
you the capability of adding 3D relief to 2D variables. In the IDV
Dashboard, Field Selector tab, examine what is under the Displays > 3D
Surface. See the attached image for an example.

As an aside, I was surprised that you had gzipped the netCDF file when
you uploaded it. This should not be necessary, but then I did see it
compressed considerably. That is pretty odd and something we should look
into separately. netCDF should handle the compression much better than
that and so you should not have to gzip.

[coordinate variable]
https://www.ncl.ucar.edu/Document/Language/cv.shtml

[OpenDAP link of the repaired netCDF file]
http://motherlode.ucar.edu/repository/opendap/461efeff-a537-4900-bf47-c5d3962977d8/entry.das

Here is that case study (rough draft).

1 Snow Data Exercise
====================

1.1 Introduction
~~~~~~~~~~~~~~~~

  We recently came across a netCDF file containing two-dimensional
  gridded data pertaining to snow accumulation. The file was not CF
  compliant so this will be a case study on how to clean up a netCDF
  file with Python.


1.2 ncdump of Original Data
~~~~~~~~~~~~~~~~~~~~~~~~~~~

  I uploaded the original data to RAMADDA. Let's do an `ncdump -h' just
  to get our bearings.

  ,----
  | ncdump -h 
https://motherlode.ucar.edu/repository/opendap/780ff0cb-8a3d-45a6-a763-b0fc94140d0e/entry.das
  `----

  netcdf entry { dimensions: lat = 1059 ; lon = 1799 ; time = 13 ;
  variables: float lat(lat, lon) ; lat:long_name = "Latitude" ; float
  lon(lat, lon) ; lon:long_name = "Longitude" ; float fcst_raw(time,
  lat, lon) ; fcst_raw:long_name = "Forecast Raw Data" ;
  fcst_raw:_FillValue = -9999.f ; float obs_raw(time, lat, lon) ;
  obs_raw:long_name = "Observed Raw Data" ; obs_raw:_FillValue = -9999.f
  ; int fcst_object_id(time, lat, lon) ; fcst_object_id:long_name =
  "Forecast Object ID" ; fcst_object_id:_FillValue = -9999 ; int
  obs_object_id(time, lat, lon) ; obs_object_id:long_name = "Observed
  Object ID" ; obs_object_id:_FillValue = -9999 ; int
  fcst_cluster_id(time, lat, lon) ; fcst_cluster_id:long_name =
  "Forecast Cluster ID" ; fcst_cluster_id:_FillValue = -9999 ; int
  obs_cluster_id(time, lat, lon) ; obs_cluster_id:long_name = "Observed
  Cluster ID" ; obs_cluster_id:_FillValue = -9999 ;

  // global attributes: :FileOrigins = "File
      out/10/mtd_snow_20160123_150000V_obj.nc generated 20160125_225534
      UTC on host yorick by the MET MTD tool" ; :MET_version = "V5.1" ;
      :MET_tool = "MTD" ; :model = "snow" ; :obtype = "ANALYS" ;
      :Projection = "Lambert Conformal" ; :scale_lat_1 = "38.5" ;
      :scale_lat_2 = "38.5" ; :lat_pin = "21.128" ; :lon_pin = "-122.72"
      ; :x_pin = "0" ; :y_pin = "0" ; :lon_orient = "-97.5" ; :d_km = "3
      km" ; :r_km = "6371.2 km" ; :nx = "1799" ; :ny = "1059" ; }

  A couple of problems immediately stand out:

  - time dimension, but no time [coordinate variable]
  - no units on the geophysical variables


  [coordinate variable]
  https://www.ncl.ucar.edu/Document/Language/cv.shtml


1.3 Using Python to Clean Up Data
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

  In order to repair this file with Python, we will need to import a
  couple of libraries.

  ,----
  | import netCDF4
  | import numpy as np
  `----

  Let's open the dataset that needs to be repaired.

  ,----
  | ds = 
netCDF4.Dataset('https://motherlode.ucar.edu/repository/opendap/780ff0cb-8a3d-45a6-a763-b0fc94140d0e/entry.das')
  `----

  Also create file, `ncfile', that will have our fixed dataset.

  ,----
  | try:
  |     ncfile.close()  # just to be safe, make sure dataset is not already 
open.
  | except:
  |     pass
  | ncfile = netCDF4.Dataset('snow.nc',
  |                          mode='w',
  |                          format='NETCDF4_CLASSIC')
  `----

  Grab the `lat', `lon', `time' dimension sizes from the original
  dataset.

  ,----
  | lat_size  = ds.dimensions['lat'].size
  | lon_size  = ds.dimensions['lon'].size
  | time_size = ds.dimensions['time'].size
  `----

  According to the [CF documentation] or [NODC template] I do not think
  `cdm_data_type' `featureType' are strictly necessary. For the sake of
  clarity, during an `ncump' for example, I am adding them.

  ,----
  | ncfile.cdm_data_type = "Grid"
  | ncfile.featureType = "Grid"
  `----

  Create latitude, longitude, and time netCDF dimensions.

  ,----
  | lat_dim = ncfile.createDimension('lat_dim', lat_size )
  | lon_dim = ncfile.createDimension('lon_dim',  lon_size)
  | time_dim = ncfile.createDimension('time_dim', time_size)
  `----

  The original data has a `time' dimension, but no `time' coordinate
  variable so let's artificially create one and populate with fake data
  just so we have something we can work with.

  ,----
  | time = ncfile.createVariable('time', np.float64, ('time_dim',))
  | time.standard_name = 'time'
  | time.units = 'days since 2015-10-01 00:00:00'
  | time.long_name = 'time of measurement'
  | time[:] = np.arange(time_size)
  `----

  Create `latitude' variable and grab the latitude data from the
  original dataset.

  ,----
  | latitude = ncfile.createVariable('latitude', np.float32, 
('lat_dim','lon_dim',))
  | latitude.standard_name = 'latitude'
  | latitude.units = 'degrees_north'
  | latitude.long_name = 'latitude of the observation'
  | latitude[:] = ds['lat'][:]
  `----

  Create `longitude' variable and grab the longitude data from the
  original dataset.

  ,----
  | longitude = ncfile.createVariable('longitude', np.float32, 
('lat_dim','lon_dim',))
  | longitude.standard_name = 'longitude'
  | longitude.units = 'degrees_east'
  | longitude.long_name = 'longitude of the observation'
  | longitude[:] = ds['lon'][:]
  `----

  Create a `fcst_raw' ("Forecast Raw Data") geophysical variable. I did
  not notice any units here so I just assumed meters, again, so we have
  something to work with.

  ,----
  | fcst_raw = ncfile.createVariable('fcst_raw', np.float64,('time_dim', 
'lat_dim', 'lon_dim'))
  | fcst_raw.units = 'm'
  | fcst_raw.standard_name = 'fcst_raw' 
  | fcst_raw.coordinates = 'time lat lon'
  | fcst_raw[:] = ds['fcst_raw'][:]
  `----

  Finally close the datasets we've been working with.

  ,----
  | ncfile.close();
  | ds.close()
  `----

  Do an `ncdump' of our new file just to make sure every thing is
  hunky-dory.

  ,----
  | ncdump -h snow.nc
  `----

  netcdf snow { dimensions: lat_dim = 1059 ; lon_dim = 1799 ; time_dim =
  13 ; variables: double time(time_dim) ; time:standard_name = "time" ;
  time:units = "days since 2015-10-01 00:00:00" ; time:long_name = "time
  of measurement" ; float latitude(lat_dim, lon_dim) ;
  latitude:standard_name = "latitude" ; latitude:units = "degrees_north"
  ; latitude:long_name = "latitude of the observation" ; float
  longitude(lat_dim, lon_dim) ; longitude:standard_name = "longitude" ;
  longitude:units = "degrees_east" ; longitude:long_name = "longitude of
  the observation" ; double fcst_raw(time_dim, lat_dim, lon_dim) ;
  fcst_raw:units = "m" ; fcst_raw:standard_name = "fcst_raw" ;
  fcst_raw:coordinates = "time lat lon" ;

  // global attributes: :cdm_data_type = "Grid" ; :featureType = "Grid"
      ; }


  [CF documentation]
  http://cfconventions.org/cf-conventions/v1.6.0/cf-conventions.html

  [NODC template]
  https://www.nodc.noaa.gov/data/formats/netcdf/v2.0/grid.cdl


1.4 Plotting the Data with the matplotlib or the Unidata IDV
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

  TODO



> Hi again -
> 
> Ok, I uploaded a (compressed) data file.  It's called
> mtd_snow_20160123_150000V_obj.nc.gz
> 
> Randy
> 
> address@hidden> wrote:
> 
> > Randy,
> >
> > Please upload some sample data here:
> >
> > http://motherlode.ucar.edu/repository/alias/idvupload
> >
> > (I cannot tell if the IDV can handle the data just by looking at the nc
> > header,
> > unfortunately.)
> >
> > If you have control over how the netCDF file is generated, this is
> > definitely
> > possible. Ideally, the data should be CF compliant. Here is one example:
> >
> > https://www.nodc.noaa.gov/data/formats/netcdf/v2.0/grid.cdl
> >
> > though there is a lot of information here that is not required by the IDV,
> > even
> > the parts marked "REQUIRED".
> >
> > and more information here:
> >
> > http://cfconventions.org/cf-conventions/v1.6.0/cf-conventions.html
> >
> > But if you are lucky, the IDV may be able to read it in as is.
> >
> > Best,
> >
> > Unidata IDV Support
> >
> >
> > > Hi -
> > >
> > > I'm one of the developers for the Model Evaluation Tools project, and we
> > > have a tool that creates a 3D gridded dataset.  We have an in-house tool
> > to
> > > view the data, but we'd like to know if other tools such as IDV can plot
> > > this data, as well.
> > >
> > > The data is in a netcdf file ("ncdump -h" output is attached to this
> > > email).  Also attached is a plot using our in-house plotting software.
> > > This should give you some idea what we're shooting for.
> > >
> > > Is there some way to plot this data using IDV?  We're willing to consider
> > > making changes to the netcdf file in order to facilitate this, if needed.
> > >
> > > Any thoughts?
> > >
> > > Randy Bullock
> > >
> > >
> >
> >
> > Ticket Details
> > ===================
> > Ticket ID: BEV-647620
> > Department: Support IDV
> > Priority: Normal
> > Status: Closed
> >
> >
> 
> 


Ticket Details
===================
Ticket ID: BEV-647620
Department: Support IDV
Priority: Normal
Status: Closed

Attachment: snow.png
Description: PNG image


NOTE: All email exchanges with Unidata User Support are recorded in the Unidata inquiry tracking system and then made publicly available through the web. If you do not want to have your interactions made available in this way, you must let us know in each email you send to us.