Hi Randy, Here at Unidata we come across many netCDF files that are problematic for one reason or another. I took the opportunity to turn this file into a case study exercise that, with your permission, I would like to publish on our Unidata developer blog. (If you don't want to, it is no problem. We also often work with scientists who don't want their data mentioned in blogs, for whatever reason). In summary, I found the following problems: - time dimension, but no time [coordinate variable] - no units on the geophysical variables Jumping to the punch line, here is an [OpenDAP link of the repaired netCDF file]. You can open that in the IDV. Also, about plotting your data in 3D, the data are 2D, but the IDV gives you the capability of adding 3D relief to 2D variables. In the IDV Dashboard, Field Selector tab, examine what is under the Displays > 3D Surface. See the attached image for an example. As an aside, I was surprised that you had gzipped the netCDF file when you uploaded it. This should not be necessary, but then I did see it compressed considerably. That is pretty odd and something we should look into separately. netCDF should handle the compression much better than that and so you should not have to gzip. [coordinate variable] https://www.ncl.ucar.edu/Document/Language/cv.shtml [OpenDAP link of the repaired netCDF file] http://motherlode.ucar.edu/repository/opendap/461efeff-a537-4900-bf47-c5d3962977d8/entry.das Here is that case study (rough draft). 1 Snow Data Exercise ==================== 1.1 Introduction ~~~~~~~~~~~~~~~~ We recently came across a netCDF file containing two-dimensional gridded data pertaining to snow accumulation. The file was not CF compliant so this will be a case study on how to clean up a netCDF file with Python. 1.2 ncdump of Original Data ~~~~~~~~~~~~~~~~~~~~~~~~~~~ I uploaded the original data to RAMADDA. Let's do an `ncdump -h' just to get our bearings. ,---- | ncdump -h https://motherlode.ucar.edu/repository/opendap/780ff0cb-8a3d-45a6-a763-b0fc94140d0e/entry.das `---- netcdf entry { dimensions: lat = 1059 ; lon = 1799 ; time = 13 ; variables: float lat(lat, lon) ; lat:long_name = "Latitude" ; float lon(lat, lon) ; lon:long_name = "Longitude" ; float fcst_raw(time, lat, lon) ; fcst_raw:long_name = "Forecast Raw Data" ; fcst_raw:_FillValue = -9999.f ; float obs_raw(time, lat, lon) ; obs_raw:long_name = "Observed Raw Data" ; obs_raw:_FillValue = -9999.f ; int fcst_object_id(time, lat, lon) ; fcst_object_id:long_name = "Forecast Object ID" ; fcst_object_id:_FillValue = -9999 ; int obs_object_id(time, lat, lon) ; obs_object_id:long_name = "Observed Object ID" ; obs_object_id:_FillValue = -9999 ; int fcst_cluster_id(time, lat, lon) ; fcst_cluster_id:long_name = "Forecast Cluster ID" ; fcst_cluster_id:_FillValue = -9999 ; int obs_cluster_id(time, lat, lon) ; obs_cluster_id:long_name = "Observed Cluster ID" ; obs_cluster_id:_FillValue = -9999 ; // global attributes: :FileOrigins = "File out/10/mtd_snow_20160123_150000V_obj.nc generated 20160125_225534 UTC on host yorick by the MET MTD tool" ; :MET_version = "V5.1" ; :MET_tool = "MTD" ; :model = "snow" ; :obtype = "ANALYS" ; :Projection = "Lambert Conformal" ; :scale_lat_1 = "38.5" ; :scale_lat_2 = "38.5" ; :lat_pin = "21.128" ; :lon_pin = "-122.72" ; :x_pin = "0" ; :y_pin = "0" ; :lon_orient = "-97.5" ; :d_km = "3 km" ; :r_km = "6371.2 km" ; :nx = "1799" ; :ny = "1059" ; } A couple of problems immediately stand out: - time dimension, but no time [coordinate variable] - no units on the geophysical variables [coordinate variable] https://www.ncl.ucar.edu/Document/Language/cv.shtml 1.3 Using Python to Clean Up Data ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ In order to repair this file with Python, we will need to import a couple of libraries. ,---- | import netCDF4 | import numpy as np `---- Let's open the dataset that needs to be repaired. ,---- | ds = netCDF4.Dataset('https://motherlode.ucar.edu/repository/opendap/780ff0cb-8a3d-45a6-a763-b0fc94140d0e/entry.das') `---- Also create file, `ncfile', that will have our fixed dataset. ,---- | try: | ncfile.close() # just to be safe, make sure dataset is not already open. | except: | pass | ncfile = netCDF4.Dataset('snow.nc', | mode='w', | format='NETCDF4_CLASSIC') `---- Grab the `lat', `lon', `time' dimension sizes from the original dataset. ,---- | lat_size = ds.dimensions['lat'].size | lon_size = ds.dimensions['lon'].size | time_size = ds.dimensions['time'].size `---- According to the [CF documentation] or [NODC template] I do not think `cdm_data_type' `featureType' are strictly necessary. For the sake of clarity, during an `ncump' for example, I am adding them. ,---- | ncfile.cdm_data_type = "Grid" | ncfile.featureType = "Grid" `---- Create latitude, longitude, and time netCDF dimensions. ,---- | lat_dim = ncfile.createDimension('lat_dim', lat_size ) | lon_dim = ncfile.createDimension('lon_dim', lon_size) | time_dim = ncfile.createDimension('time_dim', time_size) `---- The original data has a `time' dimension, but no `time' coordinate variable so let's artificially create one and populate with fake data just so we have something we can work with. ,---- | time = ncfile.createVariable('time', np.float64, ('time_dim',)) | time.standard_name = 'time' | time.units = 'days since 2015-10-01 00:00:00' | time.long_name = 'time of measurement' | time[:] = np.arange(time_size) `---- Create `latitude' variable and grab the latitude data from the original dataset. ,---- | latitude = ncfile.createVariable('latitude', np.float32, ('lat_dim','lon_dim',)) | latitude.standard_name = 'latitude' | latitude.units = 'degrees_north' | latitude.long_name = 'latitude of the observation' | latitude[:] = ds['lat'][:] `---- Create `longitude' variable and grab the longitude data from the original dataset. ,---- | longitude = ncfile.createVariable('longitude', np.float32, ('lat_dim','lon_dim',)) | longitude.standard_name = 'longitude' | longitude.units = 'degrees_east' | longitude.long_name = 'longitude of the observation' | longitude[:] = ds['lon'][:] `---- Create a `fcst_raw' ("Forecast Raw Data") geophysical variable. I did not notice any units here so I just assumed meters, again, so we have something to work with. ,---- | fcst_raw = ncfile.createVariable('fcst_raw', np.float64,('time_dim', 'lat_dim', 'lon_dim')) | fcst_raw.units = 'm' | fcst_raw.standard_name = 'fcst_raw' | fcst_raw.coordinates = 'time lat lon' | fcst_raw[:] = ds['fcst_raw'][:] `---- Finally close the datasets we've been working with. ,---- | ncfile.close(); | ds.close() `---- Do an `ncdump' of our new file just to make sure every thing is hunky-dory. ,---- | ncdump -h snow.nc `---- netcdf snow { dimensions: lat_dim = 1059 ; lon_dim = 1799 ; time_dim = 13 ; variables: double time(time_dim) ; time:standard_name = "time" ; time:units = "days since 2015-10-01 00:00:00" ; time:long_name = "time of measurement" ; float latitude(lat_dim, lon_dim) ; latitude:standard_name = "latitude" ; latitude:units = "degrees_north" ; latitude:long_name = "latitude of the observation" ; float longitude(lat_dim, lon_dim) ; longitude:standard_name = "longitude" ; longitude:units = "degrees_east" ; longitude:long_name = "longitude of the observation" ; double fcst_raw(time_dim, lat_dim, lon_dim) ; fcst_raw:units = "m" ; fcst_raw:standard_name = "fcst_raw" ; fcst_raw:coordinates = "time lat lon" ; // global attributes: :cdm_data_type = "Grid" ; :featureType = "Grid" ; } [CF documentation] http://cfconventions.org/cf-conventions/v1.6.0/cf-conventions.html [NODC template] https://www.nodc.noaa.gov/data/formats/netcdf/v2.0/grid.cdl 1.4 Plotting the Data with the matplotlib or the Unidata IDV ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ TODO > Hi again - > > Ok, I uploaded a (compressed) data file. It's called > mtd_snow_20160123_150000V_obj.nc.gz > > Randy > > address@hidden> wrote: > > > Randy, > > > > Please upload some sample data here: > > > > http://motherlode.ucar.edu/repository/alias/idvupload > > > > (I cannot tell if the IDV can handle the data just by looking at the nc > > header, > > unfortunately.) > > > > If you have control over how the netCDF file is generated, this is > > definitely > > possible. Ideally, the data should be CF compliant. Here is one example: > > > > https://www.nodc.noaa.gov/data/formats/netcdf/v2.0/grid.cdl > > > > though there is a lot of information here that is not required by the IDV, > > even > > the parts marked "REQUIRED". > > > > and more information here: > > > > http://cfconventions.org/cf-conventions/v1.6.0/cf-conventions.html > > > > But if you are lucky, the IDV may be able to read it in as is. > > > > Best, > > > > Unidata IDV Support > > > > > > > Hi - > > > > > > I'm one of the developers for the Model Evaluation Tools project, and we > > > have a tool that creates a 3D gridded dataset. We have an in-house tool > > to > > > view the data, but we'd like to know if other tools such as IDV can plot > > > this data, as well. > > > > > > The data is in a netcdf file ("ncdump -h" output is attached to this > > > email). Also attached is a plot using our in-house plotting software. > > > This should give you some idea what we're shooting for. > > > > > > Is there some way to plot this data using IDV? We're willing to consider > > > making changes to the netcdf file in order to facilitate this, if needed. > > > > > > Any thoughts? > > > > > > Randy Bullock > > > > > > > > > > > > Ticket Details > > =================== > > Ticket ID: BEV-647620 > > Department: Support IDV > > Priority: Normal > > Status: Closed > > > > > > Ticket Details =================== Ticket ID: BEV-647620 Department: Support IDV Priority: Normal Status: Closed
Attachment:
snow.png
Description: PNG image
