[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[IDV #RZP-160177]: unable to read netcdf file.



Mark,

On a Mac, I believe you can install ncdump via brew. It is really an essential
tool if you are dealing with netCDF so I would get your hands on it, one way or
another.

There are a few things I find suspicious about test3.nc

- I don't know if you can supply a comment inside the unit definition.

- Since you define the position of every data point, you probably don't need to
mention anything about the projection. At any rate, if you supply information on
the projection you should probably do it in the global attributes and in a CF
compliant manner, though again this should not be necessary.

- Most importantly, the ordering of the coordinate variables in the variable
definition looks suspicious and therefore the shape of your data is inconsistent
with what you describe in the variable definition. I have time, z, lat, lon NOT
lon, lat, pres, time. At any rate, think hard about the shape of you data.

- The probability units looks suspicious, and I'd leave out the probability 
variable
until you have figured out the rest.

Best,

Unidata IDV Support

> I donât have the ncdump routine on my mac. What do I need to do to install 
> it?
> In any event, I have uploaded the file called test3.nc
> Thanks again for all the help you have given me, I really appreciate it.
> 
> - Mark
> 
> 
> address@hidden
> Chief Scientist
> Science and Technology Corporation
> 10015 Old Columbia Road
> Suite E-250
> Columbia,  MD  21046
> phone: 410-309-0818 x 106
> cell: 240-481-7390
> http://markschoeberl.com
> 
> 
> 
> 
> > On Dec 14, 2015, at 12:25 PM, Unidata IDV Support <address@hidden> wrote:
> >
> > Mark,
> >
> > Yes you can upload the file, but another important hint is to look at the 
> > ncdump
> > -h output and compare it to what I have. My guess is you are probably pretty
> > close.
> >
> > Best,
> >
> > Unidata IDV Support
> >
> >
> >> Well, after working on this all morning, I get the same message from IDV 
> >> that it canât recognize the file.
> >> Of course, it recognizes the file you sent me (test2.nc) but not mine.  I 
> >> cannot figure out what is wrong.
> >> Shall I upload the new file?
> >>
> >> - Mark
> >>
> >> address@hidden
> >> Chief Scientist
> >> Science and Technology Corporation
> >> 10015 Old Columbia Road
> >> Suite E-250
> >> Columbia,  MD  21046
> >> phone: 410-309-0818 x 106
> >> cell: 240-481-7390
> >> http://markschoeberl.com
> >>
> >>
> >>
> >>
> >>> On Dec 11, 2015, at 11:53 AM, Unidata IDV Support <address@hidden> wrote:
> >>>
> >>> Hi Mark,
> >>>
> >>> Here, you will find a [test2.nc] file that can be read into the
> >>> IDV. (All reference links can be found at the bottom.) Also, here is the
> >>> [DAP] link that the IDV can read.
> >>>
> >>> When I need guidance from CF in the form of examples, I usually turn to
> >>> the [NODC] website, and specifically, in this case, the section on
> >>> [grids]. Using that resource as a guide, we can use a bit of [Python] to
> >>> create a data file according to these standards. I know you may be more
> >>> familiar with IDL, but Python is fairly straightforward to understand.
> >>>
> >>> First create a netCDF file to store your data.
> >>>
> >>> ,----
> >>> | import netCDF4
> >>> | import numpy as np
> >>> |
> >>> | try:
> >>> |     ncfile.close()
> >>> | except:
> >>> |     pass
> >>> | ncfile = netCDF4.Dataset('/home/python/work/test2.nc',
> >>> |                          mode='w',
> >>> |                          format='NETCDF4_CLASSIC')
> >>> | ncfile
> >>> `----
> >>>
> >>> Set some global attributes pertaining to standards and whatnot.
> >>>
> >>> ,----
> >>> | ncfile.Conventions = 'CF-1.6'
> >>> | ncfile.Metadata_Conventions = 'Unidata Dataset Discovery v1.0'
> >>> | ncfile.cdm_data_type = 'Grid'
> >>> | ncfile.featureType = "grid"
> >>> | ncfile.nodc_template_version = 'NODC_NetCDF_Grid_Template_v1.1'
> >>> `----
> >>>
> >>> Now create the `lat', `lon', `z', `time' dimensions.
> >>>
> >>> ,----
> >>> | lat_dim = ncfile.createDimension('lat', ds['latitude'].size)
> >>> | lon_dim = ncfile.createDimension('lon', ds['longitude'].size)
> >>> | z_dim = ncfile.createDimension('z', ds['pressure'].size)
> >>> | time_dim = ncfile.createDimension('time', None)
> >>> `----
> >>>
> >>> Create `lat' variable.
> >>>
> >>> ,----
> >>> | lat = ncfile.createVariable('lat', np.float32, ('lat',))
> >>> | lat.units = 'degrees_north'
> >>> | lat.long_name = 'latitude'
> >>> `----
> >>>
> >>> Create `lon' variable.
> >>>
> >>> ,----
> >>> | lon = ncfile.createVariable('lon', np.float32, ('lon',))
> >>> | lon.units = 'degrees_east'
> >>> | lon.long_name = 'longitude'
> >>> `----
> >>>
> >>> Create `z' variable
> >>>
> >>> ,----
> >>> | z = ncfile.createVariable('z', np.float32, ('z',))
> >>> | z.units = 'hPa'
> >>> | z.long_name = 'pressure'
> >>> `----
> >>>
> >>> Create `time' variable.
> >>>
> >>> ,----
> >>> | time = ncfile.createVariable('time', np.float64, ('time',))
> >>> | time.units = 'hours since 2000-01-01'
> >>> | time.long_name = 'time'
> >>> `----
> >>>
> >>> Create `temp' variable.
> >>>
> >>> ,----
> >>> | temp = ncfile.createVariable('temp',np.float64,('time','z', 
> >>> 'lat','lon'))
> >>> | temp.units = 'K' # degrees Kelvin
> >>> | temp.standard_name = 'air_temperature' # this is a CF standard name
> >>> `----
> >>>
> >>> Now set your data. The `ds' variable contains your original data. The
> >>> code for this is not shown.
> >>>
> >>> ,----
> >>> | lat[:] = ds['latitude'].data
> >>> | lon[:] = ds['longitude'].data
> >>> | z[:] = ds['pressure'].data
> >>> | time[:] = np.array([0])
> >>> | temp[0,:] = ds['temperature'].data
> >>> `----
> >>>
> >>> You are done. Now close the file.
> >>>
> >>> ,----
> >>> | ncfile.close();
> >>> `----
> >>>
> >>> Now let's look at the result of what we have done with `ncdump'.
> >>>
> >>> ,----
> >>> | ncdump -h ../../../test2.nc
> >>> `----
> >>>
> >>> ,----
> >>> | netcdf test2 {
> >>> | dimensions:
> >>> |   lat = 181 ;
> >>> |   lon = 361 ;
> >>> |   z = 100 ;
> >>> |   time = UNLIMITED ; // (1 currently)
> >>> | variables:
> >>> |   float lat(lat) ;
> >>> |     lat:units = "degrees_north" ;
> >>> |     lat:long_name = "latitude" ;
> >>> |   float lon(lon) ;
> >>> |     lon:units = "degrees_east" ;
> >>> |     lon:long_name = "longitude" ;
> >>> |   float z(z) ;
> >>> |     z:units = "hPa" ;
> >>> |     z:long_name = "pressure" ;
> >>> |   double time(time) ;
> >>> |     time:units = "hours since 2000-01-01" ;
> >>> |     time:long_name = "time" ;
> >>> |   double temp(time, z, lat, lon) ;
> >>> |     temp:units = "K" ;
> >>> |     temp:standard_name = "air_temperature" ;
> >>> |
> >>> | // global attributes:
> >>> |     :Conventions = "CF-1.6" ;
> >>> |     :Metadata_Conventions = "Unidata Dataset Discovery v1.0" ;
> >>> |     :cdm_data_type = "Grid" ;
> >>> |     :featureType = "grid" ;
> >>> |     :nodc_template_version = "NODC_NetCDF_Grid_Template_v1.1" ;
> >>> | }
> >>> `----
> >>>
> >>> Compare the results of this `ncdump' with the file you originally
> >>> uploaded to us.  Do you see the differences?
> >>>
> >>> ,----
> >>> | ncdump -h ../../../test.nc
> >>> `----
> >>>
> >>> ,----
> >>> | netcdf test {
> >>> | dimensions:
> >>> |   longitude = 361 ;
> >>> |   latitude = 181 ;
> >>> |   pressure = 100 ;
> >>> | variables:
> >>> |   float longitude(longitude) ;
> >>> |     longitude:longitude = "degrees" ;
> >>> |   float latitude(latitude) ;
> >>> |     latitude:latitude = "degrees" ;
> >>> |   float pressure(pressure) ;
> >>> |     pressure:pressure = "hPa" ;
> >>> |   float temperature(pressure, latitude, longitude) ;
> >>> |     temperature:temperature = "K" ;
> >>> |     temperature:Projection = "cylindrical" ;
> >>> |   float probability(pressure, latitude, longitude) ;
> >>> |     probability:probability = "1" ;
> >>> |
> >>> | // global attributes:
> >>> |     :Title = "test" ;
> >>> | }
> >>> `----
> >>>
> >>> At this point, you should have enough information to tidy up your netCDF
> >>> file with IDL. Let us know if you have any additional questions.
> >>>
> >>> Best,
> >>>
> >>> Unidata IDV Support
> >>>
> >>>
> >>> [test2.nc]
> >>> https://motherlode.ucar.edu/repository/entry/show/RAMADDA/Unidata/Staff/Julien+Chastang/mark/test2.nc?entryid=3cb9100e-6a62-4c5c-af37-8cd9966fcce6
> >>>
> >>> [DAP]
> >>> https://motherlode.ucar.edu/repository/opendap/3cb9100e-6a62-4c5c-af37-8cd9966fcce6/entry.das
> >>>
> >>> [NODC] https://www.nodc.noaa.gov/data/formats/netcdf/v1.1/
> >>>
> >>> [grids] https://www.nodc.noaa.gov/data/formats/netcdf/v1.1/grid.cdl
> >>>
> >>> [Python] https://github.com/Unidata/unidata-python-workshop
> >>>
> >>>
> >>>
> >>>> The data is from the AIRS instrument on AQUA. The data is basically 
> >>>> retrievals.
> >>>> There are 100 pressure levels and I am averaging 10 years of data for 
> >>>> each day of the
> >>>> year.  I sent you the data for one day.   There are two 3d fields: 
> >>>> probability and air temperature.
> >>>> Now the probability field is the probability
> >>>> that the air will decrease below some temperature.  This is a data 
> >>>> product that
> >>>> airlines would like to have if they want to avoid flying in those 
> >>>> regions. The
> >>>> goal is to visualize some of these regions as âbubblesâ of cold air 
> >>>> in the
> >>>> upper troposphere.  The other field is just the averaged temperature.
> >>>>
> >>>>
> >>>> I am creating the files using IDL netcdf commands
> >>>> [first var is probability]
> >>>> [second var is air temperature]
> >>>>
> >>>> s=size(first_var)
> >>>> dim1=ncdf_dimdef(id,'longitude',s(1))
> >>>> dim2=ncdf_dimdef(id,'latitude',s(2))
> >>>> dim3=ncdf_dimdef(id,'pressure',s(3))
> >>>>
> >>>> varlon=ncdf_vardef(id,'longitude',dim1,/float)
> >>>> varlat=ncdf_vardef(id,'latitude',dim2,/float)
> >>>> varpres=ncdf_vardef(id,'pressure',dim3,/float)
> >>>>
> >>>>
> >>>> ncdf_attput,id,varlat,'latitude','degrees',/char
> >>>> ncdf_attput,id,varlon,'longitude','degrees',/char
> >>>> ncdf_attput,id,varpres,'pressure','hPa',/char
> >>>>
> >>>> var1_data=ncdf_vardef(id,v1name,[dim1,dim2,dim3],/float)
> >>>> var2_data=ncdf_vardef(id,v2name,[dim1,dim2,dim3],/float)
> >>>>
> >>>> ; create the attributes
> >>>>
> >>>> ncdf_attput,id,var1_data,v1name,'K',/char
> >>>> ncdf_attput,id,var2_data,v2name,'1',/char
> >>>> ncdf_attput,id, /GLOBAL, 'Title', filename
> >>>> ncdf_attput,id,var1_data,'Projection','cylindrical'
> >>>>
> >>>> ncdf_control,id,/endef
> >>>>
> >>>> ; now write out data
> >>>>
> >>>> ncdf_varput,id,varlat,latgrid
> >>>> ncdf_varput,id,varlon,longrid
> >>>> ncdf_varput,id,varpres,pgrid
> >>>>
> >>>> ncdf_varput,id,var1_data,first_var
> >>>> ncdf_varput,id,var2_data,second_var
> >>>>
> >>>>
> >>>>
> >>>> address@hidden
> >>>> Chief Scientist
> >>>> Science and Technology Corporation
> >>>> 10015 Old Columbia Road
> >>>> Suite E-250
> >>>> Columbia,  MD  21046
> >>>> phone: 410-309-0818 x 106
> >>>> cell: 240-481-7390
> >>>> http://markschoeberl.com
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>> On Dec 10, 2015, at 3:44 PM, Unidata IDV Support <address@hidden> wrote:
> >>>>>
> >>>>> Hi Mark,
> >>>>>
> >>>>> Yes, it is not always as easy as we would like for data providers to 
> >>>>> adhere to
> >>>>> standards.
> >>>>>
> >>>>> Let's step back for a moment. What is the provenance of these data? 
> >>>>> Where are
> >>>>> they from?  How are you converting them into netCDF format?
> >>>>> These data are straightforward so we should hopefully be able to arrive 
> >>>>> at the
> >>>>> goal here quickly.
> >>>>>
> >>>>> Best,
> >>>>>
> >>>>> Unidata IDV Support
> >>>>>
> >>>>>> okay, actually the web site you pointed me at was helpful.  In your 
> >>>>>> documentation, I suggest that
> >>>>>> you point to this site.
> >>>>>>
> >>>>>> Just so I understand what you are saying.  My variable âlatitudeâ  
> >>>>>> should be named âgrid_latitudeâ
> >>>>>> and âtemperatureâ  should be âair_temperatureâ
> >>>>>>
> >>>>>> However, there is no variable named âprobabilityâ  â how do I 
> >>>>>> deal with that or other variables that
> >>>>>> I might invent that have no standard name.
> >>>>>>
> >>>>>> Thank you,
> >>>>>>
> >>>>>> Mark
> >>>>>>
> >>>>>>
> >>>>>> address@hidden
> >>>>>> Chief Scientist
> >>>>>> Science and Technology Corporation
> >>>>>> 10015 Old Columbia Road
> >>>>>> Suite E-250
> >>>>>> Columbia,  MD  21046
> >>>>>> phone: 410-309-0818 x 106
> >>>>>> cell: 240-481-7390
> >>>>>> http://markschoeberl.com
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>> On Dec 10, 2015, at 1:24 PM, Unidata IDV Support <address@hidden> 
> >>>>>>> wrote:
> >>>>>>>
> >>>>>>> Mark,
> >>>>>>>
> >>>>>>> The problem with these data is that they do not follow CF (or any) 
> >>>>>>> conventions
> >>>>>>> so the IDV cannot open them.
> >>>>>>>
> >>>>>>> http://www.unidata.ucar.edu/software/netcdf/conventions.html
> >>>>>>>
> >>>>>>> I would also maybe suggest specifying the central longitude for the 
> >>>>>>> cylindrical
> >>>>>>> projection. (Panoply must be assuming a central longitude of 0, I 
> >>>>>>> guess.)
> >>>>>>>
> >>>>>>>
> >>>>>>> Best,
> >>>>>>>
> >>>>>>> Unidata IDV Support
> >>>>>>>
> >>>>>>>
> >>>>>>>> File has been uploaded.
> >>>>>>>> Please let me know what I need to do to the nc file to get this IDV 
> >>>>>>>> to work.
> >>>>>>>> Thanks
> >>>>>>>>
> >>>>>>>> - mark
> >>>>>>>>
> >>>>>>>> On Dec 10, 2015, at 10:40 AM, Unidata IDV Support <address@hidden> 
> >>>>>>>> wrote:
> >>>>>>>>
> >>>>>>>> Mark,
> >>>>>>>>
> >>>>>>>> Please upload your data to
> >>>>>>>>
> >>>>>>>> http://motherlode.ucar.edu/repository/alias/idvupload
> >>>>>>>>
> >>>>>>>> and let us know when it has been uploaded.
> >>>>>>>>
> >>>>>>>> It is also possible that you simply need to select a "Data Source 
> >>>>>>>> Type" of "Grid
> >>>>>>>> files (netCDF/GRIB/OPeNDap/GEMPAK)" in the IDV Dashboard, "Data 
> >>>>>>>> Choosers" tab.
> >>>>>>>>
> >>>>>>>> Best,
> >>>>>>>>
> >>>>>>>> Unidata IDV Support
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> I want to use the idv package to visualize my data.
> >>>>>>>>
> >>>>>>>> I created a netcdf file. It can be read and and visualized by 
> >>>>>>>> PANOPLY.
> >>>>>>>> But Idv does not recognize it as a netcdf file - obviously it is 
> >>>>>>>> looking
> >>>>>>>> for something I don't have in my file.
> >>>>>>>>
> >>>>>>>> I studied your documentation and even looked at an example file. But 
> >>>>>>>> it
> >>>>>>>> beats me what I am doing wrong.
> >>>>>>>>
> >>>>>>>> Can you help?
> >>>>>>>>
> >>>>>>>> The file is too big to attach.
> >>>>>>>>
> >>>>>>>> Here is the PANOPLY screenshot. Can you help. Thank you. - Mark
> >>>>>>>>
> >>>>>>>> address@hidden
> >>>>>>>> Chief Scientist
> >>>>>>>> Science and Technology Corporation
> >>>>>>>> 10015 Old Columbia Road
> >>>>>>>> Suite E-250
> >>>>>>>> Columbia, MD 21046
> >>>>>>>> phone: 410-309-0818 x 106
> >>>>>>>> cell: 240-481-7390
> >>>>>>>> http://markschoeberl.com
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> I want to use the idv package to visualize my data.
> >>>>>>>>
> >>>>>>>> I created a netcdf file. It can be read and and visualized by 
> >>>>>>>> PANOPLY.
> >>>>>>>> But Idv does not recognize it as a netcdf file - obviously it is 
> >>>>>>>> looking
> >>>>>>>> for something I don't have in my file.
> >>>>>>>>
> >>>>>>>> I studied your documentation and even looked at an example file. But 
> >>>>>>>> it
> >>>>>>>> beats me what I am doing wrong.
> >>>>>>>>
> >>>>>>>> Can you help?
> >>>>>>>>
> >>>>>>>> The file is too big to attach.
> >>>>>>>>
> >>>>>>>> Here is the PANOPLY screenshot. Can you help. Thank you. - Mark
> >>>>>>>>
> >>>>>>>> address@hidden
> >>>>>>>> Chief Scientist
> >>>>>>>> Science and Technology Corporation
> >>>>>>>> 10015 Old Columbia Road
> >>>>>>>> Suite E-250
> >>>>>>>> Columbia, MD 21046
> >>>>>>>> phone: 410-309-0818 x 106
> >>>>>>>> cell: 240-481-7390
> >>>>>>>> http://markschoeberl.com
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> Ticket Details
> >>>>>>>> ===================
> >>>>>>>> Ticket ID: RZP-160177
> >>>>>>>> Department: Support IDV
> >>>>>>>> Priority: Normal
> >>>>>>>> Status: Closed
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> Ticket Details
> >>>>>>> ===================
> >>>>>>> Ticket ID: RZP-160177
> >>>>>>> Department: Support IDV
> >>>>>>> Priority: Normal
> >>>>>>> Status: Closed
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>
> >>>>>
> >>>>> Ticket Details
> >>>>> ===================
> >>>>> Ticket ID: RZP-160177
> >>>>> Department: Support IDV
> >>>>> Priority: Normal
> >>>>> Status: Closed
> >>>>>
> >>>>
> >>>>
> >>>>
> >>>
> >>>
> >>> Ticket Details
> >>> ===================
> >>> Ticket ID: RZP-160177
> >>> Department: Support IDV
> >>> Priority: Normal
> >>> Status: Closed
> >>>
> >>
> >>
> >>
> >
> >
> > Ticket Details
> > ===================
> > Ticket ID: RZP-160177
> > Department: Support IDV
> > Priority: Normal
> > Status: Closed
> >
> 
> 
> 


Ticket Details
===================
Ticket ID: RZP-160177
Department: Support IDV
Priority: Normal
Status: Closed


NOTE: All email exchanges with Unidata User Support are recorded in the Unidata inquiry tracking system and then made publicly available through the web. If you do not want to have your interactions made available in this way, you must let us know in each email you send to us.