[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[IDV #RZP-160177]: unable to read netcdf file.

Mark,

Yes you can upload the file, but another important hint is to look at the ncdump
-h output and compare it to what I have. My guess is you are probably pretty
close.

Best,

Unidata IDV Support


> Well, after working on this all morning, I get the same message from IDV that 
> it can’t recognize the file.
> Of course, it recognizes the file you sent me (test2.nc) but not mine.  I 
> cannot figure out what is wrong.
> Shall I upload the new file?
> 
> - Mark
> 
> address@hidden
> Chief Scientist
> Science and Technology Corporation
> 10015 Old Columbia Road
> Suite E-250
> Columbia,  MD  21046
> phone: 410-309-0818 x 106
> cell: 240-481-7390
> http://markschoeberl.com
> 
> 
> 
> 
> > On Dec 11, 2015, at 11:53 AM, Unidata IDV Support <address@hidden> wrote:
> >
> > Hi Mark,
> >
> > Here, you will find a [test2.nc] file that can be read into the
> > IDV. (All reference links can be found at the bottom.) Also, here is the
> > [DAP] link that the IDV can read.
> >
> > When I need guidance from CF in the form of examples, I usually turn to
> > the [NODC] website, and specifically, in this case, the section on
> > [grids]. Using that resource as a guide, we can use a bit of [Python] to
> > create a data file according to these standards. I know you may be more
> > familiar with IDL, but Python is fairly straightforward to understand.
> >
> > First create a netCDF file to store your data.
> >
> > ,----
> > | import netCDF4
> > | import numpy as np
> > |
> > | try:
> > |     ncfile.close()
> > | except:
> > |     pass
> > | ncfile = netCDF4.Dataset('/home/python/work/test2.nc',
> > |                          mode='w',
> > |                          format='NETCDF4_CLASSIC')
> > | ncfile
> > `----
> >
> > Set some global attributes pertaining to standards and whatnot.
> >
> > ,----
> > | ncfile.Conventions = 'CF-1.6'
> > | ncfile.Metadata_Conventions = 'Unidata Dataset Discovery v1.0'
> > | ncfile.cdm_data_type = 'Grid'
> > | ncfile.featureType = "grid"
> > | ncfile.nodc_template_version = 'NODC_NetCDF_Grid_Template_v1.1'
> > `----
> >
> > Now create the `lat', `lon', `z', `time' dimensions.
> >
> > ,----
> > | lat_dim = ncfile.createDimension('lat', ds['latitude'].size)
> > | lon_dim = ncfile.createDimension('lon', ds['longitude'].size)
> > | z_dim = ncfile.createDimension('z', ds['pressure'].size)
> > | time_dim = ncfile.createDimension('time', None)
> > `----
> >
> > Create `lat' variable.
> >
> > ,----
> > | lat = ncfile.createVariable('lat', np.float32, ('lat',))
> > | lat.units = 'degrees_north'
> > | lat.long_name = 'latitude'
> > `----
> >
> > Create `lon' variable.
> >
> > ,----
> > | lon = ncfile.createVariable('lon', np.float32, ('lon',))
> > | lon.units = 'degrees_east'
> > | lon.long_name = 'longitude'
> > `----
> >
> > Create `z' variable
> >
> > ,----
> > | z = ncfile.createVariable('z', np.float32, ('z',))
> > | z.units = 'hPa'
> > | z.long_name = 'pressure'
> > `----
> >
> > Create `time' variable.
> >
> > ,----
> > | time = ncfile.createVariable('time', np.float64, ('time',))
> > | time.units = 'hours since 2000-01-01'
> > | time.long_name = 'time'
> > `----
> >
> > Create `temp' variable.
> >
> > ,----
> > | temp = ncfile.createVariable('temp',np.float64,('time','z', 'lat','lon'))
> > | temp.units = 'K' # degrees Kelvin
> > | temp.standard_name = 'air_temperature' # this is a CF standard name
> > `----
> >
> > Now set your data. The `ds' variable contains your original data. The
> > code for this is not shown.
> >
> > ,----
> > | lat[:] = ds['latitude'].data
> > | lon[:] = ds['longitude'].data
> > | z[:] = ds['pressure'].data
> > | time[:] = np.array([0])
> > | temp[0,:] = ds['temperature'].data
> > `----
> >
> > You are done. Now close the file.
> >
> > ,----
> > | ncfile.close();
> > `----
> >
> > Now let's look at the result of what we have done with `ncdump'.
> >
> > ,----
> > | ncdump -h ../../../test2.nc
> > `----
> >
> > ,----
> > | netcdf test2 {
> > | dimensions:
> > |   lat = 181 ;
> > |   lon = 361 ;
> > |   z = 100 ;
> > |   time = UNLIMITED ; // (1 currently)
> > | variables:
> > |   float lat(lat) ;
> > |     lat:units = "degrees_north" ;
> > |     lat:long_name = "latitude" ;
> > |   float lon(lon) ;
> > |     lon:units = "degrees_east" ;
> > |     lon:long_name = "longitude" ;
> > |   float z(z) ;
> > |     z:units = "hPa" ;
> > |     z:long_name = "pressure" ;
> > |   double time(time) ;
> > |     time:units = "hours since 2000-01-01" ;
> > |     time:long_name = "time" ;
> > |   double temp(time, z, lat, lon) ;
> > |     temp:units = "K" ;
> > |     temp:standard_name = "air_temperature" ;
> > |
> > | // global attributes:
> > |     :Conventions = "CF-1.6" ;
> > |     :Metadata_Conventions = "Unidata Dataset Discovery v1.0" ;
> > |     :cdm_data_type = "Grid" ;
> > |     :featureType = "grid" ;
> > |     :nodc_template_version = "NODC_NetCDF_Grid_Template_v1.1" ;
> > | }
> > `----
> >
> > Compare the results of this `ncdump' with the file you originally
> > uploaded to us.  Do you see the differences?
> >
> > ,----
> > | ncdump -h ../../../test.nc
> > `----
> >
> > ,----
> > | netcdf test {
> > | dimensions:
> > |   longitude = 361 ;
> > |   latitude = 181 ;
> > |   pressure = 100 ;
> > | variables:
> > |   float longitude(longitude) ;
> > |     longitude:longitude = "degrees" ;
> > |   float latitude(latitude) ;
> > |     latitude:latitude = "degrees" ;
> > |   float pressure(pressure) ;
> > |     pressure:pressure = "hPa" ;
> > |   float temperature(pressure, latitude, longitude) ;
> > |     temperature:temperature = "K" ;
> > |     temperature:Projection = "cylindrical" ;
> > |   float probability(pressure, latitude, longitude) ;
> > |     probability:probability = "1" ;
> > |
> > | // global attributes:
> > |     :Title = "test" ;
> > | }
> > `----
> >
> > At this point, you should have enough information to tidy up your netCDF
> > file with IDL. Let us know if you have any additional questions.
> >
> > Best,
> >
> > Unidata IDV Support
> >
> >
> > [test2.nc]
> > https://motherlode.ucar.edu/repository/entry/show/RAMADDA/Unidata/Staff/Julien+Chastang/mark/test2.nc?entryid=3cb9100e-6a62-4c5c-af37-8cd9966fcce6
> >
> > [DAP]
> > https://motherlode.ucar.edu/repository/opendap/3cb9100e-6a62-4c5c-af37-8cd9966fcce6/entry.das
> >
> > [NODC] https://www.nodc.noaa.gov/data/formats/netcdf/v1.1/
> >
> > [grids] https://www.nodc.noaa.gov/data/formats/netcdf/v1.1/grid.cdl
> >
> > [Python] https://github.com/Unidata/unidata-python-workshop
> >
> >
> >
> >> The data is from the AIRS instrument on AQUA. The data is basically 
> >> retrievals.
> >> There are 100 pressure levels and I am averaging 10 years of data for each 
> >> day of the
> >> year.  I sent you the data for one day.   There are two 3d fields: 
> >> probability and air temperature.
> >> Now the probability field is the probability
> >> that the air will decrease below some temperature.  This is a data product 
> >> that
> >> airlines would like to have if they want to avoid flying in those regions. 
> >> The
> >> goal is to visualize some of these regions as “bubbles” of cold air in the
> >> upper troposphere.  The other field is just the averaged temperature.
> >>
> >>
> >> I am creating the files using IDL netcdf commands
> >> [first var is probability]
> >> [second var is air temperature]
> >>
> >> s=size(first_var)
> >> dim1=ncdf_dimdef(id,'longitude',s(1))
> >> dim2=ncdf_dimdef(id,'latitude',s(2))
> >> dim3=ncdf_dimdef(id,'pressure',s(3))
> >>
> >> varlon=ncdf_vardef(id,'longitude',dim1,/float)
> >> varlat=ncdf_vardef(id,'latitude',dim2,/float)
> >> varpres=ncdf_vardef(id,'pressure',dim3,/float)
> >>
> >>
> >> ncdf_attput,id,varlat,'latitude','degrees',/char
> >> ncdf_attput,id,varlon,'longitude','degrees',/char
> >> ncdf_attput,id,varpres,'pressure','hPa',/char
> >>
> >> var1_data=ncdf_vardef(id,v1name,[dim1,dim2,dim3],/float)
> >> var2_data=ncdf_vardef(id,v2name,[dim1,dim2,dim3],/float)
> >>
> >> ; create the attributes
> >>
> >> ncdf_attput,id,var1_data,v1name,'K',/char
> >> ncdf_attput,id,var2_data,v2name,'1',/char
> >> ncdf_attput,id, /GLOBAL, 'Title', filename
> >> ncdf_attput,id,var1_data,'Projection','cylindrical'
> >>
> >> ncdf_control,id,/endef
> >>
> >> ; now write out data
> >>
> >> ncdf_varput,id,varlat,latgrid
> >> ncdf_varput,id,varlon,longrid
> >> ncdf_varput,id,varpres,pgrid
> >>
> >> ncdf_varput,id,var1_data,first_var
> >> ncdf_varput,id,var2_data,second_var
> >>
> >>
> >>
> >> address@hidden
> >> Chief Scientist
> >> Science and Technology Corporation
> >> 10015 Old Columbia Road
> >> Suite E-250
> >> Columbia,  MD  21046
> >> phone: 410-309-0818 x 106
> >> cell: 240-481-7390
> >> http://markschoeberl.com
> >>
> >>
> >>
> >>
> >>> On Dec 10, 2015, at 3:44 PM, Unidata IDV Support <address@hidden> wrote:
> >>>
> >>> Hi Mark,
> >>>
> >>> Yes, it is not always as easy as we would like for data providers to 
> >>> adhere to
> >>> standards.
> >>>
> >>> Let's step back for a moment. What is the provenance of these data? Where 
> >>> are
> >>> they from?  How are you converting them into netCDF format?
> >>> These data are straightforward so we should hopefully be able to arrive 
> >>> at the
> >>> goal here quickly.
> >>>
> >>> Best,
> >>>
> >>> Unidata IDV Support
> >>>
> >>>> okay, actually the web site you pointed me at was helpful.  In your 
> >>>> documentation, I suggest that
> >>>> you point to this site.
> >>>>
> >>>> Just so I understand what you are saying.  My variable ‘latitude’  
> >>>> should be named ‘grid_latitude’
> >>>> and ‘temperature’  should be ‘air_temperature’
> >>>>
> >>>> However, there is no variable named ‘probability’  … how do I deal with 
> >>>> that or other variables that
> >>>> I might invent that have no standard name.
> >>>>
> >>>> Thank you,
> >>>>
> >>>> Mark
> >>>>
> >>>>
> >>>> address@hidden
> >>>> Chief Scientist
> >>>> Science and Technology Corporation
> >>>> 10015 Old Columbia Road
> >>>> Suite E-250
> >>>> Columbia,  MD  21046
> >>>> phone: 410-309-0818 x 106
> >>>> cell: 240-481-7390
> >>>> http://markschoeberl.com
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>> On Dec 10, 2015, at 1:24 PM, Unidata IDV Support <address@hidden> wrote:
> >>>>>
> >>>>> Mark,
> >>>>>
> >>>>> The problem with these data is that they do not follow CF (or any) 
> >>>>> conventions
> >>>>> so the IDV cannot open them.
> >>>>>
> >>>>> http://www.unidata.ucar.edu/software/netcdf/conventions.html
> >>>>>
> >>>>> I would also maybe suggest specifying the central longitude for the 
> >>>>> cylindrical
> >>>>> projection. (Panoply must be assuming a central longitude of 0, I 
> >>>>> guess.)
> >>>>>
> >>>>>
> >>>>> Best,
> >>>>>
> >>>>> Unidata IDV Support
> >>>>>
> >>>>>
> >>>>>> File has been uploaded.
> >>>>>> Please let me know what I need to do to the nc file to get this IDV to 
> >>>>>> work.
> >>>>>> Thanks
> >>>>>>
> >>>>>> - mark
> >>>>>>
> >>>>>> On Dec 10, 2015, at 10:40 AM, Unidata IDV Support <address@hidden> 
> >>>>>> wrote:
> >>>>>>
> >>>>>> Mark,
> >>>>>>
> >>>>>> Please upload your data to
> >>>>>>
> >>>>>> http://motherlode.ucar.edu/repository/alias/idvupload
> >>>>>>
> >>>>>> and let us know when it has been uploaded.
> >>>>>>
> >>>>>> It is also possible that you simply need to select a "Data Source 
> >>>>>> Type" of "Grid
> >>>>>> files (netCDF/GRIB/OPeNDap/GEMPAK)" in the IDV Dashboard, "Data 
> >>>>>> Choosers" tab.
> >>>>>>
> >>>>>> Best,
> >>>>>>
> >>>>>> Unidata IDV Support
> >>>>>>
> >>>>>>
> >>>>>> I want to use the idv package to visualize my data.
> >>>>>>
> >>>>>> I created a netcdf file. It can be read and and visualized by PANOPLY.
> >>>>>> But Idv does not recognize it as a netcdf file - obviously it is 
> >>>>>> looking
> >>>>>> for something I don't have in my file.
> >>>>>>
> >>>>>> I studied your documentation and even looked at an example file. But it
> >>>>>> beats me what I am doing wrong.
> >>>>>>
> >>>>>> Can you help?
> >>>>>>
> >>>>>> The file is too big to attach.
> >>>>>>
> >>>>>> Here is the PANOPLY screenshot. Can you help. Thank you. - Mark
> >>>>>>
> >>>>>> address@hidden
> >>>>>> Chief Scientist
> >>>>>> Science and Technology Corporation
> >>>>>> 10015 Old Columbia Road
> >>>>>> Suite E-250
> >>>>>> Columbia, MD 21046
> >>>>>> phone: 410-309-0818 x 106
> >>>>>> cell: 240-481-7390
> >>>>>> http://markschoeberl.com
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> I want to use the idv package to visualize my data.
> >>>>>>
> >>>>>> I created a netcdf file. It can be read and and visualized by PANOPLY.
> >>>>>> But Idv does not recognize it as a netcdf file - obviously it is 
> >>>>>> looking
> >>>>>> for something I don't have in my file.
> >>>>>>
> >>>>>> I studied your documentation and even looked at an example file. But it
> >>>>>> beats me what I am doing wrong.
> >>>>>>
> >>>>>> Can you help?
> >>>>>>
> >>>>>> The file is too big to attach.
> >>>>>>
> >>>>>> Here is the PANOPLY screenshot. Can you help. Thank you. - Mark
> >>>>>>
> >>>>>> address@hidden
> >>>>>> Chief Scientist
> >>>>>> Science and Technology Corporation
> >>>>>> 10015 Old Columbia Road
> >>>>>> Suite E-250
> >>>>>> Columbia, MD 21046
> >>>>>> phone: 410-309-0818 x 106
> >>>>>> cell: 240-481-7390
> >>>>>> http://markschoeberl.com
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> Ticket Details
> >>>>>> ===================
> >>>>>> Ticket ID: RZP-160177
> >>>>>> Department: Support IDV
> >>>>>> Priority: Normal
> >>>>>> Status: Closed
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>
> >>>>>
> >>>>> Ticket Details
> >>>>> ===================
> >>>>> Ticket ID: RZP-160177
> >>>>> Department: Support IDV
> >>>>> Priority: Normal
> >>>>> Status: Closed
> >>>>>
> >>>>
> >>>>
> >>>>
> >>>
> >>>
> >>> Ticket Details
> >>> ===================
> >>> Ticket ID: RZP-160177
> >>> Department: Support IDV
> >>> Priority: Normal
> >>> Status: Closed
> >>>
> >>
> >>
> >>
> >
> >
> > Ticket Details
> > ===================
> > Ticket ID: RZP-160177
> > Department: Support IDV
> > Priority: Normal
> > Status: Closed
> >
> 
> 
> 


Ticket Details
===================
Ticket ID: RZP-160177
Department: Support IDV
Priority: Normal
Status: Closed