[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[IDV #RZP-160177]: unable to read netcdf file.



Mark,

Yes you can upload the file, but another important hint is to look at the ncdump
-h output and compare it to what I have. My guess is you are probably pretty
close.

Best,

Unidata IDV Support


> Well, after working on this all morning, I get the same message from IDV that 
> it canât recognize the file.
> Of course, it recognizes the file you sent me (test2.nc) but not mine.  I 
> cannot figure out what is wrong.
> Shall I upload the new file?
> 
> - Mark
> 
> address@hidden
> Chief Scientist
> Science and Technology Corporation
> 10015 Old Columbia Road
> Suite E-250
> Columbia,  MD  21046
> phone: 410-309-0818 x 106
> cell: 240-481-7390
> http://markschoeberl.com
> 
> 
> 
> 
> > On Dec 11, 2015, at 11:53 AM, Unidata IDV Support <address@hidden> wrote:
> >
> > Hi Mark,
> >
> > Here, you will find a [test2.nc] file that can be read into the
> > IDV. (All reference links can be found at the bottom.) Also, here is the
> > [DAP] link that the IDV can read.
> >
> > When I need guidance from CF in the form of examples, I usually turn to
> > the [NODC] website, and specifically, in this case, the section on
> > [grids]. Using that resource as a guide, we can use a bit of [Python] to
> > create a data file according to these standards. I know you may be more
> > familiar with IDL, but Python is fairly straightforward to understand.
> >
> > First create a netCDF file to store your data.
> >
> > ,----
> > | import netCDF4
> > | import numpy as np
> > |
> > | try:
> > |     ncfile.close()
> > | except:
> > |     pass
> > | ncfile = netCDF4.Dataset('/home/python/work/test2.nc',
> > |                          mode='w',
> > |                          format='NETCDF4_CLASSIC')
> > | ncfile
> > `----
> >
> > Set some global attributes pertaining to standards and whatnot.
> >
> > ,----
> > | ncfile.Conventions = 'CF-1.6'
> > | ncfile.Metadata_Conventions = 'Unidata Dataset Discovery v1.0'
> > | ncfile.cdm_data_type = 'Grid'
> > | ncfile.featureType = "grid"
> > | ncfile.nodc_template_version = 'NODC_NetCDF_Grid_Template_v1.1'
> > `----
> >
> > Now create the `lat', `lon', `z', `time' dimensions.
> >
> > ,----
> > | lat_dim = ncfile.createDimension('lat', ds['latitude'].size)
> > | lon_dim = ncfile.createDimension('lon', ds['longitude'].size)
> > | z_dim = ncfile.createDimension('z', ds['pressure'].size)
> > | time_dim = ncfile.createDimension('time', None)
> > `----
> >
> > Create `lat' variable.
> >
> > ,----
> > | lat = ncfile.createVariable('lat', np.float32, ('lat',))
> > | lat.units = 'degrees_north'
> > | lat.long_name = 'latitude'
> > `----
> >
> > Create `lon' variable.
> >
> > ,----
> > | lon = ncfile.createVariable('lon', np.float32, ('lon',))
> > | lon.units = 'degrees_east'
> > | lon.long_name = 'longitude'
> > `----
> >
> > Create `z' variable
> >
> > ,----
> > | z = ncfile.createVariable('z', np.float32, ('z',))
> > | z.units = 'hPa'
> > | z.long_name = 'pressure'
> > `----
> >
> > Create `time' variable.
> >
> > ,----
> > | time = ncfile.createVariable('time', np.float64, ('time',))
> > | time.units = 'hours since 2000-01-01'
> > | time.long_name = 'time'
> > `----
> >
> > Create `temp' variable.
> >
> > ,----
> > | temp = ncfile.createVariable('temp',np.float64,('time','z', 'lat','lon'))
> > | temp.units = 'K' # degrees Kelvin
> > | temp.standard_name = 'air_temperature' # this is a CF standard name
> > `----
> >
> > Now set your data. The `ds' variable contains your original data. The
> > code for this is not shown.
> >
> > ,----
> > | lat[:] = ds['latitude'].data
> > | lon[:] = ds['longitude'].data
> > | z[:] = ds['pressure'].data
> > | time[:] = np.array([0])
> > | temp[0,:] = ds['temperature'].data
> > `----
> >
> > You are done. Now close the file.
> >
> > ,----
> > | ncfile.close();
> > `----
> >
> > Now let's look at the result of what we have done with `ncdump'.
> >
> > ,----
> > | ncdump -h ../../../test2.nc
> > `----
> >
> > ,----
> > | netcdf test2 {
> > | dimensions:
> > |   lat = 181 ;
> > |   lon = 361 ;
> > |   z = 100 ;
> > |   time = UNLIMITED ; // (1 currently)
> > | variables:
> > |   float lat(lat) ;
> > |     lat:units = "degrees_north" ;
> > |     lat:long_name = "latitude" ;
> > |   float lon(lon) ;
> > |     lon:units = "degrees_east" ;
> > |     lon:long_name = "longitude" ;
> > |   float z(z) ;
> > |     z:units = "hPa" ;
> > |     z:long_name = "pressure" ;
> > |   double time(time) ;
> > |     time:units = "hours since 2000-01-01" ;
> > |     time:long_name = "time" ;
> > |   double temp(time, z, lat, lon) ;
> > |     temp:units = "K" ;
> > |     temp:standard_name = "air_temperature" ;
> > |
> > | // global attributes:
> > |     :Conventions = "CF-1.6" ;
> > |     :Metadata_Conventions = "Unidata Dataset Discovery v1.0" ;
> > |     :cdm_data_type = "Grid" ;
> > |     :featureType = "grid" ;
> > |     :nodc_template_version = "NODC_NetCDF_Grid_Template_v1.1" ;
> > | }
> > `----
> >
> > Compare the results of this `ncdump' with the file you originally
> > uploaded to us.  Do you see the differences?
> >
> > ,----
> > | ncdump -h ../../../test.nc
> > `----
> >
> > ,----
> > | netcdf test {
> > | dimensions:
> > |   longitude = 361 ;
> > |   latitude = 181 ;
> > |   pressure = 100 ;
> > | variables:
> > |   float longitude(longitude) ;
> > |     longitude:longitude = "degrees" ;
> > |   float latitude(latitude) ;
> > |     latitude:latitude = "degrees" ;
> > |   float pressure(pressure) ;
> > |     pressure:pressure = "hPa" ;
> > |   float temperature(pressure, latitude, longitude) ;
> > |     temperature:temperature = "K" ;
> > |     temperature:Projection = "cylindrical" ;
> > |   float probability(pressure, latitude, longitude) ;
> > |     probability:probability = "1" ;
> > |
> > | // global attributes:
> > |     :Title = "test" ;
> > | }
> > `----
> >
> > At this point, you should have enough information to tidy up your netCDF
> > file with IDL. Let us know if you have any additional questions.
> >
> > Best,
> >
> > Unidata IDV Support
> >
> >
> > [test2.nc]
> > https://motherlode.ucar.edu/repository/entry/show/RAMADDA/Unidata/Staff/Julien+Chastang/mark/test2.nc?entryid=3cb9100e-6a62-4c5c-af37-8cd9966fcce6
> >
> > [DAP]
> > https://motherlode.ucar.edu/repository/opendap/3cb9100e-6a62-4c5c-af37-8cd9966fcce6/entry.das
> >
> > [NODC] https://www.nodc.noaa.gov/data/formats/netcdf/v1.1/
> >
> > [grids] https://www.nodc.noaa.gov/data/formats/netcdf/v1.1/grid.cdl
> >
> > [Python] https://github.com/Unidata/unidata-python-workshop
> >
> >
> >
> >> The data is from the AIRS instrument on AQUA. The data is basically 
> >> retrievals.
> >> There are 100 pressure levels and I am averaging 10 years of data for each 
> >> day of the
> >> year.  I sent you the data for one day.   There are two 3d fields: 
> >> probability and air temperature.
> >> Now the probability field is the probability
> >> that the air will decrease below some temperature.  This is a data product 
> >> that
> >> airlines would like to have if they want to avoid flying in those regions. 
> >> The
> >> goal is to visualize some of these regions as âbubblesâ of cold air in 
> >> the
> >> upper troposphere.  The other field is just the averaged temperature.
> >>
> >>
> >> I am creating the files using IDL netcdf commands
> >> [first var is probability]
> >> [second var is air temperature]
> >>
> >> s=size(first_var)
> >> dim1=ncdf_dimdef(id,'longitude',s(1))
> >> dim2=ncdf_dimdef(id,'latitude',s(2))
> >> dim3=ncdf_dimdef(id,'pressure',s(3))
> >>
> >> varlon=ncdf_vardef(id,'longitude',dim1,/float)
> >> varlat=ncdf_vardef(id,'latitude',dim2,/float)
> >> varpres=ncdf_vardef(id,'pressure',dim3,/float)
> >>
> >>
> >> ncdf_attput,id,varlat,'latitude','degrees',/char
> >> ncdf_attput,id,varlon,'longitude','degrees',/char
> >> ncdf_attput,id,varpres,'pressure','hPa',/char
> >>
> >> var1_data=ncdf_vardef(id,v1name,[dim1,dim2,dim3],/float)
> >> var2_data=ncdf_vardef(id,v2name,[dim1,dim2,dim3],/float)
> >>
> >> ; create the attributes
> >>
> >> ncdf_attput,id,var1_data,v1name,'K',/char
> >> ncdf_attput,id,var2_data,v2name,'1',/char
> >> ncdf_attput,id, /GLOBAL, 'Title', filename
> >> ncdf_attput,id,var1_data,'Projection','cylindrical'
> >>
> >> ncdf_control,id,/endef
> >>
> >> ; now write out data
> >>
> >> ncdf_varput,id,varlat,latgrid
> >> ncdf_varput,id,varlon,longrid
> >> ncdf_varput,id,varpres,pgrid
> >>
> >> ncdf_varput,id,var1_data,first_var
> >> ncdf_varput,id,var2_data,second_var
> >>
> >>
> >>
> >> address@hidden
> >> Chief Scientist
> >> Science and Technology Corporation
> >> 10015 Old Columbia Road
> >> Suite E-250
> >> Columbia,  MD  21046
> >> phone: 410-309-0818 x 106
> >> cell: 240-481-7390
> >> http://markschoeberl.com
> >>
> >>
> >>
> >>
> >>> On Dec 10, 2015, at 3:44 PM, Unidata IDV Support <address@hidden> wrote:
> >>>
> >>> Hi Mark,
> >>>
> >>> Yes, it is not always as easy as we would like for data providers to 
> >>> adhere to
> >>> standards.
> >>>
> >>> Let's step back for a moment. What is the provenance of these data? Where 
> >>> are
> >>> they from?  How are you converting them into netCDF format?
> >>> These data are straightforward so we should hopefully be able to arrive 
> >>> at the
> >>> goal here quickly.
> >>>
> >>> Best,
> >>>
> >>> Unidata IDV Support
> >>>
> >>>> okay, actually the web site you pointed me at was helpful.  In your 
> >>>> documentation, I suggest that
> >>>> you point to this site.
> >>>>
> >>>> Just so I understand what you are saying.  My variable âlatitudeâ  
> >>>> should be named âgrid_latitudeâ
> >>>> and âtemperatureâ  should be âair_temperatureâ
> >>>>
> >>>> However, there is no variable named âprobabilityâ  â how do I deal 
> >>>> with that or other variables that
> >>>> I might invent that have no standard name.
> >>>>
> >>>> Thank you,
> >>>>
> >>>> Mark
> >>>>
> >>>>
> >>>> address@hidden
> >>>> Chief Scientist
> >>>> Science and Technology Corporation
> >>>> 10015 Old Columbia Road
> >>>> Suite E-250
> >>>> Columbia,  MD  21046
> >>>> phone: 410-309-0818 x 106
> >>>> cell: 240-481-7390
> >>>> http://markschoeberl.com
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>> On Dec 10, 2015, at 1:24 PM, Unidata IDV Support <address@hidden> wrote:
> >>>>>
> >>>>> Mark,
> >>>>>
> >>>>> The problem with these data is that they do not follow CF (or any) 
> >>>>> conventions
> >>>>> so the IDV cannot open them.
> >>>>>
> >>>>> http://www.unidata.ucar.edu/software/netcdf/conventions.html
> >>>>>
> >>>>> I would also maybe suggest specifying the central longitude for the 
> >>>>> cylindrical
> >>>>> projection. (Panoply must be assuming a central longitude of 0, I 
> >>>>> guess.)
> >>>>>
> >>>>>
> >>>>> Best,
> >>>>>
> >>>>> Unidata IDV Support
> >>>>>
> >>>>>
> >>>>>> File has been uploaded.
> >>>>>> Please let me know what I need to do to the nc file to get this IDV to 
> >>>>>> work.
> >>>>>> Thanks
> >>>>>>
> >>>>>> - mark
> >>>>>>
> >>>>>> On Dec 10, 2015, at 10:40 AM, Unidata IDV Support <address@hidden> 
> >>>>>> wrote:
> >>>>>>
> >>>>>> Mark,
> >>>>>>
> >>>>>> Please upload your data to
> >>>>>>
> >>>>>> http://motherlode.ucar.edu/repository/alias/idvupload
> >>>>>>
> >>>>>> and let us know when it has been uploaded.
> >>>>>>
> >>>>>> It is also possible that you simply need to select a "Data Source 
> >>>>>> Type" of "Grid
> >>>>>> files (netCDF/GRIB/OPeNDap/GEMPAK)" in the IDV Dashboard, "Data 
> >>>>>> Choosers" tab.
> >>>>>>
> >>>>>> Best,
> >>>>>>
> >>>>>> Unidata IDV Support
> >>>>>>
> >>>>>>
> >>>>>> I want to use the idv package to visualize my data.
> >>>>>>
> >>>>>> I created a netcdf file. It can be read and and visualized by PANOPLY.
> >>>>>> But Idv does not recognize it as a netcdf file - obviously it is 
> >>>>>> looking
> >>>>>> for something I don't have in my file.
> >>>>>>
> >>>>>> I studied your documentation and even looked at an example file. But it
> >>>>>> beats me what I am doing wrong.
> >>>>>>
> >>>>>> Can you help?
> >>>>>>
> >>>>>> The file is too big to attach.
> >>>>>>
> >>>>>> Here is the PANOPLY screenshot. Can you help. Thank you. - Mark
> >>>>>>
> >>>>>> address@hidden
> >>>>>> Chief Scientist
> >>>>>> Science and Technology Corporation
> >>>>>> 10015 Old Columbia Road
> >>>>>> Suite E-250
> >>>>>> Columbia, MD 21046
> >>>>>> phone: 410-309-0818 x 106
> >>>>>> cell: 240-481-7390
> >>>>>> http://markschoeberl.com
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> I want to use the idv package to visualize my data.
> >>>>>>
> >>>>>> I created a netcdf file. It can be read and and visualized by PANOPLY.
> >>>>>> But Idv does not recognize it as a netcdf file - obviously it is 
> >>>>>> looking
> >>>>>> for something I don't have in my file.
> >>>>>>
> >>>>>> I studied your documentation and even looked at an example file. But it
> >>>>>> beats me what I am doing wrong.
> >>>>>>
> >>>>>> Can you help?
> >>>>>>
> >>>>>> The file is too big to attach.
> >>>>>>
> >>>>>> Here is the PANOPLY screenshot. Can you help. Thank you. - Mark
> >>>>>>
> >>>>>> address@hidden
> >>>>>> Chief Scientist
> >>>>>> Science and Technology Corporation
> >>>>>> 10015 Old Columbia Road
> >>>>>> Suite E-250
> >>>>>> Columbia, MD 21046
> >>>>>> phone: 410-309-0818 x 106
> >>>>>> cell: 240-481-7390
> >>>>>> http://markschoeberl.com
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> Ticket Details
> >>>>>> ===================
> >>>>>> Ticket ID: RZP-160177
> >>>>>> Department: Support IDV
> >>>>>> Priority: Normal
> >>>>>> Status: Closed
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>
> >>>>>
> >>>>> Ticket Details
> >>>>> ===================
> >>>>> Ticket ID: RZP-160177
> >>>>> Department: Support IDV
> >>>>> Priority: Normal
> >>>>> Status: Closed
> >>>>>
> >>>>
> >>>>
> >>>>
> >>>
> >>>
> >>> Ticket Details
> >>> ===================
> >>> Ticket ID: RZP-160177
> >>> Department: Support IDV
> >>> Priority: Normal
> >>> Status: Closed
> >>>
> >>
> >>
> >>
> >
> >
> > Ticket Details
> > ===================
> > Ticket ID: RZP-160177
> > Department: Support IDV
> > Priority: Normal
> > Status: Closed
> >
> 
> 
> 


Ticket Details
===================
Ticket ID: RZP-160177
Department: Support IDV
Priority: Normal
Status: Closed


NOTE: All email exchanges with Unidata User Support are recorded in the Unidata inquiry tracking system and then made publicly available through the web. If you do not want to have your interactions made available in this way, you must let us know in each email you send to us.