[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: FW: Re: NCEP North American Reanalysis (fwd)

On Fri, 22 Oct 2004, Dan Swank wrote:

> Robb,
> One final brief question,
> are the cdl files time independant?  (Can we make one and use it as a
> template for every other narr file)

dan,

yes, once made they can be used as long as they are valid for the
datasets. gribtocdl is just a utility to assist in making cdls.

robb...

> Or does a cdl file need to be created for every file we want to convert?
>
> -Dan
>
>
> Robb Kambic wrote:
>
> >On Thu, 21 Oct 2004, Dan Swank wrote:
> >
> >
> >
> >>Almost have it.
> >>The latest version indeed corrects the gribtocdl bug.
> >>I was able to create the narr-b .nc file going through both steps.
> >>Next i tried to convert the narr-a files, which have more then 400 grib
> >>records each.
> >>gribtocdl wrote this record:
> >>
> >>        float  03tot_atm(record,y,x) ;
> >>               03tot_atm:long_name = "Total ozone at entire atmosphere
> >>considered as a single layer" ;
> >>               03tot_atm:GRIB_parameter_number = 237 ;
> >>               03tot_atm:GRIB_level_flag = 200 ;
> >>               03tot_atm:units = "Kg/m2" ;
> >>               03tot_atm:_FillValue = -9999.f ;
> >>               03tot_atm:navigation = "nav" ;
> >>
> >>Gribtonc doesn't seem to like the variable name starting with a number
> >>zero, and throws a syntax error.
> >>
> >>
> >
> >dan,
> >
> >in netcdf no var can start with a number, guess that parameter hasn't been
> >used before.  the problem has been fixed and tested on the file you gave
> >as a test. sorry but you have to download the latest release:
> >
> >decoders-3.0.6.tar.Z
> >
> >robb...
> >
> >ps the size of netcdf output file grew to ~625mg from ~56 mg input file.
> >that;s about right
> >
> >
> >
> >
> >>This is also strange becuase the NARR doesn' tcontain any ozone fields.
> >>
> >>Suggestions?  A sample narr-a file to try:
> >>http://nomads.ncdc.noaa.gov/tmp/narr-a_221_20010101_0000_000.grb
> >>
> >>-Dan
> >>
> >>
> >>
> >>Robb Kambic wrote:
> >>
> >>
> >>
> >>>On Thu, 21 Oct 2004, Dan Swank wrote:
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>>Robb,
> >>>>
> >>>>No luck.  I've recompiled the decoders package again and gribtocdl does
> >>>>the same thing.
> >>>>We have:
> >>>>netcdf-3.5.1          (built from source)
> >>>>udunits-1.11.7       ( Used binary installation since our system had
> >>>>trouble compiling this from source)
> >>>>decoders-2.3.8n    ( built from source)
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>Dan,
> >>>
> >>>that's a really old distribution of the decoders package. Your problem
> >>>sounds like a bug that was fixed back at that time. the latest decoders
> >>>package is decoders-3.0.5.tar.Z hopefully the newest distribution will fix
> >>>your problem.
> >>>
> >>>robb...
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>>And the output from the compilation can be found here
> >>>>http://nomads.ncdc.noaa.gov/tmp/decoders-2.3.8n.logs.tar
> >>>>
> >>>>OS:
> >>>>Red Hat Linux release 7.3 (Valhalla)
> >>>>
> >>>>We also have a
> >>>>Red Hat Linux release 8.0 (Psyche)
> >>>>We can try this if you think it will really make much of a difference.
> >>>>Avbl but we dont' have root user on either of these so installing
> >>>>tpackages can be a bit tricky.
> >>>>
> >>>>If you need anymore information let me know.
> >>>>
> >>>>-Dan
> >>>>
> >>>>
> >>>>
> >>>>Robb Kambic wrote:
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>>no need to include the others now...
> >>>>>
> >>>>>
> >>>>>On Wed, 20 Oct 2004, Dan Swank wrote:
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>Getting a little closer...
> >>>>>>
> >>>>>>Set the Env var and now udunits works as expected... however gribtocdl
> >>>>>>still gets stuck in an infinate loop and don't seem to work right.
> >>>>>>when i ran it on the narr-b GRIB it got to the following section
> >>>>>>
> >>>>>>
> >>>>>>// Navigation
> >>>>>>nav_model = "GRIB1" ;
> >>>>>>grid_type_code = 3 ;
> >>>>>>grid_type = "Lambert conformal projection" ;
> >>>>>>grid_name = " " ;
> >>>>>>grid_center = 7 ;
> >>>>>>grid_number = 221 ;
> >>>>>>i_dim = "lon" ;
> >>>>>>j_dim = "lat" ;
> >>>>>>Nx = 349 ;
> >>>>>>Ny = 277 ;
> >>>>>>La1 = 1.000000 ;
> >>>>>>Lo1 = -145.500000 ;
> >>>>>>Lov = -107.000000 ;
> >>>>>>Dx = 32463.000000 ;
> >>>>>>Dy = 32463.000000 ;
> >>>>>>// Pole in proj. plane = North ;
> >>>>>>centers = 1 ;
> >>>>>>Latin1 = 50.000000 ;
> >>>>>>Latin2 = 50.000000 ;
> >>>>>>Splat = 0.000000 ;
> >>>>>>SPlon = 0.000000 ;
> >>>>>>ResCompFlag = 0 ;
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>if you notice the cdl i made didn't have lon and lats.  there must be
> >>>>>something wrong with your gribtocdl build.  for a test, i build the
> >>>>>decoders package on a linux box fedora 9 and gribtocdl created the cdl
> >>>>>fine.  remake the distribution, if you can try it on another box. whats
> >>>>>your os?
> >>>>>
> >>>>>robb...
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>lon = -145.50,-146.50,-147.50,-148.50,-149.50,-150.50,-151.50,-152.50,
> >>>>>>    -153.50,-154.50,-155.50,-156.50,-157.50,-158.50,-159.50,-160.50,
> >>>>>>    -161.50,-162.50,-163.50,-164.50,-165.50,-166.50,-167.50,-168.50,
> >>>>>>
> >>>>>>Then continues decrementing the longitue values and don't stop.  I was
> >>>>>>forced to CNTL-C it and remove the lon section completely, putting the
> >>>>>>curly bracket at the end.  Know why it might do this?
> >>>>>>After setting the env var, gribtonc successfully re-created narr.nc from
> >>>>>>the narr.cdl you supplied.  However when I tried my CDL from above i get
> >>>>>>the following:
> >>>>>>
> >>>>>>./narr-b_221_20010101_0000_000.grb
> >>>>>>ncgen: narr-b2.cdl line 6: dimension length must be positive
> >>>>>>ncgen: narr-b2.cdl line 7: dimension length must be positive
> >>>>>>ncgen: narr-b2.cdl line 73: Nx not declared as a variable, fatal error
> >>>>>>
> >>>>>>Does gribtocdl require any ENV vars to be set?
> >>>>>>
> >>>>>>Appreciate the help.
> >>>>>>-Dan
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>Robb Kambic wrote:
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>On Wed, 20 Oct 2004, Dan Swank wrote:
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>I copied the narr.cdl file from the FTP and tried:
> >>>>>>>>
> >>>>>>>>gribtonc -v -l ./log -e ./error ./narr.cdl
> >>>>>>>>narr-b_221_20010101_0000_000.nc < narr-b_221_20010101_0000_000.grb
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>here's the process
> >>>>>>>
> >>>>>>># creates cdl file narr.cdl
> >>>>>>>% gribtocdl -v narr-b_221_20010101_0000_000.grb > narr.cdl
> >>>>>>>
> >>>>>>># creates netcdf file narr.nc  in verbose mode, logging to screen
> >>>>>>>% gribtonc -vl - narr.cdl narr.nc < narr-b_221_20010101_0000_000.grb
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>Result
> >>>>>>>>Segmentation fault
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>explained below, set UDUNITS_PATH
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>and
> >>>>>>>>Oct 20 19:31:05 gribtonc[32594]: Starting Up
> >>>>>>>>in the ./log file.
> >>>>>>>>
> >>>>>>>>Any idea whats going on?  It is likely gribtonc (or more likely one of
> >>>>>>>>its dependancies) is not installed correctly on our system (RedHat 
> >>>>>>>>7.3)
> >>>>>>>>Also, what did you use to create this CDL file?  The data in cdl seems
> >>>>>>>>like a translation of the information in the grib PDS, reworked into a
> >>>>>>>>format that ncgen can understand.
> >>>>>>>>Any way i can get anymore debug information regarding this?
> >>>>>>>>
> >>>>>>>>Only hunch is that it is not happy with the udunits package
> >>>>>>>>
> >>>>>>>>-> /usr/local/udunits-1.11.7/udunits-1.11.7/bin/udunits
> >>>>>>>>udunits(3): Couldn't open units database 
> >>>>>>>>"/upc/udunits/etc/udunits.dat":
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>ahh, that's the problem. udunits can't find udunits.dat  either place
> >>>>>>>udunits.dat in dir /upc/udunits/etc/ or set environment var
> >>>>>>>
> >>>>>>>% setenv UDUNITS_PATH /your/udunits/path/udunits.dat
> >>>>>>>
> >>>>>>>replace /your/udunits/path/ with appropriate path
> >>>>>>>
> >>>>>>>robb...
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>No such file or directory
> >>>>>>>>Segmentation fault
> >>>>>>>>
> >>>>>>>>But, while building the unidata decoders package it only seemed to 
> >>>>>>>>want
> >>>>>>>>the .dat .a and .h files within these packages.
> >>>>>>>>Would this be the source of the problems?
> >>>>>>>>
> >>>>>>>>-Dan
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>Robb Kambic wrote:
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>>Thanks russ for the clarification.  i was assuming you were familar 
> >>>>>>>>>with
> >>>>>>>>>the decoders process. if i can answer any more questions let me 
> >>>>>>>>>know. i'll
> >>>>>>>>>try to be more descriptive.
> >>>>>>>>>
> >>>>>>>>>robb...
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>On Tue, 19 Oct 2004, Dan Swank wrote:
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>>Russ
> >>>>>>>>>>
> >>>>>>>>>>Actually, i was, at first, trying
> >>>>>>>>>>GRIB -( gribtocdl )-> CDL -( gribtonc )-> NetCDF
> >>>>>>>>>>Which i now understand is completely wrong, thanks for the help.
> >>>>>>>>>>As you have noticed, we are completely unfamiliar with these 
> >>>>>>>>>>programs.
> >>>>>>>>>>
> >>>>>>>>>>Attempting it the correct way now, i'll let you know how it goes.
> >>>>>>>>>>
> >>>>>>>>>>-Dan
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>Russ Rew wrote:
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>>Robb,
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>>i downloaded the file from the last message, created a cdl, and 
> >>>>>>>>>>>>decoded
> >>>>>>>>>>>>the grb file. there might be something wrong with Dan's  decoders 
> >>>>>>>>>>>>build or
> >>>>>>>>>>>>it could be a platform issue. this was done on a solaris box 5.9  
> >>>>>>>>>>>>The
> >>>>>>>>>>>>narr.cdl file is attached and the files narr.cdl, narr.grb, and 
> >>>>>>>>>>>>narr.nc are
> >>>>>>>>>>>>in the Unidata's ftp dir at
> >>>>>>>>>>>>
> >>>>>>>>>>>>ftp unidata.ucar.edu
> >>>>>>>>>>>>
> >>>>>>>>>>>>% cd pub/contrib
> >>>>>>>>>>>>% mget narr*
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>Thanks Robb.  The file sizes are:
> >>>>>>>>>>>
> >>>>>>>>>>>-rw-rw-r--   1 rkambic  ustaff   3486764 Oct 19 13:32 narr.nc
> >>>>>>>>>>>-rw-rw-r--   1 rkambic  ustaff      8560 Oct 19 13:32 narr.cdl
> >>>>>>>>>>>-rw-rw-r--   1 rkambic  ustaff   1398914 Oct 19 13:32 narr.grb
> >>>>>>>>>>>
> >>>>>>>>>>>so the netCDF file is about 2.5 times as big as the GRIB file.
> >>>>>>>>>>>
> >>>>>>>>>>>I'm guessing the source of the problem may come from using
> >>>>>>>>>>>
> >>>>>>>>>>>GRIB -> (via gribtocdl) -> CDL -> (via ncgen) -> netCDF
> >>>>>>>>>>>
> >>>>>>>>>>>(Using gribtocdl to generate a very large CDL file and then using 
> >>>>>>>>>>>ncgen
> >>>>>>>>>>>to convert that into a netCDF file.)
> >>>>>>>>>>>
> >>>>>>>>>>>I think Robb used the following tools instead:
> >>>>>>>>>>>
> >>>>>>>>>>>GRIB -> (via gribtocdl) -> CDL
> >>>>>>>>>>>GRIB and CDL -> (via gribtonc) -> netCDF
> >>>>>>>>>>>
> >>>>>>>>>>>(Using gribtocdl to generate a small CDL file describing structure 
> >>>>>>>>>>>of
> >>>>>>>>>>>the desired netCDF file and then using gribtonc to convert the GRIB
> >>>>>>>>>>>data into the netCDF file.)
> >>>>>>>>>>>
> >>>>>>>>>>>--Russ
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>--
> >>>>>>>>>>Dan Swank <address@hidden>
> >>>>>>>>>>NOMADS programmer
> >>>>>>>>>>STG, Incorporated - Government Contractor
> >>>>>>>>>>151 Patton Avenue, Room 514
> >>>>>>>>>>Asheville, NC 28801
> >>>>>>>>>>Phone: 828-271-4007
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>===============================================================================
> >>>>>>>>>Robb Kambic                             Unidata Program Center
> >>>>>>>>>Software Engineer III                           Univ. Corp for 
> >>>>>>>>>Atmospheric Research
> >>>>>>>>>address@hidden                  WWW: http://www.unidata.ucar.edu/
> >>>>>>>>>===============================================================================
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>--
> >>>>>>>>Dan Swank <address@hidden>
> >>>>>>>>NOMADS programmer
> >>>>>>>>STG, Incorporated - Government Contractor
> >>>>>>>>151 Patton Avenue, Room 514
> >>>>>>>>Asheville, NC 28801
> >>>>>>>>Phone: 828-271-4007
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>===============================================================================
> >>>>>>>Robb Kambic                               Unidata Program Center
> >>>>>>>Software Engineer III                     Univ. Corp for Atmospheric 
> >>>>>>>Research
> >>>>>>>address@hidden            WWW: http://www.unidata.ucar.edu/
> >>>>>>>===============================================================================
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>--
> >>>>>>Dan Swank <address@hidden>
> >>>>>>NOMADS programmer
> >>>>>>STG, Incorporated - Government Contractor
> >>>>>>151 Patton Avenue, Room 514
> >>>>>>Asheville, NC 28801
> >>>>>>Phone: 828-271-4007
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>===============================================================================
> >>>>>Robb Kambic                                 Unidata Program Center
> >>>>>Software Engineer III                       Univ. Corp for Atmospheric 
> >>>>>Research
> >>>>>address@hidden              WWW: http://www.unidata.ucar.edu/
> >>>>>===============================================================================
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>--
> >>>>Dan Swank <address@hidden>
> >>>>NOMADS programmer
> >>>>STG, Incorporated - Government Contractor
> >>>>151 Patton Avenue, Room 514
> >>>>Asheville, NC 28801
> >>>>Phone: 828-271-4007
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>===============================================================================
> >>>Robb Kambic                                   Unidata Program Center
> >>>Software Engineer III                         Univ. Corp for Atmospheric 
> >>>Research
> >>>address@hidden                WWW: http://www.unidata.ucar.edu/
> >>>===============================================================================
> >>>
> >>>
> >>>
> >>>
> >>--
> >>Dan Swank <address@hidden>
> >>NOMADS programmer
> >>STG, Incorporated - Government Contractor
> >>151 Patton Avenue, Room 514
> >>Asheville, NC 28801
> >>Phone: 828-271-4007
> >>
> >>
> >>
> >>
> >
> >===============================================================================
> >Robb Kambic                             Unidata Program Center
> >Software Engineer III                           Univ. Corp for Atmospheric 
> >Research
> >address@hidden                  WWW: http://www.unidata.ucar.edu/
> >===============================================================================
> >
> >
>
> --
> Dan Swank <address@hidden>
> NOMADS programmer
> STG, Incorporated - Government Contractor
> 151 Patton Avenue, Room 514
> Asheville, NC 28801
> Phone: 828-271-4007
>
>

===============================================================================
Robb Kambic                                Unidata Program Center
Software Engineer III                      Univ. Corp for Atmospheric Research
address@hidden             WWW: http://www.unidata.ucar.edu/
===============================================================================