[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [netcdf-java] Reading NCEP BUFR

Hi Kevin:

I believe i have this working in the latest 4.3.16 source; and will be in the next release.

I dont have a workaround for the problem of having multiple bufr message types in the same file, however. I could make a utility for breaking it up into seperate files.

John

On 2/25/2013 12:09 PM, Kevin Manross wrote: 

We retrieve some of these files directly from NOMADS:

I.e.,

http://nomads.ncep.noaa.gov/pub/data/nccf/com/gfs/prod/gdas.20130129/gdas1.t00z.adpsfc.tm00.bufr_d.nr

(I ran the above file through netcdfAll-4.3.jar and received similar
errors to what I sent before)

Additionally, we get a daily tarfile from NCEP that contains all the
upper-air data that we archive.  I don't know if there is any special
processing done for these files or not.  Let me know if this is
pertinent and I will track it down.  The differences are mainly the
filtering of restricted data subsets, from what I can tell.

-kevin.




On 2/25/13 11:08 AM, John Caron wrote: 
Hi Kevin:

Im looking at this file, seeing some problems, probably mine.

One thing is that the file has mixed message types in it. Is this a
standard file from NCEP, or has it been concatenated ? What exactly is
it, and where did you get it?

John

On 2/25/2013 10:04 AM, Kevin Manross wrote: 

Hi John,

Thanks for looking into this.

I'm attaching an error log and one or two files if the server will
let me.  These files are from the GTS and are the files we archive in
the NCAR RDA, such as surface data:

http://rda.ucar.edu/datasets/ds461.0/#description

and upper air data

http://rda.ucar.edu/datasets/ds351.0/#description

I am getting similar errors for all the files listed at the top of
the error log file.

I am running this against the netcdfAll-4.3.jar bundle running Java
version 1.6 on yellowstone

address@hidden/NetCDFJava] % java -version
java version "1.6.0_22"
OpenJDK Runtime Environment (IcedTea6 1.10.4)
(rhel-1.41.1.10.4.el6-x86_64)
OpenJDK 64-Bit Server VM (build 20.0-b11, mixed mode)

I have tried these on several files over several dates ranging from
2002 to 2013.  (Note, the .le suffix means that these files were
converted to little endian.  I'm getting the same errors for the
respective big endian counterparts.)

Please let me know if I can provide additional info that would be
helpful.

-kevin.


On 2/24/13 9:26 AM, John Caron wrote:
On 2/8/2013 12:03 PM, Kevin Manross wrote: 

Greetings!

I've been delving into the murky waters of BUFR for the past few
months with the goal of converting our archive to netCDF.  I'm
working mainly (actually, exclusively up to this point) with
NCEP-style BUFR files.

When I started this project, I naturally thought of netCDF java,
but received errors when trying to open up the files I was wotking
with, so I went with the NCEP BUFRLIB  library and reacquainted
myself with FORTRAN, and even learned some FORTRAN "OOP" along the way.

I'm at a point where I'm re-examining how to move forward and I've
taken a look at netCDF java again and wanted to ask about the
following errors when I open an NCEP BUFR file from the GTS:

address@hidden/java] % java -Xmx1024m -classpath
JARS_Backup/netcdfAll-4.3.jar ucar.nc2.dataset.NetcdfDataset -in
fouthsubset.bufr -out temp.nc
Feb 8, 2013 11:18:45 AM
ucar.nc2.iosp.bufr.BufrIdentificationSection <init>
WARNING: month is zero
Feb 8, 2013 11:18:45 AM
ucar.nc2.iosp.bufr.BufrIdentificationSection <init>
WARNING: day is zero
Feb 8, 2013 11:18:45 AM
ucar.nc2.iosp.bufr.BufrIdentificationSection <init>
WARNING: month is zero
Feb 8, 2013 11:18:45 AM
ucar.nc2.iosp.bufr.BufrIdentificationSection <init>
WARNING: day is zero
Feb 8, 2013 11:18:45 AM
ucar.nc2.iosp.bufr.BufrIdentificationSection <init>
WARNING: month is zero
Feb 8, 2013 11:18:45 AM
ucar.nc2.iosp.bufr.BufrIdentificationSection <init>
WARNING: day is zero
Feb 8, 2013 11:18:45 AM
ucar.nc2.iosp.bufr.BufrIdentificationSection <init>
WARNING: month is zero
Feb 8, 2013 11:18:45 AM
ucar.nc2.iosp.bufr.BufrIdentificationSection <init>
WARNING: day is zero
Feb 8, 2013 11:18:46 AM ucar.nc2.iosp.bufr.BufrIosp open
WARNING: File gdas.adpupa.t12z.20120603.bufr has different BUFR
message types msgno=325; skipping
Feb 8, 2013 11:18:46 AM ucar.nc2.iosp.bufr.BufrIosp open
WARNING: File gdas.adpupa.t12z.20120603.bufr has different BUFR
message types msgno=325; skipping
...
...
...
WARNING: File gdas.adpupa.t12z.20120603.bufr has different BUFR
message types msgno=325; skipping
Feb 8, 2013 11:18:46 AM
ucar.nc2.iosp.bufr.BufrIdentificationSection <init>
WARNING: month is zero
Feb 8, 2013 11:18:46 AM
ucar.nc2.iosp.bufr.BufrIdentificationSection <init>
WARNING: day is zero
Feb 8, 2013 11:18:46 AM
ucar.nc2.iosp.bufr.BufrIdentificationSection <init>
WARNING: month is zero
Feb 8, 2013 11:18:46 AM
ucar.nc2.iosp.bufr.BufrIdentificationSection <init>
WARNING: day is zero
Feb 8, 2013 11:18:46 AM
ucar.nc2.iosp.bufr.BufrIdentificationSection <init>
WARNING: month is zero
Feb 8, 2013 11:18:46 AM
ucar.nc2.iosp.bufr.BufrIdentificationSection <init>
Read from gdas.adpupa.t12z.20120603.bufr write to temp.nc
Exception in thread "main" java.lang.IllegalArgumentException:
Dimension length =-1 must be > 0
    at ucar.nc2.Dimension.setLength(Dimension.java:283)
    at ucar.nc2.Dimension.<init>(Dimension.java:221)
    at
ucar.nc2.NetcdfFileWriter.addDimension(NetcdfFileWriter.java:295)
    at
ucar.nc2.NetcdfFileWriter.addDimension(NetcdfFileWriter.java:273)
    at ucar.nc2.FileWriter2.addNetcdf3(FileWriter2.java:231)
    at ucar.nc2.FileWriter2.write(FileWriter2.java:176)
    at ucar.nc2.dataset.NetcdfDataset.main(NetcdfDataset.java:1682)


Obviously the error at the end is related to the inability to
properly read the file.

I am able to open the file in tools-UI and display the data, though
the header column appears non-human-readable.

I know that the NCEP BUFR files come with the BUFR table embedded
in them.  Is this the source of the problem? If so, what is the
best way around it?

-kevin.

--
Kevin Manross
NCAR/CISL/Data Support Section
Phone: (303)-497-1218
Email:address@hidden <mailto:address@hidden>
Web:http://rda.ucar.edu


_______________________________________________
netcdf-java mailing list
address@hidden
For list information or to unsubscribe, 
visit:http://www.unidata.ucar.edu/mailing_lists/


Hi Kevin:

Can you send me the file with this problem? thanks

John 

--
Kevin Manross
NCAR/CISL/Data Support Section
Phone: (303)-497-1218
Email:address@hidden <mailto:address@hidden>
Web:http://rda.ucar.edu



--
Kevin Manross
NCAR/CISL/Data Support Section
Phone: (303)-497-1218
Email:address@hidden <mailto:address@hidden>
Web:http://rda.ucar.edu

NOTE: All email exchanges with Unidata User Support are recorded in the Unidata inquiry tracking system and then made publicly available through the web. If you do not want to have your interactions made available in this way, you must let us know in each email you send to us.