Hi John,
I can imagine creating a tool to break up the BUFR subsets into
multiple files. Do you feel that is the best (or only) approach?
The reason I ask is that a typical BUFR file in our archive (an
'adpsfc' file) can have on the order of 66,000 subsets which makes
for a lot of netCDF files.
Perhaps we could break down our BUFR file by category and creating
one file per category with each of those files having multiple
records. As an example, here is the breakdown of an 'adpsfc' file
that I've been testing with:
manross@geyser11[BUFR2NC/src] % grep "MESSAGE TYPE NC" ufdump-gdas.adpsfc.t12z.20120603.bufr | uniq -c
1149 MESSAGE TYPE NC000000
18375 MESSAGE TYPE NC000001
1331 MESSAGE TYPE NC000002
45500 MESSAGE TYPE NC000007
Total: 66355
Creating a netCDF file per "MESSAGE TYPE" which share common
parameters containing all records/subsets for that type.
What I'm really interested in is: looking down the road a bit I
would like to try to use the THREDDS Data Server to handle our
archive of NCEP BUFR observations if possible. If you wouldn't
mind, I would like your ballpark estimates on A) whether this is
feasible, B) any idea on server load that this may cause, and C) If
I develop this, can I add such a feature to TDS?
(Sorry, I know this overlaps wit the thredds group, but wasn't sure
where to send the questions.)
Many thanks!
-kevin.
On 3/1/13 12:25 PM, John Caron wrote:
Hi
Kevin:
I believe i have this working in the latest 4.3.16 source; and
will be in the next release.
I dont have a workaround for the problem of having multiple bufr
message types in the same file, however. I could make a utility
for breaking it up into seperate files.
John
On 2/25/2013 12:09 PM, Kevin Manross wrote:
We retrieve some of these files directly from NOMADS:
I.e.,
http://nomads.ncep.noaa.gov/pub/data/nccf/com/gfs/prod/gdas.20130129/gdas1.t00z.adpsfc.tm00.bufr_d.nr
(I ran the above file through netcdfAll-4.3.jar and received
similar
errors to what I sent before)
Additionally, we get a daily tarfile from NCEP that contains all
the
upper-air data that we archive. I don't know if there is any
special
processing done for these files or not. Let me know if this is
pertinent and I will track it down. The differences are mainly
the
filtering of restricted data subsets, from what I can tell.
-kevin.
On 2/25/13 11:08 AM, John Caron wrote:
Hi Kevin:
Im looking at this file, seeing some problems, probably mine.
One thing is that the file has mixed message types in it. Is
this a
standard file from NCEP, or has it been concatenated ? What
exactly is
it, and where did you get it?
John
On 2/25/2013 10:04 AM, Kevin Manross wrote:
Hi John,
Thanks for looking into this.
I'm attaching an error log and one or two files if the
server will
let me. These files are from the GTS and are the files we
archive in
the NCAR RDA, such as surface data:
http://rda.ucar.edu/datasets/ds461.0/#description
and upper air data
http://rda.ucar.edu/datasets/ds351.0/#description
I am getting similar errors for all the files listed at the
top of
the error log file.
I am running this against the netcdfAll-4.3.jar bundle
running Java
version 1.6 on yellowstone
manross@yslogin3[BUFRTesting/NetCDFJava] % java -version
java version "1.6.0_22"
OpenJDK Runtime Environment (IcedTea6 1.10.4)
(rhel-1.41.1.10.4.el6-x86_64)
OpenJDK 64-Bit Server VM (build 20.0-b11, mixed mode)
I have tried these on several files over several dates
ranging from
2002 to 2013. (Note, the .le suffix means that these files
were
converted to little endian. I'm getting the same errors for
the
respective big endian counterparts.)
Please let me know if I can provide additional info that
would be
helpful.
-kevin.
On 2/24/13 9:26 AM, John Caron wrote:
On 2/8/2013 12:03 PM, Kevin Manross
wrote:
Greetings!
I've been delving into the murky waters of BUFR for the
past few
months with the goal of converting our archive to
netCDF. I'm
working mainly (actually, exclusively up to this point)
with
NCEP-style BUFR files.
When I started this project, I naturally thought of
netCDF java,
but received errors when trying to open up the files I
was wotking
with, so I went with the NCEP BUFRLIB library and
reacquainted
myself with FORTRAN, and even learned some FORTRAN "OOP"
along the way.
I'm at a point where I'm re-examining how to move
forward and I've
taken a look at netCDF java again and wanted to ask
about the
following errors when I open an NCEP BUFR file from the
GTS:
manross@yslogin2[toolbox/java] % java -Xmx1024m
-classpath
JARS_Backup/netcdfAll-4.3.jar
ucar.nc2.dataset.NetcdfDataset -in
fouthsubset.bufr -out temp.nc
Feb 8, 2013 11:18:45 AM
ucar.nc2.iosp.bufr.BufrIdentificationSection
<init>
WARNING: month is zero
Feb 8, 2013 11:18:45 AM
ucar.nc2.iosp.bufr.BufrIdentificationSection
<init>
WARNING: day is zero
Feb 8, 2013 11:18:45 AM
ucar.nc2.iosp.bufr.BufrIdentificationSection
<init>
WARNING: month is zero
Feb 8, 2013 11:18:45 AM
ucar.nc2.iosp.bufr.BufrIdentificationSection
<init>
WARNING: day is zero
Feb 8, 2013 11:18:45 AM
ucar.nc2.iosp.bufr.BufrIdentificationSection
<init>
WARNING: month is zero
Feb 8, 2013 11:18:45 AM
ucar.nc2.iosp.bufr.BufrIdentificationSection
<init>
WARNING: day is zero
Feb 8, 2013 11:18:45 AM
ucar.nc2.iosp.bufr.BufrIdentificationSection
<init>
WARNING: month is zero
Feb 8, 2013 11:18:45 AM
ucar.nc2.iosp.bufr.BufrIdentificationSection
<init>
WARNING: day is zero
Feb 8, 2013 11:18:46 AM ucar.nc2.iosp.bufr.BufrIosp open
WARNING: File gdas.adpupa.t12z.20120603.bufr has
different BUFR
message types msgno=325; skipping
Feb 8, 2013 11:18:46 AM ucar.nc2.iosp.bufr.BufrIosp open
WARNING: File gdas.adpupa.t12z.20120603.bufr has
different BUFR
message types msgno=325; skipping
...
...
...
WARNING: File gdas.adpupa.t12z.20120603.bufr has
different BUFR
message types msgno=325; skipping
Feb 8, 2013 11:18:46 AM
ucar.nc2.iosp.bufr.BufrIdentificationSection
<init>
WARNING: month is zero
Feb 8, 2013 11:18:46 AM
ucar.nc2.iosp.bufr.BufrIdentificationSection
<init>
WARNING: day is zero
Feb 8, 2013 11:18:46 AM
ucar.nc2.iosp.bufr.BufrIdentificationSection
<init>
WARNING: month is zero
Feb 8, 2013 11:18:46 AM
ucar.nc2.iosp.bufr.BufrIdentificationSection
<init>
WARNING: day is zero
Feb 8, 2013 11:18:46 AM
ucar.nc2.iosp.bufr.BufrIdentificationSection
<init>
WARNING: month is zero
Feb 8, 2013 11:18:46 AM
ucar.nc2.iosp.bufr.BufrIdentificationSection
<init>
Read from gdas.adpupa.t12z.20120603.bufr write to
temp.nc
Exception in thread "main"
java.lang.IllegalArgumentException:
Dimension length =-1 must be > 0
at ucar.nc2.Dimension.setLength(Dimension.java:283)
at
ucar.nc2.Dimension.<init>(Dimension.java:221)
at
ucar.nc2.NetcdfFileWriter.addDimension(NetcdfFileWriter.java:295)
at
ucar.nc2.NetcdfFileWriter.addDimension(NetcdfFileWriter.java:273)
at
ucar.nc2.FileWriter2.addNetcdf3(FileWriter2.java:231)
at ucar.nc2.FileWriter2.write(FileWriter2.java:176)
at
ucar.nc2.dataset.NetcdfDataset.main(NetcdfDataset.java:1682)
Obviously the error at the end is related to the
inability to
properly read the file.
I am able to open the file in tools-UI and display the
data, though
the header column appears non-human-readable.
I know that the NCEP BUFR files come with the BUFR table
embedded
in them. Is this the source of the problem? If so, what
is the
best way around it?
-kevin.
--
Kevin Manross
NCAR/CISL/Data Support Section
Phone: (303)-497-1218
Email:address@hidden <mailto:address@hidden>
Web:http://rda.ucar.edu
_______________________________________________
netcdf-java mailing list
address@hidden
For list information or to unsubscribe,
visit:http://www.unidata.ucar.edu/mailing_lists/
Hi Kevin:
Can you send me the file with this problem? thanks
John
--
Kevin Manross
NCAR/CISL/Data Support Section
Phone: (303)-497-1218
Email:address@hidden <mailto:address@hidden>
Web:http://rda.ucar.edu
--
Kevin Manross
NCAR/CISL/Data Support Section
Phone: (303)-497-1218
Email:address@hidden <mailto:address@hidden>
Web:http://rda.ucar.edu
|
