[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[IDV #HWF-642685]: Bundle created on one machine fails on others



Hi Dave-

Sorry for the delay in getting back to you.  Here's the problem in a nutshell.

- When the grib files are opened, the netCDF-Java package creates an index file 
(the .gbx9, .ncx files) which stores information about the grib file (e.g., 
offsets to products, and grib records).
- netCDF-Java has logic that says if the grib file is new than the index, 
recreate the index.  The check for this is done by looking at the last modified 
time of the file.  However, for soft links, it returns the last modified time 
of the linked file, not when the link was created.  Therein lies the problem - 
the index files are not getting regenerated, so the index information is wrong.

Let's take the case of the Latest analyses (case 2 below).  When the links are 
first created, we assume that there are no grib index files.  Once you access 
the files through the IDV either locally or through RAMADDA, the index files 
get generated.  Sometime later, the links are recreated.  When the IDV reads 
the data, the netcdf-java library compares the last modified dates.  Since the 
older linked files in the list have not changed, the indexes are not recreated. 
 However, the file that the link latest_00_GFS.grb points to is now different 
than what it was the last time and the latest_00_GFS.grb.gbx9 index file has 
the wrong information.  

For the non-grib files, the aggregations work using links because there are no 
index files involved.

So, I'm going to throw this back to the IDV group to talk over with the 
netcdf-java folks.  I think in the next version of netcdf-java they are 
switching to the java.nio package for reading files.  That has support for 
looking at the last modified time of the link, not the file being linked.  
Also, as of netcdf-java 4.3, I've been told that aggregations of grib files 
shouldn't be done using ncml.  However, I have not received an alternative that 
doesn't involve running the TDS. Sean was supposed to come up with a solution.  
Or, perhaps Unidata can think of another way to do this.  It's not a RAMADDA 
problem - the same problem exists when pointing to local files in the IDV 
(contrary to what I told you at one point because I was deleting the grib index 
files).  you could have your script that links the files delete the indexes in 
~/.ramadda/tmp/nj22, but that is a hack.

Sorry I don't have a better answer.

Don

> On 1/8/14 11:44 AM, David P Dempsey wrote:
> > Ah, I misunderstood what Yuan was talking about. I didnât realize that
> > the creation of *.gbx9 and *.ncx files had anything to do with RAMADDA.
> > Itâs nice to have the problem fixed, thoughâthanks!
> >
> > However, another problem thatâs been really consternating (rather than
> > just annoying, like the *.gbx9 and *.ncx file creation) arises when I:
> >
> > (1) create symbolic links to GRIB (model output) files that we receive
> > using the LDM, so that there is an unchanging set of names (the symbolic
> > links) always pointing to the latest set of model output files;
> >
> > (2) tell the IDV to access these data files via RAMADDA using the
> > symbolic links, and time-aggregate a series of them;
> >
> > (3) save the display I create as a bundle; and
> >
> > (4) load the bundle later, after the symbolic links have been reassigned
> > to new GRIB files.
> >
> > The bundle wonât generate a displayâthe IDV just issues error message:
> > "Couldn't get data. Error reading data from server.â and "Initializing
> > after unpersistence. java.lang.NullPointerExceptionerror messages
> > (details below).â
> >
> > However, if I create the same display by accessing the data sources as
> > local files (using the symbolic links) rather than via RAMADDA, the
> > saved bundle works fineâit will always display the latest of model
> > output. So, the problem is with the way RAMADDA handles the files using
> > the symbolic links once those links have been reassigned to new files.
> >
> > (Iâm not certain that time-aggregating the data sources is a necessary
> > condition for the problem to arise. I need to check on that.)
> 
> I would guess it is.  Does the symbolic link get created anew each time?
> 
> Jeff put in a fix for another issue related to aggregations which is in
> the 1.6a release.  Does this error still occur with the new RAMADDA release?
> 
> If so, let me know and I can look into this further.
> 
> Don
> 
> > IDV error message details:
> >
> > ucar.unidata.data.BadDataException: Error reading data from server
> > at ucar.visad.data.GeoGridFlatField.readData(GeoGridFlatField.java:248)
> > at visad.data.CachedFlatField.getMyValues(CachedFlatField.java:463)
> > at visad.data.CachedFlatField.unpackFloats(CachedFlatField.java:602)
> > at visad.data.CachedFlatField.getRanges(CachedFlatField.java:369)
> > at
> > ucar.unidata.data.grid.GeoGridAdapter.readTimeStep(GeoGridAdapter.java:1563)
> > at ucar.unidata.data.grid.GeoGridAdapter.access$100(GeoGridAdapter.java:113)
> > at ucar.unidata.data.grid.GeoGridAdapter$1.run(GeoGridAdapter.java:1455)
> > at visad.util.ThreadManager$1.run(ThreadManager.java:292)
> > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
> > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> > at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> > at
> > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
> > at
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
> > at java.lang.Thread.run(Thread.java:695)
> >>
> >> Don
> >>
> >> On 1/7/14 9:44 PM, David P Dempsey wrote:
> >>>
> >>> On Jan 7, 2014, at 9:26 AM, Unidata IDV Support
> >>> <address@hidden <mailto:address@hidden>
> >>> <mailto:address@hidden>> wrote:
> >>>
> >>>>> Dave,
> >>>>     I talked to Don and he believe it is fixed in the 1.6, you can
> >>>> reinstall the 1.6 release.
> >>>
> >>> Yuan and Don,
> >>>
> >>> My initial testing with RAMADDA v1.6a  supports the idea that Don has
> >>> fixed the problem.
> >>>
> >>> If the apparent fix holds up, Iâll be psyched, even if I have to
> >>> recreate a bunch of bundles to work with the new version of RAMADDA!
> >>>
> >>> â Dave
> >>>
> >>> P.S.
> >>>
> >>> If I understand it right, the problem arose when:
> >>>
> >>> (1) I access data sources via our RAMADDA server.
> >>> (2) The data sources selected in the IDV are symbolic links to the
> >>> actual data files.
> >>> (3) The data files contain gridded data, and I time aggregate them.
> >>>
> >>> (Iâm not totally sure if that last condition is necessary to create the
> >>> problem. Even if it is, Iâm struck by how often I created bundles that
> >>> meet all three conditions.)
> >>>
> >>> ****************************************************************
> >>> *                                      |        __ __    \|/  *
> >>> *   Dr. Dave Dempsey                   | ) ^  /|| ||\  --0-- *
> >>> *   Dept. of Earth & Climate Sciences  |) ) ^  / ||_|| \  /|\  *
> >>> *   San Francisco State University     | ) )  /  | _ |  \      *
> >>> *   1600 Holloway Ave.                 |) )/   || ||   \     *
> >>> *   San Francisco, CA   94132          | ) )   ||_||    \    *
> >>> *                                      |) ) )   | _ |     \   *
> >>> *   Phone:  (415) 338-7716             | )   )  || ||      \  *
> >>> *   FAX:    (415) 338-7705             |) )  )~~~~~~~~~~~~~~~*
> >>> *   Email: address@hidden <mailto:address@hidden>
> >>> <mailto:address@hidden>           |  )  )
> >>> )  ~  ~   ~ ~  *
> >>> *                                      |) )   ) )  ~   ~  ~   *
> >>> ****************************************************************
> >>>
> >>>
> >>>
> >>
> >> --
> >> Don Murray
> >> NOAA/ESRL/PSD and CIRES
> >> 303-497-3596
> >> http://www.esrl.noaa.gov/psd/people/don.murray/
> >>
> >
> > ****************************************************************
> > *                                      |        __ __    \|/  *
> > *   Dr. Dave Dempsey                   | ) ^  /|| ||\  --0-- *
> > *   Dept. of Earth & Climate Sciences  |) ) ^  / ||_|| \  /|\  *
> > *   San Francisco State University     | ) )  /  | _ |  \      *
> > *   1600 Holloway Ave.                 |) )/   || ||   \     *
> > *   San Francisco, CA   94132          | ) )   ||_||    \    *
> > *                                      |) ) )   | _ |     \   *
> > *   Phone:  (415) 338-7716             | )   )  || ||      \  *
> > *   FAX:    (415) 338-7705             |) )  )~~~~~~~~~~~~~~~*
> > *   Email: address@hidden <mailto:address@hidden>           |  )  )
> > )  ~  ~   ~ ~  *
> > *                                      |) )   ) )  ~   ~  ~   *
> > ****************************************************************
> >
> >
> >
> 
> --
> Don Murray
> NOAA/ESRL/PSD and CU-CIRES
> 303-497-3596
> http://www.esrl.noaa.gov/psd/people/don.murray/
> 
> 


Ticket Details
===================
Ticket ID: HWF-642685
Department: Support IDV
Priority: Critical
Status: Open


NOTE: All email exchanges with Unidata User Support are recorded in the Unidata inquiry tracking system and then made publicly available through the web. If you do not want to have your interactions made available in this way, you must let us know in each email you send to us.