[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[IDV #BDO-300797]: catalog choking on folder of OK .ncml files

Brian,

The change may not be related to Unidata.  As far as we could tell, the
"slowness" is related to the URLs aggregated in the NCML.  Bear in that these
NCML files are hitting the (NCEP?) web server quite a bit. It could that NCEP
set up some mechanisms to limit the rate of URL access on their web servers(???)
(Especially after the cyber attacks that hit the NWS recently.)

Best,

Unidata IDV Support

> Let me clarify that it used to work.
> What has changed?
> 
> Last year about this time, for example, my class used these same server side 
> ncml aggregations with no trouble. This is not experimental new 
> functionality, it is a workhorse that fell down. Always disappointing when 
> the March of progress or at least of workable fixes goes backward.
> 
> What has changed?
> 
> Thanks for your help.
> Brian
> 
> Brian Mapes
> This message was typed on a cell phone, please forgive its terseness and any 
> errors.
> 
> > On Nov 21, 2014, at 5:38 PM, Unidata IDV Support <address@hidden> wrote:
> >
> > Hi Brian,
> >
> > re: IDV Catalog listing of directory with lots of NCML files that reference 
> > data
> > through URLs
> >
> >> Yes I did this and it never came back.
> >> Help ??
> >
> > Julien and I proved that the catalog view in the IDV will _eventually_
> > show all of the NCML files.  We did this by:
> >
> > - create a new server side view of a single NCML file
> >
> >  This listing is reasonably fast.
> >
> > - create a server side view of 10 NCML files
> >
> >  This listing is slow, but not slow enough to look like the IDV is
> >  hung.
> >
> > - full set
> >
> >  We didn't bother to wait for this to complete, but we are convinced
> >  that it eventually will.
> >
> > What is going on?
> >
> > Evidently, netCDF Java is actually opening the dataset referenced in each
> > URL named in the NCML files.  It seems to take between 4 and 5 seconds to
> > get the information from the 7 URLs referenced in each NCML file.  If the
> > wait time scales linearly, then it should take 335 (5 x 67) seconds to 
> > generate
> > the list for all of the NCML files you have in the NCEP1 6 hourly pressure 
> > level
> > data server side set of files.
> >
> > Can this be sped up?
> >
> > We don't know.  We'll have to run more tests to determine if the slowness
> > is really in the netCDF Java code or simply a slowness in the ESRL web
> > server's response to the query being made.
> >
> > Sorry we didn't have better news!
> >
> > Cheers,
> >
> > Tom
> > --
> > ****************************************************************************
> > Unidata User Support                                    UCAR Unidata Program
> > (303) 497-8642                                                 P.O. Box 3000
> > address@hidden                                   Boulder, CO 80307
> > ----------------------------------------------------------------------------
> > Unidata HomePage                       http://www.unidata.ucar.edu
> > ****************************************************************************
> >
> >
> > Ticket Details
> > ===================
> > Ticket ID: BDO-300797
> > Department: Support IDV
> > Priority: Normal
> > Status: Closed
> >
> 
> 


Ticket Details
===================
Ticket ID: BDO-300797
Department: Support IDV
Priority: Normal
Status: Closed