[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[IDV #BDO-300797]: catalog choking on folder of OK .ncml files



Brian, Don, et al.,

Speaking entirely speculatively, does this have anything to do with the NWS hack
from a few weeks ago?  Could the NOAA web servers be limiting successive URL
requests (which the NCML aggregation seems to do) to prevent DOS attacks?

(No, this problem is not b/c of a change in Java.)

Best,

Unidata IDV Support

> Tom and Julien, (cc hoop)
> 
> 
> Might this (cc from Hoop) be an issue ?
> It used to be faster so we are narrowing to what might have changed?
> 
> 
> Has there been a change in java or something? About 2-3 other things went a 
> bit
> haywire this week, including on weather.rsmas. I havenât updated weather 
> recently,
> but some ISL scripts stopped working. (Updating to IDV5.0u2 fixed that).
> 
> 
> 
> Brian
> 
> 
> 
> On Nov 24, 2014, at 11:00 AM, NOAA ESRL/PSD Web & Data Team <address@hidden> 
> wrote:
> 
> > Something has changed recently, though I would be surprised if it
> > was the culprit.  Last month, we changed the format of the R1 files
> > from NetCDF3 to NetCDF4-Classic.  In general, this means the files
> > read more quickly, in our experience at least.  Since OPeNDAP renders
> > all files the same to the reading application regardless of source
> > format (I.E., essentially as NetCDF3 files), this would imply that
> > the Java NetCDF API used by the TDS server software is less efficient
> > reading NetCDF4-Classic files.  This seems unreasonable as the Java
> > API is generally the most forward-thinking and capable of the NetCDF
> > APIs.  As I recall, it spoke NetCDF4 before the others.  So, I'm
> > puzzled. (from Hoop)
> 
> 
> On Nov 21, 2014, at 5:38 PM, Unidata IDV Support <address@hidden> wrote:
> 
> > Hi Brian,
> >
> > re: IDV Catalog listing of directory with lots of NCML files that reference 
> > data
> > through URLs
> 
> > Julien and I proved that the catalog view in the IDV will _eventually_
> > show all of the NCML files.  We did this by:
> >
> > - create a new server side view of a single NCML file
> >
> >  This listing is reasonably fast.
> >
> > - create a server side view of 10 NCML files
> >
> >  This listing is slow, but not slow enough to look like the IDV is
> >  hung.
> >
> > - full set
> >
> >  We didn't bother to wait for this to complete, but we are convinced
> >  that it eventually will.
> >
> > What is going on?
> >
> > Evidently, netCDF Java is actually opening the dataset referenced in each
> > URL named in the NCML files.  It seems to take between 4 and 5 seconds to
> > get the information from the 7 URLs referenced in each NCML file.  If the
> > wait time scales linearly, then it should take 335 (5 x 67) seconds to 
> > generate
> > the list for all of the NCML files you have in the NCEP1 6 hourly pressure 
> > level
> > data server side set of files.
> >
> > Can this be sped up?
> >
> > We don't know.  We'll have to run more tests to determine if the slowness
> > is really in the netCDF Java code or simply a slowness in the ESRL web
> > server's response to the query being made.
> >
> > Sorry we didn't have better news!
> >
> > Cheers,
> >
> > Tom
> > --
> > ****************************************************************************
> > Unidata User Support                                    UCAR Unidata Program
> > (303) 497-8642                                                 P.O. Box 3000
> > address@hidden                                   Boulder, CO 80307
> > ----------------------------------------------------------------------------
> > Unidata HomePage                       http://www.unidata.ucar.edu
> > ****************************************************************************
> >
> >
> > Ticket Details
> > ===================
> > Ticket ID: BDO-300797
> > Department: Support IDV
> > Priority: Normal
> > Status: Closed
> >
> 
> *********************************************
> Brian Mapes, Professor
> Department of Atmospheric Sciences
> Meteorology and Physical Oceanography Program
> RSMAS, University of Miami
> 4600 Rickenbacker Causeway
> Miami, FL 33149-1098
> 
> phone: (305) 421-4275
> fax: (305) 421-4696
> email: address@hidden
> Web: http://www.rsmas.miami.edu/users/bmapes/
> **********************************************
> 
> 
> 
> 
> 


Ticket Details
===================
Ticket ID: BDO-300797
Department: Support IDV
Priority: Normal
Status: Closed


NOTE: All email exchanges with Unidata User Support are recorded in the Unidata inquiry tracking system and then made publicly available through the web. If you do not want to have your interactions made available in this way, you must let us know in each email you send to us.