[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[IDV #BDO-300797]: catalog choking on folder of OK .ncml files



Brian,

Tom, Yuan and I had a long pow wow about this topic today. We brought your NCML
data over to Unidata machines. Your analysis coupled with our experiments ruled
out a number of different possibilities (e.g., web servers limiting traffic,
networking issues).  BTW, we see the same behavior with older versions of the
IDV.

We are now working with a few different theories.

1/ The IDV needs to make smarter requests to RAMADDA in order to not crack open
each individual NCML aggregation in a directory.

2/ Something changed within RAMADDA and it now cracks open more of the dataset
than we would like.

We will keep you posted.

Best,

Unidata IDV Support

> It is unworkable that IDV has to parse every .ncml file in a directory
> before exposing them in Catalog view.
> 
> I have the same problem with CFSR at Albany (see screen shot),
> so the issue is not specific the PSD datasets.
> It is a change in IDV or its libraries, since this used to work.
> 
> 
> 
> I did a lot of work to build these .ncml aggregations.
> It's a shame they are now useless, or downright dangerous to the session
> (the IDV progress indicator bar remains busy, although other tasks can be 
> done).
> 
> What's the solution?
> Will this always be the way Catalog view operates?
> If so I need to refolderize my aggregations, one or two .ncml per folder, I 
> guess.
> 
> [cid:address@hidden
> 
> 
> 
> On Nov 24, 2014, at 10:51 AM, Unidata IDV Support wrote:
> 
> Brian,
> 
> The change may not be related to Unidata.  As far as we could tell, the
> "slowness" is related to the URLs aggregated in the NCML.  Bear in that these
> NCML files are hitting the (NCEP?) web server quite a bit. It could that NCEP
> set up some mechanisms to limit the rate of URL access on their web 
> servers(???)
> (Especially after the cyber attacks that hit the NWS recently.)
> 
> Best,
> 
> Unidata IDV Support
> 
> Let me clarify that it used to work.
> What has changed?
> 
> Last year about this time, for example, my class used these same server side 
> ncml aggregations with no trouble. This is not experimental new 
> functionality, it is a workhorse that fell down. Always disappointing when 
> the March of progress or at least of workable fixes goes backward.
> 
> What has changed?
> 
> Thanks for your help.
> Brian
> 
> Brian Mapes
> This message was typed on a cell phone, please forgive its terseness and any 
> errors.
> 
> On Nov 21, 2014, at 5:38 PM, Unidata IDV Support 
> <address@hidden<mailto:address@hidden>> wrote:
> 
> Hi Brian,
> 
> re: IDV Catalog listing of directory with lots of NCML files that reference 
> data
> through URLs
> 
> Yes I did this and it never came back.
> Help ??
> 
> Julien and I proved that the catalog view in the IDV will _eventually_
> show all of the NCML files.  We did this by:
> 
> - create a new server side view of a single NCML file
> 
> This listing is reasonably fast.
> 
> - create a server side view of 10 NCML files
> 
> This listing is slow, but not slow enough to look like the IDV is
> hung.
> 
> - full set
> 
> We didn't bother to wait for this to complete, but we are convinced
> that it eventually will.
> 
> What is going on?
> 
> Evidently, netCDF Java is actually opening the dataset referenced in each
> URL named in the NCML files.  It seems to take between 4 and 5 seconds to
> get the information from the 7 URLs referenced in each NCML file.  If the
> wait time scales linearly, then it should take 335 (5 x 67) seconds to 
> generate
> the list for all of the NCML files you have in the NCEP1 6 hourly pressure 
> level
> data server side set of files.
> 
> Can this be sped up?
> 
> We don't know.  We'll have to run more tests to determine if the slowness
> is really in the netCDF Java code or simply a slowness in the ESRL web
> server's response to the query being made.
> 
> Sorry we didn't have better news!
> 
> Cheers,
> 
> Tom
> --
> ****************************************************************************
> Unidata User Support                                    UCAR Unidata Program
> (303) 497-8642                                                 P.O. Box 3000
> address@hidden<mailto:address@hidden>                                   
> Boulder, CO 80307
> ----------------------------------------------------------------------------
> Unidata HomePage                       http://www.unidata.ucar.edu
> ****************************************************************************
> 
> 
> Ticket Details
> ===================
> Ticket ID: BDO-300797
> Department: Support IDV
> Priority: Normal
> Status: Closed
> 
> 
> 
> 
> 
> Ticket Details
> ===================
> Ticket ID: BDO-300797
> Department: Support IDV
> Priority: Normal
> Status: Closed
> 
> 
> Brian Mapes
> address@hidden<mailto:address@hidden>
> 
> 
> 
> 
> 


Ticket Details
===================
Ticket ID: BDO-300797
Department: Support IDV
Priority: Normal
Status: Open


NOTE: All email exchanges with Unidata User Support are recorded in the Unidata inquiry tracking system and then made publicly available through the web. If you do not want to have your interactions made available in this way, you must let us know in each email you send to us.