[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[IDV #TPP-254601]: More details on "Sunday" problem with OPeNDAP and/or THREDDS

Hi Murray,

It sounds like the IDV is waiting for the server to respond. One way to tell, 
besides the blinking light on your ethernet port, is to check the bottom 
portion of the Dashboard in the IDV. I'm trying to load the files using the URL 
you gave me and the IDV says "Loading in data source: Grids from an OPeNDAP 
Server", and the little status bar in the lower right corner of the dashboard 
is moving around. I noticed that the IDV status does not really update after 
requesting the spatial subset (or temporal) using the upper-left control, but 
again, the IDV becomes unresponsive because it it waiting to hear back from the 
server.

I think the slowness on the server is due to the fact that they are serving out 
a collection of *very* large files, and perhaps the collection is being rebuilt 
each time a client tries to access the dataset (maybe caching is turned off on 
their server? Not sure). While the collection is being built, the IDV has to 
sit and wait for the server to send it back the information on what is 
contained in the collection (same thing after you subset in the upper-left).

Unfortunately, I'm not sure there is much we can do on the IDV side in this 
case, as I'm pretty sure it's a server performance issue. I'll check with our 
THREDDS team to see if the folks at HYCOM have contacted us regarding the 
performance of their server.

Cheers,

Sean

> Hi again Sean,
> 
> Subsetting has become the big issue (for me) during the current HYCOM server
> situation.  I understand that the server might not be fixed yet, but wanted
> to let you know how it's affecting IDV.
> 
> I can use the OPeNDAP URL
> (http://tds.hycom.org/thredds/dodsC/GLBa0.08/expt_90.9) and it does "load"
> the 2D/3D grids, but at that point if I use the upper-left control to subset
> anything, then IDV seems to go dead on me.  Nothing I click "works" and
> nothing I click shows bold to indicate it was selected.  But after a turn in
> the garden this morning I found out that in fact IDV is working, and usually
> comes through (after 20+ minutes) with the expected next step (usually
> selection of a display).  The only sign something is going on is the
> blinking ethernet cable in the interim.  It seems to me that IDV might
> actually be downloading the data at this step, instead of building a local
> structure map.
> 
> On the other hand, I can make temporal and stride changes in the lower
> right.  But because I don't want to I haven't even tried to use the inexact
> rubber band areal selection down there.
> 
> I have checked my IDV with the GFS data in the motherload, and it works fine
> and fast.
> 
> Murray
> 
> ----- Original Message -----
> From: "Unidata IDV Support" <address@hidden>
> To: <address@hidden>
> Cc: <address@hidden>
> Sent: Monday, May 21, 2012 1:59 PM
> Subject: [IDV #TPP-254601]: More details on "Sunday" problem with OPeNDAP
> and/or THREDDS
> 
> 
> > Greetings Murray,
> >
> >> Good Morning Sean,
> >>
> >> Wow.  So early.  Yeps, I have msgs from the HYCOM folks about their
> >> troubles, and actually thought I was sending a note to them.  My dementia
> >> kicked in and I sent it to you folks instead.
> >
> > No problem :-)
> >
> >>
> >> I've seen their travails from my end, with the appearance and
> >> disappearance
> >> of my target data (over the hours), and even when available the spatial
> >> and
> >> temporal subsetting doesn't work, i.e. the guts of OPeNDAP.  That's show
> >> biz.  I wish somebody would buy them a better server.
> >>
> >> Anyway, while a friendly soul is listening, I want to ask you about the
> >> difference between subsetting a la the PROPERTIES of the data object (in
> >> the
> >> upper left quarter of IDV's dashboard) and the similar (but not so
> >> comprehensive) functions in the lower right corner (available just before
> >> CREATE DISPLAY).
> >
> > The Properties of the data object apply to the entire set of variables
> > within the data source, while the tools in the lower right corner apply
> > the field you've selected.
> >
> >> They are almost, but not quite identical, for example the
> >> spatial subsetting is manual only, without coordinate specification.
> >
> > The properties "Spatial Subset" tab is a combination of the "Region" and
> > "Stride" tabs in the lower right part of the dashboard, except like I said
> > before, one is dataset wide, the other is for a single variable.
> >
> >> Also,  I've seen cases where initial specification of TIMES in the upper
> >> left
> >> corner was not honored later in the lower right corner.  Which one
> >> over-rides the other, or is it merely a matter of which one was set last?
> >> And last, what is the full meaning of USE DEFAULT?
> >>
> >
> > From what I gather, "use default" will default to what is set under the
> > Properties for the data source (which is, by default, no spatial or
> > temporal subsetting). If you set a temporal and/or spatial subset via the
> > Properties interface for a data source and have "use default" selected in
> > the times, region, and stride tabs in the lower right, then the temporal
> > and/or spatial subset you've set for the data source should be honored (if
> > not, please let us know!).
> >
> > Cheers,
> >
> > Sean
> >
> >> Murray
> >>
> >> ----- Original Message -----
> >> From: "Unidata IDV Support" <address@hidden>
> >> To: <address@hidden>
> >> Cc: <address@hidden>
> >> Sent: Monday, May 21, 2012 8:13 AM
> >> Subject: [IDV #TPP-254601]: More details on "Sunday" problem with OPeNDAP
> >> and/or THREDDS
> >>
> >>
> >> > Greetings Murray,
> >> >
> >> > It looks like the HYCOM folks are having some issues with their THREDDS
> >> > server:
> >> >
> >> > http://groups.google.com/a/hycom.org/group/notice/browse_thread/thread/2351e493822b42fd#
> >> >
> >> > THe particular dataset you pointed to is not available because they are
> >> > trying to serve out a collection of datasets, and the THREDDS server
> >> > does
> >> > not 'see' any datasets to collect together. We haven't received any
> >> > support tickets from them, so it may be that they are doing some
> >> > changes
> >> > to how they serve out collections of data - you may want to drop them a
> >> > line to see if they have any more information: address@hidden
> >> >
> >> > Cheers!
> >> >
> >> > Sean
> >> >
> >> >> Full Name: Murray Brown
> >> >> Email Address: address@hidden
> >> >> Organization: UNESCO/IOC/IODE Marine Data Training Program
> >> >> Package Version: 3.0u2 build date:2012-02-10 17:21 UTC
> >> >> Operating System: Windows XP
> >> >> Hardware: Java: home: C:\Program Files\IDV_3.0u2\jre version: 1.6.0_20
> >> >> j3d:1.5.2 fcs (build4)
> >> >> Description of problem:
> >> >>
> >> >> There was an error loading the data:
> >> >>
> >> >> Error creating data source:OPENDAP.GRID with:
> >> >>
> >> >> http://tds.hycom.org/thredds/dodsC/GLBa0.08/expt_90.9
> >> >>
> >> >> Grid data source failed making data set:
> >> >> dods://tds.hycom.org/thredds/dodsC/GLBa0.08/expt_90.9
> >> >>
> >> >> Method failed:HTTP/1.1 500 Internal Server Error on URL=
> >> >> http://tds.hycom.org/thredds/dodsC/GLBa0.08/expt_90.9.dds
> >> >>
> >> >> I can actually load some older "legacy" results, but nothing current
> >> >> that is global.
> >> >>
> >> >> Murray
> >> >>
> >> >> ******************
> >> >> Stack trace:
> >> >> opendap.dap.DAP2Exception: Method failed:HTTP/1.1 500 Internal Server
> >> >> Error on URL=
> >> >> http://tds.hycom.org/thredds/dodsC/GLBa0.08/expt_90.9.dds
> >> >> at opendap.dap.DConnect2.openConnection(DConnect2.java:280)
> >> >> at opendap.dap.DConnect2.getDDS(DConnect2.java:517)
> >> >> at opendap.dap.DConnect2.getDDS(DConnect2.java:491)
> >> >> at ucar.nc2.dods.DODSNetcdfFile.<init>(DODSNetcdfFile.java:212)
> >> >> at ucar.nc2.dataset.NetcdfDataset.acquireDODS(NetcdfDataset.java:809)
> >> >> at
> >> >> ucar.nc2.dataset.NetcdfDataset.openOrAcquireFile(NetcdfDataset.java:654)
> >> >> at ucar.nc2.dataset.NetcdfDataset.openDataset(NetcdfDataset.java:424)
> >> >> at
> >> >> ucar.nc2.dataset.NetcdfDataset.acquireDataset(NetcdfDataset.java:520)
> >> >> at ucar.nc2.dt.grid.GridDataset.open(GridDataset.java:101)
> >> >> at ucar.nc2.dt.grid.GridDataset.open(GridDataset.java:87)
> >> >> at
> >> >> ucar.unidata.data.grid.GeoGridDataSource.doMakeDataSet(GeoGridDataSource.java:1094)
> >> >> at
> >> >> ucar.unidata.data.grid.GeoGridDataSource.getDataset(GeoGridDataSource.java:1116)
> >> >> at
> >> >> ucar.unidata.data.grid.GeoGridDataSource.initAfterCreation(GeoGridDataSource.java:478)
> >> >> at
> >> >> ucar.unidata.data.DataManager.createDataSource(DataManager.java:1577)
> >> >> at
> >> >> ucar.unidata.idv.IntegratedDataViewer.createDataSource(IntegratedDataViewer.java:1980)
> >> >> at
> >> >> ucar.unidata.idv.IntegratedDataViewer.makeDataSource(IntegratedDataViewer.java:1897)
> >> >> at
> >> >> ucar.unidata.idv.IntegratedDataViewer.makeDataSource(IntegratedDataViewer.java:1831)
> >> >> at
> >> >> ucar.unidata.idv.IntegratedDataViewer.makeDataSource(IntegratedDataViewer.java:1804)
> >> >> at
> >> >> ucar.unidata.idv.chooser.IdvChooser.makeDataSource(IdvChooser.java:943)
> >> >> at
> >> >> ucar.unidata.idv.chooser.XmlChooser.makeDataSource(XmlChooser.java:280)
> >> >> at
> >> >> ucar.unidata.idv.chooser.ThreddsHandler.processNodes(ThreddsHandler.java:851)
> >> >> at
> >> >> ucar.unidata.idv.chooser.ThreddsHandler.doLoad(ThreddsHandler.java:700)
> >> >> at
> >> >> ucar.unidata.idv.chooser.XmlChooser.doLoadInThread(XmlChooser.java:717)
> >> >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >> >> at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
> >> >> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
> >> >> at java.lang.reflect.Method.invoke(Unknown Source)
> >> >> at ucar.unidata.util.Misc$2.run(Misc.java:1062)
> >> >> at ucar.unidata.util.Misc$3.run(Misc.java:1090)
> >> >>
> >> >>
> >> >>
> >> >
> >> >
> >> > Ticket Details
> >> > ===================
> >> > Ticket ID: TPP-254601
> >> > Department: Support IDV
> >> > Priority: Normal
> >> > Status: Closed
> >> >
> >>
> >>
> >
> >
> > Ticket Details
> > ===================
> > Ticket ID: TPP-254601
> > Department: Support IDV
> > Priority: Normal
> > Status: Open
> >
> >
> 
> 


Ticket Details
===================
Ticket ID: TPP-254601
Department: Support IDV
Priority: Normal
Status: Open