[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[IDV #TPP-254601]: More details on "Sunday" problem with OPeNDAP and/or THREDDS



Very nice material Murray! I've contacted the HYCOM folks, so hopefully we can 
help them out a bit. I'm also looking into a good way for the IDV to let users 
know that it's waiting on a server to respond (while allowing the user to do 
other things), and possibly allow for the user to kill the request in a 
graceful way.

Cheers!

Sean

> Sorry, that link got chopped.  It's
> http://marinedataliteracy.org/ops/circ_mod_direct.htm
> 
> 
> 
> ----- Original Message -----
> From: "Unidata IDV Support" <address@hidden>
> To: <address@hidden>
> Cc: <address@hidden>
> Sent: Thursday, May 24, 2012 11:36 AM
> Subject: [IDV #TPP-254601]: More details on "Sunday" problem with OPeNDAP
> and/or THREDDS
> 
> 
> > Hi Murray,
> >
> > It sounds like the IDV is waiting for the server to respond. One way to
> > tell, besides the blinking light on your ethernet port, is to check the
> > bottom portion of the Dashboard in the IDV. I'm trying to load the files
> > using the URL you gave me and the IDV says "Loading in data source: Grids
> > from an OPeNDAP Server", and the little status bar in the lower right
> > corner of the dashboard is moving around. I noticed that the IDV status
> > does not really update after requesting the spatial subset (or temporal)
> > using the upper-left control, but again, the IDV becomes unresponsive
> > because it it waiting to hear back from the server.
> >
> > I think the slowness on the server is due to the fact that they are
> > serving out a collection of *very* large files, and perhaps the collection
> > is being rebuilt each time a client tries to access the dataset (maybe
> > caching is turned off on their server? Not sure). While the collection is
> > being built, the IDV has to sit and wait for the server to send it back
> > the information on what is contained in the collection (same thing after
> > you subset in the upper-left).
> >
> > Unfortunately, I'm not sure there is much we can do on the IDV side in
> > this case, as I'm pretty sure it's a server performance issue. I'll check
> > with our THREDDS team to see if the folks at HYCOM have contacted us
> > regarding the performance of their server.
> >
> > Cheers,
> >
> > Sean
> >
> >> Hi again Sean,
> >>
> >> Subsetting has become the big issue (for me) during the current HYCOM
> >> server
> >> situation.  I understand that the server might not be fixed yet, but
> >> wanted
> >> to let you know how it's affecting IDV.
> >>
> >> I can use the OPeNDAP URL
> >> (http://tds.hycom.org/thredds/dodsC/GLBa0.08/expt_90.9) and it does
> >> "load"
> >> the 2D/3D grids, but at that point if I use the upper-left control to
> >> subset
> >> anything, then IDV seems to go dead on me.  Nothing I click "works" and
> >> nothing I click shows bold to indicate it was selected.  But after a turn
> >> in
> >> the garden this morning I found out that in fact IDV is working, and
> >> usually
> >> comes through (after 20+ minutes) with the expected next step (usually
> >> selection of a display).  The only sign something is going on is the
> >> blinking ethernet cable in the interim.  It seems to me that IDV might
> >> actually be downloading the data at this step, instead of building a
> >> local
> >> structure map.
> >>
> >> On the other hand, I can make temporal and stride changes in the lower
> >> right.  But because I don't want to I haven't even tried to use the
> >> inexact
> >> rubber band areal selection down there.
> >>
> >> I have checked my IDV with the GFS data in the motherload, and it works
> >> fine
> >> and fast.
> >>
> >> Murray
> >>
> >> ----- Original Message -----
> >> From: "Unidata IDV Support" <address@hidden>
> >> To: <address@hidden>
> >> Cc: <address@hidden>
> >> Sent: Monday, May 21, 2012 1:59 PM
> >> Subject: [IDV #TPP-254601]: More details on "Sunday" problem with OPeNDAP
> >> and/or THREDDS
> >>
> >>
> >> > Greetings Murray,
> >> >
> >> >> Good Morning Sean,
> >> >>
> >> >> Wow.  So early.  Yeps, I have msgs from the HYCOM folks about their
> >> >> troubles, and actually thought I was sending a note to them.  My
> >> >> dementia
> >> >> kicked in and I sent it to you folks instead.
> >> >
> >> > No problem :-)
> >> >
> >> >>
> >> >> I've seen their travails from my end, with the appearance and
> >> >> disappearance
> >> >> of my target data (over the hours), and even when available the
> >> >> spatial
> >> >> and
> >> >> temporal subsetting doesn't work, i.e. the guts of OPeNDAP.  That's
> >> >> show
> >> >> biz.  I wish somebody would buy them a better server.
> >> >>
> >> >> Anyway, while a friendly soul is listening, I want to ask you about
> >> >> the
> >> >> difference between subsetting a la the PROPERTIES of the data object
> >> >> (in
> >> >> the
> >> >> upper left quarter of IDV's dashboard) and the similar (but not so
> >> >> comprehensive) functions in the lower right corner (available just
> >> >> before
> >> >> CREATE DISPLAY).
> >> >
> >> > The Properties of the data object apply to the entire set of variables
> >> > within the data source, while the tools in the lower right corner apply
> >> > the field you've selected.
> >> >
> >> >> They are almost, but not quite identical, for example the
> >> >> spatial subsetting is manual only, without coordinate specification.
> >> >
> >> > The properties "Spatial Subset" tab is a combination of the "Region"
> >> > and
> >> > "Stride" tabs in the lower right part of the dashboard, except like I
> >> > said
> >> > before, one is dataset wide, the other is for a single variable.
> >> >
> >> >> Also,  I've seen cases where initial specification of TIMES in the
> >> >> upper
> >> >> left
> >> >> corner was not honored later in the lower right corner.  Which one
> >> >> over-rides the other, or is it merely a matter of which one was set
> >> >> last?
> >> >> And last, what is the full meaning of USE DEFAULT?
> >> >>
> >> >
> >> > From what I gather, "use default" will default to what is set under the
> >> > Properties for the data source (which is, by default, no spatial or
> >> > temporal subsetting). If you set a temporal and/or spatial subset via
> >> > the
> >> > Properties interface for a data source and have "use default" selected
> >> > in
> >> > the times, region, and stride tabs in the lower right, then the
> >> > temporal
> >> > and/or spatial subset you've set for the data source should be honored
> >> > (if
> >> > not, please let us know!).
> >> >
> >> > Cheers,
> >> >
> >> > Sean
> >> >
> >> >> Murray
> >> >>
> >> >> ----- Original Message -----
> >> >> From: "Unidata IDV Support" <address@hidden>
> >> >> To: <address@hidden>
> >> >> Cc: <address@hidden>
> >> >> Sent: Monday, May 21, 2012 8:13 AM
> >> >> Subject: [IDV #TPP-254601]: More details on "Sunday" problem with
> >> >> OPeNDAP
> >> >> and/or THREDDS
> >> >>
> >> >>
> >> >> > Greetings Murray,
> >> >> >
> >> >> > It looks like the HYCOM folks are having some issues with their
> >> >> > THREDDS
> >> >> > server:
> >> >> >
> >> >> > http://groups.google.com/a/hycom.org/group/notice/browse_thread/thread/2351e493822b42fd#
> >> >> >
> >> >> > THe particular dataset you pointed to is not available because they
> >> >> > are
> >> >> > trying to serve out a collection of datasets, and the THREDDS server
> >> >> > does
> >> >> > not 'see' any datasets to collect together. We haven't received any
> >> >> > support tickets from them, so it may be that they are doing some
> >> >> > changes
> >> >> > to how they serve out collections of data - you may want to drop
> >> >> > them a
> >> >> > line to see if they have any more information:
> >> >> > address@hidden
> >> >> >
> >> >> > Cheers!
> >> >> >
> >> >> > Sean
> >> >> >
> >> >> >> Full Name: Murray Brown
> >> >> >> Email Address: address@hidden
> >> >> >> Organization: UNESCO/IOC/IODE Marine Data Training Program
> >> >> >> Package Version: 3.0u2 build date:2012-02-10 17:21 UTC
> >> >> >> Operating System: Windows XP
> >> >> >> Hardware: Java: home: C:\Program Files\IDV_3.0u2\jre version:
> >> >> >> 1.6.0_20
> >> >> >> j3d:1.5.2 fcs (build4)
> >> >> >> Description of problem:
> >> >> >>
> >> >> >> There was an error loading the data:
> >> >> >>
> >> >> >> Error creating data source:OPENDAP.GRID with:
> >> >> >>
> >> >> >> http://tds.hycom.org/thredds/dodsC/GLBa0.08/expt_90.9
> >> >> >>
> >> >> >> Grid data source failed making data set:
> >> >> >> dods://tds.hycom.org/thredds/dodsC/GLBa0.08/expt_90.9
> >> >> >>
> >> >> >> Method failed:HTTP/1.1 500 Internal Server Error on URL=
> >> >> >> http://tds.hycom.org/thredds/dodsC/GLBa0.08/expt_90.9.dds
> >> >> >>
> >> >> >> I can actually load some older "legacy" results, but nothing
> >> >> >> current
> >> >> >> that is global.
> >> >> >>
> >> >> >> Murray
> >> >> >>
> >> >> >> ******************
> >> >> >> Stack trace:
> >> >> >> opendap.dap.DAP2Exception: Method failed:HTTP/1.1 500 Internal
> >> >> >> Server
> >> >> >> Error on URL=
> >> >> >> http://tds.hycom.org/thredds/dodsC/GLBa0.08/expt_90.9.dds
> >> >> >> at opendap.dap.DConnect2.openConnection(DConnect2.java:280)
> >> >> >> at opendap.dap.DConnect2.getDDS(DConnect2.java:517)
> >> >> >> at opendap.dap.DConnect2.getDDS(DConnect2.java:491)
> >> >> >> at ucar.nc2.dods.DODSNetcdfFile.<init>(DODSNetcdfFile.java:212)
> >> >> >> at
> >> >> >> ucar.nc2.dataset.NetcdfDataset.acquireDODS(NetcdfDataset.java:809)
> >> >> >> at
> >> >> >> ucar.nc2.dataset.NetcdfDataset.openOrAcquireFile(NetcdfDataset.java:654)
> >> >> >> at
> >> >> >> ucar.nc2.dataset.NetcdfDataset.openDataset(NetcdfDataset.java:424)
> >> >> >> at
> >> >> >> ucar.nc2.dataset.NetcdfDataset.acquireDataset(NetcdfDataset.java:520)
> >> >> >> at ucar.nc2.dt.grid.GridDataset.open(GridDataset.java:101)
> >> >> >> at ucar.nc2.dt.grid.GridDataset.open(GridDataset.java:87)
> >> >> >> at
> >> >> >> ucar.unidata.data.grid.GeoGridDataSource.doMakeDataSet(GeoGridDataSource.java:1094)
> >> >> >> at
> >> >> >> ucar.unidata.data.grid.GeoGridDataSource.getDataset(GeoGridDataSource.java:1116)
> >> >> >> at
> >> >> >> ucar.unidata.data.grid.GeoGridDataSource.initAfterCreation(GeoGridDataSource.java:478)
> >> >> >> at
> >> >> >> ucar.unidata.data.DataManager.createDataSource(DataManager.java:1577)
> >> >> >> at
> >> >> >> ucar.unidata.idv.IntegratedDataViewer.createDataSource(IntegratedDataViewer.java:1980)
> >> >> >> at
> >> >> >> ucar.unidata.idv.IntegratedDataViewer.makeDataSource(IntegratedDataViewer.java:1897)
> >> >> >> at
> >> >> >> ucar.unidata.idv.IntegratedDataViewer.makeDataSource(IntegratedDataViewer.java:1831)
> >> >> >> at
> >> >> >> ucar.unidata.idv.IntegratedDataViewer.makeDataSource(IntegratedDataViewer.java:1804)
> >> >> >> at
> >> >> >> ucar.unidata.idv.chooser.IdvChooser.makeDataSource(IdvChooser.java:943)
> >> >> >> at
> >> >> >> ucar.unidata.idv.chooser.XmlChooser.makeDataSource(XmlChooser.java:280)
> >> >> >> at
> >> >> >> ucar.unidata.idv.chooser.ThreddsHandler.processNodes(ThreddsHandler.java:851)
> >> >> >> at
> >> >> >> ucar.unidata.idv.chooser.ThreddsHandler.doLoad(ThreddsHandler.java:700)
> >> >> >> at
> >> >> >> ucar.unidata.idv.chooser.XmlChooser.doLoadInThread(XmlChooser.java:717)
> >> >> >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >> >> >> at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
> >> >> >> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
> >> >> >> at java.lang.reflect.Method.invoke(Unknown Source)
> >> >> >> at ucar.unidata.util.Misc$2.run(Misc.java:1062)
> >> >> >> at ucar.unidata.util.Misc$3.run(Misc.java:1090)
> >> >> >>
> >> >> >>
> >> >> >>
> >> >> >
> >> >> >
> >> >> > Ticket Details
> >> >> > ===================
> >> >> > Ticket ID: TPP-254601
> >> >> > Department: Support IDV
> >> >> > Priority: Normal
> >> >> > Status: Closed
> >> >> >
> >> >>
> >> >>
> >> >
> >> >
> >> > Ticket Details
> >> > ===================
> >> > Ticket ID: TPP-254601
> >> > Department: Support IDV
> >> > Priority: Normal
> >> > Status: Open
> >> >
> >> >
> >>
> >>
> >
> >
> > Ticket Details
> > ===================
> > Ticket ID: TPP-254601
> > Department: Support IDV
> > Priority: Normal
> > Status: Open
> >
> 
> 


Ticket Details
===================
Ticket ID: TPP-254601
Department: Support IDV
Priority: High
Status: Closed