[netCDFDecoders #BRC-314440]: Motherlode Data Reterival
- To: address@hidden
- Subject: [netCDFDecoders #BRC-314440]: Motherlode Data Reterival
- From: "Unidata THREDDS Support" <address@hidden>
- Date: Thu, 21 Jun 2012 12:50:08 -0600
What is the IfCanOpen Java function you mention? Are you (is IfCanOpen)
checking in the THREDDS XML catalog for the NCEP/WW3/Global data:
It lists all available NCEP/WW3/Global datasets. It also has a "Latest" URL
which returns a THREDDS XML catalog which lists only the latest (and complete)
[NOTE: You can look at both URLs above as HTML by changing the ".xml" suffix to
That latest.xml catalog is probably the best way to go about this at the
moment. You could ping that every half hour or something until it shows that
the latest dataset is the one you expect. Unfortunately, we don't have
"LastModified" HTTP headers and the like working on that catalog. If that were
in place, you could use a conditional HTTP GET request to determine when the
"Latest" had changed.
If you aren't already, you should make sure you are on the address@hidden email
list. We are working on the next version of the THREDDS Data Server (version
4.3) which will introduce some changes to the URLs above and to some of the
variable names in these datasets. Once the software is released, we'll have
some time with that version available on our test server on motherlode. After
some time for users to test things out, we'll upgrade our main motherlode
server to that version. All of this activity will be communicated on the
address@hidden email list.
Hope that helps,
Liam Eastop wrote:
> Hi Ethan,
> Thanks so much for getting back to us and apologies again for the 95% failure
> I'll write down our process of how we have been obtaining the data and
> perhaps you
> might be able to suggest a preferred method.
> The file which we are wanting to grab is the:
> Current Method:
> JAVA Web App
> 1. Recreate a URL based on the current GMT date, plus forecastTime
> (e.g: /WW3_Global_currentDate_forecastTime.grib2)
> for example: /WW3_Global_20120619_1800.grib2
> 2. We then check to see if the file exists on the catalogue using the
> (IfCanOpen Java
> function). If True we download the file, Else we check again (previously
> before last
> Friday we were checking every hour, however on friday we changed stepping
> time to
> every 30min).
> Second Error:
> We also had a second error, which occurred after we reset the server on
> This initiated two checkTimers. This meant we were checking for a new file 4
> times an
> hour and possibly 20 times per six hour block. Which would have occurred in
> the 95%
> failure rate.
> Our Reason for manipulating our timers:
> We changed our checkTimer because the catalogue doesn't seem to update at any
> time based off the 'last modified' column
> although the files are strictly 6hour forecasts.
> Could you make any recommendations of a better technique or how we could
> obtain the
> grib files closer to there creation time?
> Currently we have stopped the server data retrieval until we know the best
> Thanks again Ethan, great emails and a big help.
Ticket ID: BRC-314440
Department: Support THREDDS
NOTE: All email exchanges with Unidata User Support are recorded in the
Unidata inquiry tracking system and then made publicly available
through the web. If you do not want to have your interactions made
available in this way, you must let us know in each email you send to us.