Re: [thredds] Catalog Changes on thredds.ucar.edu

Hi Sean-

On 7/2/13 7:07 PM, Sean Arms wrote:

Is there a reason to use the NAM CONUS 20km grids over the 12km grids?
I do believe these files were being FTP'd to Unidata servers, and were
not being transmitted via IDD, so perhaps something changed on the FTP
server. At any rate, no one mentioned any issue with accessing these
files (again, the other machines with much longer scour times were empty
too, so it's been down for quite some time), so I made the call to
remove them from the catalog. Perhaps it was the wrong call for me to make.

I don't use the 20km NAM, but to answer your more general question of why would I use a lower resolution grid, it would be for speed of download. For example, I use the 1.0 degree GFS for "quick, large scale looks", but if I want a little more detail over a particular area, I use the .5 degree grid. Same with the 40 vs. 13 km RAP - 40 km is good for CONUS, 13 is better for smaller area analysis.


I would prefer to see these changes announced before the fact rather
than after or on a web page that one has to search around for.

That would be the ideal situation! The idea of this page is that it will
be "the place" to look. No need to search for it if you bookmark it (and
if it's prominently linked off of our main page). What do you feel would
be the appropriate venue other than a status page? I'm assuming some
soft of real time notification? RSS, twitter (which can be used like an
rss feed), mailing list (which already exist, like nws-changes), the
blink html tag in the thredds catalogs? These have seriously crossed my
mind. Ok, the blink tag might be a bit much (and it's not being support
much anymore), but in all honestly, what would be most beneficial for
you (or anyone else - feel free to chime in!).

You already have a thredds mailing list and a datastream mailing list. And you already have a community RSS feed. All I'm asking is that you use them. It's a matter of push vs. pull. For news, I like the push model where I find out ahead of time what's going on rather than having to check (pull) every day a web page that may or may not be updated ahead of a change.

For example, the recent change from storing 21 days to 15 days impacted my
work and I only found out about it after when I tried to display the
data.

That change was a bit difficult - we could not give much of a lead time
as our disk space was ran low quick. This is in part due to us including
the new FNMOC NAVGEM 0.5 degree global model output, but level II radar
data volumes are also getting tough to manage (and continue to grow).
Also, as you mentioned in your support message, you understand that
Unidata isn't a data archive center.

However, the UPC has maintained a valuable resource in the "motherlode" TDS and ADDE servers which many in the community have come to rely on.

> You also mention NOMADS, which is a
model data archive (oversimplified statement). I do believe they are
testing out the use of TDS 4.3, which will ease the issues you have with
the way they serve out the individual forecast grids. Have you tried to
contact them to see what their plans are?

I have not, but in the past, the way they store their data is not conducive to the types of queries I can make to thredds.ucar.edu. Datasets are not aggregated very well for multi-day access.


...shameless plug...
Not necessairly news for you, but for the benefit of others...This is
also a good time to note that while the thredds servers at Unidata
(particularly the one housed on the motherlode machine) has always been
a "demonstration" installation (even though it has been "good enough"
for people to heavily rely upon). We encourage people in the community
to apply for Unidata equipment awards to setup their own
"motherlode"-like servers:

http://www.unidata.ucar.edu/community/equipaward/

I'm all for a community network of servers, but the reality is that most Unidata sites do not have the resources to maintain the type of datasets that motherlode provides. And community servers tend to come and go, whereas motherlode has been a pretty consistent source of near realtime data.

Also, if there are real-time datasets people would like to see being
served by Unidata, the best way to make the request is through our Users
Committee:

http://www.unidata.ucar.edu/committees/usercom/

We've relied on them in the past (and will continue to do so) to get a
sense of what model output are important to the community, what model
output could be cut with minimal impact, and what would be beneficial to
add, say, to the CONDUIT feed (it should be noted that NCEP has been
awesome about listening to our community!).

Were they consulted on the recent changes?

...end shameless plug...

What is the status of the longer term LEAD archive?  Will it be
upgraded to TDS 4.3 in the near future so I can switch to that for my
longer "real-time" archive needs?

We are discussing how to best proceed with our other servers. Do we
simply make motherlode clones and distribute the load across them? How
do we best allow clients, like the IDV, to have one access url that can
point to other resources behind the scenes (as to prevent bundle
breaking if a server goes down). It does not make sense to blast forward
at this point, but. we are talking about these things internally

That's good to know. In terms of LEAD, it is not a clone, but a longer term storage of some of the datasets. The big problem for me is that I had to redo all my access to account for the new names of the GRIB variables when thredds.ucar.edu went to 4.3 and if I wanted to go to the lead machine which is still running 4.2, I'd have to redo them again to go back to the old names. And when the lead machine goes to 4.3, I'd have to do that again.

Unidata provides a valuable service with the thredds.ucar.edu dataset to
the IDV and other communities, so even minor changes have impacts that
would be nice to know of before they happen.

Again, some changes we can announce with some lead-time, but these are
mostly limited to NWS or CONDUIT feed changes (for which email lists
already exist).

NWS and CONDUIT change messages do not easily translate changes in the user will experience for the data access. It would be nice if the UPC would provide the translation and disseminate that information. An example of where this worked well was the move to the RAP.

Don
--
Don Murray
NOAA/ESRL/PSD and CIRES
303-497-3596
http://www.esrl.noaa.gov/psd/people/don.murray/



  • 2013 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the thredds archives: