Hi Don,
On 7/3/13 9:04 AM, Don Murray wrote:
> Hi Sean-
>
> On 7/2/13 7:07 PM, Sean Arms wrote:
>
>> Is there a reason to use the NAM CONUS 20km grids over the 12km grids?
>> I do believe these files were being FTP'd to Unidata servers, and were
>> not being transmitted via IDD, so perhaps something changed on the FTP
>> server. At any rate, no one mentioned any issue with accessing these
>> files (again, the other machines with much longer scour times were empty
>> too, so it's been down for quite some time), so I made the call to
>> remove them from the catalog. Perhaps it was the wrong call for me to
>> make.
>
> I don't use the 20km NAM, but to answer your more general question of
> why would I use a lower resolution grid, it would be for speed of
> download. For example, I use the 1.0 degree GFS for "quick, large
> scale looks", but if I want a little more detail over a particular
> area, I use the .5 degree grid. Same with the 40 vs. 13 km RAP - 40
> km is good for CONUS, 13 is better for smaller area analysis.
I would ask why not spatially subset or stride the higher res grids?
Often, but not always (for sure!), the higher res models have slightly
different, better performing parametrization schemes running behind the
scenes, and it can be beneficial to use those grids over the lower res
models.
>
>>>
>>> I would prefer to see these changes announced before the fact rather
>>> than after or on a web page that one has to search around for.
>>
>> That would be the ideal situation! The idea of this page is that it will
>> be "the place" to look. No need to search for it if you bookmark it (and
>> if it's prominently linked off of our main page). What do you feel would
>> be the appropriate venue other than a status page? I'm assuming some
>> soft of real time notification? RSS, twitter (which can be used like an
>> rss feed), mailing list (which already exist, like nws-changes), the
>> blink html tag in the thredds catalogs? These have seriously crossed my
>> mind. Ok, the blink tag might be a bit much (and it's not being support
>> much anymore), but in all honestly, what would be most beneficial for
>> you (or anyone else - feel free to chime in!).
>
> You already have a thredds mailing list
The THREDDS mailing list exists for the users of the THREDDS software
package, not Unidata maintained instances of THREDDS. That said, we did
advertise the availability of our thredds.ucar.edu catalog changes page.
> and a datastream mailing list.
Do you mean the datastream support address, or the needdata mailing
list? Neither one are designed for or really work for getting this type
of information out.
> And you already have a community RSS feed.
The News@Unidata feed could be a possible point of dissemination. We did
discuss keeping track of things on that blog, but then figured it would
be just as much of a pain to tell users to search our blog site for a
given hashtag (blog diving for information is not the most fun
experience). I think that if a push method is what is desired, then we
need to have a separate feed to handle these changes.
> All I'm asking is that you use them. It's a matter of push vs. pull.
> For news, I like the push model where I find out ahead of time what's
> going on rather than having to check (pull) every day a web page that
> may or may not be updated ahead of a change.
>
>>> For example, the recent change from storing 21 days to 15 days
>>> impacted my
>>> work and I only found out about it after when I tried to display the
>>> data.
>>
>> That change was a bit difficult - we could not give much of a lead time
>> as our disk space was ran low quick. This is in part due to us including
>> the new FNMOC NAVGEM 0.5 degree global model output, but level II radar
>> data volumes are also getting tough to manage (and continue to grow).
>> Also, as you mentioned in your support message, you understand that
>> Unidata isn't a data archive center.
>
> However, the UPC has maintained a valuable resource in the
> "motherlode" TDS and ADDE servers which many in the community have
> come to rely on.
>
Yep, and we realize that.
> > You also mention NOMADS, which is a
>> model data archive (oversimplified statement). I do believe they are
>> testing out the use of TDS 4.3, which will ease the issues you have with
>> the way they serve out the individual forecast grids. Have you tried to
>> contact them to see what their plans are?
>
> I have not, but in the past, the way they store their data is not
> conducive to the types of queries I can make to thredds.ucar.edu.
> Datasets are not aggregated very well for multi-day access.
>
That's one of the major points of the new GRIB FeatureCollection in
THREDDS 4.3. Just direct the feature collection at a pile of GRIB files,
and we will try to do the rest, including exposing the Best and Latest
aggregations. The folks I know from NCDC are responsive, and awesome
people in general, and I'm sure they would love the feedback from users
of their system.
>>
>> ...shameless plug...
>> Not necessairly news for you, but for the benefit of others...This is
>> also a good time to note that while the thredds servers at Unidata
>> (particularly the one housed on the motherlode machine) has always been
>> a "demonstration" installation (even though it has been "good enough"
>> for people to heavily rely upon). We encourage people in the community
>> to apply for Unidata equipment awards to setup their own
>> "motherlode"-like servers:
>>
>> http://www.unidata.ucar.edu/community/equipaward/
>
> I'm all for a community network of servers, but the reality is that
> most Unidata sites do not have the resources to maintain the type of
> datasets that motherlode provides. And community servers tend to
> come and go, whereas motherlode has been a pretty consistent source of
> near realtime data.
That does not mean we should not help others in the community setup and
maintain their servers. We provide training for this, we will go on site
visits to help in person, and we can help out remotely in getting things
up and running.
>
>
>> Also, if there are real-time datasets people would like to see being
>> served by Unidata, the best way to make the request is through our Users
>> Committee:
>>
>> http://www.unidata.ucar.edu/committees/usercom/
>>
>> We've relied on them in the past (and will continue to do so) to get a
>> sense of what model output are important to the community, what model
>> output could be cut with minimal impact, and what would be beneficial to
>> add, say, to the CONDUIT feed (it should be noted that NCEP has been
>> awesome about listening to our community!).
>
> Were they consulted on the recent changes?
>
Which changes in particular? The NAM 20km surface and selectsurface
grids? No, they wern't, because the data simply stopped showing up. The
way I saw It was either remove the entries from the catalog (and try to
figure out if our community needs these grids or not, and if so, try to
find an alternative source), of leave an empty catalog online.
The changes that are listed on the webpage so far have pretty much been
out of our hands, or had to be done in a timely manner (like the disk
space issue), so we had to make the call. But for other things, like
adding the NAVGEM 0.5 degree model, as well as a slew of other model
output (like the HRRR), yes, we have and will continue to consult our
committees. In fact, Jeff Weber is currently outside of my office
talking to Mohan, giving him an update on community input regarding the
addition of new datasets to the IDD :-)
Having been on both sides (on the Users Committee and now employed at
Unidata), I can without a doubt say that our committees play a critical
role in how things at Unidata are done (shameless plug part 2 -
nominations for our committee are open, through August 16th, including
the Users Committee student representative position:
https://www.unidata.ucar.edu/blogs/news/entry/unidata_committee_nominations_open2)
>> ...end shameless plug...
>>
>>> What is the status of the longer term LEAD archive? Will it be
>>> upgraded to TDS 4.3 in the near future so I can switch to that for my
>>> longer "real-time" archive needs?
>>
>> We are discussing how to best proceed with our other servers. Do we
>> simply make motherlode clones and distribute the load across them? How
>> do we best allow clients, like the IDV, to have one access url that can
>> point to other resources behind the scenes (as to prevent bundle
>> breaking if a server goes down). It does not make sense to blast forward
>> at this point, but. we are talking about these things internally
>
> That's good to know. In terms of LEAD, it is not a clone, but a
> longer term storage of some of the datasets.
Yes, but if we implement a load balancing system, lead can play a role
in that.
> The big problem for me is that I had to redo all my access to
> account for the new names of the GRIB variables when thredds.ucar.edu
> went to 4.3 and if I wanted to go to the lead machine which is still
> running 4.2, I'd have to redo them again to go back to the old names.
> And when the lead machine goes to 4.3, I'd have to do that again.
Understandable. We hope to get lead running 4.3 after our summer
training workshops are over.
>
>
>>> Unidata provides a valuable service with the thredds.ucar.edu
>>> dataset to
>>> the IDV and other communities, so even minor changes have impacts that
>>> would be nice to know of before they happen.
>>
>> Again, some changes we can announce with some lead-time, but these are
>> mostly limited to NWS or CONDUIT feed changes (for which email lists
>> already exist).
>
> NWS and CONDUIT change messages do not easily translate changes in the
> user will experience for the data access. It would be nice if the UPC
> would provide the translation and disseminate that information. An
> example of where this worked well was the move to the RAP.
>
Yep, I know that those messages can be "fun" to read :-) In fact, as I
said, the GFS announcement sort of caught us off guard. That said, it is
my opinion that Becky Cosgrove of NWS/NCEP Central Operations does a
great job making upcoming change announcements on the CONDUIT list very
readable.
The RUC to RAP is something that required particular care, as many
potential changes to downstream LDMs had to be made, so it was a big
deal. For the NAM 20km surface and selectsurface grids, I'm not sure
those were ever going out on the IDD, so no one playing a part in the
IDD needed to know about that change.
We want to do our best to keep our community informed of changes on our
systems (thus the basic catalog change page was created). We appreciate
feedback, and are interested in hearing more about what would be the
best, most useful way to get this information out.
Cheers,
Sean
> Don
