[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [netcdf-java] Announcing Netcdf-Java / CDM version 4.3.8 BETA and emergency shutdown procedures

Hi John-

On 2/15/12 5:25 PM, John Caron wrote:

Jim Steenburg and I (and perhaps others) use the Best Time Series for
several datasets and I think as the IDV evolves its time matching
capabilities, the Best Time Series will become used more. For the Best
Time Series, we use index offsets to get the latest data. Dave Dempsey
also uses the forecast 0 data from the Constant Offset FMRC.


The FMRC will be replaced by the feature collections. FMRC is
particularly broken for GRIB files; it is essentially unmaintainable.
FMRC should be ok for netcdf files, as long as they are homogenous. The
GRIB feature collection should do the right thing for arbitrary
collections of GRIB files. The only constraint (I think) is they have to
come from the same center and subcenter.

Grib Feature collections have a "collection dataset" that is essentially
the same as the Best Time Series. There are probably cases where it is
correct where FMRC is not, but im not positive.
This might be okay in combination with the IDV time driver work. I have a couple use cases that I use the best time series for:
- match the satellite image times in a display giving either the analysis or the forecast in the series. - get the last week of analyses and the next week of forecasts of precipitable water to watch the evolution of atmospheric rivers. 
- show the last 12 hours of analyses from the RUC data.


The other variations, Constant Offset Constant Forecast and Runs will
not be supported, at least for now. So Ill have to understand how Dave
is using "forecast 0 data from the Constant Offset FMRC" to see what can
be done there. The individual files on motherlode will be equivalent to
the Runs, since we put all of one run in a single file.
I think the use case is to get the analyses for some time period (e.g. the last week). He could do this as I do from the best time series, but then you have to know the right indexes to use so you don't get the forecasts. 


For the large datasets on motherlode, there will also be daily
collections. This two-level collection (aka "Time partitions") should
scale to very large datasets. We dont really need it on motherlode, but
it makes the indexing better, plus we want to eat that dogfood.

BTW, index offsets are pretty unstable, as these datasets are changing
underneath you.
I understand the limitations, but for the use cases Jim and I have it works pretty well. 

Im hoping to prototype a "coordinate space only" Grid

feature type using cdmremote, if i every get out of the GRIB swamp. It
will solve this problem:

http://www.unidata.ucar.edu/blogs/developer/en/entry/indexed_data_access_and_coordinate





we can try to help the user choose new variable names, but the original
bundle has to be able to change, or else its going to be useless
eventually. Might as well do the right thing now, and add some UI to
help evolve the bundle.


There will need to be enough information stored in the variables so
that if the names change, a request can still find the appropriate
data. For example, if I have a bundle that is accessing temperature on
a pressure level, what can I use to always get that variable in the
dataset if the variable name changes?


Really, this is just a problem for GRIB files, which sucks so bad the
very fabric of spacetime is warped ;^( I'm certainly willing to add any
metadata that's appropriate. I doubt its possible to guarentee that one
can " always get that variable in the dataset if the variable name
changes", but probably can get pretty close. OTOH, the machinations to
do so might not be worth it. Actually, the "NCL naming scheme" below is
probably the best bet in this regard.


See the note I sent out this morning on this.





Also, how will you handle the FMRC stuff where older files were
indexed with the old names. Does 4.3 still read the old index files?


Im not sure what you mean by the index files here? Do you mean gbx8? Or
the cached xml files? In both cases, those are no longer used, and new
indexed files (gbx9 and ncx) are created.


I think this is a moot point so ignore for now. It sounds like you are
going to reindex everything for the 4.3.


One thing that would help is to generate a list of 4.2 variable names
with the corresponding 4.3 names for all the GRIB datasets on
motherlode. That could be used by the IDV for the lookup table.


unfortunately, the problem with using the human names is that they keep
getting tweaked (because the tables keep getting tweaked) by WMO and
especially NCEP. So they will just break again whenever that happens. Im
leaning towards an NCL-like variable name that will be much more stable
(though not guaranteed if we discover we are doing things wrong). The
implication is that an application will want to use the description when
letting users choose from a list, and the variable name when talking to
the API. I think IDV is already doing this?

The NCL-like syntax (still evolving) is:

VAR_%d-%d-%d[_error][_L%d][_layer][_I%s_S%d][_D%d][_Prob_%s]
L = level type
S = stat type
D = derived type



I did a quick scan and am not keen on the NCL like names. I would like
to see the VAR part replaced with a string that describes the variable
in some way like NCL does. I've been working with a lot of NCEP folks
who swear by wgrib2 which uses the names of the variables in the last
column of Table 4.2 for each parameter. These are the names that NCL
uses as well.


The VARs that NCL use are hand-maintained (by the NCL group) tables.
These "short names" are mostly only available from NCEP. They are not in
the WMO tables. They are also subject to being tweaked, and are not
always unique. So they seem only marginally better than the actual table
descriptions.
However, they are well recognized by many users of GRIB2 data at NCEP, NOAA and users of wgrib/wgrib2. Grib1 tables have a short name in the table, and I understand that Grib2 tables do now. NCAR must be maintaining a lookup list - can't you use that? 


I think using the description for human consumption is the right way to
go. Then let the variable names be as stable as possible, but not
particularly human readable.
FYI, the ECMWF grib software (http://www.ecmwf.int/publications/manuals/d/gribapi/keys/)(used in pygrib) allows users to specify aliases for parameters. For example, I can specify temperature as t, or Temperature. Perhaps the netCDF-Java library can have a facility for the user to specify an alias and other information and get back the appropriate variable (e.g. param="t", level="pressure"). 


Will you still provide these as descriptive names in the attributes?
The IDV uses them to categorize the variables in the Field Chooser.
Could you send along a comparison of a couple of variables with the
attributes listed so we can see how that has changed?


yes, i can add those.


Thanks.



> So its differrent from NCL in not using the time coordinate in the
name.

You are using the time interval in the name for accumulations, so I'm
not sure what you mean here.


We arent using any of the time interval coordinates, just the interval
lengths, in the variable name. Often, one has both a "time instant" and
a "time interval" variable in the same file. The interval length is in
the data, so not subject to getting changed in a table.

Ive prototyped making seperate variables for each time interval length.
I think thats probably wrong, but handling mixed intervals is tricky.
Probably the IDV needs to carefully consider this issue.
I would agree that's wrong. I think having the mixed_intervals identifier on the name and using the cell_bounds attribute is the way to go. I can provide you with some sample data from the grib2 datasets that I've been working with that illustrates this if you want. 


one thing to note is that the mapping is dataset dependent 15-20% of the
time.


Could you give examples of where these differ? I'd like to understand
if this is just in accumulation intervals or something more.


its mostly where the 4.2 name leaves off the level or other info, to
make a nice name. This is now Considered Harmful.


I agree per the note I sent out this morning.

Don
--
Don Murray
NOAA/ESRL/PSD and CIRES
303-497-3596
http://www.esrl.noaa.gov/psd/people/don.murray/