Re: [idvusers] Important changes regarding the Unidata IDV

Sean-

One further note. Because of the GRIB variable name changes between TDS 4.2 and 4.3, using option 2 below might still cause some issues for users of old bundles with or without the Best Time Series. While Sean has done a lot of work to make sure that old names are mapped automatically to new names, there are some cases where there is not a 1:1 match between the old/new variables. This will cause a dialog to pop up where the user can select the appropriate variable, but that may be confusing in a classroom setting.

Don

On 3/25/13 10:14 AM, Sean Arms wrote:
Hi All,

First, thanks Don for sending out the additional information. What Don
has found is a reflection of the fact that the IDV 4.0 was released
prior to the formation of the timeline outlined in the initial message.
The end result is that if you want to use IDV 4.0, you have two options.
1) install the "Motherlode TDS 4.2 persistence (for IDV 4.0)" plugin,
which will keep bundles pointing to the older TDS server accessed using
motherlode.ucar.edu, or 2) install the "Test
Unidata TDS 4.2 -> 4.3 Upgrade" plugin, which will allow for a full
remapping of motherlode.ucar.edu urls to the new thredds.ucar.edu access
url.

My recommendation is as follows:

1) If you are using the IDV for a purpose that is time sensitive (like
end of the semester projects, or critical projects that will end before
the June 1st deadline), you should:

     a) Stick with IDV 3.1 (making sure to upgrade to the latest IDV
release before June 1st), or
     b) Use IDV 4.0, but with the "Motherlode TDS 4.2 persistence (for
IDV 4.0)" plugin.

Note that after June 1st, neither a) nor b) will work, and you will need
to upgrade to the latest version of the IDV.

2) If you do not have an absolute need to stick with the older TDS
server, you should use the IDV 4.0 with the "Test Unidata TDS 4.2 -> 4.3
Upgrade" plugin installed. Old bundles that use the Best Time Series
datasets *will continue to work*, but will use data from the new TDS
server thredds.ucar.edu instead of the older, problematic
motherlode.ucar.edu TDS server.

Running the IDV 4.0 without one of the two plugins installed will result
in errors like Don described.

It it our intent to release a new version of the IDV prior to the June
1st deadline that will negate the need to use any plugin to properly
access data from the Unidata TDS server. However, if you decided to
stick with IDV 4.0 at that time rather than upgrading, your IDV will
continue to work as long as you keep the "Test Unidata TDS 4.2 -> 4.3
Upgrade" plugin installed.

As a side note, most of the IDV development team is out of the office
this week due to Boulder Valley School District's Spring Break. That
said, I will be checking email and support to ensure your questions and
concerns regarding this transition are addressed.

Cheers,

Sean

On 3/25/13 8:45 AM, Don Murray (NOAA Affiliate) wrote:
All-

One further note on this.  If you are running IDV 4.0 and are using
bundles that access "Best Time Series" datasets, you will need to
install the "Data Sources">"Motherlode TDS 4.2 persistence (for IDV
4.0)" for them to continue working.  Many of my bundles do not work
without this plugin, but the note below doesn't address this issue.
You may get an error without the plugin, or it may just not load the
Best Time Series data.  If you are running IDV 3.1, the Best Time
Series bundles will work until June 1, but after that you will need to
switch to IDV 4.0 (and remove the plugin).

Don Murray

On 3/22/13 5:18 PM, Sean Arms wrote:
Greetings IDV community!

This is a message to let you know that the transition of the official
Unidata TDS server, thredds.ucar.edu, to TDS version 4.3 is complete.
Also note that the main IDV catalog,

http://www.unidata.ucar.edu/georesources/idvcatalog.xml

has been updated to reflect this change. You will notice that there are
now two entries under "Unidata IDD Model Data":

"THREDDS Model Data" and "IDV Model Data (Available until 1 June 2013)".

The "THREDDS Model Data" catalogs point to thredds.ucar.edu (the latest
TDS server); the "IDV Model Data" catalogs point to motherlode.ucar.edu,
which is the older TDS.

The remaining timeline is as follows:

Saturday, June 1st, 2013 - the previous data access URL,
motherlode.ucar.edu, will be changed to use the TDS 4.3 server. This
marks the end of TDS 4.2 access on Unidata systems.

Thursday, August 1st, 2013 - the data access URL motherlode.ucar.edu
will no longer function. All users of the Unidata TDS should transition
to using thredds.ucar.edu by this time. All users will need to upgrade
to at least IDV 4.0 by this date.

Please see the message below for the gory details. Please note that we
remain committed to making this process go as smoothly as possible. If
at any point you have questions, please direct them to
support-idv@xxxxxxxxxxxxxxxxx

Cheers,

The IDV and THREDDS Development Teams

On 3/19/13 1:09 PM, Sean Arms wrote:
Greetings IDV community!

The IDV and THREDDS teams have been working hard on transitioning the
IDV to the latest version of netCDF-Java, as well as making
modifications to allow the IDV to take advantage of the newest version
of the THREDDS Data Server (TDS version 4.3).

The time has arrived for the official Unidata TDS to transition to
version 4.3. There are very important improvements in this version,
especially fixing incorrect GRIB data handling, affecting ~25% of all
GRIB parameters from NCEP models. However, we understand that moving
to a new version of the IDV and accessing data from a new data server
during the school semester may cause some uneasiness. We've come up
with a transition plan that we hope will address your concerns for the
remainder of the Spring 2013 semester, while at the same time allowing
us to make the necessary upgrades to the Unidata TDS.

Why are we doing this? Why is an upgrade necessary?
---------------------------------------------------

For more information on why the move to netCDF-Java and TDS 4.3 is
necessary (including fixing incorrect handling of GRIB-1/2 data
formats), please browse a very small selection of the discussion that
has taken place over the past year or so:

http://www.unidata.ucar.edu/blogs/news/entry/netcdf_java_library_and_tds

http://www.unidata.ucar.edu/blogs/developer/en/entry/grib_renaming
http://www.unidata.ucar.edu/blogs/developer/en/entry/on_the_suitability_of_grib


http://www.unidata.ucar.edu/blogs/developer/en/entry/grib_table_case_studies


http://www.unidata.ucar.edu/mailing_lists/archives/netcdf-java/2012/msg00048.html


http://www.unidata.ucar.edu/mailing_lists/archives/netcdf-java/2012/msg00060.html


http://www.unidata.ucar.edu/mailing_lists/archives/netcdf-java/2012/msg00123.html


http://www.unidata.ucar.edu/mailing_lists/archives/netcdf-java/2012/msg00240.html



(yes, this is a 'very small' portion of the discussion!)

The bottom line is that you WANT to make this transition, and we
strongly encourage you to do so now. If you need to delay this until
the semester is over, then we have ways for you to do that, detailed
below.

Ok, so when will this happen?
-----------------------------

The timeline for transitioning from the TDS 4.2 to the 4.3 server at
Unidata is as follows:

Friday, March 22nd, 2013 - the official data access URL for the
Unidata TDS will be thredds.ucar.edu. This address will resolve to a
TDS 4.3 server (the latest version).

Saturday, June 1st, 2013 - the previous data access URL,
motherlode.ucar.edu, will be changed to use the TDS 4.3 server. This
marks the end of TDS 4.2 access on Unidata systems.

Thursday, August 1st, 2013 - the data access URL motherlode.ucar.edu
will no longer function. All users of the Unidata TDS should
transition to using thredds.ucar.edu by this time.

What does this mean for IDV users?
----------------------------------

Well, that depends on which version of the IDV you are using.

1) If you or your lab are currently using the IDV version 3.1 (or
earlier), then you will not need to do anything special for the
remainder of the Spring Semester. By June 1st, 2013, you must upgrade
to the latest version of the IDV (currently 4.0).

2) If you are currently using IDV 4.0 and are ready to make the
transition, simply start accessing the Unidata server at
thredds.ucar.edu by typing the following URL into the Catalog address
box in the Data Choosers tab:

http://thredds.ucar.edu/thredds/catalog.xml

Note that the default catalog used by the IDV,
http://www.unidata.ucar.edu/georesources/idvcatalog.xml, will include
a link to the thredds.ucar.edu under "Unidata IDD Model Data" once the
transition to TDS 4.3 has been made this Friday.

3) If you are currently using IDV 4.0, use bundles, and would like to
continue using the older version of the Unidata TDS (version 4.2), you
will need to install a plugin to allow IDV 4.0 to keep using the older
TDS. The required plugin is available in the IDV plugin manager under
the plugin category:

"Data Sources">"Motherlode TDS 4.2 persistence (for IDV 4.0)"

Using this plugin should ensure that the IDV continues to interface
with the TDS 4.2 server as it did in previous versions, at least until
June 1st 2013. After June 1st, you should uninstall this plugin.

Will all of my stuff break?
---------------------------

We don't believe so. Many of the changes between TDS 4.2 and 4.3, such
as new GRIB variable names and new access URLs, are handled 'under the
hood' of the IDV and netCDF-Java libraries, which is why older bundles
should continue to work. However, there are some features of the TDS
4.2 that are not yet implemented in TDS 4.3. This decision was based
on initial feedback that these features were labeled confusing or
unnecessary - specifically, the "constant forecast offset" and
"constant forecast hour" features of the TDS Forecast Model Run
Feature collections (FMRC), which have been replaced by the GRIB
feature collections. We know this will be missed by some, but we hope
to have these features back in the TDS in the future.

Want to test your bundles with TDS 4.3 and the IDV 4.0?
-------------------------------------------------------

If you are using IDV version 4.0 and would like to test how the new
TDS 4.3 server works with previous bundles, you can install the "Test
Unidata TDS 4.2 -> 4.3 Upgrade" plugin, also found in the IDV plugin
manager. If you've downloaded this plugin previously, you should
remove it and install it again, as we needed to make some changes
needed to the plugin to accommodate the timeline above. This plugin
will work once the official Unidata TDS, thredds.ucar.edu, moves to
4.3 (scheduled for this Friday, March 22). Note: if you install this
plugin, you will need to remove the TDS 4.2 persistence plugin - you
may use one plugin or the other, but not both.

Lastly, we know that many IDV users are not subscribed to this list.
If you know of someone who uses the IDV, please contact them to make
sure they are aware of these very important upcoming changes. As
always, feel free to contact us if you have questions, comments, or
concerns, at support-idv@xxxxxxxxxxxxxxxxx

Cheers,

The IDV and THREDDS Development Teams




--
Don Murray
NOAA/ESRL/PSD and CIRES
303-497-3596
http://www.esrl.noaa.gov/psd/people/don.murray/