[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

20050223: IDV - Reading from OPeNDAP URLs with IDV

>From: "Roland Schweitzer" <address@hidden>
>Organization: Weathertop Consulting, LLC
>Keywords: 200502231559.j1NFx2d7026126   IDV Ferret

Hi Roland-

Good to hear from you.

>Institution: Weathertop Consulting, LLC
>Package Version: 1.2b1
>Operating System: Windows XP
>Hardware Information: 2.4GHz P4 w/ 512 MB of Ram
>Inquiry: Hi Don, Jeff and Co.
>
>We have just completed development on the first release of the Ferret Data Ser
> ver (FDS) an OPeNDAP server based on the Anagram framework and GDS developed 
> at COLA.  Of course, one of the first tests of our system was to attempt to r
> ead the data being served with IDV.  After working through several bugs at ou
> r end, we've finally reached a point where we're stuck.  By examining the FDS
>  request logs it appears to us as if IDV is requesting the entire contents of
>  the OPeNDAP served netCDF file.  For very large files the server refuses to 
> serve the entire file and returns an error message instead.  

> Leaving aside fo
> r a moment the fact that we don't believe that IDV really wants to request th
> e entire dataset, we noticed that  IDV responds to this condition by adding t
> he data source to the list, but the list of times, fields and displays is bla
> nk instead of passing on the error message returned by the server.  I've past
> ed an email from our internal discussion of this problem which i!
> ncludes URLs that you can test.

Looking at the scenario below, I'm guessing the user is trying to use
the URL Chooser and is using the Default Data Source Type.  In that case, 
the IDV is defaulting to the VisAD Data form and it just tries several 
different ways of opening the file.

There are two ways around this:

1) Set the Data Source Type in the URL chooser to be netCDF Grid Files.
This is changed in the next release and will be listed as
"Grid Files (netCDF/GRIB/OPeNDAP)".

2) Keep the type as Default and change the URL to be:

dods://tmap.pmel.noaa.gov/RHS-las-FDS/LAS/gfdl_ocean_simulation/temp

> Leaving aside fo
> r a moment the fact that we don't believe that IDV really wants to request th
> e entire dataset, we noticed that  IDV responds to this condition by adding t
> he data source to the list, but the list of times, fields and displays is bla
> nk instead of passing on the error message returned by the server.  I've past
> ed an email from our internal discussion of this problem which i!
> ncludes URLs that you can test.

Adding the data source to the list should only occur when another is
successfully loaded.  That is a bug.  It may be fixed in the new
release.

Using the default setting though should have popped up an error
dialog with the following message:

An error has occurred: There was an error loading the data: Failed to open 
http://tmap.pmel.noaa.gov/RHS-las-FDS/LAS/gfdl_ocean_simulation/temp
Data object 
"http://tmap.pmel.noaa.gov/RHS-las-FDS/LAS/gfdl_ocean_simulation/temp";
not compatible with "VisADDataSource" data family

As for specifics:

>----------------
>Hi, Roland, 
>
>    I downloaded IDV and installed it. After running IDV against FDS and looki
> ng at the log file. 
>I found it is a IDV implementation problem. 
>
>   For your example, after I pasted http://tmap.pmel.noaa.gov/RHS-las-FDS/LAS/
> gfdl_ocean_simulation/temp 
>into the data source. IDV asks for following thing: 
>(1) asks for http://tmap.pmel.noaa.gov/RHS-las-FDS/LAS/gfdl_ocean_simulation/t
> emp without any extension and constraint 
>   several times. I don't understand why it does this, it will get error messa
> ge from other DODS server and info from GDS and FDS. 

This is the VisAD form trying to read it through it's different
adapters.  I got the following listings:

DConnect getData failed; retry (1,100) http://tmap.pmel.noaa.gov/RHS-las-FDS/LAS
/gfdl_ocean_simulation/temp.dods
DConnect getData failed; retry (2,200) http://tmap.pmel.noaa.gov/RHS-las-FDS/LAS
/gfdl_ocean_simulation/temp.dods
DConnect getData failed; retry (3,400) http://tmap.pmel.noaa.gov/RHS-las-FDS/LAS
/gfdl_ocean_simulation/temp.dods
DConnect getData failed; retry (4,800) http://tmap.pmel.noaa.gov/RHS-las-FDS/LAS
/gfdl_ocean_simulation/temp.dods
DConnect getData failed; retry (5,1600) http://tmap.pmel.noaa.gov/RHS-las-FDS/LA
S/gfdl_ocean_simulation/temp.dods

>(2) asks for http://tmap.pmel.noaa.gov/RHS-las-FDS/LAS/gfdl_ocean_simulation/t
> emp.das 
>(3) asks for http://tmap.pmel.noaa.gov/RHS-las-FDS/LAS/gfdl_ocean_simulation/t
> emp.dods without any constraint. This is problematic, 
>    the dataset could be very big. In our example it is 340MB. So FDS refused 
> to give that data and sent back a DODS error message 
>    saying the requested size is too big. IDV does not process this DODS error
>  message. 

Not sure where this is happening.

>    To confirm the error message is sent, please try: http://tmap.pmel.noaa.go
> v/RHS-las-FDS/LAS/gfdl_ocean_simulation/temp.asc 
>The ascii output display in FDS is generated by DODS library by sending a url 
> request to the local FDS for dods data. From the result we can 
>see DODS library does receive the error message correctly.  
>   IDV asks for everything from the start, it is good for performance for smal
> l dataset but can't handle big dataset. In this aspect, Ferret 
>is better:  Ferret asks for metadata first then asks for subset of the data. 

When the correct Data Source Type is used, it does just go off and
reads the metadata.  It only populates the Data Selector with the list
of times and variables.

When you request data for a time, it will ask the server for all data
for that time for only the parameter requested.

>   To confirm if this is right, I give a IDV virtual dataset which is a subset
>  of the original: 
>http://tmap.pmel.noaa.gov/RHS-las-FDS/LAS/gfdl_ocean_simulation/_expr_{temp}{A
> =TEMP[k=20,l=100]} 
>
>    IDV gives the good plot of it.

Don Murray
NOTE: All email exchanges with Unidata User Support are recorded in the
Unidata inquiry tracking system and then made publicly available
through the web.  If you do not want to have your interactions made
available in this way, you must let us know in each email you send to us.