[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [njtbx-users] Problem accessing NcML file via OpenDAP

Hi Giles:

Some questions are embedded  below:

On 5/25/2010 10:33 AM, Rich Signell wrote:

Giles,

I actually don't have that much experience with this type of data, but
I'm cc'ing this to John Caron for his opinion.   He is very actively
working on time series stuff and I'm sure he has better ideas than I
on the appropriate approach to take.

John, please group reply, as I'm interested in the reply as well!

-Rich

On Mon, May 24, 2010 at 2:22 PM,<address@hidden>  wrote:

Hi Rich

Good to hear from you. Fun stuff this netCDF/NcML. I've got a lot to learn
and would welcome your opinions/advice...

I/we are exploring this path (using remote NcML) because we would like to
set up a server which serves up many (several dozen) time-series datasets
("live" data from wave buoys and tide gauges and met sensors as well as
WAM model forecasts, etc) from several (around 20) ports. The datasets
will be continuously updated - every 30mins for wave data, every 6-10mins
for tide data either by appending existing files or by "rolling" to new
files every... not sure yet, will depend on source and data type - lets
say month, as a guess... we will then need to aggregate the sequences of
monthly files when we want to retrieve longer records.

A number of other requirements:
1. Speed/efficiency is important, when we query for data it must come back
fast. Scan aggregations without caching will definitely be too slow.


what will the queries be? What will be the queries be that need to be fast?

In what format do you want the data to be returned?


2. It is essential that we can re-create the exact state of the datasets
at specific times - for re-creating queries at a later time in case
questions arise. This makes me wary of the caching built into TDS - unless
the "refresh every" time is set very small, in which case what is the
point...?
If data has not yet arrived, you will get different results. How does that fit into the need for reproducibility? 


3. I don't like the idea of having to restart the TDS every time a dataset
definition is updated in the catalog.xml (it would need to be restarted
very frequently).


Not sure what
you mean by the dataset definition? Why is it getting updated?


The above leads me to attempt a system where the NcML (joinExisting)
aggregations are done explicitly in remote files. I am making them nice
efficient aggregations where the coordinate dimensions are explicitly
specified in the NcML. These NcML files are being updated by the same
python scripts that are adding/appending the new data as it comes in. In
this way the TDS catalog.xml only needs to be changed and the TDS
restarted when new datasets are defined.

This also means that we can point our NetCDF data clients (Matlab, Python,
and we will somehow need to get something working in C#.NET) directly at
the NcML files rather than the TDS if we feel like it - not quite sure why
we would at this stage though...

Hope that paints the scene. I welcome you comments.

The answer to your second question is yes. We want to serve the virtual
datasets via OpenDAP.
This whole process is an exploration of whether NetCDF/NcML/TDS/OpenDAP 
would offer a better (flexible/robust/transparent/extensible) solution
than the SQL databases we currently use for the same purpose...


what issues do you have with the SQL database?


Kind regards


Giles




Giles,

Interesting finding.   And<address@hidden>  is
definitely the right place to ask.   But why do you want to use remote
NcML if you are running a THREDDS Data Server?
Do you want to embed that NcML into a thredds catalog so you can serve
the virtual dataset created by your NcML via OpenDAP and other
services?

-Rich

On Mon, May 24, 2010 at 4:40 AM, Giles Lesser
<address@hidden>  wrote:

All

Further to my earlier posts, I have done some more investigation (using
toolsUI-4.0.49.jar) and have determined that:

a) if I point either the njTbx or toolsUI at a URL such as
http://******:8080/thredds/dodsC/enviroData/regional/australasia/bom/wam/allSpec1d.ncml
then they both fail, however toolsUI returns the rather more helpful
error message
"Server returned HTTP response code: 400 for URL:...."
where as njTbx returns
"read error: stream is closed"

however,

b) if I simply change the "http" at the start of the URL to "dods" then
everything works just fine. ie
dods://******:8080/thredds/dodsC/enviroData/regional/australasia/bom/wam/allSpec1d.ncml
works fine in both njTbx and toolsUI.

This leads me to the (poorly understood) conclusion that the problem is
most
likely somewhere in the configuration of Tomcat/thredds where an
extension of
.ncml on an HTTP URL causes the server confusion, however an .ncml
extension on
a DODS URL is passed on (to thredds?) and evaluates OK.

I would like to understand why this occurs, however I appreciate that
this forum
is probably not the place for help. I will attempt to post to the
unidata-thredds mailing list, however if someone here can explain why
this
occurs I would be very grateful.

Thanks for your help.


Giles
______________________________________________________________________________
Giles Lesser, PhD | Research and Development Manager&  Senior Coastal
Engineer


OMC-International | 6 Paterson St | Abbotsford, VIC 3067

Melbourne | Australia

Phone +61 (3) 9412 6501 | Fax +61 (3) 9415 9105

http://www.omc-international.com.au

Dedicated to safer and more efficient shipping.

CONFIDENTIAL COMMUNICATIONS. The information contained in this e-mail is
confidential and may be subject to legal professional privilege. It is
intended solely for the addressee. If you received this correspondence
by
mistake, please promptly inform us by reply e-mail or by telephoning +61
3
9412 6500 and then delete the e-mail and destroy any printed copy. You
must
not disclose, copy or rely on any part of this correspondence if you are
not
the intended recipient.

On 14/05/2010 11:13 AM, address@hidden wrote:

Hi Sachin

My catalog.xml file follows. As you can see this is a test server we
are
experimenting with to see if OpenDAP serves the needs we have (and that
I
am just a beginner....)

I was/am attempting to access the LAST dataset declared in this
catalog.xml file.

You will also notice that I was experimenting with making the dataType
of
the datasets "Point" or "Grid" - as I'm not sure which is appropriate.
The
.nc files are NOT grid results in the numerical modelling sense of the
word, they contain output (wave spectra) for several forecast times at
several discrete stations.

The NcML files aggregate multiple .nc files along the analysisTime
dimension. I will also include a .ncml file FYI.

Many thanks


Giles

------------------------------------------------------------
start of catalog.xml
------------------------------------------------------------
<?xml version="1.0" encoding="UTF-8"?>
<catalog name="Giles' Simple Little THREDDS Server Catalog"

  xmlns="http://www.unidata.ucar.edu/namespaces/thredds/InvCatalog/v1.0";
          xmlns:xlink="http://www.w3.org/1999/xlink";>

    <service name="all" base="" serviceType="compound">
      <service name="odap" serviceType="OpenDAP" base="/thredds/dodsC/"
/>
      <service name="http" serviceType="HTTPServer"
base="/thredds/fileServer/" />
    </service>

    <datasetRoot path="enviroData/regional/australasia/bom/wam"
location="/home/enviroData/regional/australasia/bom/wam/" />
    <datasetRoot path="enviroData/regional/australasia/bom/wam/2008"
location="/home/enviroData/regional/australasia/bom/wam/2008/" />
    <datasetRoot path="enviroData/regional/australasia/bom/wam/2009"
location="/home/enviroData/regional/australasia/bom/wam/2009/" />
    <datasetRoot path="enviroData/regional/australasia/bom/wam/2010"
location="/home/enviroData/regional/australasia/bom/wam/2010/" />

    <dataset name="The first WAM forecast" ID="WAM2008072200"
urlPath="enviroData/regional/australasia/bom/wam/2008/spec1d_2008072200.nc"

      <serviceName>odap</serviceName>
      <dataType>Point</dataType>
    </dataset>

    <dataset name="BOM WAM forecasts 2008 - 1D" ID="BOMWAM2008_1D"
urlPath="enviroData/regional/australasia/bom/wam/2008/allSpec1d.ncml">
      <serviceName>odap</serviceName>
      <dataType>Point</dataType>
    </dataset>
    <dataset name="BOM WAM forecasts 2008 - 2D" ID="BOMWAM2008_2D"
urlPath="enviroData/regional/australasia/bom/wam/2008/allSpec2d.ncml">
      <serviceName>odap</serviceName>
      <dataType>Point</dataType>
    </dataset>

<dataset name="BOM WAM forecasts 2009 - 1D" ID="BOMWAM2009_1D"
urlPath="enviroData/regional/australasia/bom/wam/2009/allSpec1d.ncml">
      <serviceName>odap</serviceName>
      <dataType>Point</dataType>
    </dataset>
<dataset name="BOM WAM forecasts 2009 - 2D" ID="BOMWAM2009_2D"
urlPath="enviroData/regional/australasia/bom/wam/2009/allSpec2d.ncml">
      <serviceName>odap</serviceName>
      <dataType>Point</dataType>
    </dataset>

<dataset name="BOM WAM forecasts 2010 - 1D" ID="BOMWAM2010_1D"
urlPath="enviroData/regional/australasia/bom/wam/2010/allSpec1d.ncml">
      <serviceName>odap</serviceName>
      <dataType>Point</dataType>
    </dataset>
<dataset name="BOM WAM forecasts 2010 - 2D" ID="BOMWAM2010_2D"
urlPath="enviroData/regional/australasia/bom/wam/2010/allSpec2d.ncml">
      <serviceName>odap</serviceName>
      <dataType>Point</dataType>
    </dataset>

    <dataset name="BOM WAM forecasts ALL - 1D" ID="BOMWAM_ALL_1D"
urlPath="enviroData/regional/australasia/bom/wam/allSpec1d.ncml">
      <serviceName>odap</serviceName>
      <dataType>Grid</dataType>
      <!--
      <netcdf
xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2";>
        <aggregation dimName="validTime" type="joinExisting"
recheckEvery="1
min">
          <netcdf
location="/home/enviroData/regional/australasia/bom/wam/2008/allSpec1d.ncml"/>
          <netcdf
location="/home/enviroData/regional/australasia/bom/wam/2009/allSpec1d.ncml"/>
          <netcdf
location="/home/enviroData/regional/australasia/bom/wam/2010/allSpec1d.ncml"/>
        </aggregation>
      </netcdf>
      /-->
    </dataset>

    <dataset name="BOM WAM forecasts ALL - 2D" ID="BOMWAM_ALL_2D"
    urlPath="enviroData/regional/australasia/bom/wam/allSpec2d.ncml">
      <serviceName>odap</serviceName>
      <dataType>Grid</dataType>
    </dataset>


</catalog>


------------------------------------------------------------
end of catalog.xml
------------------------------------------------------------

------------------------------------------------------------
start of allSpec1d.ncml
------------------------------------------------------------
<?xml version="1.0" encoding="UTF-8"?>
    <netcdf
xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2";>
      <aggregation dimName="analysisTime" type="joinExisting">
        <scan location="\." suffix="spec1d_*.nc" subdirs="true" />
       </aggregation>
    </netcdf>
------------------------------------------------------------
end of allSpec1d.ncml
------------------------------------------------------------



Giles,

I can see from your previous post that you can access ncml locally via
njtbx, so it seems
that your TDS may have some issues as it is unable to open dataset.

Can you provide your catalog.xml ?

-sachin

On Thu, May 13, 2010 at 6:34 PM,<address@hidden>
  wrote:


Hi Sachin

Wow, quick response!

No, sorry I can't - well I could, but it wouldn't be any use. It is a
(trial) server running on our internal network. No access from the
outside.

Anything else I can do to help?

Thanks


Giles




Giles,

Can you provide the complete url ?

http://


******:8080/thredds/dodsC/enviroData/regional/australasia/bom/wam/allSpec1d.ncml

thanks,

-sachin

On Thu, May 13, 2010 at 6:13 PM,<address@hidden>

wrote:

Dearest njtbx-users

I am using njtbx to access data aggregated (joinExisting) using a

NcML

file. When I point njtbx at the file directly all works fine. When
I
serve
the (same) NcML file using a THREDDS OpenDAP server and then try
accessing
the data via the OpenDAP URL njtbx fails (despite being able to

browse

the
OpenDAP dataset quite happily using a web browser). I would

appreciate

any
help in figuring out how to debug this problem.

The problem seems to be caused by njtbx failing to create a
mDataset
from
the OpenDAP URL. For example, when the mDataset fucntion is
provided

the

file location it returns:

               URI:
'\\*******\envirodata/regional/australasia/bom/wam/allSpec1d.ncml'
          numGrids: 0
     numDimensions: 6
      numVariables: 10

However, when pointed to the OpenDAP URL mDataset returns:

Unable to open the dataset:
http://


******:8080/thredds/dodsC/enviroData/regional/australasia/bom/wam/allSpec1d.ncml

read error: stream is closed
Attempt to reference field of non-structure array.

Debugging further leads to the discovery that the (Matlab) error

occurs

in
the function njGetInfo

The java object NCid provided as input to that function appears to
be
OK:
K>>    NCid

NCid =

msstate.cstm.data.JDataset@476e0993

However the getGridDataset() method fails to return anything
K>>    NCid.getGridDataset()

ans =

      []

and hence line 21 of njGetInfo fails:
K>>    nGrid = size(NCid.getGridDataset().getGrids());
??? Attempt to reference field of non-structure array.


My question is: Is this clearly a bug in the Java toolbox? or is it
possible that something about either my underlying NetCDF files,
NcML
aggregation file, or TDS dataset definition could be at fault?

I can post/provide examples of .nc files, NcML file, and TDS

catalog.xml

file if that would help anyone.

Any pointers very gratefully received.

Many thanks


Giles







------------------------------------------------------------------------------

_______________________________________________
njtbx-users mailing list
address@hidden
https://lists.sourceforge.net/lists/listinfo/njtbx-users




--
Sachin Bhate, Research Associate III
MSU-High Performance Computing Collaboratory's NGI
John C. Stennis Space Center, MS 39529
http://njtbx.sourceforge.net/






--
Sachin Bhate, Research Associate III
MSU-High Performance Computing Collaboratory's NGI
John C. Stennis Space Center, MS 39529
http://njtbx.sourceforge.net/




------------------------------------------------------------------------------

_______________________________________________
njtbx-users mailing list
address@hidden
https://lists.sourceforge.net/lists/listinfo/njtbx-users




--
Dr. Richard P. Signell   (508) 457-2229
USGS, 384 Woods Hole Rd.
Woods Hole, MA 02543-1598