Re: [thredds] OLFS and/or BES configuration to obtain a dynamic refresh of existing files ...

  • To: "Benno Blumenthal" <benno@xxxxxxxxxxxxxxxx>
  • Subject: Re: [thredds] OLFS and/or BES configuration to obtain a dynamic refresh of existing files ...
  • From: "Claude DEROGNAT" <cderognat@xxxxxxx>
  • Date: Thu, 23 Feb 2012 22:21:50 +0100
Hi Benno,

 

I just have to change the name of dimension to have a continuous update of the 
targeted file, we mean ?

Thank you for the correction in my interpretation of the CF convention.

 

Regards, 

 

Ing. Claude DEROGNAT, PhD
ARIA TECHNOLOGIES SA
8-10, rue de la Ferme - 92100  Boulogne-Billancourt  - France
tél. / phone: 33 (0)1 46 08 68 60
ligne directe / direct line: 33 (0)1 46 08 68 65
fax: 33 (0)1 41 41 93 17
e-mail: cderognat@xxxxxxx <mailto:cderognat@xxxxxxx>  - http : www.aria.fr 

P Avant d'imprimer ce courriel, réfléchissez à l'impact sur l'environnement, et 
dans la mesure du possible éditez les documents en recto verso, merci!

 

 

De : bennoblumenthal@xxxxxxxxx [mailto:bennoblumenthal@xxxxxxxxx] De la part de 
Benno Blumenthal
Envoyé : jeudi 23 février 2012 22:12
À : Claude DEROGNAT
Cc : Nathan Potter; support@xxxxxxxxxxx; Patrick West; Nathan Potter; 
thredds@xxxxxxxxxxxxxxxx
Objet : Re: [thredds] OLFS and/or BES configuration to obtain a dynamic refresh 
of existing files ...

 

While I haven't seen the original file, my point is that you are violating the 
CF convention (assuming I am correct in thinking that the TDS matches the 
netcdf structure):  variables that have the same name as a dimension need to be 
one dimensional.

Benno

On Thu, Feb 23, 2012 at 3:57 PM, Claude DEROGNAT <cderognat@xxxxxxx> wrote:

Hi Benno,

 

Thank you for your answer. The corresponding name between grid coordinates and 
array dimension are driven by the NetCdf CF convention...

What about my real time constrain, so the require continuous update of the 
provides data ? How can I reach the property from my TDS catalog ?

 

Regards,

 

Claude

 

Ing. Claude DEROGNAT, PhD
ARIA TECHNOLOGIES SA
8-10, rue de la Ferme - 92100  Boulogne-Billancourt  - France
tél. / phone: 33 (0)1 46 08 68 60
ligne directe / direct line: 33 (0)1 46 08 68 65
fax: 33 (0)1 41 41 93 17
e-mail: cderognat@xxxxxxx <mailto:cderognat@xxxxxxx>  - http : www.aria.fr 

P Avant d'imprimer ce courriel, réfléchissez à l'impact sur l'environnement, et 
dans la mesure du possible éditez les documents en recto verso, merci!

 

 

De : bennoblumenthal@xxxxxxxxx [mailto:bennoblumenthal@xxxxxxxxx] De la part de 
Benno Blumenthal
Envoyé : jeudi 23 février 2012 19:45
À : Nathan Potter
Cc : Claude DEROGNAT; support@xxxxxxxxxxx Support; Patrick West; Nathan Potter; 
thredds@xxxxxxxxxxxxxxxx
Objet : Re: [thredds] OLFS and/or BES configuration to obtain a dynamic refresh 
of existing files ...

 

As for the two different structures -- the TDS generated a pure flat structure, 
Hyrax made Grid structures, from the netcdf API point of view they are very 
similar, though there is a significant difference w.r.t. height.

I wonder if the height variable is the reason TDS did not try to generate Grid 
structures -- It is a netcdf convention that the height dimension should have a 
height variable that is also one-dimensional.  In this case, the height 
variable is four-dimensional.  If the height dimension were renamed slightly, 
e.g. heightdim, you might get very different (and more consistent) results.

Benno

On Thu, Feb 23, 2012 at 1:09 PM, Nathan Potter <ndp@xxxxxxxxxxx> wrote:

 

 

 

On Feb 23, 2012, at 4:24 AM, Claude DEROGNAT wrote:

 

Hi, 

My IT said that the file I send you is an old one. My problem is probably link 
to my THREEDS catalog.
The question is so probably to threeds mailing list. I send them my question on 
Monday just after your message and I still have no response back ...
So, I send you my catalog if you can have a look to it you may find why there a 
mismatch between 
http://web.aria.fr:443/thredds/dodsC/CHIMERE@CAMAC@reference@p02/CF_CHIMERE_20120115_d03.nc.html
and 
http://web.aria.fr:443/LENVIS/CHIMERE/CAMAC/reference/p02/CF_CHIMERE_20120115_d03.nc.html

 

 

Claude,

I look at the first URL, saw that it was a instance of THREDDS Data Server (a 
fairly recent version 4.0.26) and I forwarded you message to their support list.

It was only later that I realized the second URL is what appears to be a VERY 
old (circa version 1.5.0), locally compiled, version of our Hyrax server.

So you are running two entirely different implementations of an OPeNDAP server. 

You can see this clearly when you look at one of the collections with a browser:

    TDS: http://web.aria.fr:443/thredds/dodsC/CHIMERE@CAMAC@reference@p02/

    Hyrax: http://web.aria.fr:443/LENVIS/CHIMERE/CAMAC/reference/p02/

 

        why the second is continuously updated but doesn't allow the access to 
the whole set of variables stored in NetCdf file.
        Why the first one are not continuously updated even if the date in the 
threeds catalogs presentation always mentioned the right modified time ... 


So with regards to how often datasets are "updated"  - this is a function of 
how the servers are implemented. I don't know what is going on in the TDS, but 
in the Hyrax server each new request gets the file information regarding file 
size and last modified times from the files system on which the dataset 
resides. Hyrax also doesn't cache responses, so it will re-interrogate the file 
every time you ask for something  in or about the file.

 

The TDS folks are better informed to comment on what's going on in the TDS with 
regards to this.

Now I think you are also saying that these two URLs:

 

http://web.aria.fr:443/thredds/dodsC/CHIMERE@CAMAC@reference@p02/CF_CHIMERE_20120115_d03.nc

http://web.aria.fr:443/LENVIS/CHIMERE/CAMAC/reference/p02/CF_CHIMERE_20120115_d03.nc

 

Reference the exact same file on the exact same file system. Is that the case?

 

If so, the fact that the data structure representations generated by each 
server for this file are so different is somewhat surprising:

 

 

>From the TDS:

 

http://web.aria.fr:443/thredds/dodsC/CHIMERE@CAMAC@reference@p02/CF_CHIMERE_20120115_d03.nc.dds

 

Dataset {

    String transverse_mercator;

    Float32 x[lon = 101];

    Float32 y[lat = 101];

    Float32 lat[time = 48][lat = 101][lon = 101];

    Float32 lon[time = 48][lat = 101][lon = 101];

    String projectionCoordSys;

    Float32 height[time = 48][height = 25][lat = 101][lon = 101];

    Float32 time[time = 48];

    Float32 Time[Time = 48];

    Float32 REL[time = 48][lat = 101][lon = 101];

    Float32 PHI[time = 48][height = 25][lat = 101][lon = 101];

    Float32 M001S001[time = 48][height = 25][lat = 101][lon = 101];

} CHIMERE%40CAMAC%40reference%40p02/CF_CHIMERE_20120115_d03.nc;

 

 

 

>From (very old) Hyrax:

 

http://web.aria.fr:443/LENVIS/CHIMERE/CAMAC/reference/p02/CF_CHIMERE_20120115_d03.nc.dds

 

Dataset {

    String transverse_mercator;

    Grid {

      Array:

        Float32 x[lon = 101];

      Maps:

        Float32 lon[lon = 101];

    } x;

    Grid {

      Array:

        Float32 y[lat = 101];

      Maps:

        Float32 lat[lat = 101];

    } y;

    String projectionCoordSys;

    Grid {

      Array:

        Float32 REL[time = 48][lat = 101][lon = 101];

      Maps:

        Float32 time[time = 48];

        Float32 lat[lat = 101];

        Float32 lon[lon = 101];

    } REL;

    Grid {

      Array:

        Float32 PHI[time = 48][height = 25][lat = 101][lon = 101];

      Maps:

        Float32 time[time = 48];

        Float32 height[height = 25];

        Float32 lat[lat = 101];

        Float32 lon[lon = 101];

    } PHI;

    Grid {

      Array:

        Float32 M001S001[time = 48][height = 25][lat = 101][lon = 101];

      Maps:

        Float32 time[time = 48];

        Float32 height[height = 25];

        Float32 lat[lat = 101];

        Float32 lon[lon = 101];

    } M001S001;

    Float32 Time[Time = 48];

} CF_CHIMERE_20120115_d03.nc;

 

 

This may be expected behavior, or not. I am tempted to say that you should 
upgrade both the TDS Hyrax to the latest releases before we dig to deeply...

 

 

Anyone else care to comment?

 

 

 

Thanks,

 

Nathan

 

 

 

 

 

 

 

 

 

 

 

For instance, 

why the second is continuously updated but doesn't allow the access to the 
whole set of variables stored in NetCdf file.
Why the first one are not continuously updated even if the date in the threeds 
catalogs presentation always mentioned the right modified time ... 

Regards

Ing. Claude DEROGNAT, PhD
ARIA TECHNOLOGIES SA
8-10, rue de la Ferme - 92100  Boulogne-Billancourt  - France
tél. / phone: 33 (0)1 46 08 68 60
ligne directe / direct line: 33 (0)1 46 08 68 65
fax: 33 (0)1 41 41 93 17
e-mail: cderognat@xxxxxxx - http : www.aria.fr 

P Avant d'imprimer ce courriel, réfléchissez à l'impact sur l'environnement, et 
dans la mesure du possible éditez les documents en recto verso, merci!



-----Message d'origine-----
De : Nathan Potter [mailto:ndp@xxxxxxxxxxx] 

Envoyé : mardi 21 février 2012 17:02
À : Claude DEROGNAT
Cc : Nathan Potter; Nathan Potter; support@xxxxxxxxxxx; Patrick West
Objet : Re: OLFS and/or BES configuration to obtain a dynamic refresh of 
existing files ...




Claude,

It would appear that you are using the THREDDS Data Server (TDS). I say this 
because the configuration file that you attached is unique to the TDS and is 
not a part of Hyrax. The TDS is a UNIDATA product and I think that they would 
be better positioned to answer your questions regarding this, you xan reach 
them here: thredds@xxxxxxxxxxxxxxxx

If you are pretty sure that you are trying to use Hyrax then I think we may 
have entered a twilight zone where parts of the two servers are being mashed up 
into one...

Can you confirm which server software you are attempting to configure and use?


Thanks,


Nathan



On Feb 21, 2012, at 5:03 AM, Claude DEROGNAT wrote:



HI, 

         

        Sorry for the delay of my answer but I had to deal with another huge 
project.

         

        My system will be dedicated to real time plume modeling.

        The model runs every 30 minutes. At the beginning of each day, it 
creates a result file then every 30 minutes the result file is overwritten with 
a new one containing the additional time frame.

         

        I observed à strange behavior between Opendap and thredds.

        The file is continuously updated in the OpenDap server.

        The threeds catalog notices in the 'Dates' field that the file are 
updates but the Access/OpenDap target file are not updated. The available time 
doesn't be continuously updated. May I have to perform any modification in the 
thredds  configuration ? A attached you my threddsConfig.xml file...

         

        Regards       

         

        Ing. Claude DEROGNAT, PhD

        ARIA TECHNOLOGIES SA

        8-10, rue de la Ferme - 92100  Boulogne-Billancourt  - France

        tél. / phone: 33 (0)1 46 08 68 60

        ligne directe / direct line: 33 (0)1 46 08 68 65

        fax: 33 (0)1 41 41 93 17

        e-mail: cderognat@xxxxxxx - http : www.aria.fr 

        P Avant d'imprimer ce courriel, réfléchissez à l'impact sur 
l'environnement, et dans la mesure du possible éditez les documents en recto 
verso, merci!

         

         

         

        -----Message d'origine-----

        De : Nathan Potter [mailto:npotter@xxxxxxxxxxx] 

        Envoyé : mardi 11 octobre 2011 19:31

        À : Claude DEROGNAT; support@xxxxxxxxxxx support

        Cc : Nathan Potter; Patrick West

        Objet : Re: OLFS and/or BES configuration to obtain a dynamic refresh 
of existing files ...

         

         

        On Oct 11, 2011, at 9:22 AM, Claude DEROGNAT wrote:

         

                Dear all,

                 

                With a correct title it should be better.

                 

                Regards,

                ...

                 

                Dear all,

                 

                I have a question about OpenDap functionality.

                Is-it possible to configure OLFS and/or BES to obtain a dynamic 

                refresh of existing files.

         

        If your model is writing to files that are within the scope of the 
BES.Catalog.catalog.RootDirectory, then Hyrax will just pick up the changes in 
the filesystem and refresh it's output. Little of that is cached, and if it is, 
it's triggered to refresh by a change in a files last modified time.

         

        If writing your model output into the

        BES.Catalog.catalog.RootDirectory doesn't work you might be able to use 
disk mounts and soft links to achieve what you want.

         

        But I have a feeling James and I haven't answered quite the question 
you were asking. The fact that James's earlier reply didn't help makes me think 
that repeating his point in more detail probably didn't either.

         

        - Are you perhaps wanting to define aggregations for a periodically 
changing file set? So that the aggregation definition always reflects the 
current state of the model?

         

        - Are you talking about automatically updating thredds catalog metadata 
for aggregations?

         

        - Updating NcML definitions of aggregations to reflect the new changing 
model results?

         

         

         

                I performed daily forecast of 48h and used Opendap to provide 
model 

                results to partners.

                Thank to assimilation each 30 minutes, I want to

                - add a new time frame on an existing file

         

        Your model can just write to the file within the scope of 
BES.Catalog.catalog.RootDirectory. Hyrax will see that the last- modified time 
of the file has changed.

         

                - replace (modified) some fields stored few hours ago.

         

        Again, I think this is just your model writing to files that are within 
the scope of BES.Catalog.catalog.RootDirectory

         

                 

                Is-is possible ?

                 

                Regards,

                 

                Ing. Claude DEROGNAT, PhD

                ARIA TECHNOLOGIES SA

                8-10, rue de la Ferme - 92100  Boulogne-Billancourt  - France

                tél. / phone: 33 (0)1 46 08 68 60

                ligne directe / direct line: 33 (0)1 46 08 68 65

                fax: 33 (0)1 41 41 93 17

                e-mail: cderognat@xxxxxxx - http : www.aria.fr

                P Avant d'imprimer ce courriel, réfléchissez à l'impact sur  

                l'environnement, et dans la mesure du possible éditez les 
documents  

                en recto verso, merci!

                 

                 

                 

                -----Message d'origine-----

                De : Nathan Potter [mailto:ndp@xxxxxxxxxxx]

                Envoyé : vendredi 18 septembre 2009 18:14

                À : Patrick West

                Cc : Nathan Potter; Claude DEROGNAT; support@xxxxxxxxxxx

                Objet : [SPAM?] Re: [Support] Bad Request 400 after a complete 
re- 

                installation ....

                 

                 

                 

                Claude,

                 

                The OLFS relies on Tomcat's built in authentication mechanism. 
REad

                this section:

                 

                
http://docs.opendap.org/index.php/Hyrax_-_OLFS_Configuration#Authentication_.26_Authorization

                 

                Of our documents for more information on that, although you 
will need

                to read the Tomcat documentation to get it sorted out. You can 
connect

                the authentication service to an LDAP server (using the 
JNDIRealm) if

                you wish, or you could use the example "MemoryRealm" if all you 
want

                is something simple but not as secure as LDAP.

                 

                 

                Nathan

                 

                 

                 

                On Sep 18, 2009, at 8:26 AM, Patrick West wrote:

                 

                         

                        On Sep 18, 2009, at 5:38 AM, Claude DEROGNAT wrote:

                         

                                Patrick,

                                 

                                Thank you very much for your help ... Thank 
also Nathan and all the

                                opendap support !!!!

                                I never success without your help !!!!

                                 

                         

                        Glad we could help. We are always working to make the 
installs

                        easier, and have formed a working group around the 
issue.

                         

                                the configuration similar to 
http://opendap.rpi.edu is perfect for

                                my application.

                                 

                                A last question ... the last I promise ;-))).

                                I didn't find simple way to introduce 
"authentication" to restrict

                                access to the opendap (OLFS) server.

                                You look very familiar with tomcat. Do you know 
simple way to only

                                introduce one (or more) login/password in the 
tomcat server access.

                                I have in mind something similar to ftp access 
: http://login:password@IP

                                :port .

                         

                        I'm not sure about this last part, whether the OLFS 
supports this,

                        can get the username and password and do anything with 
it. Nathan?

                         

                        Patrick

                         

                                 

                                Regards,

                                 

                                Claude Derognat * Ingénieur R&D

                                ARIA TECHNOLOGIES SA

                                8-10, rue de la Ferme - 92100  
Boulogne-Billancourt  - France

                                tél. / phone: 33 (0)1 46 08 68 60

                                ligne directe / direct line: 33 (0)1 46 08 68 65

                                fax: 33 (0)1 41 41 93 17

                                e-mail: cderognat@xxxxxxx - http : www.aria.fr

                                P Avant d'imprimer ce courriel, réfléchissez à 
l'impact sur

                                l'environnement, et dans la mesure du possible 
éditez les documents

                                en recto verso, merci!

                                 

                                 

                                -----Message d'origine-----

                                De : Patrick West [mailto:pwest@xxxxxxxxxxx]

                                Envoyé : jeudi 17 septembre 2009 19:38

                                À : Nathan Potter

                                Cc : Claude DEROGNAT; support@xxxxxxxxxxx

                                Objet : Re: [Support] Bad Request 400 after a 
complete re-

                                installation ....

                                 

                                For one of our projects here, actually two of 
them, we have a  

                                default

                                context. We set this in the server.xml file, 
before the closing Host

                                tag:

                                 

                                      <Context docBase="opendap" path="" 
reloadable="true"

                                source="org.eclipse.j    
st.j2ee.server:opendap"/>

                                </Host>

                                 

                                And, if you don't want to have to use :8080 
when going to the  

                                server,

                                and you don't already have an apache server 
running, you can use  

                                this

                                in your server.xml file.

                                 

                                <Connector port="80" address="128.113.28.182" 
protocol="HTTP/1.1"

                                           connectionTimeout="20000"

                                           useIPVHosts="true" 
redirectPort="443" />

                                 

                                This configuration gives us 
http://opendap.rpi.edu, where we serve

                                some sample data.

                                 

                                Might need to change this in other locations as 
well (port 80 for  

                                the

                                port tomcat listens on, and 443 for ssl 
connections.)

                                 

                                But if you have apache running, then this last 
part won't work,

                                unless

                                you change the apache port to something other 
than 80. There is a  

                                way

                                to redirect in httpd.conf, but I'm not sure how 
to do that.

                                 

                                In our case, we have one machine with three 
different IP addresses,

                                each one with either an apache server or tomcat 
server running, each

                                listening on port 80 for that IP address, which 
is why we specify  

                                the

                                address in the Connector.

                                 

                                Patrick

                                 

                                On Sep 17, 2009, at 12:58 PM, Nathan Potter 
wrote:

                                 

                                         

                                        On Sep 17, 2009, at 9:23 AM, Claude 
DEROGNAT wrote:

                                         

                                        Nathan,

                                         

                                        Thank you for your help.

                                        But your mail induces additional 
questions ...

                                         

                                        How do you configure Tomcat to use 
Hyrax (running in

                                        it's context of "opendap") as the 
"default" servlet ?

                                         

                                        At this point I would have to go read 
through the Tomcat

                                        documentation in order to answer your 
question.

                                         

                                        I think the answer is on this page:

                                         

                                        
http://tomcat.apache.org/tomcat-6.0-doc/default-servlet.html

                                         

                                         

                                        But I haven't tried this and I have no 
idea if it's actually

                                        possible (or even advisable...)

                                         

                                         

                                         

                                         

                                        I tried to couple Tomcat with Apache:

                                        NC Services (link to BES) work properly 
but I lost the most part  

                                        of

                                        the layout: (no logo, no blue 
horizontal bar, not link to

                                        documentation...).

                                         

                                        Did you read the section in the Hyrax 
documentation on integrating

                                        Hyrax with Apache??

                                         

                                        
http://docs.opendap.org/index.php/Hyrax_-_Apache_Integration

                                         

                                        I know people have done this and they 
have been able to make all of

                                        it work.

                                         

                                         

                                         

                                        What's why, I tried to deploy OLFS on 
the "default" Tomcat URL (http://IP:8080

                                        ) and NAT the external IP adresse to 
the LAN IP(:8080) of my

                                        OpenDap sever. Any idea about that ?

                                         

                                        That's always worked for me.

                                         

                                        I just set up the firewall so that all 
requests sent to  http://

                                        WAN_ADDRESS:80 are forwarded to  
http://NAT_ADDRESS:8080

                                         

                                        I hope that helps,

                                         

                                         

                                        Nathan

                                         

...

[Message clipped]  




-- 
Dr. M. Benno Blumenthal          benno@xxxxxxxxxxxxxxxx
International Research Institute for climate and society
The Earth Institute at Columbia University
Lamont Campus, Palisades NY 10964-8000   (845) 680-4450 

  • 2012 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the thredds archives: