Re: [thredds] OpenDAP constraint expression

Hi Ashwin,

If you have Netcdf library, version above 4.1 and compile with opendap capabilities you can read opendap URLs as like a local netcdf:

http://www.unidata.ucar.edu/software/netcdf/docs/dap_support.html

This means that you can use as "file"name the opendap URL, including the constraint expressions.

For example, you can use ncdump to see the remote metadata:

$ ncdump -h "http://www.esrl.noaa.gov/psd/thredds/dodsC/Datasets/ncep.reanalysis2/pressure/air.2014.nc";


or you can use James's CE:

$ ncdump -h "http://www.esrl.noaa.gov/psd/thredds/dodsC/Datasets/ncep.reanalysis2/pressure/air.2014.nc?air[480:603][2][20:34][26:40]";

(Important: I have removed the .ascii after .nc, this is required)

Or copy the remote data to a local netcdf file using nccopy:

$ nccopy "http://www.esrl.noaa.gov/psd/thredds/dodsC/Datasets/ncep.reanalysis2/pressure/air.2014.nc?air[480:603][2][20:34][26:40]"; air.nc

but the auxiliary coordinates are missing, therefore you also need to "copy them"

$ nccopy "http://www.esrl.noaa.gov/psd/thredds/dodsC/Datasets/ncep.reanalysis2/pressure/air.2014.nc?air[480:603][2][20:34][26:40],time[480:603],level[2],lat[20:34],lon[26:40]"; air.nc

And you will have a local subset netcdf file:

$ ncdump -h air.nc

As I mention this depends on the netcdf library do you have, but if it's the right one, if your Fortran, C program it's linked against it you will able to opendap URLS as local files. This includes R and Python packages using this library.

Netcdf-java has these features but many others interesting ones.

Regards

Antonio


--
Antonio S. Cofiño
Grupo de Meteorología de Santander
Dep. de Matemática Aplicada y
        Ciencias de la Computación
Universidad de Cantabria
http://www.meteo.unican.es



El 27/05/2016 a las 11:13, ashwinD12 . escribió:
Hi Don,
Thanks for your detailed response. Please allow me to confirm what you have written in my own words so that I can begin programming at the earliest.

As an aside I would like to say my ultimate goal is to automate this process i.e. the download of NOAA reanalysis data and the subsequent uptake by the Fortran program will have NO human intervention whatsoever. The subsetting and generation of the netCDF file will be automated and it will be part of a server process run every day(same subset will be generated four times a day). From my understanding of what you have written this is certainly and clearly possible with what you have written.

Regarding the specifics of what you have written in relation to the programming language of choice

1) If I use java netcdf(the latest version) I can use the OPeNDAP URL that James explained to me and send it to the URL - http://www.esrl.noaa.gov/psd/thredds/dodsC/Datasets/ncep.reanalysis2/pressure/air.2014.nc.ascii?air[480:603][2][20:34][26:40] (as an example) the data returned will be ascii but the java netcdf library will allow me to generate a netcdf file containing that data precisely that can be written to disk. Am I correct in this ?

2) The other option is to use the URL you gave me and generate the URL with the required parameters and then use Python and pycurl and write out the netcdf file to disk.

Your input is appreciated,
Regards,
Ashwin.


On Fri, May 27, 2016 at 8:16 AM, Don Murray <don.murray@xxxxxxxx <mailto:don.murray@xxxxxxxx>> wrote:

    Hi Ashwin-

    On 5/26/16 7:12 PM, ashwinD12 . wrote:

        Don/Ryan,
                          Thanks for your response. I suppose I was
        not clear on
        my question. All other doubts have been resolved except this one.

        I am wanting to download a subset of the May 2014 file for air
        temperature programmatically(either python or java) and I
        would like to
        store this file as a netcdf file on my disk that will be read
        downstream
        by a Fortran program. The URL Ryan gave me will give ASCII
        file but I am
        wanting netcdf binary format. Regarding the URL Don gave me
        would it be
        possible to construct an OpenDap constraint expression and
        then send it
        to the RAMADDA server ?


    The subset URL I sent will return a netCDF file. You can replace
    certain parameters in the url arguments for different levels and
    regions and times, or replace the entry id with that of another
    file on the server.  You can use the pycurl module in Python to
    retrieve the file using that URL. The subset from that URL is
    attached.

    You could also use the netcdf-java library to make the subsets
    programmatically in Java, pointing to the OPeNDAP URL.  You can do
    the subsets in real world space (e.g. lat/lon, time (with a
    stride), etc) and it will convert that to the appropriate index
    space constraints.

    You cannot use the constraint expressions with netcdf-java or
    through the RAMADDA url (which in the end calls the netcdf-java
    library). I'm not sure why you would need that when you can use a
    higher level interface.

    Don

        This is the only outstanding question.

        Regards,
        Ashwin.

        On Fri, May 27, 2016 at 12:57 AM, Don Murray (NOAA Affiliate)
        <don.murray@xxxxxxxx <mailto:don.murray@xxxxxxxx>
        <mailto:don.murray@xxxxxxxx <mailto:don.murray@xxxxxxxx>>> wrote:

            Ashwin-

            We do not expose the netCDF subset service (NCSS) on the
        PSD THREDDS
            server, but the same files can be accessed through the PSD
        RAMADDA
            server, starting from here:

        
http://www.esrl.noaa.gov/psd/repository/entry/show?entryid=e570c8f9-ec09-4e89-93b4-babd5651e7a9

            If you drill down to the 2014 file, you'll get to here:

        
http://www.esrl.noaa.gov/psd/repository/entry/show?entryid=synth%3Ae570c8f9-ec09-4e89-93b4-babd5651e7a9%3AL25jZXAucmVhbmFseXNpczIvcHJlc3N1cmUvYWlyLjIwMTQubmM%3D

            where you can download the entire file, or select the
        "Subset Grid"
            option on the right side of the page.  There you can
        subset by time,
            level and region.  (however, you can't pick specific
        hours, just a
            range in this interface).

            For for month of May 2014, latitude between 5 N
            and 40 N and longitude 65 E and 100 E for the pressure level
            850 hPa, the URL would be:

        
http://www.esrl.noaa.gov/psd/repository/entry/show/air.2014_subset.nc?submit=Subset&output=data.gridsubset&entryid=synth%3Ae570c8f9-ec09-4e89-93b4-babd5651e7a9%3AL25jZXAucmVhbmFseXNpczIvcHJlc3N1cmUvYWlyLjIwMTQubmM%3D&hstride=1&area_north=40&area_west=65&area_east=100&area_south=5&calendar=gregorian&fromdate=2014-05-01%2000%3A00%3A00%20UTC&todate=2014-05-31%2018%3A00%3A00%20UTC&addlatlon=true&level=2&variable=air

            (shown at the bottom of the Grid Subset form).

            Don Murray
            --
            Don Murray
            NOAA/ESRL/PSD and CU-CIRES
            303-497-3596
        http://www.esrl.noaa.gov/psd/people/don.murray/

            On 5/26/16 12:32 PM, Ryan May wrote:

                Regarding #2, the URL from thredds to just download
        the file
                would be:

        
http://www.esrl.noaa.gov/psd/thredds/dodsC/Datasets/ncep.reanalysis2/pressure/air.2014.nc.ascii?air[480:603][2][20:34][26:40]

                You can find that in the catalog here:

        
http://www.esrl.noaa.gov/psd/thredds/catalog/Datasets/ncep.reanalysis2/pressure/catalog.html?dataset=Datasets/ncep.reanalysis2/pressure/air.2014.nc

                (The HTTPServer link.)

                Ryan


                On Thu, May 26, 2016 at 11:10 AM, ashwinD12 .
                <winash12@xxxxxxxxx <mailto:winash12@xxxxxxxxx>
        <mailto:winash12@xxxxxxxxx <mailto:winash12@xxxxxxxxx>>
                <mailto:winash12@xxxxxxxxx <mailto:winash12@xxxxxxxxx>
        <mailto:winash12@xxxxxxxxx <mailto:winash12@xxxxxxxxx>>>> wrote:

                    James,
                               What an OpenDap primer from the person
        who invented
                    OpenDap ! Thank you very much indeed. I absorbed
        all of that
                    information.

                    I had three more follow up questions(maybe related)
                    1) I presume this URL relates to air temperatures -

        
http://www.esrl.noaa.gov/psd/thredds/dodsC/Datasets/ncep.reanalysis2/pressure/air.2014.nc.ascii?air[480:603][2][20:34][26:40]

                    Why are the values negative ?

                    2) I presume I can download this file as a netcdf
        file in
                Python or
                    Java ?

                    3) Finally is there a way in the constraint
        expression that
                I can
                    add what time of day I want(00,06,12,18) ?

                    Regards,
                    Ashwin.

                    On Thu, May 26, 2016 at 8:41 PM, James Gallagher
                    <jgallagher@xxxxxxxxxxx
        <mailto:jgallagher@xxxxxxxxxxx> <mailto:jgallagher@xxxxxxxxxxx
        <mailto:jgallagher@xxxxxxxxxxx>>
                <mailto:jgallagher@xxxxxxxxxxx
        <mailto:jgallagher@xxxxxxxxxxx> <mailto:jgallagher@xxxxxxxxxxx
        <mailto:jgallagher@xxxxxxxxxxx>>>>
                wrote:



                        On May 26, 2016 at 07:06:38, ashwinD12 .
                (winash12@xxxxxxxxx <mailto:winash12@xxxxxxxxx>
        <mailto:winash12@xxxxxxxxx <mailto:winash12@xxxxxxxxx>>
                        <mailto:winash12@xxxxxxxxx
        <mailto:winash12@xxxxxxxxx> <mailto:winash12@xxxxxxxxx
        <mailto:winash12@xxxxxxxxx>>>)

                wrote:

                            Hello,
                                       I am not sure whether I will
        get any help for
                            asking such a basic question on OpenDAP
        constraint
                    expression
                            but if this is not the forum for getting
                    clarifications on
                            OpenDAP constraint expression please let
        me know the
                            appropriate forum.


                            I I have this URL
                            -
        
http://www.esrl.noaa.gov/psd/thredds/dodsC/Datasets/ncep.reanalysis2/pressure/air.2014.nc.ascii?air[408:603][2][20:34][26:40]'



                            This is the output of a program that
        fetches data from a
                            THREDDS server. I am wanting to replicate the
                    functionality by
                            coming up with that URL myself.

                            Can somebody explain to me what the values in
                    parenthesis are ?

                            This is the input I give

                            variable='air',level=850,
                            months.minmax=c(5,5),
        years.minmax=c(2014,2014),
                            lat.southnorth=c(5, 40),
        lon.westeast=c(65, 100),

                            I am wanting data for month of May 2014,
        latitude
                    between 5 N
                            and 40 N and longitude 65 E and 100 E for the
                    pressure level
                            850 hPa.

                            How does that input get translated to the
        above URL ?


                        Here’s how the URL breaks down: air.2014.nc
        <http://air.2014.nc>
                <http://air.2014.nc> .ascii ? air

                        [408:603 ] [2] [20:34] [26:40]

<http://www.esrl.noaa.gov/psd/thredds/dodsC/Datasets/ncep.reanalysis2/pressure/air.2014.nc.ascii?air[408:603][2][20:34][26:40]>

                        The dataset (which is a file in this case) is
                ‘air.2014.nc <http://air.2014.nc> <http://air.2014.nc>
                        <http://air.2014.nc>’. You’re asking the server to
                subset that

                        dataset and return just the variable ‘air’ and to
                translate the
                        result into ASCII (text). When the variable
        ‘air’ is
                extracted
                        from the data set its a four-dimensional array and
                you’re asking
                        to have those dimensions ‘sliced’ (or subset)
        so that,
                for the
                        first dimension, you see only elements 408 to
        603, for the
                        second dim only element 2, for the third
        elements 20 to
                34 and
                        elements 26 to 40 for the fourth.

                        You can look at the dataset to see how your
        request
                (level=850,
                        …) translated into those array indices by
        looking at the
                        datasets’s metadata. To do that, removed the
        .ascii and
                replace
                        it with .dds, .das and/or .info and don’t
        include the query
                        string (the part after the ‘?’). Like this:


        
http://www.esrl.noaa.gov/psd/thredds/dodsC/Datasets/ncep.reanalysis2/pressure/air.2014.nc.dds

                        Dataset {
                            Float32 level[level = 17];
                            Float32 lat[lat = 73];
                            Float32 lon[lon = 144];
                            Float64 time[time = 1460];
                            Grid {
                             ARRAY:
                                Int16 air[time = 1460][level = 17][lat
        = 73][lon
                = 144];
                             MAPS:
                                Float64 time[time = 1460];
                                Float32 level[level = 17];
                                Float32 lat[lat = 73];
                                Float32 lon[lon = 144];
                            } air;
                        }
        Datasets/ncep.reanalysis2/pressure/air.2014.nc
        <http://air.2014.nc>
                <http://air.2014.nc> <http://air.2014.nc>;



                        One thing you’ll see is that ‘air’ is a Grid -
        so it’s not
                        really a simple array, but rather a collection of
                arrays: air,
                        time, level, lat and lon. The ‘air’ array
        holds the data
                while
                        the other four hold what could be described as the
                independent
                        variables (or dimensions).

                        You can see more information about the
        variables by
                looking at
                        ‘semantic metadata’ like this:


        
http://www.esrl.noaa.gov/psd/thredds/dodsC/Datasets/ncep.reanalysis2/pressure/air.2014.nc.das

                        Attributes {
                            level {
                                String units "millibar";
                                Float32 actual_range 1000.0, 10.0;
                                String long_name "Level";
                                String positive "down";

                        …

                        Hope this helps,
                        James




                            Regards,
                            Ashwin.
        _______________________________________________
                            NOTE: All exchanges posted to Unidata
        maintained
                    email lists are
                            recorded in the Unidata inquiry tracking
        system and
                    made publicly
                            available through the web. Users who post
        to any of
                    the lists we
                            maintain are reminded to remove any personal
                    information that
                            they
                            do not want to be made public.


                            thredds mailing list
        thredds@xxxxxxxxxxxxxxxx <mailto:thredds@xxxxxxxxxxxxxxxx>
                    <mailto:thredds@xxxxxxxxxxxxxxxx
        <mailto:thredds@xxxxxxxxxxxxxxxx>>
                    <mailto:thredds@xxxxxxxxxxxxxxxx
        <mailto:thredds@xxxxxxxxxxxxxxxx>
                    <mailto:thredds@xxxxxxxxxxxxxxxx
        <mailto:thredds@xxxxxxxxxxxxxxxx>>>
                            For list information or to unsubscribe, visit:
        http://www.unidata.ucar.edu/mailing_lists/


                        --
                        James Gallagher
        jgallagher@xxxxxxxxxxx <mailto:jgallagher@xxxxxxxxxxx>
        <mailto:jgallagher@xxxxxxxxxxx <mailto:jgallagher@xxxxxxxxxxx>>
                <mailto:jgallagher@xxxxxxxxxxx
        <mailto:jgallagher@xxxxxxxxxxx> <mailto:jgallagher@xxxxxxxxxxx
        <mailto:jgallagher@xxxxxxxxxxx>>>



        _______________________________________________
                    NOTE: All exchanges posted to Unidata maintained
        email lists are
                    recorded in the Unidata inquiry tracking system
        and made
                publicly
                    available through the web.  Users who post to any
        of the
                lists we
                    maintain are reminded to remove any personal
        information
                that they
                    do not want to be made public.


                    thredds mailing list
        thredds@xxxxxxxxxxxxxxxx <mailto:thredds@xxxxxxxxxxxxxxxx>
        <mailto:thredds@xxxxxxxxxxxxxxxx
        <mailto:thredds@xxxxxxxxxxxxxxxx>>
                <mailto:thredds@xxxxxxxxxxxxxxxx
        <mailto:thredds@xxxxxxxxxxxxxxxx>
        <mailto:thredds@xxxxxxxxxxxxxxxx
        <mailto:thredds@xxxxxxxxxxxxxxxx>>>
                    For list information or to unsubscribe, visit:
        http://www.unidata.ucar.edu/mailing_lists/




                --
                Ryan May, Ph.D.
                Software Engineer
                UCAR/Unidata
                Boulder, CO


                _______________________________________________
                NOTE: All exchanges posted to Unidata maintained email
        lists are
                recorded in the Unidata inquiry tracking system and
        made publicly
                available through the web.  Users who post to any of
        the lists we
                maintain are reminded to remove any personal
        information that they
                do not want to be made public.


                thredds mailing list
        thredds@xxxxxxxxxxxxxxxx <mailto:thredds@xxxxxxxxxxxxxxxx>
        <mailto:thredds@xxxxxxxxxxxxxxxx
        <mailto:thredds@xxxxxxxxxxxxxxxx>>
                For list information or to unsubscribe,  visit:
        http://www.unidata.ucar.edu/mailing_lists/




-- Don Murray
    NOAA/ESRL/PSD and CIRES
    303-497-3596
    http://www.esrl.noaa.gov/psd/people/don.murray/




_______________________________________________
NOTE: All exchanges posted to Unidata maintained email lists are
recorded in the Unidata inquiry tracking system and made publicly
available through the web.  Users who post to any of the lists we
maintain are reminded to remove any personal information that they
do not want to be made public.


thredds mailing list
thredds@xxxxxxxxxxxxxxxx
For list information or to unsubscribe, visit:http://www.unidata.ucar.edu/mailing_lists/

  • 2016 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the thredds archives: