[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[netCDFJava #UXM-196070]: Regarding NCML Aggregating ( montly axis at fixed date )



Greetings Akshay,

NcML aggregation works by sorting the files in lexicographical order first,
and follows that for the order in which the files are aggregated. Note that
the output from the ls command is in lexicographical order, so you should
be good to go given the file naming convention you've used for your files.

Cheers,

Sean

address@hidden> wrote:

> New Client Reply: Regarding NCML Aggregating  ( montly axis at fixed date )
>
> Hi!
>
> Thank you so much Sean, its working fine. I have one doubt if I am not
> specifying dateFormatMark, are you sure will it aggregate files in right
> order ?
>
> Have a nice day
>
> Regards,
> Akshay
>
> ----- Original Message -----
> From: "Unidata netCDF Java Support" <address@hidden>
> To: address@hidden
> Cc: address@hidden
> Sent: Saturday, April 9, 2016 3:36:11 PM
> Subject: [netCDFJava #UXM-196070]: Regarding NCML Aggregating  ( montly
> axis at fixed date )
>
> Greetings Akshay,
>
> So I think I have the ncml fixed to do the right thing. Since each nc file
> had time already defined and this was a joinExsiting aggregation, there was
> actually no need to do anything with time, like extracting the date/time
> from the file name. Please try the attached ncml and see if that works for
> you.
>
> Cheers,
>
> Sean
>
>
> On Sat, Apr 9, 2016 at 9:08 AM, Sean Arms <address@hidden> wrote:
>
> > Oops, I see you did that already. Let me take a look real quick.
> >
> > Thanks!
> >
> > Sean
> >
> > On Sat, Apr 9, 2016 at 9:06 AM, Sean Arms <address@hidden> wrote:
> >
> >> Greeting Aksha,
> >>
> >> One thing I see is that the time format is not quite right - what you
> >> have is extracting year, minute, day. I'd change yyyymmdd to yyyyMMdd.
> >> However, there is something else going on here that isn't quite right.
> >> Would it be possible to get a copy of two or three of your files so
> that I
> >> can test locally? If not, the output of  `ncdump -h` for one of them
> would
> >> be helpful.
> >>
> >> Thanks!
> >>
> >> Sean
> >>
> >>
> >> address@hidden> wrote:
> >>
> >>> New Client Reply: Regarding NCML Aggregating  ( montly axis at fixed
> >>> date )
> >>>
> >>> Hi!
> >>>
> >>> Thanks for reply and for pointing what I am doing wrong, I just
> modified
> >>> ncml file, and executed, I am getting netcdf file as output but though
> >>> variable named time is created in netcdf but data its wrong, also
> timing
> >>> information like, months since 1960-01-01 00:00:00 is not added please
> go
> >>> through attachment.
> >>>
> >>> This is what I did
> >>>
> >>> akshay@las:~/Desktop$ pwd
> >>> /home/akshay/Desktop
> >>>
> >>> akshay@las:~/Desktop$ ls air/* -1
> >>> air/air_19600101.nc
> >>> air/air_19600201.nc
> >>> air/air_19600301.nc
> >>> air/air_19600401.nc
> >>> air/air_19600501.nc
> >>> air/air_19600601.nc
> >>> air/air_19600701.nc
> >>> air/air_19600801.nc
> >>> air/air_19600901.nc
> >>> air/air_19601001.nc
> >>> air/air_19601101.nc
> >>> air/air_19601201.nc
> >>>
> >>> akshay@las:~/Desktop$ cat test.ncml
> >>> <netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2
> ">
> >>> <aggregation dimName="TIME" type="joinExisting">
> >>>                 <variableAgg name="T"/>
> >>>              <scan dateFormatMark="air_#yyyymmdd"
> >>> location="/home/akshay/Desktop/air" suffix=".nc" subdirs="false" />
> >>> </aggregation>
> >>> </netcdf>
> >>>
> >>> akshay@las:~/Desktop$ java -Xmx1g -classpath netcdfAll-4.6.5.jar
> >>> ucar.nc2.dataset.NetcdfDataset -in test.ncml -out test.nc
> >>> SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
> >>> SLF4J: Defaulting to no-operation (NOP) logger implementation
> >>> SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for
> >>> further details.
> >>> NetcdfDatataset read from test.ncml write to test.nc finished=true
> >>>
> >>>
> >>> akshay@las:~/Desktop$ ncdump -h test.nc
> >>> netcdf test {
> >>> dimensions:
> >>>         LON = 120 ;
> >>>         LAT = 115 ;
> >>>         bnds = 2 ;
> >>>         TIME = 12 ;
> >>>         TIME_strlen = 20 ;
> >>> variables:
> >>>         float LON(LON) ;
> >>>                 LON:long_name = "Longitude" ;
> >>>                 LON:units = "degrees_east" ;
> >>>                 LON:axis = "X" ;
> >>>                 LON:modulo = 360.f ;
> >>>                 LON:point_spacing = "even" ;
> >>>                 LON:standard_name = "longitude" ;
> >>>         float LAT(LAT) ;
> >>>                 LAT:long_name = "Latitude" ;
> >>>                 LAT:units = "degrees_north" ;
> >>>                 LAT:axis = "Y" ;
> >>>                 LAT:point_spacing = "even" ;
> >>>                 LAT:standard_name = "latitude" ;
> >>>         double TIME_bnds(TIME, bnds) ;
> >>>         float AIR(TIME, LAT, LON) ;
> >>>                 AIR:missing_value = 32766.f ;
> >>>                 AIR:_FillValue = 32766.f ;
> >>>                 AIR:units = "degC" ;
> >>>                 AIR:long_name = "Air Temperature Monthly Mean at
> >>> Surface" ;
> >>>                 AIR:add_offset = 239.65f ;
> >>>                 AIR:scale_factor = 0.01f ;
> >>>                 AIR:Ferret_Precision_Note = "This variable written
> using
> >>> Ferret was converted from SHORT to FLOAT" ;
> >>>                 AIR:history = "From
> >>>
> http://www.esrl.noaa.gov/psd/thredds/dodsC/Datasets/coads/1degree/global/enh/air.mean.nc
> "
> >>> ;
> >>>         char TIME(TIME, TIME_strlen) ;
> >>>                 TIME:_CoordinateAxisType = "Time" ;
> >>>                 TIME:long_name = "time coordinate" ;
> >>>                 TIME:standard_name = "time" ;
> >>>
> >>> // global attributes:
> >>>                 :history = "FERRET V6.93   1-Apr-16" ;
> >>>                 :Conventions = "CF-1.6" ;
> >>> }
> >>>
> >>>
> >>> akshay@las:~/Desktop$ ncdump  test.nc | awk '/^ TIME =/{f=1}f'
> >>>  TIME =
> >>>   "1960-01-01T00:01:00Z",
> >>>   "1960-01-01T00:02:00Z",
> >>>   "1960-01-01T00:03:00Z",
> >>>   "1960-01-01T00:04:00Z",
> >>>   "1960-01-01T00:05:00Z",
> >>>   "1960-01-01T00:06:00Z",
> >>>   "1960-01-01T00:07:00Z",
> >>>   "1960-01-01T00:08:00Z",
> >>>   "1960-01-01T00:09:00Z",
> >>>   "1960-01-01T00:10:00Z",
> >>>   "1960-01-01T00:11:00Z",
> >>>   "1960-01-01T00:12:00Z" ;
> >>> }
> >>>
> >>>
> >>> akshay@las:~/Desktop$ ferret
> >>>         NOAA/PMEL TMAP
> >>>         FERRET v6.96
> >>>         Linux 2.6.32-573.7.1.el6.x86_64 64-bit - 12/02/15
> >>>          9-Apr-16 14:36
> >>>
> >>> yes? use test.nc
> >>>  **TMAP ERR: Time axis info missing or incomplete for dataset
> >>>              unrecognizable time encoding
> >>> yes? sh d
> >>>      currently SET data sets:
> >>>     1> ./test.nc  (default)
> >>>  name     title                             I         J         K
> >>>  L         M         N
> >>>  TIME_BNDS
> >>>                                            1:2       ...       ...
> >>>  1:12      ...       ...
> >>>  AIR      Air Temperature Monthly Mean at  1:120     1:115     ...
> >>>  1:12      ...       ...
> >>>  TIME     time coordinate                  ...       ...       ...
> >>>  1:12      ...       ...
> >>>
> >>> yes? sh g air
> >>>     GRID GMH2
> >>>  name       axis              # pts   start                end
> >>>  LON       LONGITUDE          120mr   20.5E                139.5E
> >>>  LAT       LATITUDE           115 r   74.5S                39.5N
> >>>  normal    Z
> >>>  TIME      T                   12 r   1                    12
> >>>  normal    E
> >>>  normal    F
> >>> yes? list time
> >>>              VARIABLE : time coordinate
> >>>              FILENAME : test.nc
> >>>              SUBSET   : 12 points (T)
> >>>  1    /  1:"1960-01-01T00:01:00Z"
> >>>  2    /  2:"1960-01-01T00:02:00Z"
> >>>  3    /  3:"1960-01-01T00:03:00Z"
> >>>  4    /  4:"1960-01-01T00:04:00Z"
> >>>  5    /  5:"1960-01-01T00:05:00Z"
> >>>  6    /  6:"1960-01-01T00:06:00Z"
> >>>  7    /  7:"1960-01-01T00:07:00Z"
> >>>  8    /  8:"1960-01-01T00:08:00Z"
> >>>  9    /  9:"1960-01-01T00:09:00Z"
> >>>  10   / 10:"1960-01-01T00:10:00Z"
> >>>  11   / 11:"1960-01-01T00:11:00Z"
> >>>  12   / 12:"1960-01-01T00:12:00Z"
> >>> yes?
> >>>
> >>>
> >>> I expect output like this, above as you can see time index are numeric
> 1
> >>> to 12, and also array time contains datetime stamp but its not correct
> >>>
> >>> yes? sh g air
> >>>     GRID GMH2
> >>>  name       axis              # pts   start                end
> >>>  LON       LONGITUDE          120mr   20.5E                139.5E
> >>>  LAT       LATITUDE           115 r   74.5S                39.5N
> >>>  normal    Z
> >>>  TIME      T                   12 r   1960-01-01           1960-12-01
> >>>  normal    E
> >>>  normal    F
> >>>
> >>>
> >>> Thanks in advance.
> >>>
> >>> Regards,
> >>> Akshay
> >>>
> >>>
> >>>
> >>> ----- Original Message -----
> >>> address@hidden>
> >>> To: address@hidden
> >>> Cc: address@hidden
> >>> Sent: Friday, April 8, 2016 10:54:57 PM
> >>> Subject: [netCDFJava #UXM-196070]: Regarding NCML Aggregating  ( montly
> >>> axis at fixed date )
> >>>
> >>> Greetings Akshay,
> >>>
> >>> I think the issue is that you are extracting the date from the filename
> >>> (good), but are rewriting the information using the NCML ouside of the
> >>> aggregation (i.e. <values start="0" increment="1" />). Can you try
> using
> >>> the same NcML you have, but remove this part of the NcML:
> >>>
> >>> <variable name="TIME" type="double" shape="TIME">
> >>>         <attribute name="units" value="months since 1960-01-01
> 00:00:00"
> >>> />
> >>>         <values start="0" increment="1" />
> >>>          <attribute name="axis" value="T" />
> >>> </variable>
> >>>
> >>> Thanks!
> >>>
> >>> Sean
> >>>
> >>>
> >>> address@hidden> wrote:
> >>>
> >>> > New Ticket: Regarding NCML Aggregating  ( montly axis at fixed date )
> >>> >
> >>> > Hi!
> >>> >
> >>> > This query is regarding time variable declaration in NCML, here my
> aim
> >>> > is to define time axis at 1th of every month that is same as that of
> >>> > date stamp in filename.
> >>> >
> >>> >
> >>> > My Input :
> >>> >
> >>> > I have list of files like below
> >>> >
> >>> > # ls -1 | head -10
> >>> > air_19600101.nc
> >>> > air_19600201.nc
> >>> > air_19600301.nc
> >>> > air_19600401.nc
> >>> > air_19600501.nc
> >>> > air_19600601.nc
> >>> > air_19600701.nc
> >>> > air_19600801.nc
> >>> > air_19600901.nc
> >>> > air_19601001.nc
> >>> >
> >>> >
> >>> > This is what I tried :
> >>> >
> >>> > Following is my ncml
> >>> >
> >>> > <netcdf xmlns="
> http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2
> >>> ">
> >>> > <variable name="TIME" type="double" shape="TIME">
> >>> >         <attribute name="units" value="months since 1960-01-01
> >>> 00:00:00" />
> >>> >         <values start="0" increment="1" />
> >>> >          <attribute name="axis" value="T" />
> >>> > </variable>
> >>> > <aggregation dimName="TIME" type="joinExisting">
> >>> >              <scan dateFormatMark="air_#yyyymmdd"
> >>> location="/path/to/AIR"
> >>> > suffix=".nc" subdirs="false" />
> >>> > </aggregation>
> >>> > </netcdf>
> >>> >
> >>> >
> >>> > This is what I received after restarting thredds data server:
> >>> >
> >>> > I use ferret for data analysis, this is how time indexes are created
> >>> >
> >>> > yes? set data "http://127.0.0.1/thredds/dodsC/test/test.nc";
> >>> > yes? sh g air
> >>> >     GRID GFP1
> >>> >  name       axis              # pts   start                end
> >>> >  LON       LONGITUDE          120mr   20.5E                139.5E
> >>> >  LAT       LATITUDE           115 r   74.5S                39.5N
> >>> >  normal    Z
> >>> >  TIME      TIME               674 r   01-JAN-1960 12:00
> 31-JAN-2016
> >>> > 12:24
> >>> >  normal    E
> >>> >  normal    F
> >>> > yes? list/l=1:10 t[gt=air]
> >>> >              VARIABLE : T
> >>> >                         axis TIME
> >>> >              FILENAME : test.nc
> >>> >              FILEPATH : http://127.0.0.1/thredds/dodsC/test/
> >>> >              SUBSET   : 10 points (TIME)
> >>> >  01-JAN-1960 /  1:  0.000
> >>> >  31-JAN-1960 /  2:  1.000
> >>> >  02-MAR-1960 /  3:  2.000
> >>> >  01-APR-1960 /  4:  3.000
> >>> >  02-MAY-1960 /  5:  4.000
> >>> >  01-JUN-1960 /  6:  5.000
> >>> >  02-JUL-1960 /  7:  6.000
> >>> >  01-AUG-1960 /  8:  7.000
> >>> >  31-AUG-1960 /  9:  8.000
> >>> >  01-OCT-1960 / 10:  9.000
> >>> >
> >>> >
> >>> > This is expected output :
> >>> >
> >>> > Here I expect time index to be created like below, at first of every
> >>> > month, or else 15th of every month
> >>> >
> >>> >  01-JAN-1960 /  1:  0.000
> >>> >  01-FEB-1960 /  2:  1.000
> >>> >  01-MAR-1960 /  3:  2.000
> >>> >  01-APR-1960 /  4:  3.000
> >>> >  01-MAY-1960 /  5:  4.000
> >>> >  01-JUN-1960 /  6:  5.000
> >>> >  01-JUL-1960 /  7:  6.000
> >>> >  01-AUG-1960 /  8:  7.000
> >>> >  01-SEP-1960 /  9:  8.000
> >>> >  01-OCT-1960 / 10:  9.000
> >>> >
> >>> >
> >>> > Hope someone can guide me
> >>> >
> >>> > Thanks in advance.
> >>> >
> >>> > Have a nice day !
> >>> >
> >>> > Regards,
> >>> > Akshay Hegde
> >>> >
> >>> >
> >>> > Ticket Details
> >>> > ===================
> >>> > Ticket ID: UXM-196070
> >>> > Department: Support netCDF Java
> >>> > Priority: Normal
> >>> > Status: Open
> >>> > Link:
> >>> >
> >>>
> https://andy.unidata.ucar.edu/esupport/staff/index.php?_m=tickets&_a=viewticket&ticketid=26875
> >>> >
> >>> >
> >>>
> >>>
> >>>
> >>> Ticket Details
> >>> ===================
> >>> Ticket ID: UXM-196070
> >>> Department: Support netCDF Java
> >>> Priority: Normal
> >>> Status: Open
> >>>
> >>>
> >>>
> >>>
> >>> Ticket Details
> >>> ===================
> >>> Ticket ID: UXM-196070
> >>> Department: Support netCDF Java
> >>> Priority: Normal
> >>> Status: Open
> >>> Link:
> >>>
> https://andy.unidata.ucar.edu/esupport/staff/index.php?_m=tickets&_a=viewticket&ticketid=26875
> >>
> >>
> >>
> >
>
>
>
> Ticket Details
> ===================
> Ticket ID: UXM-196070
> Department: Support netCDF Java
> Priority: Normal
> Status: Open
>
>
>
> Ticket Details
> ===================
> Ticket ID: UXM-196070
> Department: Support netCDF Java
> Priority: Normal
> Status: Open
> Link:
> https://andy.unidata.ucar.edu/esupport/staff/index.php?_m=tickets&_a=viewticket&ticketid=26875
>
>



Ticket Details
===================
Ticket ID: UXM-196070
Department: Support netCDF Java
Priority: Normal
Status: Open