[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: MODIS HDF-EOS Data in the NetCDF-Java CDM and THREDDS



update stardate 2349873.432874

ok, enhance actually already works, but i ran into another problem with 
joinExisting.

So I retreated to joinNew using the kludge of extracting the date from the 
filename. 

this only works if the filenames are uniform:

<?xml version='1.0' encoding='UTF-8'?>
<netcdf xmlns='http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2'>
  <aggregation dimName='time' type='joinNew'>
    <variableAgg 
name='MODIS_Grid_16DAY_250m_500m_VI/Data_Fields/250m_16_days_NDVI' />
    <scan location='G:/work/blodgett/' suffix='.hdf' subdirs='false' 
dateFormatMark="MOD13Q1.A#yyyyDDD"/> 
  </aggregation>
</netcdf>

you also have to list the variables you want aggregated explicitly; i only did 
one, you can add the others:

    <variableAgg 
name='MODIS_Grid_16DAY_250m_500m_VI/Data_Fields/250m_16_days_NDVI' />

this form should work with the 4.3.16 release.

let me know if this is adequate for you for now, and any problems.

john

ps: use this as reference:

http://www.unidata.ucar.edu/software/netcdf/ncml/v2.2/AnnotatedSchema4.html#aggregation



On 3/26/2013 1:29 PM, Blodgett, David wrote:
> ... enhance ... enhance ... go back ... enhance ...
> 
> Love it!
> 
> Thanks John, so no easy path forward on aggregations yet, but the master 
> plan includes it?
> 
> I'm happy with that.
> 
> 
> On Tue, Mar 26, 2013 at 1:59 PM, John Caron <address@hidden 
> <mailto:address@hidden>> wrote:
> 
>     Hi David:
> 
>     I had a look at this again. I changed HdfEosModisConvention to use a
>     1D instead of a scalar time coordinate.
> 
>     However, joinExisting isnt working because the dataset needs to be
>     "enhanced" (with HdfEosModisConvention) first, before the time
>     coordinate is available. Currently joinExisting aggregation doesnt
>     do that.
> 
>     I can probably hack that in. Really I want to get the GRID feature
>     collection working, but thats going to be a few months. Aggregation
>     is old and crufty.
> 
>     Anyway, Ill let you know if i can get "enhanced" working.
> 
> 
>     John
> 
> 
> 
>     On 3/20/2013 7:50 AM, Blodgett, David wrote:
> 
>         Morning John,
> 
>         I pulled down the latest thredds war and have been testing a few
>         things
>         out. Did you get temporal aggregation working on the structured
>         data at
>         all? Any hints on how to structure the ncml for that?
> 
>         Looks like the spatial stuff is working. Fantastic.
> 
>         Thanks!
> 
>         - Dave
> 
> 
> 
>         On Mon, Mar 4, 2013 at 9:07 AM, Blodgett, David
>         <address@hidden <mailto:address@hidden>
>         <mailto:address@hidden <mailto:address@hidden>>> wrote:
> 
>              OK, we'll keep an eye out for the release, thanks so much John!
> 
>              No worries on the aggregation, the tiles are big enough
>         that I think
>              we can deal without aggregation.
> 
> 
>              On Mon, Mar 4, 2013 at 8:43 AM, John Caron
>         <address@hidden <mailto:address@hidden>
>              <mailto:address@hidden
>         <mailto:address@hidden>__>> wrote:
> 
>                  On 3/1/2013 12:03 PM, Blodgett, David wrote:
> 
>                      John,
> 
>                      Really happy you added this stuff in, what's your
>             plan for
>                      releasing it?
> 
> 
>                  going to try to release 4.3.16 in next week or two.
> 
> 
> 
>                      You think there's any chance that tiled aggregation
>             will work?
> 
> 
>                  im reluctant to recommend using whats there currently.
>         But doing
>                  it right is a challenge and wont be done soon.
> 
> 
> 
> 
>                      - Dave
> 
> 
>                      On Tue, Feb 26, 2013 at 8:30 AM, address@hidden
>             <mailto:address@hidden>
>                      <mailto:address@hidden <mailto:address@hidden>>
>             <address@hidden <mailto:address@hidden>
> 
>                      <mailto:address@hidden <mailto:address@hidden>>>
>             wrote:
> 
>                          Yes the tiling scheme is consistent for all the
>             MODIS tiles.
> 
>                          Thanks,
> 
>                          Jason Werpy
>                          Enterprise Architect
>                          Information Dynamics
>                          USGS/EROS
>                          Sioux Falls, SD 57198
>                          phone: 605-594-2723 <tel:605-594-2723>
>                          cell: 605-690-3576 <tel:605-690-3576>
>                          fax: 605-594-2530 <tel:605-594-2530>
>             address@hidden <mailto:address@hidden>
>             <mailto:address@hidden <mailto:address@hidden>>
> 
> 
>                          NOTICE: This email may contain confidential,
>             proprietary,
>                          or competition
>                          sensitive bid or proposal procurement
>                          information. Unauthorized
>                          disclosure of this information may carry
>                          criminal penalties as set forth
>                          in the Procurement Integrity Act, 41 U.S.C. 423, as
>                          amended. Further,
>                          the unauthorized disclosure of certain
>                          commercial information by civil
>                          servants may result in fines or imprisonment
>             under the
>                          Trade Secrets Act
>                          (18 U.S.C. 1905). If you have received this
>             information in
>                          error, please
>                          delete it, including all copies, and notify the
>             sender of
>                          the error
>                          immediately.
> 
> 
>                          On Feb 26, 2013, at 8:26 AM, "Blodgett, David"
>                          <address@hidden <mailto:address@hidden>
>             <mailto:address@hidden <mailto:address@hidden>>> wrote:
> 
>                              In this case, the data is 16 day from about
>                 2000 to
>                              present. So, on the order of 275 time
>                 steps. There are
>                              daily products and annual products though,
>                 in wich case,
>                              we are looking 12 to 4000+ time steps.
> 
>                              Jason, is the tile scheme you sent over
>                 used for all the
>                              different resolutions? Looking at a few of
>                 them it looks
>                              like that is the case.
> 
>                              - Dave
> 
> 
> 
>                              On Tue, Feb 26, 2013 at 7:48 AM, John Caron
>                              <address@hidden
>                 <mailto:address@hidden>
>                 <mailto:address@hidden
>                 <mailto:address@hidden>__>>
> 
>                              wrote:
> 
>                                  Heres an image of this file.
> 
>                                  questions about aggregations: how many
>                 time steps
>                                  will be in each aggregation ? how many
>                 different
>                                  aggregations (tiles?) will you serve?
> 
> 
> 
>                                  On 2/24/2013 6:55 PM, David Blodgett wrote:
> 
>                                      Sweet, thanks John. There was data
>                 in the tile,
>                                      but was hard to find... BORING, I
>                 know. Sorry
>                                      about that.
> 
>                                      Here's a tile over the great lakes
>                 to give you a
>                                      coast to compare locations to.
> 
>                 
> http://e4ftl01.cr.usgs.gov/__MOLT/MOD13Q1.005/2000.03.05/__BROWSE.MOD13A1.A2000065.__h11v04.005.2008238024700.1.jpg
>                 
> <http://e4ftl01.cr.usgs.gov/MOLT/MOD13Q1.005/2000.03.05/BROWSE.MOD13A1.A2000065.h11v04.005.2008238024700.1.jpg>
>                 
> http://e4ftl01.cr.usgs.gov/__MOLT/MOD13Q1.005/2000.03.05/__MOD13Q1.A2000065.h11v04.005.__2008238031620.hdf
>                 
> <http://e4ftl01.cr.usgs.gov/MOLT/MOD13Q1.005/2000.03.05/MOD13Q1.A2000065.h11v04.005.2008238031620.hdf>
>                 
> http://e4ftl01.cr.usgs.gov/__MOLT/MOD13Q1.005/2000.03.05/__MOD13Q1.A2000065.h11v04.005.__2008238031620.hdf.xml
>                 
> <http://e4ftl01.cr.usgs.gov/MOLT/MOD13Q1.005/2000.03.05/MOD13Q1.A2000065.h11v04.005.2008238031620.hdf.xml>
> 
>                                      - Dave
> 
>                                      On Feb 24, 2013, at 6:33 PM, John
>                 Caron wrote:
> 
>                                          Hi:
> 
>                                          Im hacking away at this, but
>                 the datafiles i
>                                          downloaded _seem_ to have
>                 nothing but missing
>                                          values. Or im doing something
>                 wrong. Can you
>                                          send me a file that you know
>                 has real data in
>                                          it. thanks
> 
>                                          John
> 
>                                          On 2/24/2013 9:22 AM, David
>                 Blodgett wrote:
> 
>                                              Hi John,
> 
>                                              Actually, I was assuming
>                 that the
>                                              sinusoidal projection was
>                 in there, doh!
>                                              That was based on
>                 gdal_translate creating
>                                              a geotiff with this
>                 projection info:
> 
>                                              PROJCS["unnamed",
>                                                   GEOGCS["Unknown datum
>                 based upon the
>                                              custom spheroid",
>                                             
>                   DATUM["Not_specified_based_on___custom_spheroid",
>                                                           SPHEROID["Custom
>                                              spheroid",6371007.181,0]],
>                                                       PRIMEM["Greenwich",0],
>                                                     
>                   UNIT["degree",0.__0174532925199433]],
>                                                   PROJECTION["Sinusoidal"],
>                                                 
>                   PARAMETER["longitude_of___center",0],
>                                                 
>                   PARAMETER["false_easting",0],
>                                               PARAMETER["false_northing",0],
>                                                   UNIT["metre",1,
>                                               AUTHORITY["EPSG","9001"]]]
> 
>                                              There's a georeferencing
>                 section here
>                                             
>                 (http://www.gdal.org/frmt___hdf4.html
>                 <http://www.gdal.org/frmt_hdf4.html>) that
>                                              has a bit more info about
>                 gdal's
>                                              implementation. Might be
>                 something
>                                              helpful there? Looks like
>                 the ODL text
>                                              you point to is where its
>                 coming from.
> 
>                                              4 and 5 are questions for
>                 Jason and maybe
>                                              Jordan. Guys?
> 
>                                              - Dave
> 
> 
> 
> 
> 
>                                              On Feb 23, 2013, at 1:22
>                 PM, John Caron
>                                              wrote:
> 
> 
>                                                  On 2/22/2013 2:07 PM,
>                 David Blodgett
>                                                  wrote:
> 
>                                                      Welcome back John.
>                 Hope you had a
>                                                      good trip!
> 
>                                                  thanks!
> 
>                                                      I actually just
>                 spent a bunch of
>                                                      time with this the
>                 other day and have
>                                                      been meaning to
>                 summarize what
>                                                      I'm finding.
> 
>                                                      The first issue we
>                 are running
>                                                      into is your #1
>                 below. In this
>                                                      case there
>                                                      is an external xml
>                 file that
>                                                      contains a lot of
>                 metadata, including
>                                                      information about
>                 the projection
>                                                      (sinusoidal), which
>                 is currently
>                                                      unsupported by
>                 NetCDF-Java?
> 
>                                                  1) where do you get
>                 this xml file?
> 
>                                                  2) im looking at
> 
>                                                 
>                 MOD13Q1.A2000065.h00v08.005.__2008238080422.hdf
>                                                 
>                 MOD13Q1.A2000065.h00v08.005.__2008238080422.hdf.xml
> 
>                                                  in your union
>                 subdirectory. The xml
>                                                  file does not seem to
>                 have projection
>                                                  info in it.
> 
>                                                  3) the internal ODL text
>                                                  (StructMetadata) has
> 
>                                                  ...
>                                                 
>                 UpperLeftPointMtrs=(-20015109.__354000,1111950.519667)
>                                                 
>                 LowerRightMtrs=(-18903158.__834333,-0.000000)
>                                                  Projection=GCTP_SNSOID
>                                                 
>                 ProjParams=(6371007.181000,0,__0,0,0,0,0,0,0,0,0,0,0)
>                                                                 
>                 SphereCode=-1
> 
>                                                  ...
> 
>                                                  which may be enough to
>                 calculate the
>                                                  lat/lon values. Since
>                 this is the
>                                                  best choice, lets
>                 investigate it
>                                                  before we go down any
>                 other paths.
> 
>                                                  4) Can anyone track
>                 down the code in
>                                                  the hyrax handler that
>                 does this? Or
>                                                  does the hyrax handler
>                 actually link
>                                                  in the hds-eos libraries ??
> 
>                                                  5) how confident are
>                 you that all the
>                                                  files that you want to
>                 aggregate have
>                                                  the same lat/lon
>                 geolocation ?
> 
> 
>                                                  John
> 
> 
> 
> 
> 
> 
> 
>