Re: [thredds] TDS 4.6 - I still can't get GRIB aggregations to work

  • To: Roy Mendelssohn - NOAA Federal <roy.mendelssohn@xxxxxxxx>
  • Subject: Re: [thredds] TDS 4.6 - I still can't get GRIB aggregations to work
  • From: John Caron <caron@xxxxxxxx>
  • Date: Thu, 9 Apr 2015 12:52:29 -0600
Hi Roy:

java.lang.RuntimeException: SimpleDateFormat = yyyyMMddHH fails on

FNMOC_NAVGEM_0.5-degree_6-hourly_Pressure-201302.ncx3
ParseException:Unparseable date: "ssure-2013"

So it looks like the index files (ncx3) are in the data directory, so those
need to get excluded. If all of your files end in "pres", you could use:

  <collection spec="/u00/FNMOC/NAVGEM/pressure/**/.*pres$"
timePartition="directory"
 olderThan="5 min"/>


The problem is that you need to exclude the non GRIB files fron your
collection, so change your collection element to:

  <collection spec="/u00/FNMOC/NAVGEM/pressure/**/.*grib1$"
timePartition="directory"
 olderThan="5 min"/>

Note that GRIB collections no longer need the date filename parser (unless
you are using timePartition=<time duration>) so we dont need the





<collection spec="/u00/FNMOC/NAVGEM/pressure/**/US058GMET-GR1mdl.
0018_0056_00000F0..#yyyyMMddHH#_0102_000000-000000pres$"
>            timePartition="directory"    olderThan="5 min"/>



On Thu, Apr 9, 2015 at 12:34 PM, Roy Mendelssohn - NOAA Federal <
roy.mendelssohn@xxxxxxxx> wrote:

> Hi All:
>
> I have a bunch of FNMOC Grib1 pressure field.  My aggregations work fine
> in TDS 4.3.x, but I have not been able to get them to work in either 4.5 or
> now 4.6.  I am probably being brain-dead or missing something obvious, but
> here is the error message I am getting:
>
>
> > 2015-04-09T08:29:13.343 -0700 INFO  - Schedule startup scan force=always
> for 'FNMOC_NAVGEM_0.5-degree_6-hourly_Pressure' at
> 2015-04-09T08:29:16.339-0700
> > 2015-04-09T08:29:16.817 -0700 ERROR - UpdateCollectionJob.execute
> startup failed collection=FNMOC_NAVGEM_0.5-degree_6-hourly_Pressure
> > java.lang.RuntimeException: SimpleDateFormat = yyyyMMddHH fails on
> FNMOC_NAVGEM_0.5-degree_6-hourly_Pressure-201302.ncx3
> ParseException:Unparseable date: "ssure-2013"
> >       at
> ucar.nc2.units.DateFromString.getDateUsingCompleteDateFormatWithOffset(DateFromString.java:253)
> ~[cdm-4.6.0.jar:4.6.0]
> >       at
> ucar.nc2.units.DateFromString.getDateUsingDemarkatedCount(DateFromString.java:99)
> ~[cdm-4.6.0.jar:4.6.0]
> >       at
> thredds.inventory.DateExtractorFromName.getCalendarDateFromPath(DateExtractorFromName.java:86)
> ~[cdm-4.6.0.jar:4.6.0]
> >       at
> ucar.nc2.grib.collection.PartitionCollectionMutable$Partition.<init>(PartitionCollectionMutable.java:364)
> ~[grib-4.6.0.jar:4.6.0]
> >       at
> ucar.nc2.grib.collection.PartitionCollectionMutable.addPartition(PartitionCollectionMutable.java:411)
> ~[grib-4.6.0.jar:4.6.0]
> >       at
> ucar.nc2.grib.collection.GribPartitionBuilder.createPartitionedIndex(GribPartitionBuilder.java:139)
> ~[grib-4.6.0.jar:4.6.0]
> >       at
> ucar.nc2.grib.collection.GribCdmIndex.updatePartition(GribCdmIndex.java:376)
> ~[grib-4.6.0.jar:4.6.0]
> >       at
> ucar.nc2.grib.collection.GribCdmIndex.updateDirectoryCollectionRecurse(GribCdmIndex.java:465)
> ~[grib-4.6.0.jar:4.6.0]
> >       at
> ucar.nc2.grib.collection.GribCdmIndex.updateGribCollection(GribCdmIndex.java:341)
> ~[grib-4.6.0.jar:4.6.0]
> >       at
> ucar.nc2.grib.collection.GribCdmIndex.openGribCollection(GribCdmIndex.java:673)
> ~[grib-4.6.0.jar:4.6.0]
> >       at
> thredds.catalog.InvDatasetFcGrib.updateCollection(InvDatasetFcGrib.java:151)
> ~[tdcommon-4.6.0.jar:4.6.0]
> >       at
> thredds.catalog.InvDatasetFeatureCollection.checkState(InvDatasetFeatureCollection.java:309)
> ~[tdcommon-4.6.0.jar:4.6.0]
> >       at
> thredds.catalog.InvDatasetFeatureCollection.update(InvDatasetFeatureCollection.java:330)
> ~[tdcommon-4.6.0.jar:4.6.0]
> >       at
> thredds.catalog.InvDatasetFeatureCollection.sendEvent(InvDatasetFeatureCollection.java:226)
> ~[tdcommon-4.6.0.jar:4.6.0]
> >       at
> thredds.inventory.CollectionUpdater$UpdateCollectionJob.execute(CollectionUpdater.java:242)
> [cdm-4.6.0.jar:4.6.0]
> >       at org.quartz.core.JobRunShell.run(JobRunShell.java:207)
> [quartz-2.2.0.jar:?]
> >       at
> org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:560)
> [quartz-2.2.0.jar:?]
> > 2015-04-09T08:34:00.754 -0700 INFO  - FeatureCollection added =
> FeatureCollectionConfig name ='FNMOC NAVGEM 0.5-degree 6-hourly Pressure'
> collectionName='FNMOC_NAVGEM_0.5-degree_6-hourly_Pressure' type='GRIB1'
> >
>  
> spec='/u00/FNMOC/NAVGEM/pressure/**/US058GMET-GR1mdl.0018_0056_00000F0..#yyyyMMddHH#_0102_000000-000000pres$'
> >   olderThan =5 min
> >   timePartition =directory
> >   updateConfig =UpdateConfig{userDefined=true, recheckAfter='null',
> rescan='null', triggerOk=true, updateType=always}
> >   tdmConfig =UpdateConfig{userDefined=true, recheckAfter='null',
> rescan='null', triggerOk=true, updateType=always}
> >   ProtoConfig{choice=Penultimate, change='null', param='null',
> outerNcml='null', cacheAll=true}
> >   hasInnerNcml =false
> >   GribConfig{datasets=[Best, Latest], useGenType=false,
> useTableVersion=false, intvMerge=true, useCenter=false,
> filesSortIncreasing=true}
>
>
> Here is the relevant part of  the catalog:
>
> >      <featureCollection name="FNMOC NAVGEM 0.5-degree 6-hourly Pressure"
> featureType="GRIB1" harvest="true" path="FNMOC/navgem/pressure/6hr">
> >        <metadata inherited="true">
> >          <dataType>GRID</dataType>
> >           <dataFormat>GRIB-1</dataFormat>
> >           <serviceName>all</serviceName>
> >           <documentation type="summary">
> >               FNMOC 1-degree pressure fields from the NOGAPS model
> >            </documentation>
> >        </metadata>
> >
> >        <collection
> spec="/u00/FNMOC/NAVGEM/pressure/**/US058GMET-GR1mdl.0018_0056_00000F0..#yyyyMMddHH#_0102_000000-000000pres$"
> >            timePartition="directory"    olderThan="5 min"/>
> >
>
>
>
> This is my test directory structure:
>
> > root@localhost pressure]# pwd
> > /u00/FNMOC/NAVGEM/pressure
> > [root@localhost pressure]# ls
> > 201302
> >
>
>
> Here are typical files  (note some have F0FL and some F0OF which why the
> "F0..” in the time string
>
> > US058GMET-GR1mdl.0018_0056_00000F0RL2013021400_0102_000000-000000pres
> > US058GMET-GR1mdl.0018_0056_00000F0OF2013021306_0102_000000-000000pres
>
>
> Some things on the internet suggest that files have to have “.grib1” at
> the end,  Is this true?  If not, if anyone can see what my error is I sure
> would appreciate it.  This is why I haven’t upgraded.
>
> Thanks,
>
> -Roy
>
> **********************
> "The contents of this message do not reflect any position of the U.S.
> Government or NOAA."
> **********************
> Roy Mendelssohn
> Supervisory Operations Research Analyst
> NOAA/NMFS
> Environmental Research Division
> Southwest Fisheries Science Center
> ***Note new address and phone***
> 110 Shaffer Road
> Santa Cruz, CA 95060
> Phone: (831)-420-3666
> Fax: (831) 420-3980
> e-mail: Roy.Mendelssohn@xxxxxxxx www: http://www.pfeg.noaa.gov/
>
> "Old age and treachery will overcome youth and skill."
> "From those who have been given much, much will be expected"
> "the arc of the moral universe is long, but it bends toward justice" -MLK
> Jr.
>
> _______________________________________________
> thredds mailing list
> thredds@xxxxxxxxxxxxxxxx
> For list information or to unsubscribe,  visit:
> http://www.unidata.ucar.edu/mailing_lists/
  • 2015 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the thredds archives: