Re: [thredds] TDS 4.6 - I still can't get GRIB aggregations to work

  • To: John Caron <caron@xxxxxxxx>
  • Subject: Re: [thredds] TDS 4.6 - I still can't get GRIB aggregations to work
  • From: Roy Mendelssohn - NOAA Federal <roy.mendelssohn@xxxxxxxx>
  • Date: Thu, 9 Apr 2015 14:30:04 -0700
Yes.  I have to give a password.  Also I have tried launching with -showOnly 
and it shows the featureCollections but doesn’t exit.

-Roy

> On Apr 9, 2015, at 2:21 PM, John Caron <caron@xxxxxxxx> wrote:
> 
> The tdm is already in a seperate process from the tds, and is typically 
> started from some batch file or shell script.
> 
> To put it into the background after starting, from the shell I started it in:
> 
> ^Z
> bg
> 
> One can also start it up with the usual "&" 
> If you need to enter the password, then firsst way probably better.
> 
> Is that what you are asking?
> 
> On Thu, Apr 9, 2015 at 1:00 PM, Roy Mendelssohn - NOAA Federal 
> <roy.mendelssohn@xxxxxxxx> wrote:
> Thanks.  I was just writing that it was unclear what I don’t need to have.  
> now it is.  I would prefer having the ncx3 and gbx9 files in a different 
> directory, can you point me to where the information to do so is.
> 
> While I have you there doesn’t seem to be a way to get the tdm to run in the 
> background successfully.  Am I missing something?
> 
> -Roy
> 
> > On Apr 9, 2015, at 11:57 AM, John Caron <caron@xxxxxxxx> wrote:
> >
> > Sorry, lets try that again:
> >
> > Hi Roy:
> >
> > java.lang.RuntimeException: SimpleDateFormat = yyyyMMddHH fails on 
> > FNMOC_NAVGEM_0.5-degree_6-hourly_Pressure-201302.ncx3 
> > ParseException:Unparseable date: "ssure-2013"
> >
> > So it looks like the index files (ncx3) are in the data directory, and 
> > those need to get excluded. If all of your files end in "pres", you could 
> > use:
> >
> >   <collection spec="/u00/FNMOC/NAVGEM/pressure/**/.*pres$" 
> > timePartition="directory"  olderThan="5 min"/>
> >
> > If not all of your files end in "pres", then we need to get more clever 
> > with the regexp.
> >
> > Note that GRIB collections no longer need the date filename parser (unless 
> > you are using timePartition=<time duration>) so we dont need the # 
> > rigamorole:
> >
> >    US058GMET-GR1mdl.0018_0056_00000F0..#yyyyMMddHH#_0102_000000-000000pres$
> >
> > So its a litttle easier to write the regexp that will get just the files 
> > you want.
> >
> > Another thing one could do is to place the index files in a seperate 
> > directory. But I assume you have chosen to keep the indexes with the data?
> >
> > John
> >
> >
> >
> >
> >
> > On Thu, Apr 9, 2015 at 12:34 PM, Roy Mendelssohn - NOAA Federal 
> > <roy.mendelssohn@xxxxxxxx> wrote:
> > Hi All:
> >
> > I have a bunch of FNMOC Grib1 pressure field.  My aggregations work fine in 
> > TDS 4.3.x, but I have not been able to get them to work in either 4.5 or 
> > now 4.6.  I am probably being brain-dead or missing something obvious, but 
> > here is the error message I am getting:
> >
> >
> > > 2015-04-09T08:29:13.343 -0700 INFO  - Schedule startup scan force=always 
> > > for 'FNMOC_NAVGEM_0.5-degree_6-hourly_Pressure' at 
> > > 2015-04-09T08:29:16.339-0700
> > > 2015-04-09T08:29:16.817 -0700 ERROR - UpdateCollectionJob.execute startup 
> > > failed collection=FNMOC_NAVGEM_0.5-degree_6-hourly_Pressure
> > > java.lang.RuntimeException: SimpleDateFormat = yyyyMMddHH fails on 
> > > FNMOC_NAVGEM_0.5-degree_6-hourly_Pressure-201302.ncx3 
> > > ParseException:Unparseable date: "ssure-2013"
> > >       at 
> > > ucar.nc2.units.DateFromString.getDateUsingCompleteDateFormatWithOffset(DateFromString.java:253)
> > >  ~[cdm-4.6.0.jar:4.6.0]
> > >       at 
> > > ucar.nc2.units.DateFromString.getDateUsingDemarkatedCount(DateFromString.java:99)
> > >  ~[cdm-4.6.0.jar:4.6.0]
> > >       at 
> > > thredds.inventory.DateExtractorFromName.getCalendarDateFromPath(DateExtractorFromName.java:86)
> > >  ~[cdm-4.6.0.jar:4.6.0]
> > >       at 
> > > ucar.nc2.grib.collection.PartitionCollectionMutable$Partition.<init>(PartitionCollectionMutable.java:364)
> > >  ~[grib-4.6.0.jar:4.6.0]
> > >       at 
> > > ucar.nc2.grib.collection.PartitionCollectionMutable.addPartition(PartitionCollectionMutable.java:411)
> > >  ~[grib-4.6.0.jar:4.6.0]
> > >       at 
> > > ucar.nc2.grib.collection.GribPartitionBuilder.createPartitionedIndex(GribPartitionBuilder.java:139)
> > >  ~[grib-4.6.0.jar:4.6.0]
> > >       at 
> > > ucar.nc2.grib.collection.GribCdmIndex.updatePartition(GribCdmIndex.java:376)
> > >  ~[grib-4.6.0.jar:4.6.0]
> > >       at 
> > > ucar.nc2.grib.collection.GribCdmIndex.updateDirectoryCollectionRecurse(GribCdmIndex.java:465)
> > >  ~[grib-4.6.0.jar:4.6.0]
> > >       at 
> > > ucar.nc2.grib.collection.GribCdmIndex.updateGribCollection(GribCdmIndex.java:341)
> > >  ~[grib-4.6.0.jar:4.6.0]
> > >       at 
> > > ucar.nc2.grib.collection.GribCdmIndex.openGribCollection(GribCdmIndex.java:673)
> > >  ~[grib-4.6.0.jar:4.6.0]
> > >       at 
> > > thredds.catalog.InvDatasetFcGrib.updateCollection(InvDatasetFcGrib.java:151)
> > >  ~[tdcommon-4.6.0.jar:4.6.0]
> > >       at 
> > > thredds.catalog.InvDatasetFeatureCollection.checkState(InvDatasetFeatureCollection.java:309)
> > >  ~[tdcommon-4.6.0.jar:4.6.0]
> > >       at 
> > > thredds.catalog.InvDatasetFeatureCollection.update(InvDatasetFeatureCollection.java:330)
> > >  ~[tdcommon-4.6.0.jar:4.6.0]
> > >       at 
> > > thredds.catalog.InvDatasetFeatureCollection.sendEvent(InvDatasetFeatureCollection.java:226)
> > >  ~[tdcommon-4.6.0.jar:4.6.0]
> > >       at 
> > > thredds.inventory.CollectionUpdater$UpdateCollectionJob.execute(CollectionUpdater.java:242)
> > >  [cdm-4.6.0.jar:4.6.0]
> > >       at org.quartz.core.JobRunShell.run(JobRunShell.java:207) 
> > > [quartz-2.2.0.jar:?]
> > >       at 
> > > org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:560)
> > >  [quartz-2.2.0.jar:?]
> > > 2015-04-09T08:34:00.754 -0700 INFO  - FeatureCollection added = 
> > > FeatureCollectionConfig name ='FNMOC NAVGEM 0.5-degree 6-hourly Pressure' 
> > > collectionName='FNMOC_NAVGEM_0.5-degree_6-hourly_Pressure' type='GRIB1'
> > >   
> > > spec='/u00/FNMOC/NAVGEM/pressure/**/US058GMET-GR1mdl.0018_0056_00000F0..#yyyyMMddHH#_0102_000000-000000pres$'
> > >   olderThan =5 min
> > >   timePartition =directory
> > >   updateConfig =UpdateConfig{userDefined=true, recheckAfter='null', 
> > > rescan='null', triggerOk=true, updateType=always}
> > >   tdmConfig =UpdateConfig{userDefined=true, recheckAfter='null', 
> > > rescan='null', triggerOk=true, updateType=always}
> > >   ProtoConfig{choice=Penultimate, change='null', param='null', 
> > > outerNcml='null', cacheAll=true}
> > >   hasInnerNcml =false
> > >   GribConfig{datasets=[Best, Latest], useGenType=false, 
> > > useTableVersion=false, intvMerge=true, useCenter=false, 
> > > filesSortIncreasing=true}
> >
> >
> > Here is the relevant part of  the catalog:
> >
> > >      <featureCollection name="FNMOC NAVGEM 0.5-degree 6-hourly Pressure" 
> > > featureType="GRIB1" harvest="true" path="FNMOC/navgem/pressure/6hr">
> > >        <metadata inherited="true">
> > >          <dataType>GRID</dataType>
> > >           <dataFormat>GRIB-1</dataFormat>
> > >           <serviceName>all</serviceName>
> > >           <documentation type="summary">
> > >               FNMOC 1-degree pressure fields from the NOGAPS model
> > >            </documentation>
> > >        </metadata>
> > >
> > >        <collection 
> > > spec="/u00/FNMOC/NAVGEM/pressure/**/US058GMET-GR1mdl.0018_0056_00000F0..#yyyyMMddHH#_0102_000000-000000pres$"
> > >            timePartition="directory"    olderThan="5 min"/>
> > >
> >
> >
> >
> > This is my test directory structure:
> >
> > > root@localhost pressure]# pwd
> > > /u00/FNMOC/NAVGEM/pressure
> > > [root@localhost pressure]# ls
> > > 201302
> > >
> >
> >
> > Here are typical files  (note some have F0FL and some F0OF which why the 
> > "F0..” in the time string
> >
> > > US058GMET-GR1mdl.0018_0056_00000F0RL2013021400_0102_000000-000000pres
> > > US058GMET-GR1mdl.0018_0056_00000F0OF2013021306_0102_000000-000000pres
> >
> >
> > Some things on the internet suggest that files have to have “.grib1” at the 
> > end,  Is this true?  If not, if anyone can see what my error is I sure 
> > would appreciate it.  This is why I haven’t upgraded.
> >
> > Thanks,
> >
> > -Roy
> >
> > **********************
> > "The contents of this message do not reflect any position of the U.S. 
> > Government or NOAA."
> > **********************
> > Roy Mendelssohn
> > Supervisory Operations Research Analyst
> > NOAA/NMFS
> > Environmental Research Division
> > Southwest Fisheries Science Center
> > ***Note new address and phone***
> > 110 Shaffer Road
> > Santa Cruz, CA 95060
> > Phone: (831)-420-3666
> > Fax: (831) 420-3980
> > e-mail: Roy.Mendelssohn@xxxxxxxx www: http://www.pfeg.noaa.gov/
> >
> > "Old age and treachery will overcome youth and skill."
> > "From those who have been given much, much will be expected"
> > "the arc of the moral universe is long, but it bends toward justice" -MLK 
> > Jr.
> >
> > _______________________________________________
> > thredds mailing list
> > thredds@xxxxxxxxxxxxxxxx
> > For list information or to unsubscribe,  visit: 
> > http://www.unidata.ucar.edu/mailing_lists/
> >
> 
> **********************
> "The contents of this message do not reflect any position of the U.S. 
> Government or NOAA."
> **********************
> Roy Mendelssohn
> Supervisory Operations Research Analyst
> NOAA/NMFS
> Environmental Research Division
> Southwest Fisheries Science Center
> ***Note new address and phone***
> 110 Shaffer Road
> Santa Cruz, CA 95060
> Phone: (831)-420-3666
> Fax: (831) 420-3980
> e-mail: Roy.Mendelssohn@xxxxxxxx www: http://www.pfeg.noaa.gov/
> 
> "Old age and treachery will overcome youth and skill."
> "From those who have been given much, much will be expected"
> "the arc of the moral universe is long, but it bends toward justice" -MLK Jr.
> 
> 

**********************
"The contents of this message do not reflect any position of the U.S. 
Government or NOAA."
**********************
Roy Mendelssohn
Supervisory Operations Research Analyst
NOAA/NMFS
Environmental Research Division
Southwest Fisheries Science Center
***Note new address and phone***
110 Shaffer Road
Santa Cruz, CA 95060
Phone: (831)-420-3666
Fax: (831) 420-3980
e-mail: Roy.Mendelssohn@xxxxxxxx www: http://www.pfeg.noaa.gov/

"Old age and treachery will overcome youth and skill."
"From those who have been given much, much will be expected" 
"the arc of the moral universe is long, but it bends toward justice" -MLK Jr.



  • 2015 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the thredds archives: