Re: [thredds] Time Partition Syntax

  • To: Kevin Manross <manross@xxxxxxxx>
  • Subject: Re: [thredds] Time Partition Syntax
  • From: Dan Swank - NOAA Affiliate <dan.swank@xxxxxxxx>
  • Date: Wed, 1 May 2013 09:49:34 -0400
Just a quick plug...

Been closely working with Jahn Caron (up until he left for EGU), and
trying to get featureCollections working for all the major NCDC-NOMADS
 Archive datasets.  Both Grib1 & 2 ~ With moderate success.
Alot (but not all) of the issues I found were due to problems on our
own file system. Not permissions, but rather
zero-byte and/or corrupt/damaged GRIB records wrecking havoc
with the GRIB scanner and cause it to fail to create the master
time partition / index files.

  Caron added some modifications in the last few .wars to make
the GRIB scanner more tolerant, but as you mentioned there are still
lingering issues with it.

The "too many open files" sounds like you may need to adjust
an OS or JAVA env setting.   Not an expert on this and perhaps
someone else can chime in.  In short, it may not necessarily be
related to the featureCollectionScan.

     -Dan
   NCDC-NOMADS project




On Mon, Apr 29, 2013 at 4:17 PM, Kevin Manross <manross@xxxxxxxx> wrote:

>
> Here's an update on attempting a Time Partition for model analyses that
> are in GRIB1:
>
> I have created a local data structure for testing where I can write to the
> file directories (until I can get permission to write to the file
> directories of our main archive).  The structure is as follows:
>
> /glade/nwsc_scratch/manross/ds083.2/grib1/yyyy/yyyy.MM/fnl_yyyyMMdd_HH_00_c
>
>
> When I found the message thread :
> http://www.unidata.ucar.edu/support/help/MailArchives/thredds/msg01820.html
>
> I took the advice mentioned therein and grabbed the 4.3.17 version of the
> thredds war, and set my catalog.xml config as follows:
>
> <?xml version="1.0" encoding="UTF-8"?>
> <catalog name="THREDDS Server Default Catalog : You must change this to
> fit your server!"
>         xmlns=
> "http://www.unidata.ucar.edu/namespaces/thredds/InvCatalog/v1.0";<http://www.unidata.ucar.edu/namespaces/thredds/InvCatalog/v1.0>
>         
> xmlns:xlink="http://www.w3.org/1999/xlink";<http://www.w3.org/1999/xlink>
> >
>
>   <service name="all" base="" serviceType="compound">
>     <service name="odap" serviceType="OpenDAP" base="/thredds/dodsC/" />
>     <service name="http" serviceType="HTTPServer"
> base="/thredds/fileServer/" />
>     <!--service name="wcs" serviceType="WCS" base="/thredds/wcs/" /-->
>     <service name="wms" serviceType="WMS" base="/thredds/wms/" />
>     <service name="ncss" serviceType="NetcdfSubset"
> base="/thredds/ncss/grid/" />
>   </service>
>
>   <!--datasetRoot path="FNL" location="content/ds083.2" /-->
>
>   <featureCollection name="LOCAL_FNL"
>                      path="LOCALFNL"
>                      harvest="true"
>                      featureType="GRIB">
>
>
>     <metadata inherited="true">
>       <serviceName>all</serviceName>
>       <documentation type="summary">LOCAL FNLS FOR TESING
>       </documentation>
>       <dataFormat>GRIB-1</dataFormat>
>     </metadata>
>
>     <collection
>         spec="/glade/nwsc_scratch/manross/ds083.2/grib1/**/fnl_.*_c$"
>         timePartition="1 day"
>         dateFormatMark="#fnl_#yyyyMMdd_HH"
>
>         />
>
>     <update startup="true" rescan="0 0 0 * * ? *"/>
>     <gribConfig datasetTypes="Best Files"/>
>
>   </featureCollection>
> </catalog>
>
> Upon restarting tomcat, the directory was scanned and I was met with
> numerous "Too many open files" errors:
>
> 2013-04-29T13:12:17.795 -0600 [     36341][        ] ERROR - fc.LOCAL_FNL
> - Grib1CollectionBuilder LOCAL_FNL-20130416 : reading/Creating gbx9 index
> for file
> /glade/nwsc_scratch/manross/ds083.2/grib1/2013/2013.04/fnl_20130416_18_00_c
> failed
> java.io.FileNotFoundException:
> /usr/share/tomcat6/content/thredds/cache/grib/glade-nwsc_scratch-manross-ds083.2-grib1-2013-2013.04-fnl_20130416_18_00_c.gbx9
> (Too many open files)
>     at java.io.FileOutputStream.open(Native Method)
>     at java.io.FileOutputStream.<init>(FileOutputStream.java:209)
>     at java.io.FileOutputStream.<init>(FileOutputStream.java:160)
>     at ucar.nc2.grib.grib1.Grib1Index.makeIndex(Grib1Index.java:199)
>     at
> ucar.nc2.grib.GribIndex.readOrCreateIndexFromSingleFile(GribIndex.java:143)
>     at
> ucar.nc2.grib.grib1.Grib1CollectionBuilder.makeAggregatedGroups(Grib1CollectionBuilder.java:469)
>     at
> ucar.nc2.grib.grib1.Grib1CollectionBuilder.createIndex(Grib1CollectionBuilder.java:434)
>     at
> ucar.nc2.grib.grib1.Grib1CollectionBuilder.readOrCreateIndex(Grib1CollectionBuilder.java:167)
>     at
> ucar.nc2.grib.grib1.Grib1CollectionBuilder.factory(Grib1CollectionBuilder.java:85)
>     at ucar.nc2.grib.GribCollection.factory(GribCollection.java:177)
>     at
> ucar.nc2.grib.TimePartition$Partition.makeGribCollection(TimePartition.java:172)
>     at
> ucar.nc2.grib.grib1.Grib1TimePartitionBuilder.createPartitionedIndex(Grib1TimePartitionBuilder.java:163)
>     at
> ucar.nc2.grib.grib1.Grib1TimePartitionBuilder.readOrCreateIndex(Grib1TimePartitionBuilder.java:128)
>
>     at
> ucar.nc2.grib.grib1.Grib1TimePartitionBuilder.factory(Grib1TimePartitionBuilder.java:73)
>     at ucar.nc2.grib.TimePartition.factory(TimePartition.java:96)
>     at
> thredds.catalog.InvDatasetFcGrib.updateCollection(InvDatasetFcGrib.java:224)
>     at thredds.catalog.InvDatasetFcGrib.update(InvDatasetFcGrib.java:163)
>     at
> thredds.catalog.InvDatasetFeatureCollection.handleCollectionEvent(InvDatasetFeatureCollection.java:236)
>
>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>     at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>     at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>     at java.lang.reflect.Method.invoke(Method.java:616)
>     at ucar.nc2.util.ListenerManager.sendEvent(ListenerManager.java:177)
>     at
> thredds.inventory.CollectionManagerAbstract.sendEvent(CollectionManagerAbstract.java:213)
>     at
> thredds.inventory.MFileCollectionManager.scan(MFileCollectionManager.java:392)
>     at
> thredds.inventory.CollectionUpdater$UpdateCollectionJob.execute(CollectionUpdater.java:287)
>     at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
>     at
> org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557)
> 2013-04-29T13:12:17.796 -0600 [     36342][        ] ERROR - fc.LOCAL_FNL
> -  Failed to open partition LOCAL_FNL-20130416
> java.io.FileNotFoundException:
> /usr/share/tomcat6/content/thredds/cache/grib/glade-nwsc_scratch-manross-ds083.2-grib1-LOCAL_FNL-20130416.ncx
> (Too many open files)
>     at java.io.RandomAccessFile.open(Native Method)
>     at java.io.RandomAccessFile.<init>(RandomAccessFile.java:233)
>     at java.io.RandomAccessFile.<init>(RandomAccessFile.java:118)
>     at ucar.unidata.io.RandomAccessFile.<init>(RandomAccessFile.java:277)
>     at ucar.unidata.io.RandomAccessFile.<init>(RandomAccessFile.java:259)
>     at
> ucar.nc2.grib.grib1.Grib1CollectionBuilder.createIndex(Grib1CollectionBuilder.java:569)
>     at
> ucar.nc2.grib.grib1.Grib1CollectionBuilder.createIndex(Grib1CollectionBuilder.java:435)
>     at
> ucar.nc2.grib.grib1.Grib1CollectionBuilder.readOrCreateIndex(Grib1CollectionBuilder.java:167)
>     at
> ucar.nc2.grib.grib1.Grib1CollectionBuilder.factory(Grib1CollectionBuilder.java:85)
>     at ucar.nc2.grib.GribCollection.factory(GribCollection.java:177)
>     at
> ucar.nc2.grib.TimePartition$Partition.makeGribCollection(TimePartition.java:172)
>     at
> ucar.nc2.grib.grib1.Grib1TimePartitionBuilder.createPartitionedIndex(Grib1TimePartitionBuilder.java:163)
>     at
> ucar.nc2.grib.grib1.Grib1TimePartitionBuilder.readOrCreateIndex(Grib1TimePartitionBuilder.java:128)
>
>     at
> ucar.nc2.grib.grib1.Grib1TimePartitionBuilder.factory(Grib1TimePartitionBuilder.java:73)
>     at ucar.nc2.grib.TimePartition.factory(TimePartition.java:96)
>     at
> thredds.catalog.InvDatasetFcGrib.updateCollection(InvDatasetFcGrib.java:224)
>     at thredds.catalog.InvDatasetFcGrib.update(InvDatasetFcGrib.java:163)
>     at
> thredds.catalog.InvDatasetFeatureCollection.handleCollectionEvent(InvDatasetFeatureCollection.java:236)
>
>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>     at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>     at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>     at java.lang.reflect.Method.invoke(Method.java:616)
>     at ucar.nc2.util.ListenerManager.sendEvent(ListenerManager.java:177)
>     at
> thredds.inventory.CollectionManagerAbstract.sendEvent(CollectionManagerAbstract.java:213)
>     at
> thredds.inventory.MFileCollectionManager.scan(MFileCollectionManager.java:392)
>     at
> thredds.inventory.CollectionUpdater$UpdateCollectionJob.execute(CollectionUpdater.java:287)
>     at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
>     at
> org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557)
> 2013-04-29T13:12:17.803 -0600 [     36349][        ] ERROR - fc.LOCAL_FNL
> - Fail to create/update collection
> java.io.FileNotFoundException:
> /glade/nwsc_scratch/manross/ds083.2/grib1/LOCAL_FNL-20120605.ncx (Too many
> open files)
>     at java.io.RandomAccessFile.open(Native Method)
>     at java.io.RandomAccessFile.<init>(RandomAccessFile.java:233)
>     at java.io.RandomAccessFile.<init>(RandomAccessFile.java:118)
>     at ucar.unidata.io.RandomAccessFile.<init>(RandomAccessFile.java:277)
>     at ucar.unidata.io.RandomAccessFile.<init>(RandomAccessFile.java:259)
>     at
> ucar.nc2.grib.grib1.Grib1CollectionBuilder.readIndex(Grib1CollectionBuilder.java:196)
>     at
> ucar.nc2.grib.grib1.Grib1CollectionBuilder.readOrCreateIndex(Grib1CollectionBuilder.java:163)
>     at
> ucar.nc2.grib.grib1.Grib1CollectionBuilder.factory(Grib1CollectionBuilder.java:85)
>     at ucar.nc2.grib.GribCollection.factory(GribCollection.java:177)
>     at
> ucar.nc2.grib.TimePartition$Partition.makeGribCollection(TimePartition.java:172)
>     at
> ucar.nc2.grib.grib1.Grib1TimePartitionBuilder.checkPartitions(Grib1TimePartitionBuilder.java:215)
>     at
> ucar.nc2.grib.grib1.Grib1TimePartitionBuilder.createPartitionedIndex(Grib1TimePartitionBuilder.java:187)
>     at
> ucar.nc2.grib.grib1.Grib1TimePartitionBuilder.readOrCreateIndex(Grib1TimePartitionBuilder.java:128)
>
>     at
> ucar.nc2.grib.grib1.Grib1TimePartitionBuilder.factory(Grib1TimePartitionBuilder.java:73)
>     at ucar.nc2.grib.TimePartition.factory(TimePartition.java:96)
>     at
> thredds.catalog.InvDatasetFcGrib.updateCollection(InvDatasetFcGrib.java:224)
>     at thredds.catalog.InvDatasetFcGrib.update(InvDatasetFcGrib.java:163)
>     at
> thredds.catalog.InvDatasetFeatureCollection.handleCollectionEvent(InvDatasetFeatureCollection.java:236)
>
>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>     at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>     at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>     at java.lang.reflect.Method.invoke(Method.java:616)
>     at ucar.nc2.util.ListenerManager.sendEvent(ListenerManager.java:177)
>     at
> thredds.inventory.CollectionManagerAbstract.sendEvent(CollectionManagerAbstract.java:213)
>     at
> thredds.inventory.MFileCollectionManager.scan(MFileCollectionManager.java:392)
>     at
> thredds.inventory.CollectionUpdater$UpdateCollectionJob.execute(CollectionUpdater.java:287)
>     at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
>     at
> org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557)
>
>
> So I pared back the dataset from 3.5 years worth of 4 daily files to just
> the first month (still 4 files daily) of 2010-2013.  This eliminated the
> "Too many open files" errors, and created the respective *.gbx9 and .ncx
> files, but threw a cast error:
>
> 2013-04-29T13:40:04.188 -0600 [    108434][        ] INFO  - fc.LOCAL_FNL
> - fnl_20130131_00_00_c: createIndex
> /glade/nwsc_scratch/manross/ds083.2/grib1.1/2013/2013.01/fnl_20130131_00_00_c.ncx
> 2013-04-29T13:40:04.351 -0600 [    108597][        ] INFO  - fc.LOCAL_FNL
> - fnl_20130131_06_00_c: createIndex
> /glade/nwsc_scratch/manross/ds083.2/grib1.1/2013/2013.01/fnl_20130131_06_00_c.ncx
> 2013-04-29T13:40:04.508 -0600 [    108754][        ] INFO  - fc.LOCAL_FNL
> - fnl_20130131_12_00_c: createIndex
> /glade/nwsc_scratch/manross/ds083.2/grib1.1/2013/2013.01/fnl_20130131_12_00_c.ncx
> 2013-04-29T13:40:04.649 -0600 [    108895][        ] INFO  - fc.LOCAL_FNL
> - fnl_20130131_18_00_c: createIndex
> /glade/nwsc_scratch/manross/ds083.2/grib1.1/2013/2013.01/fnl_20130131_18_00_c.ncx
> 2013-04-29T13:40:06.936 -0600 [    111182][        ] ERROR - fc.LOCAL_FNL
> - Fail to create/update collection
> java.lang.ClassCastException: ucar.nc2.grib.GribCollection$VariableIndex
> cannot be cast to ucar.nc2.grib.TimePartition$VariableIndexPartitioned
>     at
> ucar.nc2.grib.grib1.Grib1TimePartitionBuilder.writeGroupProto(Grib1TimePartitionBuilder.java:445)
>     at
> ucar.nc2.grib.grib1.Grib1TimePartitionBuilder.writeIndex(Grib1TimePartitionBuilder.java:413)
>     at
> ucar.nc2.grib.grib1.Grib1TimePartitionBuilder.createPartitionedIndex(Grib1TimePartitionBuilder.java:197)
>     at
> ucar.nc2.grib.grib1.Grib1TimePartitionBuilder.readOrCreateIndex(Grib1TimePartitionBuilder.java:128)
>
>     at
> ucar.nc2.grib.grib1.Grib1TimePartitionBuilder.factory(Grib1TimePartitionBuilder.java:73)
>     at ucar.nc2.grib.TimePartition.factory(TimePartition.java:96)
>     at
> thredds.catalog.InvDatasetFcGrib.updateCollection(InvDatasetFcGrib.java:224)
>     at thredds.catalog.InvDatasetFcGrib.update(InvDatasetFcGrib.java:163)
>     at
> thredds.catalog.InvDatasetFeatureCollection.handleCollectionEvent(InvDatasetFeatureCollection.java:236)
>
>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>     at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>     at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>     at java.lang.reflect.Method.invoke(Method.java:616)
>     at ucar.nc2.util.ListenerManager.sendEvent(ListenerManager.java:177)
>     at
> thredds.inventory.CollectionManagerAbstract.sendEvent(CollectionManagerAbstract.java:213)
>     at
> thredds.inventory.MFileCollectionManager.scan(MFileCollectionManager.java:392)
>     at
> thredds.inventory.CollectionUpdater$UpdateCollectionJob.execute(CollectionUpdater.java:287)
>     at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
>     at
> org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557)
>
>
>
> And when trying to access my TDS from a browser, the catalog could not be
> made:
>
>
> 2013-04-29T13:40:33.330 -0600 [    137576][       7] INFO  -
> threddsServlet - Remote host: 128.117.8.232 - Request: "GET
> /thredds/catalog/LOCALFNL/catalog.html HTTP/1.1"
> 2013-04-29T13:40:33.335 -0600 [    137581][       7] ERROR - fc.LOCAL_FNL
> - Error making catalog for LOCALFNL
> java.lang.NullPointerException
>     at
> thredds.catalog.InvDatasetFcGrib.makeCatalogTop(InvDatasetFcGrib.java:453)
>     at
> thredds.catalog.InvDatasetFcGrib.makeCatalog(InvDatasetFcGrib.java:377)
>     at
> thredds.servlet.DataRootHandler.makeDynamicCatalog(DataRootHandler.java:1467)
>     at
> thredds.servlet.DataRootHandler.getCatalog(DataRootHandler.java:1430)
>
>     at
> thredds.server.catalogservice.LocalCatalogServiceController.handleRequestInternal(LocalCatalogServiceController.java:191)
>     at
> org.springframework.web.servlet.mvc.AbstractController.handleRequest(AbstractController.java:153)
>     at
> org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter.handle(SimpleControllerHandlerAdapter.java:48)
>     at
> org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:925)
>     at
> org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:856)
>     at
> org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:936)
>     ...
>
>
> For grins, I tried changing the partition to
>
>         timePartition="directory"
>         dateFormatMark="#grib1/#yyyy"
>
> And got the same ClassCastException.
>
>
> If I'm missing something obvious, please let me know.  I understand that
> the GRIB feature collection is still in flux, so please consider this some
> testing info.  I'm able to make a GRIB collection without the time
> partition, but for file browsing, that partition would be a feature we
> would really want to incorporate.
>
> Cheers!
>
>
> -kevin.
>
> --
> Kevin Manross
> NCAR/CISL/Data Support Section
> Phone: (303)-497-1218
> Email:manross@xxxxxxxx
> Web:http://rda.ucar.edu
>
> _______________________________________________
> thredds mailing list
> thredds@xxxxxxxxxxxxxxxx
> For list information or to unsubscribe,  visit:
> http://www.unidata.ucar.edu/mailing_lists/
>



-- 
=======================================
Dan Swank
NCDC-NOMADS Project
Team ERT/STG
(NCDC Government Contractor)
Data Access & Applications Branch
Veach-Baley Federal Building
151 Patton Avenue
Asheville, NC 28801-5001
Email: Dan.Swank@xxxxxxxx
Phone: 828-271-4007
============================================
Any opinions expressed in this message are mine personally and
do not represent any position from STG, ERT, NCDC, or NOAA
============================================
  • 2013 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the thredds archives: