[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: New Client Reply - [THREDDS !GHR-456072]: nullPointerException in when accessing 4.3.15 GRIB featureCollection's



wow, i didnt envision this scale. It should work if you dont run out of memory, but probably need to think of better algorithms.

however, once its indexed, shouldnt do it again, if you use

<update startup="nocheck" trigger="allow" />

On 5/28/2013 9:17 AM, Dan Swank - NOAA Affiliate wrote:
Morning John,
Assume your still waiting on logs.. will get thta
to you shortly.   Looks like I stepped into
a 14 day process with indexing the NDFD
4667691 gbx9's created over the weekend and its
still got about 15 million to go.     The spin-up I
mentioned took approximate 15 hours.   So unless
I comment the NDFD away, restarting will cost alot of time.


On Thu, May 23, 2013 at 4:06 PM, Dan Swank - NOAA Affiliate <address@hidden <mailto:address@hidden>> wrote:

    One quick note
    the ruc130anl FC is actually working from
    the openDap form.   ruc252anl is not.   the
    errors still exist in the ruc130anl logs .. however.
    But it does not seem to be affecting functionality.




On Thu, May 23, 2013 at 3:37 PM, Dan Swank - NOAA Affiliate <address@hidden <mailto:address@hidden>> wrote:

        John,

        After opening it up to the entire NDFD tree.... the scan log
        seems to
        cease here.... wondering if there is a "spin up" period while its
        finding files ... and nothing is written to the log for a period
        of time ?
        (talking scale of millions of files )

        Probably just need to give it more time.

        2013-05-23T15:12:35.655 -0400 INFO  - FeatureCollection added =
        name ='NDFD' type='GRIB'
           spec='/global/nomads/nexus/ndfd/**/......_...._#yyyyMMdd#....$'
           timePartition =1 day
           updateConfig =UpdateConfig{startup=true, recheckAfter='null',
        rescan='null', triggerOk=true, deleteAfter=null}
           protoConfig =ProtoConfig{choice=Penultimate, change='null',
        param='null', outerNcml='null', cacheAll=true}
           hasInnerNcml =false
           gribConfig =GribConfig: datasetTypes=[Best, Files]


NNR2 :

        After battling a bit with the spec and mark, I got this:
        -- seemingly odd exception.

        ---------

        2013-05-23T15:23:55.682 -0400 INFO  - FeatureCollection added =
        name ='NNR2pgrbft' type='GRIB'

        spec='/global/nomads/nexus/reanalysis-2/6hr/pgb.ft06/pgb.ft06.#yyyyMM'
           timePartition =1 year
           updateConfig =UpdateConfig{startup=true, recheckAfter='null',
        rescan='null', triggerOk=true, deleteAfter=null}
           protoConfig =ProtoConfig{choice=Penultimate, change='null',
        param='null', outerNcml='null', cacheAll=true}
           hasInnerNcml =false
           gribConfig =GribConfig: datasetTypes=[Best, Files]

        2013-05-23T15:27:21.062 -0400 INFO  - NNR2pgrbft : was scanned
        nfiles= 360
        2013-05-23T15:27:21.062 -0400 INFO  - NNR2pgrbft: createIndex
        
/home/tomcat/tomcat-main.4.3.15/content/thredds/cache/GribIndex/global/nomads/nexus/reanalysis-2/6hr/pgb.ft06/NNR2pgrbft.ncx
        2013-05-23T15:27:21.063 -0400 ERROR - Fail to create/update
        collection
        java.lang.RuntimeException: SimpleDateFormat = yyyyMM fails on
        pgb.ft06.198306 ParseException:Unparseable date: "pgb.ft06.198306"
                 at
        
ucar.nc2.units.DateFromString.getDateUsingCompleteDateFormatWithOffset(DateFromString.java:214)
                 at
        
ucar.nc2.units.DateFromString.getDateUsingDemarkatedCount(DateFromString.java:99)
                 at
        
thredds.inventory.DateExtractorFromName.getDate(DateExtractorFromName.java:67)
                 at
        
thredds.inventory.DateExtractorFromName.getCalendarDate(DateExtractorFromName.java:74)
                ~~~~

        ---
        This is probably all from me today,
        See you Tuesday.





        On Thu, May 23, 2013 at 8:56 AM, Dan Swank - NOAA Affiliate
        <address@hidden <mailto:address@hidden>> wrote:

            Okay, I will flush the .ncx for ruc.

            NDFD is purring now for 1 days worth.
            It will take a long time to generate gbx9 for the entire
            online period,
              we will see how far it gets over the extended weekend.
              That will be a great stress test.
            In the meantime,
            I will take a shot at the GDS hashes for NDGD.


On Wed, May 22, 2013 at 9:04 PM, John Caron <address@hidden <mailto:address@hidden>> wrote:

                havent been able to reproduce this one.

                delete all the collection ncx and restart tds. then send
                me the logs.

                im gone starting tommorrow for a long weekend, so we
                will resume next tuesday

                ciao!


On 5/22/2013 11:57 AM, Dan Swank - NOAA Affiliate wrote:

                    Got the correct version running this morning.

                    Heres some logs, some insight on the fc.ruc252,
                         but not too much
                    
http://nomads.ncdc.noaa.gov/__data/gfsanl/201001-test/__tdsLogs-20130522.tar.bz2
                    
<http://nomads.ncdc.noaa.gov/data/gfsanl/201001-test/tdsLogs-20130522.tar.bz2>




On Tue, May 21, 2013 at 5:38 PM, John Caron <address@hidden <mailto:address@hidden> <mailto:address@hidden <mailto:address@hidden>__>> wrote:



                         On 5/21/2013 2:40 PM, Dan Swank - NOAA
                    Affiliate wrote:

                             Running Version 4.3.17-SNAPSHOT - 20130515.1958


then it should say:

                         Version 4.3.17-SNAPSHOT - 20130517.1111




-- ==============================__========= Dan Swank NCDC-NOMADS Project Team ERT/STG (NCDC Government Contractor) Data Access & Applications Branch Veach-Baley Federal Building 151 Patton Avenue Asheville, NC 28801-5001 Email: address@hidden <mailto:address@hidden> <mailto:address@hidden <mailto:address@hidden>>

                    Phone: 828-271-4007 <tel:828-271-4007>
                    ==============================__==============
                    Any opinions expressed in this message are mine
                    personally and
                    do not represent any position from STG, ERT, NCDC,
                    or NOAA
                    ==============================__==============




-- ======================================= Dan Swank NCDC-NOMADS Project Team ERT/STG (NCDC Government Contractor) Data Access & Applications Branch Veach-Baley Federal Building 151 Patton Avenue Asheville, NC 28801-5001 Email: address@hidden <mailto:address@hidden> Phone: 828-271-4007 <tel:828-271-4007> ============================================ Any opinions expressed in this message are mine personally and do not represent any position from STG, ERT, NCDC, or NOAA ============================================




-- ======================================= Dan Swank NCDC-NOMADS Project Team ERT/STG (NCDC Government Contractor) Data Access & Applications Branch Veach-Baley Federal Building 151 Patton Avenue Asheville, NC 28801-5001 Email: address@hidden <mailto:address@hidden> Phone: 828-271-4007 <tel:828-271-4007> ============================================ Any opinions expressed in this message are mine personally and do not represent any position from STG, ERT, NCDC, or NOAA ============================================




-- ======================================= Dan Swank NCDC-NOMADS Project Team ERT/STG (NCDC Government Contractor) Data Access & Applications Branch Veach-Baley Federal Building 151 Patton Avenue Asheville, NC 28801-5001 Email: address@hidden <mailto:address@hidden> Phone: 828-271-4007 <tel:828-271-4007> ============================================ Any opinions expressed in this message are mine personally and do not represent any position from STG, ERT, NCDC, or NOAA ============================================




-- ======================================= Dan Swank NCDC-NOMADS Project Team ERT/STG (NCDC Government Contractor) Data Access & Applications Branch Veach-Baley Federal Building 151 Patton Avenue Asheville, NC 28801-5001 Email: address@hidden <mailto:address@hidden> Phone: 828-271-4007 ============================================ Any opinions expressed in this message are mine personally and do not represent any position from STG, ERT, NCDC, or NOAA ============================================


NOTE: All email exchanges with Unidata User Support are recorded in the Unidata inquiry tracking system and then made publicly available through the web. If you do not want to have your interactions made available in this way, you must let us know in each email you send to us.