[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Reingestion problem at LSU

Robert Leche wrote:
> 
> Hello Unidata, Anne and all,
> 
> I observed something related to a problem I reported yesterday. I
> found that data is being parsed out of the LDM queue as the pqact.conf
> tree defines 'what goes where'. But the dates on the files  where
> re-ingested where wrong. The date number part (of the original raw
> file date) was correct, but the year and month parts where wrong,
> showing up as the current year and month. Mistral's  pqact.conf is
> attached.
> 
> Selected examples of files re-ingested into our digested data product
> tree for January 16 ( Raw files generated 09/26/2001):
> Model products:
> -rw-r--r--    1 ldm5     data      11652236 Jan 16 13:17 02012600.mod
> -rw-r--r--    1 ldm5     data       5476036 Jan 16 14:22 02012612.mod
> Extra products:
> -rw-r--r--    1 ldm5     data         75064 Jan 16 14:21
> 020126.SRCC.summary
> -rw-r--r--    1 ldm5     data        563567 Jan 16 21:08
> 020126.climate
> -rw-r--r--    1 ldm5     data      43699336 Jan 16 21:08 020126.extra
> -rw-r--r--    1 ldm5     data       2968501 Jan 16 21:07
> 020126.rainavg
> -rw-r--r--    1 ldm5     data      55564356 Jan 16 21:08
> 020126.rainfall
> -rw-r--r--    1 ldm5     data          8445 Jan 16 15:11
> 020126.rainrep
> -rw-r--r--    1 ldm5     data         33081 Jan 16 15:07
> 020126.satellite
> -rw-r--r--    1 ldm5     data       3558468 Jan 16 21:08 020126.sfcfor
> 
> -rw-r--r--    1 ldm5     data       2678798 Jan 16 21:08 020126.ship
> -rw-r--r--    1 ldm5     data       1382776 Jan 16 21:08
> 020126.shiprpt
> -rw-r--r--    1 ldm5     data        375025 Jan 16 21:08 020126.ships
> -rw-r--r--    1 ldm5     data        242609 Jan 16 21:06
> 020126.summary
> -rw-r--r--    1 ldm5     data       2122028 Jan 16 21:06
> 020126.tropical
> -rw-r--r--    1 ldm5     data        589226 Jan 16 21:07
> 020126.tropics
> -rw-r--r--    1 ldm5     data          3002 Jan 16 14:23 020126.upair
> -rw-r--r--    1 ldm5     data           936 Jan 16 13:16
> 02012600.reconn
> -rw-r--r--    1 ldm5     data           396 Jan 16 13:20
> 02012601.reconn
> -rw-r--r--    1 ldm5     data          1846 Jan 16 14:21
> 02012612.reconn
> -rw-r--r--    1 ldm5     data           496 Jan 16 14:24
> 02012613.reconn
> -rw-r--r--    1 ldm5     data           468 Jan 16 21:07
> 02012618.reconn
> -rw-r--r--    1 ldm5     data           240 Jan 15 18:27
> 02012620.reconn
> -rw-r--r--    1 ldm5     data           488 Jan 15 19:16
> 02012622.reconn
> -rw-r--r--    1 ldm5     data           158 Jan 15 19:30
> 02012623.reconn
> 
> An example of a proper 'file date' format should be 010926xx.abcdef.
> 
> I will away from the phone between 11:00am - 2:00pm (central time)
> 
> Ideas?
> 
> Bob
> 
> -----------------------------------------------------------
> 
> The following text where reported yesterday:
> 
> -----------------------------------------------------------
> Hello Anne,
> It been awhile since problems have creep up and this is a good thing.
> 
> All is well with real time data acquisition at LSU, but I am running
> into problems reingesting data on mistral.srcc.lsu.edu with pqing.
> For years, the following script was used to reingest raw data.
> 
> #!/bin/csh
> # run this as root on mistral only!
> 
> cd /sirocco/rawfiles
> 
> # Clone this line and cut to hours needed...don't run too many at once!
> #foreach hr ( 00 01 02 03 04 05 06 07 08 09 10 11 )
> #foreach hr ( 12 13 14 15 16 17 18 19 20 21 22 23 )
> 
> foreach day ( 26 )
>         #foreach hr ( 13 14 15 16 17 18 19 20 21 22 23 )
>         foreach hr ( 11 12 13 14 15 16 17 )
>     #  if(($day == 10) && ($hr <= 10) ) continue
>         foreach fi (`ls 200109$day$hr.raw`)
>         set FN=`echo $fi | cut -d. -f1,2`
>         echo "Reingesting $FN"
>         pqing -l - -f ddplus $FN
>         echo "Sleeping..."
>         sleep 90
>         end
>     end
> end
> 
> When the script is invoked, the output looks normal:
> Sleeping...
> Reingesting 2001092612.raw
> Jan 16 20:15:34 pqing[757509]: Starting Up
> Jan 16 20:15:34 pqing[757509]: FILE "2001092612.raw"
> Jan 16 20:19:58 pqing[757509]: Lone ETX error
> 9         -01      00      162019
> Jan 16 20:21:06 pqing[757509]: Lone ETX error
> 9         -01      00      162021
> Jan 16 20:21:20 pqing[757509]: Lone ETX error
> 137         -01  FAGG26 1200 162021
> Jan 16 20:21:21 pqing[757509]: Lone SOH error
> 514         288  SAHO31 MHTG 261200 /pMETAR
> Jan 16 20:21:21 pqing[757509]: Lone ETX error
> 691         -01  FADN26 1200 162021
> Jan 16 20:21:29 pqing[757509]: Lone SOH error
> 258         888  SMGY01 SYCJ 261200
> Jan 16 20:22:15 pqing[757509]: Exiting
> Jan 16 20:22:15 pqing[757509]:   Queue usage (bytes):214410488
> Jan 16 20:22:15 pqing[757509]:            (nregions):   65505
> Jan 16 20:22:15 pqing[757509]:   Duplicates rejected:       0
> Jan 16 20:22:15 pqing[757509]:   WMO Messages seen:     10822
> Jan 16 20:22:15 pqing[757509]:   SOH/ETX missing  :         6
> Jan 16 20:22:15 pqing[757509]:   parity/chksum err:         0
> Jan 16 20:22:15 pqing[757509]:   WMO format errors:         0
> Jan 16 20:22:15 pqing[757509]:   FILE Bytes read:    11050352
> Sleeping...
> Reingesting 2001092613.raw
> Jan 16 20:23:45 pqing[763240]: Starting Up
> Jan 16 20:23:45 pqing[763240]: FILE "2001092613.raw"
> Jan 16 20:24:04 pqing[763240]: Lone ETX error
> 263         -01  FADN26 1200 162024
> Jan 16 20:25:05 pqing[763240]: Lone SOH error
> 258         105  SAHO31 MHTG 261300 /pMETAR
> Jan 16 20:25:09 pqing[763240]: Exiting
> Jan 16 20:25:09 pqing[763240]:   Queue usage (bytes):224751256
> Jan 16 20:25:09 pqing[763240]:            (nregions):   72374
> Jan 16 20:25:09 pqing[763240]:   Duplicates rejected:       0
> Jan 16 20:25:09 pqing[763240]:   WMO Messages seen:      6019
> Jan 16 20:25:09 pqing[763240]:   SOH/ETX missing  :         2
> Jan 16 20:25:09 pqing[763240]:   parity/chksum err:         0
> Jan 16 20:25:09 pqing[763240]:   WMO format errors:         0
> Jan 16 20:25:09 pqing[763240]:   FILE Bytes read:     4864625
> Sleeping...
> 
> I've set the -v verbose flag and gathered the following messages:
> Produces a message that the product is queue.
> Jan 16 20:00:50 pqing[672520]: Product already in queue
> Jan 16 20:00:50 pqing[672520]:      110 20020116200050.071  DDPLUS 892
> SACN89 CWAO 260000 RRC
> Jan 16 20:00:50 pqing[672520]: Product already in queue
> Jan 16 20:00:50 pqing[672520]:       77 20020116200050.073  DDPLUS 893
> CSCN03 CWVR 260000
> Jan 16 20:00:50 pqing[672520]: Product already in queue
> Jan 16 20:00:50 pqing[672520]:     3480 20020116200050.076  DDPLUS 901
> USUS50 KWBC 260000 RRE
> Jan 16 20:00:50 pqing[672520]: Product already in queue
> Jan 16 20:00:50 pqing[672520]:     3324 20020116200050.081  DDPLUS 902
> USUS50 KWBC 260000 RRF
> Jan 16 20:00:50 pqing[672520]: Product already in queue
> Jan 16 20:00:50 pqing[672520]:      350 20020116200050.085  DDPLUS 903
> USUS50 KWBC 260000 CCB
> Jan 16 20:00:50 pqing[672520]: Product already in queue
> 
> So, the ingested data is making it to the ldmqueue but looks as if these
> data are not processed out of it. The results of this is a file system
> tree that should be expanded with the reingested data but is not. This
> is strange, as I said before as the pqing process use to work. I already
> tried deleting and remaking the ldmqueue, no difference. Also, 'ldmadmin
> watch' has not revealed data processed from the selected date only
> current ingestion. We are processing real-time data with pqact. Mistral
> is an SGI IREX system running Ldm version ldm-5.0.8.
> 
> --
> 
> ----------------------------------------------------------------
> Robert Leche
> System Administrator
> Louisiana State University - Southern Regional Climate Center
> 260 Howe-Russell Building
> Baton Rouge, La. 70803
> address@hidden
> 225 578 5023
> ----------------------------------------------------------------

Hi Bob,

I apologize for not responding sooner.  I was at the AMS conference in
Orlando last week.

Let me make sure I understand your problem correctly.  In the example
you gave me you were trying to ingest a stream of products from a raw
file that were presumably created in September, but it appears that
pqact created files from the products in the raw file that had the
current date.

Are you sure this raw file actually contains products that were created
in September?  It is possible that even though the filename indicates a
September date, such as '2001092612.raw', the products contained in the
file are from some other date.  Your pqact.conf entries, such as

DDS|PPS ^WHXX.* .... ([0-3][0-9])
        FILE    data/ddplus/severe/(\1:yy)(\1:mm)\1.mod
and

DDS|PPS ^FO.* .... ([0-3][0-9])([0-2][0-9])
        FILE    data/ddplus/model/(\1:yy)(\1:mm)\1\2.mod

are creating file names based on the product bulletin time.  So if
2001092612.raw contained products from the current day, that's how they
would be filed.

Or, are the September products possibly being filed somewhere where you
aren't finding them while new products are also being filed?  

Anne
-- 
***************************************************
Anne Wilson                     UCAR Unidata Program            
address@hidden                 P.O. Box 3000
                                  Boulder, CO  80307
----------------------------------------------------
Unidata WWW server       http://www.unidata.ucar.edu/
****************************************************