[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Reingestion problem at LSU

Robert Leche wrote:
> 
> Hello Anne,
> I am indeed attempting to re-ingest a stream, a stream we simple call
> 'raw file data' which comes from the NOAAport or IDD. As described
> before, When data are reprocessed with the reing.csh scriptI(pqing) I
> am finding data showing up in the ldm tree but with an incorrect date
> in the file name. The example I related in the previous email is a
> good example. The process was repeated with a raw file stream dated
> 10/01/2001 and the results where nearly the same, only the date of the
> file (01) was correct. The pqing produced a file with this months
> number and this years number: 020101 . This actually overwrote good
> current data.  I include a sample of th 2001092600.raw file data. At
> this point, I am not sure of the date feild in the raw file IDD stream
> data.
> 
> As far as the question about files showing up else where..... I
> looked, Anne and I do not see them.
> 
> 
> 874
> WWMM30 LERT 260000 Z
> SUBJ/260000Z SEP 01 MEDITERRANEAN, BLACK, AND BALTIC
> SEA HIGH WIND AND SEAS WARNING//
> REF/A/RMG/NAVEURMETOCCEN ROTA SP/242000ZSEP01//
> AMPN/REF A IS HIGH WIND/SEAS WARNING VALID 251200ZSEP01.//
> 1. NO WINDS/SEAS 35 KTS/12 FT OR GREATER FORECAST FOR
> THE 12 HR PERIOD BEGINNING 260000ZSEP01.
> 2. THIS WARNING WILL SUPERCEDE AND CANCEL REF A AFTER
> 250000Z SEP 01 AND THE NEXT MEDITERRANEAN/BLACK/BALTIC
> SEA HIGH WIND AND SEAS WARNING WILL BE NAVEURMETOCCEN
> ROTA WWMM31 261200Z.//
> 
> 
> 897
> FABZ24 SBPA 260000
> SBCW GAMET VALID 260000/260600 UTC SBPA-
> CURITIBA FIR BLW FL100
> SECN I
> SFC VIS : 5000M RA S OF 23 DEG S.
> SIG WX : N OF 28 DEG S ISOL TS.
> MT OBSC : MT SERRA DO MAR N OF 30 DEG S.
> SECN II
> PSYS : COLD FRONT 0300 30 DEG S 54 DEG W AND
> 31 DEG S 50 DEG W MOV NE 10KT INTSF.
> WIND/T : 2000FT 18015KT PS18 5000FT 33020KT PS15
> 10000FT 28015KT PS04.
> CLD : BKN CUSC 1000/5000FT S OF 23 DEG S.
> MNM QNH : 1009HPA.
> VA : NIL.=
> 
> 
> 475
> UXAK47 PAJK 260000
> &ZCZC JNUFZLANN
> RADAT 62057=
> $$
> 
> 
> NNNN
> 
> 
> 
> 054
> FTID20 WIII 260000
> MF 1072
> WIII 260618 01010KT 9000 SCT018
>      BECMG 0810 16005KT 8000 FEW018CB SCT018
>      BECMG 1416 00000KT 5000 HZ=
> 
> 
> 792
> SMVA01 KWBC 260000
> BBXX
> ELXS4 26003 99193 10395 41498 33406 10310 2028/ 40060 58005 70111
>  834// 22274 20201 334// 40301 80290=
> 
> 
> 793
> SMVB01 KWBC 260000
> BBXX
> ELPT3 26004 99492 11621 41/96 90820 10080 2004/ 40100 57020 72166
>  22214 00100 20403 80060=
> ELQB8 26004 99534 11735 41296 82425 10090 2005/ 40109 52010 70022
>  882// 22263 00080 20707 80070=
> 
> 
> 796
> SMVD01 KWBC 260000
> BBXX
> WQVY 26004 99439 70396 43398 63116 10190 2018/ 40165 52011 868//
>  22264 20404 330// 40603 80170=
> 
> 
> 
> 
> --
> 
> ----------------------------------------------------------------
> Robert Leche
> System Administrator
> Louisiana State University - Southern Regional Climate Center
> 260 Howe-Russell Building
> Baton Rouge, La. 70803
> address@hidden
> 225 578 5023
> ----------------------------------------------------------------
> 
> 

Hi Bob,

First, it is pqact that files the data, not pqing.  pqing only breaks up
the stream into products and inserts them into the queue.

Second, (and I should have said this yesterday), the products themselves
only carry the day of the month, hour, and minute.  For example, from
the last product above:

> SMVD01 KWBC 260000
              ^^^^^^
This shows the product time stamp as 00:00 on the 26th.  Since the LDM
was designed to handle current data, pqact extrapolates the month and
year based on today's date.  This is why it is filing the product using
the current month and the current year.

If you really want to break up your old raw stream into separate files,
this could be done by "cheating".  For example, you could make a
pqact.conf entry using a pattern that only matched those products
(assuming this is possible - it may not be), and hard wire the year and
month into the path yourself.  If you can't match only those products,
you'll probably have to stop your ldm, turn off the requests to other
sites, add the new pqact.conf entry with the date hard wired in, restart
the LDM, ingest the raw file with pqing, then stop the LDM and reverse
all that to get back to ingesting from your usual servers.  

Sorry, it's not pretty, but the LDM was not designed to handle mixing of
current products and "very old" products.   Indeed, it was designed to
handle current products only.  There are a few ways to get around this
limitation (like I described above) although depending on what you want
to do it may also require changing some default values to rpc.ldmd and
pqact.   

Anne
-- 
***************************************************
Anne Wilson                     UCAR Unidata Program            
address@hidden                 P.O. Box 3000
                                  Boulder, CO  80307
----------------------------------------------------
Unidata WWW server       http://www.unidata.ucar.edu/
****************************************************