[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[LDM #GOE-694008]: Request for temporary IDD for NWA Demo

Hi Brad,

Good to see you at AMS in New Orleans...

re:
> Yeah I may have diluted the question - see if you can follow this logic:
> 
> (1)  a site issues the FXUS10 KWNH product on January 20th at 0528Z.
> 
> (2) The product is sent over our smtp setup and uplinked at the NCF.
> 
> (3) Another site receiving the noaaport feed and running LDM /readnoaaport
>     for ingest from the AWIPS SBN noaaport feed sees the product as follows on
>     the LDM connected directly to the DVBs:
> 
> Jan 20 05:28:10 cpsbn1-hpcn readnoaaport[29052] NOTE: FXUS10 KWNH 200528
> /pPMDHMD inserted [cat 1 type 4 ccb 2/0 seq 68910095 size 4791]

OK.  For reference: the only piece of this that is visible to downstream
requests is the product ID 'FXUS10 KWNH 200528 /pPMDHMD'.  The other
pieces of the readnoaaport log entry:

"[cat 1 type 4 ccb 2/0 seq 68910095 size 4791]"

are not part of what can be used in a downstream's ~ldm/etc/ldmd.conf
REQUEST.

re:
> (4) A downstream LDM client requests the product per the pqact.conf entry:
> 
> IDS|DDPLUS      ^([ABCFMNRSUVW][A-Z]{3}[0-9]{2}) ([KPTMC].{3}) (..)(..)(..)
>      FILE    -overwrite -log -close -edex
>      /data_store/text/\3/\4/\1_\2_\3\4\5_(seq).%Y%m%d

For those who might read this exchange after it has been HTMLized and
made available to web crawlers and then made accessible through something
like Google, etc.:

- (the '-edex' and '(seq)' pieces of the above action are
  _NOT_ part of a standard Unidata LDM distribution. Instead, they were added
  by Raytheon (or one of its subcontractors), AND, if what we saw in the 
ldm-6.8.1
  distribution bundled with the AWIPS-II installation we were running at the
  AMS 2012 Annual Meeting in New Orleans holds, the non-Unidata code broke a
  variety of features in normal 'pqact' processing.  I don't think that
  this is causing the problem being discussed in this exchange, but I offer
  it as background information "just in case"...

re:
> (5) And the downstream LDM requests the product and performs the FILE
>     actions as follows:
> 
> Jan 20 05:28:10 dx2-hpcn pqact[9934] NOTE: Filed in
> "/data_store/text/20/05/FXUS10_KWNH_200528_68910095.20120120":     4791
> 20120120052810.048 IDS|DDPLUS 68910095  FXUS10 KWNH 200528 /pPMDHMD

OK.

re:
> Then, the same site, the same way, issues the same product at 1514Z, it
> gets across the AWIPS smtp MHS system to the NCF and is uplinked, but this
> time, it comes into the LDM running on the device connected to the DVB
> directly as follows:
> 
> Jan 20 15:14:50 cpsbn1-hpcn readnoaaport[29052] NOTE: FXUS10 KWNH 201514
> /pPMDHMD inserted [cat 101 type 4 ccb 2/0 seq 69705248 size 2236]

OK.  The Product ID for this product seems only to differ from the
one sent earlier by its time (201514 vs 200528 assuming, of course
that there is nothing strange like unprintable characters, etc.).
Also, since the products have different sizes (4791 for the earlier
one and 2236 for the later one), the MD5 signatures for the two products
should be different.  So even if the earlier product was still in
the receiving LDM's product queue, it should not have been rejected
as being a duplicate.

re:
> And the downstream LDM does not register anything, no request for the
> product, and thus it isn't stored.

I would assume that the REQUEST for the data was sufficient broad
enough to allow for the fields '200528' and '201514', but I can't
be 100% sure since you did not send the REQUEST entry being used.

re:
> This problem was brought to my attention well after the fact, so I could
> not pqcat the product queue,

Too bad!

> but according to the above entry it is inserted,

In the LDM queue on the machine running the NOAAPort ingest, yes.

re:
> the only difference I noticed was all previous PMDHMD products
> came in as cat 1 but the one that did not store came in as cat 101, so I
> was wondering if it wasn't being "seen" as an IDS/DDSPLUS per the
> pqact.conf entry, and thus it wasn't requested, 

This is a very good question which I can't answer off of the top of
my head.  It may have been inserted in the HDS (aka HRS) datastream.

re:
> and if so, then why did that happen?

Good question.

re:
> Does that make sense?

Yes, your question makes perfect sense.

Just for reference:

- I assume that the REQUEST line for IDS|DDPLUS is ".*" (everything)

  Please let us know if this is not the case.

Comment:

- I took a look to see if we still had NOAAPort ingest log files for Jan 20,
  but we only keep a couple of days at a time.  However, I note the exact
  same situation on Jan 25:

  Here is the single instance of what I found:

Jan 25 17:41:08 burns.unidata.ucar.edu noaaportIngester[14794] NOTE: FXUS10 
KWNH 251740 /pPMDHMD inserted [cat 1 type 4 ccb 2/0 seq 126923 size 4550]
 ...
Jan 25 18:34:27 burns.unidata.ucar.edu noaaportIngester[14794] NOTE: FXUS10 
KWNH 251834 /pPMDHMD inserted [cat 101 type 4 ccb 2/0 seq 199208 size 4560]

  Since we have a situation exactly analogous to yours, it should be
  possible capture the products and sort out what is going on.

  I setup a REQUEST for everything (ANY ".*") from one of our NOAAPort
  ingest machines and a simple pattern-action file/action to capture
  all FXUS10 products:

ANY     ^([ABCFMNRSUVW][A-Z]{3}[0-9]{2}) ([KPTMC].{3}) (..)(..)(..)
        FILE    -overwrite      -close  -log
        data/text/\3/\4/\1_\2_\3\4\5_.%Y%m%d

  We should see over the next couple of days if products with 'cat 101'
  are being put into a datastream other than IDS|DDPLUS.  If they are,
  I can look at the readnoaaport code to see why.

Cheers,

Tom
--
****************************************************************************
Unidata User Support                                    UCAR Unidata Program
(303) 497-8642                                                 P.O. Box 3000
address@hidden                                   Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage                       http://www.unidata.ucar.edu
****************************************************************************


Ticket Details
===================
Ticket ID: GOE-694008
Department: Support IDD
Priority: High
Status: Closed