[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[LDM #CZE-486853]: GFS grid question

Hi,

re:
> Note, Tom suggested I change:
> request CONDUIT|NGRID ".*.gfs*" idd.unidata.ucar.edu
> request CONDUIT|NGRID ".*.GFS*" idd.unidata.ucar.edu
> request NGRID|CONDUIT "(gfs|GFS)" idd.unidata.ucar.edu
> 
> Yes - the first two were commented out during the test.

Very good.  One of the primary reasons for the change was
a ".*" at the beginning of a regular expression is not needed/redundant.
This is what is referred to as a pathalogical regular expression.  Here
is Steve's explanation:

A pathological regular expression is one that starts with ".*". Such a
prefix changes nothing (the same product-identifiers are matched either
way) but for some regular expression libraries, the matching can be orders
of magnitude slower.

re:
> I'm not sure it is a coincidence, but ldm connections greatly increase
> after that change 

I don't understand what you are saying here:

"ldm connections greatly increase after the change"

What greatly increased?

- the number of connections?

  This should not be the case as the change combined two REQUESTs in
  one.

- the latency of product receipt?

  The rtstats plot for latencies for NGRID on on-dsserve do not look
  like this was the case:

  
https://rtstats.unidata.ucar.edu/cgi-bin/rtstats/iddstats_nc?NGRID+on-dcserve.ssec.wisc.edu

- the volume of data being received in the NGRID feed?

  The volume rtstats volume plot for on-dsserve _does_ show this:

  
https://rtstats.unidata.ucar.edu/cgi-bin/rtstats/iddstats_vol_nc?NGRID+on-dcserve.ssec.wisc.edu

  It should not be the case that combining two REQUESTs into one where
  the regular expression used in the combined REQUEST is, in essence,
  the same as the union of the regular expressions used in the two
  would result in more data being received!

re:
> and we dropped a lot of grib data that come in on NGRID.

I don't know what this means either.  Do you mean that it was
not processed by XCD, or are you saying that a lot of grib data
was not received?  The rtstats volume plot for NGRID on on-dsserve
indicates that you received _more_ data (and all of it is in GRIB2
format) in NGRID than before the change.

re:
> Our grib data is processed through XCD, so there maybe another
> underlying problem. 

Yes, it is (near) impossible for us to make any comments about
any data processing "downstream" of the LDM with the information
at hand.

re:
> This may be of interest.
> https://qcweb.ssec.wisc.edu/web/xcd/2021239/index.html

It is interesting that data missing in the XCD processing pipeline
is well documented.

re:
> As Tom mentioned, it's hard for you to comment if problems are XCD related.

Correct.  My comment should have been more generic - it is (near) impossible
to make any specific comments about processing that is done downstream
of the LDM receipt of data.  All I can suggest is that someone there (you?)
spend the time comparing what was actually received by the LDM (mine the
log file for the 'notifyme' invocation that is looking at the local
LDM) and want ended up being processed by your XCD processing pipeline.

re:
> I'm going back to the previous version of ldmd.conf to catch the
> incoming 12Z run

...


Cheers,

Tom
--
****************************************************************************
Unidata User Support                                    UCAR Unidata Program
(303) 497-8642                                                 P.O. Box 3000
address@hidden                                   Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage                       http://www.unidata.ucar.edu
****************************************************************************


Ticket Details
===================
Ticket ID: CZE-486853
Department: Support LDM
Priority: Normal
Status: Closed
===================
NOTE: All email exchanges with Unidata User Support are recorded in the Unidata 
inquiry tracking system and then made publicly available through the web.  If 
you do not want to have your interactions made available in this way, you must 
let us know in each email you send to us.