[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[LDM #CZE-486853]: GFS grid question

Hi Rick,

re:
> So I can get random results?

No, not random.  Steve comment was made to point out that the regular expression
you were (are still?) using will not be doing exactly what you think it is
doing, so you need to be aware of the difference.  It may be helpful to 
hear that a REQUEST of "GFS" is a supset of a REQUEST for "GFS*".  What 
portion of a subset that my be can only be determined by examination of
the log files produced by comparative 'notifyme' invocations.

Aside:

For interest, I ran 'notifyme' invocations on one of our data server machines
to see if there were Product IDs in NGRID that matched "gfs" or "GFS" and
Product IDs in CONDUIT that match  either "gfs" or "GFS".  My casual observation
was that the CONDUIT Product IDs for GFS products match both "gfs" and "GFS"
while the NGRID Product IDs match "GFS".  If these casual observation are
complete/true/how ever you want to phrase it, it may be useful to use
to split the feed REQUESTs.

Another aside:

The current latencies being reported by on-dcserv do not look large enough to 
definitively indicate that on-dcserv is/should be losing data.

re:
> I'm getting the expected results for both machines for fhour 36 and not
> fhour 72

That is likely due to something other than the regular expression IF
you are using the same regular expressions and REQUEST line(s) on
all of the machines you are comparing.

re:
> I checked the 72 hour forecast for the 6Z model run and and got my
> expected results for both machines.

OK.  Are they using the exact same REQUEST line(s) and does(do) that(those)
REQUEST(s) use the exact same regular expression?  If yes, then the problem
is due to something else.  What comes to mind is a) your comment that the
machine that you are having problems with is a VM and b) my aside comment
about another site running an LDM in a VM having bad problems with data being
written to disk, and the problem was eventually traced down to the underlying
SSD on which the file system being written to being worn out, so writes were
getting slower and slower (and SLOWER).

The other thing that I noticed (and reported in a previous email) was the
number of ssec.wisc.edu machines that are independently requesting all or part
of the CONDUIT feed.  Since CONDUIT volumes can reach up to and exceed 50 GB/hr,
having multiple machines REQUEST the same data _may_ be saturating the
network at/near SSEC.  I do not know that this is, in fact, the case, but we've
seen a number of institutions impose per-connection limiting on network 
bandwidth,
so this possibility (even if it is remote) came to mind as being a potential
cause of the problem you have been struggling with.  Along those lines is the
question of network throughput in the VM you are running.  Are there other
VMs that are also doing high volume network transfers on the same host hardware?
If yes, might the competition for network bandwidth be causing the problem with
CONDUIT ingestion in the VM?  Please remember that the GFS products _dominate_
the CONDUIT volume, so "just" REQUESTing GFS from CONDUIT is not much different
from REQUESTing all of CONDUIT in terms of data volumes.

Cheers,

Tom
--
****************************************************************************
Unidata User Support                                    UCAR Unidata Program
(303) 497-8642                                                 P.O. Box 3000
address@hidden                                   Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage                       http://www.unidata.ucar.edu
****************************************************************************


Ticket Details
===================
Ticket ID: CZE-486853
Department: Support LDM
Priority: Normal
Status: Closed
===================
NOTE: All email exchanges with Unidata User Support are recorded in the Unidata 
inquiry tracking system and then made publicly available through the web.  If 
you do not want to have your interactions made available in this way, you must 
let us know in each email you send to us.