[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[CONDUIT #SSL-833782]: LDM - conduit files >16MB being blocked?

> Greetings.
> 
> We're seeing a rather strange issue here at UND:
> any files being sent via CONDUIT that are over 16 megabytes
> in length are not showing up.  As a workaround we're fetching
> them via the telecom gateway, which is how we know they exist
> and are > 16MB.

John,

The data on the CONDUIT feed are the individual grib messages that exist within
the large files, so no product is bigger than a few hundred kbytes (eg not
a 16MB product such as the aggregate file on the ftp server). Therefore, nothing
can be blocked by the original file size of the NWS disk as all LDM products in
CONDUIT are individual grib products regardless.


> 
> Our primary ingestor is running LDM 6.3.0 under FreeBSD
> 4.9.  This machine feeds from bigbird.tamu.edu via I2 and most
> of the time runs little latency:
> 
> http://www.unidata.ucar.edu/cgi-bin/rtstats/iddstats_nc?CONDUIT+adiabat.rwic.und.edu
> 
> Any idea why this only seems to affect CONDUIT files over
> 16MB?

To the contrary, you feed shows large latencies during each 6 hourly data cycle.
Since the data is being transmitted as individual grib products, that is 
independent of
the size of the original data file. By comparison, the latency at bigbird is 
almost
always less than 60 seconds:
http://www.unidata.ucar.edu/cgi-bin/rtstats/iddstats_nc?CONDUIT+bigbird.tamu.edu
so, the holdup here is definitely attributable to your feed.

Your latency plot shows a consitent latency of 2500 seconds at the peak (with 
one
over 4000s). If Bigbird's queue holds less than 2500 seconds of data at those 
times,
it will get overwritten in their queue by newer data before you receive it.
The other possibility is that it takes your own system so long to process the 
data out of
your local queue that it would get overwritten locally by new data before your 
pqact
process gets around to fileing or piping it.

To get the data from bigbird to your machine faster, you may be able to use 
multiple
request lines. If you are currently feeding with a single CONDUIT pattern of 
".*",
you can try instead using 5 request lines each getting 20% of the data such as:
request CONDUIT "[09]$" bigbird.tamu.edu
request CONDUIT "[18]$" bigbird.tamu.edu
request CONDUIT "[27]$" bigbird.tamu.edu
request CONDUIT "[36]$" bigbird.tamu.edu
request CONDUIT "[45]$" bigbird.tamu.edu


If on the other hand, you are seeing adta appear on your disk with significant 
time
bewteen when it arrives in the data queue, then you need to split up your
pqact processing so that the pqact processes can keep up with the rate of data 
coming in to your system.

The last option is to cut down on the data you are receiving if you are not
utilizing all of the data in the CONDUIT feed. This would not solve your
underlying problem with keeping up with the volume however and that may
still become an issue in the future as more data are added to CONDUIT.

Steve Chiswell
Unidata User Support


> 
> Thanks.
> 
> =========================================================================
> John Nordlie                    |     Regional Weather Information Center
> |              University of North Dakota
> =========================================================================
> 
> 
> Greetings.
> 
> We're seeing a rather strange issue here at UND:
> any files being sent via CONDUIT that are over 16 megabytes
> in length are not showing up.  As a workaround we're fetching
> them via the telecom gateway, which is how we know they exist
> and are > 16MB.
> 
> Our primary ingestor is running LDM 6.3.0 under FreeBSD
> 4.9.  This machine feeds from bigbird.tamu.edu via I2 and most
> of the time runs little latency:
> 
> http://www.unidata.ucar.edu/cgi-bin/rtstats/iddstats_nc?CONDUIT+adiabat.rwic.und.edu
> 
> Any idea why this only seems to affect CONDUIT files over
> 16MB?
> 
> Thanks.
> 
> =========================================================================
> John Nordlie                    |     Regional Weather Information Center
> |              University of North Dakota
> =========================================================================
> 
> 


Ticket Details
===================
Ticket ID: SSL-833782
Department: Support CONDUIT
Priority: Normal
Status: Closed