[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

20040712: FILEing NNEXRAD/NIDS on Linux reiserfs



>From: Harry Edmon <address@hidden>
>Organization: University of Washington
>Keywords: 200407122128.i6CLScaW020180 LDM FILE NNEXRAD reiserfs

Hi Harry,

>I am in the process of setting up a new weather data ingestor.  I am using the
>suggested pqact files for gempak.  I notice that the process dealing with the
>NIDS data falls behind in its processing.  Everything else is okay.  I am
>wondering if the problem is the large number of files this process has to
>create.

It is typically a bottleneck in the file I/O when one is ingesting
all Level III products and FILEing them to disk.

>It is writing to a reiserfs.  Have you seen this behavior?

Yes, we have, but not on reiserfs since we havn't used this file
system.  We found that it is hard to keep up with the FILEing of all
NEXRAD Level III products to a disk that is not a striped RAID.

If you have one ldmd.conf pqact invocation processing just the NNEXRAD
data (it is not looking at any other feed type), and it is falling
behind, you might try splitting the processing load among several
pqacts like is being done for the CRAFT data.  This may or may not
help.  Writing to a fast, striped RAID really does help -- that is how
we process the Level III data here.

Cheers,

Tom
--
NOTE: All email exchanges with Unidata User Support are recorded in the
Unidata inquiry tracking system and then made publically available
through the web.  If you do not want to have your interactions made
available in this way, you must let us know in each email you send to us.