[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[IDD #MIY-175774]: "WARN" Log Message in Concatenate NEXRAD2 Script

Hi Ziv,

re:
> I checked the ldmd.log just now and it is filled with the same "WARN"
> message.

OK, this _should_ mean that the writing of the NEXRAD2 products by the
FILE actions in your pattern-action file are not able to keep up with
the data that is flowing into your VM.   Perhaps you should run the
processing actions (FILE and then reconstitution) in/on a file system
that you know to be fast.  What I have in mind is in a RAM disk or
something equivalent.

re:
> I am aware of the Amazon NEXRAD2 files, but as we are setting up our own
> scientific data cloud my supervisor prefers to pull the data directly from
> the source.

OK.  Again, we are reconstituting the NEXRAD2 volume scans in AWS by doing
essentially the same thing as the procedure that we provided in the
pattern-action file actions and Perl script.

Question:

- can you collect some IO statistics from within your VM while the data
  ingest is running?

  You may have to install a package that contains 'iostat'.

re:
> I want to get the fully rendered NEXRAD2 scans which I then can import into
> our data cloud. However, I do not think the fully rendered scans are being
> produced due to the "WARN" message.

The WARN messages should indicate that some pieces are being missed since
they are not being processed out of your LDM queue fast enough.

re:
> What is also strange is that the Perl
> script does not seem to be producing the log files that it should be.
> 
> Have you had a chance to look over the script? Are there any other
> outputs/files you would like from my end. Thanks Tom!

A quick look at the hhmmssRadarII.pl Perl script indicates that you
should be able to specify the output data directory, log file
and log verbosity using flats to the script.  This would be preferable
to modifying the code in the script especially if one is not adept
at Perl.  The flags I am referring to are:

-d <data directory>
-l <log file>
-v

So, you could change the pattern-action file action that runs hhmmssRadarII.pl
from:

NEXRAD2   
^L2-BZIP2/(....)/([0-9][0-9][0-9][0-9][0-1][0-9][0-3][0-9])([0-2][0-9][0-5][0-9][0-9][0-9])/[0-9]*/([0-9]*)/E
        EXEC    perl etc/TDS/util/hhmmssRadarII.pl \1 \2 \3 \4

to:

NEXRAD2   
^L2-BZIP2/(....)/([0-9][0-9][0-9][0-9][0-1][0-9][0-3][0-9])([0-2][0-9][0-5][0-9][0-9][0-9])/[0-9]*/([0-9]*)/E
        EXEC    perl etc/TDS/util/hhmmssRadarII.pl -d <your output directory> 
-l /mnt/ldm/logs \1 \2 \3 \4

where:

<your output directory> needs to be replaced by the path where you want your 
data written.

Again, if the problem is that writing to /mnt is too slow to keep up with the
products that are being received, you need to change to using a faster file 
system
(like a RAM disk), and you would then need to copy the reconstituted volume 
scans
to the directory you want.

If you would like to discuss this "in person", you can call me at 303.497.8642.
I will be out of the office from 2 pm until about 4 pm MST today, but I will
be available after that.

Cheers,

Tom
--
****************************************************************************
Unidata User Support                                    UCAR Unidata Program
(303) 497-8642                                                 P.O. Box 3000
address@hidden                                   Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage                       http://www.unidata.ucar.edu
****************************************************************************


Ticket Details
===================
Ticket ID: MIY-175774
Department: Support IDD
Priority: Normal
Status: Closed