Re: [ldm-users] ucnids doesn't always decompress single site radar data

  • To: Daniel Vietor - NOAA Affiliate <dan.vietor@xxxxxxxx>
  • Subject: Re: [ldm-users] ucnids doesn't always decompress single site radar data
  • From: Gregory Grosshans <gregory.grosshans@xxxxxxxx>
  • Date: Thu, 25 Feb 2021 21:50:19 -0600
Hi Dan,

Yes, if you could send me your version of ucnids, it would be appreciated.

Thanks,
Gregg


On Thu, Feb 25, 2021 at 4:59 PM Daniel Vietor - NOAA Affiliate <
dan.vietor@xxxxxxxx> wrote:

> I know ucnids has gone through several iterations over the years. I
> checked my ingest for TLX NVW and it seems to be decompressing them well:
>
> -rw-r--r--. 1 ldm ops 8936 21-02-25 21:31:53 202102252122_NVW.nids
> -rw-r--r--. 1 ldm ops 2753 21-02-25 21:31:53 202102252122_NVW.nidz
> -rw-r--r--. 1 ldm ops 8796 21-02-25 21:41:35 202102252131_NVW.nids
> -rw-r--r--. 1 ldm ops 2727 21-02-25 21:41:35 202102252131_NVW.nidz
> -rw-r--r--. 1 ldm ops 8984 21-02-25 21:51:09 202102252141_NVW.nids
> -rw-r--r--. 1 ldm ops 2823 21-02-25 21:51:09 202102252141_NVW.nidz
> -rw-r--r--. 1 ldm ops 9090 21-02-25 22:00:46 202102252151_NVW.nids
> -rw-r--r--. 1 ldm ops 2888 21-02-25 22:00:46 202102252151_NVW.nidz
> -rw-r--r--. 1 ldm ops 9110 21-02-25 22:10:21 202102252200_NVW.nids
> -rw-r--r--. 1 ldm ops 2893 21-02-25 22:10:21 202102252200_NVW.nidz
> -rw-r--r--. 1 ldm ops 8200 21-02-25 22:20:01 202102252210_NVW.nids
> -rw-r--r--. 1 ldm ops 2688 21-02-25 22:20:01 202102252210_NVW.nidz
> -rw-r--r--. 1 ldm ops 7946 21-02-25 22:29:35 202102252219_NVW.nids
> -rw-r--r--. 1 ldm ops 2700 21-02-25 22:29:35 202102252219_NVW.nidz
> -rw-r--r--. 1 ldm ops 7864 21-02-25 22:39:10 202102252229_NVW.nids
> -rw-r--r--. 1 ldm ops 2569 21-02-25 22:39:10 202102252229_NVW.nidz
> -rw-r--r--. 1 ldm ops 7940 21-02-25 22:48:49 202102252239_NVW.nids
> -rw-r--r--. 1 ldm ops 2592 21-02-25 22:48:49 202102252239_NVW.nidz
>
> I don't know if there is a newer version out there? I could give you my
> ucnids source.
>
> Dan.
>
>
> On Thu, Feb 25, 2021 at 3:57 PM Gregory Grosshans via ldm-users <
> ldm-users@xxxxxxxxxxxxxxxx> wrote:
>
>> We are spinning up a new Vad Wind Profiler dataflow and using the
>> "ucnids" program to decompress the data via LDM.  Sometimes the output file
>> is decompressed, and other times it is NOT decompressed.  I don't see any
>> orphaned ucnids processes or entries in the ldmd.log file until I turn on
>> verbose logging.  The pqact entry is:
>>
>> NEXRAD3 ^SDUS3. .... ([0-3][0-9])([0-2][0-9])([0-5][0-9]).*/p(NVW)(...)
>>         PIPE    -close /nfsops/ops_users/ldmopsng1/util/ucnids -n -
>> radar/nflowvad/\5/prod48/(\1:yy)(\1:mm)\1\2.\3
>>
>> For the TLX 88D site in the past 24 hours there are about 151 files saved
>> (i.e. data is scrubbed after 24 hours).  Of these 151 files, 73 files (i.e.
>> ~half of the files) are not UNcompressed.
>>
>>
>> -rw-r--r--. 1 ldmopsng1 users 8598 Feb 25 20:53 21022520.43
>> -rw-r--r--. 1 ldmopsng1 users 8780 Feb 25 21:03 21022520.53
>> -rw-r--r--. 1 ldmopsng1 users 8968 Feb 25 21:12 21022521.03
>> -rw-r--r--. 1 ldmopsng1 users 3946 Feb 25 21:22 21022521.12  NOT
>> decompressed
>> -rw-r--r--. 1 ldmopsng1 users 3946 Feb 25 21:31 21022521.22  NOT
>> decompressed
>>
>>
>> The version of LDM is 6.13.11.
>>
>> When I have set ldmd.log to be verbose for the pqact processing this data
>> I see entries like this, but the file was NOT decompressed (2131Z
>> 21022521.22 file):
>>
>> 20210225T212217.975508Z pqact[11328]
>>  palt.c:processProduct:1340          INFO        2650 20210225212217.885403
>> NEXRAD3 1145245  SDUS34 KOUN 252112 /pNVWTLX
>> 20210225T212217.978774Z pqact[11328]
>>  filel.c:fl_removeAndFree:425        INFO  Deleting closed PIPE entry:
>> pid=6990, cmd="-close /nfsops/ops_users/ldmopsng1/util/ucnids -n -
>> radar/nflowvad/TLX/prod48/21022521.12"
>> 20210225T212217.978880Z pqact[6990]
>> filel.c:pipe_open:1798              INFO  Executing decoder
>> "/nfsops/ops_users/ldmopsng1/util/ucnids"
>>
>> It is odd the INFO for executing the decoder is in the log file after the
>> entry about LDM deleting the closed PIPE.
>>
>> Below are the log entries for the 2113Z 21022521.03 file that WAS
>> decompressed:
>>
>> 20210225T211240.207544Z pqact[11328]
>>  palt.c:processProduct:1340          INFO        2699 20210225211240.103390
>> NEXRAD3 1125246  SDUS34 KOUN 252103 /pNVWTLX
>> 20210225T211240.209361Z pqact[17988]
>>  filel.c:pipe_open:1798              INFO  Executing decoder
>> "/nfsops/ops_users/ldmopsng1/util/ucnids"
>> 20210225T211240.211086Z pqact[11328]
>>  filel.c:fl_removeAndFree:425        INFO  Deleting closed PIPE entry:
>> pid=17989, cmd="-close /nfsops/ops_users/ldmopsng1/util/ucnids -n -
>> radar/nflowvad/TLX/prod48/21022521.03"
>>
>>
>> So both log entries look similar, and I'm not sure why one file is
>> decompressed and another is not.  The only thing I see different is the
>> order of the log entries, the one out of order isn't DEcompressed, but I
>> suspect this has more to do the with system logging and is a coincidence.
>>
>> The limits for the number of open files has been increased on this
>> server.  The OPEN_MAX, ulimit -a, limit descriptors, was changed from 1021
>> for the LDM user to 40,000 soft limit and 40960 for the hard limit.
>>
>> Has anyone seen something like this with ucnids, or another type of
>> decoder, determined what was going on and have suggestions?
>>
>> Thanks,
>> Gregg
>>
>> --
>> **********************
>> *I'm primarily telecommuting.  If you need to reach me, call my work cell
>> 405-249-5310.*
>>
>>
>>
>>
>> *========================================================================Email
>> seems to be generating increasing inefficiencies in organizations.  I
>> learned from a manager a Stanford Computer Science professor no longer uses
>> email for communication, but uses SNAIL mail, telephone calls, and person
>> to person visits.  I'm considering the same.  405-325-2462*
>>
>> *Storm Prediction Center*
>>
>> *120 David L. Boren Blvd, Suite 2330Norman, OK 73072*
>> _______________________________________________
>> NOTE: All exchanges posted to Unidata maintained email lists are
>> recorded in the Unidata inquiry tracking system and made publicly
>> available through the web.  Users who post to any of the lists we
>> maintain are reminded to remove any personal information that they
>> do not want to be made public.
>>
>>
>> ldm-users mailing list
>> ldm-users@xxxxxxxxxxxxxxxx
>> For list information or to unsubscribe,  visit:
>> https://www.unidata.ucar.edu/mailing_lists/
>>
>
>
> --
> *Dan Vietor*
> *Senior Research Meteorologist*
> CIRA, Colorado State Univ
> Aviation Weather Center
> Kansas City, MO
> 816.584.7211
>
>

-- 
**********************
*I'm primarily telecommuting.  If you need to reach me, call my work cell
405-249-5310.*




*========================================================================Email
seems to be generating increasing inefficiencies in organizations.  I
learned from a manager a Stanford Computer Science professor no longer uses
email for communication, but uses SNAIL mail, telephone calls, and person
to person visits.  I'm considering the same.  405-325-2462*

*Storm Prediction Center*

*120 David L. Boren Blvd, Suite 2330Norman, OK 73072*
  • 2021 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the ldm-users archives: