Re: [ldm-users] level2 via LDM

  • To: Clinton Rowe <crowe1@xxxxxxx>, "Herzmann, Daryl E [AGRON]" <akrherz@xxxxxxxxxxx>, "ldm-users@xxxxxxxxxxxxxxxx" <ldm-users@xxxxxxxxxxxxxxxx>
  • Subject: Re: [ldm-users] level2 via LDM
  • From: Victor Gensini <vgensini@xxxxxxx>
  • Date: Sun, 8 Nov 2020 23:33:43 +0000
  • Arc-authentication-results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=niu.edu; dmarc=pass action=none header.from=niu.edu; dkim=pass header.d=niu.edu; arc=none
  • Arc-message-signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Yv+8gxetGyfzVBRYkFARp9lRNPoMsUamcGEIfOavQRU=; b=BB/jbhXnfdYzkNAd1IKgcsqxtVeHklmrI/LnC9unkXLdIHBrarg5+sdSo3nZGvBHlYSU2sbnnYb69oKmugZVeTROvzjPfS50Sx6wbtKUmVtg5EmqEI44/gxhwyiw3eSRc9ojND+Nhp6soGIbL5oj3U4WakVtaxLSZTxM1ZtC7J+x1itChzNGU+7PBNDNJSCU3J5gCyXNRSlSC9bno6mzZCD4jNIaT7lC1x+n/WQj070t6bJeRW6zNyiXEhQ6xnuGBnK46r0hxnIt/dWS0OgmBnQqEkNSLOxWwMBs9Ow/sV4+hmDfIl5lUfi/ryztBUZCGHAypBXr3qEAGC614O2QJQ==
  • Arc-seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=MpmZkZdrIFUaFTk8lLZD7mjaS2ei1geydS+63sonV+opZ8sOHokWVNIYxVdVlcoFzhkpbV1lf3rtZ0pROZfHLRitVz7mDYGBVWmf7ueJxqZtLWOmtOyPo1Q8bs7xv4tq4KFj/afhEeM2WsPpO0iroZd3+xeqll9a31F6Y4Uv93+6hP5h3xkQbtIJge04VArvuzTn+WjlZrYJsUZj61ejkvHcK6EiAJ/i2J5qtc2xgSII66Eih/BWtGYqqH1WFr5Le3d9FGGCNnVXYGXz7ScsC++t0rvssMU2n1F/106PuMS33VW1vsxpPxkDKHpqCZvF9zEsureOy4OqcBhnYave4w==
  • Authentication-results: unl.edu; dkim=none (message not signed) header.d=none;unl.edu; dmarc=none action=none header.from=niu.edu;
Hi Clint!

No resolution as GEMPAK is no longer supported by UNIDATA (only the community 
via GitHub). I switched all of my scripts to use the coarser resolution level3 
data 😕👎

I will let you know if there is a fix for SAILS_1 in GEMPAK.

Best,


-----------------------------------------------

Vittorio (Victor) A. Gensini, Ph.D., CCM

Associate Professor

Department of Geographic and Atmospheric Sciences

Northern Illinois University

DeKalb, IL 60115

https://atlas.niu.edu<http://atlas.niu.edu>

https://wcs.niu.edu<http://wcs.niu.edu/>

<http://a><http://atlas.niu.edu>

________________________________
From: Clinton Rowe <crowe1@xxxxxxx>
Sent: Sunday, November 8, 2020 5:29 PM
To: Victor Gensini <vgensini@xxxxxxx>; Herzmann, Daryl E [AGRON] 
<akrherz@xxxxxxxxxxx>; ldm-users@xxxxxxxxxxxxxxxx <ldm-users@xxxxxxxxxxxxxxxx>
Subject: RE: level2 via LDM


Victor, et al,



Was there any resolution of this issue? I didn’t see anything in the mailing 
lists and I’m still seeing seg faults with gpnexr2_gf (LDM 6.13.11 and GEMPAK 
v7.? – not sure of sub-version, but definitely not the latest) for some sites.



Thanks,

Clint



[https://ucomm.unl.edu/images/email-signature/Nebraska_N_RGB_small_on_white.gif]

Clinton Rowe

Professor & Chair

University of Nebraska–Lincoln

Earth and Atmospheric Sciences<https://eas.unl.edu/>

402.472.1946<tel:4024721946>





From: ldm-users <ldm-users-bounces@xxxxxxxxxxxxxxxx> On Behalf Of Victor Gensini
Sent: Tuesday, September 29, 2020 12:13 PM
To: Herzmann, Daryl E [AGRON] <akrherz@xxxxxxxxxxx>; ldm-users@xxxxxxxxxxxxxxxx
Subject: Re: [ldm-users] level2 via LDM



Ohhhh..the fun continues. I have been troubleshooting offline with Daryl.



So, from what I can tell, the level2 files for some radars (for whatever 
reason) are transmitting two volume scans per file. The radars in question 
appear random, but it might be related to a recent upgrade in ROC software or 
something. EDIT: the files that blow things up are SAILS_1!!!



The file sizes are double because (surprise!) there are two timesteps due to 
this SAILS_1.



Copy of my pqact here, but I assume this is going to create somewhat of a 
headache for others as well.



https://atlas.niu.edu/vgensini/temp/pqact.split.radar<https://urldefense.proofpoint.com/v2/url?u=https-3A__atlas.niu.edu_vgensini_temp_pqact.split.radar&d=DwMFAg&c=Cu5g146wZdoqVuKpTNsYHeFX_rg6kWhlkLF8Eft-wwo&r=ubRAum_abJ_VeyWwkZyVzQ&m=YGiYowSlf11EYs4Iv7UoJRkpuOr3jxOGiNQ-V92hCYk&s=H-HgSFBgoOCg6ZfObthZZv9PwLNb1zxSSicnkQrrcKA&e=>



-----------------------------------------------

Vittorio (Victor) A. Gensini, Ph.D., CCM

Associate Professor

Department of Geographic and Atmospheric Sciences

Northern Illinois University

DeKalb, IL 60115

https://atlas.niu.edu<https://urldefense.proofpoint.com/v2/url?u=http-3A__atlas.niu.edu&d=DwMFAg&c=Cu5g146wZdoqVuKpTNsYHeFX_rg6kWhlkLF8Eft-wwo&r=ubRAum_abJ_VeyWwkZyVzQ&m=YGiYowSlf11EYs4Iv7UoJRkpuOr3jxOGiNQ-V92hCYk&s=EetUJnuPXBeYKK2_H2bIyPWa79sQE9SwHaKxjKqYqZE&e=>

https://wcs.niu.edu<https://urldefense.proofpoint.com/v2/url?u=http-3A__wcs.niu.edu_&d=DwMFAg&c=Cu5g146wZdoqVuKpTNsYHeFX_rg6kWhlkLF8Eft-wwo&r=ubRAum_abJ_VeyWwkZyVzQ&m=YGiYowSlf11EYs4Iv7UoJRkpuOr3jxOGiNQ-V92hCYk&s=P5g8d4fiypeLWleTEvEUChZbM0smOp4Dt33hnAVik6s&e=>

________________________________

From: Herzmann, Daryl E [AGRON] 
<akrherz@xxxxxxxxxxx<mailto:akrherz@xxxxxxxxxxx>>
Sent: Tuesday, September 29, 2020 11:38 AM
To: Victor Gensini <vgensini@xxxxxxx<mailto:vgensini@xxxxxxx>>; 
ldm-users@xxxxxxxxxxxxxxxx<mailto:ldm-users@xxxxxxxxxxxxxxxx> 
<ldm-users@xxxxxxxxxxxxxxxx<mailto:ldm-users@xxxxxxxxxxxxxxxx>>
Subject: Re: level2 via LDM



Well Howdy,

This sounds like a queue that is not holding an hour's worth of data and thus 
getting duplicated / triplicated product transmisions.  My questions:

1) Is your LDM queue holding an hour's worth of data `pqmon`?  Do you know if 
your upstream sources are holding an hour's worth of data?  Is your LDM logging 
anything exciting about switching upstreams etc ?

2) Do the file write timestamps on the files make sense?  Such that the write 
time coincides with the end of the VCP transmission?

3) Do your file byte sizes match what is found on the files at AWS / my 
realtime site?

https://mesonet-nexrad.agron.iastate.edu/level2/raw/KDVN/<https://urldefense.proofpoint.com/v2/url?u=https-3A__mesonet-2Dnexrad.agron.iastate.edu_level2_raw_KDVN_&d=DwMFAg&c=Cu5g146wZdoqVuKpTNsYHeFX_rg6kWhlkLF8Eft-wwo&r=ubRAum_abJ_VeyWwkZyVzQ&m=YGiYowSlf11EYs4Iv7UoJRkpuOr3jxOGiNQ-V92hCYk&s=kPnL1ECWMi6TOqj_F5_waUineWOSRDGN44ssfsddL5k&e=>

Do you have a recent example (today / yesterday) of a 2x / 3x file?  Can you 
share that file for others to interrogate!

daryl

--
/**
 * daryl herzmann
 * Systems Analyst III -- Iowa Environmental Mesonet
 * 
https://mesonet.agron.iastate.edu<https://urldefense.proofpoint.com/v2/url?u=https-3A__mesonet.agron.iastate.edu&d=DwMFAg&c=Cu5g146wZdoqVuKpTNsYHeFX_rg6kWhlkLF8Eft-wwo&r=ubRAum_abJ_VeyWwkZyVzQ&m=YGiYowSlf11EYs4Iv7UoJRkpuOr3jxOGiNQ-V92hCYk&s=5-OS1Jc88tlN5MfD8mWw3gl7m9ZbL1oE2rQYaknZ-Kk&e=>
 */

________________________________________
From: ldm-users 
<ldm-users-bounces@xxxxxxxxxxxxxxxx<mailto:ldm-users-bounces@xxxxxxxxxxxxxxxx>> 
on behalf of Victor Gensini <vgensini@xxxxxxx<mailto:vgensini@xxxxxxx>>
Sent: Tuesday, September 29, 2020 11:30 AM
To: ldm-users@xxxxxxxxxxxxxxxx<mailto:ldm-users@xxxxxxxxxxxxxxxx>
Subject: [ldm-users] level2 via LDM

Tom/Steve and LDM junkies,

Have there been any recent changes to the format of LEVEL2 radar files coming 
over LDM that I perhaps missed?

A couple of weeks ago, I had a couple scripts start to break that are using 
gpnexr2_gf with:

Program received signal SIGSEGV: Segmentation fault - invalid memory reference.

Backtrace for this error:
#0  0x7FD1DF3C1697
#1  0x7FD1DF3C1CDE
#2  0x7FD1DE8BC3AF
#3  0x7FD1DE9F0B54
#4  0x4BCFCB
#5  0x4B8E59
#6  0x4B93C7
#7  0x42A510
#8  0x4288FE
#9  0x40F5A7
#10  0x40903D
#11  0x40554A
#12  0x405588
#13  0x7FD1DE8A8504
#14  0x4046D7
#15  0xFFFFFFFFFFFFFFFF
Segmentation fault (core dumped)

A quick peek of the file size for these stations (relative to others) indicate 
nearly a doubling (in some cases tripling) of the file size versus other 
stations that are still working. As an example, compare your file sizes today 
for KDVN and KILX, both of which are in VCP 35.

I'm using the standard hhmmssRadarII perl script to create the volumes off of 
LDM. Again, no issues for some sites, and then this memory error is popping up 
for others. I sense a change in file contents or format...

Any/all ideas welcome.

Victor


-----------------------------------------------

Vittorio (Victor) A. Gensini, Ph.D., CCM

Associate Professor

Department of Geographic and Atmospheric Sciences

Northern Illinois University

DeKalb, IL 60115

https://atlas.niu.edu<http://atlas.niu.edu<https://urldefense.proofpoint.com/v2/url?u=https-3A__atlas.niu.edu-253Chttp-3A__atlas.niu.edu&d=DwMFAg&c=Cu5g146wZdoqVuKpTNsYHeFX_rg6kWhlkLF8Eft-wwo&r=ubRAum_abJ_VeyWwkZyVzQ&m=YGiYowSlf11EYs4Iv7UoJRkpuOr3jxOGiNQ-V92hCYk&s=6tIOGB82VkkW2FsCxx4kQr0wKMe-PKswUcyKf_arEyw&e=>>

https://wcs.niu.edu<http://wcs.niu.edu/<https://urldefense.proofpoint.com/v2/url?u=http-3A__wcs.niu.edu_&d=DwMFAg&c=Cu5g146wZdoqVuKpTNsYHeFX_rg6kWhlkLF8Eft-wwo&r=ubRAum_abJ_VeyWwkZyVzQ&m=YGiYowSlf11EYs4Iv7UoJRkpuOr3jxOGiNQ-V92hCYk&s=P5g8d4fiypeLWleTEvEUChZbM0smOp4Dt33hnAVik6s&e=>>

<http://a><http://atlas.niu.edu<https://urldefense.proofpoint.com/v2/url?u=http-3A__a-253E-253Chttp-3A__atlas.niu.edu&d=DwMFAg&c=Cu5g146wZdoqVuKpTNsYHeFX_rg6kWhlkLF8Eft-wwo&r=ubRAum_abJ_VeyWwkZyVzQ&m=YGiYowSlf11EYs4Iv7UoJRkpuOr3jxOGiNQ-V92hCYk&s=MsqfuBjzOnKYnlkZGSCSldp5xgRWmSJ4wR_Z5AInErg&e=>>

GIF image

  • 2020 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the ldm-users archives: