[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[LDM #YZC-779273]: One product in a data feed stops writing to disk

Gilbert,

What was the product that stopped being written to disk?

What REQUEST entry was responsible for that product?

Are there any relevant log messages from the corresponding downstream LDM?

Are there any relevant log messages from the corresponding upstream LDM-s?

> I have another head-scratcher for you. The LDM doesn't crash, but an MRMS
> product just suddenly stops being written to disk. Other MRMS products are
> retrieved and written fine. An LDM restart corrects the issue. This has
> happened to us twice.
> 
> 1326Z was the last MRMS BR QC product received until the LDM was
> restarted at 1818Z.
> Any ideas?
> 
> [ldm@nfs-central1-b logs]$ grep MRMS ldmd.log.1 | grep 20160615T13
> 20160615T130152.215128Z pqact[16167] ERROR filel.c:2790:reap() Deleting
> failed EXEC entry "/home/ldm/decoders/store_MRMS.sh
> RadarOnly_QPE_24H_20160615_130000.grib2.gz"
> 20160615T133031.229931Z 140.90.98.15[16216] NOTE
> requester6.c:588:req6_new() LDM-6 desired product-class:
> 20160615124928.229854 TS_ENDT {{EXP,
> "/data/realtime/outgoing/grib2/MRMS_SeamlessHSR_00.00_"},{NONE,
> "SIG=6accb55a548fd87660623ab7b2543920"}}
> 20160615T133032.218429Z 140.90.98.15[16223] NOTE
> requester6.c:588:req6_new() LDM-6 desired product-class:
> 20160615124929.218361 TS_ENDT {{EXP,
> "/data/realtime/outgoing/grib2/GUAM/MRMS_MergedReflectivityQComposite_00.50"},{NONE,
> "SIG=d668c360b57eb76a14b863a1e44b50ba"}}
> 20160615T133032.318353Z 140.90.98.15[16225] NOTE
> requester6.c:588:req6_new() LDM-6 desired product-class:
> 20160615124929.318264 TS_ENDT {{EXP,
> "/data/realtime/outgoing/grib2/CARIB/MRMS_MergedReflectivityQComposite_00.50"},{NONE,
> "SIG=9ee686630c92f93cf7937a7dec581641"}}
> 20160615T133033.227060Z 140.90.98.15[16224] NOTE
> requester6.c:588:req6_new() LDM-6 desired product-class:
> 20160615124930.226986 TS_ENDT {{EXP,
> "/data/realtime/outgoing/grib2/ALASKA/MRMS_MergedReflectivityQComposite_00.50"},{NONE,
> "SIG=56f035650081f800bd73ad6f132ba1e2"}}
> 20160615T133033.279336Z 140.90.98.15[16222] NOTE
> requester6.c:588:req6_new() LDM-6 desired product-class:
> 20160615124930.279269 TS_ENDT {{EXP,
> "/data/realtime/outgoing/grib2/HAWAII/MRMS_MergedReflectivityQComposite_00.50"},{NONE,
> "SIG=69ebb81bce2442a998f4b9a260f1dfed"}}
> 20160615T133038.421835Z 140.90.98.15[16210] NOTE
> requester6.c:588:req6_new() LDM-6 desired product-class:
> 20160615124935.421779 TS_ENDT {{EXP,
> "/data/realtime/outgoing/grib2/MRMS_MESH_Max_60min_00.50_"},{NONE,
> "SIG=76e3d10b17fa827f132467b9d8e30c47"}}
> 20160615T133042.324873Z 140.90.98.15[16207] NOTE
> requester6.c:588:req6_new() LDM-6 desired product-class:
> 20160615124939.324798 TS_ENDT {{EXP,
> "/data/realtime/outgoing/grib2/MRMS_RadarOnly_QPE_01H_00.00_"},{NONE,
> "SIG=cfba2b2c187876f65e810950ee6814bf"}}
> 20160615T133042.327465Z 140.90.98.15[16213] NOTE
> requester6.c:588:req6_new() LDM-6 desired product-class:
> 20160615124939.327403 TS_ENDT {{EXP,
> "/data/realtime/outgoing/grib2/MRMS_MergedAzShear_"},{NONE,
> "SIG=348eaa70b858fccdecf35cc1ab4852fa"}}
> 20160615T133044.233619Z 140.90.98.15[16217] NOTE
> requester6.c:588:req6_new() LDM-6 desired product-class:
> 20160615124941.233552 TS_ENDT {{EXP,
> "/data/realtime/outgoing/grib2/MRMS_EchoTop_18_00.50_"},{NONE,
> "SIG=47941773b52699f2d533abbfbd313184"}}
> 20160615T133044.238398Z 140.90.98.15[16211] NOTE
> requester6.c:588:req6_new() LDM-6 desired product-class:
> 20160615124941.238356 TS_ENDT {{EXP,
> "/data/realtime/outgoing/grib2/MRMS_MESH_Max_1440min_00.50_"},{NONE,
> "SIG=2a5df5275b3604320f5fa252f70a24e7"}}
> 20160615T133046.112519Z 140.90.98.15[16218] NOTE
> requester6.c:588:req6_new() LDM-6 desired product-class:
> 20160615124943.112462 TS_ENDT {{EXP,
> "/data/realtime/outgoing/grib2/MRMS_VIL_00.50_"},{NONE,
> "SIG=459d80d431cacf9fe9cb66aecb343720"}}
> 20160615T133046.528099Z 140.90.98.15[16212] NOTE
> requester6.c:588:req6_new() LDM-6 desired product-class:
> 20160615124943.528043 TS_ENDT {{EXP,
> "/data/realtime/outgoing/grib2/MRMS_MergedReflectivityQCComposite_00.50_"},{NONE,
> "SIG=49f146ce1bf8ae734f355948de833d28"}}
> 20160615T133047.301507Z 140.90.98.15[16219] NOTE
> requester6.c:588:req6_new() LDM-6 desired product-class:
> 20160615124944.301448 TS_ENDT {{EXP,
> "/data/realtime/outgoing/grib2/MRMS_LightningProbabilityNext30min_scale_0_"},{NONE,
> "SIG=de91a572dc190e49122da15be65e8962"}}
> 20160615T133049.444297Z 140.90.98.15[16209] NOTE
> requester6.c:588:req6_new() LDM-6 desired product-class:
> 20160615124946.444240 TS_ENDT {{EXP,
> "/data/realtime/outgoing/grib2/MRMS_MESH_00.50_"},{NONE,
> "SIG=9d69a1d104b97fb136e5453f265c9be3"}}
> 20160615T133051.296322Z 140.90.98.15[16214] NOTE
> requester6.c:588:req6_new() LDM-6 desired product-class:
> 20160615124948.296267 TS_ENDT {{EXP,
> "/data/realtime/outgoing/grib2/MRMS_RotationTrackML60min_00.50_"},{NONE,
> "SIG=3ae652224617d867f5667b845033f1dd"}}
> 20160615T133053.240838Z 140.90.98.15[16220] NOTE
> requester6.c:588:req6_new() LDM-6 desired product-class:
> 20160615124950.240789 TS_ENDT {{EXP,
> "/data/realtime/outgoing/grib2/MRMS_RadarOnly_QPE_24H_00.00_"},{NONE,
> "SIG=bbc3f1c2e289ffaf4193440a1b992353"}}
> 20160615T133053.265838Z 140.90.98.15[16215] NOTE
> requester6.c:588:req6_new() LDM-6 desired product-class:
> 20160615124950.265745 TS_ENDT {{EXP,
> "/data/realtime/outgoing/grib2/MRMS_RadarOnly_QPE_12H_00.00_"},{NONE,
> "SIG=5e16b72421bd767fe303f3a6c503e33b"}}
> 20160615T134047.650860Z 217.40.148.146.bc.googleusercontent.com(feed)[13834]
> NOTE uldb.c:1533:sm_vetUpstreamLdm() Terminated obsolete upstream LDM
> (addr=146.148.40.217, pid=31101, vers=6, type=feeder, mode=alternate,
> sub=(20160615103056.207061 TS_ENDT {{EXP,
> "MRMS_SeamlessHSR_00.00_.*000.grib2.gz"}}))
> 20160615T134048.653597Z 217.40.148.146.bc.googleusercontent.com(feed)[13834]
> NOTE up6.c:445:up6_run() Starting Up(6.13.1/6): 20160615124047.617850
> TS_ENDT {{EXP, "MRMS_SeamlessHSR_00.00_.*000.grib2.gz"}},
> SIG=4087cd7c4c14e072d98d2cf33b081fb5, Primary
> 20160615T135053.749171Z 217.40.148.146.bc.googleusercontent.com(feed)[28615]
> NOTE uldb.c:1533:sm_vetUpstreamLdm() Terminated obsolete upstream LDM
> (addr=146.148.40.217, pid=13834, vers=6, type=feeder, mode=primary,
> sub=(20160615124047.617850 TS_ENDT {{EXP,
> "MRMS_SeamlessHSR_00.00_.*000.grib2.gz"}}))
> 20160615T135054.752344Z 217.40.148.146.bc.googleusercontent.com(feed)[28615]
> NOTE up6.c:445:up6_run() Starting Up(6.13.1/6): 20160615125053.715553
> TS_ENDT {{EXP, "MRMS_SeamlessHSR_00.00_.*000.grib2.gz"}},
> SIG=a26e0cdf1f0cebe05a364e73ef7e4543, Alternate
> 
> There are no messages that mention RALA, which is the product we started to
> miss beginning with these logs.
> 
> [ldm@nfs-central1-b logs]$ grep 20160615T13 ldmd.log.1 | grep 140.90.98.15
> 20160615T133031.229765Z 140.90.98.15[16216] NOTE error.c:236:err_log()
> Upstream LDM died: pid=31053
> 20160615T133031.229931Z 140.90.98.15[16216] NOTE
> requester6.c:588:req6_new() LDM-6 desired product-class:
> 20160615124928.229854 TS_ENDT {{EXP,
> "/data/realtime/outgoing/grib2/MRMS_SeamlessHSR_00.00_"},{NONE,
> "SIG=6accb55a548fd87660623ab7b2543920"}}
> 20160615T133031.311790Z 140.90.98.15[16216] NOTE
> requester6.c:311:make_request() Upstream LDM-6 on 140.90.98.15 is willing
> to be a primary feeder
> 20160615T133032.218017Z 140.90.98.15[16223] NOTE error.c:236:err_log()
> Upstream LDM died: pid=31294
> 20160615T133032.218429Z 140.90.98.15[16223] NOTE
> requester6.c:588:req6_new() LDM-6 desired product-class:
> 20160615124929.218361 TS_ENDT {{EXP,
> "/data/realtime/outgoing/grib2/GUAM/MRMS_MergedReflectivityQComposite_00.50"},{NONE,
> "SIG=d668c360b57eb76a14b863a1e44b50ba"}}
> 20160615T133032.282274Z 140.90.98.15[16223] NOTE
> requester6.c:311:make_request() Upstream LDM-6 on 140.90.98.15 is willing
> to be a primary feeder
> 20160615T133032.318191Z 140.90.98.15[16225] NOTE error.c:236:err_log()
> Upstream LDM died: pid=31316
> 20160615T133032.318353Z 140.90.98.15[16225] NOTE
> requester6.c:588:req6_new() LDM-6 desired product-class:
> 20160615124929.318264 TS_ENDT {{EXP,
> "/data/realtime/outgoing/grib2/CARIB/MRMS_MergedReflectivityQComposite_00.50"},{NONE,
> "SIG=9ee686630c92f93cf7937a7dec581641"}}
> 20160615T133032.382661Z 140.90.98.15[16225] NOTE
> requester6.c:311:make_request() Upstream LDM-6 on 140.90.98.15 is willing
> to be a primary feeder
> 20160615T133033.226902Z 140.90.98.15[16224] NOTE error.c:236:err_log()
> Upstream LDM died: pid=31372
> 20160615T133033.227060Z 140.90.98.15[16224] NOTE
> requester6.c:588:req6_new() LDM-6 desired product-class:
> 20160615124930.226986 TS_ENDT {{EXP,
> "/data/realtime/outgoing/grib2/ALASKA/MRMS_MergedReflectivityQComposite_00.50"},{NONE,
> "SIG=56f035650081f800bd73ad6f132ba1e2"}}
> 20160615T133033.279198Z 140.90.98.15[16222] NOTE error.c:236:err_log()
> Upstream LDM died: pid=31344
> 20160615T133033.279336Z 140.90.98.15[16222] NOTE
> requester6.c:588:req6_new() LDM-6 desired product-class:
> 20160615124930.279269 TS_ENDT {{EXP,
> "/data/realtime/outgoing/grib2/HAWAII/MRMS_MergedReflectivityQComposite_00.50"},{NONE,
> "SIG=69ebb81bce2442a998f4b9a260f1dfed"}}
> 20160615T133033.290215Z 140.90.98.15[16224] NOTE
> requester6.c:311:make_request() Upstream LDM-6 on 140.90.98.15 is willing
> to be a primary feeder
> 20160615T133033.342906Z 140.90.98.15[16222] NOTE
> requester6.c:311:make_request() Upstream LDM-6 on 140.90.98.15 is willing
> to be a primary feeder
> 20160615T133038.421724Z 140.90.98.15[16210] NOTE error.c:236:err_log()
> Upstream LDM died: pid=30825
> 20160615T133038.421835Z 140.90.98.15[16210] NOTE
> requester6.c:588:req6_new() LDM-6 desired product-class:
> 20160615124935.421779 TS_ENDT {{EXP,
> "/data/realtime/outgoing/grib2/MRMS_MESH_Max_60min_00.50_"},{NONE,
> "SIG=76e3d10b17fa827f132467b9d8e30c47"}}
> 20160615T133038.485341Z 140.90.98.15[16210] NOTE
> requester6.c:311:make_request() Upstream LDM-6 on 140.90.98.15 is willing
> to be a primary feeder
> 20160615T133042.324728Z 140.90.98.15[16207] NOTE error.c:236:err_log()
> Upstream LDM died: pid=30590
> 20160615T133042.324873Z 140.90.98.15[16207] NOTE
> requester6.c:588:req6_new() LDM-6 desired product-class:
> 20160615124939.324798 TS_ENDT {{EXP,
> "/data/realtime/outgoing/grib2/MRMS_RadarOnly_QPE_01H_00.00_"},{NONE,
> "SIG=cfba2b2c187876f65e810950ee6814bf"}}
> 20160615T133042.327330Z 140.90.98.15[16213] NOTE error.c:236:err_log()
> Upstream LDM died: pid=30798
> 20160615T133042.327465Z 140.90.98.15[16213] NOTE
> requester6.c:588:req6_new() LDM-6 desired product-class:
> 20160615124939.327403 TS_ENDT {{EXP,
> "/data/realtime/outgoing/grib2/MRMS_MergedAzShear_"},{NONE,
> "SIG=348eaa70b858fccdecf35cc1ab4852fa"}}
> 20160615T133042.388192Z 140.90.98.15[16207] NOTE
> requester6.c:311:make_request() Upstream LDM-6 on 140.90.98.15 is willing
> to be a primary feeder
> 20160615T133042.391303Z 140.90.98.15[16213] NOTE
> requester6.c:311:make_request() Upstream LDM-6 on 140.90.98.15 is willing
> to be a primary feeder
> 20160615T133044.233505Z 140.90.98.15[16217] NOTE error.c:236:err_log()
> Upstream LDM died: pid=30785
> 20160615T133044.233619Z 140.90.98.15[16217] NOTE
> requester6.c:588:req6_new() LDM-6 desired product-class:
> 20160615124941.233552 TS_ENDT {{EXP,
> "/data/realtime/outgoing/grib2/MRMS_EchoTop_18_00.50_"},{NONE,
> "SIG=47941773b52699f2d533abbfbd313184"}}
> 20160615T133044.238314Z 140.90.98.15[16211] NOTE error.c:236:err_log()
> Upstream LDM died: pid=30994
> 20160615T133044.238398Z 140.90.98.15[16211] NOTE
> requester6.c:588:req6_new() LDM-6 desired product-class:
> 20160615124941.238356 TS_ENDT {{EXP,
> "/data/realtime/outgoing/grib2/MRMS_MESH_Max_1440min_00.50_"},{NONE,
> "SIG=2a5df5275b3604320f5fa252f70a24e7"}}
> 20160615T133044.297620Z 140.90.98.15[16217] NOTE
> requester6.c:311:make_request() Upstream LDM-6 on 140.90.98.15 is willing
> to be a primary feeder
> 20160615T133044.302390Z 140.90.98.15[16211] NOTE
> requester6.c:311:make_request() Upstream LDM-6 on 140.90.98.15 is willing
> to be a primary feeder
> 20160615T133046.112392Z 140.90.98.15[16218] NOTE error.c:236:err_log()
> Upstream LDM died: pid=31097
> 20160615T133046.112519Z 140.90.98.15[16218] NOTE
> requester6.c:588:req6_new() LDM-6 desired product-class:
> 20160615124943.112462 TS_ENDT {{EXP,
> "/data/realtime/outgoing/grib2/MRMS_VIL_00.50_"},{NONE,
> "SIG=459d80d431cacf9fe9cb66aecb343720"}}
> 20160615T133046.175881Z 140.90.98.15[16218] NOTE
> requester6.c:311:make_request() Upstream LDM-6 on 140.90.98.15 is willing
> to be a primary feeder
> 20160615T133046.527958Z 140.90.98.15[16212] NOTE error.c:236:err_log()
> Upstream LDM died: pid=31139
> 20160615T133046.528099Z 140.90.98.15[16212] NOTE
> requester6.c:588:req6_new() LDM-6 desired product-class:
> 20160615124943.528043 TS_ENDT {{EXP,
> "/data/realtime/outgoing/grib2/MRMS_MergedReflectivityQCComposite_00.50_"},{NONE,
> "SIG=49f146ce1bf8ae734f355948de833d28"}}
> 20160615T133046.590428Z 140.90.98.15[16212] NOTE
> requester6.c:311:make_request() Upstream LDM-6 on 140.90.98.15 is willing
> to be a primary feeder
> 20160615T133047.301392Z 140.90.98.15[16219] NOTE error.c:236:err_log()
> Upstream LDM died: pid=31315
> 20160615T133047.301507Z 140.90.98.15[16219] NOTE
> requester6.c:588:req6_new() LDM-6 desired product-class:
> 20160615124944.301448 TS_ENDT {{EXP,
> "/data/realtime/outgoing/grib2/MRMS_LightningProbabilityNext30min_scale_0_"},{NONE,
> "SIG=de91a572dc190e49122da15be65e8962"}}
> 20160615T133047.365182Z 140.90.98.15[16219] NOTE
> requester6.c:311:make_request() Upstream LDM-6 on 140.90.98.15 is willing
> to be a primary feeder
> 20160615T133049.444185Z 140.90.98.15[16209] NOTE error.c:236:err_log()
> Upstream LDM died: pid=31236
> 20160615T133049.444297Z 140.90.98.15[16209] NOTE
> requester6.c:588:req6_new() LDM-6 desired product-class:
> 20160615124946.444240 TS_ENDT {{EXP,
> "/data/realtime/outgoing/grib2/MRMS_MESH_00.50_"},{NONE,
> "SIG=9d69a1d104b97fb136e5453f265c9be3"}}
> 20160615T133049.508170Z 140.90.98.15[16209] NOTE
> requester6.c:311:make_request() Upstream LDM-6 on 140.90.98.15 is willing
> to be a primary feeder
> 20160615T133051.296206Z 140.90.98.15[16214] NOTE error.c:236:err_log()
> Upstream LDM died: pid=31069
> 20160615T133051.296322Z 140.90.98.15[16214] NOTE
> requester6.c:588:req6_new() LDM-6 desired product-class:
> 20160615124948.296267 TS_ENDT {{EXP,
> "/data/realtime/outgoing/grib2/MRMS_RotationTrackML60min_00.50_"},{NONE,
> "SIG=3ae652224617d867f5667b845033f1dd"}}
> 20160615T133051.359991Z 140.90.98.15[16214] NOTE
> requester6.c:311:make_request() Upstream LDM-6 on 140.90.98.15 is willing
> to be a primary feeder
> 20160615T133053.240727Z 140.90.98.15[16220] NOTE error.c:236:err_log()
> Upstream LDM died: pid=30704
> 20160615T133053.240838Z 140.90.98.15[16220] NOTE
> requester6.c:588:req6_new() LDM-6 desired product-class:
> 20160615124950.240789 TS_ENDT {{EXP,
> "/data/realtime/outgoing/grib2/MRMS_RadarOnly_QPE_24H_00.00_"},{NONE,
> "SIG=bbc3f1c2e289ffaf4193440a1b992353"}}
> 20160615T133053.265674Z 140.90.98.15[16215] NOTE error.c:236:err_log()
> Upstream LDM died: pid=30703
> 20160615T133053.265838Z 140.90.98.15[16215] NOTE
> requester6.c:588:req6_new() LDM-6 desired product-class:
> 20160615124950.265745 TS_ENDT {{EXP,
> "/data/realtime/outgoing/grib2/MRMS_RadarOnly_QPE_12H_00.00_"},{NONE,
> "SIG=5e16b72421bd767fe303f3a6c503e33b"}}
> 20160615T133053.303848Z 140.90.98.15[16220] NOTE
> requester6.c:311:make_request() Upstream LDM-6 on 140.90.98.15 is willing
> to be a primary feeder
> 20160615T133053.329992Z 140.90.98.15[16215] NOTE
> requester6.c:311:make_request() Upstream LDM-6 on 140.90.98.15 is willing
> to be a primary feeder

> These are our request lines for the data in ldmd.conf.
> 
> # MRMS
> request EXP 
> "/data/realtime/outgoing/grib2/MRMS_ReflectivityAtLowestAltitude_00.50_" 
> 140.90.98.15
> request EXP "/data/realtime/outgoing/grib2/MRMS_RadarOnly_QPE_01H_00.00_" 
> 140.90.98.15
> request EXP "/data/realtime/outgoing/grib2/MRMS_RotationTrack60min_00.50_" 
> 140.90.98.15
> request EXP "/data/realtime/outgoing/grib2/MRMS_MESH_00.50_" 140.90.98.15
> request EXP "/data/realtime/outgoing/grib2/MRMS_MESH_Max_60min_00.50_" 
> 140.90.98.15
> request EXP "/data/realtime/outgoing/grib2/MRMS_MESH_Max_1440min_00.50_" 
> 140.90.98.15
> request EXP 
> "/data/realtime/outgoing/grib2/MRMS_MergedReflectivityQCComposite_00.50_" 
> 140.90.98.15
> request EXP "/data/realtime/outgoing/grib2/MRMS_MergedAzShear_" 140.90.98.15
> request EXP "/data/realtime/outgoing/grib2/MRMS_RotationTrackML60min_00.50_" 
> 140.90.98.15
> request EXP "/data/realtime/outgoing/grib2/MRMS_RadarOnly_QPE_12H_00.00_" 
> 140.90.98.15
> request EXP "/data/realtime/outgoing/grib2/MRMS_SeamlessHSR_00.00_" 
> 140.90.98.15
> request EXP "/data/realtime/outgoing/grib2/MRMS_EchoTop_18_00.50_" 
> 140.90.98.15
> request EXP "/data/realtime/outgoing/grib2/MRMS_VIL_00.50_" 140.90.98.15
> request EXP 
> "/data/realtime/outgoing/grib2/MRMS_LightningProbabilityNext30min_scale_0_" 
> 140.90.98.15
> request EXP "/data/realtime/outgoing/grib2/MRMS_RadarOnly_QPE_24H_00.00_" 
> 140.90.98.15
> request EXP "/data/realtime/outgoing/grib2/MRMS_Reflectivity_-20C_00.50_" 
> 140.90.98.15
> 
> # Non-CONUS MRMS
> request EXP 
> "/data/realtime/outgoing/grib2/HAWAII/MRMS_MergedReflectivityQComposite_00.50"
>  140.90.98.15
> request EXP 
> "/data/realtime/outgoing/grib2/GUAM/MRMS_MergedReflectivityQComposite_00.50" 
> 140.90.98.15
> request EXP 
> "/data/realtime/outgoing/grib2/ALASKA/MRMS_MergedReflectivityQComposite_00.50"
>  140.90.98.15
> request EXP 
> "/data/realtime/outgoing/grib2/CARIB/MRMS_MergedReflectivityQComposite_00.50" 
> 140.90.98.15

Regards,
Steve Emmerson

Ticket Details
===================
Ticket ID: YZC-779273
Department: Support LDM
Priority: Normal
Status: Closed
===================
NOTE: All email exchanges with Unidata User Support are recorded in the Unidata 
inquiry tracking system and then made publicly available through the web.  If 
you do not want to have your interactions made available in this way, you must 
let us know in each email you send to us.