[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: 20080216: IDD burstiness (cont.)

Hi Tom,

Thanks for your follow up.
The spikes being seen by all are undoubtedly "second trip" or "recirculation" products being received on idd.unidata.ucar.edu from mistral.srcc.lsu.edu. The products being received on idd.unidata.ucar.edu from dvb1.ssec.wisc.edu uniformly have very low latencies. It would seem that LDM queues for data being received on both oliver.unidata.ucar.edu and emo.unidata.ucar.edu -- the accumulators for the idd.unidata.ucar.edu cluster -- are not large enough to account for/reject "old" products from mistral.srcc.lsu.edu. This is understandable given that both olvier and emo are ingesting the entire volume of data available on the IDD multiply redundantly from upstream sites. 

Ah, makes sense.
The recirculation argument does not, however, account for Jeff's report of burstiness in the NNEXRAD products.
I have noticed similiar problems with IDS products that started yesterday afternoon. I can't attest to the burstiness, just that seemingly random products come really late (10-20 minutes). 

The last concrete example was:

553
WWUS43 KDMX 162149
WSWDMX

This didn't hit my LDM until 5 minutes later.
I think that if you study the latency plots in conjunction with the volume plots, you will see that the entire content of the IDS|DDPLUS datastream is being sent to downstreams with low latency. The hiccup is the set of`high-latency, second-trip products being intoduced into the system from mistral.srcc.lsu.edu.
I dunno, I have lots of users complaining to me and have noticed similiar issues with COD's website. I have lots of NWS users, so they will squawk to me when a product doesn't hit iembot, since they issued the product and can sense latency in real time :)
By the way, the explanation for why we are now seeing the relatively high latency products from mistral on oliver/emo is that the two NOAAPort ingesters here at UCAR are not working at the moment. A service call was placed this morning; we expect to get the problem diagnosed and resolved on Tuesday morning when the service is made. 

Ahh, I also noticed this ADM today:

NOUS72 KNCF 111452
ADMNCF
THE NWSTG IS EXPERIENCING TECHNICAL DIFFICULTIES WITH A FRONT END
PROCESSOR.  THEY ARE TROUBLSHOOTING THE ISSUE AND HOPE TO HAVE
THE PROBLEM RESOLVED IN A TIMELY FASHION.  THANK YOU FOR YOUR
PATIENCE.
I tried to verify any problems with other data vendors and couldn't get any affirmative responses. 

thanks,
  daryl