[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[LDM #CSZ-262645]: errant LDM REQUESTs on sirocco.srcc.lsu.edu



Hi Jeffry,

We wanted to thank you for your clear and concise description of the cause
and solution the the firewall-related problem that was interfering with
LSU/SRCC LDMs from reliably getting data from machines outside of the
LSU campus.  The information provided will likely help some other site
down the road!

Thanks again...

Jeffry Handal wrote:
> Summary:
> Issue at LSU were caused by the campus firewall.
> 
> Equipment:
> Juniper SRX-5800 with IDP
> 
> Technical Details of Issue:
> All traffic at LSU is subject to an IDP to block p2p. Since January
> 18th, the IDP was turned off because it was causing high CPU
> utilization, slowing the entire campus traffic. Unfortunately, there was
> a rule that still would point to an inactive IDP. The way this works,
> the flows that match this IDP policy will be asked to redirect to the
> IDP module. They will go into a queue waiting for IDP inspection.
> Because the IDP module is not enabled, they are momentarily stuck there
> waiting for timeout. All this waiting will being to clog up the
> buffer-queue, which eventually triggers the log message we saw on the
> SRX (i.e. Feb 22 14:03:24  csc-118-l7-srx5800-edgefrwl fpc1 Cobar: XLR1
> flow_held_mbuf 500, raise above 500, 1000th time. ). When the queue gets
> full, packets were dropped. Most connections will not see a problem
> because TCP will recover and start a new connection. For other
> applications, this may look like a DoS due to the constant creation of
> new connections, e.g. Unidata.
> 
> LSU future plans:
> * Feature request to Juniper: if IDP is turned off, send a warning
> message or error trigger that a rule is still pointing to the IDP even
> though it is off.
> 
> * We are in the process of creating a Science DMZ that will no be
> subject to a campus firewall.
> 
> Thanks to everyone involved for the support and patience in this
> process. By far, hardware issues are the hardest to troubleshoot; here,
> we went down to the buffer level to diagnose the problem.
> 
> Regards,
> 
> Jeffry J.Handal
> Convergence Specialist, M.S.E.E., P.E.
> University Networking and Infrastructure - LSU
> 200 Computing Services Building
> Baton Rouge, LA 70803
> Office:       (225)578-1966
> Fax:  (225)578-6400

Cheers,

Tom
--
****************************************************************************
Unidata User Support                                    UCAR Unidata Program
(303) 497-8642                                                 P.O. Box 3000
address@hidden                                   Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage                       http://www.unidata.ucar.edu
****************************************************************************


Ticket Details
===================
Ticket ID: CSZ-262645
Department: Support LDM
Priority: Normal
Status: Open


NOTE: All email exchanges with Unidata User Support are recorded in the Unidata inquiry tracking system and then made publicly available through the web. If you do not want to have your interactions made available in this way, you must let us know in each email you send to us.