[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[THREDDS #IXX-362335]: Urgent: UMASS Production Tomcat/THREDDS server shut down due to flood of DNS requests



On 4/21/2014 2:37 PM, Signell, Richard wrote:
> New Ticket: Urgent: UMASS Production Tomcat/THREDDS server shut down due to 
> flood of DNS requests
>
> Thredds guys,
>
> UMASS shutdown their production tomcat/thredds and disabled the tomcat
> user on Saturday, which of course is causing an interuption in ocean
> forecast products in New England used by the US Coast Guard, US IOOS
> and the local Weather Service Offices.
>
> Here is there message about why they shut it down.
>
> Any ideas about what was happening and how to get this back up and running?
>
> >From Kent Gardner at UMASSD:
>
> It appears that the SMAST host system that is running Thredds was
> generating a storm of DNS requests to our campus name server. When
> Mike shut Thredds down and disabled the tomcat account the storm
> stopped.
>
> I can think of no legitimate reason why Thredds would be doing this.
> The only thing that remotely comes to mind would be someone trying to
> look up IP numbers in a log file to get the host name for
> informational purposes. Has anyone come across this behavior before in
> Thredds/Tomcat?
>
> Also looking in /tmp we see the following:
>
>   ls -al /tmp|grep tomcat
>
> drwxr-xr-x   2 tomcat       tomcat           4096 Apr 15 19:42 adiandian
>
> -rwxr-xr-x   1 tomcat       tomcat              5 Apr 18 13:11 bill.lock
>
> drwxr-xr-x   3 tomcat       tomcat           4096 Apr 11 14:32 dEDVea
>
> drwxr-xr-x   3 tomcat       tomcat           4096 Apr 14 10:30 dvcdNo
>
> drwxr-xr-x   3 tomcat       tomcat           4096 Apr  8 09:35 fkuQAx
>
> -rwxr-xr-x   1 tomcat       tomcat              5 Apr 18 13:11 gates.lock
>
> drwxr-xr-x   2 tomcat       tomcat           4096 Apr 18 21:52 
> hsperfdata_tomcat
>
> drwxr-xr-x   2 tomcat       tomcat           4096 Mar 28 23:59 httpdlog
>
> --wx--Sr--   1 tomcat       tomcat             51 Apr 16 11:46 notify.file
>
>
> I do not know of any files that Thredds/Tomcat would put in /tmp. Does
> anyone know if any of these files are legitimate?
>
> As far a game plan goes I will need to confer with Mike. At the very
> least we need to scan for and delete all suspicious files, and change
> the password on the tomcat account. After that we start things up and
> monitor the network traffic. "
>
>
>
> Thanks,
> Rich
>
Hi Rich:

One could (mis)configure tomcat to do ns lookup domain name for each 
request. Check or send me *${tomcat_home}/conf/server.xml*

just got back from vacation so catching up.

John



Ticket Details
===================
Ticket ID: IXX-362335
Department: Support THREDDS
Priority: Normal
Status: Open


NOTE: All email exchanges with Unidata User Support are recorded in the Unidata inquiry tracking system and then made publicly available through the web. If you do not want to have your interactions made available in this way, you must let us know in each email you send to us.