[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[THREDDS #IXX-362335]: Urgent: UMASS Production Tomcat/THREDDS server shut down due to flood of DNS requests



Rich,

Is there a competent sys admin for this TDS instance?  I'm not really a 
security expert, but yet, upgrading OpenSSL to eliminate the 
vulnerability would seem to be sufficient.  I can't speak to the 
necessity of re-compiling the APR or Tomcat Native library. Checking 
against a service that identifies vulnerabilities would seem to be a 
good idea.

Turning off OpenSSL and remote access (i.e., "locking down" the server) 
is probably still the safest bet.  This is done in the 
{$tomcat_home}/conf/server.xml file.

-Lansing

On 4/22/2014 9:40 AM, Signell, Richard wrote:
> New Client Reply: Urgent: UMASS Production Tomcat/THREDDS server shut down 
> due to flood of DNS requests
>
> I saw this comment:
>
> "Confirmed my app was vulnerable according to filippo.io/Heartbleed
> After upgrading OpenSSL, simply restarting Tomcat eliminated the
> vulnerability. Re-compiling APR or Tomcat Native was not necessary"
>
> Do you agree that this simple approach is sufficient?
>
> -Rich
>
> On Tue, Apr 22, 2014 at 11:35 AM, Unidata THREDDS Support
> <address@hidden> wrote:
>> My experience is with Windows, since that is what I have.  I'll try to
>> find out what is included in downloads for other o/s. Information
>> directly from apache is here:
>>
>> https://issues.apache.org/bugzilla/show_bug.cgi?id=56363
>>
>> and a discussion on security.stackexchange is here:
>>
>> http://security.stackexchange.com/questions/55139/does-the-heartbleed-vulnerability-affect-apache-tomcat-servers-using-tomcat-nati
>>
>> -Lansing
>>
>> On 4/22/2014 9:31 AM, Signell, Richard wrote:
>>> New Client Reply: Urgent: UMASS Production Tomcat/THREDDS server shut down 
>>> due to flood of DNS requests
>>>
>>> Does this pertain to Tomcat/THREDDS on Windows machines only, or all 
>>> machines?
>>>
>>> On Tue, Apr 22, 2014 at 11:28 AM, Unidata THREDDS Support
>>> <address@hidden> wrote:
>>>> Hi Rich,
>>>>
>>>> We noticed that with the most recent jdk and tomcat downloads (1.7u55
>>>> and 7u53, respectively), our Windows machines had an old SSL, as
>>>> evidenced here:
>>>>
>>>> Apr 21, 2014 2:34:22 PM org.apache.catalina.core.AprLifecycleListener 
>>>> initializeSSL
>>>> INFO: OpenSSL successfully initialized (OpenSSL 1.0.1e 11 Feb 2013)
>>>>
>>>> Clearly, the OpenSSL library predates the Heartbleed fix.  We asked Jen 
>>>> about this, and her advice was to keep the APRLifecycleListener commented 
>>>> out (as we have on our production machines).  This avoids the 
>>>> vulnerability until Apache Tomcat updates.
>>>>
>>>> -Lansing
>>>>
>>>> On 4/22/2014 9:24 AM, Signell, Richard wrote:
>>>>> New Client Reply: Urgent: UMASS Production Tomcat/THREDDS server shut 
>>>>> down due to flood of DNS requests
>>>>>
>>>>> Are there things we should do to respond to Heartbleed on public
>>>>> THREDDS servers?
>>>>>
>>>>> For folks running public IPython notebook servers, we received this
>>>>> advice:"If you run a public #IPython notebook server over HTTPS, shut
>>>>> it down, upgrade OpenSSL, regenerate your TLS certificate & restart."
>>>>>
>>>>> Does the same apply to tomcat/thredds?
>>>>>
>>>>> -Rich
>>>>>
>>>>> On Mon, Apr 21, 2014 at 5:12 PM, Unidata THREDDS Support
>>>>> <address@hidden> wrote:
>>>>>> Hi Rich, all,
>>>>>>
>>>>>> Nothing is jumping out at me. Do you have the access log and 
>>>>>> threddsServlet.log files for the time period when the DNS requests were 
>>>>>> being generated?
>>>>>>
>>>>>> The only /tmp file listed below that looks legit (i.e., likely produced 
>>>>>> by the TDS) is the hsperfdata_tomcat/ directory (more below [1]). The 
>>>>>> only other one that seems like it might be OK is the httpdlog file. But 
>>>>>> even that seems a bit odd unless you are running an Apache server as the 
>>>>>> tomcat user. Is the tomcat user only used to run Tomcat instances 
>>>>>> running TDS?
>>>>>>
>>>>>>
>>>>>> [1] The hsperfdata_tomcat/ directory is the only one from the file 
>>>>>> listing below that we have on our main server. It, according to [2], is 
>>>>>> created by Java and used to allow/improve monitoring capabilities with 
>>>>>> jconsole and the like.
>>>>>>
>>>>>> [2] 
>>>>>> http://stackoverflow.com/questions/76327/how-can-i-prevent-java-from-creating-hsperfdata-files
>>>>>>
>>>>>>> Thredds guys,
>>>>>>>
>>>>>>> UMASS shutdown their production tomcat/thredds and disabled the tomcat
>>>>>>> user on Saturday, which of course is causing an interuption in ocean
>>>>>>> forecast products in New England used by the US Coast Guard, US IOOS
>>>>>>> and the local Weather Service Offices.
>>>>>>>
>>>>>>> Here is there message about why they shut it down.
>>>>>>>
>>>>>>> Any ideas about what was happening and how to get this back up and 
>>>>>>> running?
>>>>>>>
>>>>>>>    From Kent Gardner at UMASSD:
>>>>>>>
>>>>>>> It appears that the SMAST host system that is running Thredds was
>>>>>>> generating a storm of DNS requests to our campus name server. When
>>>>>>> Mike shut Thredds down and disabled the tomcat account the storm
>>>>>>> stopped.
>>>>>>>
>>>>>>> I can think of no legitimate reason why Thredds would be doing this.
>>>>>>> The only thing that remotely comes to mind would be someone trying to
>>>>>>> look up IP numbers in a log file to get the host name for
>>>>>>> informational purposes. Has anyone come across this behavior before in
>>>>>>> Thredds/Tomcat?
>>>>>>>
>>>>>>> Also looking in /tmp we see the following:
>>>>>>>
>>>>>>> ls -al /tmp|grep tomcat
>>>>>>>
>>>>>>> drwxr-xr-x   2 tomcat       tomcat           4096 Apr 15 19:42 adiandian
>>>>>>>
>>>>>>> -rwxr-xr-x   1 tomcat       tomcat              5 Apr 18 13:11 bill.lock
>>>>>>>
>>>>>>> drwxr-xr-x   3 tomcat       tomcat           4096 Apr 11 14:32 dEDVea
>>>>>>>
>>>>>>> drwxr-xr-x   3 tomcat       tomcat           4096 Apr 14 10:30 dvcdNo
>>>>>>>
>>>>>>> drwxr-xr-x   3 tomcat       tomcat           4096 Apr  8 09:35 fkuQAx
>>>>>>>
>>>>>>> -rwxr-xr-x   1 tomcat       tomcat              5 Apr 18 13:11 
>>>>>>> gates.lock
>>>>>>>
>>>>>>> drwxr-xr-x   2 tomcat       tomcat           4096 Apr 18 21:52 
>>>>>>> hsperfdata_tomcat
>>>>>>>
>>>>>>> drwxr-xr-x   2 tomcat       tomcat           4096 Mar 28 23:59 httpdlog
>>>>>>>
>>>>>>> --wx--Sr--   1 tomcat       tomcat             51 Apr 16 11:46 
>>>>>>> notify.file
>>>>>>>
>>>>>>>
>>>>>>> I do not know of any files that Thredds/Tomcat would put in /tmp. Does
>>>>>>> anyone know if any of these files are legitimate?
>>>>>>>
>>>>>>> As far a game plan goes I will need to confer with Mike. At the very
>>>>>>> least we need to scan for and delete all suspicious files, and change
>>>>>>> the password on the tomcat account. After that we start things up and
>>>>>>> monitor the network traffic. "
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Rich
>>>>>>>
>>>>>>> --
>>>>>>> Dr. Richard P. Signell
>>>>>> Ticket Details
>>>>>> ===================
>>>>>> Ticket ID: IXX-362335
>>>>>> Department: Support THREDDS
>>>>>> Priority: Normal
>>>>>> Status: Open
>>>>>>
>>>>
>>>> Ticket Details
>>>> ===================
>>>> Ticket ID: IXX-362335
>>>> Department: Support THREDDS
>>>> Priority: Normal
>>>> Status: Open
>>>>
>>>
>>
>>
>> Ticket Details
>> ===================
>> Ticket ID: IXX-362335
>> Department: Support THREDDS
>> Priority: Normal
>> Status: Open
>>
>
>



Ticket Details
===================
Ticket ID: IXX-362335
Department: Support THREDDS
Priority: Normal
Status: Open


NOTE: All email exchanges with Unidata User Support are recorded in the Unidata inquiry tracking system and then made publicly available through the web. If you do not want to have your interactions made available in this way, you must let us know in each email you send to us.