Thanks Rich, I'll try that. I had another incident this weekend with a
chinese IP sending a number of large requests. I guess I could
restrict the number of requests per single ip, but I'm worried that
might effect a valid user's experience.
On Mon, Sep 26, 2011 at 8:12 AM, Rich Signell <rsignell@xxxxxxxx> wrote:
> Tom,
> Do you have a robots.txt file on your server?
> This helps keep legitimate crawlers from hitting your thredds server
> (but it can't stop hackers who don't pay attention to robots.txt of
> course)
>
> I patterned mine after what Unidata's motherlode server used to look like:
> http://geoport.whoi.edu/robots.txt
>
> But I noticed that now, it looks much simpler:
> http://motherlode.ucar.edu/robots.txt
>
> # disallow everything
> User-agent: *
> Disallow: /
>
> The problem with disallowing anything is that things like the
> GetCapabilities can't be picked up by crawlers from your TDS WMS
> server, which means that tools like ASA's OGC Search
> http://staging.asascience.com/OGCSearch-OL/
> won't pick up your WMS-served data.
>
> -Rich
>
> On Thu, Sep 22, 2011 at 7:42 PM, tom cook <tcook@xxxxxxxxxxxx> wrote:
>> HI, the more I investigated, I saw a lot of large requests originating
>> from a Chinese address. I wasn't able to resolve the address to a
>> specific institution so I blocked it in Tomcat's server.xml.
>> We'll see if that was the problem or not.
>> Thanks,
>> Tom
>>
>> On Thu, Sep 22, 2011 at 3:35 PM, tom cook <tcook@xxxxxxxxxxxx> wrote:
>>> HI,
>>> I just recently started seeing a large amount of requests from my
>>> server address for various TDS services. For example, I'm seeing ALOT
>>> of these type of messages:
>>>
>>> 2011-09-22T20:02:08.679 +0000 [ 17778388][ 697] INFO -
>>> thredds.server.opendap.OpendapServlet - doGet(): Remote host:
>>> 132.239.127.187 - Request: "GET
>>> /thredds/dodsC/HFRNet/USEGC/1km/hourly/GNOME.das HTTP/1.1"
>>> 2011-09-22T20:02:14.703 +0000 [ 17784412][ 698] INFO -
>>> thredds.server.opendap.OpendapServlet - doGet(): Remote host:
>>> 132.239.127.187 - Request: "GET
>>> /thredds/dodsC/HFRNet/USWC/1km/hourly/GNOME.das HTTP/1.1"
>>>
>>> AND
>>>
>>> java.lang.OutOfMemoryError: GC overhead limit exceeded
>>>
>>> 132.239.127.187 is my host machine, and the requests are also showing
>>> up in the access log as initiated from 132.239.127.187. There are a
>>> ton of these, and I'm concerned why this started and what its doing?
>>> I can send logs if needed, but they are pretty large, so I'd rather
>>> see if there is something you can explain to me that may help
>>> understand where my problem lies. All these requests/overhead limit
>>> are making my server unusable.
>>>
>>> Thanks,
>>> Tom
>>>
>>
>> _______________________________________________
>> thredds mailing list
>> thredds@xxxxxxxxxxxxxxxx
>> For list information or to unsubscribe, visit:
>> http://www.unidata.ucar.edu/mailing_lists/
>>
>
>
>
> --
> Dr. Richard P. Signell (508) 457-2229
> USGS, 384 Woods Hole Rd.
> Woods Hole, MA 02543-1598
>
