[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[Datastream #WSL-696975]: NEXRAD data on AWS

Hi,

re:
> On Saturday there was a gap in the real-time NEXRAD data on AWS.

Yes, there were two periods of high latency in the delivery of both
NEXRAD Level 3 (products) and NEXRAD Level 2 (full volume scan) data
to AWS.  Investigations showed that the problem/slowdown was likely
related to a bridge from Internet 2 to AWS, something that is not
under our control.

re:
> Not knowing whom to contact, I e-mailed the NOAA Big Data Project,
> and they referred me to you.

OK.  We received support inquiries by a couple of other sites that
are using the data on AWS, one for the NEXRAD Level 3 data and the
other for the NEXRAD Level 2 data.

re:
> We are planning to use the AWS NEXRAD data in real-time for an upcoming
> NASA project (DCOTSS), and I was hoping you could answer a few questions
> for me.

Yes, of course.

re:
> Whom should I contact in the event of future problems?  UNIDATA or
> someone at AWS?  I have looked over the UNIDATA support e-mail listservs
> and don't see anything appropriate there.

Unless the problem is somewhere in AWS, it would be best to contact
us:

address@hidden

re:
> Is there a way to get real-time status info on UNIDATA data feeds in
> general and the NEXRAD feed in particular?

We maintain a website that shows real-time statistics for machines
running the LDM that have chosen to report back to us:

https://rtstats.unidata.ucar.edu/cgi-bin/rtstats/siteindex

The machine in AWS that is receiving both NEXRAD Level 3 (IDD feedtype
NEXRAD3) and NEXRAD Level 2 (IDD feedtype NEXRAD2) from our top level
IDD relay cluster is:

amazon.ec2w2_2.unidata.ucar.edu

Scroll down in the list presented by the page for the URL above, and
click on the machine name.  In the page that will be presented:

https://rtstats.unidata.ucar.edu/cgi-bin/rtstats/siteindex?amazon.ec2w2_2.unidata.ucar.edu

you will see two entries, one for the NEXRAD Level 3 feed and the other for the
NEXRAD Level 2 feed.  When selected, the 'latency' link for each feed will
produce a time series plot of the latency for the feed.  For instance:

NEXRAD 3 latency:
https://rtstats.unidata.ucar.edu/cgi-bin/rtstats/iddstats_nc?NEXRAD3+amazon.ec2w2_2.unidata.ucar.edu

NEXRAD 2 latency
https://rtstats.unidata.ucar.edu/cgi-bin/rtstats/iddstats_nc?NEXRAD2+amazon.ec2w2_2.unidata.ucar.edu

If you display these plots now (or soon), you will see the high period of
latencies that occurred on Saturday.

Knowing that there is high latency on the receiving machine is not enough to 
determine
where the slowdown is, however.  The next step in the process of determining 
where
the slowdown is occurring (and, therefore who would be best to contact) is to
look at the latencies for these feeds on any of the real-server backend machines
of our top level IDD relay cluster, idd.unidata.ucar.edu.  The names of these
real server backend machines are:

node0.unidata.ucar.edu
node1.unidata.ucar.edu
 ...
node7.unidata.ucar.edu

Looking at the latencies for the NEXRAD2 and/or NEXRAD3 feeds in the page
for any of these machines will show if they too are experiencing high
latencies.  If they are, then the problem is somewhere upstream of the
machines.  For the NEXRAD2 feed, the top level relays are machines that
are maintained by NOAA.  For the NEXRAD3 feed, the top levels are
machines operated by us (Unidata/UCAR), the University of Wisconsin
Space Science and Engineering Center (UW/SSEC) and Louisiana State
University Southern Regional Climate Center (LSU/SRCC).

If, for instance, the latencies being reported on any/all of the real-server
backend machines of our top level IDD relay cluster are reporting low
receipt latencies, the slowdown is somewhere between us and our machine
in AWS.  To determine if the problem is at/near us is tricky - we need
to figure out which real server backend is currently feeding the AWS
machine and then find out what other machines that node is also feeding
and then lookup delivery latencies to that/those machines.  I did just
this on Saturday and determined that data was flowing without added
latencies to machines at Unidata user sites while the deliver to the
AWS machine was impacted.  This told me that the problem was somewhere
in the data path from us to the AWS machine, but it did not tell us
exactly where other than it was not with/immediately near us.  Since
the process of determining which real-server backend is currently 
serving the AWS instance and what other machines, if any, that node
is also feeding requires login access to the real server backend
machines, the end-user can do no more sleuthing, so contacting us
is in order.

re:
> It appears that the missing files were recovered later in the weekend.
> Is that the standard procedure,

Yes, we try to backfill to AWS as soon as we are aware of an outage and
the network permits transfer of the missing data.

re:
> or should we plan to get the data from
> a different source?

In the support form you filled out and sent us, you listed your organization
as Texas A&M.  Since Texas A&M (TAMU) is a Unidata university, they could
receive a real-time feed of either/both the NEXRAD2 or NEXRAD3 feeds.  This
feed would act as a redundant way to get the data.

We also make the the NEXRAD2 and NEXRAD3 data available via our THREDDS Data
Server (TDS) technology on two public facing server instances:

thredds.ucar.edu
thredds-test.unidata.ucar.edu

Unlike the NEXRAD Level 2 data that is in the NEXRAD2 feed (the products are
pieces of volume scans that need to be stitched back together to form a full
volume scan), the NEXRAD Level 2 data available from our TDS instances are
fully reconstituted volume scans.

re:
> Thanks for your help

I hoped that the above helped...

Cheers,

Tom
--
****************************************************************************
Unidata User Support                                    UCAR Unidata Program
(303) 497-8642                                                 P.O. Box 3000
address@hidden                                   Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage                       http://www.unidata.ucar.edu
****************************************************************************


Ticket Details
===================
Ticket ID: WSL-696975
Department: Support Datastream
Priority: Normal
Status: Closed
===================
NOTE: All email exchanges with Unidata User Support are recorded in the Unidata 
inquiry tracking system and then made publicly available through the web.  If 
you do not want to have your interactions made available in this way, you must 
let us know in each email you send to us.