[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[Datastream #QOK-427508]: NPS: New "manager", Date problem

Hi Ryan,

My apologies for the tardy reply...

address@hidden> wrote:

> New Client Reply: NPS: New "manager", Date problem
>
> Hi Tom,
> I wanted to follow-up, perhaps with the IDD support email…Not sure if it
> went through on the address@hidden email.
>

It did, but there have been a LOT of things going on in the past several
days...

re:

> See questions below.


These will be answered from our inquiry tracking system.

re:

> Also, can you please add idd.foggy.nps.edu to your ALLOW list on
> idd.unidata.ucar.edu?
>

As far as I can tell, this is a VM in AWS, correct?  Here is what I found:

 $ nslookup idd.foggy.nps.edu
Server: 127.0.1.1
Address: 127.0.1.1#53

Non-authoritative answer:
Name: idd.foggy.nps.edu
Address: 18.236.13.104
Name: idd.foggy.nps.edu
Address: 52.38.107.188

mcidas@odroidU3:~/Documents$ nslookup 52.38.107.188
Server: 127.0.1.1
Address: 127.0.1.1#53

Non-authoritative answer:
188.107.38.52.in-addr.arpa name =
ec2-52-38-107-188.us-west-2.compute.amazonaws.com.

Authoritative answers can be found from:

So, there is forward DNS (name -> IP), but no reverse DNS (IP -> name).  In
order to
ALLOW you we added an entry to our hosts file that defines your machine's
IP address
as the name you specified.  This approach is "brittle" as it will stop
working if the IP
address of your VM changes.  It would be much better if reverse DNS was
setup for
your machine such that the IP address is globally known as idd.foggy.nps.edu.
In
the interim, you should be ready to go, but you should check as follows:

<as 'ldm' on your machine>
notifyme -vl- -f ANY -h idd.unidata.ucar.edu

Cheers,

Tom

> . Thanks!
> Best regards,
> Ryan/Mark
>
> --
> Ryan Yamaguchi
> Faculty Associate - Research
> Department of Meteorology
> Naval Postgraduate School
> 589 Dyer Road, Room 259
> Monterey, CA 93943
> Office: (831) 656-7645 DSN: 756-7645
> Cell: (831) 682-5159
>
>
> From: "Boothe, Mark (CIV)" <address@hidden>
> Date: Friday, September 6, 2019 at 2:02 PM
> To: "Yamaguchi, Ryan (CIV)" <address@hidden>, "
> address@hidden" <address@hidden>
> Subject: Re: [Datastream #QOK-427508]: NPS: New "manager", Date problem
>
> Hello Tom,
>
> Here at the Naval Postgraduate School, Ryan and I have been coordinating
> with our local IT folks as we prepare the transition of our LDM feed from a
> set of on-site linux machines to AWS.
>
> Thanks for your previous message pointing out the RTSTATS link.  That
> gives a good depiction of the data stream.
>
> Do you have any suggestions as we spin up an AWS instance?  For instance:
>
> Is there a preferred OS (i.e.: Amazon linux, Red Hat, Suse, Ubuntu, etc.)?
> What other specifications (i.e. memory, computing power, etc.)?
> How do we inform you of our new AWS instance?
> We currently receive our data via the University of Washington. Would this
> continue with a new AWS instance, or would we connect directly with the IDD?
> Is there a particular data stream that serves as a good initial test?
>
> Thanks,
> Mark Boothe and Ryan Yamaguchi
> Naval Postgraduate School
> Monterey CA 93943
> ________________________________
> From: Yamaguchi, Ryan (CIV) <address@hidden>
> Sent: Friday, June 21, 2019 9:48 AM
> address@hidden>; Boothe, Mark (CIV) <address@hidden
> >
> Subject: Re: [Datastream #QOK-427508]: NPS: New "manager", Date problem
>
> Hi Tom,
>
> Attached is a basic diagram and list of requirements I've sent to our IT
> folks.  At a minimum, we are trying to get an LDM server and AWIPS EDEX
> ingest/decoder server spun in the AWS cloud.  We would like to archive case
> studies with Amazon S3.
>
> As for the visualization suite, I am planning to host that software on
> Amazon WorkSpaces (i.e. bring your own thin client).  The students can
> login to that system and complete all of their analyses there, without
> pulling out data to their local machine (trying to save costs).
>
> Sorry, I do not have an explicit roadmap for cloud computing.  However, it
> seems we have full support from our IT to transition our aging systems to
> the cloud.
>
> Thanks!
> Ryan
>
> address@hidden> wrote:
>
>     Hi,
>
>     re:
>     > Thanks for your information this last week as I begin to log into
> machines here at NPS
>     > as an LDM manager for the first time.
>
>     No worries.
>
>     re:
>     > I'd like to shift gears a bit and help get the ball rolling faster
> on our transition
>     > here at NPS from our relatively limited ingest into linux machines
> on campus to a larger
>     > ingest on a new Amazon platform.
>
>     OK.
>
>     re:
>     > The IT/cybersecurity department here on campus requires,
>     > of course, specifics on the technical aspects of what this Amazon
> instance/storage would
>     > require before we can spin it up.
>
>     Of course.
>
>     re:
>     > Do you have, or can you direct us toward a person or
>     > website, that has specifics that we would need in order to ingest
> and store a useful
>     > (maybe a year's worth?) archive?
>
>     The volume of data flowing in the IDD is a moving target, so any
> estimate made today
>     will be wrong tomorrow.  That being said, you could do some "back of
> the envelope"
>     calculations by extrapolating the average volume of data available in
> the datastreams
>     desired per hour and then multiplying by the number of hours in a
> year.  For
>     instance, a randomly chosen real-server back end of our
> idd.unidata.ucar.edu
>     top level IDD relay cluster can be found in:
>
>
> http://rtstats.unidata.ucar.edu/cgi-bin/rtstats/siteindex?cluster2.unidata.ucar.edu
>
>     The link at the lower left of this page will generate a snapshot list
> of
>     the volume of data flowing in each of the datastreams that this node is
>     handling:
>
>     Cumulative volume summary
>
> http://rtstats.unidata.ucar.edu/cgi-bin/rtstats/rtstats_summary_volume?cluster2.unidata.ucar.edu
>
>     This looks like:
>
>     Data Volume Summary for cluster2.unidata.ucar.edu
>
>     Maximum hourly volume 100803.397 M bytes/hour
>     Average hourly volume  65493.578 M bytes/hour
>
>     Average products per hour     523876 prods/hour
>
>     Feed                           Average             Maximum     Products
>                          (M byte/hour)            (M byte/hour)
>  number/hour
>     SATELLITE             14892.385    [ 22.739%]    19959.534     6659.762
>     CONDUIT               11783.612    [ 17.992%]    33345.747   105400.619
>     NGRID                 10437.259    [ 15.936%]    14778.178    64434.857
>     NEXRAD2                9293.592    [ 14.190%]    11192.046   102743.286
>     NOTHER                 6984.060    [ 10.664%]    10042.353    12000.310
>     NIMAGE                 3973.837    [  6.068%]     9144.665     3507.000
>     FNMOC                  3181.508    [  4.858%]    11549.530     8347.071
>     NEXRAD3                2975.614    [  4.543%]     3511.048   127587.762
>     HDS                    1369.725    [  2.091%]     1754.030    44231.143
>     GEM                     306.029    [  0.467%]     3307.336     1779.452
>     FNEXRAD                 119.299    [  0.182%]      134.965      104.548
>     UNIWISC                  85.293    [  0.130%]      142.226       49.857
>     IDS|DDPLUS               71.118    [  0.109%]       84.169    46548.738
>     EXP                      12.109    [  0.018%]       18.775      126.310
>     LIGHTNING                 8.033    [  0.012%]       15.598      354.429
>     GPS                       0.105    [  0.000%]        1.102        0.929
>
>     As you can see, there are hourly average and hourly peak volumes
> listed.
>
>     re:
>     > We realize that the recent transition to GOES-16/17 is
>     > changing all of the calculations, so making this transition now
> seems like good timing.
>
>     The volume of data flowing in the IDD changes all of the time as new
> model output
>     is added, new satellite imagery is added, higher resolution radar data
> becomes
>     available, etc.  The IDD truly is a fire hose, and its stream is
> continually growing.
>
>     re:
>     > By the way, Ryan xxxxx has been here a few years and has already
> made some headway
>     > in getting this transition going. You may recall earlier
> correspondence with him, and
>     > I've cc'ed him. As of now, the two of us are working together on
> this project to help
>     > cover this transition to Amazon during any other temporary projects
> that require weeks
>     > worth of field work out of the office.
>
>     OK, sounds good.
>
>     re:
>     > Thanks for any technical advice you have,
>
>     No worries.
>
>     Do you/NPS have a road map for where you are heading wrt cloud
> computing?  If yes, are
>     you willing to share it?
>
>     Cheers,
>
>     Tom
>     --
>
> ****************************************************************************
>     Unidata User Support                                    UCAR Unidata
> Program
>     (303) 497-8642                                                 P.O.
> Box 3000
>     address@hidden                                   Boulder,
> CO 80307
>
> ----------------------------------------------------------------------------
>     Unidata HomePage                       http://www.unidata.ucar.edu
>
> ****************************************************************************
>
>
>     Ticket Details
>     ===================
>     Ticket ID: QOK-427508
>     Department: Support Datastream
>     Priority: Normal
>     Status: Closed
>     ===================
>     NOTE: All email exchanges with Unidata User Support are recorded in
> the Unidata inquiry tracking system and then made publicly available
> through the web.  If you do not want to have your interactions made
> available in this way, you must let us know in each email you send to us.
>
>
>
>
>
> Ticket Details
> ===================
> Ticket ID: QOK-427508
> Department: Support Datastream
> Priority: Normal
> Status: Open
> Link:
> https://andy.unidata.ucar.edu/staff/index.php?_m=tickets&_a=viewticket&ticketid=30459
>
>

-- 
+----------------------------------------------------------------------+
* Tom Yoksas                                      UCAR Unidata Program *
* (303) 497-8642 (last resort)                           P.O. Box 3000 *
* address@hidden                            Boulder, CO 80307 *
* Unidata WWW Service                     http://www.unidata.ucar.edu/ *
+----------------------------------------------------------------------+



Ticket Details
===================
Ticket ID: QOK-427508
Department: Support Datastream
Priority: Normal
Status: Open
===================
NOTE: All email exchanges with Unidata User Support are recorded in the Unidata 
inquiry tracking system and then made publicly available through the web.  If 
you do not want to have your interactions made available in this way, you must 
let us know in each email you send to us.