Re: [awips2-users] [EXTERNAL] Re: EDEX to CAVE latency with multiple simultaneous users

Robert,

Thank you for your informative experience. Out of curiosity, would you mind 
sharing what the brand and model number is for NVMe SSDs you use? With EDEX 
constantly performing significant amounts of disk writing, I've read that for 
SSDs, the underlying NAND flash type may be an important consideration when 
determining long-term SSD reliability/endurance (i.e. Drive Writes Per Day).

Jason N. T. Kaiser
Atmospheric Sciences Data Systems Administrator
Northern Vermont University-Lyndon


-----Original Message-----
From: Haley, Robert E <haley787@xxxxxxxx> 
Sent: Thursday, April 4, 2019 1:46 PM
To: Kaiser, Jason N. <jason.kaiser@xxxxxxxxxxxxxxxxxxx>; Michael James 
<mjames@xxxxxxxx>
Cc: awips2-users@xxxxxxxxxxxxxxxx
Subject: RE: [EXTERNAL] Re: [awips2-users] EDEX to CAVE latency with multiple 
simultaneous users

Jason,

We experienced an issue similar to what you're describing and for us the 
culprit was insufficient disk I/O on the EDEX server, even with an array of 
eight 10K RPM 12G SAS hard disks.

When a class started launching CAVE and loading data not only would their 
clients slow down (even menus would take time to populate), but we also saw 
data processing latency on EDEX start climbing until the class was done loading 
their initial data sets.  In a few cases the EDEX server could not catch up 
with processing and we had to stop the LDM and give EDEX a chance to clear the 
backlog.

Monitoring with top we saw IO-waits typically between 5% and 10%, with 
instances as high as 20%

It's worth noting we originally had possibly the most inefficient disk set up 
imaginable: EDEX was running on a VM and virtual storage, so the hypervisor was 
dealing with two layers of file systems on a RAID 5 array.  That was a lot of 
extra work...

We replaced the hard disk array with a PCIe RAID card with two NVMe SSDs, 
directly attached the SSD array to the VM, and the difference was MIND BLOWING. 
 IO-wait stays below 1% and data processing latency messages have disappeared 
entirely, even when a class of 30 students are using CAVE.  We even saw CPU 
usage drop significantly, probably because very little time is being wasted 
waiting for read/write ops now.

Robert Haley
Weather Systems Administrator
Applied Aviation Sciences, College of Aviation
600. S. Clyde Morris Blvd.
Daytona Beach, FL 32114
386.323.8033
haley787@xxxxxxxx
Embry-Riddle Aeronautical University
Florida | Arizona | Worldwide

-----Original Message-----
From: awips2-users-bounces@xxxxxxxxxxxxxxxx 
<awips2-users-bounces@xxxxxxxxxxxxxxxx> On Behalf Of Kaiser, Jason N.
Sent: Wednesday, April 3, 2019 12:36 PM
To: Michael James <mjames@xxxxxxxx>
Cc: awips2-users@xxxxxxxxxxxxxxxx
Subject: [EXTERNAL] Re: [awips2-users] EDEX to CAVE latency with multiple 
simultaneous users

Hi Michael,

/awips2/cave/ is locally mounted, on each SSD. Only the home directories are 
NFS-mounted. Multiple sessions of CAVE are run as different users (i.e. 
students are each logged in to Linux with their own user account), meaning that 
you’re correct, no two users should be reading/writing to the same ~/caveData 
directory at the same time. I will try turning off data cacheing and see if 
that alleviates the problem.

-Jason


From: Michael James <mjames@xxxxxxxx>
Sent: Wednesday, April 3, 2019 10:52 AM
To: Kaiser, Jason N. <jason.kaiser@xxxxxxxxxxxxxxxxxxx>
Cc: awips2-users@xxxxxxxxxxxxxxxx
Subject: Re: [awips2-users] EDEX to CAVE latency with multiple simultaneous 
users

Hi Jason,

I don't believe that CAVE using an NFS-mounted user home directory should 
result in the performance issues you are experiencing, but I wonder if multiple 
users running the same CAVE executable over NFS could cause this... is that how 
the application is being used? (meaning /awips2/cave/ is on an NFS mount and 
each users is running the app from that mount?). In our classrooms we have seen 
no issues with multiple CAVE clients connecting to a single server and I have 
not seen network latency caused by multiple clients connecting at the same time.

Can we confirm that the multiple session of CAVE are run as different users, 
meaning no two users would be reading/writing the same ~/caveData directory at 
the same time?

Perhaps turning off data cacheing (CAVE > Preferences > Cache) would reduce the 
latency to an acceptable level?


_______________________________________________
NOTE: All exchanges posted to Unidata maintained email lists are recorded in 
the Unidata inquiry tracking system and made publicly available through the 
web.  Users who post to any of the lists we maintain are reminded to remove any 
personal information that they do not want to be made public.


awips2-users mailing list
awips2-users@xxxxxxxxxxxxxxxx
For list information, to unsubscribe, or change your membership options, visit: 
http://www.unidata.ucar.edu/mailing_lists/
  • 2019 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the awips2-users archives: