Hello,
We have a single EDEX server (CentOS 6, running the latest 18.1.1-6 release)
with fairly good specs: 10 physical CPU cores, 128GB RAM, and a 1.2TB 10K RPM
SAS 6Gb/s hard drive (not an SSD).
We have a lab of CAVE clients (running CentOS 7, that I haven't yet had a
chance to update to the latest AWIPS release), with 6 physical CPU cores, 32 GB
RAM, ~120GB SSD, and a 2GB NVIDIA graphics card.
With about 6 clients trying to load CAVE simultaneously, just trying to start
CAVE, without loading any products, resulted in CAVE "not responding" after the
splash screen, but before the CAVE window could completely load, and CentOS 7
asking the user whether they wanted to "wait" or "quit" CAVE for about half of
the clients. I tried deleting ~/caveData for one user, but this didn't appear
to resolve the problem. On the clients that were able to load CAVE
successfully, it was still was noticeably slow (took a couple minutes) to load
any products (they were trying to load the GOES-16 IR CONUS sector with 5-min
GLM data overlaid). Given the relatively good specs for both the EDEX server
and CAVE clients, my guess is the bottleneck is occurring in the network
connectivity between EDEX server and CAVE clients, but that it's only
noticeable when several CAVE clients are trying to simultaneously connect (as
occurs frequently in a classroom environment). The professor attempted to load
a single CAVE instance prior to class (without anyone else trying to load
CAVE), and was able to open CAVE relatively quickly, and load products as
usual. Perhaps exacerbating the problem is that the home directories of all of
the lab machines are NFS mounted on a separate file server, which could be
causing significant latency to occur in network and/or drive I/O speeds
(associated with reading and writing to/from the NFS-mounted ~/caveData
directories). Is there a file that logs the back-and-forth EDEX to CAVE
communication/file syncing that could show any latency? Or something else I
should be looking at to troubleshoot this? If anyone else has had trouble
loading CAVE on 6+ Linux machines, all at once, how was that resolved?
Thanks,
Jason
_________________________________________
Jason N. T. Kaiser
Atmospheric Sciences Data Systems Administrator
Northern Vermont University-Lyndon (formerly Lyndon State College)
