[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[LDM #MIW-275261]: some questions on hardware configs for our LDM server



Greg,

That was helpful. Thanks.

In order to speed things up, would it be possible for us to log onto the system 
in question as the LDM user? If so, would you please add the following ssh(1) 
credential to the file "~ldm/.ssh/authorized_keys" (NB: the entry has to be one 
line only):

from="128.117.144.*" ssh-rsa 
AAAAB3NzaC1yc2EAAAADAQABAAACAQCsVY8X6SQ8v4yXyWE7pD4jtn1HznYUlsOZm9AjGOSR5Vi0oFDCjM3Ip/5dm8sMoYPe+DvAOAbbxk0HhdIq3tRjzVNkndM4c4NxvnNrbM0IQMnWPPTEeWMAkXWXLrHQyI+qJJvz3yc7ivX/UJg5BSIzrsMjjEvquHTDpx/iuS0GnCmadk/847hzo8AagPp3gGA5t7iCqh4Riv+obbRYA1foWZApGpF+LPP0QV22W+CllOMAJC+eXY/9zX3wN6hsUJfS+4CANCN6WzaY4w3HgYiFfPoEAYO/I8B/JHphT1VFyc8TjO0fc3+0TFzLYp0M8CH8R7+iA7jlStjDpdZFDWT7idK0afHJaa+gdos+YyIUcIHa5vdh1oiF0Ho8cab7v/HQMhnKitEmtXMEfwU85mSegXxwZt4bl/3MyvOg6OAEo0g0riKVUPUJSXameacGKwalkDFY0nGb0V0UoWmOheNEuBtvOSpWgU24EHWCuC3qWN5TrVgJZxHWugoAi7wIfzEMXRAWq0tBUZtIzBRB9dXSknyaLkSAXn89Mud4b6fR20iQW+bltHHhod6jzVegqNfPI+roIYREDHD7m2P2WQvZtG9SBMo+qv+d/uRYEZPJFxYWZIvUOU6eMiLNstQTJXE6Ov+mYiOGdkGceleMJQUaa471Q/ZiGMmh0KzwdPTLEw==
 address@hidden

One thought: if the system is I/O bound, then you might think about offloading 
the filing and decoding of data-products to back-end LDM systems that request 
from the original system, which would then be relegated to requesting and 
relaying only.

> Here's what I've been given ...
> =====================================================================================
> Sen Chiao Tue, Jul 17, 11:21 AM (21 hours ago) to me
> 
> We havenât stream GOES 16 (high def data). ITâs a huge amount and higher
> frequency.
> 
> Sen
> 
> > On Jul 17, 2018, at 11:19 AM, Greg Bulman <address@hidden> wrote:
> >
> > And this includes what you intend to process in the near future ... the
> high def data?
> >
> > On Tue, Jul 17, 2018 at 11:10 AM Sen Chiao <address@hidden> wrote:
> > I don't exactly know the amount, but you can see everything (data coming
> in) is under /ldm_data
> > We process mainly models (GFS, RUC, HRRR & NAM), satellite (GOES West and
> GOES East) and radio sondes (4 sites around the northern California).
> >
> > Sen
> >
> > On Tue, Jul 17, 2018 at 11:04 AM Greg Bulman <address@hidden> wrote:
> > Sen,
> >
> > ?In case they ask ... do you have a canned answer to ?
> > "amount or type of data we download and then how its processed"
> > ?
> > -Greg?

> address@hidden ~]$ /usr/local/ldm/bin/ldmadmin plotmetrics
> No matching data
> 
> TITAN:
> (4) 2.0TB SATA 7.2K RPM in a RAID 5     /dev/sdb1       5.4T  2.4T  2.8T  47% 
> /data
> Processor = Xeon(R) X5660 2.80GHz
> NumCPU = 24 (probably hyperthreading on)
> RAM = 24576
> load average: 3.65, 4.26, 4.34
> 
> DSK |sdb |  busy    101%  | read     278  | write   3765  |  KiB/r     17 || 
> KiB/w     11  |  MBr/s   0.46 |  MBw/s   4.20 |  avq   152.78  | avio 2.47 ms 
>  |
> 
> 3370                 -                 0K             120.1M    0K            
>   49%             pqact
> 3368                 -                 0K             69316K    0K            
>   28%             pqact
> 29553                 -             25376K                 0K    0K           
>    10%             clamscan
> 3378                 -                 0K              7344K    0K            
>    3%             pqact
> 1645                 -                 0K              6772K    0K            
>    3%             jbd2/sdb1-8
> 9677                 -                 0K              5608K    0K            
>    2%             dcgrib2
> 13547                 -              5032K                 0K    0K           
>     2%             find
> 3377                 -               284K              4016K   20K            
>    2%             pqact
> 9202                 -              1512K                 8K    4K            
>    2%             gdplot2
> 
> Everything seems to run mainly on cpu01
> PRC |  sys    7.80s |  user  17.56s  |               | #proc    649  | #trun  
>     2 |  #tslpi   764  | #tslpu     0  |  #zombie    0 |  clones 37 |         
>        | #exit      36  |
> CPU |  sys      62% |  user    161%  | irq       4%  |               | idle   
> 2118% |  wait     56%  |               |  steal     0% |  guest  0% |  curf 
> 1.75GHz  | curscal  62%  |
> cpu |  sys       3% |  user     44%  | irq       0%  |               | idle   
>   19% |  cpu000 w 34%  |               |  steal     0% |  guest  0% |  curf 
> 1.60GHz  | curscal  57%  |
> cpu |  sys       6% |  user     28%  | irq       0%  |               | idle   
>   59% |  cpu001 w  7%  |               |  steal     0% |  guest  0% |  curf 
> 2.53GHz  | curscal  90%  |
> cpu |  sys       9% |  user     12%  | irq       0%  |               | idle   
>   78% |  cpu004 w  0%  |               |  steal     0% |  guest  0% |  curf 
> 2.00GHz  | curscal  71%  |
> cpu |  sys       6% |  user     11%  | irq       1%  |               | idle   
>   82% |  cpu003 w  1%  |               |  steal     0% |  guest  0% |  curf 
> 1.60GHz  | curscal  57%  |
> cpu |  sys       6% |  user      9%  | irq       0%  |               | idle   
>   85% |  cpu016 w  0%  |               |  steal     0% |  guest  0% |  curf 
> 2.79GHz  | curscal 100%  |
> cpu |  sys       4% |  user     10%  | irq       1%  |               | idle   
>   83% |  cpu002 w  2%  |               |  steal     0% |  guest  0% |  curf 
> 1.60GHz  | curscal  57%  |
> cpu |  sys       4% |  user     10%  | irq       1%  |               | idle   
>   84% |  cpu013 w  1%  |               |  steal     0% |  guest  0% |  curf 
> 1.60GHz  | curscal  57%  |
> cpu |  sys       1% |  user      9%  | irq       0%  |               | idle   
>   85% |  cpu012 w  5%  |               |  steal     0% |  guest  0% |  curf 
> 2.79GHz  | curscal 100%  |
> cpu |  sys       3% |  user      3%  | irq       1%  |               | idle   
>   93% |  cpu007 w  0%  |               |  steal     0% |  guest  0% |  curf 
> 1.60GHz  | curscal  57%  |
> cpu |  sys       3% |  user      4%  | irq       0%  |               | idle   
>   93% |  cpu005 w  0%  |               |  steal     0% |  guest  0% |  curf 
> 1.60GHz  | curscal  57%  |
> cpu |  sys       3% |  user      2%  | irq       0%  |               | idle   
>   95% |  cpu006 w  0%  |               |  steal     0% |  guest  0% |  curf 
> 1.60GHz  | curscal  57%  |
> cpu |  sys       3% |  user      4%  | irq       0%  |               | idle   
>   93% |  cpu009 w  0%  |               |  steal     0% |  guest  0% |  curf 
> 1.60GHz  | curscal  57%  |
> cpu |  sys       2% |  user      3%  | irq       0%  |               | idle   
>   95% |  cpu021 w  0%  |               |  steal     0% |  guest  0% |  curf 
> 1.60GHz  | curscal  57%  |
> cpu |  sys       3% |  user      2%  | irq       0%  |               | idle   
>   95% |  cpu008 w  1%  |               |  steal     0% |  guest  0% |  curf 
> 1.60GHz  | curscal  57%  |
> cpu |  sys       2% |  user      3%  | irq       0%  |               | idle   
>   95% |  cpu011 w  1%  |               |  steal     0% |  guest  0% |  curf 
> 1.60GHz  | curscal  57%  |
> cpu |  sys       1% |  user      1%  | irq       0%  |               | idle   
>   96% |  cpu015 w  1%  |               |  steal     0% |  guest  0% |  curf 
> 1.60GHz  | curscal  57%  |
> cpu |  sys       1% |  user      1%  | irq       0%  |               | idle   
>   97% |  cpu010 w  0%  |               |  steal     0% |  guest  0% |  curf 
> 1.60GHz  | curscal  57%  |
> cpu |  sys       1% |  user      2%  | irq       0%  |               | idle   
>   97% |  cpu017 w  0%  |               |  steal     0% |  guest  0% |  curf 
> 1.60GHz  | curscal  57%  |
> cpu |  sys       1% |  user      1%  | irq       0%  |               | idle   
>   97% |  cpu014 w  1%  |               |  steal     0% |  guest  0% |  curf 
> 1.60GHz  | curscal  57%  |
> cpu |  sys       1% |  user      1%  | irq       0%  |               | idle   
>   98% |  cpu019 w  1%  |               |  steal     0% |  guest  0% |  curf 
> 1.60GHz  | curscal  57%  |
> cpu |  sys       1% |  user      1%  | irq       0%  |               | idle   
>   98% |  cpu022 w  0%  |               |  steal     0% |  guest  0% |  curf 
> 1.60GHz  | curscal  57%  |
> cpu |  sys       1% |  user      1%  | irq       0%  |               | idle   
>   98% |  cpu018 w  0%  |               |  steal     0% |  guest  0% |  curf 
> 1.60GHz  | curscal  57%  |
> cpu |  sys       0% |  user      0%  | irq       0%  |               | idle   
>  100% |  cpu020 w  0%  |               |  steal     0% |  guest  0% |  curf 
> 1.60GHz  | curscal  57%  |
> cpu |  sys       0% |  user      0%  | irq       0%  |               | idle   
>  100% |  cpu023 w  0%  |               |  steal     0% |  guest  0% |  curf 
> 1.60GHz  | curscal  57%  |
> CPL |  avg1    2.04 |  avg5    2.10  |               | avg15   2.20  |        
>   |                | csw  3068954  |  intr  228250 |  |                | 
> numcpu    24
> 
> ?If you need more info please let me know.  I can bring in our researcher
> too who
> hopefully can add info.?

Regards,
Steve Emmerson

Ticket Details
===================
Ticket ID: MIW-275261
Department: Support LDM
Priority: Normal
Status: Closed
===================
NOTE: All email exchanges with Unidata User Support are recorded in the Unidata 
inquiry tracking system and then made publicly available through the web.  If 
you do not want to have your interactions made available in this way, you must 
let us know in each email you send to us.



NOTE: All email exchanges with Unidata User Support are recorded in the Unidata inquiry tracking system and then made publicly available through the web. If you do not want to have your interactions made available in this way, you must let us know in each email you send to us.