[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

20000330: McIDAS-X, DSSERVE hanging during system test phase

Hi Tom,

You probably were hoping NOT to hear from me about McIDAS but I need
your help again.  With all the weird problems I have been having  with
this installation it is almost funny, or would be if I didn't have a
deadline to get McIDAS up and running.


I decided to start the installation from scratch (again!).  Compiling
went fine and I am now in the pre-install "testing" section.  I created
the /tmp/mctmp directory and linked .mctmp to it.  After setting
MCDATA, MCPATH, MCGUI, PATH and DISPLAY as per the web page, I can fire
up mcidas just fine. (Which is more than I could do the last time.)
The `MAP NA`, `EG`, `PANEL 3 3`, and `MAP USA PAN=5` tests work fine,
however when I try:

`DSSERVE ADD TOPO/CONF AREA 9011 9011 "North America (Conformal)`  

it hangs forever, similar to what I saw when trying to run McIDAS
withough the link from .mctmp to /tmp/mctmp.  Could this be a similar
problem?

I appreciate any help you can give!

Dave
 
>From address@hidden  Thu Mar 30 14:15:58 2000
Subject: 20000330: McIDAS-X and dsserve 

Dave,

We logged onto twister to try and find out what is going on (like before).
One thing that we see is that snowball is _not_ running /usr/lib/nfs/nfsd
AND it _SHOULD_ be!  Can you explain this?  We don't understand how
any of the NFS stuff could be working given that nfsd is not running.

Also, there is an NFS patch that could be installed, but this is probably
not the cause of your problems.

Tom

>From address@hidden  Fri Mar 31 11:37:11 2000
>Subject: McIDAS etc
>Cc: address@hidden, address@hidden

Tom,

Yes your right, nfsd is NOT running on snowball...  weird!  The startup
script /etc/init.d/nfs.server calls it correctly, and I've stopped and
started it by hand a number of times to no avail, but looking in my
/var/adm/messages file I see the following:

Mar 31 12:51:45 snowball unix: WARNING: nfsauth: RPC: Unitdata error
Mar 31 12:51:45 snowball unix:
Mar 31 12:51:46 snowball /usr/lib/nfs/nfsd[21206]: t_bind to wrong address
Mar 31 12:51:46 snowball /usr/lib/nfs/nfsd[21206]: Cannot establish NFS service
over /dev/udp: transport setup problem.
Mar 31 12:51:46 snowball /usr/lib/nfs/nfsd[21206]: t_bind to wrong address
Mar 31 12:51:46 snowball /usr/lib/nfs/nfsd[21206]: Cannot establish NFS service
over /dev/tcp: transport setup problem.
Mar 31 12:51:46 snowball /usr/lib/nfs/nfsd[21206]: Could not start NFS service f
or any protocol. Exiting.

My /etc/services file is correct in that it lists nfsd for udp and tcp
to port 2049.   We are running NIS+ here but my /etc/nsswitch.conf
file  tells services to look at local files first.  The ONLY other
thing I know that has been changed recently is that the networking
people have been putting in new switches around campus, and one is
sitting right behind me, snowball and twister both go through it.

Let me do some more digging in the sun archives and other resources to
see if I can find out how to solve this.

statd, mountd, and lockd all are running fine though and I have not been
getting any complaints from my users about not being able to run any programs
(other than McIDAS!)

I don't expect you to toubleshoot my networking problems for me, but if
you  or your sys admin knows off the top of your heads why nfsd may not
be running I'd appreciate your help, otherwise I'll dig into this from
here.

Thanks!!

Dave

>From address@hidden  Fri Mar 31 16:35:28 2000
>Subject: 20000331: McIDAS etc 

Dave,

Chiz just mentioned that you (Millersville) had gotten hacked into at
some point in the recent past; true?  Going on that, and thinking that
your 'ps' command may have been changed, I logged in and ran:

/usr/ucb/ps -ax | grep nfs
   160 ?        S  0:00 /usr/lib/nfs/statd
   163 ?        S  0:00 /usr/lib/nfs/lockd
   429 ?        S  0:00 /usr/lib/nfs/msgserv
   430 ?        S  0:00 /usr/lib/nfs/rpc.pcnfsd
  1057 ?        S  0:00 /usr/lib/nfs/nfsd -a 16
  6671 ?        S  0:00 /usr/lib/nfs/mountd

This gives a much different list than /bin/ps:

/bin/ps -eaf | grep nfs
  daemon   160     1  0 17:32:41 ?        0:00 /usr/lib/nfs/statd
    root   163     1  0 17:32:41 ?        0:00 /usr/lib/nfs/lockd
    root   429     1  0 17:33:12 ?        0:00 /usr/lib/nfs/msgserv
  mcidas 22202 16643  0 18:33:41 pts/8    0:00 grep nfs
    root  6671     1  0 17:49:58 ?        0:00 /usr/lib/nfs/mountd

Thinking that your system has been compromised, we FTPed a version of
'ps' from one of our Solaris 2.6 machines to /tmp on snowball and
ran it:

/tmp% /tmp/ps -eaf | grep nfs
  daemon   160     1  0 17:32:41 ?        0:00 /usr/lib/nfs/statd
    root   163     1  0 17:32:41 ?        0:00 /usr/lib/nfs/lockd
    root  1057     1  0 17:35:34 ?        0:00 /usr/lib/nfs/nfsd -a 16
    root   429     1  0 17:33:12 ?        0:00 /usr/lib/nfs/msgserv
    root   430     1  0 17:33:12 ?        0:00 /usr/lib/nfs/rpc.pcnfsd
  mcidas 22400 16643  0 18:34:31 pts/8    0:00 grep nfs
    root  6671     1  0 17:49:58 ?        0:00 /usr/lib/nfs/mountd

Looks like your 'ps' command has been replaced by a bogus one!  A hacked
system _could_ explain all of the weirdness that you are seeing!!

Tom

>From address@hidden  Mon Apr  3 13:20:02 2000
>Subject: 20000403: Compromised system

Dave,

Mike Schmidt and I just looked at snowball in a little more detail.
Your system has been compromised, and there are at least two routines
'ps' and 'netstat' that have been replaced by hackers.

Here is how we proved this:

<login>
which ps
/bin/ps
cd /bin
ls -l ps
-rwxr-xr-x   1 root     root        6664 Sep 24  1999 ps
<ps is suspiciously small>

strings ps
++sh0w
/bin/procs
/usr/bin/.ssh/P-T

<something is going on in the /usr/bin/.ssh directory>
cd /usr/bin/.ssh
ls -l
-rw-r--r--   1 root     root          57 Sep 24  1999 P-T
-rwxr-xr-x   1 root     root       15640 Sep 24  1999 cnb
-rwxr-xr-x   1 root     root        6860 Sep 24  1999 klim
-rw-r--r--   1 root     root       45076 Nov 17 18:20 out

file P-T
P-T:            ascii text
more P-T
milk
core
live
ppr0
eggdrop
sched
nfsd
pageout
bnc
procs

<this looks like a list of processes to hide; notice 'nfsd'>

file out:            ascii text
more out

Log started at => Wed Oct 27 21:11:01 [pid 21900]

Log started at => Wed Oct 27 21:12:10 [pid 21991]

-- TCP/IP LOG -- TM: Thu Oct 28 08:05:00 --
 PATH: snowball(58860) => ftp.wwb.noaa.gov(ftp)
 STAT: Thu Oct 28 08:05:12, 11 pkts, 138 bytes [DATA LIMIT]

<passwords are being sniffed; you should check out this file: /usr/bin/.ssh/out>

which netstat
/bin/netstat
-rwxr-xr-x   1 root     root        6672 Sep 24  1999 netstat

<this is too small!>
++sh0w
/bin/neterm
/usr/bin/.ssh/P-N

<same thing going on with netstat!!>

Mike says that given the time stamps for the passwords in out, it is likely
that you successfully turned off the attacks.  Unfortunately, at least
two routines remain that were compromised.  The easiest way to get your
system back to a solid configuration (which will probably take care of
the McIDAS NFS file locking problems) is to upgrade your operating system.

Sorry for the bad news...

>From address@hidden  Mon Apr  3 13:23:56 2000
>Subject: RE: 20000403: Compromised system
>Date: Mon, 3 Apr 2000 15:19:19 -0400 

Tom,

Yep I found those this morning as well after I saw your earlier message
about the possibility of being hacked.  I will be rebuilding snowball
tomorrow evening (can't do it earlier due to classes).  I will take this
opportunity to upgrade to Solaris 7 so hopefully that will take care of the
weirdness we've been seeing.  Now time for a refresher on security!

thanks for your help!

Dave