[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Huge Files

Hi Brian,
With some help from others I've tracked the problem down to a problem with /home/ldm/util/ucnids. Those very large files are filled not with data but with this:
ldm@ldm:/mnt/ldmdata/NNEXRAD_NCR/KOAX/20041123> tail KOAX_SDUS53_NCR_200411231638.raw 

Read: 0 21 1
Cpy: 21
Read: 0 21 1
Cpy: 21
Read: 0 21 1
Cpy: 21
Read: 0 21 1
Cpy: 21
Read: 0 21 1
Cpy: 21

which is a ucnids issue.  If you execute this:
        strings /home/ldm/util/ucnids
you'll see that the above is ucnids output, i.e., output from this command includes these lines: 
        Read: %d %d %d
        Cpy: %d
which are the lines that are generating the above. Maybe ucnids is trying to simultaneously read and write the same file? I would carefully examine the relevant pqact.conf entries to see if you're having some sort of a race condition that is causing this.
Please note that this has nothing to do with the upgrade. It must have been occuring before the upgrade as well.
Are you using WXP? We don't support WXP and haven't for some time. FWIW, our GEMPAK expert here pointed out that if you're using GEMPAK or MCIDAS you don't need to uncompress the radar data upon receipt as both those packages will uncompress for you when you ask them to display the data.
You could try to debug this problem by running ucnids on a good version of the problematic data. We have the correct data files available on our archive - you can go to http://motherlode.ucar.edu/ and get the correct version of the corrupted files and try running ucnids on them, which might reveal the problem. If ucnids runs properly on the good data from the command line then the problem is with interaction across or within pqact.conf entries.
If this is really a problem for you I may be able to help you after the holiday. Unfortunately, I can't debug this problem for you now. I was planning on starting my vacation over an hour ago. For now could you deal with the problem by turning off the decompression? 

Anne

Miller, Brian S wrote:

Anne-
I removed all of the huge files from /home, but there are still a few in
/mnt/ldmdata/NNEXRAD_NCR
In that directory there are 6 radars.  Look at the data for today in
each radar folder and you will see what I am talking about.
Thanks again.
Brian

-----Original Message-----
From: Anne Wilson [mailto:address@hidden] Sent: Tuesday, November 23, 2004 3:25 PM 
To: Miller, Brian S
Subject: Re: Huge Files

Hi Brian,

Where are these files?  Please respond ASAP if you can.

Anne

Miller, Brian S wrote:


Anne-

Since the upgrade there are random HUGE files that are being created
on 
our LDM server.  Every now and then instead of having a 1MB file there




is a 2GB file in its place.
Can you take a look at what it is doing and let me know if I can do anything to stop the "bad" data files? It is filling up my storage 

fast.


Brian






--
***************************************************
Anne Wilson                     UCAR Unidata Program            
address@hidden                 P.O. Box 3000
                                  Boulder, CO  80307
----------------------------------------------------
Unidata WWW server       http://my.unidata.ucar.edu/
****************************************************