[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

20050422: building ldm-mcidas on Debian Linux

>From: Gilbert Sebenste <address@hidden>
>Organization: NIU
>Keywords: 200504221620.j3MGK3v2005961 ldm-mcidas Debian Linux build

Gilbert and Dave,

I logged onto climate.cod.edu and built the ldm-mcidas v2004 decoders
after defining needed environment variables.  The key to the successful
build was defining CC to be 'gcc' and FC to be 'g77'.  The build attempt
that Dave tried before did not define CC so 'configure' was defaulting
to use C89.

I put the environment variable definitions in
/home/ldm/ldm-mcidas/ldm-mcidas.environment so they can be used
in the future.

For "the files", the commands I used to set the environment variables were:

setenv CC         gcc
setenv CPPFLAGS   '-DNDEBUG -Df2cFortran'
setenv CFLAGS     -O
setenv FC         g77
setenv FFLAGS     '-O -Wuninitialized -fno-automatic -ff90-intrinsics-hide'
setenv CXX        ''
setenv F90        ''
setenv CPP_LDM    -I/home/ldm/include
setenv LD_LDM     '-L/home/ldm/lib -lldm'
setenv CPP_MCIDAS -I/home/mcidas/include
setenv LD_MCIDAS  '-L/home/mcidas/lib -lmcidas'
setenv CPP_NETCDF -I/home/mcidas/include
setenv LD_NETCDF  '-L/home/mcidas/lib -lnetcdf'
setenv LD_X11     '-L/usr/X11R6/lib -lX11'

I also:

- created the /home/ldm/ldm-mcidas directory and put all ldm-mcidas 
distributions
  under it

- installed the newly built ldm-mcidas decoders in /home/ldm/decoders

- copied /home/ldm/ldm-mcidas/ldm-mcidas-2004/etc/SAT* to /home/ldm/etc

- deleted obsolete ldm-mcidas decoders from /home/ldm/decoders

- turned on ldm-mcidas NLDN lightning decoding

- reregistered the WPRO and WPR6 schema in the SCHEMA file:

  <as 'mcidas'>
  cd ~mcidas/workdata
  dmap.k SCHEMA
  sche.k DCWPRO
  sche.k DCWPR6

  I did this because the versions in /home/data/mcidas/SCHEMA did not
  allow for the creation of MD files with enough columns to handle all
  of the profiler sites that are now reporting

- tweeked the scouring setup in /home/mcidas/workdata/mcscour.sh.  The
  changes were minor:

  - add scouring of MD files 101 - 120
  - add scouring of XCD files over the past 100 days (was 10)

Everything appears to be working correctly now.  Please check to make
sure you agree with this.

Recommendation:

- you are not rotating your ldm-mcidas log files, and they are getting
  very large.  I strongly advise you to setup rotation of the files
  in a crontab entry for 'ldm'.  Here is one example:

#
# Shift LDM-McIDAS log files
#
0 17 * * * bin/newlog logs/ldm-mcidas.log

  Since you are logging ldm-mcidas decoded products to different files, you
  will need to modify this example entry to match your setup.

FYI, I tried building a v2004 binary distribution of ldm-mcidas on our
Fedora Core 3 32-bit Linux machine and using it on climate.  The
proftomd and nldn2md decoders failed in the exact same way as the
binary distribution you had already grabbed.  Verdict:  Debian is
significantly different from RedHat and Fedore Core Linux.

Please let me know if you see anything amiss on climate.

Cheers,

Tom

>> >From: Gilbert Sebenste <address@hidden>
>> >Organization: NIU
>> >Keywords: 200504221620.j3MGK3v2005961 ldm-mcidas proftomd
>> 
>> Hi Gilbert,
>> 
>> >What would cause this in the ldmd.conf error log:
>> >
>> >Apr 22 15:49:31 climate pqact[32292]: pbuf_flush (5) write: Broken pipe
>> >Apr 22 15:49:31 climate pqact[32292]: pipe_put:
>> >-closedecoders/proftomd-vllogs/ldm-mcidas.profiler.log-d/home/data/mcidas-v
> U6W
>> > P$
>> >write error
>> >Apr 22 15:49:31 climate pqact[32292]: pipe_prodput: trying again
>> >Apr 22 15:49:31 climate pqact[32292]: pbuf_flush (5) write: Broken pipe
>> >Apr 22 15:49:31 climate pqact[32292]: pipe_put:
>> >-closedecoders/proftomd-vllogs/ldm-mcidas.profiler.log-d/home/data/mcidas-v
> U6W
>> > P$
>> >write error
>> >Apr 22 15:49:31 climate pqact[32292]: child 503 exited with status 1
>> >Apr 22 15:49:31 climate pqact[32292]: child 505 exited with status 1
>> 
>> This is most likely caused by one of two things:
>> 
>> - the output MD file into which proftomd wants to write exists and is
>>   not writable by the user attempting the write ('ldm')
>> 
>> - more likey, the McIDAS files needed for the decoder do not exist, or
>>   are not readable/viable in the directory in which the decoder wants
>>   to write its output
>> 
>> I am betting that the problem is the second bullet.  The solution (if this
>> is the problem) is to copy the needed files to the output directory:
>> 
>> <as 'mcidas'>
>> cd ~mcidas
>> cp data/SCHEMA /home/data/mcidas         <- the file most probably missing/b
> ad
>> cp data/SYSKEY.TAB /home/data/mcidas     <- copy may not be needed if file i
> s OK
>> cp workdata/ROUTE.SYS /home/data/mcidas  <- copy may not be needed if file i
> s OK
>> cd /home/data/mcidas
>> chmod 666 SCHEMA SYSKEY.TAB ROUTE.SYS
>
>Those were already there and they appear to be fine.
> 
>> After these changes the decoder should start working _if_ the output MD
>> file does not already exist (if it already exists, it may be full or
>> not writable by 'ldm').
>
>It isn't full, and it is writable by the LDM. I just verified both.
> 
>> >FSL2    ^FSL\.NetCDF\.NOAAnet\.windprofiler\.06min
>> >        PIPE
>> >        -close decoders/proftomd -vl logs/ldm-mcidas.profiler.log
>> >        -d /home/data/mcidas -v U6 WPR6 91
>> 
>> No problems with this entry.
>
>Yep, double-checked it.
> 
>> >I am helping out COD because they are having problems with this, and I am 
>> >donating a little time to get this fixed. I can verify that in 
>> >/home/data/mcidas, the files are getting written to, and file sizes are 
>> >normal. The MDXX008* files are fine and the pqact entry generates no 
>> >errors.
>> 
>> So, you are saying that the hourly summary profiler data is being decoded
>> correctly, but the 6 minute profiler data is not?
>
>No, because now, as I have just noticed, the MDXX0080 files aren't 
>decoding either. So what I did was:
>
>Went into pqact.conf and hard-coded all locations:
>
># FSL2 wind profiler
>FSL2    ^FSL\.NetCDF\.NOAAnet\.windprofiler\.01hr
>        PIPE
>        -close /home/ldm/decoders/proftomd -vl 
>/home/ldm/logs/ldm-mcidas.profil$
>        -d /home/data/mcidas -v U2 WPRO 81
>
>FSL2    ^FSL\.NetCDF\.NOAAnet\.windprofiler\.06min
>        PIPE
>        -close /home/ldm/decoders/proftomd -vl 
>/home/ldm/logs/ldm-mcidas.profil$
>        -d /home/data/mcidas -v U6 WPR6 91
>
>(There are tabs there where there should be in front of the ^FSL,
>PIPE, and close and -d lines above in each, not spaces). But I still get:
>
>Apr 22 21:37:35 climate pqact[32292]: pbuf_flush (5) write: Broken pipe 
>Apr 22 21:37:35 climate pqact[32292]: pipe_put: 
>-close/home/ldm/decoders/proftomd-vl/home/ldm/logs/ldm-mcidas.profiler.log-d/h
> ome/data/mcidas-vU2WPRO81 
>write error 
>Apr 22 21:37:35 climate pqact[32292]: pipe_prodput: trying again 
>Apr 22 21:37:35 climate pqact[32292]: pbuf_flush (5) write: Broken pipe 
>Apr 22 21:37:35 climate pqact[32292]: pipe_put: 
>-close/home/ldm/decoders/proftomd-vl/home/ldm/logs/ldm-mcidas.profiler.log-d/h
> ome/data/mcidas-vU2WPRO81 
>write error
>
> > >I am 
>wondering if they didn't include something in a 
>McIDAS config > >file?
>> 
>> Hmm...  if hourly summary decoding is OK, it should mean that the copies
>> of SCHEMA, SYSKEY.TAB, and ROUTE.SYS in /home/mcidas/data are OK.  If
>> this is the case, the example copies I listed above should not be needed.
>
>No it is not, unfortunately.
> 
>> Let's see if the schema for the 6 minute profiler data is correctly included
>> in /home/data/mcidas/SCHEMA:
>> 
>> <as 'mcidas'>
>> cd workdata
>> redirect.k ADD SCHEMA \"/home/data/mcidas
>> lsche.k WPR6
>> 
>> The listing should look like:
>> 
>> /home/mcidas/workdata% lsche.k WPR6
>> 
>> NAME: WPR6  VERSION:  2  DATE: 1998338  TEXTID: "6-MIN WIND PROFILER UPPER A
> IR D
>> ----        -------
>>    DEFAULT NUMBER OF ROWS:  480                        INTEGER ID: 0
>>                      COLS:     50                   MISSING DATA VALUE:  -21
> 39062144
>>    REPEAT GROUP:  NUMBER OF REPETITIONS:  37
>>                       STARTING POSITION:  16
>>                                    SIZE:   4
>>    NUMBER OF KEYS IN ROW HEADER:     4
>>                      COL HEADER:     0   STARTING AT POSITION   5
>>                     DATA RECORD:   159   STARTING AT POSITION   5
>>                                   ----
>>                                    163 TOTAL
>> 
>>         KEY  SCALE UNIT     KEY  SCALE UNIT     KEY  SCALE UNIT
>>         ---- ----- ----     ---- ----- ----     ---- ----- ----
>>         DAY    0   CYD      TIME   0   HMS      MODE   0   CHAR
>>         CMAX   0            MOD    0            IDA    0   CHAR
>>         IDB    0   CHAR     IDN    0            LAT    4   DEG
>>         LON    4   DEG      ZS     0   M        PSL    2   MB
>>         T      2   K        TD     2   K        PCP    2   MM
>>         Z      0   M        SPD    2   MPS      DIR    0   DEG
>>         VWND   2   MPS
>> lsche.k: DONE
>
>yep, looks fine to me.
> 
>> If it does not (like nothing gets listed), then do the following:
>> 
>> sche.k DCWPR6
>> lsche.k WPR6
>> 
>> The lsche.k invocation should then show that appropriate listing for the 6
>> minute profiler schema.  If it didn't to begin with but does after registeri
> ng
>> the schema, and IF the output MD files for the 6 minute data (MDXX009*) do
>> not exist, they should now be created.
>> 
>> Cheers,
>
>Hmm. Still not being created. Not sure what is happening here.

>From address@hidden  Fri Apr 22 15:51:51 2005
>
>Tom,
>
>You should also know that all of the other MD files are updating fine. 
>But...they upgraded from ldm-mcidas 2003 to 2004 last night. Everything 
>else is working fine except for the profiler data. All other MD's are 
>being created and are being updated. Is there a patch we need to know 
>about?

>From address@hidden  Fri Apr 22 16:00:25 2005
>
>Tom,
>
>Found this.
>
>/home/ldm/decoders/ >proftomd
>
>proftomd: /lib/libc.so.6: version `GLIBC_2.3' not found (required by 
>proftomd)
>
>Well, THAT says it all. He's running the latest version of Debian,
>3.0R5, which came out April 16.
>
>******************************************************************************
> *
>Gilbert Sebenste                                                     ********
>(My opinions only!)                                                  ******
>Staff Meteorologist, Northern Illinois University                      ****
>E-mail: address@hidden                                  ***
>web: http://weather.admin.niu.edu                                      **
>Work phone: 815-753-5492                                                *
>******************************************************************************
> *
>
>From address@hidden  Fri Apr 22 16:05:11 2005
>
>Tom,
>
>I have permission for you to log into climate.cod.edu as ldm and mcidas
>and find out how to compile the LDM decoders 
>on his Debian system. The Linux binaries will not work for many of the 
>binaries, as it turns out. /home/ldm/decoders is where they are.
>
>Gilbert

--
NOTE: All email exchanges with Unidata User Support are recorded in the
Unidata inquiry tracking system and then made publicly available
through the web.  If you do not want to have your interactions made
available in this way, you must let us know in each email you send to us.