[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[netCDF #OML-578835]: AREA to NetCDF conversion



Hi Tom,

re:
> Hey Tom... I noticed you must have generated all those netcdf
> files on cacimbo. Did you do that with a script or individually?

As you found out later (next email), I did this with a script last night.

> If it was a script, can you tell me where it is? IMGCPY seems
> pretty straightforward, but I'm not clear about getting the
> images into an ADDE dataset.

I created the script 'area2ncdf.sh' in the ~mcidas/workdata directory
as an edited copy of ~mcidas/data/mcrun.sh:

<as 'mcidas'>
cd $MCDATA
cp ../data/mcrun.sh area2ncdf.sh
-- edit area2ncdf.sh and put in the IMGCOPY command

I had not written to you about this because I intended to enhance the
script to create more descriptive output file names than the ones
produced currently.  If you do not need this enhancement, I will stop
now.

> I turned the netcdf files over to my student, and the look almost
> identical to the G8 file we created with AreaToNetCDF
> (which is good!).

Yes, this is very good :-)

> I figured this would be something that we would spend a couple
> of weeks just trying to figure out how to do. Thanks for jumping
> on there!

No worries.  I'm glad to help if/when I can.

> Actually, I think I found it. I saw your script in the workdata directory.
> Thanks!

Exactly.  Now here is what I did as a precursor to creating and running the
script:

<as 'mcidas'>

cd /data/goesarchive
mkdir AREA netcdf
mv goes12*.BAND_04 AREA

cd $MCDATA

dsserve.k ADD GARCH/G12IR AREA DIRFILE=/data/goesarchive/AREA/goes12.\*.BAND_04 
\"GOES-12 10.7 um Archive images
dsserve.k ADD GARCH/NCDF NCDF 1 9999 TYPE=IMAGE \"netCDF GOES image files

redirect.k ADD NCDF\* \"/data/goesarchive/netcdf

-- test the input dataset setup:

imglist.k NCDF/G12IR.ALL

-- test the conversion from AREA to netCDF:

imgcopy.k GARCH/G12IR.1 GARCH/NCDF.1 SIZE=674 1188 BAND=4 UNIT=TEMP
/home/mcidas/mcidas2007/netcdf/ncdump/ncdump 
/data/goesarchive/netcdf/NCDF0001.nc

What I think is lacking in the 'area2ncdf.sh' script is:

- getting the size of the input image so that the SIZE= keyword specification is
  not hardwired

- getting a number of useful pieces of information from the input image (e.g., 
date,
  time, etc.) and using them in an informative name for the output file name

  For instance, I would have preferred to change the output file names after 
their
  creation to something like:

  goes12_20030429_0245_band4.nc

  etc.  In order to do this, I have to sharpen up my Bourne shell scripting 
skills :-)

  The information I was wanting to use comes from an expanded listing output.  
For
  instance:

imglist.k GARCH/G12IR.1 FORM=ALL

Image file directory listing for:GARCH/G12IR
 Pos Satellite/         Date       Time      Center      Res (km)   Image_Size
     sensor                                 Lat  Lon    Lat   Lon
 --- -------------  ------------  --------  ---- ----  ----- ----- ------------
   1  G-12 IMG      29 APR 03119  02:45:00     0   75
   Band: 4    10.7 um IR Surface/Cloud-top Temp        16.12  9.15   674 x 1188
     proj:    0 created: 2007241  64416  memo: RT GVAR
     type:GVAR     cal type:RAW
     offsets:  data=    3328 navigation=  256 calibration= 2816 auxiliary=    0
     doc length: 228   cal length:   0   lev length:   0 PREFIX= 232
     valcod:  119024500 zcor:  0 avg-smp: N
     start yyddd: 2003119  start time: 24514  start scan:  306
     lcor: 2469  ecor:  5881  bytes per pixel: 2  ss: 78
     Resolution Factors (base=1):   Line=   16.0   Element=   16.0
imglist.k: done

  The output file would be renamed after its creation by imgcopy.k using 
something
  like (pseudocode):

  mv /data/goesarchive/netcdf/NCDF0001.sh 
/data/goesarchive/netcdf/goes12_20030429_0245_band4.nc

  etc.

Now on to other comments:

I was taking a look at how you are processing data using LDM's pqact, and I 
noticed that
in some cases you are processing the same data (e.g., NIMAGE) three times.  
Evidently
you want to store the data for use in GEMPAK and also write it off to some 
larger
store.  Here is one instance of the latter case:

NIMAGE  ^TIGQ52 .... ([0-3][0-9])([0-2][0-9])([0-5][0-9])
        FILE    -close  
data/gempak/images/sat/goeseast/w_atlantic/06sounding/WATSND06_(\1:yyyy)(\1:mm)\1_\2\3

There is nothing wrong in processing the same data more than once in different 
ways, but
in the NIMAGE and UNIWISC (aka MCIDAS) cases, you could actually be stepping 
out the output
data files.  Consider:

-- pqact.conf line 1967
MCIDAS  ^pnga2area Q. (U[^ACXY1]) (.*) (.*)_IMG (0.65)um (.*) (........) (....)
        PIPE    -close
        /usr/local/ldm/ldm-mcidas/bin/pnga2area -vl 
/usr/local/ldm/logs/ldm-mcidas.log
        /usr/local/ldm/data/gempak/images/sat/\3/\5/VIS/VIS_\6_\7


-- and pqact.conf line 3428
UNIWISC ^pnga2area Q. (U[^ACXY13]) (.*) (.*)_IMG (0.65)um (.*) (........) (....)
        PIPE    -close
        /usr/local/ldm/ldm-mcidas/bin/pnga2area -vl logs/ldm-mcidas.log
        -a etc/SATANNOT
        -b etc/SATBAND
        data/gempak/images/sat/\3/\5/VIS/VIS_\6_\7

These two actions decode the same product into the same output file since
/usr/local/ldm/data/gempak/images/sat is the same directory as 
~ldm/data/gempak/images/sat.

This kind of double processing of the same product into the same output file 
can have
two kinds of problems:

- unnecessary use of CPU resources 
- if the actions were in two different pqact.conf files, the write by one 
invocation
  of pnga2area could stomp on the other invocation of pnga2area

I suggest that it would be a good idea to do some housekeeping in the way that
you are processing products using pqact.  In particular, I would:

- move processing actions needed by McIDAS into a pqact.conf_mcidas 
pattern-action file
- move processing actions needed by GEMPAK into a pqact.conf_gempak 
pattern-action file
- leave all other actions in the pqact.conf file

After doing this more, I would modify ~ldm/etc/ldmd.conf to have three pqact 
invocations:

#
# McIDAS-XCD actions
#
exec    "xcd_run MONITOR"
exec    "pqact -f FSL2|FNEXRAD|NLDN|UNIDATA etc/pqact.conf_mcidas"
#
# GEMPAK actions
#
exec    "pqact -f FSL2|FNEXRAD|NIMAGE|NLDN|UNIDATA etc/pqact.conf_gempak"
#
# Other actions
exec    "pqact -f <list of relevant feed types> etc/pqact.conf"
#
# rtstats uses the LDM to send product latency statistics to the UPC.
# For more info on rtstats please see the man page.
EXEC    "rtstats -h rtstats.unidata.ucar.edu"

If you adopt this approach, you must make sure that the list of feeds specified
in the ldmd.conf 'exec' invocation contains all of the feeds for which there
are patterns in the corresponding pqact.conf_x pattern action file and no 
others.
Also, you must make sure that there is no duplication of processing in the 
separate
pattern-action files, or at least no duplication of the same processing in the
separate pattern-action files.

Segmenting the processing this way will help to conserve your system resources
and make your processing as efficient as possible.  It may be the case that
you do not need to worry about system resources at the moment, but the volume
of data relayed by the IDD is continuously increasing, so a little tweaking now
will have significant benefits in the future.

Also, since I am not able to logon as 'ldm', I was not able to review how you
have data scouring setup.  Data scouring is one of the big areas where one can
significantly save on system resources by doing some planning.

Last item:

The reason I asked you to run 'sh ./mcinet2007.sh install mcadde' as 'root' was
to re-enable access to cacimbo via ADDE.  Since this is now working nicely, I
am wondering if you are amenable to cacimbo once again acting as one of the
cooperating a community servers?

Cheers,

Tom
****************************************************************************
Unidata User Support                                    UCAR Unidata Program
(303) 497-8642                                                 P.O. Box 3000
address@hidden                                   Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage                       http://www.unidata.ucar.edu
****************************************************************************


Ticket Details
===================
Ticket ID: OML-578835
Department: Support McIDAS
Priority: Normal
Status: Closed