Hi Tom, re: > Hey Tom... I noticed you must have generated all those netcdf > files on cacimbo. Did you do that with a script or individually? As you found out later (next email), I did this with a script last night. > If it was a script, can you tell me where it is? IMGCPY seems > pretty straightforward, but I'm not clear about getting the > images into an ADDE dataset. I created the script 'area2ncdf.sh' in the ~mcidas/workdata directory as an edited copy of ~mcidas/data/mcrun.sh: <as 'mcidas'> cd $MCDATA cp ../data/mcrun.sh area2ncdf.sh -- edit area2ncdf.sh and put in the IMGCOPY command I had not written to you about this because I intended to enhance the script to create more descriptive output file names than the ones produced currently. If you do not need this enhancement, I will stop now. > I turned the netcdf files over to my student, and the look almost > identical to the G8 file we created with AreaToNetCDF > (which is good!). Yes, this is very good :-) > I figured this would be something that we would spend a couple > of weeks just trying to figure out how to do. Thanks for jumping > on there! No worries. I'm glad to help if/when I can. > Actually, I think I found it. I saw your script in the workdata directory. > Thanks! Exactly. Now here is what I did as a precursor to creating and running the script: <as 'mcidas'> cd /data/goesarchive mkdir AREA netcdf mv goes12*.BAND_04 AREA cd $MCDATA dsserve.k ADD GARCH/G12IR AREA DIRFILE=/data/goesarchive/AREA/goes12.\*.BAND_04 \"GOES-12 10.7 um Archive images dsserve.k ADD GARCH/NCDF NCDF 1 9999 TYPE=IMAGE \"netCDF GOES image files redirect.k ADD NCDF\* \"/data/goesarchive/netcdf -- test the input dataset setup: imglist.k NCDF/G12IR.ALL -- test the conversion from AREA to netCDF: imgcopy.k GARCH/G12IR.1 GARCH/NCDF.1 SIZE=674 1188 BAND=4 UNIT=TEMP /home/mcidas/mcidas2007/netcdf/ncdump/ncdump /data/goesarchive/netcdf/NCDF0001.nc What I think is lacking in the 'area2ncdf.sh' script is: - getting the size of the input image so that the SIZE= keyword specification is not hardwired - getting a number of useful pieces of information from the input image (e.g., date, time, etc.) and using them in an informative name for the output file name For instance, I would have preferred to change the output file names after their creation to something like: goes12_20030429_0245_band4.nc etc. In order to do this, I have to sharpen up my Bourne shell scripting skills :-) The information I was wanting to use comes from an expanded listing output. For instance: imglist.k GARCH/G12IR.1 FORM=ALL Image file directory listing for:GARCH/G12IR Pos Satellite/ Date Time Center Res (km) Image_Size sensor Lat Lon Lat Lon --- ------------- ------------ -------- ---- ---- ----- ----- ------------ 1 G-12 IMG 29 APR 03119 02:45:00 0 75 Band: 4 10.7 um IR Surface/Cloud-top Temp 16.12 9.15 674 x 1188 proj: 0 created: 2007241 64416 memo: RT GVAR type:GVAR cal type:RAW offsets: data= 3328 navigation= 256 calibration= 2816 auxiliary= 0 doc length: 228 cal length: 0 lev length: 0 PREFIX= 232 valcod: 119024500 zcor: 0 avg-smp: N start yyddd: 2003119 start time: 24514 start scan: 306 lcor: 2469 ecor: 5881 bytes per pixel: 2 ss: 78 Resolution Factors (base=1): Line= 16.0 Element= 16.0 imglist.k: done The output file would be renamed after its creation by imgcopy.k using something like (pseudocode): mv /data/goesarchive/netcdf/NCDF0001.sh /data/goesarchive/netcdf/goes12_20030429_0245_band4.nc etc. Now on to other comments: I was taking a look at how you are processing data using LDM's pqact, and I noticed that in some cases you are processing the same data (e.g., NIMAGE) three times. Evidently you want to store the data for use in GEMPAK and also write it off to some larger store. Here is one instance of the latter case: NIMAGE ^TIGQ52 .... ([0-3][0-9])([0-2][0-9])([0-5][0-9]) FILE -close data/gempak/images/sat/goeseast/w_atlantic/06sounding/WATSND06_(\1:yyyy)(\1:mm)\1_\2\3 There is nothing wrong in processing the same data more than once in different ways, but in the NIMAGE and UNIWISC (aka MCIDAS) cases, you could actually be stepping out the output data files. Consider: -- pqact.conf line 1967 MCIDAS ^pnga2area Q. (U[^ACXY1]) (.*) (.*)_IMG (0.65)um (.*) (........) (....) PIPE -close /usr/local/ldm/ldm-mcidas/bin/pnga2area -vl /usr/local/ldm/logs/ldm-mcidas.log /usr/local/ldm/data/gempak/images/sat/\3/\5/VIS/VIS_\6_\7 -- and pqact.conf line 3428 UNIWISC ^pnga2area Q. (U[^ACXY13]) (.*) (.*)_IMG (0.65)um (.*) (........) (....) PIPE -close /usr/local/ldm/ldm-mcidas/bin/pnga2area -vl logs/ldm-mcidas.log -a etc/SATANNOT -b etc/SATBAND data/gempak/images/sat/\3/\5/VIS/VIS_\6_\7 These two actions decode the same product into the same output file since /usr/local/ldm/data/gempak/images/sat is the same directory as ~ldm/data/gempak/images/sat. This kind of double processing of the same product into the same output file can have two kinds of problems: - unnecessary use of CPU resources - if the actions were in two different pqact.conf files, the write by one invocation of pnga2area could stomp on the other invocation of pnga2area I suggest that it would be a good idea to do some housekeeping in the way that you are processing products using pqact. In particular, I would: - move processing actions needed by McIDAS into a pqact.conf_mcidas pattern-action file - move processing actions needed by GEMPAK into a pqact.conf_gempak pattern-action file - leave all other actions in the pqact.conf file After doing this more, I would modify ~ldm/etc/ldmd.conf to have three pqact invocations: # # McIDAS-XCD actions # exec "xcd_run MONITOR" exec "pqact -f FSL2|FNEXRAD|NLDN|UNIDATA etc/pqact.conf_mcidas" # # GEMPAK actions # exec "pqact -f FSL2|FNEXRAD|NIMAGE|NLDN|UNIDATA etc/pqact.conf_gempak" # # Other actions exec "pqact -f <list of relevant feed types> etc/pqact.conf" # # rtstats uses the LDM to send product latency statistics to the UPC. # For more info on rtstats please see the man page. EXEC "rtstats -h rtstats.unidata.ucar.edu" If you adopt this approach, you must make sure that the list of feeds specified in the ldmd.conf 'exec' invocation contains all of the feeds for which there are patterns in the corresponding pqact.conf_x pattern action file and no others. Also, you must make sure that there is no duplication of processing in the separate pattern-action files, or at least no duplication of the same processing in the separate pattern-action files. Segmenting the processing this way will help to conserve your system resources and make your processing as efficient as possible. It may be the case that you do not need to worry about system resources at the moment, but the volume of data relayed by the IDD is continuously increasing, so a little tweaking now will have significant benefits in the future. Also, since I am not able to logon as 'ldm', I was not able to review how you have data scouring setup. Data scouring is one of the big areas where one can significantly save on system resources by doing some planning. Last item: The reason I asked you to run 'sh ./mcinet2007.sh install mcadde' as 'root' was to re-enable access to cacimbo via ADDE. Since this is now working nicely, I am wondering if you are amenable to cacimbo once again acting as one of the cooperating a community servers? Cheers, Tom **************************************************************************** Unidata User Support UCAR Unidata Program (303) 497-8642 P.O. Box 3000 address@hidden Boulder, CO 80307 ---------------------------------------------------------------------------- Unidata HomePage http://www.unidata.ucar.edu **************************************************************************** Ticket Details =================== Ticket ID: OML-578835 Department: Support McIDAS Priority: Normal Status: Closed
NOTE: All email exchanges with Unidata User Support are recorded in the Unidata inquiry tracking system and then made publicly available through the web. If you do not want to have your interactions made available in this way, you must let us know in each email you send to us.