>From: Pepo Juega <address@hidden> >Organization: Instituto Nacional de Meteorologia >Keywords: 200006220829.e5M8TZT11974 LDM ldm-mcidas Pepo, >Understood the FILE action. OK. I thought you did, but I felt obligated to include comments regarding the '-d', '-l', and '-v' flags. All email transactions to/from support get put into our inquiry tracking system. The more completely a question gets answered, the easier it is for the person searching the tracking system to apply the information to his/her situation. If at any time anyone does not want his/her comments/questions put into the tracking system, all they have to do is let us know and we will exclude them. >Changed my pqact.conf at alfa: >---------------------------------------------------- ># MCIDAS products: ># ># GRID files: ># >MCIDAS ^.*GRID.* > FILE -close data/GRIB/%m%d%H%M.grid > ># AREA files: ># >MCIDAS ^.*AREA.* > FILE -close data/AREA/%m%d%H%M.area >---------------------------------------------------- >ldmadmin pqactcheck >...ldm/etc/pqact.conf is syntactically correct Looks good. >I fed manually from meteosat: >pqinsert -vx -f MCIDAS -q $HOME/data/ldm.pq AREA7028 >Mapping 25595904 >pqinsert: 3e6a273cac4f483c2f757d1663509031 263424 20000629135738.024 >MCIDAS 000 AREA7028 > >...and... I got 06291357.area on alfa !!!!! Success! The really nice thing is the fact that there can be many, many machines to which the product will be sent and delivery is guaranteed up until the point that the product gets scoured out of the sending machine's queue. >Also tried out the zip/unzip business with good results Very good. >So, YOU got things moving al last. Now, as long as you suggested, >I will be very interested in the rest i.e.: an 'intelligent' >file namimg schema, and PNG. Good. re: PNG compression/uncompression routines >Please tell me all about it. I developed the PNG compression and uncompression routines for McIDAS AREA files for our use in our Internet Data Delivery (IDD: http://www.unidata.ucar.edu/projects/idd/index.html) system. I chose the PNG technique after reasonably extensive comparisons of it against a number of other compression methods. It was the overall winner even though I chose to not compress the file headers and comment cards. A "graphical" comparison of an AREA file and PNG compressed AREA file is shown here: AREA file PNG compressed AREA file +--------------------+ +----------------------+ | area directory | | area directory | +--------------------+ +----------------------+ +--------------------+ +----------------------+ | navigation codicil | | navigation codicil | +--------------------+ +----------------------+ +--------------------+ +----------------------+ | calibration codicil| | calibration codicil | +--------------------+ +----------------------+ +--------------------+ +----------------------+ | aux block | | aux block | +--------------------+ +----------------------+ +--------------------+ +----------------------+ | | | AREA file comment | | | | cards | | image lines | +----------------------+ | | +----------------------+ | | | | +--------------------+ | PNG compressed | +--------------------+ | image | | AREA file comment | | | | cards | | | +--------------------+ +----------------------+ The PNG compression is only done on the image line portion of the AREA file. The comment cards, if they exist, are moved to just after all of the other directories/codicils in the PNG compressed file. The compression and uncompression routines can extract information from the AREA file header and use it as indicies into the McIDAS SATANNOT and SATBAND files to extract descriptive information that can be used for file naming. This turns out to be very important/useful for Unidata GEMPAK users and will be very useful for McIDAS users once they upgrade to Version 7.70 (I am working on my release right now). Our ldm-mcidas package source contains PNG interface routines that make it very easy to read/write the PNG compressed portion of the image. >I don't know too much about McIDAS 7.7 >yet, as we haven't got the release. One of the nice things about 7.7 is the ability to get away from the arcane McIDAS file naming conventions. For instance, suppose that you have a set of infrared images in AREA file format from METEOSAT. The pre-7.7 way of creating a dataset was to: o configure your McIDAS-X session to know how to find the AREA files that the images are stored in. As you well know, this is done by use of either or both of REDIRECT and MCPATH. REDIRECT allowed one to locate AREA files on an image-by-image basis. MCPATH was typically used to locate groups of files in a single directory (this is not strictly true, but it is effectively true). o define a dataset something like: DSSERVE ADD METEOSAT/IR AREA 1525 1600 "METEOSAT IR images It has always been my goal to not have to name the images (or MD or GRID) files AREAnnnn. A more descriptive name might be something like: METEOSAT_IR_20000630.1530 7.7 now allows this. Assuming that you had a set of images in AREA file format named like this example and located in, say, the /data/mcidas directory, the DSSERVE command to setup the dataset would be: DSSERVE ADD METEOSAT/IR AREA DIRFILE=/data/mcidas/METEOSAT_IR* REDIRECT and MCPATH do not come into play at all with this scheme, AND you can have more than 9999 files! >But you can imagine that what >I intend is to transfer McIDAS images to a bunch (=~60) of hungry >users. We do not want all of them to make ADDE requests to meteosat >at the very same minute every half hour! Right. >So, we decided to build >regional McIDAS image servers, and now what we do is plain old ftp >sessions to feed them offline the McIDAS ports. Once the images rest >at their regional server, they can remap, reload, etc. with ease. The LDM will be very useful for this activity. This setup is analagous to our IDD, and that is moving on the order of 250 GB per day when the reception at all sites is aggregated. >Now, this will be tricky with the LDM I got now. I tried to feed >alfa from meteosat simulating arrival of a new meteosat slot, and >what I get from that is three bands (VIS+IR+WV) which means 3 >images to send over. The transfer is so fast that the name for the >file is the same, and the last overwrites the first two. If/when you start using the PNG compress/uncomress routines, you can name the output files so that they won't match for different bands. >Besides, >I have no way to differentiate bands. Of course, I could start by >giving them different names at meteosat, like VIS001 IR001... >and prepare a different pattern to be recognized, but once you >told me, I would prefer this improved PNG feature. I think that you will like this. >BTW the band >information, being also at the image header, could be used to >implement the file naming strategy, right? Absolutely correct. This is exactly what we are doing right now to send image products created by the CIMSS group in SSEC to our sites. As a quick example of using the PNG compression routine, I offer the following simple Korn shell script that we are using to create PNG compressed CIMSS products that are then sent through our IDD: #!/bin/ksh cd $HOME/mcidas/png area=$1 typeset -Z4 area dialprod=$2 file=`area2png -n -f $HOME/mcidas/data/AREA${area} -a /home3/mcidas/data/SATANNOT -b /home3/mcidas/data/SATBAND "pnga2area Q0 ${dialprod} $area \\SAT \\BAND \\RES %Y%m%d %H%M"` pqinsert -f MCIDAS "${file}" date=`date -u ` echo "${file}" " [ sent $date ] " >> png_file.log /bin/rm -f "${file}" An example file name generated by the file=`area2png ...` line in the script is: pnga2area Q0 CB 1110 GOES-9_IMG UNKBAND 10km 19970709 1800 The relevant pieces of this are: Q0 - this is not really relevant for you. We use it to tag the image with the portion of the hour from which it was scanned: Q0 - any time of the hour Q1 - H to H+15 Q2 - H+15 to H+30 Q3 - H+30 to H+45 Q4 - H+45 to H+50 CB - an arbitrary two letter "product code". This was added for support of the McIDAS routing table, ROUTE.SYS, that some of our sites still use. You probably don't use this notion, so you should ignore it in this example \\SAT GOES-9_IMG - this comes from a lookup of the satellite sensor source number (McIDAS AREA header word 2 (zero based) in the McIDAS satellite annotation file SATANNOT \\BAND UNKBAND - the band number for this image is not one of the ones available in the McIDAS satellite band information file, SATBAND. Since there is no information, area2png expands this as UNKBAND \\RES 10km - base resolution of the satellite from the McIDAS satellite band information file, SATBAND. If the information on the resolution is not in SATBAND, this is the LINRES value in the AREA file header (header word 11 (zero based)) expressed in km %Y%m%d 19970709 - date information from the AREA file header (header word 3 (zero based)) %H%M 1800 - time information from the AREA file header (header word 4 (zero based)) A PNG file with the name: "pnga2area Q0 CB 1110 GOES-9_IMG UNKBAND 10km 19970709 1800" is created from the input AREA. Its name is then used as the header when the product is 'pqinsert'ed into the LDM product queue. The receiving machine can key on this product with pqact entries that might look like: # broadcast header format: # pnga2area Q0 Cx AREA SAT BAND RES CCYYMMDD HHMM MCIDAS ^pnga2area Q. CA (.*) (.*) (.*) (.*) (........) (....) PIPE pnga2area /var/data/mcidas/CIMSS/\2_CTP_\4_\5_\6 I have made binary distributions of the ldm-mcidas package available for a variety of operating systems: Operating System pub/binary subdirectory ----------------------+--------------------------------------------------- DEC OSF/1 4.0E osf1_4.0-alpha HP-UX 11.0 hpux_11.00-hp9000 IBM AIX 4.3 aix_4.3-rs6000 RedHat 6.[012] Linux linux_2.2-i686 SGI IRIX 6.5 irix_6.5-mips SGI IRIX64 6.5 irix64_6.5-mips Sun SOLARIS 5.7 SPARC sunos_5.6-sparc Sun SOLARIS 5.7 SPARC sunos_5.7-sparc Sun SOLARIS 5.7 x86 sunos_5.7-i86pc Sun SOLARIS 5.8 x86 sunos_5.8-i86pc In all cases, you can ftp the distribution (using binary mode!) by anonymous FTP to our ftp server, ftp.unidata.ucar.edu. The latest release is always accessible in ldm-mcidas.tar.Z; this is a link to the most recent version which is currently ldm-mcidas-7.6.3.tar.Z. If your machine(s) is(are) not using one of the above OSes, you can build the distribution from source. In this case, you should grab the file ldm-mcidas.tar.Z (a link to the current source distribution ldm-mcidas.7.6.3.tar.Z) from the pub/ldm-mcidas directory using anonymous ftp on the same ftp server, ftp.unidata.ucar.edu. In order to build the package from source, you will need to have installed: o LDM o netCDF since modules need include files from these packages for compiling and the libraries from these for linking. The good news for you is that you already have the LDM built and the netCDF is included as part of the McIDAS-X distribution! After you grab the distribution, you can investigate how the PNG compression and uncompression routines work. You can do this through the area2png.1 and pnga2area.1 man page files; by running either routine with no arguments (or with -h); or you can look at my ldm-mcidas web pages: http://www.unidata.ucar.edu/packages/mcidas/mcidd/ldm-mcidas.html If things don't make sense, please let me know. >Txs a lot Tom, I'm in debt with you. Ask for anything. Whenever Well, you probably don't really mean this, but you could: o fly me to Spain for a month :-) o send vast amounts of money to my private Swiss bank account :-) Seriously though, I would be interested in you considering giving Unidata users access to one of your ADDE servers so they could occasionally look at the METEOSAT data that you undoubtedly have in abundance. Tom
NOTE: All email exchanges with Unidata User Support are recorded in the Unidata inquiry tracking system and then made publicly available through the web. If you do not want to have your interactions made available in this way, you must let us know in each email you send to us.