>From: Owen Cooper <address@hidden> >Organization: Aeronomy Laboratory/NOAA >Keywords: 200306112200.h5BM0fLd024027 McIDAS TXT2MD PTCON Hi Owen, >Thank you for the advice on reading ASCII text into MCIDAS for >contouring onto GOES imagery. No worries. >I tried registering a new SCHEMA file but ran into a problem. >Please see the attached text file which should explain all. >Please let me know if you have any ideas as to what the problem may be, >and let me know if you need any more info. As I said on the phone yesterday, the missing piece is knowing what a schema definition file should look like. A description of an existing schema will probably help: -- DCNLDN -- " NAME VSN DATE ID TEXTID " ---- --- ---- -- ------ SCHEMA NLDN 2 98337 0 "NLDN DATA OBS UNIDATA FORMAT "KEY SCALE UNITS DESCRIPTION "--- ----- ----- ----------- ROWS 1000 "1000 ROWS EACH WITH 1000 COLS DAY CYD "YEAR AND JULIAN DAY TIME HMS "TIME OF FIRST OBSERVATION IN ROW CMAX "LAST COLUMN CONTAINING DATA COLUMNS 1000 "DEFAULT # OF COLS DATA "START OF DATA SECTION HMS HMS "ACTUAL TIME OF OBSERVATION LAT 4 DEG "LATITUDE (DECIMAL DEG) LON 4 DEG "LONGITUDE (DECIMAL DEG) SGNL 1 "SIGNAL STRENGTH (150 measures ~ 30 kamps) MULT "MULTIPLICITY ENDSCHEMA Each schema description: - starts with a line that begins with SCHEMA - the SCHEMA line defines the 4-character name of the schema; the version of the schema (schema definitions can evolve); the date the version was created; an ID for the schema (not quite sure what this is!); and a textual description as a quote field - definitions of #ROWS, #COLUMNS, and DATA keys (e.g. DAY, TIME, ..., MULT in the listing above). MD file keys are each 4 characters or less in length - specification of how many significant digits of a key value is to be kept in the MD file (all MD file values are scaled integers) - units of the keys (recognized units are defined in the file FLDWDTH) - comments about the keys (these comments are not used for anything, but are useful for the reader) Your first attempt of defining a schema was an attempt to replicate the output format that you would see when listing a schema that is registered (i.e., has been added to SCHEMA using SCHE) using LSCHE. This was a good guess, but will not work: ############################################################################ NAME: FLEXP VERSION: 1 DATE: 2003167 TEXTID: "FLEXPART CO CONCENTRATIONS ---- ------- DEFAULT NUMBER OF ROWS: 33000 INTEGER ID: 0 COLS: 3 MISSING DATA VALUE: -999 REPEAT GROUP: NUMBER OF REPETITIONS: 0 STARTING POSITION: 1 SIZE: 3 NUMBER OF KEYS IN ROW HEADER: 3 COL HEADER: 0 STARTING AT POSITION 1 DATA RECORD: 3 STARTING AT POSITION 1 ---- 6 TOTAL KEY SCALE UNIT ---- ----- ---- LON 6 DEG LAT 5 DEG CO 7 CON ################################################################################ >I'm not sure what the following parameters are for: > >NUMBER OF REPETITIONS (seeing as I only want to read each column once, I >assume the # of repititions is 0) The number of repetitions is used for schema like for soundings where the same set of data are available for multiple levels. >STARTING POSITION ?? >SIZE ??? >INTEGER ID ???? >COL HEADER: STARTING AT POSITION ??? >DATA RECORD: STARTING AT POSITION ??? >Can you take a look at this SCHEMA and let me know where I'm going wrong? These are artifacts of the LSCHE output. The schema example above (DCNLDN) should go a long way towards explaining what needs to be done. The schema I came up with below should galvanize some of the concepts. I played around with the data file you gave me access to yesterday, and came up with a schema that, while pretty simple minded, works for what you want: " NAME VSN DATE ID TEXTID " ---- --- ---- -- ------ SCHEMA FLXP 1 01168 0 "FLEXPART CO CONCENTRATIONS "KEY SCALE UNITS DESCRIPTION "--- ----- ----- ----------- ROWS 90 "90 ROWS LAT 4 DEG "LATITUDE (DECIMAL DEG) COLUMNS 360 "DEFAULT # OF COLS LON 4 DEG "LONGITUDE (DECIMAL DEG) DATA "START OF DATA SECTION CO 2 CON "COD CONCENTRATION ENDSCHEMA This schema defines an MD file that is 90 rows long; each row has 360 columns; and each column has one data point, the CO concentration. Notice that I chose the LAT as a ROW header; the LON as a COLumn header, and CO as the data key. The reason for this was mostly so that the TXT2MD conversion utility would work without complaint. What you end up with is a MD file where each ROW represents all data at a single Latitude, each COLumn represents all data at a single a single Longitude, and there is exactly one datum for each ROW,COL cell. I then used this schema to create an output MD file and contour its values on top of a world map: - create DCFLXP with the contents above in the ~mcidas/workdata directory. If you are doing this as a user other than 'mcidas', then you would create this file in your ~/mcidas/data directory - since I didn't want to change the SCHEMA file that I send out in the distribution, I made a copy of it into my McIDAS working directory: cd ~mcidas/workdata cp ~mcidas/data/SCHEMA . redirect.k ADD SCHEMA \". If you are doing this as a user other than 'mcidas', do the same steps in your McIDAS working directory: cd ~/mcidas/data cp ~mcidas/data/SCHEMA . redirect.k ADD SCHEMA \". - next, I registered the schema in the copy of SCHEMA that I can modify: sche.k DCFLXP I verified the schema registry using LSCHE: lsche.k FLXP NAME: FLXP VERSION: 1 DATE: 2001168 TEXTID: "FLEXPART CO CONCENTRATIONS ---- ------- DEFAULT NUMBER OF ROWS: 90 INTEGER ID: 0 COLS: 360 MISSING DATA VALUE: -2139062144 REPEAT GROUP: NUMBER OF REPETITIONS: 1 STARTING POSITION: 3 SIZE: 1 NUMBER OF KEYS IN ROW HEADER: 1 COL HEADER: 1 STARTING AT POSITION 2 DATA RECORD: 1 STARTING AT POSITION 3 ---- 3 TOTAL KEY SCALE UNIT KEY SCALE UNIT KEY SCALE UNIT ---- ----- ---- ---- ----- ---- ---- ----- ---- LAT 4 DEG LON 4 DEG CO 2 CON LSCHE: DONE - the file you gave me FTP access to looked like one long line in Unix: there were carriage returns at the end of each logical line, not new lines. TXT2MD expects the textual data file to be in ASCII format with each line ending in a new line. Given this, I translated your file into a format I could use: tr '\015' '\012' < flexpart_20030604030000 > flexprt.txt The 'tr' invocation converts CR characters (octal 015) into NL characters (octal 012). I wrote the output into a file whose name follows the old DOS 8.3 name format since that is what TXT2MD demands (this should be changed, but...). - after converting your file, I edited it to add one additional line: change: LON LAT CO -178.5 0.5 1.63 ... to: LON LAT CO UNIT=*-1 X X -178.5 0.5 1.63 ... The reason for this addition is that I _assumed_ that your LON values were using the West negative convention, while McIDAS uses a West positive convention. The *-1 syntax for the first positional value for UNIT tells TXT2MD to multiply each value by -1 thus converting its sign. BTW the ability to convert units in the UNIT= line was an addition I made to TXT2MD a number of years ago; it does not exist in the SSEC version (just in case you find yourself using an SSEC distribution of McIDAS and wonder why you can't do unit conversions). - finally, I was able to run TXT2MD and convert your file: <I was still in ~mcidas/workdata, you may well be in ~/mcidas/data> mdu.k DEL 1000 <- just to make sure that the output MD file doesn't exist yet txt2md.k flexprt.txt 1000 FLXP All data values were read in from your text file and stored in MDXX1000. The next step was to verify that I could contour the data on top of a navigated display. In order to do this, I first made sure that I had the MYDATA datasets defined: batch.k MYDATA.BAT Then I contoured the data and saved the output into a GIF (tm) file for you to take a look at: mcenv erase.k map.k WORLD ptcon.k MYDATA/PTSRCS.1000 PAR=CO CINT=15 NAV=C LSIZE=5 frmsave.k 1 flexprt.gif exit Have a look at the result in: http://www.unidata.ucar.edu/staff/tom/gifs/owen/flexprt.gif I guess it is now an exercise for the reader (you ;-) to verify that the display of the data is correct. >Thanks! Please let me know if there is anything that I did that does not make sense. >Thanks! No worries. Tom >From address@hidden Wed Jun 18 12:14:43 2003 Hi Tom, This looks awesome, and I will give it a try very soon. Next week I will post near real-time GOES-east and Meteosat IR images on the web and will overlay the carbon monoxide data from the FLEXPART model. As far as I know this will be the first realtime display of pollution plumes travelling across the Atlatntic and superimposed onto geostationary satellite data. Once everything is working I'll send you the URL so you can see the animations. Thanks! Owen Owen R. Cooper Cooperative Institute for Research in Environmental Sciences (CIRES) University of Colorado/NOAA Aeronomy Laboratory NOAA Aeronomy Laboratory, R/AL4 325 Broadway Boulder CO 80305 ph: (303) 497-3599 fax: (303) 497-5686
NOTE: All email exchanges with Unidata User Support are recorded in the Unidata inquiry tracking system and then made publicly available through the web. If you do not want to have your interactions made available in this way, you must let us know in each email you send to us.