[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[McIDAS #HZP-657281]: Moving McIDAS-XCD server



Hi Hsie,

re: multiple grids that look the same in listing is confusing

> You misunstandd what I mean.  If I want the data from GFS analysis at
> 09/07/2007 00Z for 500 MB U wind.  I get 8 hits from the database.  What
> are the difference between those 8 data?

Sorry for misunderstanding what you meant.  You can see that the grids
are different (or the same and received multiple times) by using an
expanded form of the listing.

For instance:

GRDLIST RTGRIBS/GFS-ALL PAR=U FHO=0 LEV=500 GPR=PS NUM=10
Dataset position 1      Directory Title= /GFS.81.2007250.0.0.201.grib
PAR  LEVEL      DAY          TIME     SRC  FHR  FDAY         FTIME    GRID  PRO
---- ---------- ------------ -------- ---- ---- ------------ -------- ----- ----
U      500 MB   07 SEP 07250 00:00:00  GFS    0 07 SEP 07250 00:00:00   N/A PS
U      500 MB   07 SEP 07250 00:00:00  GFS    0 07 SEP 07250 00:00:00   N/A PS
U      500 MB   07 SEP 07250 00:00:00  GFS    0 07 SEP 07250 00:00:00   N/A PS
U      500 MB   07 SEP 07250 00:00:00  GFS    0 07 SEP 07250 00:00:00   N/A PS
U      500 MB   07 SEP 07250 00:00:00  GFS    0 07 SEP 07250 00:00:00   N/A PS
U      500 MB   07 SEP 07250 00:00:00  GFS    0 07 SEP 07250 00:00:00   N/A PS
U      500 MB   07 SEP 07250 00:00:00  GFS    0 07 SEP 07250 00:00:00   N/A PS
U      500 MB   07 SEP 07250 00:00:00  GFS    0 07 SEP 07250 00:00:00   N/A PS
U      500 MB   07 SEP 07250 06:00:00  GFS    0 07 SEP 07250 06:00:00   N/A PS
U      500 MB   07 SEP 07250 06:00:00  GFS    0 07 SEP 07250 06:00:00   N/A PS
Number of grids listed = 10
GRDLIST - done

The first 8 grids in this list look the same.  An expanded listing, however,
shows the difference:

GRDLIST RTGRIBS/GFS-ALL PAR=U FHO=0 LEV=500 GPR=PS NUM=10 FORM=ALL
Dataset position 1      Directory Title= /GFS.81.2007250.0.0.201.grib
PAR  LEVEL      DAY          TIME     SRC  FHR  FDAY         FTIME    GRID  PRO
---- ---------- ------------ -------- ---- ---- ------------ -------- ----- ----
U      500 MB   07 SEP 07250 00:00:00  GFS    0 07 SEP 07250 00:00:00   N/A PS
Total pts=   4225 Num rows=  65  Num columns=  65   received:  2007249 212319Z
u-component of wind
GRIB ID numbers: Geographic =  201; PAR = 33; Model ID = 81; Level type =100
Units of gridded variable are MPS  Scale of variable is:   2
Polar Stereographic Projection
Row num of pole=   33.00  Col num of pole=   33.00  Col spacing (m)=381000.0
Standard Latitudes=   60.00     60.00   Standard Longitude=  105.00
PAR  LEVEL      DAY          TIME     SRC  FHR  FDAY         FTIME    GRID  PRO
---- ---------- ------------ -------- ---- ---- ------------ -------- ----- ----
U      500 MB   07 SEP 07250 00:00:00  GFS    0 07 SEP 07250 00:00:00   N/A PS
Total pts=   2795 Num rows=  43  Num columns=  65   received:  2007249 212319Z
u-component of wind
GRIB ID numbers: Geographic =  202; PAR = 33; Model ID = 81; Level type =100
Units of gridded variable are MPS  Scale of variable is:   2
Polar Stereographic Projection
Row num of pole=   -2.00  Col num of pole=   33.00  Col spacing (m)=190500.0
Standard Latitudes=   60.00     60.00   Standard Longitude=  105.00
PAR  LEVEL      DAY          TIME     SRC  FHR  FDAY         FTIME    GRID  PRO
---- ---------- ------------ -------- ---- ---- ------------ -------- ----- ----
U      500 MB   07 SEP 07250 00:00:00  GFS    0 07 SEP 07250 00:00:00   N/A PS
Total pts=   1755 Num rows=  39  Num columns=  45   received:  2007249 212320Z
u-component of wind
GRIB ID numbers: Geographic =  203; PAR = 33; Model ID = 81; Level type =100
Units of gridded variable are MPS  Scale of variable is:   2
Polar Stereographic Projection
Row num of pole=    3.00  Col num of pole=   27.00  Col spacing (m)=190500.0
Standard Latitudes=   60.00     60.00   Standard Longitude=  150.00
PAR  LEVEL      DAY          TIME     SRC  FHR  FDAY         FTIME    GRID  PRO
---- ---------- ------------ -------- ---- ---- ------------ -------- ----- ----
U      500 MB   07 SEP 07250 00:00:00  GFS    0 07 SEP 07250 00:00:00   N/A PS
Total pts=  10965 Num rows=  85  Num columns= 129   received:  2007249 212340Z
u-component of wind
GRIB ID numbers: Geographic =  213; PAR = 33; Model ID = 81; Level type =100
Units of gridded variable are MPS  Scale of variable is:   2
Polar Stereographic Projection
Row num of pole=   -3.00  Col num of pole=   65.00  Col spacing (m)= 95250.0
Standard Latitudes=   60.00     60.00   Standard Longitude=  105.00
PAR  LEVEL      DAY          TIME     SRC  FHR  FDAY         FTIME    GRID  PRO
---- ---------- ------------ -------- ---- ---- ------------ -------- ----- ----
U      500 MB   07 SEP 07250 00:00:00  GFS    0 07 SEP 07250 00:00:00   N/A PS
Total pts=   2795 Num rows=  43  Num columns=  65   received:  2007250 003703Z
u-component of wind
GRIB ID numbers: Geographic =  202; PAR = 33; Model ID = 82; Level type =100
Units of gridded variable are MPS  Scale of variable is:   2
Polar Stereographic Projection
Row num of pole=   -2.00  Col num of pole=   33.00  Col spacing (m)=190500.0
Standard Latitudes=   60.00     60.00   Standard Longitude=  105.00
PAR  LEVEL      DAY          TIME     SRC  FHR  FDAY         FTIME    GRID  PRO
---- ---------- ------------ -------- ---- ---- ------------ -------- ----- ----
U      500 MB   07 SEP 07250 00:00:00  GFS    0 07 SEP 07250 00:00:00   N/A PS
Total pts=   1755 Num rows=  39  Num columns=  45   received:  2007250 003703Z
u-component of wind
GRIB ID numbers: Geographic =  205; PAR = 33; Model ID = 82; Level type =100
Units of gridded variable are MPS  Scale of variable is:   2
Polar Stereographic Projection
Row num of pole=  -17.00  Col num of pole=   27.00  Col spacing (m)=190500.0
Standard Latitudes=   60.00     60.00   Standard Longitude=   60.00
PAR  LEVEL      DAY          TIME     SRC  FHR  FDAY         FTIME    GRID  PRO
---- ---------- ------------ -------- ---- ---- ------------ -------- ----- ----
U      500 MB   07 SEP 07250 00:00:00  GFS    0 07 SEP 07250 00:00:00   N/A PS
Total pts=   1755 Num rows=  39  Num columns=  45   received:  2007250 003703Z
u-component of wind
GRIB ID numbers: Geographic =  203; PAR = 33; Model ID = 82; Level type =100
Units of gridded variable are MPS  Scale of variable is:   2
Polar Stereographic Projection
Row num of pole=    3.00  Col num of pole=   27.00  Col spacing (m)=190500.0
Standard Latitudes=   60.00     60.00   Standard Longitude=  150.00
PAR  LEVEL      DAY          TIME     SRC  FHR  FDAY         FTIME    GRID  PRO
---- ---------- ------------ -------- ---- ---- ------------ -------- ----- ----
U      500 MB   07 SEP 07250 00:00:00  GFS    0 07 SEP 07250 00:00:00   N/A PS
Total pts=   4225 Num rows=  65  Num columns=  65   received:  2007250 003703Z
u-component of wind
GRIB ID numbers: Geographic =  201; PAR = 33; Model ID = 82; Level type =100
Units of gridded variable are MPS  Scale of variable is:   2
Polar Stereographic Projection
Row num of pole=   33.00  Col num of pole=   33.00  Col spacing (m)=381000.0
Standard Latitudes=   60.00     60.00   Standard Longitude=  105.00
PAR  LEVEL      DAY          TIME     SRC  FHR  FDAY         FTIME    GRID  PRO
---- ---------- ------------ -------- ---- ---- ------------ -------- ----- ----
U      500 MB   07 SEP 07250 06:00:00  GFS    0 07 SEP 07250 06:00:00   N/A PS
Total pts=   4225 Num rows=  65  Num columns=  65   received:  2007250 032232Z
u-component of wind
GRIB ID numbers: Geographic =  201; PAR = 33; Model ID = 81; Level type =100
Units of gridded variable are MPS  Scale of variable is:   2
Polar Stereographic Projection
Row num of pole=   33.00  Col num of pole=   33.00  Col spacing (m)=381000.0
Standard Latitudes=   60.00     60.00   Standard Longitude=  105.00
PAR  LEVEL      DAY          TIME     SRC  FHR  FDAY         FTIME    GRID  PRO
---- ---------- ------------ -------- ---- ---- ------------ -------- ----- ----
U      500 MB   07 SEP 07250 06:00:00  GFS    0 07 SEP 07250 06:00:00   N/A PS
Total pts=   2795 Num rows=  43  Num columns=  65   received:  2007250 032242Z
u-component of wind
GRIB ID numbers: Geographic =  202; PAR = 33; Model ID = 81; Level type =100
Units of gridded variable are MPS  Scale of variable is:   2
Polar Stereographic Projection
Row num of pole=   -2.00  Col num of pole=   33.00  Col spacing (m)=190500.0
Standard Latitudes=   60.00     60.00   Standard Longitude=  105.00
Number of grids listed = 10
GRDLIST - done

For instance, compare the expanded listing (FORM=ALL) for the first two grids 
in this
list:

Dataset position 1      Directory Title= /GFS.81.2007250.0.0.201.grib
PAR  LEVEL      DAY          TIME     SRC  FHR  FDAY         FTIME    GRID  PRO
---- ---------- ------------ -------- ---- ---- ------------ -------- ----- ----
U      500 MB   07 SEP 07250 00:00:00  GFS    0 07 SEP 07250 00:00:00   N/A PS
Total pts=   4225 Num rows=  65  Num columns=  65   received:  2007249 212319Z
u-component of wind
GRIB ID numbers: Geographic =  201; PAR = 33; Model ID = 81; Level type =100
Units of gridded variable are MPS  Scale of variable is:   2
Polar Stereographic Projection
Row num of pole=   33.00  Col num of pole=   33.00  Col spacing (m)=381000.0
Standard Latitudes=   60.00     60.00   Standard Longitude=  105.00
PAR  LEVEL      DAY          TIME     SRC  FHR  FDAY         FTIME    GRID  PRO
---- ---------- ------------ -------- ---- ---- ------------ -------- ----- ----
U      500 MB   07 SEP 07250 00:00:00  GFS    0 07 SEP 07250 00:00:00   N/A PS
Total pts=   2795 Num rows=  43  Num columns=  65   received:  2007249 212319Z
u-component of wind
GRIB ID numbers: Geographic =  202; PAR = 33; Model ID = 81; Level type =100
Units of gridded variable are MPS  Scale of variable is:   2
Polar Stereographic Projection
Row num of pole=   -2.00  Col num of pole=   33.00  Col spacing (m)=190500.0
Standard Latitudes=   60.00     60.00   Standard Longitude=  105.00

The first grid has Geographic ID = 201; the second has 202.  The grib messages
that came in were different and they were correctly entered into the database
as different grids.  I think you will find the same thing if you look carefully
at expanded listings.

By the way, your comments point out something that NOAAPORT is famous for:
sending the same field in multiple projections for more or less the same
area.

re:
> It seems to me that xcdscour only scour the data, NOT the database. I
> just run it manually. It deleted the data files in
> /home/mcidas/savedata/grib. But when I issue the grdlist.k command. It
> tell me that the data is not changed:

'xcdscour' is supposed to scour the database in addition to removing the
files from disk.  Here is its code:

 ...

GRIB|grib)
        i=$max_days_to_delete

        # MySQL
        # Do this before deleting files so client doesnt get false positives
        LD_LIBRARY_PATH=/usr/local/lib:$LD_LIBRARY_PATH
        export LD_LIBRARY_PATH
        daynum=`days_ago_year $days`
        MBIN=`which mysql |grep -v no`
        if [ -n "$MBIN" ]; then
                strSQL="DELETE FROM file_list WHERE j_day <= \"$daynum\";"
                echo "$strSQL" |$MBIN --user=gribwrite --password=gribwrite 
mcrtgrib >/dev/null 2>&1
                strSQL="OPTIMIZE table file_list;"
                echo "$strSQL" |$MBIN --user=gribwrite --password=gribwrite 
mcrtgrib >/dev/null 2>&1
        fi

        while [[ $i -ge $days ]]; do
                daynum=`days_ago_year $i`
                hournum=0
                while [[ $hournum -le 23 ]]; do
                        rm -f *.*.$daynum.$hournum??.*.*.grib
                        rm -f *.*.$daynum.$hournum??.*.grb2
                        hournum=$((hournum+1))
                done
                rm -f *.*.$daynum.*.*.*.grib
                rm -f *.*.$daynum.*.*.grb2
                i=$((i-1))
        done
        ;;

 ...

If it is not actually scouring the MySQL database, it means that the 'mysql'
executable that is being used by virtue of the PATH defined in the copy of
'xcdscour' that is being run (~ldm/util/xcdscour) does not work with the
64-bit database being used.  This means that we need to adjust the path
setting:

PATH=/home/oper/mcidas/bin:/home/mcidas/bin:/opt/SUNWspro/bin:/usr/ccs/bin:/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/openssh/bin:/usr/ccs/bin:/opt/openssh/bin:/opt/openssh/sbin:/usr/local/bin:/usr/ucb:/usr/dt/bin:/usr/local/mysql/bin
export PATH

so that the proper 'mysql' is found and used.

re:
> How do you define 1 day here?

The numerical value passed into 'xcdscour' should be the number of days to keep 
online:

1 -> 1 day
2 -> 2 days
etc.

I run this all of the time on several machines that I use for testing, and I 
have never had
a problem with it.

> When I run it manually, it almost wipe everything clean.

Strange.  I will need to take a look at the 'xcdscour' being used on your 
machine to see what
is going on.

Cheers,

Tom
****************************************************************************
Unidata User Support                                    UCAR Unidata Program
(303) 497-8642                                                 P.O. Box 3000
address@hidden                                   Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage                       http://www.unidata.ucar.edu
****************************************************************************


Ticket Details
===================
Ticket ID: HZP-657281
Department: Support McIDAS
Priority: Normal
Status: Closed