[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[IDD #TTR-495389]: Slow access to local datasets through ADDE
- To: zehel@xxxxxxx
- Subject: [IDD #TTR-495389]: Slow access to local datasets through ADDE
- From: "Unidata McIDAS Support" <support-mcidas@xxxxxxxxxxxxxxxx>
- Date: Thu, 07 Feb 2008 16:55:54 -0700
- Delivered-to: support-mcidas@unidata.ucar.edu by laraine.unidata.ucar.edu (Postfix) with ESMTP id 95622CB199; Thu, 7 Feb 2008 16:55:54 -0700 (MST) id 6BCC0D50BC; Thu, 7 Feb 2008 16:55:54 -0700 (MST)
Hi Samuel,
re:
> Do I still need to put the 'CURDAY' option into the dsserve.k command string?
> (e.g. dsserve.k ADD RTIMAGES/ANTARCTIC AREA TYPE=IMAGE DIRFILE=/data/ldm/mcidas/images/sat/ANTARCTIC/4km/IR/\\CURDAY/IR_* \"Antarctic IR Composite)
The \CURDAY "replaceable" gets replaced by the CCYYMMDD value for the current day (UTC) in the
DIRFILE= regular expression that defines a datasets content. This replacement occurs on the
server side when the ADDE request is received. If you want to only access data from the current day, then
you would use \CURDAY. If you want to access data from all days, you would use '*'. Remember,
the discussion of \CURDAY arose from my comment that you would likely want to serve data
from the current day for the very numerous GINIEAST/GE1KVIS images. This way, instead of having
to read and sort 'n*m' (number of days of data kept for a type of image TIMES number of images per day)
files, the ADDE server would only have to read 'n' files. This becomes very significant when one tries
to keep a number of days of GINIEAST/GINIWEST Visible images online since there are 4 of each of
these each hour.
> If so, will it work this time?
It should, but... I just tried creating an ADDE dataset for the Antarctic IR images using
the \CURDAY construct, and it didn't work. This means that the mod I made to support \CURDAY
is not being used by the ADDE server for images in AREA format. Rats! I apologize for this
oversight on my part!!
The \CURDAY construct does work for datasets of images of type GINI (images in the GINIEAST,
GINIWEST, GINICOMP, and NEXRCOMP datasets) and images of type NEXR (images in the RTNEXRAD
dataset), so I suggest that you use it in those first. I will work on the mods needed to add
\CURDAY support to datasets of type AREA and make an addendum.
Here is an example I just ran on your machine that uses three different "replaceables" to define
a dataset for NEXRAD Level III data:
<as 'mcidas' on your machine>
cd $MCDATA
dsserve.k ADD CURNIDS/N0R NEXR TYPE=IMAGE DIRFILE=/data/ldm/mcidas/images/NIDS/\\ID/\\TYPE/\\TYPE_\\CURDAY_\* \"Test of CURDAY for NIDS
The "replaceables" used are:
\ID -> NEXRAD Level III station ID (e.g., FTG, DIX, PBZ, etc.)
\TYPE -> NEXRAD Level III product type (e.g., N0R, N0S, etc.)
\CURDAY -> current data represented as CCYYMMDD
After making the CRUNIDS/N0R dataset definition on your machine, I am able to see the just the
current day's NEXRAD Level III base reflectivity (N0R) images from the station FTG:
-bash-2.05b$ imglist.k CURNIDS/N0R.ALL ID=FTG
Image file directory listing for:CURNIDS/N0R
Pos Satellite/ Date Time Center Band(s)
sensor Lat Lon
--- ------------- ------------ -------- ---- ---- ------------
1 RADAR 7 FEB 08038 00:06:00 FTG 1
2 RADAR 7 FEB 08038 00:16:00 FTG 1
3 RADAR 7 FEB 08038 00:25:00 FTG 1
4 RADAR 7 FEB 08038 00:35:00 FTG 1
5 RADAR 7 FEB 08038 00:45:00 FTG 1
6 RADAR 7 FEB 08038 00:55:00 FTG 1
7 RADAR 7 FEB 08038 01:04:00 FTG 1
8 RADAR 7 FEB 08038 01:14:00 FTG 1
9 RADAR 7 FEB 08038 01:24:00 FTG 1
10 RADAR 7 FEB 08038 01:33:00 FTG 1
11 RADAR 7 FEB 08038 01:43:00 FTG 1
12 RADAR 7 FEB 08038 01:53:00 FTG 1
13 RADAR 7 FEB 08038 02:22:00 FTG 1
14 RADAR 7 FEB 08038 03:30:00 FTG 1
15 RADAR 7 FEB 08038 04:39:00 FTG 1
16 RADAR 7 FEB 08038 06:37:00 FTG 1
17 RADAR 7 FEB 08038 07:25:00 FTG 1
18 RADAR 7 FEB 08038 07:35:00 FTG 1
19 RADAR 7 FEB 08038 10:50:00 FTG 1
20 RADAR 7 FEB 08038 11:48:00 FTG 1
21 RADAR 7 FEB 08038 11:58:00 FTG 1
22 RADAR 7 FEB 08038 12:08:00 FTG 1
23 RADAR 7 FEB 08038 12:18:00 FTG 1
24 RADAR 7 FEB 08038 12:27:00 FTG 1
25 RADAR 7 FEB 08038 12:37:00 FTG 1
26 RADAR 7 FEB 08038 12:47:00 FTG 1
27 RADAR 7 FEB 08038 12:57:00 FTG 1
28 RADAR 7 FEB 08038 13:06:00 FTG 1
29 RADAR 7 FEB 08038 13:16:00 FTG 1
30 RADAR 7 FEB 08038 13:26:00 FTG 1
31 RADAR 7 FEB 08038 13:36:00 FTG 1
32 RADAR 7 FEB 08038 15:34:00 FTG 1
33 RADAR 7 FEB 08038 17:50:00 FTG 1
34 RADAR 7 FEB 08038 18:00:00 FTG 1
35 RADAR 7 FEB 08038 18:10:00 FTG 1
36 RADAR 7 FEB 08038 18:19:00 FTG 1
37 RADAR 7 FEB 08038 18:29:00 FTG 1
38 RADAR 7 FEB 08038 18:39:00 FTG 1
39 RADAR 7 FEB 08038 18:49:00 FTG 1
40 RADAR 7 FEB 08038 18:58:00 FTG 1
41 RADAR 7 FEB 08038 19:08:00 FTG 1
42 RADAR 7 FEB 08038 19:18:00 FTG 1
43 RADAR 7 FEB 08038 19:28:00 FTG 1
44 RADAR 7 FEB 08038 19:37:00 FTG 1
45 RADAR 7 FEB 08038 19:47:00 FTG 1
46 RADAR 7 FEB 08038 19:57:00 FTG 1
47 RADAR 7 FEB 08038 20:07:00 FTG 1
48 RADAR 7 FEB 08038 20:16:00 FTG 1
49 RADAR 7 FEB 08038 20:26:00 FTG 1
50 RADAR 7 FEB 08038 20:36:00 FTG 1
51 RADAR 7 FEB 08038 20:46:00 FTG 1
imglist.k: done
To demonstrate that it really is accessing a subset of the files, note the number of
days of images in the directory containing FTG N0R images:
-bash-2.05b$ ls /data/ldm/mcidas/images/NIDS/FTG/N0R
N0R_20080120_1640 N0R_20080125_0754 N0R_20080130_0121 N0R_20080204_0002
N0R_20080120_1650 N0R_20080125_0931 N0R_20080130_0131 N0R_20080204_0012
N0R_20080120_1709 N0R_20080125_1021 N0R_20080130_0141 N0R_20080204_0022
N0R_20080120_1719 N0R_20080125_1041 N0R_20080130_0150 N0R_20080204_0031
N0R_20080120_1729 N0R_20080125_1050 N0R_20080130_0200 N0R_20080204_0041
N0R_20080120_1738 N0R_20080125_1208 N0R_20080130_0210 N0R_20080204_0051
...
N0R_20080125_0350 N0R_20080130_0032 N0R_20080203_2254 N0R_20080207_2036
N0R_20080125_0410 N0R_20080130_0042 N0R_20080203_2304 N0R_20080207_2046
N0R_20080125_0518 N0R_20080130_0052 N0R_20080203_2333
N0R_20080125_0537 N0R_20080130_0102 N0R_20080203_2343
N0R_20080125_0547 N0R_20080130_0111 N0R_20080203_2352
\CURDAY, like all "replaceables" can be used anyplace in the DIRFILE= regular expression
where part of the fully qualified pathname is the date in CCYYMMDD format. This means
that you really don't have to organize images into daily directories... but experience
has shown that it is useful to do so.
Again, I apologize for the \CURDAY construct not working (as I had thought that it
did) for images of type AREA (e.g., UNIWISC images; the ones organized into the
RTIMAGES and CIMSS datasets). I will find and fix the problem and make an addendum.
> Will scourBYnumber be able to identify and remove old directores? (e.g. 20080206 should be deleted in March)
No. You will need to switch to use of scourBYday. scourBYday can be found in the same
directory as scourBYnumber on your machine: ~ldm/util. The directory specified on the
scourBYday invocation command line will be the directory down to, but not including
the various days. For instance, the invocation for scourBYday that keeps 5 days
of the Antarctic IR images would be:
scourBYday /data/ldm/mcidas/images/sat/ANTARCTIC/4km/IR 5
> P.S. I'm in the process of moving our NNEXRAD (NIDS) data to /usr/data (a different file system).
> This should free up some disk I/O.
Excellent. This should really help your I/O bottleneck as there are on the order of 17,500
NNEXRAD (NIDS) files received every hour!
> We are still in the process of ordering a computer with more RAM, and CALU would like to utilize
> bandwidth to the fullest (even if it isn't always stored)
Very good.
> P.P.S. I've noticed Shu pulls down NEXRAD2 data, but it doesn't seem to be in the ADDE set, or on file.
> Am I missing something?
No, you are not missing anything. McIDAS has no ADDE server for NEXRAD Level 2 data. The
IDV, however, can use the NEXRAD Level 2 data directly, so you may still want to ingest
the images and then make them available to machines where the IDV is used (NFS mount, samba, etc.).
Cheers,
Tom
****************************************************************************
Unidata User Support UCAR Unidata Program
(303) 497-8642 P.O. Box 3000
support@xxxxxxxxxxxxxxxx Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage http://www.unidata.ucar.edu
****************************************************************************
Ticket Details
===================
Ticket ID: TTR-495389
Department: Support McIDAS
Priority: Normal
Status: Closed
