>From: "Paul L. Sirvatka" <address@hidden> >Organization: COD >Keywords: 200110292153.f9TLrs128399 McIDAS-X ADDE server Paul, >In attempting to automate everything I have run into some problems. I have >a good handle on the scripts, but have noticed pretty strong >inconsistencies in the availability of images. > >ON RTGINI, when I use ADDE.UCAR.EDU I sometimes get nothing when I run >DSINFO IMAGES RTGINI. This would happen right after 0Z. >Also...do you scour the images at 00Z? If so, the 23:45 image goes away >pretty rapidly. The images are not scoured, but the set of images in the dataset is changed. We have 15 days of GINI images online on adde.ucar.edu stored in daily directories. In order to make access quick, we setup the GINICOMP, GINIEAST, GINIWEST, and RTGINI datasets so that they contain images from the "current" day only (done by changing a pointer at 0Z to the directory containing the current day's images). So, at 0Z, the number of images for the "current" day drops to zero as the pointer switches to a directory that will be filled by the data ingestion process. Now, one can ask to search through the full set of GINI images (through another dataset that is the set of all days data), but those searches use alot of machine resources, so we discourage their use. On a machine like papagayo.unl.edu, the number of images in each of the above datasets is kept small and fixed. This way, the set of images in RTGINI, GINICOMP, GINIEAST, and GINIWEST, is always the same size. Since you are trying to keep things as realtime as possible, I would strongly encourage you to point to papagayo.unl.edu since the set of images will be more like what you are looking for. >If you have any info on why there is inconsistent availability on other >sites as well I would appreciate it. How a dataset is setup is a site-specific option. Since papagayo.unl.edu gets all of its images from adde.ucar.edu, and since the attempt on papagayo is to keep the most recent 32 images -regardless of day/time-, you would do better to point at it for your automated processes. This is what I was getting at in a previous email. >Should I expect (if no local >errors) the other ADDE sites to have all images in RTGINI? No. Again, the name of the dataset is the same, but the contents can be very different (numbers of images). The number of images kept on a particular server depends on how much disk space that server has and how many images the administrator of that server wants to keep. >And with GINIEAST and GINIWEST? The exact same comment(s) apply. As far as datasets go, GINIEAST is the set of images that a site has from NOAAPORT channel 1; GINIWEST is the set of images a site has from NOAAPORT channel 2; GINICOMP is the set of composite images that a site has from either or both of NOAAPORT channels 1 and 2; RTGINI is the union of the sets GINICOMP, GINIEAST, and GINIWEST. >Should I use one over the other in the commands to >get the images? Yes, I think that papagayo.unl.edu's holdings would better suit your purposes. >In a related question: will the McIDAS data stream have more than an >hourly image? As you know, it doesn't right now. It _could_ have more images, but given how easy it is to access imagery remotely (with ADDE and DODS), and given that remote access methods do not require that full images be sent, it is much more efficient to simply have sites use remote access methods to get the portions of images that they are interested in from machines that can handle the load. The point is that a large number of sites simply do not have the network bandwidth or storage facilities to receive and save the images that are currently available on NOAAPORT. Each VIS image (and there are 8 per hour: 4 from GOES-East and 4 from GOES-West) is about 26 MB in size. If a site was to ask for just these two image sectors, they would have to be able to handle (receive and store) 200 MB of images each hour. And, this is on top of all of the other data that they want to receive. And, the amount of satellite data that will be available is increasing by many orders of magnitude. Again, the number of sites that can handle this kind of data load is small and will be getting smaller as the data volume goes up. Tom
NOTE: All email exchanges with Unidata User Support are recorded in the Unidata inquiry tracking system and then made publicly available through the web. If you do not want to have your interactions made available in this way, you must let us know in each email you send to us.