[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[IDD #CIV-170475]: LDM Inquery

Hi Weidong,

re:
> Thanks a lot for your reply,

No worries.

> Since my computer is protected by the College Firewall, it is not accessible
> from outside, it take ages to open the port outside.

OK.  Is the firewall open for you to make requests to machines' port 388?

> I wonder it is possible
> to give me a temporary access to the GOES image data through LDM see how quick
> and how reliable the data steam will be.

Done.  Please try feeding UNIWISC from idd.cise-nsf.gov.  An appropriate
~ldm/etc/ldmd.conf REQUEST line will look like:

request UNIWISC ".*"  idd.cise-nsf.gov


> Another thing is we are particularly interesting in GOES image data, do you
> have any information about the format and data size?

The GOES image sectors we send in the IDD UNIWISC feed are PNG-compressed
McIDAS AREA files:

- the header information of each sector is not compressed
- the data portion of each sector is PNG-compressed

The simplest way to turn these images back into standard McIDAS AREA
files is by using the ldm-mcidas decoder, pnga2area.  The ldm-mcidas
package is available by anonymous FTP from our FTP server as follows:

machine:    ftp.unidata.ucar.edu
user:       anonymous
pass:       <your email address>
directory:  pub/ldm-mcidas/ldm-mcidas-2008.tar.gz  <- source distribution

Binary releases for select platforms are available in subdirectories
of pub/binary again through anonymous FTP:

darwin_9.6-i386/ldm-mcidas-2008.tar.gz
darwin_8.11-power_macintosh/ldm-mcidas-2008.tar.gz
aix_5.1-powerpc/ldm-mcidas-2008.tar.gz
sunos_5.9-sparc/ldm-mcidas-2008.tar.gz
irix64_6.5-mips/ldm-mcidas-2008.tar.gz
sunos_5.10-i386/ldm-mcidas-2008.tar.gz
linux_2.6-i686/ldm-mcidas-2008.tar.gz
linux_2.6-x86_64/ldm-mcidas-2008.tar.gz

The decoder itself is located in the bin subdirectory of the
binary release.  Example LDM pqact.conf actions can be found
in the etc subdirectory.

As far as the structure of McIDAS AREA files, this is described in
links off of:

http://www.unidata.ucar.edu/software/mcidas/miscdocs/misc_docs.html

Now, on to how to get the data:

The first order of business is to see if you can access the LDM on
idd.cise-nsf.gov.  Please do the following and let us know your findings:

<as 'ldm' on 137.73.24.45>

ldmping idd.cise-nsf.gov   <- you will get a RESPONDING indication if
                              this is working

notifyme -vl- -f UNIWISC -h idd.cise-nsf.gov -o 3600  <- try this if the
                                                         ldmping worked

If the ldmping and notifyme worked, then try starting your
LDM (after putting in the REQUEST for UNIWISC data as outlined
above).  After starting the LDM, monitor the comments being
written in the LDM log file ~ldm/logs/ldmd.log -- they should
show that idd.cise-nsf.gov is willing to feed you the data
you have REQUESTed.

If this looks good, you can monitor the receipt of the UNIWISC
products using the LDM's notifyme utility:

<as 'ldm'>
notifyme -vl- -f UNIWISC

This should show products as they are received.

Back to the size of products:

The size of the products ** as sent in the IDD ** can be seen in the
notifyme listing.  Here is an example from my workstation here at
Unidata:

% notifyme -vl- -f UNIWISC -o 3600
Sep 29 18:04:24 notifyme[22697] NOTE: Starting Up: localhost: 
20090929170424.765 TS_ENDT {{UNIWISC,  ".*"}}
Sep 29 18:04:24 notifyme[22697] NOTE: LDM-5 desired product-class: 
20090929170424.765 TS_ENDT {{UNIWISC,  ".*"}}
Sep 29 18:04:24 notifyme[22697] INFO: Resolving localhost to 127.0.0.1 took 
0.000653 seconds
Sep 29 18:04:24 notifyme[22697] NOTE: NOTIFYME(localhost): OK
Sep 29 18:04:25 notifyme[22697] INFO:    94571 20090929171906.508 UNIWISC 000  
pnga2area Q0 CE 1140 GOES-11_SND UNKBAND 1km 20090929 1600
Sep 29 18:04:26 notifyme[22697] INFO:   804046 20090929173045.407 UNIWISC 000  
pnga2area Q1 UE 1231 GOES-12_IMG 13.3um 4km 20090929 1715
Sep 29 18:04:26 notifyme[22697] INFO:    94571 20090929173410.757 UNIWISC 000  
pnga2area Q0 CE 1140 GOES-11_SND UNKBAND 1km 20090929 1600
Sep 29 18:04:26 notifyme[22697] INFO:  1085894 20090929173417.036 UNIWISC 000  
pnga2area Q1 UW 211 GOES-12_IMG 6.5um 4km 20090929 1715
Sep 29 18:04:26 notifyme[22697] INFO:  2029752 20090929173418.331 UNIWISC 000  
pnga2area Q1 UI 151 GOES-12_IMG 10.7um 4km 20090929 1715
Sep 29 18:04:26 notifyme[22697] INFO:  2125292 20090929173420.585 UNIWISC 000  
pnga2area Q1 UD 1221 GOES-12_IMG 3.9um 4km 20090929 1715
Sep 29 18:04:26 notifyme[22697] INFO:  3045657 20090929173719.523 UNIWISC 000  
pnga2area Q1 UV 141 GOES-12_IMG 0.65um 4km 20090929 1715
Sep 29 18:04:26 notifyme[22697] INFO:   159810 20090929174131.020 UNIWISC 000  
pnga2area Q3 UB 171 GOES-11_IMG 6.8um 8km 20090929 1730
 ...^C

The field directly under NOTIFYME is the size of each product.  The
sizes of the products after being decoded (the decoding is actually
more of uncompressing) are listed in:

http://www.unidata.ucar.edu/committees/usercom/users/prop.sat.06.03.html

The second order of business is to set up your LDM with a
default sized queue; add the example REQUEST line above to the
~ldm/etc/ldmd.conf file; and start your LDM.  Please leave the
line that reports real time statistics in your ~ldm/etc/ldmd.conf
file (i.e., do not delete it or comment it out):

EXEC    "rtstats -h rtstats.unidata.ucar.edu"

The reporting of real time stats back to us will allow us to
monitor the latencies for the products are receiving.

> Thanks a lot,

No worries.

Questions:

- what sort of coverages do you need for your fire detection
  investigation?

- does your technique require 10-bit GOES data?

  The sectors in the UNIWISC datastream are 8-bit.

If your research requires the full resolution data, then we will
need to discuss a different method for your access to GOES data.

Cheers,

Tom
--
****************************************************************************
Unidata User Support                                    UCAR Unidata Program
(303) 497-8642                                                 P.O. Box 3000
address@hidden                                   Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage                       http://www.unidata.ucar.edu
****************************************************************************


Ticket Details
===================
Ticket ID: CIV-170475
Department: Support IDD
Priority: Normal
Status: Closed