Re: new NOAAPORT NIDS products

Stonie,

We've got things reversed--our NTP server is the final receptor and our
four NOAAport systems synch off that server. The times in our filenames
are pulled right from the headers. The directory listing times are times
that the files were received by the server. As I indicated, the NOAAPORT
machine feeding the LDM on the server buffers the data in one minute
files and then dumps the buffer files at one minute intervals for LDM
ingest on the server. It seems to do this at 30 seconds after the
previous minute's buffer has been built. Hence, the data actually do
come in "at least" a minute (since seconds aren't recorded) earlier.
It's certainly close enough to real-time for our purposes.

We used to grab the data via NFS directly from the NOAAport ingestor,
but our hard drives are now way too small to hold very much data because
of the deluge from the new NIDS products, so we now move them over to a
server where we have a large RAID system, which allows us to store all
products from all the NIDS sites for a much longer period.

                Jim
--
James P. Koermer             E-Mail: koermer@xxxxxxxxxxxxxxxxx
Professor of Meteorology     Office Phone: (603)535-2574
Natural Science Department   Office Fax: (603)535-2723
Plymouth State College       WWW: http://vortex.plymouth.edu/
Plymouth, NH 03264
 

"Stonie R. Cooper" wrote:
> 
> Dr. Koermer and Robert,
> 
> Are you experiencing any clock drift?  I've matched your entries below with
> what we are seeing on our MetBox, which is separated by a couple of LDM jumps
> from our NOAAPort.  It may be a hardware difference, also - big hardware,
> obviously, will keep up better.  Our NOAAPort is the NTP server for our
> network, syncing to the NOAAPort time sync messages - we are seeing a two
> minute difference from what you are seeing:
> 
> On Thursday 29 November 2001 17:45, Jim Koermer wrote:
> > From what I see over our NOAAport system, there is several minutes of
> > delay for some products. Here is part of a directory listing for one
> > site (GYX):
> >
> > -rw-r--r--  1 ldm  usr    4937 Nov 29 17:35 0111291730_NTP.nid
> 
> -rw-r--r--   1 ldm      pdi          4923 Nov 29 17:33 ./NTP/NTP_20011129_1730
> 
> > -rw-r--r--  1 ldm  usr    7123 Nov 29 17:35 0111291730_N3S.nid
> 
> -rw-r--r--   1 ldm      pdi          7109 Nov 29 17:33 ./N3S/N3S_20011129_1730
> 
> > -rw-r--r--  1 ldm  usr    5553 Nov 29 17:35 0111291730_N1P.nid
> 
> -rw-r--r--   1 ldm      pdi          5539 Nov 29 17:33 ./N1P/N1P_20011129_1730
> 
> > -rw-r--r--  1 ldm  usr    9689 Nov 29 17:34 0111291730_N3R.nid
> 
> -rw-r--r--   1 ldm      pdi          9675 Nov 29 17:33 ./N3R/N3R_20011129_1730
> 
> > -rw-r--r--  1 ldm  usr    7948 Nov 29 17:34 0111291730_N2S.nid
> 
> -rw-r--r--   1 ldm      pdi          7934 Nov 29 17:32 ./N2S/N2S_20011129_1730
> 
> > -rw-r--r--  1 ldm  usr   10856 Nov 29 17:34 0111291730_N2R.nid
> 
> -rw-r--r--   1 ldm      pdi         10842 Nov 29 17:32 ./N2R/N2R_20011129_1730
> 
> > -rw-r--r--  1 ldm  usr    9663 Nov 29 17:33 0111291730_N1S.nid
> 
> -rw-r--r--   1 ldm      pdi          9649 Nov 29 17:32 ./N1S/N1S_20011129_1730
> 
> > -rw-r--r--  1 ldm  usr    9788 Nov 29 17:33 0111291730_N1V.nid
> 
> -rw-r--r--   1 ldm      pdi          9774 Nov 29 17:31 ./N1V/N1V_20011129_1730
> 
> > -rw-r--r--  1 ldm  usr   11717 Nov 29 17:33 0111291730_N1R.nid
> 
> -rw-r--r--   1 ldm      pdi         11703 Nov 29 17:32 ./N1R/N1R_20011129_1730
> 
> > -rw-r--r--  1 ldm  usr   10595 Nov 29 17:33 0111291730_N0S.nid
> 
> -rw-r--r--   1 ldm      pdi         10581 Nov 29 17:31 ./N0S/N0S_20011129_1730
> 
> > -rw-r--r--  1 ldm  usr   13112 Nov 29 17:32 0111291730_N0V.nid
> 
> -rw-r--r--   1 ldm      pdi         13098 Nov 29 17:31 ./N0V/N0V_20011129_1730
> 
> > -rw-r--r--  1 ldm  usr    9398 Nov 29 17:32 0111291730_N0Z.nid
> 
> -rw-r--r--   1 ldm      pdi          9384 Nov 29 17:30 ./N0Z/N0Z_20011129_1730
> 
> > -rw-r--r--  1 ldm  usr   14439 Nov 29 17:32 0111291730_N0R.nid
> 
> -rw-r--r--   1 ldm      pdi         14425 Nov 29 17:30 ./N0R/N0R_20011129_1730
> 
> > -rw-r--r--  1 ldm  usr   19051 Nov 29 17:32 0111291724_NCR.nid
> 
> -rw-r--r--   1 ldm      pdi         19037 Nov 29 17:30 ./NCR/NCR_20011129_1724
> 
> > -rw-r--r--  1 ldm  usr     634 Nov 29 17:31 0111291724_NVL.nid
> 
> -rw-r--r--   1 ldm      pdi           620 Nov 29 17:29 ./NVL/NVL_20011129_1724
> 
> > -rw-r--r--  1 ldm  usr    2447 Nov 29 17:31 0111291724_NVW.nid
> 
> -rw-r--r--   1 ldm      pdi          2433 Nov 29 17:29 ./NVW/NVW_20011129_1724
> 
> > -rw-r--r--  1 ldm  usr    1243 Nov 29 17:31 0111291724_NET.nid
> 
> -rw-r--r--   1 ldm      pdi          1229 Nov 29 17:29 ./NET/NET_20011129_1724
> 
> > -rw-r--r--  1 ldm  usr    4964 Nov 29 17:29 0111291724_NTP.nid
> 
> -rw-r--r--   1 ldm      pdi          4950 Nov 29 17:27 ./NTP/NTP_20011129_1724
> 
> > -rw-r--r--  1 ldm  usr    5530 Nov 29 17:29 0111291724_N1P.nid
> 
> -rw-r--r--   1 ldm      pdi          5516 Nov 29 17:27 ./N1P/N1P_20011129_1724
> 
> > -rw-r--r--  1 ldm  usr    6764 Nov 29 17:29 0111291724_N3S.nid
> 
> -rw-r--r--   1 ldm      pdi          6750 Nov 29 17:27 ./N3S/N3S_20011129_1724
> 
> > -rw-r--r--  1 ldm  usr    9597 Nov 29 17:28 0111291724_N3R.nid
> 
> -rw-r--r--   1 ldm      pdi          9583 Nov 29 17:27 ./N3R/N3R_20011129_1724
> 
> > -rw-r--r--  1 ldm  usr    7767 Nov 29 17:28 0111291724_N2S.nid
> 
> -rw-r--r--   1 ldm      pdi          7753 Nov 29 17:26 ./N2S/N2S_20011129_1724
> 
> > -rw-r--r--  1 ldm  usr   10727 Nov 29 17:28 0111291724_N2R.nid
> 
> -rw-r--r--   1 ldm      pdi         10713 Nov 29 17:26 ./N2R/N2R_20011129_1724
> 
> > -rw-r--r--  1 ldm  usr    9639 Nov 29 17:27 0111291724_N1V.nid
> 
> -rw-r--r--   1 ldm      pdi          9625 Nov 29 17:26 ./N1V/N1V_20011129_1724
> 
> > -rw-r--r--  1 ldm  usr    9108 Nov 29 17:27 0111291724_N1S.nid
> 
> -rw-r--r--   1 ldm      pdi          9094 Nov 29 17:26 ./N1S/N1S_20011129_1724
> 
> > -rw-r--r--  1 ldm  usr   11581 Nov 29 17:27 0111291724_N1R.nid
> 
> -rw-r--r--   1 ldm      pdi         11567 Nov 29 17:25 ./N1R/N1R_20011129_1724
> 
> > -rw-r--r--  1 ldm  usr   11126 Nov 29 17:27 0111291724_N0S.nid
> 
> -rw-r--r--   1 ldm      pdi         11112 Nov 29 17:25 ./N0S/N0S_20011129_1724
> 
> > The filetimes can be about a minute later than when the data actually
> > arrive on our actual NOAAport system, since the raw data for a 1-minute
> > interval are put into a file and then sent to my server (listing above)
> > and acquired via ldm. There is so much data that it can't be
> > instantaneous, especially since the NWSTG channel also handles all
> > conventional data and model data. The precip data does seem to lag the
> > N0R data and probably due to the order inwhich the products are
> > transmitted. Note: the times in the filenames above do correspond to
> > data observation times.
> >
> >                Jim
> 
> Our LDM server is also distributing the other three NOAAPort channels - both
> GOES channels, and the DCP channel (has great AK Eta output).  As Dan had
> explained earlier, the N0* products will be the quickest, as the CCB/WMO/RFP
> time stamp is applied to the beginning of the entire volume scan - not each
> elevation sweep.
> 
> We are seeing the N0* products in near real-time on NOAAPort, so it could be
> a clock issue.
> --
> Stonie R. Cooper,
> Science Officer
> Planetary Data, Incorporated
> 3495 Liberty Road
> Villa Rica, Georgia  30180
> ph. (770) 456-0700; pg. (888) 974-5017; fx. (770) 459-0016