[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[Datastream #QJW-779379]: NLDN lightning data

Hi Rick,

re:
> I hope you can answer this question. We're using NLDN data
> and there are quite a few holes in our archive. I'm not
> sure if this is because the data files aren't created when
> there's zero lightning in the data domain, or if we just have
> an unreliable data feed. I realize this is somewhat ambiguous
> but would like to know if the files are produced even when
> there's no data. Thanks.

The NLDN IDD products are no longer produced when there is no data;
they used to be, but this was changed a couple of years ago.

David Knight said:
> As you know, the data is binned in 6 minute raw files

This also is no longer true; the 6-minute binning of the data
was discontinued at the same time as the removal of blank, "heartbeat"
products were removed.

I hope that this helps!

Cheers,

Tom

> I'm not positive about this - I did not write the code, and don't have time
> right now to dig through it. In fact we are really no longer maintaining
> this code, the person who wrote it left, Unidata has been doing it for some 
> years
> now. I'll cc Tom Yoksas, the last person to touch the code.
> As you know, the data is binned in 6 minute raw files. I'm pretty sure that
> if the file is empty it is not added to the LDM queue. We archive what we
> put in the ldm queue (even though we don't use it) and there are several
> holes in Jan 2013. Several days with apparently no lightning in the 
> continental
> US. I don't think this is uncommon in the winter. We store by hour and Julian
> day. Here are the days we have data for in Jan 2013.
> 
> striker2:/nldnarchive/ingest/nldn2013> ls -l
> total 152
> drwxr-xr-x   2 ldmnldn  unidata     1024 Jan  4 05:05 001
> drwxr-xr-x   2 ldmnldn  unidata      512 Jan  6 05:05 003
> drwxr-xr-x   2 ldmnldn  unidata      512 Jan  7 05:05 004
> drwxr-xr-x   2 ldmnldn  unidata     1024 Jan  8 05:05 005
> drwxr-xr-x   2 ldmnldn  unidata     1024 Jan  9 05:05 006
> drwxr-xr-x   2 ldmnldn  unidata      512 Jan 10 05:05 007
> drwxr-xr-x   2 ldmnldn  unidata     1024 Jan 11 05:05 008
> drwxr-xr-x   2 ldmnldn  unidata     1024 Jan 12 05:05 009
> drwxr-xr-x   2 ldmnldn  unidata     1024 Jan 13 05:05 010
> drwxr-xr-x   2 ldmnldn  unidata      512 Jan 14 05:05 011
> drwxr-xr-x   2 ldmnldn  unidata     1024 Jan 15 05:05 012
> drwxr-xr-x   2 ldmnldn  unidata     1024 Jan 16 05:05 013
> drwxr-xr-x   2 ldmnldn  unidata     1024 Jan 17 05:05 014
> drwxr-xr-x   2 ldmnldn  unidata      512 Jan 18 05:05 015
> drwxr-xr-x   2 ldmnldn  unidata      512 Jan 19 05:05 016
> drwxr-xr-x   2 ldmnldn  unidata      512 Jan 20 05:05 017
> drwxr-xr-x   2 ldmnldn  unidata      512 Jan 20 05:05 018
> drwxr-xr-x   2 ldmnldn  unidata      512 Jan 22 05:05 019
> drwxr-xr-x   2 ldmnldn  unidata     1024 Jan 23 05:05 020
> drwxr-xr-x   2 ldmnldn  unidata      512 Jan 24 05:05 021
> drwxr-xr-x   2 ldmnldn  unidata     1024 Jan 25 05:05 022
> drwxr-xr-x   2 ldmnldn  unidata      512 Jan 26 05:05 023
> drwxr-xr-x   2 ldmnldn  unidata     1024 Jan 27 05:05 024
> drwxr-xr-x   2 ldmnldn  unidata      512 Jan 28 05:05 025
> drwxr-xr-x   2 ldmnldn  unidata     1024 Jan 29 05:05 026
> drwxr-xr-x   2 ldmnldn  unidata     1024 Jan 30 05:05 027
> drwxr-xr-x   2 ldmnldn  unidata     1024 Jan 31 05:05 028
> drwxr-xr-x   2 ldmnldn  unidata     1024 Feb  1 05:05 029
> drwxr-xr-x   2 ldmnldn  unidata     1024 Feb  2 05:05 030
> drwxr-xr-x   2 ldmnldn  unidata      512 Feb  3 05:05 031
> 
> And here are the hours we have data for on Jan 1 2013:
> striker2:/nldnarchive/ingest/nldn2013> ls -l 001
> total 132
> -rw-r--r--   1 ldmnldn  unidata      224 Jan  1 00:51 201300100.ingest
> -rw-r--r--   1 ldmnldn  unidata      784 Jan  1 02:31 201300102.ingest
> -rw-r--r--   1 ldmnldn  unidata      980 Jan  1 03:55 201300103.ingest
> -rw-r--r--   1 ldmnldn  unidata     2212 Jan  1 05:00 201300104.ingest
> -rw-r--r--   1 ldmnldn  unidata     2408 Jan  1 05:48 201300105.ingest
> -rw-r--r--   1 ldmnldn  unidata     2632 Jan  1 07:00 201300106.ingest
> -rw-r--r--   1 ldmnldn  unidata     3024 Jan  1 08:00 201300107.ingest
> -rw-r--r--   1 ldmnldn  unidata     5040 Jan  1 09:00 201300108.ingest
> -rw-r--r--   1 ldmnldn  unidata     4788 Jan  1 10:01 201300109.ingest
> -rw-r--r--   1 ldmnldn  unidata     8904 Jan  1 11:00 201300110.ingest
> -rw-r--r--   1 ldmnldn  unidata     7672 Jan  1 11:59 201300111.ingest
> -rw-r--r--   1 ldmnldn  unidata     5488 Jan  1 12:59 201300112.ingest
> -rw-r--r--   1 ldmnldn  unidata     4368 Jan  1 13:55 201300113.ingest
> -rw-r--r--   1 ldmnldn  unidata     2912 Jan  1 15:01 201300114.ingest
> -rw-r--r--   1 ldmnldn  unidata     4480 Jan  1 15:59 201300115.ingest
> -rw-r--r--   1 ldmnldn  unidata     2576 Jan  1 17:00 201300116.ingest
> -rw-r--r--   1 ldmnldn  unidata      868 Jan  1 17:46 201300117.ingest
> -rw-r--r--   1 ldmnldn  unidata      196 Jan  1 23:04 201300123.ingest
> 
> 
> Hope this helps,
> David
> 
> 
> On Mar 19, 2013, at 6:58 PM, Rick Steed wrote:
> 
> > Hi David-
> >
> > I hope you can answer this question. We're using NLDN data and there are 
> > quite a few holes in our archive. I'm not sure if this is because the data 
> > files aren't created when there's zero lightning in the data domain, or if 
> > we just have an unreliable data feed. I realize this is somewhat ambiguous 
> > but would like to know if the files are produced even when there's no data. 
> > Thanks.
> >
> > -Rick Steed
> > University of Washington
> >
> > On 03/19/13 15:42, Harry Edmon wrote:
> >> Don't know the answer.    Our contact for that data is Dr. David Knight at 
> >> SUNY Albany - address@hidden.   He is a UW grad.
> >>
> >> On 03/19/13 15:30, Rick Steed wrote:
> >>> Harry-
> >>>
> >>> That file (just an example) DOES exist. But there are numerous dates 
> >>> which do not throughout the winter. For example, Dec 2012 only has 640 
> >>> and Jan 2013 only 513 out of a possible 744 files each.
> >>>
> >>> -Rick
> >>>
> >>> On 03/19/13 15:27, Harry Edmon wrote:
> >>>> I'm confused (easy to do) - that file exists:
> >>>>
> >>>> 112 moist% ls -l /home/disk/archive/nldn/raw/201207/2012073018.nldn
> >>>> -rw-rw-r-- 1 ldm ldm 384440 Jul 30  2012 
> >>>> /home/disk/archive/nldn/raw/201207/2012073018.nldn
> >>>>
> >>>>
> >>>> On 03/19/13 15:23, Rick Steed wrote:
> >>>>> Hey Mark-
> >>>>>
> >>>>> There is a fair number of times (esp winter months) when the nldn data 
> >>>>> is missing from the archive. Is it possible that no file is created 
> >>>>> when there is ZERO lightning anywhere in the US? I can see how that 
> >>>>> might happen but I need to confirm. I'm looking at the files such as:
> >>>>>
> >>>>> /home/disk/archive/nldn/raw/201207/2012073018.nldn
> >>>>>
> >>>>> Thanks.
> >>>>>
> >>>>> -Rick
> >>>>>
> >>>>
> >>>>
> >>
> >>
> >
> 
> David Knight
> Dept. of Atmospheric and Environmental Sciences  Tel: (518)-442-4204
> University at Albany, ES-228                     Fax: (518)-442-4494
> Albany, NY  12222                              Email: address@hidden
> 
> 
> 

Cheers,

Tom
--
****************************************************************************
Unidata User Support                                    UCAR Unidata Program
(303) 497-8642                                                 P.O. Box 3000
address@hidden                                   Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage                       http://www.unidata.ucar.edu
****************************************************************************


Ticket Details
===================
Ticket ID: QJW-779379
Department: Support Datastream
Priority: Normal
Status: Closed