[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[IDD #TTR-495389]: Slow access to local datasets through ADDE



Hi Samuel,

re:
> O.K. Just to let you know:
> 
> I have tried to reduce some of our data processing, to reduce how far behind 
> shu's
> processing times are, to eliminate gaps in the data we currently store.  
> Unfortunately
> it seems as though there are still gaps in the GINIEAST/GE1KVIS (GOES-EAST 
> 1km VISUAL)
> satellite imagery.

OK.  It may still be necessary to do some more tweaking of your processing 
(pqact) setup.

> Overnight, shu hasn't had any significant "Behind" messages in ldmd.log(only 
> 1/2 second or so),
> and pqmon -l- indicates the oldest queue entry is 2670 seconds.

This sounds much better.

> I have eliminated the ingesting of PR,AK, and HI imagery, and moved the 
> NNEXRAD file processing
> to a separate file system.  Unfortunately, this hasn't seemed to eliminate 
> all of our missing data.

You may want to create a pqact.conf_nnexrad file that only contains the NNEXRAD 
processing.
Then, remove NNEXRAD as one of the feedtypes in pqact.conf_mcidasB processing 
and add a
new ldmd.conf 'exec' line to run a new 'pqact' instance for the NNEXRAD 
processing:

change:

exec    "pqact -f FSL2|NLDN|NNEXRAD|UNIWISC|HDS|CONDUIT etc/pqact.conf_mcidasB"

to:

exec    "pqact -f FSL2|NLDN|UNIWISC|HDS|CONDUIT etc/pqact.conf_mcidasB"
exec    "pqact -f NNEXRAD etc/pqact.conf_nnexrad"

> Is there a way to prune down our data, to avoid processing data from outside 
> the lower 48
> United States?

This may be possible in theory, but in practice it would be pretty much of a 
nightmare.

As a first step, let's try moving the NNEXRAD processing into its own pqact 
invocation
as I outlined above.

If this doesn't do all that is needed, let's move the NIMAGE processing out of 
pqact.conf_mcidasA
into its own pqact.conf_nimage file and add another invocation of pqact to 
ldmd.conf
that just processes the NIMAGE data:

change:

exec    "pqact -f IDS|DDPLUS|HRS|NGRID|FNEXRAD|NIMAGE etc/pqact.conf_mcidasA"

to:

exec    "pqact -f IDS|DDPLUS|HRS|NGRID|FNEXRAD etc/pqact.conf_mcidasA"
exec    "pqact -f NIMAGE etc/pqact.conf_nimage"

Cheers,

Tom
****************************************************************************
Unidata User Support                                    UCAR Unidata Program
(303) 497-8642                                                 P.O. Box 3000
address@hidden                                   Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage                       http://www.unidata.ucar.edu
****************************************************************************


Ticket Details
===================
Ticket ID: TTR-495389
Department: Support McIDAS
Priority: Normal
Status: Closed


NOTE: All email exchanges with Unidata User Support are recorded in the Unidata inquiry tracking system and then made publicly available through the web. If you do not want to have your interactions made available in this way, you must let us know in each email you send to us.