[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

RE: writing files in the product queue



Hi Larry,

Well, there would need to be data in your queue that matches your reg ex
for pqact to "act" on anything....I do not know the size of your queue, if
it is large enough to still hold the 12Z CONDUIT run, then running pqact
from line command will work. You can add a " -o " to give it a time
offset long enough to find the past run.

Run pqmon, as LDM from LDM machine, to see how long of a residency time
the data has in your queue, if it is not longer than the time from the
previous model run you will likely need to wait until the 18Z run to see
how things are working.

Keep in mind we do store the IDD/LDM feed for a few days at:

http://motherlode.ucar.edu:8080/thredds/catalog.html

then

http://motherlode.ucar.edu:8080/thredds/idd/models.html

http://motherlode.ucar.edu:8080/thredds/catalog/fmrc/NCEP/NAM/Alaska_45km/conduit/catalog.html

So you can pull grids from here until things get working well :)

Hope this helps,

Jeff
---------------------------------------------------------------------
Jeff Weber                                    address@hidden        :
Unidata Program Center                        PH:303-497-8676        :
University Corp for Atmospheric Research      3300 Mitchell Ln       :
http://www.unidata.ucar.edu/staff/jweber      Boulder,Co 80307-3000  :
---------------------------------------------------------------------

On Tue, 6 Apr 2010, Larry Mastin wrote:

> Thanks, Jeff.  I made the change.   Hopefully it will work.
>
> By the way, is it possible to test pqact by running it directly, rather than
> waiting for the cron job to do it?  I tried, for example, typing the
> following at the command line while in ~/data:
>
> pqact -v -q ldm.pq
>
> After hitting the "return" key I got no response.  I waited ten minutes and
> still nothing.
>
> Thanks,
>
> Larry
>
> David A. Johnston Cascades Volcano Observatory
> 1300 SE Cardinal Court, Bldg. 10, Suite 100
> Vancouver, WA 98683 USA
> tel. 360-993-8925  fax 360-993-8980
> http://vulcan.wr.usgs.gov/Projects/Mastin
> -----Original Message-----
> From: Jeff Weber [mailto:address@hidden]
> Sent: Tuesday, April 06, 2010 8:33 AM
> To: Larry Mastin
> Subject: Re: writing files in the product queue
>
> Hi Larry,
>
> Apologies for the delay, got hectic here yesterday aft :)
>
> I would add a -close arg after the FILE command.
>
> i.e.
>
> CONDUIT ^data/nccf/com/nam.*awipak*
> FILE  -close    ~/data/nam/Alaska_45km/\1
>
>
> Please keep in mind that there should only be tabs in pqact.conf and no
> "blank space" to separate the entry.
>
> Please let me know if this does not solve the problem, I think it should
> :)
>
> Cheers,
>
> Jeff
> ---------------------------------------------------------------------
> Jeff Weber                                    address@hidden        :
> Unidata Program Center                        PH:303-497-8676        :
> University Corp for Atmospheric Research      3300 Mitchell Ln       :
> http://www.unidata.ucar.edu/staff/jweber      Boulder,Co 80307-3000  :
> ---------------------------------------------------------------------
>
> On Mon, 5 Apr 2010, Larry Mastin wrote:
>
> > Dear Jeff:
> >
> > You may recall our recent correspondence (below).  I was trying to set
> > up the ldm to ingest NAM files from AWIPS grids 212, 216, and 104.  I
> > think the files are being successfully loaded into the product queue, as
> > indicated by the following information.
> >
> > ldm@phobos:~> pqmon
> > Apr 05 19:34:22 pqmon NOTE: Starting Up (8530)
> > Apr 05 19:34:22 pqmon NOTE: nprods nfree  nempty      nbytes  maxprods
> > maxfree  minempty    maxext  age
> > Apr 05 19:34:22 pqmon NOTE:  54672     1   67397   499993096     55609
> > 2     66459      9720 39018
> > Apr 05 19:34:22 pqmon NOTE: Exiting
> >
> > Also, with the command plotMetrics I can see that data are being
> > downloaded every six hours or so.
> >
> > In the pqact.conf file I have commands to write these data out to
> > separate files, but for some reason they're not being written.
> >
> > Here's the line in the pqact.conf file that's supposed to write out the
> > NAM AWIPS grid 216 file:
> >
> > CONDUIT ^data/nccf/com/nam.*awipak*    FILE    ~/data/nam/Alaska_45km/\1
> >
> > I've tried a few different ERE syntaxes, all of which have been tested
> > using the regex command and found to match NCEP file names of the format
> > given at:
> >
> > http://www.unidata.ucar.edu/data/conduit/ldm_idd/nam_files.html
> >
> > Unfortunately, none of the expressions I've tried have worked in getting
> > the files written from the product queue.
> >
> > So, I wonder if I'm perhaps missing something else.
> >
> > The ldmd.conf and pqact.conf files I'm using are attached, as well as a
> > shell script I've been using to test the regular expressions.
> >
> > If you have any suggestions on how to get this to work, I'd appreciate
> > it.
> >
> > Thanks,
> >
> > Larry Mastin
> >
> > On Thu, 2010-03-11 at 14:57 -0700, Jeff Weber wrote:
> >
> > > Hi Larry,
> > >
> > > For sanity, please try:
> > >
> > > notifyme -vl - -h idd.unidata.ucar.edu -f ANY
> > >
> > > Thanks,
> > >
> > > Jeff
> > > ---
> > >
> > >
> > > On Thu, 11 Mar 2010, Larry Mastin wrote:
> > >
> > > > Jeff:
> > > >
> > > > Here's what I get:
> > > >
> > > > ldm@phobos:~> notifyme -vl - -h freshair.atmos.washington.edu -f ANY
> > > > Mar 11 21:47:35 notifyme[5173] NOTE: Starting Up:
> > > > freshair.atmos.washington.edu: 20100311214735.906 TS_ENDT {{ANY,
> ".*"}}
> > > > Mar 11 21:47:35 notifyme[5173] NOTE: LDM-5 desired product-class:
> > > > 20100311214735.906 TS_ENDT {{ANY,  ".*"}}
> > > > Mar 11 21:47:35 notifyme[5173] INFO: Resolving
> > > > freshair.atmos.washington.edu to 128.95.89.10 took 0.00027 seconds
> > > > Mar 11 21:47:36 notifyme[5173] ERROR:
> > > > NOTIFYME(freshair.atmos.washington.edu): 7: Access denied by remote
> > > > server
> > > > Mar 11 21:48:01 notifyme[5173] NOTE: LDM-5 desired product-class:
> > > > 20100311214735.906 TS_ENDT {{ANY,  ".*"}}
> > > > Mar 11 21:48:01 notifyme[5173] INFO: Resolving
> > > > freshair.atmos.washington.edu to 128.95.89.10 took 1e-05 seconds
> > > > Mar 11 21:48:01 notifyme[5173] ERROR:
> > > > NOTIFYME(freshair.atmos.washington.edu): 7: Access denied by remote
> > > > server
> > > > Mar 11 21:48:26 notifyme[5173] NOTE: LDM-5 desired product-class:
> > > > 20100311214735.906 TS_ENDT {{ANY,  ".*"}}
> > > > Mar 11 21:48:26 notifyme[5173] INFO: Resolving
> > > > freshair.atmos.washington.edu to 128.95.89.10 took 7e-06 seconds
> > > > Mar 11 21:48:26 notifyme[5173] ERROR:
> > > > NOTIFYME(freshair.atmos.washington.edu): 7: Access denied by remote
> > > > server
> > > > Mar 11 21:48:51 notifyme[5173] NOTE: LDM-5 desired product-class:
> > > > 20100311214735.906 TS_ENDT {{ANY,  ".*"}}
> > > > Mar 11 21:48:51 notifyme[5173] INFO: Resolving
> > > > freshair.atmos.washington.edu to 128.95.89.10 took 9e-06 seconds
> > > > Mar 11 21:48:52 notifyme[5173] ERROR:
> > > > NOTIFYME(freshair.atmos.washington.edu): 7: Access denied by remote
> > > > server
> > > > Mar 11 21:49:17 notifyme[5173] NOTE: LDM-5 desired product-class:
> > > > 20100311214735.906 TS_ENDT {{ANY,  ".*"}}
> > > > Mar 11 21:49:17 notifyme[5173] INFO: Resolving
> > > > freshair.atmos.washington.edu to 128.95.89.10 took 9e-06 seconds
> > > > Mar 11 21:49:17 notifyme[5173] ERROR:
> > > > NOTIFYME(freshair.atmos.washington.edu): 7: Access denied by remote
> > > > server
> > > > Mar 11 21:49:42 notifyme[5173] NOTE: LDM-5 desired product-class:
> > > > 20100311214735.906 TS_ENDT {{ANY,  ".*"}}
> > > > Mar 11 21:49:42 notifyme[5173] INFO: Resolving
> > > > freshair.atmos.washington.edu to 128.95.89.10 took 7e-06 seconds
> > > > Mar 11 21:49:42 notifyme[5173] ERROR:
> > > > NOTIFYME(freshair.atmos.washington.edu): 7: Access denied by remote
> > > > server
> > > >
> > > > It looks like access is being denied.
> > > >
> > > > Larry
> > > >
> > > >
> > > > On Thu, 2010-03-11 at 14:25 -0700, Jeff Weber wrote:
> > > >
> > > > > Hi Larry,
> > > > >
> > > > > We have our User's Committee meeting today, apologies for the late
> > > > > response:
> > > > >
> > > > >
> > > > > As ldm run:
> > > > >
> > > > > notifyme -vl - -h freshair.atmos.washington.edu -f ANY
> > > > >
> > > > >
> > > > > ..this will determine if freshair is willing to feed you, if so,
> products
> > > > > should scroll by.
> > > > >
> > > > > I see your e-mail in support now as well, we'll figure this one out
> :)
> > > > >
> > > > > Cheers,
> > > > >
> > > > > Jeff
> > > > >
> > > > > On Thu, 11 Mar 2010, Larry Mastin wrote:
> > > > >
> > > > > > Dear Unidata (Jeff?):
> > > > > >
> > > > > > Six months ago you assisted me in setting up an ldm server at the
> USGS
> > > > > > (the message below may jog your memory).  It turns out I didn't
> finish
> > > > > > the job, but am now working on it again.  I tried activating the
> ldm
> > > > > > server yesterday (using "ldmadmin start") but thus far haven't
> received
> > > > > > any data and have seen no messages in the log file
> ~/logs/ldmd.log.  I'm
> > > > > > trying to troubleshoot my installation, some of which was done
> last
> > > > > > September.
> > > > > >
> > > > > > I found one issue that I could use advice one.  I'm looking at the
> ldm
> > > > > > pre-installation instructions:
> > > > > >
> > > > > >
> http://www.unidata.ucar.edu/software/ldm/ldm-6.8.1/basics/preinstallation.ht
> ml
> > > > > >
> > > > > > under "Edit file /etc/syslog.conf".  It says to "Add the following
> entry
> > > > > > to file /etc/syslog.conf (or /etc/rsyslog.conf, depending on your
> > > > > > operating system):
> > > > > >
> > > > > >     --localn.debug /usr/local/ldm/logs/ldmd.log
> > > > > >
> > > > > > On my Linux Suse 11.0 system, my /etc folder contains neither a
> > > > > > syslog.conf nor an rsyslog.conf file.  There is however a
> /etc/syslog-ng
> > > > > > folder which contains a syslog-ng.conf file.  Is that where this
> line is
> > > > > > supposed to go?
> > > > > >
> > > > > > Thanks for your help.
> > > > > >
> > > > > > Larry Mastin
> > > > > >
> > > > > >
> > > > > > On Tue, 2009-09-29 at 11:27 -0600, Jeff Weber wrote:
> > > > > > > Hi Harry,
> > > > > > >
> > > > > > > Great, thanks for the quick response :)
> > > > > > >
> > > > > > > Larry, please let Unidata know if we can help configure anything
> for you.
> > > > > > >
> > > > > > > One caveat, CONDUIT is voluminous, so it is better to throttle
> back the
> > > > > > > request to only what you desire (we can help with that).
> > > > > > >
> > > > > > > Cheers,
> > > > > > >
> > > > > > > Jeff
> > > > > > >
> ---------------------------------------------------------------------
> > > > > > > Jeff Weber                                    address@hidden
> :
> > > > > > > Unidata Program Center                        PH:303-497-8676
> :
> > > > > > > University Corp for Atmospheric Research      3300 Mitchell Ln
> :
> > > > > > > http://www.unidata.ucar.edu/staff/jweber      Boulder,Co
> 80307-3000  :
> > > > > > >
> ---------------------------------------------------------------------
> > > > > > >
> > > > > > > On Tue, 29 Sep 2009, Harry Edmon wrote:
> > > > > > >
> > > > > > > > freshair.atmos.washington.edu is the alias to our current
> operational
> > > > > > > > system and should be the address they should use.   It is all
> set up.
> > > > > > > >
> > > > > > > > Jeff Weber wrote:
> > > > > > > > > Hi Harry,
> > > > > > > > >
> > > > > > > > > The good people at the USGS David A. Johnston Cascades
> Volcano
> > > > > > > > > Observatory, would like a feed:
> > > > > > > > >
> > > > > > > > > They are requesting:
> > > > > > > > >
> > > > > > > > > HDS
> > > > > > > > > CONDUIT
> > > > > > > > > CMC (GEM)
> > > > > > > > > FNMOC
> > > > > > > > >
> > > > > > > > > They also want NGRID, but I see you do not have that feed,
> so Larry,
> > > > > > > > > please point your NGRID request at us:
> > > > > > > > >
> > > > > > > > > idd.unidata.ucar.edu
> > > > > > > > >
> > > > > > > > > Larry,
> > > > > > > > >
> > > > > > > > > I know I suggested getting the CMC (GEM) data and FNMOC from
> us, but I see
> > > > > > > > > U of W has those, so let's not ship it over to Washington
> twice, if Harry
> > > > > > > > > will place the allow for you let's feed those 4 feeds from U
> of W and we
> > > > > > > > > will feed you the NGRID feed...sound good?
> > > > > > > > >
> > > > > > > > > Harry, which machine would you like Larry to feed from if
> you choose to
> > > > > > > > > grant access to the feed?
> > > > > > > > >
> > > > > > > > > freshair2.atmos.washington.edu or
> freshair1.atmos.washington.edu?
> > > > > > > > >
> > > > > > > > > Thanks for considering this request, I have cc'd Larry
> Mastin, contact at
> > > > > > > > > USGS on this e-mail. The machine name to "allow" is:
> > > > > > > > >
> > > > > > > > > phobos.wr.usgs.gov
> > > > > > > > >
> > > > > > > > > Cheers,
> > > > > > > > >
> > > > > > > > > Jeff
> > > > > > > > >
> ---------------------------------------------------------------------
> > > > > > > > > Jeff Weber
> address@hidden        :
> > > > > > > > > Unidata Program Center
> PH:303-497-8676        :
> > > > > > > > > University Corp for Atmospheric Research      3300 Mitchell
> Ln       :
> > > > > > > > > http://www.unidata.ucar.edu/staff/jweber      Boulder,Co
> 80307-3000  :
> > > > > > > > >
> ---------------------------------------------------------------------
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > >  Dr. Harry Edmon                        E-MAIL:
> address@hidden
> > > > > > > >  206-543-0547 FAX: 206-543-0308
> address@hidden, address@hidden
> > > > > > > >  Director of IT, College of the Environment and
> > > > > > > >  Director of Computing, Dept of Atmospheric Sciences
> > > > > > > >  University of Washington, Box 351640, Seattle, WA 98195-1640
> > > > > > > >
> > > > > >
> > > >
> >
>
>