[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: 20050623: python + LDM

Hi Daryl,

> To: address@hidden
> From: Daryl Herzmann <address@hidden>
> Subject: python + LDM
> Organization: UCAR/Unidata
> Keywords: 200506231613.j5NGDeZu022545

The above message contained the following:

> I mucked up a small peice of python code that bridges pqact and python via 
> Twisted Python.  Basically it allows for non-blocking processing of 
> products while the STDIN pipe remains open.  Currently, I am invoking 
> python for each product received, which is not good (Bad daryl!)

It's my understanding that a Python decoder will run about three
orders of magnitude slower (or, eqivalently, use about three orders of
magnitude more CPU) than the equivalent C decoder.

> I wanted to run this by you folks, before I sent a more polished version 
> to the ldm-users list.  Perhaps somebody else has already done this and I 
> suspect better!  Or there are some python hacks there that can comment on 
> this code...

It appears that your code would only be good for textual products
because it looks for an STX control-character -- and binary products
would have plenty of those.

> There are many reasons why this is interesting.  The main one being the 
> non-blocking nature and the ability for a long running python process that 
> is not reinvoked constantly....

I'll run it past our Python expert.

> later,
>    daryl
> 
> #=cut==== cat bridgefe.py =====
> 
> #!/usr/bin/env python
> 
> import ldmbridge
> from twisted.internet import reactor
> 
> class myProductIngestor(ldmbridge.LDMProductReceiver):
> 
>      # Here is where you would write your processing code!
>      def processData(self, data):
>          print "I would like to do something here"
> 
> fact = ldmbridge.LDMProductFactory( myProductIngestor() )
> 
> reactor.run()
> 
> #=cut==== cat ldmbridge.py =====
> 
> import sys, re
> 
> from twisted.internet import stdio
> from twisted.protocols import basic
> from twisted.internet import reactor
> 
> class LDMProductReceiver(basic.LineReceiver):
>      delimiter = '\n'
>      productDelimiter = '\003'
> 
>      def __init__(self):
>          self.productBuffer = ""
>          self.setRawMode()
> 
>      def rawDataReceived(self, data):
>          pos = data.find(self.productDelimiter)
>          if (pos == -1):
>              self.productBuffer += data
>          else:
>              self.productBuffer += data[:pos]
>              reactor.callLater(0, self.processData, self.productBuffer)
>              self.productBuffer = data[pos:]
> 
>      def connectionLost(self, reason):
>          reactor.stop()
> 
>      def processData(self, data):
>          raise NotImplementedError
> 
> class LDMProductFactory( stdio.StandardIO ):
> 
>      def __init__(self, protocol):
>          stdio.StandardIO.__init__(self, protocol)
> 
>      def connectionLost(self, reason):
>          reactor.stop()
> 
> 

Regards,
Steve Emmerson

> NOTE: All email exchanges with Unidata User Support are recorded in the
> Unidata inquiry tracking system and then made publicly available
> through the web.  If you do not want to have your interactions made
> available in this way, you must let us know in each email you send to us.