[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: 20050623: python + LDM



Hi Daryl,

> To: address@hidden
> From: Daryl Herzmann <address@hidden>
> Subject: python + LDM
> Organization: UCAR/Unidata
> Keywords: 200506231613.j5NGDeZu022545

The above message contained the following:

> I mucked up a small peice of python code that bridges pqact and python via 
> Twisted Python.  Basically it allows for non-blocking processing of 
> products while the STDIN pipe remains open.  Currently, I am invoking 
> python for each product received, which is not good (Bad daryl!)

It's my understanding that a Python decoder will run about three
orders of magnitude slower (or, eqivalently, use about three orders of
magnitude more CPU) than the equivalent C decoder.

> I wanted to run this by you folks, before I sent a more polished version 
> to the ldm-users list.  Perhaps somebody else has already done this and I 
> suspect better!  Or there are some python hacks there that can comment on 
> this code...

It appears that your code would only be good for textual products
because it looks for an STX control-character -- and binary products
would have plenty of those.

> There are many reasons why this is interesting.  The main one being the 
> non-blocking nature and the ability for a long running python process that 
> is not reinvoked constantly....

I'll run it past our Python expert.

> later,
>    daryl
> 
> #=cut==== cat bridgefe.py =====
> 
> #!/usr/bin/env python
> 
> import ldmbridge
> from twisted.internet import reactor
> 
> class myProductIngestor(ldmbridge.LDMProductReceiver):
> 
>      # Here is where you would write your processing code!
>      def processData(self, data):
>          print "I would like to do something here"
> 
> fact = ldmbridge.LDMProductFactory( myProductIngestor() )
> 
> reactor.run()
> 
> #=cut==== cat ldmbridge.py =====
> 
> import sys, re
> 
> from twisted.internet import stdio
> from twisted.protocols import basic
> from twisted.internet import reactor
> 
> class LDMProductReceiver(basic.LineReceiver):
>      delimiter = '\n'
>      productDelimiter = '\003'
> 
>      def __init__(self):
>          self.productBuffer = ""
>          self.setRawMode()
> 
>      def rawDataReceived(self, data):
>          pos = data.find(self.productDelimiter)
>          if (pos == -1):
>              self.productBuffer += data
>          else:
>              self.productBuffer += data[:pos]
>              reactor.callLater(0, self.processData, self.productBuffer)
>              self.productBuffer = data[pos:]
> 
>      def connectionLost(self, reason):
>          reactor.stop()
> 
>      def processData(self, data):
>          raise NotImplementedError
> 
> class LDMProductFactory( stdio.StandardIO ):
> 
>      def __init__(self, protocol):
>          stdio.StandardIO.__init__(self, protocol)
> 
>      def connectionLost(self, reason):
>          reactor.stop()
> 
> 

Regards,
Steve Emmerson

> NOTE: All email exchanges with Unidata User Support are recorded in the
> Unidata inquiry tracking system and then made publicly available
> through the web.  If you do not want to have your interactions made
> available in this way, you must let us know in each email you send to us.


NOTE: All email exchanges with Unidata User Support are recorded in the Unidata inquiry tracking system and then made publicly available through the web. If you do not want to have your interactions made available in this way, you must let us know in each email you send to us.