Re: fiber(cooperative multi-threading)

Akihiro KAYAMA <kayama@xxxxxxxxxxxx> wrote:

Thanks for your replies.

But current Python generator specification requires me:

def f1():
for x in foo(f2, "foo"): yield x
for x in foo(f2, "foo"): yield x
# XXX v = ... (I don't know how to do this)
for x in foo(f2, "v=%s world" % v, "OK"): yield x

I think it is not straitforward. Single level function which generator
impose is impractical for real use.

Not just impractical, there are plenty of situations where you simply
cannot do that at all.

Here's a greenlet example (from
http://socal-piggies.org/presentations/grig/2005_07_21/):

from py.magic import greenlet
import xml.parsers.expat

def send(arg):
greenlet.getcurrent().parent.switch(arg)

def start_element(name, attrs):
send(('START', name, attrs))
def end_element(name):
send(('END', name))
def char_data(data):
data = data.strip()
if data: send(('DATA', data))

def greenparse(xmldata):
p = xml.parsers.expat.ParserCreate()
p.StartElementHandler = start_element
p.EndElementHandler = end_element
p.CharacterDataHandler = char_data
p.Parse(xmldata, 1)

def iterxml(xmldata):
g = greenlet(greenparse)
data = g.switch(xmldata)
while data is not None:
yield data
data = g.switch()

if __name__ == "__main__":
for data in iterxml("somexmldata"):
# do something with data


The greenlet here calls expat, but the 'yield' happens inside an expat
callback. To get this to work you would need to rewrite expat to use its
callback as a generator. Expat is coded in C, so you also need to find
some way to get it to save the state of the C functions when it has
yielded.

I think this example should answer Arnaud's question ("In fact it is not
clear to me at the moment what can be done (sensibly :) with the OP's
Fiber class that cannot be achieved with the run() function I
suggested.") Arnaud's run function cannot be used by anything which needs
to save C stack state.

I am happy if I could use convenient coroutine features via standard
or simple extension library. py.magic.greenlet may be what I'm
looking for, but I wonder why this is named "magic" :-)

I think it is called "magic" because it does something which is at first
glance impossible: saving the C stack state as well as the Python stack.
Possibly it does something underhand to achieve an impressive effect.
I haven't looked at the implementation to see how it does it: creating
threads and then scheduling them cooperatively would be one way but I
expect it just switches the C stack around between different areas of
memory.

A long time ago I ported the BCPL coroutine library and there is one
function in the middle (the one which does the stack switch) that
definitely felt like magic.

One interesting question raised by all this is whether those of us using
Windows can usefully call Microsoft's Fiber functions through ctypes and
get the benefit of coroutines without having to load any additional C
extensions at all.
.

Relevant Pages

  • Re: Nested generator caveat
    ... Here's what's actually going on in your generator. ... def gen1: ... yield i, gen1 ... the for loop in gen0 is suspended each iteration while we do some ...
    (comp.lang.python)
  • Re: First attempt at a Python prog (Chess)
    ... EMPTY = 0 ... def display_board: ... In Python 2 you can get this effect by using: ... yield board ...
    (comp.lang.python)
  • Re: trouble with generators
    ... def records(self, cls): ... yield cls ... It's not quadratic - you add the same B to a.b in each generator run, so the first run shows 3 times the B, then 6, then 9. ...
    (comp.lang.python)
  • Re: Just for fun: Countdown numbers game solver
    ... Ops is vanilla, cleverops only returns canonical expressions as defined in Dan's email. ... def getop: return 'n' if isinstance ... yield x + y, ... "Return all ways of reaching target with nums" ...
    (comp.lang.python)
  • Re: Augmented generators?
    ... AFAIK there's no way to have "yield" produce anything other than a ... generator. ... def value: ... dict.iteritemsenumerates tuples for dictionaries, ...
    (comp.lang.python)