Progress async and await

It's been some time, but I made quite some progress in the new async feature of Python 3.5! There is still a bit to be done though and the end of this years Google Summer of Code is pretty close already. If I can do it in time will mostly be a luck factor, since I don't know how much I will still have to do in order for asyncio to work. The module is dependent of many new features from Python 3.3 up to 3.5 that have not been implemented in PyPy yet.

Does async and await work already?
Not quite. PyPy now accepts async and await though, and checks pretty much all places where it is allowed and where it is not. In other words, the parser is complete and has been tested.
The code generator is complete as well, so the right opcodes get executed in all cases.
The new bytecode instructions I need to handle are: GET_YIELD_FROM_ITER, GET_AWAITABLE, GET_AITER, GET_ANEXT, BEFORE_ASYNC and SETUP_ASYNC.
These opcodes do not work with regular generators, but with coroutine objects. Those are based on generators, however they do not imlement __iter__ and __next__ and can therefore not be iterated over. Also generators and generator based coroutines (@asyncio.coroutines in asyncio) cannot yield from coroutines. [1]
I started implementing the opcodes, but I can only finish them after asyncio is working as I need to test them constantly and can only do that with asyncio, because I am unsure what the values normally lying on the stack are. That is also valid for some functions in coroutine objects. Coroutine objects are working, however they are missing a few functions needed for the async await-syntax feature.
These two things are the rest I have to do though, everything else is tested and should therefore work.

What else has been done?
Only implementing async and await would have been too easy I guess. With it comes a problem I already mentioned, and that is the missing dependencies of Python 3.3 up to 3.5.
The module sre (offers support for regular expressions) was missing a macro named MAXGROUPS (from Python 3.3), the magic number standing for the number of constants had to be updated as well. The memoryview objects also got an update from Python 3.3 that is needed for an import. It has a function called “cast” now, which converts memoryview objects to any other predefined format.
I just finished implementing this as well, now I am at the point where it says inside threading.py:
_set_sentinel = _thread._set_sentinel
AttributeError: 'module' object has no attribute '_set_sentinel'

What to do next?
My next goal is that asyncio works and the new opcodes are implemented. Hopefully I can write about success in my next blog post, because I am sure I will need some time to test everything afterwards.

A developer tip for execution of asyncio in pyinteractive (--withmod)
(I only write that as a hint because it gets easily skipped in the PyPy doc, or at least it happened to me. The PyPy team already thought about a solution for that though :) )
Asyncio needs some modules in order to work which are by default not loaded in pyinteractive. If someone stumbles across the problem where PyPy cannot find these modules, –withmod does the trick [2]. For now, –withmod-thread and –withmod-select are required.

[1] https://www.python.org/dev/peps/pep-0492/
[2] http://doc.pypy.org/en/latest/getting-started-dev.html#pyinteractive-py-options


Update (23.07.): asyncio can be imported and works! Well that went better than expected :)
For now only the @asyncio.coroutine way of creating coroutines is working, so for example the following code would work:

import asyncio

@asyncio.coroutine

def my_coroutine(seconds_to_sleep=3):

    print('my_coroutine sleeping for: {0} seconds'.format(seconds_to_sleep))

    yield from asyncio.sleep(seconds_to_sleep)

loop = asyncio.get_event_loop()

loop.run_until_complete(

    asyncio.gather(my_coroutine())

)

loop.close(

(from http://www.giantflyingsaucer.com/blog/?p=5557)

And to illustrate my goal of this project, here is an example of what I want to work properly:

import asyncio

async def coro(name, lock):
    print('coro {}: waiting for lock'.format(name))
    async with lock:
        print('coro {}: holding the lock'.format(name))
        await asyncio.sleep(1)
        print('coro {}: releasing the lock'.format(name))

loop = asyncio.get_event_loop()
lock = asyncio.Lock()
coros = asyncio.gather(coro(1, lock), coro(2, lock))
try:
    loop.run_until_complete(coros)
finally:
    loop.close()

(from https://docs.python.org/3/whatsnew/3.5.html#whatsnew-pep-492)

The async keyword replaces the @asyncio.coroutine, and await is written instead of yield from. "await with" and "await for" are additional features, allowing to suspend execution in "enter" and "exit" methods (= asynchronous context manager) and to iterate through asynchronous iterators respectively.

Unpacking done! Starting with Coroutines

It took a bit longer than anticipated, but the additional unpacking generalizations are finally completed.

Another good thing: I am now finally ready to work full time and make up for the time I lost, as I don't have to invest time into studying anymore.

The unpackings are done slightly different than in cpython, because in PyPy I get objects with a different structure for the maps. So I had to filter them for the keys and values and do a manual type check if it really is a dict. For the map unpack with call I had to implement the intersection check. For that I just check if a key is already stored in the dict that gets returned.

Now it's time to implement coroutines with async and await syntax. There is still a problem with the translation of PyPy, which is connected to the missing coroutines syntax feature. I will need to get this working as well, starting with implementing async in the parser.