Hack Me

Various types of arguments and cache

2010-07-09T20:31:00.001+02:00

Good news!

So, I have finally managed how to omit the limitation of RPython to pass arguments of various type to the called functions. The limitation was due to passing the arguments as a list (which has to have elements of the same type in RPython). Instead of this, internally the jitffi module fishes elements from the passed list (this is a wrapped list from app-level) and pushes them directly to the calling stack.

I have added a simple cache for compiled loops. It's a bit primitive but it works. Simple, once the loop is compiled it has been stored in the cache and then before the whole compiling operation, we check if there is already compiled code (if the result type and arguments are the same).

It still needs some extra work, e.g. once we pushed something to the cache, it has never been removed from. So the cache can grow infinitely. I will try to add some cleaning operation later. It's not really important for now.

And this week..

I will continue my work to have C pointers working as arguments for calling functions.

That's all.

Have a nice weekend!

bart.

What have I done...

2010-07-05T10:18:00.001+02:00

...sa far :)

As you see I'm not really into blogging however I would like to share about the progress with PyPy - fast ctypes project.

The code lives in a branch at codespeak and you can simple take a look at:

https://codespeak.net/viewvc/pypy/branch/fast-ctypes/

or even better check it out:

svn co http://codespeak.net/svn/pypy/branch/fast-ctypes

What is already done?

I guess not too much :-) I've started a new Foreign Function Interface called jitffi which you can find in pypy/rlib/rjitffi.py. For object space go to pypy/module/jitffi/. It uses the PyPy JIT to call functions.

With this module we can simply load dynamic C libraries and call functions from the loaded library. There are some limitations such as all function arguments have to be the same type. Other limitation is that jitffi supports only int and float types (and void obviously) so far.

Here is a sample to see how we can use it:

import jitffi
lib = jitffi.CDLL('library_name.so')

func = lib.get('func_name1', ['int', 'int'], 'int')
func.call([1,2]) # return int

func = lib.get('func_name2', ['int'])
func.call([10]) # return None (as it's equivalent to C void)

This API is completely unstable and changes are made from time to time, or rather from commit to commit :-) The best way is to check appropriate tests to see the current API and how to use it.

What am I doing now?

The next step is to drop the limitation of same type of passing arguments. The problem is that the RPython lists can store only elements of the same type in one instance.

The idea is to fish the passed arguments from app-level and push it onto the stack. I'm working on it now.

Next? The PyPy JIT supports ints, floats and pointers. So, I will go to give a support for the latter.

This is it. The plan for the next week.

Google Summer of Code 2010 - fact ctypes

2010-05-26T23:10:00.004+02:00

So, this is my third time with GSoC program. Last year I was chosen by Umit Project and PyPy (through Python Software Foundation) and I had to choose where to go. I decided to go with Umit Project and finish my project which was born during GSoC 2007 edition. The main reason was that I needed it to my Master Thesis.

So, this year I gave a try with PyPy again and fortunately I was chosen again! Thanks guys!

And here is my project proposal for this year...

Project Name

PyPy - fast ctypes

Introduction

The PyPy project aims at producing a flexible and fast Python implementation. The guiding idea is to translate a Python-level description of the Python language itself to lower level languages.

PyPy project's main part is not Python interpreter implementation, but its configurable translator. It provides a good way to avoid writing interpreter for every language and platform. It provides tool-chain for translating front-end dynamic language into one of the back-ends (e.g. C, CLI).

Since 1.2 release[1], PyPy has provided Just-In-Time compiler (actually it's a 5th generation, also called pyjitpl5)[2]. Currently the speed up for Python applications is massive[3], however it doesn't mean that the PyPy has reached the limits. One of the way to give a boost is to speed up ctypes[4]. And this is a main goal of the proposal.

Project Details

Current implementation of pypy/lib/_ctypes uses libffi[5] library through _rawffi module. The idea is to give a chance to JIT in supporting ctypes. This can be reached by rewriting ctypes in RPython and avoiding to use libffi.

The rewriting process ain't that simple due to restrictions in RPython language[6]. One of the obvious issues which has to be solved in the project is dynamic amount of arguments in calling functions. Python allows to use dynamic amount of arguments, so does JIT. Unfortunately, there is RPython in the middle which is enough restricted and static to forbid this. The issue to solve is how to transport dynamic amount of arguments through RPython.

In fact that the goal is to speed up ctypes especially by using JIT, special attention is needed during the porting process. From the beginning, RPython's ctypes should be written with the JIT in mind. The JIT should be able to trace ctypes since first incarnation of the new ctypes. This fact will impact on some implementation details.

References

[1] http://codespeak.net/pypy/trunk/pypy/doc/release-1.2.0.html

[2] http://codespeak.net/pypy/trunk/pypy/doc/jit/pyjitpl5.html

[3] http://speed.pypy.org

[4] http://python.net/crew/theller/ctypes/

[5] http://sourceware.org/libffi/

[6] http://codespeak.net/pypy/dist/pypy/doc/coding-guide.html#restricted-python

The Blog

2010-05-26T19:32:00.000+02:00

Seems like I was more or less forced to start a technical blog. So, here I am :)

Mostly, I will use this blog for status updates of my GSoC project. But we will see what future will bring :-)

Enjoy hacking!