Overview

My mentor asked me to write about what we’ve done so far, how we got here, and where we’re going.

Most of the story of what’s happened from May 24 to now, is chronicled in this blog, however I’m going to make an attempt to summarize it a little to make a nice, digestible byte for you to consume.

Past

To start, my school didn’t end until May 28th, and so I didn’t get anything done during my final’s week. Then, the next week or so my family from Nebraska and San Diego came to stay at my house for my younger brother’s celebrations. That is, my little brother achieved the rank of Eagle Scout in Boy Scouts, which is the highest rank, and he had also graduated highschool. Both celebrations were in the span of one hectic weekend, so, though I did shut myself in my room to get work done, my work really began after everyone left.

On the technical side of things, I’d been working on getting NumPy working on top of PyPy. Unfortunately, CPyExt, PyPy’s CPython compatibility layer is not as mature or complete as we had hoped, and so I spent the better part of June hammering on it and NumPy. My work on NumPy/CPyExt has been very tough. Implementing mostly wrapper functions in CPyExt for the sake of NumPy was the easiest part, but there are bugs lurking in CPyExt, and subtle incompatibilities with the CPython interface. I have a few functions sitting in my working copy that I want to commit to PyPy, but won’t until I take another look at them.

On top of this, NumPy doesn’t always play within the API and will go around touching your structs in their private members. I tried to eliminate NumPy’s bad touch, and those changes, (most heinous of which was in PyArray_Scalar()) are in my github repository for NumPy.

Present

In an attempt to make up for lost time, I’ve been doing a lot of work lately. I quickly (re)implemented micronumpy and it passes 50% of its tests. It does seem to have a somewhat obscure bug where it segfaults if you allocate (and subsequently collect?) too many micronumpy arrays. I haven’t had a chance to look into this just yet, but that’s my #1 bug to squash right now. Thanks to the awesome work from the PyPy guys with the JIT generator, my incredibly naive implementation of micronumpy arrays is already twice as fast as normal NumPy on CPython for the convolve benchmark. This is part of the beauty of PyPy, I didn’t do any JIT specific work, and the JIT has yielded a significant improvement. There are hints that I can and will give the JIT, to hopefully further improve performance. (After all, we want to beat Cython (well, that’d be nice, I’ll be happy to be within 20% or so))

A little about Subversion. I attempted to merge trunk back into the micronumpy branch. Mind you, not one file has a conflicting edit. The merge resulted in a series of conflicts. Subsequently, I tried to commit my latest changes including the convolve benchmark I used, and svn would not co-operate. Eventually I got a bit of help on #svn on irc.freenode.net and started svn reverting, rming, and svn updateing, and after around three hours, I finally was able to commit again. During the whole process I ended up writing a few bash one-liners and brainlessly svn reverted my working copy of micronumpy. Luckily I still had vim open with the latest changes that I could write back out, but that had me scared…

Future

I’ve been developing some tools to make debugging CPyExt easier, so I’m very hopeful that I’ll have NumPy working on PyPy. The first is a little indented printer, which allows you to push/pop levels of indentation. I implemented it as a decorator which outputs function arguments, so I’m able to somewhat usefully, see the CPyExt call stack. Right now I’m interested in implementing a more useful printer for lltype.Struct, which currently only tells us it’s a struct, and what its fields are. I’d like to make it certain key fields from the struct as name/value pairs.

On top of this, micronumpy is coming together nicely, I still have to implement some of the tougher array access methods (multi-dimensional slices for instance) and those will be coming, hopefully I can scavenge them from the original micronumpy. My first priority, for micronumpy is to sort out this segfault that keeps occuring.

Closing

So that’s what has been, what is, and what will be. I hope someone out there is finding this interesting, and eventually people will find my work useful. Also writing this post has made me think more closely about how much time I’ve put into this summer of code. For a while I was feeling that maybe I wasn’t working enough, or hard enough, but writing this, and realizing how much time I’ve actually had to put in, puts it in perspective, and I feel much better.

Cheers,

Dan
Advertisements
Overview

Unofficial Results

So, in order to make up for lost time, I’ve been coding a significant amount the past 48 hours, near non-stop. My labor has finally bore fruit.

Micronumpy

Micronumpy found at https://codespeak.net/svn/pypy/branch (Click to browse the source, copy the link name for the checkout address, full instructions here) has been rewritten to use lltype.Array and now supports enough to run the convolve benchmark found in pypy/tool/numpybenchmark.py which is a slightly modified version of this. Unfortunately, pypy segfaults if the test is run more than 20 times by timeit. However, with 20 repetitions, a 200 by 200 image, and a 3×3 kernel, PyPy handily beats CPython and plain NumPy.

Command Average Time per run (Seconds)
./pypy-c ~/Projects/micronumpy/pypy/tool/numpybench.py 200 200 3 3 0.38495
python ~/Projects/micronumpy/pypy/tool/numpybench.py 200 200 3 3 0.85705

That’s an improvement of about 55% shorter runtime!

That’s not bad for a first iteration (Even if the iteration’s been a while in the works). Now note that the benchmark as it is right now does not provide for a warmup period, which should improve PyPy’s score even more. Whether or not a warmup period is more or less of a measurement of what we care about, I’m not sure. Also note that, while the times have remained stable, they are likely susceptible to my other processes on this computer. I plan on writing a follow up article which will hopefully have more interesting results, and maybe even a graph!

NumPy on PyPy

Though it’s taken far longer than any of us expected(Except maybe Maciej). I’m very hopeful that in the next week or so I can have NumPy running in a somewhat stable fashion on PyPy. Currently I’m hacking on CPyExt/lltype to give more useful str() values, rather than <* <Array of Char> at 0xDEADBEEF> I hope it will look more like <* <Char array = "spam"> at 0xDEADBEEF> which I think will be a useful debugging tool for everyone.

More on everything after the midterm reviews ūüôā

Cheers,
Dan
Unofficial Results

A Bit Behind Schedule

Life seems intent on limiting the time I can spend on my GSoC project. The first wave of family arrived at my house today, and more still will be coming. I have a lot to do to help my family host. Last week we set the goal that I should have NumPy working on PyPy by the end of last week. (which we’ll call Tuesday since that’s when we set those goals forward) In addition I was intended to have started fixing up micronumpy.

While I haven’t started on micronumpy, I have made significant strides in getting NumPy to work on PyPy. Unfortunately, CPyExt isn’t as mature as we’d hoped, and rather than working on NumPy, I’m spending my time implementing C API functions in PyPy. NumPy still doesn’t import, but I can see from the output of nm -u the number of missing symbols is being slowly whittled down. I should also note that NumPy’s setup.py has caused me significant grief since its ‘clean’ target doesn’t clean up after setup.py build_ext -i at the moment I haven’t identified whether that’s a bug in NumPy’s copy of distutils (gross in and of itself) or if it’s a nasty interaction between it and CPyExt’s presetup.py which creates a stub dll/so for testing purposes.

Now on to the positive side of things, since I began writing this post (about a week ago) I’ve satisfied the last of NumPy’s Symbol dependencies it seems, now there are still issues with importing the module, for instance some API functions get passed non-Python objects (probably being double freed) but I’m not sure how best to track this all down. I’m not sure whether I just haven’t found the source, or if it has to do with output redirection and buffering, but no matter where i’ve put printfs (in c) and prints (in python) they always end up after the exception is thrown in the output…

EDIT: It’s almost certainly turned out to be a buffering/stderr vs. stdout issue, how I’m going to get Python buffering to play nice with C is a mystery to me though… Perhaps if I turn off buffering…

So, sorry for the absence, things are still moving along, though I definitely have a bit of catching up to do.

Cheers,
Dan
A Bit Behind Schedule

Planning

To start, there was a question where my progress could be followed. There are several, unique projects which affect my ultimate goal, and each one is going to be hosted accordingly. There’s nothing interesting there right now but a woefully out of sync trunk and some basic array code (hey, it works though). but modifications to PyPy will live at codespeak.net http://codespeak.net/svn/pypy/branch/micronumpy is the url, to browse the source you can navigate a browser to http://codespeak.net/viewvc/pypy/branch/micronumpy. My modifications to NumPy will live here though the trivial changes are already in trunk (Changeset #8448 and Changeset #8418). I will have a few changes for Cython by the time the summer is through, but for the moment I may mostly ignore the mtrand.c file which has alot of problems for non-CPython implementations.

As far as plans go, it’s been decided that this week I should finish up my NumPy compatibility work as fast as possible, I am quite close, though I noticed I had some other hackish things I did to get it to build, which I need to fix. After that it will be time to port micronumpy to lltype arrays, which are PyPy’s way of creating raw arrays/buffers. Without this, the lists used in the current implementation can be moved by the garbage collector (a very useful thing). However, normal C code does not have any method for dealing with moving objects, and so in order to interact with NumPy’s C code, we need to migrate to lltype arrays.

Planning

CPyExt Tie-Ins

In order to get NumPy to build against PyPy I’ve had to make numerous changes to the CPyExt module, many of them ugly.

It’s also become apparent that Cython and NumPy both depend directly on CPython, and as such they touch structures that they shouldn’t (as dictated by the C API) and will have to be modified. Unfortunately, not everything NumPy does appears possible through the C API and therefore extensions to the C API may be necessary, I’ll need further guidance on this, but I think it’s likely that I will add PyPy*() functions which may then be proposed for CPython (at which point they’d be renamed to Py*()). The only problem with that is the possibility that these functions would become relied upon with their PyPy*() name, which would create another problem. Of course, that’s still the right way to go, as it’s not guaranteed that any extensions to the API will be accepted by the CPython folks, especially not in their originally proposed forms.

I have to apologize, this past week was my finals week, so I didn’t really accomplish much during the first week of the GSoC, but starting today I’m getting going.

Cheers,
Dan

CPyExt Tie-Ins

Google Summer of Code!

I’m extremely happy to say that my Google Summer of Code project proposal was approved! ¬†That means I get to spend my summer working on my favorite project, PyPy, and combine it with NumPy. ¬†Hopefully the result will be an extra efficient NumPy implementation on PyPy.

This morning I met with Maciej FijaŇākowski and St√©fan van der Walt, my unofficial mentor, representing PyPy, and my official mentor, representing NumPy respectively. ¬†Since exposing my proposal to the NumPy mailing list, it became apparent that my project, in order to best serve the existing NumPy users, would need to be approached differently. ¬†It seems that everyone uses their obscure corners of the library, and it would be tough to address only the most used parts and still make anyone happy.

Because of these needs I couldn’t really just hack away on a small subset of NumPy and have my project be useful at all. Originally it was proposed that we try and replace NumPy in a piecemeal approach with RPython code, however there were numerous technical problems and practical problems with this approach. (Such as the JIT not being so useful when there’s C code, and the extra overhead of trying to keep my code “plugged into” NumPy’s existing code) St√©fan came up with a sane approach, however. ¬†We will first port NumPy to PyPy by way of CPyExt (a growing PyPy sub-project allowing CPython extensions to be compiled to work with PyPy, described a bit here). ¬†This way, we will be completely compatible with NumPy on CPython. ¬†This allows us to do whatever we please with micronumpy. ¬†The idea as it is today is to provide both side-by side, and allow converting between the two via micronumpy.asarray() and numpy.asarray() or something of that sort. ¬†micronumpy arrays will likely be lacking in features, but blazingly fast by comparison, and NumPy arrays will be mostly identical in speed, and as CPyExt matures, I expect that speed to converge to CPython speeds). ¬†In the long term it would be great to make micronumpy arrays completely compatible with NumPy arrays. I plan to re-use as much of NumPy as I can for micronumpy arrays.

To Do

The question remains, then. ¬†How should I move forward from today? ¬†Well, I’m going to stay on top of my school work, and then on the 16th I’ll be able to really start working on this. ¬†Starting on the 16th this is what I need to accomplish.

  1. Clean up the existing micronumpy code. ¬†I worked on it during school and the result is functional, mostly, but it’s quite ugly, and is less than ideal in many ways.
  2. Eliminate  trivial errors with building NumPy with CPyExt.  Stéfan created a bug here.
  3. Port existing micronumpy code to low level RPython arrays, maybe should be done during the cleanup.
  4. Add __array_interface__ to micronumpy arrays to facilitate interoperability.
Google Summer of Code!