Myself and another Freenet volunteer will be representing the Freenet Project at the Decentralized Web Summit next week. We will be giving at least one Freenet-related lightning talk, potentially more. I encourage everyone interested in Freenet, decentralization, and internet freedom to check out the decentralized web summit being held next week 2016-06-08, 2016-06-09 in San Francisco at the Internet Archive Offices. While the even Wednesday costs money, it’s my understanding that the meetup on Thursday when we’ll be giving our talk(s?) will be free admission. For anyone who cannot physically be present, I’m told there will be a live stream. Check it all out here: decentralizedweb.net
Congratulations to the OpenBazaar team for their hard work and on their recent release! I firmly believe that global, decentralized, anonymous marketplaces will liberate millions. Whether it loosens the grip of a despotic state, or breaks the monopoly of a private company, OpenBazaar will help many, many people. For this reason I consider OpenBazaar’s work incredibly important, and therefore important to implement correctly. Unfortunately, the first production OpenBazaar release is conspicuously missing anonymity. Lack of anonymity severely hampers OpenBazaar’s ability to do good. Due to lack of information, I was under the mistaken impression that OpenBazaar stopped prioritizing anonymity. Recently, however, I was happy to read their recent blog post. The post enumerates five key limitations of OpenBazaar’s initial release. They address my two key concerns and three other important concerns I had not considered. Most importantly for me, they acknowledge the importance of anonymity for OpenBazaar. While I was overall encouraged, I was a bit disappointed to see IPFS favored over a project I’ve recently become passionate about: Freenet. Freenet has been battle-tested for 15 years and directly addresses three of the five limitations of OpenBazaar, and indirectly supports a robust solution for a fourth. On the other hand, IPFS is young and only addresses one of the five issues. Though IPFS is experimenting with integrating third party anonymization solutions, that only addresses two of five. Even if/when these changes are completed, I believe Freenet is a superior solution even for the two. These two are anonymity and offline storage, something Freenet is designed to do from the ground up. Freenet allows users to anonymously publish and receive pieces of static data, which fits well with OpenBazaar’s model.
The OpenBazaar post correctly recognizes that many users may not want to run OpenBazaar’s server at all times. Currently, if a user turns off their OpenBazaar server their store becomes inaccessible. This problem is doubly important because I want OpenBazaar to help people worldwide, which must include people without reliable internet access. Freenet directly solves this by not storing data on the computer which publishes it. Using Freenet’s basic data insertion, OpenBazaar users could anonymously publish contracts which could be accessed even after the user went offline. When data is inserted into Freenet, it is stored on other computers according to a derived routing key. When that data is accessed, it becomes replicated across additional computers in accordance with its popularity. For this reason, data on Freenet cannot be DDoSed by repeatedly querying it. Contrast this with OpenBazaar’s current model, where identifying the correct computer to DDoS is facilitated by the network, and the data is not replicated elsewhere. IPFS does satisfactorily solve this issue, but it is still a relatively young project.
On the other hand, Freenet simultaneously solves multiple problems today, with over a decade of testing.
As I alluded to in the previous section, on Freenet, computers and IP addresses have nothing to do with the content they store, anonymity is built in. Freenet further improves anonymity by dividing and encrypting all data, limiting knowledge that other nodes may have about the data you are inserting or requesting. There exist (statistical) caveats to this, Freenet’s default behavior obfuscates data insertion and access quite effectively. More importantly, Freenet provides a mechanism for gaining stronger anonymity by only connecting to trusted friend nodes. By doing this, you further increase the difficulty of analyzing your traffic. The larger your network of trusted friends is, the better you are protected. In this way, privacy can scale with one’s desire for increased privacy. Freenet is also exploring a tunneling concept to provide yet another layer of protection. All said, Freenet provides adequate anonymity by default, and provides the ability to improve anonymity as needed by adding trusted friends. IPFS on the other hand does not support anonymity natively, and any support will be experimental for the foreseeable future. Further, Freenet is designed to simultaneously support anonymity and offline caching, and the two properties compliment each other. This is opposed to the way IPFS would interact with privacy layered on top. Consider that Torrenting via Tor is discouraged due to traffic requirements. Instead, popularity of a piece of data on Freenet makes its retrieval more efficient as it is replicated in more places.
Freenet includes a third feature OpenBazaar requires in the form of an official plugin developed directly by the Freenet Project Inc. WebOfTrust is a plugin for Freenet that provides spam-resistant data publishing on the Freenet network. It accomplishes this by defining the concepts of identities, and trust between identities. An identity (optionally) lists other identities which it trusts or distrusts, and by how much. By considering all known identities, this produces a directed, weighted graph with identities as vertices, and trust relationships as edges. By traversing this graph from one identity to another, WebOfTrust calculates how much that identity trusts another. OpenBazaar can utilize this framework by setting trust based on their seller reviews. If Alice is scammed by Eve, and Alice writes a negative review, Alice’s WebOfTrust identity would indicate distrust to Eve’s identity. If Bob has positive or neutral trust to Alice, Bob would then transitively distrust Eve. Each trust relationship also may have arbitrary data attached to it, allowing a proof to be included. Furthermore, if Bob determines that Alice’s bad review of Eve is fraudulent (lacking a proof), Bob may choose to lower his trust of Alice in response. Those who trust Bob would then have lower trust of Alice, and potentially higher trust of Eve.
Finally, Search can also be facilitated by the above Reputation algorithm. If a user is well trusted, and they claim a contract belongs to a particular keyword, that will be considered “correct”. If a user is not well trusted, the keyword may be ignored when searching. Conflicting keywords would be resolved by some weighted summing and a threshold value, and spamming keywords could be discouraged by automatically distrusting identities which do this (such spam would be identified manually by the user). In fact, prevention of spam like this is the core purpose of Freenet’s WebOfTrust!
Because Freenet adeptly addresses three of five significant concerns of OpenBazaar, I argue that the OpenBazaar team ought to carefully evaluate Freenet as a potential backend platform for OpenBazaar. I have a decent working knowledge of Freenet and am more than happy to lend my efforts to the OpenBazaar project (though there are far more knowledgable individuals in the Freenet project). I did not seriously pursue this earlier because I was under the impression OpenBazaar was de-prioritizing anonymity, but I am happy to find that I was very mistaken. This document will undergo further revisions, but I am going to publish it as soon as possible.
I intend to follow this post with a more concrete proposal for a Freenet backend. I also welcome any questions, comments, or corrections for this article.
P.S. Thank you to the redditors that brought the OpenBazaar blog post to my attention, and encouraged me to publish my thoughts. (And thank you to the OpenBazaar team for making the world a better place)
The original OpenBazaar blog post: Current Limitations of the OpenBazaar Software
It’s been an absurdly long time since the last time I updated this blog. For the moment life has allowed me some free time, some of which I want to devote to blogging. Since my last blog post almost three years ago I fell down the rabbit hole of 3D Printing, Electronic Engineering, and Robotics, all of which I want to share. With the remainder of my free time I’ll be working on software, hopefully some of which I can contribute to the world. Whenever I do that I intend to provide writeups here for documentation and entertainment purposes. It’s good to be back.
A little about my life, just for a second, I’ve come down with some sort of sickness, which sucks. And I’ll be gone the 23rd through the 25th on a camping trip
On a more technical note, I was struggling a little bit with NumPy’s handling of Fortran ordered arrays conceptually, so my stride_column code didn’t seem to work right, but I think I’ve fixed it now. It looks like the “right” thing to do, however, is going to be to precompute the strides of an array at creation time, that should provide a pretty significant speed boost for multi-dimensional indexing. (NumPy has always been doing this, I haven’t)
I want to keep pushing on NumPy on PyPy, but I haven’t made any progress since my last blog post, that’s a very important part of my GSoC project though, so it must get done.
I’ll be updating this post later, for now I want to get some work done.
Don’t be alarmed, it’s the same blog as before.
Subversion hates me.
As evidenced by my previous blog posts, I’m not particularly happy with my own progress. Hopefully my recent progress will be sufficient to appease the Google gods. (Maybe with some benchmarks too)
I can report that I’ve completed quite a bit since my last blog post, for instance I implemented a toy micronumpy array implementation using
lltype.Array and have now grown it into a “full” implementation. I hope that my approach will work well with the JIT, simple element lookups are a couple of dynamic dispatches away (which is a little less than ideal in my opinion) but it should JIT well since the JIT will remove the overhead of those calls.
Unfortunately, this implementation isn’t re-using much of the original code, considering that the core storage of the array is being altered, not a whole lot of the original code is proving useful, but I will be attempting to re-use as much as possible (if for no other reason than not having to reconsider all of the math). I currently have one of our original tests passing now, which is better than nothing, but not particularly satisfying…
I’ve had to deal with alot of bugs in PyPy’s CPyExt CPython extension compatibility layer, which is certainly what slowed progress the most. The time spent on that was almost entirely unexpected, and represents the biggest setback (other than family obligations…)
Trying to keep this short so that I can get back to work, I want to have some benchmarks before evaluations are over to justify my existence 🙂
To start, there was a question where my progress could be followed. There are several, unique projects which affect my ultimate goal, and each one is going to be hosted accordingly. There’s nothing interesting there right now but a woefully out of sync trunk and some basic array code (hey, it works though). but modifications to PyPy will live at codespeak.net
http://codespeak.net/svn/pypy/branch/micronumpy is the url, to browse the source you can navigate a browser to http://codespeak.net/viewvc/pypy/branch/micronumpy. My modifications to NumPy will live here though the trivial changes are already in trunk (Changeset #8448 and Changeset #8418). I will have a few changes for Cython by the time the summer is through, but for the moment I may mostly ignore the mtrand.c file which has alot of problems for non-CPython implementations.
As far as plans go, it’s been decided that this week I should finish up my NumPy compatibility work as fast as possible, I am quite close, though I noticed I had some other hackish things I did to get it to build, which I need to fix. After that it will be time to port micronumpy to lltype arrays, which are PyPy’s way of creating raw arrays/buffers. Without this, the lists used in the current implementation can be moved by the garbage collector (a very useful thing). However, normal C code does not have any method for dealing with moving objects, and so in order to interact with NumPy’s C code, we need to migrate to lltype arrays.