Toad's blog: Older stuff is largely Freenet-related; since then I've moved on. Currently there are no comments or tags, I will move this to a proper blog at some point. Note that this may cover a wide range of topics! This is static content because historically it was mirrored to Freenet.
Some links on the NHSX Covid-19 appTags: covid, android, privacy, decentralisation
Since leaving Freenet, I went to get a degree at Cambridge, then got a job with StarLeaf, who make awesome video conferencing solutions. I'm in the happy position of having a job and being able to work from home... and in fact classed as a key worker, though that's highly dubious, and anyway I can work from home.
I've been vaguely watching the situation with the NHSX app. The privacy issues are interesting, but well-covered elsewhere, and there are some technical issues, that I may or may not have run into in other contexts.
The source code and some of the development docs have been published:
One interesting technical question is how they translate signal strength into proximity with different devices. Bluetooth Low Energy signal strengths vary enormously from one phone to another, some of them just have better aerials than others. So the measured RSSI, the "signal strength", does not necessarily tell you the distance between two phones. Some early papers just glossed over this - one of them was based in a university where everyone seemed to have exactly the same university-supplied phone. It looks like the NHS centralised approach does solve this:
This allows the server to run some stats to try to build a database of signal strength by device, so they can try to compensate for that. Apparently they did test this, though what exactly the results were was not given in the docs - probably just an indication that they need such a database.
It is not clear how Google/Apple's decentralised approach solves this particular problem. They give the individual health authorities the measured signal strength, do they include the device model for both users' phones, or have they built their own signal strength database? Digging so far does not include this information. If they don't, expect a lot more false positives or false negatives... meaning people being told to isolate when they're not at risk, or missing out on people who don't need to isolate. Even with such a database, we have to assume that radio makes this distinctly fuzzy - phone in back pocket vs in hand etc. But the main reason for false negatives is getting enough people to use the app, which seems unlikely at this stage...
How does that affect the centralised vs decentralised model question? In countries which publish an app using the standard APIs, it will likely be used much more widely; the key components are pushed automatically the app stores, although you still have to install the per-country app to get notified and send in a report of symptoms.
I agree with the standard criticisms; it's basically impossible to fully anonymise this sort of data. The social graph can be mapped to other social graphs. All we can do is ask for legislative guarantees that it won't be sold to marketers. Which does matter - some of them are trying to sell you things like Boris Johnson. The tinfoil hat brigade can go home though - Google already have your location history. So we'll see. For more analysis see the mainstream media.
In any case, platform limitations are a real problem - even in an Android-only world, foreground notifications can still get killed if the user is using other services, and using a notification to restart the app is really very naughty and not going to work well on Android 9 onwards. On iOS it's much worse. So the NHS will probably have to switch to using Google's and Apple's systems. Interesting example of how US political realities determine the rest of the world's fate. I hope they have a good answer for radio accuracy issues. Also, it illustrates the general principle that centralised implementations are usually easier - but they create a different set of problems.
Any advice here? Nothing you haven't heard already. Follow the government advice. Stay home, wash your hands, self isolate if you have symptoms etc. I'll probably install the app on one of my phones, though I tend to have a cough most of the time anyway so I'm very cautious. We'll probably get better privacy with the Google/Apple version, but the NHSX one could work - if we have the right guarantees from the politicians. Google and Apple already have loads of data about you anyway, and so does the government. And the NHS takes privacy pretty seriously on the whole - it's the politicians who periodically try to sell our data. The app likely won't be entirely accurate: some people won't be notified even though they were close to somebody infected, and some people will be told to self-isolate when they don't need to. Which is why there's also going to be manual contact tracing. Is it enough to lift the lockdown? That depends on the data - how good it is in practice. Thankfully not my call.
Meanwhile direct your paranoia in more productive directions! Now is the perfect time to hide bad news, and the recovery is the perfect time to bail out obsolete, destructive, unsustainable and long-term unprofitable fossil fuel dinosaurs. Government via video isn't really happening yet, but it'll be interesting to see; the problems with electronic voting aren't quite the same for legislators as they are for public elections, since your MP's votes are public knowledge.
Take it seriously. Stay safe, and avoid spreading the disease if you can: it's not just your safey that matters. Some of us are going to get through this.
A straightforward rant
I don't like to pick fights with old people doing what they believe is right. But remember that most people who die in modern wars are civilians. And many of those wars are blatent acts of illegal aggression. For example, in Iraq:
Both wars were the direct result of colossal incompetence and imperial warmongering on the part of politicians, and at least in WWI the senior military too. And WWI led directly to WWII; the only reason there wasn't a third was that on August 6th 1945 we learned that we really can't afford one.
For context, 4.6 million people die every year from air pollution (indoor and outdoor), which is largely preventable. Numbers aren't everything. But while I support the existence of the armed forces and believe the government should look after them and their families (including foreign contractors), I won't be wearing a red poppy.
Part 2 project
My third year project was on improving the performance and reliability of Freenet simulations. See my dissertation for documentation, and the branch fast-simulator-insert-on-multiple-nodes on my repository (pull request or source code).
I have finally finished my degree. I may or may not do a Master's. If I do, my project will probably be related to CHERI or LowRISC, i.e. hardware assisted security (128-bit compressed fat pointers/capabilities are looking very promising). I spent last summer hacking LLVM to try to prevent control flow hijacking using tagged memory on LowRISC; see the details here. Ultimately it was a proof of concept and that approach (tagging code pointers etc) has some serious limitations while it may be quite costly, so I'm leaning towards CHERI (capabilities) being the better answer, which provides full memory safety and wipes out several huge classes of remote code execution attacks. Even so, LowRISC is very cool: it's an effort to build, and ship, open source System-on-Chip's using the RISC-V ISA. Current (Rocket) cores are relatively low-end but should include some interesting features such as minion cores and tagged memory. And there will be actual silicon!
I still think computer security is hopeless in practice for the next 20 years or so, but building more secure systems is worthwhile, even if politically and economically they cannot be deployed until society is ready for a grown-up debate over surveillance, covert versus overt backdoors, and the apparent need to be able to hack everyone all at once. Not to mention the fact that the business model of most of the Internet relies on nobody having any privacy. But if there is no computer security there certainly won't be any privacy: it's a necessary but not sufficient condition. Right now there is no security and no privacy, and certainly no anonymity. Assuming that your threat model includes people like agencies - and in many important cases it should.
At the moment I should probably not be involved in Freenet, or any other open source project, even in my own free time, because under the terms of my current employment it would potentially compromise any project I am involved in. So that's that. Nextgens and Steve seem to be making good progress, and the world has changed considerably since I first started working on Freenet. Have fun folks. Please do not contact me about any technical issues in Freenet that might conceivably be patentable, for the time being, and note that more or less every useful idea is patentable!
As for the rest, the world is going to hell, but I will survive. For now anyway.
Final release candidate for purge-db4o
The snapshot below is feature complete, and reasonably well tested. Hopefully it will be merged and deployed soon. It certainly needs more testing. See the post a few pages down for explanation of exactly what this does: The client layer is rewritten, downloads and uploads are stored in a new way, hopefully resulting in much more reliability and less disk I/O. Also lots of bugs fixed.
The latest version is here (github).
Signed tag "purge-db4o-snapshot-28"
Commit hash d7ad6fccb89e8f36b3711c3fdb82b8ec3f92fc50
You will need to download the new jar (save it as freenet.jar), and the bcprov library, and edit wrapper.conf to point to them. See below for more detailed instructions.
Also, you should backup your master.keys, node.db4o[.crypt] and persistent-temp* before trying the new code. It should be fairly stable now and there shouldn't be any more format changes. Previous snapshots may not be compatible with this snapshot however, so you might lose your downloads/uploads if you've been testing previous snapshots. If you want to switch back to mainline fred and keep your old downloads you will need to restore from backups, hence the advice to backup before trying it out (and it may take a while for the code to be merged given its size, so you probably will want to switch back to mainline fred at some point).
This hasn't been all I've done on Freenet this summer; I have also contributed to the link length fix, which seems to be making a big difference to data persistence and performance (thanks mainly to the volunteers involved and especially the authors of the paper on opennet performance!), and posted a couple of implementations of old load management ideas that need to be tested. Also the purge-db4o-crypto code above includes lots and lots of bug fixes, mostly to the client layer, though no doubt it introduces new bugs too.
Thanks! I will be on holiday for a few days and then back to university, so I won't be working for Freenet in a paid capacity, although I will likely respond to reasonable queries. I may come back to Freenet next summer, depending on what other opportunities arise. Have a great year!
Another release candidate for purge-db4o
The client layer rewrite aka purge-db4o is finished. That is, it is feature complete, and has had a reasonable amount of testing (but needs more).
The latest version is here (github).
Signed tag "purge-db4o-snapshot-24"
Commit hash bc5ce630c68db657bfcc726e7591d9ec9e88ae53
Bouncycastle crypto library (needed if using disk encryption i.e. physical security level of LOW or higher).
Please test, and please consider for integration. Thanks!
I will do some more testing and might do better password encryption, but I will have to stop working on Freenet within the next few days because of uni... I will answer any reasonable queries by email even then.
It will automatically migrate from node.db4o*, and hopefully we won't need to make further incompatible changes, but it's possible e.g. if operhiem1 doesn't like the class names. Also there may be more bugs, including in migration. So ...
NOTE: This is not compatible with previous snapshots, so you may have to restore the backups of your node.db4o* etc and then re-migrate if you have been testing previously.
NOTE: It *is* compatible with disk encryption (physical security level above NONE, "encrypt temp buckets" / "encrypt persistent temp buckets" enabled in advanced config). Please test this, this is the newest riskiest bit!
Plugins have not been updated. There are updated versions available but they may need further changes after the refactoring in snapshot 19.
Enjoy! And please let me know if you find any bugs in the next few days.
New readers: Read the previous post to see what purge-db4o is and what this is all about...
Please test the latest snapshot of purge-db4o. This will automatically migrate your old downloads and uploads, but you should back them up first (copy master.keys, node.db4o* and persistent-temp*). The only thing it doesn't support is disk encryption, and there is some dispute over whether we want to keep that feature. It includes a new metadata type, COMPAT_1465, but it defaults to the old COMPAT_1416.
There has been a lot of discussion on Frost, amid the usual racist slurs and other nonsense, about why purge-db4o needs a new metadata type. The answer is that in the process of rewriting large parts of the client layer I discovered some bugs in how we use metadata for storing files ("splitfiles") on Freenet. The metadata that describes how to fetch a file includes some "top blocks" fields, so that we can determine accurately how big it is, how to reinsert it etc. One of these fields contains the "compatibility mode" for the file (e.g. COMPAT_1416). This is an overall, bug-for-bug-compatible version number which tells the node how exactly to do an insert: What kind of keys to use, exactly how to split up a file into segments, and so on. For example, a 129 block file should be split up into two segments of roughly equal size, rather than (as old versions of Freenet did) a segment of 128 blocks and then a segment of 1 block; but if we are reinserting an old file, we want to produce the same key, so we should divide it up in the old way. Another of the "top block" metadata fields indicates whether compression was turned off when we uploaded the data. Hence it is possible to upload the same data and get exactly the same CHK, provided we use the same settings, which are included in the metadata. Unfortunately, we were not filling in the compatibility mode or the don't-compress flag (on smaller splitfiles). I fixed this, but it results in new keys. Since we have new keys, we need a new compatibility mode (COMPAT_1466), so that reinserts of files inserted with 1416 to 1465 give the same CHK. And then we run into another metadata bug: If the compatibility mode is something the node doesn't know about, it fails to fetch the data (even though it's only included for convenient reinserts; there are separate versions for stuff that would make it hard to read). I have fixed this in new code, but existing 1465 or earlier nodes will fail to fetch data inserted as COMPAT_1466; so 1466 (or whichever build includes purge-db4o) will by default insert data as COMPAT_1416. This is a temporary measure and will change in the following build.
None of this really matters, and the reason I've posted test builds is precisely to find such bugs by having a larger number of people testing the pre-release code. These are not even official snapshots; update.sh won't fetch them, you have to download the jars manually. (Although if Steve's update channels mechanism was working you might be able to use that in a similar situation in future). Release early, release often; and a lot of users (such as TheSeeker) will happily try the new code and find bugs in it, which can then be fixed before release. Also, note that auto-updates are inserted with COMPAT_1416, and that fixed compatibility mode is only changed when the auto-update key is updated, so updates will work in any case, although they will be tested before releasing a build with purge-db4o merged.
I am confident that people will continue to find things to whine about. But since there were legitimate questions to answer I have answered them. Also, purge-db4o will be well worth the wait; it solves a number of long-lived bugs as well as reducing disk I/O and greatly increasing reliability (at least for downloads) relative to the old situation where queues would regularly corrupt themselves for many users. Please test it if you can.
IMHO purge-db4o is working fairly well at this point, and hopefully it can be merged soon. It was originally the plan to include support for disk encryption ("physical security level" of NORMAL, HIGH or MAXIMUM). Lately there has been some discussion of whether this is a good idea; I am not sure about this. The unix philosophy is a reasonable argument when considering tools for geeks, but "usable security" is IMHO more important. Input welcome on this... I have posted a message explaining the major arguments as I see them. Unfortunately, "use TAILS" is not really a good answer for Freenet, because we want users to run Freenet in the background for long periods of time (this is essential for darknet to work well).
On 05/09/14 19:43, localghost at eOC4Zm8KjRpMFhNBp6DmI8K4URaq8bQZH45y0dLHEnI wrote:> creamsoda at 0vpcRHZV1ftyj4mJpZnuYaG8wpkNIvf3qa3b-LUcsZs wrote : >> >> TAILS is meant to be for short-lived sessions of minutes to hours right? That doesn't lend itself well to freenet which works better over a longer time. > > Ya, I kinda figured that.. just curious if anybody had done it. I was asking for a friend, as stated. I prefer to let Freenet run constantly, to help the network- and have not used TAILS, personally.. > This is exactly why "leave disk crypto to the operating system" isn't so obviously the right policy for Freenet. The arguments for not doing disk crypto: - We're likely to get it wrong. - If they download video files etc there will probably be leaks. (But it IS possible to limit this) - Memorising a good password is hard, and the users who are willing to do so may be the same group as the users who will install a secure linux distro just to run Freenet, or at least do full disk encryption on Windows (presumably using BitLocker?) - If we do disk crypto we need to turn on swap encryption. This is trivial on recent Windows but arguably not a good idea. The arguments for doing disk crypto: - We want people to run Freenet long-term. They usually won't install a new OS just to run Freenet, and they can't run it long-term from a livecd. This is one of the reasons we support Windows! - People who are willing to remember a long password may not have the technical know-how to set up a secure system. - People who install Freenet casually and then discover something interesting may not want to reinstall at that point. - Tempfiles and other evidence of your past browsing should not be recoverable even if you *have* the password. Most full disk crypto setups don't have this property, but some do (e.g. /tmp on encrypted swap). - We can do some interesting things in the long run such as combining a password with a local nonce and data fetched from the network to decrypt local data, giving "hidden identities" e.g. for WoT/FMS. - In practice physical attacks are likely to be more common than network attacks, in most environments. A long term solution might be to have a separate node (which doesn't store anything private), possibly on dedicated hardware, and client (which stores your downloads and runs from a livecd or whatever, and uses tunnels through the node). But we need a decision in the short run. I used to think that disk crypto in Freenet made sense. I'm half convinced by the "good passwords are hard" thing. operhiem1 (Steve) is in favour of removing it, as is nextgens and most of the old guard (unix philosophy etc). I'm not convinced one way or the other. I will defer; I'll be away very soon; but it's worth discussing again maybe... Technically keeping passworded disk crypto isn't particularly difficult: - Review and merge the crypto pull request. We need this anyway. - Integrate with purge-db4o. (Fairly easy) - Use a proper iterated password function. - Think about changing the UI e.g. ask for a password when first start a download? But what about client-cache? Removing it will mean decrypting people's data and posting a useralert. It slightly speeds up the schedule for purge-db4o.
Finally, I will be leaving soon. I will answer emails occasionally, but I am unlikely to have time to contribute much to Freenet in term time. It is likely that I will be back next summer. Being able to work consistently on a single project (sorting out the client layer) and have other people deal with routine maintenance and releasing (thanks Steve!), has been a good thing. As has helping out with the link length changes, which others did most of the work on, and seems to have been a major success. I have also pushed a couple of proposed changes to load management. These are small and will need testing but might have a big impact. ArneBab's recent simulations confirm quantitatively that reducing the proportion of rejects could have a big impact on effective routing.
Please try the new unofficial Freenet snapshot!
For most of the summer, and a fair amount of time last year, I've been working on fixing one of Freenet's most complicated architectural bugs. This concerns the client layer, the part of Freenet that deals with putting together files and freesites from blocks (as opposed to actually fetching blocks), and in particular the code supporting persistent downloads and uploads (long-term downloads on the Downloads page for example). The current client layer does far more disk I/O than it needs to, leaks information and space in node.db4o, and above all it is unreliable: Many users report losing their download queue regularly, especially on Java 1.7.
The old code stores the requests in a database, and stores blocks from the downloads in a big file called persistent-temp.blob. This has severe robustness problems, as if one byte is lost in the database file (which is huge), there is a tendency for the entire database to be lost, with the result that Freenet either fails to start up or loses all your downloads. And it also results in far more disk seeks, and fsync's, than are necessary, because the largest, most important data is actually very simple: For a big file download, we need to track which blocks we have downloaded, and where we've put them. The corresponding database structures are complicated and there is lots of overhead, and lots of unnecessary seeking related to transactions on structures that are actually simple, robust, and easily recovered from the other data stored. And finally, the blocks aren't stored together on disk, and we have to copy the data when we finish the download, even if it doesn't need to be decompressed or filtered.
Hence the new code keeps all the downloaded blocks, and the status information for each block etc, together in a single temporary file, which will be slightly larger than the actual download. The overall list of which files we are downloading and where to, as well as complicated but small details related to e.g. downloading parts of freesites, is kept in a new file "client.dat", using Java's built-in serialization, which is much smaller than the old "node.db4o".
The result is it should be much more reliable (corruption will only lose you one download at worst, but mostly we should be able to recover from it at least for downloads), cause much less disk I/O and thus slow down your computer, and wear out your hard disk, much less than the current code, and it also appears to be noticeably faster in some cases (but don't expect it to affect network performance). Note that inserts are slightly less robust than downloads, but then they should be over within a couple of weeks or they won't be much use.
The current prototype works for downloads, uploads and site uploads. It does not automatically migrate your old downloads, and it does clean up persistent-temp automatically for you, which means if you just start it without making a backup you will lose data. If you want to keep your download and upload queues you need to backup node.db4o* and persistent-temp* first.
Also, it doesn't support disk crypto ("physical security level" in freenet configuration). There is code to do this (part of a larger Summer of Code project by another student on simplifying our crypto) but it hasn't been fully reviewed and integrated yet. Whether we should actually provide password protection for the client cache and the downloads in Freenet is unclear; it's true that it provides some protection to users who install Freenet casually and then find a sensitive file they want to download (they can set a password at that point), but it's also true that remembering a strong password is pretty onerous, and the same people who are willing to do that are probably also willing to install full disk encryption. So we may get rid of this dubious feature.
Prototype new client layer: please test!
Now the important part ... the working prototype download! Download the jar file, shut down Freenet, backup node.db4o* and persistent-temp* and replace your freenet.jar (or freenet.jar.new, check wrapper.conf) with the jar, then start it back up. Your download/upload queues will be empty, so download / upload some files / sites, try stuff out, and let me know how it goes! I read the Freenet board on FMS, amongst other ways of contacting me ...
What else is going on?
Another important development this summer in Freenet: A paper (freenet) explaining that opennet has real routing problems, with a solution. This is something that we've suspected for a while, partly because nodes that download a lot tend to have very widely spread out peer locations, while for good routing we need most of our connections to be close to us in the keyspace (every node has a "location" between 0.0 and 1.0; we route requests for keys/blocks to the node nearest the key, converted to a location). We have deployed a slightly crude fix, forcing 70% of our opennet connections to be "near" links (less than 0.01 distance away), but the results so far show a substantial performance gain.
I actually implemented the code, but ArneBab and operhiem1 were instrumental in it happening. You can see more about it on ArneBab's site. He has also recently published this gem, and has been working on fundraising. Steve (operhiem1) has been handling day to day code review and releases (which has been a huge help, allowing me to focus on sorting out the client layer), and p0s continues to improve WoT, with a couple of big breakthroughs well on the way.
Things are looking interesting for Freenet ... just as I have to go back to uni in a month. I may be back next summer, we'll see how it works out; I might get an internship, but most of them barely pay the rent, and I do have other overheads. I'm doing well at university, in spite of numerous minor difficulties, and looking forward to (or is that trying not to think about?) next year, but thankfully first I will take a brief holiday. One interesting aspect of this was that in the first year of Cambridge Computer Science you study 50% CS, 25% maths (same as Natural Sciences maths) and 25% Physics (or a few other options). Physics is an interesting and important subject, and I did well at A-level; I seriously considered transferring, but concluded eventually that a career in Physics probably doesn't make sense. This consumed a great deal of my time and energy earlier in the summer. University is worthwhile because there are some things you just have to study intensively, you can't easily pick them up on the job; and it opens up some new career options. And as universities go, Cambridge is of course brilliant (and therefore also terrible!). I don't know where I will eventually go career-wise... I could come back to Freenet full time, but only if we have enough funded people and volunteers to make a serious dent in the numerous not-quite-intractable problems (see ArneBab's proposals). Academia is a possibility. Some industry jobs may be interesting, in spite of the tendency to ageism, London-centrism and the all-devouring financial cancer. We'll see.
Meanwhile the world tears itself apart and many of the really big problems are ignored as enthusiastically as ever, with the tabloids publishing yet more climate denial nonsense. Islamist scum using Diaspora but so far have ignored us ... this is good. Hoping we don't get obviously involved in the distributed online marketplaces thing any time soon (we can't deal with that sort of attention just yet IMHO).
It looks like the government apparently was right when it said the DRIP act doesn't give it any new surveillance powers - it was, probably illegally, interpreting the old powers to give it most of what the (blocked) Snooper's Charter supposedly contained anyway, and it needed the DRIP act to make this legal. So the UK government has powers to demand intercepts from web services (even if it means architectural changes) and presumably from peer to peer networks too, and to pay foreign services for their co-operation. As do the americans, at least in the web services case. Thankfully Freenet is open source, so there's a limit to what they can practically do to us; "secret" backdoors don't last long, or at least they need to be very subtle (think Heartbleed). The fact of the matter is right now Freenet is too small to be worth attacking directly, and in opennet mode, what security it provides is easily circumvented at the three-letter-agency level (and it may be vulnerable to much less well funded adversaries too). But we can change this. Darknet allows for some very interesting possibilities.
In the unlikely event of Scotland voting yes, I'm going to have to find out exactly what will be involved in emigrating, but then as we'd be out of the EU (and ECHR, which even Russia is part of!), it would all change anyway ... Government policy proposals that are so far out of the legal norms that they must be pure political posturing ... and two potentially world-reshaping wars ... Life goes on ... for most ...
The key is that the bad guys don't always win; life is more complicated than that. For example the network neutrality directive as passed by the European Parliament was surprisingly progressive (though it won't save Freenet from arbitrary blocking); this means it'll either be rewritten into the usual weasel-words, or it'll be dropped. See the software patents stalemate in 2005 for an example of the flawed, but at least somewhat democratic EU legislative process. When the powers that be don't get what they want that way, they find alternative routes; so software patents are now (or will be soon) legal here, not because of an EU directive (they tried that, and lost, in 2005), but because of an international treaty. Similarly TTIP threatens to massively reduce the range of possible government policies by instigating a parallel court system so corporations can sue governments for environmental or health regulations that threaten their profit margins, as well as preventing us from undoing pointless and inefficient privatisations and "harmonising" food safety down to the lowest common denominator. But this can be stopped. ACTA was stopped. The enemy is not omnipotent; there is a political or economic cost associated with any action they take. You just have to try...
So much to do, so little time...
It's been a long time since I've updated my flog. Part of the reason for this update is just to post my new Freemail 0.2 address.
I've had more time for Freenet since I finished my Physics course last August, although Maths has still been a big demand on my time until June this year. I am confident that I got the A I need to get into Cambridge (but I'll find out soon), so will be starting university in October, and therefore stopping paid work on Freenet. I may work on Freenet in a paid full-time capacity next summer, but I'm not ruling out doing something else. Which means that the Freenet side-conference in Berlin (probably but not certainly part of CTS IV) will mark the end of my current role on Freenet, at least for some time.
So firstly, what's happened since the last update?
Of course, this is not as much as I'd hoped we'd achieve in the period. Part of the problem is a lack of core developers. Who is going to maintain Freenet itself ("fred")? I'm hoping to get one or two people doing releases; the scripts have been mostly tidied up and we're ready to go. But it's still true that the majority of development on fred is by me, and much of the code is poorly documented, often over-complex, and hard to get into. And there are nowhere near enough unit tests. One priority is fixing the client layer database, see below.
Also, I've been working on a lot of "miscellaneous" stuff - a huge backlog of minor but important bugs, stuff to merge, stuff to move to the bug tracker, and so on, combined with frequently being distracted to help people with their projects (which is usually highly productive, don't get me wrong!). This means that the big stuff I mention below hasn't moved as fast as I'd hoped. But I will implement at least some of the big changes before I get to uni.
So what are people working on other than the stuff already mentioned? Two major projects:
What are the larger problems?
Much of this has been discussed in various places, and there have been various roadmaps posted, for example here (a bit strong on the IP-scarcity opennet changes that IMHO are dubious), here (beware of vandalism on both these links) and here. Feel free to discuss on FMS, or Freemail/email (with GPG) me if you want to talk in private. There are of course thousands of tasks, large and small, on the bug tracker. Seriously, we need more volunteer developers, and better now than later as I'll still be around to help you if you get stuck.
Of course there are lots of other things I'd like to see on Freenet: Various ideas to further improve performance (such as Bloom filter sharing), many of which have been bouncing around for years; filters for more content types, or a distro/VM/package that solves the same problem in a different way; lots and lots and lots of bugfixes; "eating our own dogfood" and secure release channels; and so on. There has been some discussion of using Bitcoin bounties to not only fund but also pay for development; if handled correctly this might make our development process more robust as well as allowing us to hire people who aren't always long-term contributors. However, there are some major drawbacks too, so we'll see where it goes. But IMHO we have a good handle on many of the key, fundamental problems in Freenet, but insufficient resources with which to implement them. Drop us a few million dollars and we could make an interesting Freenet 1.0! I'd be prepared to put off Cambridge for a year to help set things up, though I do intend to move on eventually (but moving on when so many key things are still unresolved is very unsatisfactory, and I do, still, passionately believe in Freenet, in spite of valuing many other things too)... Of course this is only half serious; I don't think I could manage 10 paid devs, Ian is more interested in Tahrir lately (partly because of repeated disagreements with me!), and an outsider would likely spend $2M on a new web interface and a Facebook plugin and declare job done! Or I could come back after uni - but I won't come back as paid staff unless we have a lot more resources (volunteer developers or paid ones), although I intend to work for Freenet over some of the summer vacations while I'm at uni.
The political climate is interesting. On the one hand, Snowden and apparently other people have confirmed what us paranoid privacy geeks and conspiracy theorists have suspected for years: that pretty much everything you do online is monitored and analysed in bulk, by various governments as well as by those selling advertising. "Economic wellbeing" is a well-established justification for intelligence - is this stuff used for inter-corporate espionage? Probably! On the other hand, people are openly talking about blocking darknets, and there is a good deal of interest in censoring even legal content on the web, social networks and so on. Both trends could play into our hands - provided that we don't see a really heavy-handed clampdown on peer to peer. This seems unlikely as it has generally failed so far, at least to prevent copyright infringement, and as p2p may yet be strategically significant for business with WebRTC. Unfortunately, to raise significant resources from this situation, we need something new and shiny. Most people who might care about Freenet are already disillusioned with it. Also, Kickstarter works best for commercial projects, and explicitly forbids social networks (Freenet might be seen as one); Freenet cannot be a commercial project AFAICS, because trust requires open source code, and long term persistence requires that the network not be dependant on its original authors staying around.
Now for some more politics: Some time ago somebody sent me some stuff on Zeitgeist. Post scarcity economics, transhumanism and all the rest, are fascinating, and in the long run probably have some important truths. The catch is, in the short to medium term, scarcity is going to increase, and not just by a little either; food prices, climate change, economic collapse or stagnation, increasing instability and so on, there are plenty of storm clouds on the horizon. This is one of the reasons I work on Freenet: We need safeguards, even new ones that go beyond the traditional democratic safeguards, because as we've seen before, all checks and balances are fallible.
I'm not a traditional libertarian or ancap; most of them are stuck in ancient battles which may be obsoleted by events that aren't that far away, and I believe that any system that doesn't limit poverty is going to lead to slavery and oppression, and furthermore environmental problems (especially global ones) can be difficult to deal with without a strong state. However, I'm not entirely a statist socialist either; some problems can be solved best by the state, some can't. I do think that "new" digital freedoms, which are translations of the old freedoms into new contexts and new possibilities, are something we need to seriously consider: Should there be an absolute right to exchange data of all kinds privately, and publish it anonymously? Freenet goes some way towards that, and while any sufficiently motivated state could crush Freenet and all other p2p, it would require political capital, and with WebRTC coming up probably some economic impact too. Should there be a right to be able to buy unrestricted 3d printers and other programmable fabrication tools, as long as you don't use them to hurt other people? Enforcing the old world order means preventing people from buying general purpose tools, and this may be both impractical and oppressive: If you can buy a 3d printer, should it be capable of printing whatever you design or download within its practical limitations, including items that violate intellectual property, firearms, drugs, etc, or should it only be able to create oligopoly-approved content downloaded from an official app store? That is one of many questions we will need to grapple with in the near future. It is analogous to the questions of purely virtual intellectual property: I don't support copyright infringement, though I'm skeptical about patents (especially software patents), but I object to limiting my freedom to use my own computing resources as I see fit, which are an extension of my mind, just to prevent me from violating copyrights. Of course some of these "digital rights" (a concept that is not fully formed in my mind at least) have technical aspects, and can survive as a safeguard provided they have a sufficiently large community, and provided the opposition isn't too fierce. We'll see where it goes!
I have some sympathy for Extropism, though I'm not convinced about abolition of money. In any case, it's going to be quite some time before we can talk meaningfully about post-scarcity anything. Of course the future is impossible to predict, mainly because while you can extrapolate existing trends, you can never tell what the order of events will be, what will happen before what, what will turn out to be really hard, what could work but just doesn't happen because of lack of opportunity, vision, or business sense, and so on. What was true for the last century wasn't necessarily true for the one before or for this one or the one after; assumptions we take for granted about life, politics, economics and human nature sometimes turn out to be completely wrong, or just obsolete. My relatively poor social skills, lack of photogenicity and record of working on Freenet are not the only reasons why I work in technology and not politics; technology can often make a bigger difference to people's lives. Which is part of the reason why I'm off to university in October: To learn to make things better (and exactly why and how) from one of the best universities in the world, to find out whether academia is for me, and to allow me a good chance of getting a job doing something really useful and interesting (like I have for Freenet, but hopefully with more resources!), rather than being limited to the handful of jobs that don't need a degree and don't reject my eccentric experience.
My last post referred, in a somewhat panicky way, to a new law currently being debated in the UK which, as I saw it at the time, might have been used to compel Freenet's release managers (notably me) to release builds with backdoors for tracking. The details were unclear, but the intent was for law enforcement, intelligence agencies etc to have access to logs of who you communicate with - even if it's through an online game, social networking site, peer to peer network etc. I had feared that this might lead to me being forced to implement backdoors in Freenet. This has happened in other european countries. However, it now appears that the powers will simply require ISPs (apparently only the biggest 6 who account for 94% of UK internet users) to surveil or block p2p, SSL-encrypted social networks etc. This doesn't mean they will block Facebook and thus provoke a mass exodus; there would need to be some voluntary involvement by the social networks, to avoid being blocked, but we know already that some of them have been scanning their users' posts for references to criminality and passing it on. However, Freenet certainly would not cooperate. Which means if the legislation passes in its current form, Freenet may be blocked for 94% of the UK. It does not mean I will be forced to put in a back door, which I certainly will not do under any circumstances. I apologise for not digging deeper last time.
However, the key point is this: Ideally, security should not depend on one person's trustworthiness. There are a number of things we can do to ensure that if a release manager's computer is compromised, bad guys can't release backdoored builds. We should do some of them, and my comments were in part a plea for help. These issues are likely to be common to any open source project with limited resources that needs to publish auto-updates... They are an interesting problem, and one that certainly must be solved eventually, although there is no need for immediate panic given the above.
Right now, any of the release managers for Freenet can deploy a build, and all the nodes with auto-update enabled will update themselves fairly quickly. This is a good thing: It allows us to fix bugs quickly, to change the network protocol to try to improve performance and measure the effect of changes. This is why there are "mandatory" builds occasionally (which won't talk to older versions). However, it means in theory corrupted builds could be distributed.
IMHO there are two components to any meaningful safeguard:
Both of these are important. A partial solution to the first is provided by ensuring that the source from github repository and on Freenet corresponds to the binaries released in the update. This is the purpose of the "verify-build" script in the Maintenance scripts repository. At the moment this can verify the published jar on the web against the published source code on the web and the git repository - it doesn't yet fetch the jar and source from Freenet. operhiem1 has been very helpful, and finishing it shouldn't be a big job, Ideally we would like several well-known anonymous FMS posters to run the script, as well as developers and other traceable people.
The script requires linux, and the exact same version of Java as the person doing the release (not always me) used to build it. This suggests it should be used in a virtual machine. Integrating this into the update process (creating a disk image and then calling the VM with a platform-specific command) for those users who don't mind the extra overhead (and maintenance) would be interesting.
Or we could use it as part of a "honey trap": One problem with just using the script is that once an update has happened, the corrupted nodes may send you a "clean" binary. But they will have to send the corrupted binary out at some point, and provided the attacker doesn't know which nodes are "honey traps", this could be effective: the honey-trap could detect, and prove, that there has been a bogus build distributed.
Even better would be for a release to require not only the signature from the release manager, but also from some automated signature services run by third parties. These would run the same script, and certify that 1) the binary corresponds to the source code, 2) the source code published on Freenet corresponds to that published on github, and 3) there has only been one such signature for any single build (the sigs would be published). A requirement for multiple signatures could be implemented using the proposed PSK key type, but since the release manager isn't likely to be a spammer (he has better things to do even if he's malicious), we could just use a separate key-stream for the signatures.
This brings us to the second part: How do we establish that the code is "clean"? In the first place, publishing it means that there is at least the potential for anyone to read it and find bugs, and shout about them, if necessary via means other than Freenet. So it is certainly not useless to be able to verify the binary. But ideally we'd like more than one person to have explicitly reviewed the source and signed it before release. However, requiring a second developer to read the diffs and sign them just before a release is rather onerous. Normally a release takes some time, so it would be better if the snapshots (or individual commits) could be signed by those developers who are keeping track of the code whenever is convenient to them, using the tools Git provides already, or using some Freenet-specific tool. Testers could also sign snapshots - not guaranteeing that they are not malicious, but at least providing a basic sanity check. It is likely that we will need to release builds quickly on occasion (e.g. to fix an urgent bug), at least for some time to come, although we could treat them differently; but this is an interesting direction in the long run.
There have been fairly detailed discussions on this on FMS, I recommend reading the threads there; some of it is little better than flaming but much of it is interesting and relevant.
A new build will be released shortly, which includes major changes to the auto-updater system. These are not related to the above, but will allow us to include separate files and not just update freenet.jar and freenet-ext.jar. For example, we will be shipping the Bouncycastle crypto library, which we will soon use to upgrade some of our encryption code, to be faster, smaller (in bytes) and more secure. Thanks to Nextgens (Florent) for his work on the latter and prodding on the former! This will also enable us to finally implement Infinity0's (Ximin's) ambition to split up freenet-ext.jar.
Another big project on the horizon is a new keytype, Programmable Signed Keys (PSKs). This has the potential to greatly improve performance of forums and lots of similar applications over Freenet. I won't go into detail just yet. I'm hoping that Sadao, who originally proposed a simpler version of the key type (where you can post if your identity is signed by the key's pubkey), will use this to implement a new messaging system. Also, this is of course similar to a proposal by one of the original developers around 0.4 or even earlier, which wasn't acted on for simplicity's sake (it's also inspired by bitcoin's workings); the complexity of messaging systems built on Freenet means its time has come. Making chat, forums, social networking / microblogging, even some forms of search, scale over Freenet is one of the 7 or so "impossible things" that Freenet still needs to do to make serious progress. I now have a fair idea how to solve most of the others too. :)
And of course, there are various small but annoying admin-like issues to deal with, such as how to automatically upgrade the wrapper (especially on Windows), updating the windows installer, and so on. There is lots of code to merge. There are the ever present client layer bugs and db4o corruption problems, various ideas for how to reduce disk I/O, how to safely gather data to understand why Freenet's data persistence isn't what we might hope it to be, and a whole stack of major security improvements which should make tracing content uploaders very difficult even on opennet. And then there's the stuff that's been in the queue for longer, for example making darknet easier, and sorting out load management. So much to do, so little time, but I really think we are poised to make some serious progress, including on the "impossible" issues or "grand challenges", such as scalable-enough chat, fixing the Pitch Black attack on darknet, beating Mobile Attacker Source Tracing, substantially improving data persistence, and so on...
Meanwhile I have interviews for university in the near future. I'm having to spend a significant amount of time on that, mostly sorting out my maths (I have one more A2 to do next year). I have one interview booked so far, no word from the others yet. But I'm also managing to make some significant progress on Freenet. Thanks to all our volunteers!
How safe is Freenet anyway?
There has been a lot of discussion of security lately, initially because of these academics (Dong, Baumeister, Duan et al). They published two attacks on Freenet, and then appear to have got funding for an effort to unmask Freenet users, as an "Experimental Study of Accountability in Existing Anonymous Networks". Their goals are apparently as political as ours, but in the opposite direction; read the Project Summary. Which is fine by me, see the end of this post! Anyway, I'm absolutely delighted that somebody is doing serious research on Freenet's security, thanks guys!
Their first paper, "A Traceback Attack on Freenet", concerns a genuinely new attack based on request UIDs. It was possible to trace a single request back to its originator, and this seems to go back some years; the conditions required are 1) that Freenet gives a distinct response, RejectedLoop, when a request UID has already passed a node, 2) that it remembers the UIDs of prior completed requests, and 3) that you can somehow probe a node for a UID without the request being routed further. We have now eliminated #2, since it isn't necessary; this eliminates 99% of the threat from that attack, because the only way to do the attack is if you can connect to nodes behind the bit that reached you before the request finishes. This might occasionally be possible for inserts, but we will improve it further; in any case it's likely that other known attacks are more powerful than this now.
We have planned further improvements, and a more radical solution that might have an impact on routing performance on darknet. Their paper suggests that we use a single indistinguishable failure mode; we do need to look at this but probably can't reduce it that far due to the need to distinguish between fatal failures and non-fatal ones, limit the total hops visited, and identify overloaded nodes.
Sadly I haven't had time to read their second paper yet. Nextgens tells me it's largely a confirmation of what we already know: An attacker can take over your routing table on opennet. This is unfortunate, but difficult to prevent, and is one of many good reasons why in the long run we need a large darknet. This will require improvements to both performance and especially usability, which are still planned. It will also require solving the Pitch Black attack. We have a solution, which appears to work, but the person working on it hasn't got back to us yet. If he doesn't we may need to just implement our own simulations eventually to prove it works.
I have often said publicly that opennet sucks. This is true, in that its security is much lower than is achievable on darknet. I have also said that opennet's security is hopeless. While Freenet is still experimental, so you probably shouldn't rely on it, this is perhaps a slight exaggeration. I explained a bit on FMS recently so will post details here too.
Freenet's threat model (very basic version!)
We assume that the attacker is initially distant. That is, he knows that of an anonymous identity (a chat identity or the author of a freesite etc) which he wants to trace back to its real-world identity. We want to make this task as difficult as possible.
In other words, Freenet is about hiding in the crowd. It's not much use if you are already on their shortlist of suspects. In the more unpleasant regimes, and even in the west quite often, you have bigger problems at this point (e.g. dawn raids!). In particular, Freenet does not provide strong protection if you are already connected to the target: Provided you can recognise the content (possibly after the fact from logs), you can identify with reasonable confidence that they are the originator. One consequence of this is you need a reasonably large network to hide in, and if the bad guy can afford to connect to everyone (or in some cases exclude a group of nodes at a time) he can probably trace you.
We do not provide the sort of anonymity guarantees that mixnets such as Tor can in theory provide. In practice a scalable mixnet is a difficult problem for similar reasons to why opennet on Freenet is hard, e.g. route selection attacks. But it's a different, and difficult, problem. We do plan eventually to provide stronger protection against nearby attackers, using some form of tunneling (classic mixnet sadly doesn't work on darknet). However, in the near future we have bigger fish to fry, as I will explain below.
The other part of the Freenet threat model is blocking: We want to make it hard to block Freenet on, for example, a national firewall. Again, blocking opennet is straightforward, because you can simply harvest the nodes and block them. We try to make it as difficult as possible to identify the Freenet protocol, and one of our Summer of Code students this year has been working on transport plugins, which will enable us to disguise Freenet traffic as VoIP, HTTP, etc.
Of course, traffic flow analysis can identify any peer to peer network, including Freenet, even if we have very good steganography. The brute force approach of blocking all packets between two "consumer" IP addresses (similar to many of the spam RBLs) will also crush any fully decentralised network. But this is likely to be somewhat expensive and have some collateral damage; provided that VoIP remains important, and largely decentralised in its actual traffic flows, hopefully this won't happen soon, outside of places like Iran.
Obviously traffic analysis can be used to try to identify what is going on on Freenet, combined with malicious nodes, and so on. This is a problem for Tor too. Freenet is designed to have higher latency than Tor, mostly used for big downloads, which is a slight advantage, and there are a lot of things we can do in future. I'm not going to make any promises about traffic analysis here, and no doubt there are other classes of attacks.
The bottom line is use darknet. No, really, use darknet! It's vastly more secure because it's virtually impossible for the attacker to connect to everyone, and very expensive for him even to connect to one more node: he has to either hack your computer or social engineer you. Which isn't necessarily all that expensive - it's just that compared to opennet, where you can just announce to a location, it's very expensive.
Also, for reasons explained below, for now, we only provide strong protection for original content uploaders (and that only if they insert to a random key i.e. SSK@). If you download content, or reinsert it, you are potentially vulnerable. However, we will improve on this; see the next section.
Mobile attacker source tracing and how to fix it
Unfortunately, it's actually slightly worse than the above. One attack we have been trying to deal with for some time is called "mobile attacker source tracing". This potentially enables an attacker to trace content authors without having to connect to a lot of nodes at once. The basic principle is if you can identify requests, you listen for them, and work out where on the network the originator couldn't have been (based on routing). You correlate this over many requests, and then announce to a location closer to where you think he might be. If you get it right, you will see more requests, enabling you to approach him rapidly.
This has been documented on the FAQ and the wiki for some time, so it's not news. And it is not as bad as it sounds, even on opennet: It relies on being able to identify the keys before the (large) insert/request finishes. Which means it only applies to 1) downloads of files known publicly and 2) uploads to predictable keys. In other words, if you always insert as SSK@, your uploads can't be traced - the attacker can only use your chat posts, the block at the top of your freesite uploads, etc. It ought to be possible for original content authors to remain reasonably safe, although they will have to completely change their identity from time to time, and high frequency posting makes you more vulnerable. Sadly we don't yet know how to quantify this. And downloaders are vulnerable, as is anyone who reinserts a file to a known key. But we definitely provide some protection for original content uploaders, provided that re-inserts are done by other people.
After the recent discussion of the new attacks, we (mainly me and Evan) had a long brainstorming session concerning attacks. IMHO we can improve the situation dramatically. It's a combination of two ideas, and it can potentially protect both inserters and requesters.
First, we can greatly slow down the attack by ensuring that high HTL requests are not routed to new connections (say peers that have connected in the last hour). This will have some impact on routing, but we can make the time limit reduce rapidly with HTL, as it is less powerful the further away you are from the originator. Also, we will need to tweak the time limits based on our peers' uptimes. There are of course targeted attacks against this, but it's still an improvement. The result is an attacker cannot move towards the target quickly: On each new node it connects to, it will have to wait a long time before it can capture more data. This makes small, quick requests much safer, and even for larger requests it means the attacker will need more time or more resources or both.
Secondly, routing is predictable, and this is the whole basis of the source tracing attack. We already treat the first few hops after a request starts specially: Above HTL 16, we don't cache requests, and above HTL 15, we don't cache inserts. This is to prevent datastore probing, both for what you have requested directly (a targeted attack, arguably outside our threat model), and to try to trace a download or insert after the fact (a more serious threat). The initial, maximum HTL is 18, and we spend an average of 2 hops (randomly) at 18 (we then decrement each hop down to 2).
So the proposal is, when we are in "high HTL", simply route randomly. This greatly blurs the original location of the requestor, because the request doesn't start until we are several hops away from it. Assuming the HTL thresholds are correct, the attacker will probably now need to rely on a completely different, and more expensive, attack: Try to intercept requests before they end the random routing/high HTL stage. And because they are routed to a random node at each hop, they have much less information.
For inserts, this is actually extremely cheap. We reach the "ideal" node by around HTL 16 anyway, so there is plenty of space left. And we're not visiting any more nodes. So we might reach a few less nodes around the target, but we'll still visit the important ones. We might lose a bit of performance depending on how we deal with backed off peers in our random routing. Plus it may help us to escape "rabbit holes" etc.
However, for requests, this may cause a significant increase in latency, since the extra hops are at the beginning. But on the other hand, the fact that we already don't cache until HTL is below this point, may mean we are currently caching a lot less than we need to, since we will often reach the "ideal" node before we exit the "high HTL" no-caching mode. Also, if we do this we won't path fold while we are routing randomly (that would be silly). There is a popular theory that a lot of nodes which do many local requests have too many long connections and not enough short connections, so optimising for their local traffic ("long connections", across the keyspace), but ruining routing from a global point of view. The proposed solution (by TheSeeker IIRC?) is to not path fold close to the originator. And further it improves security since the data source can never directly see the request source. So while random routing on requests would cost us some latency, it may well improve performance in other ways. IMHO it's worth a try, but it could be a bumpy ride.
In other news...
Summer of Code has finished. Chetan is keen on getting his work on transport plugins merged and it looks like he's gonna stick around for a while, yay! Sometimes we pick up devs from Summer of Code, and when we do that's always awesome. Operhiem1's work on probe requests has both improved security (the old probes gave away far too much data) and given us more useful data to chew on. He and Evan have been working on simulations and confirmed that there is some sort of problem with routing, along with gathering data to inform simulations of possible solutions. I'm not sure what's going on with Pausb's web interface rewrite.
There have been a lot of complaints about disk I/O and resource usage in general. I have implemented one improvement which should be deployed soon, which is a small tweak to the "slot filters" which took over from Bloom filters (an important optimisation to the datastore); this should save at least one disk seek per block write. The bigger changes, notably queueing up a bunch of blocks to write to the datastore, will be implemented later.
We are still having problems with db4o corrupting itself, especially when there are uploads queued. The next thing to try is to upgrade the database. A newer version may have more bugs, as seen in the past occasionally, but it's the logical next step. Automatic defrag on startup has been turned off as a stop-gap, which may help a little with stability, but does mean the database will get bigger and bigger and bigger...
Finally, I have two major sources of stress out of the way: First, after much deliberation, I decided I want to apply for Computer Science, the topic I'm most likely to do well in. I was concerned that I would not be able to make a significant contribution to sustainability, but it turns out there are lots of sustainability-related computing problems, see e.g. here, here, here and especially this blog.
Second, as of yesterday, I've completed my university application, although it may need some changes depending on college reviewing it. I still need to prepare for interview, including reading and an online course and especially working on my maths; I have three maths exams next year, and it will be important at interview. This means I will only be working part time on Freenet until January. After that I should be able to do more hours, up to starting at uni in September/October.
Of course I may stay in networking; I will certainly have a look at networking/security issues at uni, and contribute to Freenet from time to time in the holidays (and maybe more than that depending on placements etc). From what I've heard, it should be possible to run Freenet in China, for example; always-on broadband connections are rapidly expanding, and don't have severe monthly transfer limits. And in the west there are many clear threats to freedom of speech online; Freenet is a useful additional safeguard. Regarding paedophiles, well, computers don't abuse children; paedophiles abuse children. Studies show that the massive expansion in the availability of child porn in the 80s in eastern europe had no measurable effect on rates of child abuse in general, although there is evidence that convicted abusers who have access to child porn online are more likely to reoffend. For western purposes, they are the canaries in the coal mine; if Freenet is safe enough for them, it's probably safe enough for your blog that you can't publish openly without getting fired. Go read the Philosophy page!
I wish there weren't so many paedophiles (and hatred groups, climate deniers etc) using Freenet, but it is an inevitable consequence of the absolute freedom of speech that it tries to provide. I do think in the West we probably have many years before the government would consider the sorts of radical measures needed to block a darknet. And for the record I'm not an extremist libertarian on other issues. I'm deeply skeptical on bitcoin, and the radical anarchocapitalist/agorist/cypherpunk agenda in general; I accept the scientific consensus on climate change even though it requires some limited expansion of the state to deal with it (e.g. in the form of carbon taxes etc); and generally care about social justice.
But I also believe that if we can make it harder for the state to say who is allowed to say what, that would be a good thing. Of course it's not the state, it's the corporations, or anyone who can write a threatening letter. Especially as technology, with its current centralising trends, and increasingly powerful AI tools, allows for potentially very powerful monitoring and censorship mechanisms, for example Facebook's automatic fishing expeditions for criminal activity. Or the china-style self-censorship climate of fear on the centralised filesharing sites, which was always inevitable; if you put all your files on third party sites, one dubious copyright claim and they're gone, as happened to NASA recently. If you put it on your own domain, you need to fund it (which rather harder if the credit card companies block you), and deal with DDoS attacks (which is harder if the big hosters won't touch you for political reasons), and then they block your domain. We need long term solutions before the censorship infrastructure online becomes so powerful, so blunt, so centralised, and with so few safeguards that it makes the censorship our constitutions are supposed to prevent look positively pointless. Freenet is one such solution, in some places, at some times.
Decisions good and bad
Education and time
Summary: Due to education issues, I have done very little work on Freenet this year. Delaying some exams should mean I can do a lot more work on Freenet very soon, although I still won't be full time as I was for 9 years.
Last August I decided to retake a couple of A-levels in order to get into a good university. This was partly because of persistent worries about funding of Freenet coming to an end, partly because I've worked on it for 9 years, with substantial progress but still nowhere near where we need to be, and never having really made the breakthrough in performance or numbers of users. The plan was that I would retake 2 A-levels over 1 year, work part time on Freenet in the meantime, and apply for university already having my grades.
This requires me to simultaneously:
I have not managed to balance all these objectives: I have done much less work on Freenet this year than is necessary for my own financial and psychological stability, or for the Freenet Project's continued success. Meanwhile, FPI has accumulated a fair amount of funds - not enough for me to work full time, but I don't need to work full time. And we have been gaining users too - and many of the old ones are as enthusiastic as ever, and even one or two new devs here and there.
The reason for this is maths has taken a lot more time and effort than expected. There are a few parts of the core maths that I need to revise, most students do Further Maths over 2 years along with Maths, and I need to go up 4 grades (although I did an Open University course in 2001 which I did well at). Hence I am worried that while I might get an A, I won't necessarily get an A*, and I won't do well in STEP (although the AS modules I've done I probably did reasonably well in).
So the solution is to postpone the Further Maths A2 modules until next year (January and June), rather than doing them this June. This will allow me to work regularly, albeit part time, on Freenet, and do well in both subjects, hopefully doing STEP as well.
I'm still not sure what I want to apply for at university, but the above should help.
The main thing I need to say here is that it is still the plan to try to make Freenet's development more distributed and less dependant on me. Bombe has put out a test build, and will soon release build 1406. I will still be involved in release management, but hope that Bombe can do some of it. We still need volunteers in many other areas, see the discussions on the mailing lists etc. And we have been having many patches contributed via FMS, which is really awesome.
There are also rumours about a port of FMS to a Freenet plugin, which would be very interesting. Meanwhile significant optimisations to WoT will be deployed very shortly, but they sadly don't fix all the performance problems for Freetalk.
There has been some discussion of how Freenet and Tahrir relate to each other. Tahrir is a project by Ian Clarke to build a distributed twitter-like system. It is especially important now that not only are the Chinese Weibo services doing keyword filtering and user traceability, but Twitter itself is doing country-specific censorship (what this entails is unclear, it would certainly include URL blocking as it does in the west re child porn).
One of the proposals was that Ian take a Summer of Code student for Tahrir under the Freenet Project Inc umbrella. IMHO the two projects are pretty close together in terms of their political goals, and even in the long run many of their technical goals. Tahrir needs routing, darknet, and eventually some sort of filesharing capability. Freenet needs microblogging. How exactly this unfolds is not yet clear, but the two are complementary for the time being, and shouldn't clash.
One thing that both Freenet and Tahrir need is fast distributed spam-proof keyword search (possibly with social aspects). I'm not sure how to do this in either case, but a solution would be immensely helpful for both.
Finally, we appear to have had some press coverage lately, the latest graphs are encouraging.
See you soon ... !
And now for something completely Freenet... (Freenet)
Problems with build 1389
As a few people have been reporting for a little while now, 1389 has problems. I had not been able to understand it properly, and was busy fixing the last few timeout bugs related to NLM, but now having looked at some reasonably objective data generated by the auto-tests that run twice a day, I can confirm that something bad happened around the 24th of July (although I'm not sure what), and since the 1st of August or thereabouts, tests have consistently failed. These appear to correspond to build 1386-1388, and 1389, respectively. Build 1392 includes a few fixes for possible issues, mostly in 1389, two of which are fairly serious (see the detailed changelog for details). I'm not sure I 100% understand what went wrong, but there is good evidence that 1) slower inserts were very badly affected and 2) requests could get very confused after a timeout; these are both fixed in 1392. The most annoying thing is everything I've done in recent builds has itself been fixing various bugs - mostly timeout- and bogus reject- related issues which jeopardise New Load Management, and show up in testing on testnet. However, there is a reasonable chance that 1392 will solve the problems, and if it doesn't I'll have a look at what else could have caused it. Two other threads going on: First, important opennet fixes in 1391 - it might take a while for opennet to sort itself out (and bootstrapping is still rather slow, we always need more seednodes, this may be partly due to other recent problems). Second, newly bootstrapped nodes are very variable. Sometimes, they work really fast immediately. Sometimes (rather often) they take ages to bootstrap, as above. Sometimes they get a lot of transfer failures - this is not consistent, it seems to show up some days and not others, although it seems to have sorted itself out in the last test I did today, after initially being very bad. I don't think there is a serious low level issue with transfer failures (it would have shown up on testnet probably), but it's an outside possibility... I hope not, having spent many weeks debugging this earlier this year...
By the way, testnet is at the latest count a grand total of 14 nodes. We need more testers. It's not anonymous, but it helps me to debug Freenet, especially the harder bits (routing, updating, link level, etc), which can't be tested on just one node. See the installers for Windows, linux/mac. Please consider it if you have spare bandwidth, spare CPU cycles and spare disk space (it needs at least 5GB for logs, and logs quite heavily, so may need a fair bit of CPU).
New Load Management
Is nearly ready! And I really mean it this time! I had planned to deploy it last Monday, but last weekend me and ArneBab had a very helpful discussion about NLM and timeouts, which resulted in some significant changes (including making timeouts proportional to HTL, which will be deployed early next week), and then there was lots more debugging. It may be delayed further if 1392 doesn't solve the problems from 1389. But it should happen within a week or so. The main reason it has taken so long is that it has stirred up endless minor bugs (many of them related to timeouts and ordering of requests) that become important when we are predicting whether our requests are going to be accepted rather than just sending them and misrouting when they're not. Many other biggish changes happened in the process - fairness between peers (essential for security and for performance of low end nodes); new packet format (not strictly vital for NLM, but certainly related); the split between bulk requests and realtime requests (absolutely vital for NLM given it queues requests), and so on... Many thanks to ArneBab for helping me to understand the last few issues...
Other priorities remain roughly the same: Before 0.8.0, we need the darknet enhancements, which should make it far more convenient, and much better performance, to connect to your friends (and, automatically, but with an opt out, to their friends). Long term we need a global darknet because this is the only option that is not easily blocked, as well as being a lot easier to secure on many levels. We still haven't fixed the Pitch Black attack, but we have a friend working on it for an academic project, as well as a reasonably firm suggestion from our main old-timer theoretician (but this has not been tested yet). But darknet is the future, and not just because it's buzzword compliant ("social" ...).
One thing that we arguably need is for the new multi-container freesite insert code to be enabled. In 1392 it is possible to use it for transient site inserts, provided the client asks for it as an FCP option. We may make it default soon, but for huge sites (or for cases where you want to insert a medium sized site persistently), we need it to work with persistence. That means a lot of debugging, and especially de-leaking, which will probably have to fall to me ...
Help wanted: Run a search index Spider
If you can run the Spider plugin and generate a search index, anonymously, please let us know. It is very resource hungry, but it does just about everything for you - just load it, configure it, and then get the USK@ from the logs (wrapper.log) once it's inserted the first edition (after a couple of days). The new search code is greatly improved on the old search code. Meanwhile, on FMS, Jeriadoc (probably this hobbit) has written a new search index which may improve performance but we are still discussing how to make it scalable. In any case it should be made official soon.
More broadly, I still need to make Sone, and probably Jfniki, official plugins. This mostly involves code review and a little bit of admin. Beyond that, there are loads of bugs to fix, and I hope that Freetalk (p0s) and Freemail (zidel) will be ready for a release in September. operhiem1 is dealing with various small but important changes on the web interface (such as the first-time-wizard alternative discussed on devl), and infinity0 has done useful work on packaging and splitting freenet-ext.jar which has not yet reached its full fruition. Further optimisations will probably have to wait, along with client layer auto-backups, and there are zillions of bugs to fix as always... And if you can help in any way, let us know - apart from general coding, testing and statistics, translations, and of course contributing content etc, we also need with specific platforms, especially Windows (AHK experience a big bonus), and Mac.
Funding and 0.8
In approximately a week I'm going to have to ask for more funds. Including paypal, bitcoin (estimated), and bank, we have another $3539.81. If I charge my standard rate to FPI, this is good for around 5-6 weeks (those of you who think that's a lot of money remember I'm living in the UK, although I may be able to give FPI a discount now that I've paid off my various debts). Google has in the past been extraordinarily generous, but I doubt they will be again; even if they are, we should have something concrete to show for their last lot of money: Freenet 0.8! Right now we are some way away from that, but hopefully recent regressions can be cleared away quickly, opennet bootstrapping will stop being such a pain, new load management will work, and there will be enough time to debug as well as finish the few remaining critical features...
IMHO there is a good chance they won't come through this time. Last year I suspect had much to do with their battles with China. In any case it's not exactly a sustainable long term solution! Therefore, if you have money (or bitcoins) that you can give us, this would be very helpful. See the donate page or send coins to 1966U1pjj15tLxPXZ19U48c99EJDkdXeqb. Hopefully we'll get enough interest with 0.8 to keep me going for a little while longer and reverse the recent decline in the number of people running Freenet - especially if 0.8 is really good! If not, I'll be looking for other work, and Freenet will continue (I'll stay on as a volunteer, or work part time/intermittently depending on what I can get elsewhere). Right now there aren't many core developers, but hopefully that will improve with more users...
Personally, if I can't continue as is, I'd prefer something I can mix with Freenet, so either contract work that I can do mostly offsite or commute to from Scarborough, or perhaps a more permanent post with enough space that I can continue to contribute occasionally when it isn't crunch time (e.g. Google allow you one day a week for personal projects). I'm also considering university in October 2012... The fact that I don't have a degree makes it harder to get a job, but, from the right place, it could be well worth the effort in lots of ways; while I have picked up a lot of things, there are areas where I would clearly have benefited from the right courses... Of course, I have access to quite a lot via ACM and will probably spend more time on professional development in the near future... I would certainly like Freenet to achieve its main technical goals before I sign off on it, although I would like to do something else eventually, there are many fascinating things out there ... equally importantly, if we achieve our main goals for Freenet, hopefully that means we have a lot more users, content, and developers...
Some of my goals for 1.0 are:
I suspect Ian would say that we only really need a good user interface, and we can call it 1.0 now! Others would certainly have other priorities. We'll see. Also, to those who say this is over-ambitious, they may be right, but a great many things we have planned for and talked about for years have finally happened - for instance, the new packet format, new load management, the changes to USKs, and many other things.
The more people we have, the faster it will happen - and if it doesn't happen quickly enough, it probably won't happen at all, because we either won't have enough people, or governments will do horrible things. If we succeed earlier we have a greater chance of surviving, and can perhaps influence things a little too by weight of opinion; if we remain a tiny network with a reputation for foul content simply because there isn't much content overall, we are much more vulnerable. Plus, even if we get continued funding, I don't actually want to spend the rest of my working life on Freenet - and certainly not as the sole core coder, main debugger, architect, coordinator, backup windows maintainer, and almost everything else! So many thanks to all the volunteers who have been helping recently!
Watch this space...
The Big Climate Reconnection in Scarborough! (climate, local)
On June 3rd, a bunch of us visited our MP, Robert Goodwill again to talk to him about climate change. This included me and Jane (both Scarborough Climate Alliance / Scarborough Climate Action Network), Sue (a friend of Jane's), Liz (Green Party sympathiser), Kevin Allen (Scarborough Private Tenants Rights Group), Gilbert (Greenpeace), and one other person whose name I have forgotten. A Scarborough Evening News reporter turned up to photograph us on his day off, although there was some disagreement about who should be in the photo (we got everyone without Goodwill, and me and Jane and Goodwill).
Our main demand was about insulation. Apart from the obvious climate change benefits (heating is nearly half the UK's carbon footprint), Scarborough is one of the worst places in England for fuel poverty, with 23% of households fuel poor (spending over 10% of their income on energy). Also, a lot of homes in Scarborough have solid, hard to insulate walls, and are rented. The government's Green Deal should help, but despite optimistic projections they refuse to tell us how many homes will be refitted with modern insulation. The Warm Homes Amendment requires the government to tell us exactly how many homes will be upgraded, which should help to build the local businesses involved, allow us to hold the government to account, and ensure they do whatever it takes to meet the targets they have already set themselves on fuel poverty and climate change, along with fairly strong targets for the 2020s, which we were going to ask for this time. Mr Goodwill was helpful, explaining how such an amendment might make it into the bill. He has a good track record, but I suspect as a whip his hands are tied a bit. Last time we met him we asked for a minimum insulation standard for rented homes - which was granted in recent weeks, along with fairly strong targets for the 2020s, which we were going to ask for this time.
Our other main demand was that the government support funding for poor countries to adapt to climate change and develop their economies with less fossil fuels than we have. We can't afford 9 billion people who all live like brits, let alone americans, and most of the countries which are already being hit hard by climate change are too poor to adapt, and have very low carbon emissions. This will help unlock the negotiations, and is a matter of justice. And the governments of the world have already promised $100B/year by 2020 (but haven't anywhere near delivered it yet). This will have to come from new global taxes on flights, shipping, and banking (the Robin Hood Tax).
We also talked briefly about local action - local councils can achieve great things when they want to, but with budgets slashed, most won't unless it is made a core responsibility; it falls to the bottom, in spite of the chance to create jobs locally in insulation, transport, and other green industries, and the people involved get the sack. They should also have local carbon budgets and the funding to achieve them.
Beyond that, we talked about various things, including high speed rail, electric and hybrid cars, heat pumps and the recent carbon targets. We especially talked about transport, since Mr Goodwill was shadow roads minister. It appears that he was responsible in part for the abolition of speed cameras. Oh well, lost that one (more important losses are the Green Investment Bank's inability to borrow until 2015 and the farcial changes to solar panel subsidies - both will cost time and jobs). But overall it went pretty well, we kept the pressure up and discussed the issues. He pointed out, rightly, that the greenest mode of travel right now is the coach. But by 2030 all our electricity should be green, so the train will be even greener than it is now. This could be done with all renewables - mostly wind - but the government wants 40% nuclear.
Next month we will be jumping in the sea waving placards demanding that all net carbon emissions in the UK end by 2030. This is quite possible and won't require us to go back to the stone age - in fact it will create jobs, although we may not be able to fly much. See you then!
Gory details (longer account written sooner after the event)
Robert Goodwill is a frontbencher and a government whip. He generally votes in favour of action on climate change, has a good grasp of the issues, and gave us a good hour of his time today. On the other hand, he didn't commit to any specific actions resulting from the lobby. So on an objective level perhaps we achieved rather less than we might hope for (okay, I was tired!) - but as a government whip his options are probably pretty limited. In any case we made it clear that we care, and discussed various climate issues. An american president once said "I agree with you, I want to do it, now make me do it" - whatever an MP's instincts are, keeping up the pressure and linking it with constituency issues is a good thing. Still, we may have not been robust enough and certainly erred on the side of discussion of everything and anything rather than extracting specific commitments. On the other hand, they may not have been forthcoming had we pushed for them (being a government whip limits his options). Still, I'm sure it was beneficial on some level. Another criticism I would make is I talked too much!
After preliminaries we started with our core demands, starting with the Warm Homes Amendment, which requires the government to set a target for how much insulation will actually happen under the Green Deal and related measures, to provide confidence for the industry that will deliver them and ensure the government makes sure the policies are in place if things go wrong, and is supported by everyone from the TUC to the FSB to the FMB to Tearfund. He told us that it's unlikely that a non-government amendment will be accepted, although it might slip in in the Lords. I touched on the fact that there may not be enough funding for the Green Deal to insulate everything (it was in the press recently but maybe I shouldn't have mentioned it). Kevin raised various points related to the rented sector: A recent victory (which was one of our demands last time), the minimum insulation standard from 2018, may be diluted by poor enforcement related to difficulties over tenants rights. Goodwill is a landlord, and government generally wants to avoid extra regulation on landlords, but it may be possible to improve legislation. He also pointed out that people tend to be fairly good at reporting problems to councils, although of course all councils complain of resource problems!
We discussed adaptation funding (although we didn't mention the World Bank). This is essential for progress in the international negotiations, and given the difficulties with funding (e.g. the recent fight over the 0.7% aid target), will likely have to come mostly from internationally agreed taxes on banking (which he doubts will happen), aviation and shipping (which are less clear). I pointed out that high speed rail will probably make only a small difference, but we were broadly supportive of it. We did not extract the commitment to write to Andrew Mitchell about this, although that was an official goal of the Big Climate Reconnection.
We briefly touched on local council action: the Big Climate Reconnection demand is that climate action be made a core responsibility of councils; Friends of the Earth goes beyond that, asking for Local Carbon Budgets. Robert's view is that this is largely a matter for local lobbying - i.e. that there isn't a need for any statutory prodding. Clearly this is a mistake (FoE is right about local carbon budgets), but loading extra duties onto councils (no matter how good they are at them - some have done brilliantly) without extra resources is always going to be a problem - and extra resources are unlikely. However, in principle, it is wrong - it will get pushed to the bottom and the responsible people will lose their jobs, which is a shame as councils can do a lot when they want to (and have the resources to). Taking money away from councils while "empowering" them is a rather curious aspect of government policy. Sadly we didn't raise these issues in detail...
We discussed the carbon targets. The CCC says the 2020s targets will require by 2025 that we have 2.6 million heat pumps and 31% of new cars are electric. The former seems unlikely: They are technically eligible for the Green Deal, but are likely way too expensive (unless prices come down) for the limited subsidy available over that period (they certainly won't pay for themselves at any realistic interest rate without a subsidy in most cases, although there are corner cases such as people with a convenient lake). Goodwill said that's likely to be up to the market for the time being. The government seems to hope that prices for solid wall insulation will come down with scale, so the under-resourcing of the Green Deal/Energy Company Obligation will be transformed into a comprehensive and rapid refit; if they are right, why not set a target as per the Warm Homes Amendment? We'll see. We briefly touched on wood and other biofuels - he seemed to support the general consensus that only a few biofuels are sustainable, while pointing out that some are.
We talked about transport for some length (Robert being the former shadow roads minister, and transport being a non-emissions-traded sector along with heat i.e. more amenable to government policy). He doubts that the target above is feasible. There is work going on on infrastructure, and batteries will be better, so we'll see; it is probably a question of technology and price, we didn't discuss specific subsidies.
We later discovered that he was partly responsible for the government's abolition of speed cameras, and grilled him a bit about this, discussing some options including limiters and traffic lights linked to speed cameras; nothing that was said makes me doubt my view that abolishing speed cameras was a purely populist measure! He is in favour of widening the A61 at least in parts to avoid overtaking safety issues, as were some of the people present (there's a rule that you should avoid disagreeing with each other - but it's harmless enough here given the scope and time). We discussed transport targets, and hybrids, which hold some promise, but improvements in efficiency, just as improvements in traffic congestion, seem to be mostly eaten up by increased mileage, whereas electric cars avoid this; he seemed to accept this point, and pointed out that roads are important for the economy - a point that we didn't dispute, although I believe we discussed rail freight at our last meeting in November. Some of us are committed (and bordering on militant!) pedestrians, some are clearly drivers, but there is little point in picking a fight on that.
Robert pointed out that the coach is the greenest mode of transport right now, although this will change with green electricity. He seemed to accept that it's likely that fewer people will have cars in future and the long term trend is for petrol/diesel prices to rise. He said we can't just ban petrol cars - I countered that we can do exactly that, but only once there is a realistic alternative, which unfortunately most electrics at present are not. He still believes in the hydrogen economy. I don't, but I hope he's right. It would have been a good point to poke holes in the government's cuts to public transport in general, especially buses and coaches, but also while HS2 and some electrification is going ahead, rail is continuing to get more expensive and needs more help. As I've already mentioned we weren't all that combative, and I'm not convinced it would have been all that helpful - the government isn't about to reintroduce the subsidies it just cut. It was certainly worth talking about transport, we might have done a little better but it's still useful for him to hear from us.
As we pointed out, the government has done some great things and some terrible things. Great things include the new carbon targets. Terrible things include the Green Investment Bank being unable to lend until 2015, which we discussed a little (and he gave me a paper), and the shambles of the solar PV reform. He mentioned a local business involved in PV (having earlier mentioned a local business recycling plastic), and was sympathetic to both concerns - in any case there isn't much to be done about either now. We agreed that the objective with PV is for the technology to develop and become cheaper (some PV companies expect radical improvements) - but it is necessary to keep the pressure up with subsidies for the time being, especially with Spain at least abolishing them due to the bail-out. There are also worries about rare earth supply. It was pointed out that the size limit on PV prevents larger, more efficient installations.
We discussed nuclear and the grid. I explained that intermittency generally can be managed: Given big interconnects, a small amount of storage, and smart appliances, a 100% renewable grid is possible (according to a number of studies, both here and elsewhere), and we can and should export renewable electricity as we have the biggest offshore resources in Europe. And offshore wind is coming down in price quickly, unlike nuclear. Jane made the usual attacks on nuclear - Robert pointed out that Canada is much further down the road of what to do with its nuclear waste than we are. He is pro-nuclear, and broadly supports the CCC's vision of 40% nuclear, 40% renewables and 20% coal with carbon capture by 2030; he is a little doubtful about carbon capture (so am I!). We agreed that as long as the grid is 100% low carbon by 2030 we'll be happy.
So all in all, we got together and reminded our MP that some of his constituents really do care about climate change, that fuel poverty is a huge issue here with 23% of residents affected (as mentioned in the letter we handed in), we discussed some specific ideas in accordance with the Big Climate Reconnection, and many related things, and gained some new contacts - notably Sue, her husband Ross (who I met later), both of whom have volunteered for our next event, and Liz. And we hopefully got some local press coverage (probably on next Thursday's Green Page) - which in Scarborough is pretty easy, but has an impact.
Next month, we will be participating in the Campaign against Climate Change's Zero Carbon Britain Day. This will be purely a publicity stunt, aiming for some press coverage and maybe to chat to a few people; we don't currently have the resources for a full blown outreach event. On the 16th of July we will be wading into the sea with (waterproof!) placards spelling out "Zero Carbon Britain/Scarborough by 2030 Before It's Too Late"; the photos will be circulated in the local press and on CCC's national website. We now have half a dozen volunteers and a photographer, so this should go well, and should raise some attention on the need for radical action. We are not solely concerned with sea level rise here, although that's a big concern for a lot of countries, but with all of the impacts - many of which are happening today, and will get much worse. This is mainly aimed at the local people (and the national audience via the CCC's website collage), but pushing a radical but technically feasible demand is a good thing. The government targets are for 60% by 2030, including full decarbonisation of the electricity supply. We can and will ask for more - 100% by 2030 is feasible - but it is equally important to ask for specific policies, such as the Warm Homes Amendment. Aggressive long term targets are great, and form the framework for short term policies, and aid in holding the government to account for broken promises, but it's mostly the short term policies that will actually change things in the UK.
Globally, we're all waiting for the US - but the more we do, the better, partly because it builds technologies (which is not only a matter of basic research but also of deployment), partly because it shows what is feasible, and partly because we have a responsibility to act about our part of the problem. And the US is not the only part of the negotiations: Most of the Africa block and the Least Developed Countries and Small Island States support a 1.5 degree target, which technically would be extremely difficult, as they will be hardest hit - and are being hit hard already. Adaptation funding is critical for them: It must be raised in a predictable way and in sufficient quantity (hundreds of billions of dollars per annum by 2020), and dispersed fairly in accordance with need to adapt and grow out of poverty sustainably and lack of ability to do this themselves. And then there's China, which has recently improved considerably, investing vast amounts in the new renewable industries that we should be investing in, and for much the same reason - to be a market leader, but also because they want to grow sustainably (and are openly admitting that some aspects of growth may need to be managed to avoid problems). They have a long way to go, but they're a long way from where the US is. All we can do is do our part locally, push for stronger targets in the EU (which fortunately the UK government is doing, at least in public), and sort out adaptation/mitigation funding for those hit hardest who are least able to adapt. Or we could implement a comprehensive trade embargo against the US, but unfortunately they're such a huge part of the economy that this won't happen; and if it did it would probably cause all manner of problems (and not just economic ones). Better to do what we can. If we despair and do nothing, it will certainly be significantly worse.
To clarify that last point: Two degrees is an important threshold. The IEA says we're probably going to miss it. Loads of analyses say we're probably going to miss it based on current international targets (these need to increase dramatically!). Even at 2 degrees, a lot of people will be hit very hard. The risk of feedbacks and a "point of no return" increases with the temperature rise (with some of them already having an effect, for instance methane and carbon dioxide releases from melting permafrost). It is essential that we do everything possible to avoid a 2 degree rise in temperature. In fact it is essential that we cut our carbon emissions as quickly as possible - but this is speaking globally, so developing and deploying technology is more important than any primitivist urge to end industrial society that no other country would follow (behaviour will have to change quite radically though e.g. air travel is unacceptable, period). The fact that we are close to the edge doesn't mean we should give up, it means we should double our efforts. Of course we should think about adaptation - many countries need to adapt right now - and IMHO also about geoengineering, as an expensive and risky last resort (ranging from expensive and very slow measures such as extracting carbon from the atmosphere, to expensive, fast and very risky measures such as engineering a mini-nuclear winter) - but it will be vastly cheaper to cut our carbon emissions now. And it may even grow our economy - and in a sustainable way, unlike the financial services and house prices bubble!
So what's going on in Freenet? (freenet)
Freetalk: Xor is still working on optimisations. Freetalk and WoT currently use far more resources than they need to. Some database related changes have been deployed already, then hopefully he will be working on finishing the event-driven changes that will avoid a lot of unnecessary work (disk and cpu) in Freetalk and also fix the problem of messages appearing and disappearing (which I've seen recently and others report as a common and major problem). After that, hopefully some work on scalability, so that people who you've given a high trust get polled more often. All of this is necessary before we release Freenet 0.8.
New Load Management: There are still a few more bugs, but this is rapidly approaching a point where we can deploy it. One difficulty is I am still seeing very high latency on some requests. New load management introduces queueing to make sure we can get to a reasonably good route, but the impact should be limited. Having been unable to work out the likely impact from first principles I will have to write a simple simulator soon to estimate the likely impact on latency. We may keep old load management for realtime requests and just use new load management on bulk requests, which are more tolerant of higher latencies. Overall throughput should improve, and when it is applied to inserts, data retention should improve significantly. And it should also improve security, making Freenet less vulnerable to DoS attacks via spamming requests (although this is greatly reduced thanks to last year's fair sharing changes already). Hopefully new load management will be sorted out fairly soon i.e. within weeks not months. Assuming we can make it work, this is the ultimate answer to NowWhat's thoughts on performance.
Darknet enhancements: Much more needs to be done. This is important for 0.8 because opennet is insecure. Recently we have deployed code to allow you to set a trust level and visibility level for each of your friends; this will be used in the darknet enhancements, the core feature being connecting to friends of friends, if they are visible (which enables better connectivity, better performance, and better invites). We really, really need to build the darknet!
Library search plugin: It has become increasingly clear that presenting users with a search box which takes ages, usually fails, and occasionally causes the whole node to crash (because they searched for a too popular term) is unacceptable. We need to fix this. And if we can't fix it, we should remove it.
User interface enhancements: Two volunteers are doing good work on this, including operhiem1's recent changes to make it easier to choose folders and where you want to save a download to. Pouyanster will be working on this too, his last change being the country detection code. ArneBab and others have been working on the themes.
Freemail: Zidel is working on this for his Summer of Code project, including giving it a proper user interface, Web of Trust integration and anti-spam measures, and lots of bug fixes (probably including shorter addresses).
New freenet-ext.jar and 1.6 support: infinity0's (Ximin's) work on freenet-ext.jar has finally become official, although there is another stage yet. The next stage will involve breaking up freenet-ext.jar into many smaller jars, bundling a few extra libraries, and merging some code dependant on them (including new filters and code to reduce our disk I/O footprint on Windows).
Plugins: Sone needs to be made official as soon as possible. It's an important piece of functionality. It also needs lots of optimisation work. We should also think about jfniki.
What about China and Iran?
The rumours that China has deployed a whitelist system that kicks in once you have exceeded some threshold for international traffic so far appear to be bogus. However, it is vital that we build a friend-to-friend darknet in China. It appears that generally connectivity is pretty good, and in the past we've been very popular when there was a lot of chinese content on Freenet. We need more, and we need easy to use and ideally close to realtime chat or microblogging systems. Making it easier to mirror large sites onto Freenet would probably help too.
Iran is talking about having its own national internet, with only critical businesses allowed access to the external internet - joining such places as Burma and North Korea. We'll see. A friend to friend darknet could of course function on a national intranet, but if they accept that level of collateral damage they probably are quite capable of wiping out a darknet too. On the other hand, the economic impact is likely to be so huge that they will be a useful example to the rest of the world of how not to regulate the internet - but that's not much of a silver lining.
What about performance?
People are still complaining about Freenet having less performance than it used to. Inserts are, apparently, very slow, and downloads are still slower than they used to be. This may be partly because of better data retention, and hopefully new load management will help in a significant way. Also, the network has been shrinking, and this may be disruptive. Obviously we want to reverse that trend, but the main way to do that is to get 0.8 out.
What about Tahrir?
Your guess is as good as mine. I wish Ian luck - building darknets that can work in hostile environments is a good thing. Although IMHO for China Freenet is probably a usable tool, or will be very soon.
What about Bitcoin?
After a fairly serious crisis, I concluded that Bitcoin is unlikely to survive long once it becomes legally impossible to exchange it for bank money. A criminal network could provide exchange services, but given what they'd be likely to charge, there isn't going to be enough demand: Agorism is not going to happen (much to my relief). I expect such a crackdown will happen within the next few years - just as it did with e-gold and all the other pseudo-anonymous online currencies. New Scientist ran an article about it this week, apparently the total currency supply is worth $50M. I am also of the view that Bitcoin's dollar exchange rate is somewhere between a bubble and a pyramid scheme...
In principle, money is a form of information - but there is a difference between "freedom of expression in any form" and "freedom of exchange of money". But it seems highly unlikely that Bitcoin will cause any major damage (money laundering, open markets for assassinations, illegal porn and terrorism, tax evasion etc), because it can't survive illegally: There isn't enough demand for anything you can get through bitcoin (given the likely tens of percentage points margins) to justify a large criminal network exchanging it.
Do we understand Freenet?
Sort of! My reply to a question on FMS follows. A lot of things in Freenet are just a matter of good engineering (i.e. having enough resources), but some are not. Here goes:
Do the developers themselves still understand all their code, their design and the architecture of Freenet? Did anyone ever draw sketches, diagrams or similar? Can anyone explain the internals of Freenet to other new people who want to get involved? There seem to be that many bugs in Freenet, that I hardly know where to start to begin with. Bugreporting, but what area, code dissecting but what parts, and the list goes on and on.
There are some aspects of Freenet that are at least a little empirical/experimental in nature. Given limited data gathering we are not very good at this, but the nature of the beast is that we don't necessarily understand every detail in advance. This is especially painful given that all our theoreticians leave after a while, so we often don't have adequate simulations (oh and I'm not allowed to make simulations or people get really upset and in any case I don't have time).
Sometimes we just have to try things. In the case of RecentlyFailed, we have had the theory more or less understood for many years (but not simulated), but not got around to implementing it until recently. It was necessary to implement it because we needed to significantly increase our capacity for USK subscriptions. It is one part of a package that will hopefully allow chat apps to be adequately scalable, which is IMHO critical. It WAS tested in very limited simulations (just to check that ULPRs work i.e. that on a small network if everyone has requested a key they will get it reasonably quickly after it is dropped onto one of the node), it was tested on the whole of the testnet, and some testing on the main network, but right now the testnet is less than 15 nodes at any given time, so it's not an adequate testbed for network level changes.
So we just deployed it to see what happens. Freenet is not yet 1.0, this change does not have any significant security impact, it seemed to more or less work in all the ways we were able to easily test it, and it was - as I saw it - absolutely vital in the short to medium term due to the likely expansion of chat applications and other USK polling and SSK requesting stuff.
On top of all that, even if the theory is perfect, there are two more problems:
Bottom line is we have severely limited manpower, inadequate testing resources, and some parts of what we are trying to do are extremely difficult. You can help by reporting bugs, by running a testnet node, and so on.
Recycled post consumer waste (freenet, climate)
Unfortunately, "The Great Global Warming Swindle" has come to Freenet. This "documentary" consists largely of recycled climate denier arguments which have to be debunked every time they are wheeled out again. It was originally made in the 90s and reconstructed with the same largely bogus experts in 2007, and aired by Channel 4 to grab some viewing figures. They added in some misrepresented quotes from a real expert, who subsequently complained about it, some extremely bogus and massively outdated graphs with mislabelled axes, and a total lack of any care at Channel 4 for anything other than viewing figures, and hey presto, a great propaganda piece that did wonders for public opinion - in 2007. A lot of effort was put into debunking it since then, notably a detailed formal complaint to the regulator. Now, maybe TV shouldn't be regulated - the americans take this view, and Freenet clearly isn't regulated. But, especially in an unregulated medium, you should never believe anything just because it looks authoritative. People believed it because it was broadcast on what they expected to be an authoritative, regulated medium. Google it, read the rebuttals from Real Climate, CCC's collection (including one from Sir John Houghton), the official complaint to Ofcom, the misrepresented scientist, and more; it's not all that hard to check the facts, but he who controls Google controls the world... All of this should not need to be repeated, and the documentary itself is in no danger of being censored, but the wierd community that is Freenet attracts conspiracy nuts of all kinds, as well as more unpleasant people. It is depressing to think that ultimately almost everyone believes what they want to believe and the facts be damned, but it appears to be the default state of human nature: for instance, a great many americans believed that Saddam Hussein was behind 9/11 for years afterwards.
Check the facts yourself, believe some specific person, or follow the consensus. Those are your options. Me, I prefer to check the facts, and if I can't do that I'll usually go with the consensus. Of course, that's the theory - in practice I'm amenable to social pressure like everyone else, but only up to a point. For instance, I tolerate my friends' belief in homoeopathy to the point of using it as a first line - but I don't avoid medical treatment as a result, because most likely homoeopathy is a placebo.
The sort of conspiracies that would be needed for climate change to be a hoax are considerably larger than would be needed to fake the moon landings. Look at the weather over the last 10 years for instance: one event is not evidence, but the global trend is very clear, and as a result of that, there are so many record events one on top of another to make the point very clearly. Even the cold winters lately in europe and america (at the same time as record high temperatures, floods and fires in the rest of the world) are likely partly due to the melting ice caps.
Hope after all? (climate)
Climate Progress has published a summary of rebuttals to the paper that spawned the New Scientist story that the last post was based on (I read this but not the longer paper). Basically it was arguing that there is a limit on how much wind we can extract from the atmosphere, that it's disturbingly low compared to current and likely demand, and that if we use more than a fraction of it we will see major disruptions to rainfall and similar problems.
Surprisingly, Climate Progress's prescription for clean energy is probably unrealistic as it is based on 450ppm and requires a fair bit of biomass, which is looking very doubtful. But they also link to this piece, which provides a short and sharp rebuttal, explaining that the models are bogus because modelling a wind turbine as a slight increase in the friction coefficient within a particular simulation cell (= vast chunk of the virtual earth) is bogus, and the estimates for maximum available power are also bogus.
One more thing: The New Scientist article and the related editorial hint at energy use itself becoming a warming effect. This is what a lot of the quotes on the Climate Progress page refer to. This is orders of magnitude below the warming resulting from CO2 emissions.
So perhaps we can have our cake and eat it after all. By which I mean (at least in Europe, with slight changes elsewhere) a Green Grid composed of approximately 60% wind, much of it onshore, mostly the existing hydro, a bit of solar and marine power and so on, with big interconnects and some storage and dynamic demand technology, providing cheap clean electricity. Not only is it the cheapest option, and therefore the easiest to get past politicians, it's also the best option in terms of keeping the environmental movement together.
Finally I'd like to clarify that I'm not a climate scientist. I'm a human being who cares about many things. This blog has always been about whatever I happen to feel like writing about. Mostly that's Freenet, but often it's climate, sometimes it's online politics or ordinary politics or religion (or anti-religion) or something else entirely. Speaking of which...
Brief Freenet update
I've updated the roadmap. You might be interested.
Currently I am working on USKs, to speed up Freetalk: Implementing bug 4660 (only tell the client when we're fairly sure we have the latest version), and then I will be working on date-based hints for USKs (we've inserted them since a few months back but we haven't used them; the result will be we can get up to the current version of a site much more quickly and cheaply). There appear to be some bugs in the USK polling code; I'm not sure if these are bugs in what I've just added or deeper issues, but they're gonna get fixed either way, and if it is the latter, then that's pretty important as p0s has mentioned stalls...
Then the next thing is a little more work on the connection level, related to network congestion. After that, I'm not sure whether we will need network-level changes for Freetalk (e.g. RecentlyFailed) in the short term; if not, probably some more on new load management, maybe finally get some form of it deployed. Or even skip it in favour of stabilising towards a release, since Freenet seems to be working reasonably well at the moment...
We now have only $6400 in the bank, meaning apart from what I just got paid ($5000, about 2 months), we only have another 2 and a bit months left of funding. So donations would be nice, whether by bitcoin or paypal or some other means. But content, code contributions and the like can be even more valuable. Seeya folks.
All hell? (climate) (UPDATE: SEE FOLLOWING POST FOR REBUTTAL)
This article is extremely worrying. The claims it makes, and related issues, are that:
The consequences of this would be serious: Right now we only use 30GW or so of wind globally, but a lot of studies on how to power the world without nuclear power or fossil fuels relies heavily on wind. These are called the "green grid" (full article, diagram): If you link up all the national electricity grids, using high voltage DC where you have really long links, then for example you can provide Europe's electricity needs with a lot of wind, the existing Northern countries' hydro, a bit of solar and a bit of biomass. With the overall cost being reasonable - comparable to wholesale electricity prices, a bit more than coal - although requiring massive investment (which would have to happen anyway as most of the electricity infrastructure needs replacing soon). More storage wasn't strictly necessary but may be available through electric cars and new batter technologies, and "dynamic demand" helps a lot too; neither of these were taken into account in the early studies so there's probably some flexibility. One thing that was never clear to me was whether this included increased demand for transport and heating.
Many green groups had bought into that agenda quite heavily, partly because it doesn't need nuclear, which is problematic in a lot of ways, or carbon capture, which was only ever going to be a transitional technology and is much dirtier than the alternatives when you consider the full fuel cycle. I did personally. If it doesn't work, we have serious problems. First, what are we going to build our electricity grids on? (Considering only generation, assuming that big grids, dynamic demand and a little more storage happen)
All this is extremely annoying to radical but technocratic greens like me (as opposed to Paul Kingsnorth's deep ecology), and may do a lot of damage to people like the UK Green Party and Friends of the Earth. It's actually not far from James Lovelock, who is an iconic eco-pessimist. In the shorter term it will provide yet more fodder for those who want to do nothing. And most importantly of all, it will increase the costs of eliminating fossil fuels and avoiding a 4 degrees or worse world by at least a factor of 2 and probably more. Right now onshore wind is significantly cheaper than nuclear, and arguably much of the subsidy it receives is simply a correction for the odd way that the UK energy markets work (when the wind blows we don't need to turn on the expensive gas plants, and on those days even the coal plants get the same fee as the most expensive gas plant; I can't find the article on this, I think it was in the Guardian's Comment is Free), although it is no longer falling in price due to issues with materials. Offshore wind, nuclear and coal with carbon capture are all rather more expensive, and most of the rest (e.g. tidal) are early days and very expensive; solar photovoltaics even more so.
So the situation is extremely worrying. Doing enough of the easy things was meeting with a lot of opposition (for example, the EU will not move to a 30% target in spite of it probably increasing their GDP). With the new data, the great clean energy transformation that most of the greens had been asking for is going to be a lot more expensive and a lot less clean, a lot harder to argue for and likely lose a lot of campaigners. So it's all rather depressing. If it's real - there was another study a few years back that forecast some changes to wind circulation; it's early days, one recently published paper, it may be that better models yield significantly different answers for instance.
The article was published in the print edition of April 2nd, and on the website on March 31st; the paper was published in February. So I'm fairly confident it's not an April fools. And if it was it would be a horrendously irresponsible one: stuff like this tends to get quoted endlessly even after it's been retracted. Maybe I'm contributing to that process here ... I will certainly post an update if I hear that it is bogus.
Another interesting note, vaguely Freenet-related: The Met Office 4 degrees map appears to have disappeared. First it was moved to archives, now it's gone completely.
Which is a shame because it illustrated the problem quite well: First, temperature changes are not evenly distributed across the world. In the arctic, and in some parts of asia we'd see over 10 degrees, and 5 plus in a lot of other places; Africa hit hard, but also a lot of mediterranean europe. Second, it's not just about temperature: Temperature does directly affect food production, but it also drives more extreme weather, changes in precipitation, sea level rise and so on. Remember last year? No, not the record cold temperatures in the West (probably caused by arctic melting), the record high temperatures in much of the rest of the world and drastic consequences in Pakistan, Russia, Australia; severe droughts followed by flooding have been happening in large parts of Africa more or less every other year for the last decade. For some more doom and gloom, see Climate Progress's summary of big climate science stories of 2010 which were largely invisible thanks to a lot of bullshit about some stolen emails...
That map was regarded as authoritative, and everyone linked to it. And now it's gone, and it's pretty hard to find anything like it. What do you expect from the "greenest government ever" that has declared an end to the war on the motorist (in particular essentially abolishing speed cameras - which I hold my MP personally responsible for as he was shadow roads in opposition - and the fuel duty escalator)? Some people in the government clearly do care a lot about climate change; this is probably one of the better things resulting from the coalition. It still supports a 30% target for Europe, although this is not going to happen; it will introduce a floor price for carbon, but initially very low, and only in the power sector (heavy industry, despite its claims, has extremely favourable treatment, usually at the expense of the power sector, see
Scary stuff, depressing. Just thought I'd share it with you, to try to increase the general level of understanding and spread the pain... I will still be doing the Big Climate Reconnection, as its demands are perfectly valid and even more urgent than ever (especially if the implementation will be more expensive than expected), but for now, on with Freenet!
Lots going on
A whole lot of code has landed lately in Freenet:
It's great to have a lot of volunteers! It makes it a lot easier to get on with Freenet if you're not alone, and ultimately Freenet will only be sustainable if there are plenty of people working on it - without being paid. I get paid for this, out of the money that Freenet Project Incorporated manages to raise - most of it from Google in the last few years, but we can't keep relying on them (I doubt they'd give us any more if we asked for it!). We have raised a few hundred through bitcoin, probably more than that through paypal in the same period. We'll need more if I'm to keep going, which IMHO is a good thing at this stage. But long term, Freenet needs to be able to stand on its own two feet, and if necessary, to be developed underground - that is, over Freenet itself - by anonymous developers. Of course, the flipside is Freenet can't afford for me to be doing volunteer coordination, quality control and release management, I'm supposed to be a full time coder, and most of the stuff that volunteers work on - with the notable exception of Freetalk - isn't essential to the critical path to the next release. So there is some tension there, especially with the current policy that all released code will be briefly reviewed for security issues at some point; but mostly it works out.
Meanwhile, I've been working on low- and mid- level code, mostly debugging timeouts on the testnet. Most of that code is in build 1358. But there is a big chunk of it on master and testnet still, which will be deployed soon. The problem is, we could accept a load of realtime requests for a single peer - up to half our capacity - because a bit of burstiness is a good thing for realtime requests; in fact it's what makes realtime requests work. But the packet scheduler was still strict round-robin between all peers that have data queued - even if this data is only bulk data, which has much longer timeouts. The result was timeouts on the realtime data, even when there is no link level congestion. The initial fix was to just not overallocate to a peer. That worked well in terms of avoiding timeouts, but unfortunately it caused a significant slowdown especially on local inserts. The better solution is to change packet scheduling to go by deadlines - which has been implemented and is being tested on testnet. More testnet testers are of course always welcome, as is inserting and fetching stuff - actually testing testnet!
What next? Most of the above should be deployed early next week. Hopefully the low level stuff will work. Unfortunately there is more low level stuff that needs dealing with, particularly related to congestion control. After that, hopefully fatal timeouts will be a largely solved problem, so we can enable disconnect on fatal timeout on opennet. After that, deploying more chunks of new load management. If we eventually get that out the door, no doubt there will be endless debugging and tuning; the next release critical tasks will be the darknet enhancements and Freetalk-related SSK/USK changes. Hopefully we can get away with DBR hint support, bugfixes, and RecentlyFailed/failure tables request quenching, and won't need full blown passive requests until well into 0.9. Also some people would greatly appreciate deploying the new low-I/O datastore, and no doubt there will be lots more debugging. That's (some of) my TODO list, other people are working on other things. For instance, hopefully Freetalk will have good support for attaching files in the not too distant future, and (optional) image boards, as well as a lot of optimisation work.
I've had a lot of positive feedback lately. It appears that 1358 has made a significant difference. I don't know whether download speeds are back up to where they should be, but we seem to be moving in the right direction. I read Freetalk and FMS, so let me know your experiences of Freenet performance, and indeed any other problems you find; usability testing (get somebody new to install Freenet, don't help them unless they get really stuck, and tell us what they didn't understand or got stuck on) is particularly helpful.
Wandering in and out of the wilderness
For a while I've been wandering in the dark, blocked unsure of what connects to what, when we will see major progress, whether Freenet will ever amount to anything, finding it difficult to get any work done. There are other factors but I believe a large part of this is directly about work.
However, I'm pretty sure I'm back on track now. After implementing the testnet, I've been able to identify a wide range of low-level to mid-level bugs, related to packets, messages, and requests. Most recently I have some significant leads on the trail of "why do realtime requests all fail when you do a bunch of them at once", which has bugged us ever since the realtime/bulk flag was introduced. I have fixed lots of bugs along the way, some minor, some serious. The next step will be more testing locally, more debugging, deploying it on the testnet, even more debugging, and then deploying the fixes in a new build. 1357-pre1 contains a lot of the fixes and I'd appreciate testing of that, but there will likely be significant changes before 1357 is released.
Hopefully the current phase of low level debugging will come to an end fairly soon, with significant performance and stability gains deployed widely. One particularly important issue is eliminating all bogus fatal timeouts. Timeouts are tricky, mostly because new load management needs us to know exactly how many of our requests are running on our peers. Originally we would send a request and then if we didn't get a response in a set period, we would move on to the next node. We still do this, but we also wait in parallel a longer period (long enough that if the node has just gone offline we will disconnect it), and if it doesn't reply in this longer period, we declare a "fatal timeout". Currently we disconnect from darknet nodes when we get a fatal timeout (they immediately reconnect). The plan was always to disconnect from opennet nodes too (and not reconnect), but there were lots of bugs causing bogus fatal timeouts. I have now fixed most of those bugs and see very few fatal timeouts; the next step will be to reinstate dropping opennet peers when they get a fatal timeout, which will allow us to proceed with new load management.
New load management has of course caused all sorts of side issues, some of which have been huge (like new packet format, or the work on timeouts I mentioned above). The testnet has paid off enormously with the low level stuff, well worth the effort of (re)implementing it. If you can run a testnet node please do; there is a separate installer, see the previous post.
Hopefully new load management will make a significant difference to routing accuracy and hence, eventually, all kinds of performance but particularly data retention. A network that stores data for 20 weeks is far more useful than a network that stores data for 2 weeks. This is a big issue. It will also significantly reduce the network's vulnerability to some kinds of attack. Last year it was a big issue, and I introduced some bugs which caused major problems for the network, and then decided (foolishly, but after having tried to investigate the problems for quite some time) that load management was the problem and got on with implementing it; eventually I figured out what the problems were. Some of the side-issues involved in implementing new load management have caused serious knock-on effects due to temporary disruption, or bugs in those changes. New packet format was a Summer of Code project, seemed to be pretty much ready, and had a very large impact, but it solved some long standing problems, increasing the average payload percentage dramatically especially for slow nodes. Since then I've spent a fair amount of time debugging it, but it's still been worth it. Anyhow, I'm sorry that I didn't realise that the problems last year were due to bugs; but I make no apologies for focusing on new load management, which remains vitally important, and which I really think will land in the relatively near future, in some form.
There has been a significant reduction in performance over the last year or so. Some of it is due to bugs created during new load management, but some of it may be due to FMS and Freetalk having scaling issues. Fundamentally Web of Trust style chat apps rely on outbox polling, which in a naive implementation (like the current one) can't scale. One reason we merged new packet format when we did was to increase the network's capacity for failed SSK requests, which these chat applications do a lot of. Eventually it will likely be necessary to make fairly significant changes both to the chat apps and to Freenet to support this - especially if Freetalk's being deployed by default results in a lot more active users. The chat apps will need to poll a trusted subset of the identities rather than trying to regularly poll everyone, and will likely rely on hints to propagate new versions. Freenet will need to provide more efficient polling - in the short to medium term this means implementing RecentlyFailed, which is related to the old failure tables that some oldies may remember - since we already get a notification if a key turns up within half an hour, we can safely stop requests from other nodes for the same key, as long as it's been requested a few times recently; if it is found, we will propagate it to those requesters too.
In the longer term, we will probably need to implement full blown passive requests. These will be a lot of work, but will be worth it, greatly improving performance for chat-style apps and anything else that needs fast notifications of updates, greatly reducing their impact on the network, and opening up new possibilities such as real publish/subscribe. They will rely on something close to new load management, and will pave the way for long-term persistent requests, which may make high-latency transports such as sneakernet viable. For instance swapping USB keys with your friends when you meet them, or getting your phone to do the same automatically with theirs; this would work even if the internet is completely locked down, or not available at all. Haggle does something like the latter, but does it in an "opennet" fashion, so it works as long as it's tolerated; Freenet could do it just with friends, allowing a large amount of data to be accessible within a longish period, but probably with fairly good bandwidth, minimising spam, allowing publish/subscribe broadcasts of new content, and so on, although there are many big challenges before we reach that point.
Will this be enough to make chat scale to millions of users? Probably not, although "millions of users" is a long way off! But combined with some tricks on the client side it should work well enough. Long term, Twitter-like functionality (especially if you're doing more following than searching) may actually work better with Freenet than chat forums do - although of course this is at the opposite end of the latency scale. Or is it? Ian's work on Tahrir sparked some interesting discussions on how to protect a small upload - a tweet, an SSK - where it doesn't matter if it's delayed by an hour or so. This will likely result in some basic onion routing in Freenet eventually, to protect chat posts, SSK top blocks for splitfiles, and so on. Which would enhance security significantly for some very interesting scenarios - specifically a large darknet where the attacker cannot easily connect to all nodes, but can work his way across the network at some expense - without costing much performance, because the CHKs are just inserted as normal. As I've explained many times, if your peers are compromised, so are you, and it will be a long time before we can do much about this, and if we can it'll likely be at a major performance cost. But if you use darknet, if the bad guy is initially a long way away, and if he can only hack his way across the social network slowly, compromising or social engineering at each hop, Freenet provides not only near invisibility and unblockability short of really drastic steps such as prohibiting all p2p connections (assuming we implement some basic stego), but also a reasonably interesting level of security. One key point in Freenet's favour is because it is high level, because it doesn't try to anonymise connections, it is distributed, robust (at least it should be if we store data efficiently and sort out the Pitch Black attack), and can use whatever latency the user is happy with - which varies from one application to another. For large downloads, sending the request out over some low bandwidth stego and then getting the data back on a USB key may be a viable option for a lot of people. Likewise, provided we have a good UI, blog updates don't mind latency that much.
Which brings me to Freemail. Zidel, after writing the new packet format (which I had to do a lot of work on afterwards, but his code was good nonetheless, a lot of what I did was extending it), has done a fair amount of work on debugging Freemail. This will be deployed as soon as I get around to it. IMHO Freemail is really key for a lot of users - maybe not by numbers, but by need and dedication, and it will help to cement the community and keep people onboard. Now it just needs a decent web interface - assuming he's fixed the bugs that made it very unreliable for me, which I think he may well have done.
Freetalk is at RC2, and RC3 is imminent, according to p0s; RC3 may very well be final, although there are some hard to reproduce trust recalculation bugs. When it is deployed we will really see whether the scalability issues I mentioned above are going to kill the network...
There are also discussions going on about wiki's and backups. I won't cover them here, for now - go read them on Freetalk and FMS (which is remarkably hard to setup for a not very techie newbie I met, we need Freetalk now!)
My view of release policy remains essentially the same: We should not Freenet 0.8 until the network has settled down, new load management has been deployed and debugged, and the long-promised darknet enhancements have been implemented. Ian would like to release 0.8 as soon as possible; I agree with him that we need a release soon, for lots of reasons, but we can't release with a severely broken network, I'm convinced it's worth completing new load management, and the darknet enhancements are both essential (in that opennet is far less secure than darknet and we don't want to encourage people who really need freenet to use opennet), and are relatively small changes. Deadlines are good for preventing feature creep, and we will need a deadline soon, but not yet; feature creep is not the main problem at the moment.
The planned darknet enhancements should be really helpful for anyone trying to use the darknet, and should help it to expand much more quickly; the key features are the ability to make your friends visible to your friends, to connect to your friends' friends automatically, to manually upgrade them to full friends (exchanging a password to confirm identities), and to invite friends to Freenet, with the invite containing the installer, the noderefs of your node and some of your friends, enough to get online even if you are behind a nasty NAT or are even offline at the time. We will probably add some opennet support too, in case the seednodes are blocked, although this will require turning on seed mode on many more nodes. Long term, we will need this: Opennet is just far too easy to block, or worse, to comprehensively surveil. We must build a global darknet, or Freenet is finished, or worse, provides a false sense of security. Which is not to say that opennet isn't useful in the meantime.
Why Freenet, again
The question of "Why Freenet?" has come up recently. I posted an explanation of my reasoning way down near the bottom of this blog, but I have changed a lot since then; most notably I have ceased to call myself a Christian and have developed a great skepticism about how some kinds of Christians view circumstances, anecdotes, and divine sovereignty. Plus, some of the places I'd hoped Freenet might help have started to erupt into revolutions without it. So do I still have an answer? Yes.
The simple answer is freedom of speech matters. A world with Freenet is better than a world without Freenet. Freedom of speech is vital for democracy. In Western countries, freedom of speech is constantly under threat, especially online, from both corporate and government agendas. Blocking websites, attacking peer to peer, and so on; he who controls Google, Twitter and Facebook controls the world as it is perceived by many. Plus, it is very hard to host a controversial and popular website. If it is on a small hoster, it is vulnerable to DoS attacks, and if it is on a big hoster, it is vulnerable to political interference; this we saw with Wikileaks (who are doing a great service to the world). Wherever it is it is vulnerable to legal attacks and paralegal harassment (think for instance anti-Scientology sites). And probably to copyright-based blocking too in the not too distant future - as the MPAA suggested during discussions on ACTA.
In the rest of the world, where democracy is a daydream, there is overt political censorship, and speaking your mind usually means at the very least ignoring Facebook's real name rule - and you can probably still be traced (and then imprisoned, tortured, disappeared etc). For instance, China not only has extensive (if eccentric) political censorship, they've also managed to block most of the Tor bridges (not only the public nodes). If it is technically possible to guarantee freedom of speech online, in the broadest possible sense of consensual data exchange, then it's a good thing. Of course, a lot of what Freenet is used for today is pornography (exploitative and degrading of both the producers and the consumers), illegal pornography (far worse), or casual copyright infringement (undermines efforts to build a real alternative), plus some legitimate but very distasteful political speech (holocaust denial e.g.; looking more broadly, the whole Climategate farce was seen as a "leak"). That's an unavoidable consequence of freedom of speech. The US courts recognise that free speech is sometimes anonymous speech, partly for historical reasons; I doubt that european courts do, but it is fundamental.
Fortunately most of the old technological-level threats to freedom of speech online have receded somewhat (e.g. trusted computing is nowhere), although new ones have popped up in their place. National blocking systems, legislative requirements on peer to peer and web services, three strikes laws (extrajudicial punishment!), and the increasing centralisation of just about everything on the internet, all clouds on the horizon. Freenet needs to be secure, but it also needs to be popular. It could perhaps be blocked by excessive force on the part of governments - such as blocking all peer to peer - but that is probably a very long way off, and may not happen at all, and in places where there is a greater need, some sneakernet-based network may be viable...
Arguably Freenet isn't for revolutions. It's for maintaining freedom of speech for the long period before a revolution, for making it expensive to lock down the internet in supposedly free countries, for testing some interesting technologies, and more.
Bitcoin is not likely to be a serious threat to Freenet, nor is it in any danger of establishing an anarchocapitalist utopia (shudder). If mixing services make bitcoin anonymous, money laundering authorities will simply target the exchangers - freeze their accounts, their customers accounts, make life difficult for them, or regulate them and make it impossible to exchange coins that have been through anonymising services. The most likely government overreaction is to target the exchangers through the banks; not to take excessive steps against peer to peer software. Of course governments may do that anyway - but not because of bitcoin, more likely because of copyright, paedophilia, terrorism and all the usual nonsense.
We are still planning to see our MP again about climate change, in May. If you're in Scarborough, contact me! The organisers of the Big Climate Reconnection have said that this is fine (it had been expected for April); the Energy Bill and other things will still be being discussed and we can catch up on last time's promises.
The wikipedia articles on molecular nanotech (known in some science fiction as universal nanoconstructors) and mechanosynthesis are worth reading. The fact that such tools have been simulated accurately and the results published in peer reviewed journals suggests something will come of it eventually. Roll on the replicators! They would transform society, eventually eliminating most industrial production and agriculture, taking many jobs but also ensuring everyone's basic needs can be met; and that's above and beyond the vast benefits of nanotechnology, most of which we will see long before any such atom-by-atom assembly is possible. They would probably also require a lot of power, but technology would likely advance, and anyway that problem would likely solve itself real fast... There would also be a great pressure to suppress such technology, and/or to make it impossible to obtain designs for objects... Plus you have all the issues with synthesising dangerous items, but we'll go through most of them in cheap synthetic biology long before that point; IMHO it won't be the end of the world, the good guys are far better resourced than any hypothetical bioterrorists and the research to produce a useful weapon will likely be very expensive once there are good vaccines for the traditional (and not easily targetable!) enemies such as ebola, smallpox etc.
I repeat that I am not a transhumanist, I am not even a tech-optimist; I want the right technology: For instance, I strongly believe that we need to do something about carbon dioxide emissions right now. The fact that we might be able to pull carbon out of the atmosphere on a large scale eventually, with industrial air capture, biotech or nanotech, or that in 100 years we may have weather control technology, doesn't help us if we are only a few years away from the precipice (especially as there is a big timelag even once we have the technology): Feedback effects are large, somewhat uncertain (which is a reason to do something, not a reason to ignore them) and already beginning to be visible, and hundreds of thousands of people are already affected severely by climate change - by some estimates, hundreds of thousands are killed every year. Last year was one of the hottest on record globally (and in many places), and the UK's cold weather was likely an indirect result of the global high. And then, leaving the question of the future habitability of the earth aside for a moment, there are side benefits like clean air, health improvements, and avoiding peak oil - which, short of China's economy collapsing, is almost upon us. Although if peak oil results in using more coal - to make diesel or to power electric transport - then it will be an unmitigated disaster. Sometimes the market cannot see past its own nose, and it needs some prodding. Finally, pressing ahead with green technology - which means not only funding its development but also forcing its deployment - is good even in the short term because it creates jobs and industries, and is a large and sustainable new market that those countries who intervene in will lead in - as we've seen in e.g. Denmark and Germany, and as China is rapidly taking on.
For another example, Japan is illustrating once again that nuclear fission is dangerous; it is also expensive, a proliferation risk, and is not the best technology to balance out and back up renewables. The way forward on electricity has been clear for some years now, at least in Europe: A lot of onshore wind, a fair bit of offshore wind, huge international interconnects, the existing hydro, some Saharan solar if possible, demand management (appliances that know when electricity is cheapest), and a little bit of storage (for instance in electric cars). The old models ignored the last part of storage, and assumed some biomass; this is probably impossible, sustainable biomass is likely to be rare and expensive and reserved for aviation, long distance haulage and heating (hopefully the first two will be transitioned to rail as much as possible). Nuclear is a last resort, and probably not necessary at all.
Obviously there is far more going on in Japan than a moderately serious nuclear incident. I don't want to seem callous, although many horrible things happen in the world regularly which get less publicity and kill a lot more people - and many of them we are responsible for to some degree. I wish the Japanese people a speedy recovery, and hope that the consensus that global warming only weakly increases seismic activity is true...
And then there's Libya ... retaking Ras Lanuf is a long way from taking Benghazi, but it is very unfortunate, especially if Gaddafi manages to sell some oil (illegally). Looks like it will take a long time to sort out. No sign of progress in Iran, but hopefully the next election will trigger something. We'll see...
Return of the Test Network, Part Two
The testnet has returned! This is a separate network, incompatible with Freenet, which has a separate repository, separate auto-update keys etc. This is ABSOLUTELY NOT ANONYMOUS. It logs aggressively, and all nodes connect to my central coordinator node. I can then request logs for any period, along with a few other status indicators, from any node. The network is intended for two tasks:
To use the testnet, you need to download the installer:
(It should be at least version 12)
Note that the testnet uses different ports for clients by default:
Also, Fproxy will stubbornly declare on every generated page that you are using the testnet. However there is no protection for FCP.
Opennet *may* work, as my testnet node is a seednode and listed in the testnet seednodes file. However it hasn't been tested yet. Also I haven't yet tested the auto-update, and given experimental code may be deployed, you may have to update your node manually from time to time. However the adapted update scripts appear to work.
Obviously, don't share anything illegal on the testnet. Logs will sometimes include filenames of downloads etc. So we might discover illegal files (by accident), and this would put us in a difficult position, not to mention making you traceable!
Don't share identities (FMS, site keys etc) between the testnet and Freenet either, unless you want them to be traceable.
The more technical folks might like to look into copying content from one network to another via binary blobs; otherwise just insert stuff etc.
While the network is small it may be necessary to set the store everything in the datastore option to make inserts work.
(You shouldn't use Freenet itself for stuff illegal under US/UK law either; if you tell us this we must refuse to give tech support, ban you from IRC, lists etc; this is long-established policy, based on the Grokster judgement).
If we have a few hundred people on the testnet, it should make it significantly easier to debug various low level bugs and test new potentially disruptive code before deploying it. So please consider installing a testnet node! Note that it may require some manual maintenance, and is much more likely to break badly than the main Freenet. Thanks.
Freenet now accepts donations in bitcoin!
Freenet now accepts Bitcoin for donations. We've had various wierd donation options over the years and with very few exceptions nobody has ever used anything except Paypal (and wire transfers for big donations, which have recently become a big part of our income). However there was a lot of community demand (possibly partly driven by a speculative bubble), and it is technically a rather fascinating system. Nagging the devs usually helps; this is a basic principle, we are bad at prioritising (even me, who being a paid codemonkey has to prioritise). And if your nag comes with a promise to make a moderate sized donation, all the better! We've had 205 BTC so far. One BTC is worth just under a dollar, and exchanging them for dollars is somewhat tricky; it becomes a lot easier once we reach $800, so if you want to make a BTC donation now is the time! We will try to avoid keeping a large bitcoin balance since the price varies quite a bit (see here).
Bitcoin is not strictly anonymous, although strongly anonymous systems can be built on top of it without too much difficulty. Actually it provides complete transparency - all transactions are broadcast to all nodes, although there is some privacy as tracing accounts to users is nontrivial.
One big difficulty for me is that Bitcoin distributes the process of creating scarce currency (which is a good thing) by hashcash puzzles which get ever harder (which is a bad thing). There is a fixed schedule for the rate at which new currency is introduced (the inflation rate), which is 30% this year, 10% next year and falls rapidly, completely stopping in 2033. But in the meantime, assuming rational behaviour, we can expect (currency inflation) * (value of total money supply) to be spent every year on minting new coins by solving seemingly pointless math puzzles. This could get significant if the economy is large, although to be fair the banking system is also pretty significant. This is the main reason why I am somewhat skeptical about bitcoin. Mitigating factors: First, currency inflation falls rapidly. Second, green electricity is possible. Third, it may eventually be possible to make the pointless math puzzles into useful math puzzles (i.e. distributed protein folding or similar); I've discussed this at some length with the bitcoin folks, but it seems they've discussed it in the past, so we'll see. In any case I'm not 100% sold, but it's Ian's decision, we need the money, and there are many reasons to think we might get quite a few donations in it at least in the short term; and it is definitely an interesting technology.
Another difficulty is that it is deflationary in the long term. It is also deflationary when it is expanding faster than the programmed increase in the money supply, which is to say right now. Long term this may cause problems.
The main difficulty is that like all new and somewhat anonymous currency systems it's something a pain to get money out of it. There are a lot of options for doing so, but reputation is an issue, and in small quantity you have to go via other money transfer services such as Liberty Reserve; in large quantity you're risking more money...
In any case, if you have BTC, send us some! Our address is:
Including it here should help to authenticate it; if donating via the website, click the link to go through SSL; I have also posted to chat and devl a signed message including the above address.
If you don't have BTC, and you want to donate to Freenet (most of FPI's donations fund my work at the moment), you should probably use paypal, Google Checkout, or a wire transfer (especially for large transfers), as it is more efficient.
Another interesting alternative money technology is Ripple. See here. The main problems with this are 1) whether it would actually work (psychologically, risk analysis of our friends etc), 2) how to bootstrap it (this is the big one, my guess is it will eventually be bootstrapped from social networking sites when people start buying stuff directly through recommendations on such sites), 3) the fact that a fully p2p implementation is unlikely, due to intractable issues with timeouts meaning that nodes need to be online, and that perfect routes are essential if there are variable fees and interest; hence I'd expect maybe 1000 datacenter type servers exchanging full data constantly, although it might be possible to construct good routes in a distributed manner with a lot of data exchange...
Both ripple and bitcoin are essentially experimental at this stage. They could fizzle out or possibly fail spectacularly. Or they could perhaps be used for mass tax evasion, which is probably what a lot of the people behind Bitcoin want. Personally I believe avoiding taxes is unacceptable, but that doesn't mean that we should avoid all opportunities to make our currency systems more decentralised, more accountable, more local, more efficient (in different senses) and better at doing what money is actually supposed to be for (enabling people to exchange energy fairly). I have a fair amount of sympathy for traditional alternative money, but its flaws are well known; technology may offer some interesting new options.
In terms of politics, arguably a lot comes down to crowdfunding. For instance, UK political parties have a largely apathetic and greatly reduced support base, and rely largely on big donors - unions but especially rich people. One solution is state funding for political parties; this goes a long way to permanently solving the reasonable claim that all parties do what the super-rich tell them to do; unfortunately this is not going to have support both from enough politicians and enough of the people any time soon. The other solution is to energise the supporters and get a lot of small donations. Obama managed this in the US. UK politics would be a lot more representative and credible if this happened here. Unfortunately it needs a leader people can really believe in, and (s)he doesn't seem to exist. The broader point is that economic democracy matters, in lots of ways.
Freenet, climate and miscellaneous politics
Apologies for the lack of obvious progress on Freenet, the testnet will be operational soon, and then I plan to deploy the new Freemail which has a number of important bugfixes. Then it'll be low level bug fixes for a while, and then hopefully we can deploy new load management at last - or at least test it on the testnet.
On climate, we will be lobbying our MP again (Robert Goodwill), in May, possibly about the Energy Bill and post-Cancun situation. There are people organising public meetings but I don't have the time/energy at the moment. See The Big Climate Reconnection.
Oh and if you have a vote (UK), please vote for the Alternative Vote (Wikipedia definition) on May 5th. Tactical voting sucks, and AV eliminates the need for tactical voting while preserving all the good things about First Past the Post (most notably one constituency, one MP). Even if you always vote for a mainstream party within the top two in your constituency, you should seriously consider it. If you don't it's a no-brainer. It's true that very few countries use AV, but look at what they use instead, for instance Germany uses something like AMS and has effectively a 5 party system. IMHO that is a good thing: While there are disadvantages to coalition negotiations, there are advantages too (look at the coalition agreement), and two party systems suck: Neoliberal deregulatory ideology created the banking crisis, but a neoliberal-infested Labour (left of center) government was in charge, so the only option is to vote for the Tories, who supported banking deregulation even more enthusiastically and are even more in bed with the bloated and unsustainable City of London financial services sector. Even if you're approaching it from the other end of the political spectrum (like many of the ancaps I mentioned earlier) I'm sure you can still come up with reasons why two party systems suck. Of course, using AMS is Britain, as most small parties would like (for obvious reasons), and as is already used in Scottish elections, would have some larger issues. Most notably, constituency support becomes more of a problem (in such countries this is often handled at the party level), but also in a hybrid system the directly elected MPs would have constituencies that were twice the size they are now. Another option is STV, which can't be easily counted by hand on site (which is a legitimate security issue), and involves potentially ranking an awful lot of candidates - but has some interesting advantages, in that it is a reasonable tradeoff between proportionality, being able to vote for and against individuals, and constituency representation. AV has one of the benefits of fairer systems - no need for tactical voting, which will help smaller parties a little while not being fully proportional - with none of the disadvantages (although IMHO AMS or STV is worth the disadvantages in the longer term). Trivial forms of electronic voting are grossly insecure, although a paper trail may allow secure touch-screen voting and mitigate some of the disadvantages of STV; serious cryptographically safe voting is a long term project, there have been some interesting proposals...
Return of the test network: Part One
Years ago, when 0.7 was first implemented, we had a "testnet mode". You can still find traces of it in the code. It reported node locations and connections to devs, and provided some limited access to logs etc.
It has recently become clear that there are bugs in the current code - mainly related to new packet format and new load management - that can only be debugged with access to detailed logs on both sides of the connection. It has also become clear that trying stuff out on Freenet proper is a dangerous strategy.
So I will be rebuilding the testnet shortly:
The immediate goal is to be able to see both sides of the story when something bad happens at the packet or message level. For instance I spent a long time trying to debug a problem where a node was acknowledging packets containing SSK requests, but was never sending an Accepted or Rejected at the message level. This could be a packet level problem or a message level problem, but the fact that the packets are acked rules out a lot of obvious possibilities.
The medium term goal is to sort out the remaining issues with new load management (particularly fatal timeouts). The longer term goal is to be able to develop stuff on master, then merge it to the testnet branch, have the testnet network upgrade, and see whether it severely breaks everything, before deploying large potentially disruptive changes. Hopefully the testnet will grow to hundreds or perhaps eventually thousands of nodes where we can test stuff.
In the middle of a rather stressful week hacking on various frustrating Freenet problems, I discovered that FMS does the same thing that client authors have eternally done: It sees Freenet fail to answer on a request for some reason (either bugs or overload), and then shuts down the connection and opens a new one. In doing so it increases the total load and makes more timeouts likely. Although with recent builds FCP does actually pick up the disconnect pretty quickly, so the damage is limited. Anyway, I blew my top, saying some foolish things as well as pointing out the technical issues.
Right now FMS is the best stable, spam resistant chat system available on Freenet. This is why we link to it from the homepage. And Somedude has done a great deal for Freenet - without a spam resistant chat system, Freenet would have been in deep trouble long ago. He also helped find the big XML bug, prototyped a realtime chat client FLIP, and so on. I use FMS via TheSeeker's node because I don't want to run it locally.
One problem with FMS is it is not portable, because it is written in C++. It has in the past had both high level anonymity exploits and remote code execution bugs. The former could have easily happened in Java, the latter could not. But in any case we cannot bundle FMS, period. The lack of reliable private messaging on Freenet has been a problem with reporting bugs - this is a generic issue not only for FMS or for anonymously authored software but for all Freenet users; we need a reliable, working Freemail. Freetalk is based on a generic Web of Trust that will be used by a number of clients other than forum-style chat; between this and it being a Java-based plugin, IMHO Freetalk is the future, though people will probably still use FMS; I will likely use both.
To address the technical issues that started the rant above: I was aware of a problem with USK polling, which I had generally assumed to be due to a lack of capacity and unnecessary probing of early editions. The latter will be improved significantly for Freetalk and Sone and so on (but not FMS) by implementing usage of the date-based hints. Capacity was one of the reasons why I merged new packet format, which mostly seems to be working reasonably well now, showing massive improvements in payload at least, although it may still be related to the high level timeouts. Ratchet was working on the USKs issue so I assumed I could ignore it for now and focus on the numerous other problems going on...
It appears there is also some sort of FCP- or client- level bug which causes a whole bunch of SSKs to timeout. More information is needed to resolve this, I have posted on FMS about this. It might be due to losing FCP messages; it might be due to a bug at the FCP level (but it is doubtful IMHO as it is very simple); it might be due to a bug at the client level losing all the requests in a given bundle (I doubt it, it doesn't seem to affect Freetalk, but such things have happened in the past), or it might indeed be a capacity or priorities issue...
Current progress and other stuff
For the last week I've been struggling with opennet announcement being very slow, and for the last half of it, Update Over Mandatory. What was happening was that a large number of nodes got stuck at versions around 1320, but nonetheless had auto-update enabled (otherwise they'd stop announcing eventually). So they constantly announced, and tried to update. Unfortunately the UOM transfers were failing. That has now been fixed, as far as I can see. It is likely that we will regain a significant number of nodes, as well as the announcement load reducing dramatically. Plus, there were various issues with announcement itself. In any case, by 1352, all of this appears to be solved: announcement is fast, UOM is fast, it all works, and I no longer suspect attacks.
Also, fproxy performance seems significantly improved, and there is anecdotal evidence that downloads are up too. Inserts are definitely working significantly faster.
That just leaves the zillions of other bugs and some major feature work:
Long live Freenet, or something!
Have finished antibiotics for now, hopefully I won't need any more... Egypt is fascinating.
Interesting times: Please test new load management!
There have been problems with the network recently. There are lots of theories floating about: a deliberate attack, realtime requests causing loads of backoff, ...
I have posted to FMS to try to gather information on what the main symptoms are. The main things seem to be very slow downloads (estimates varying from a factor of 3-4 slowdown over a long period, but with recent builds particularly bad, to a factor of 40 since new packet format), slowish fproxy, and slow bootstrapping.
Build 1339 separates backoff for realtime (fproxy) from backoff for bulk (everything else). This seems to help IMHO but others have said negative things. Part of its purpose is to see if this is the problem - does bulk recover some lost speed?
However, new load management is really close to being ready. I will try to release it this coming week or the week after. Much more widespread testing would be really helpful. I have released a snapshot, so you can update.sh testing (linux/mac) or update.cmd testing (windows). Testing it on the main network is enormously helpful. Expect errors in the logs; reporting a selection of these would be helpful, but I probably know about most of them; some of them include timings, which would be interesting, especially the worst case ones for bulk or realtime. Some log messages are just obviously broken - anything with "ArrayIndex", "Illegal", "NullPointer" etc in it, for instance - and these really need to be reported. Likewise, deadlocks, severe performance loss etc are things I need to know about. Please report performance on FMS or elsewhere. Don't turn off AIMDs for now, as that won't happen until new load management is fully deployed (and then initially testing isolated nodes).
Organising a parallel testing network via exchanging darknet connections would also be a very useful thing to do. So we can see what the effects are of a network composed entirely of new-load-management. Try to keep this in parallel to the main network rather than instead of it though: If you have the bandwidth, run nodes on both. Feel free to exchange friend references via FMS, Frost, IRC etc, purely for the testing network. I may reinstate the full blown testnet (a non-anonymous network set up specifically to help developers track down difficult bugs), soon, whether or not we deploy new load management.
So that's what you can do to help get new load management out of the door. Why would you want to? Well, we've been building up to it for the last 6 months, and much of the current problems of the network (although many useful features too) are closely related to it. It should significantly improve throughput, darknet performance, work better with realtime requests (although probably at a slightly higher latency), routing accuracy (and hence data persistence, at least once I implement the insert part), and generally should solve a lot of long standing problems. It will also be far less easy for a third party to attack it (the current load management model is somewhat fragile). We will still need a lot of debugging on new load management, a small amount of additional implementing work (e.g. inserts), and a lot of other debugging (e.g. realtime transfer failures). The last part may only be fixable with a full testnet. However, IMHO the longer we leave the network in limbo half way between old and new load management the worse things will be.
So, thanks folks, and please test the new version! If you're going to build from source you want the tag testing-build-1340-maybe-merge-new-load-management-pre1, on the branch merge-new-load-management, rev 47edfe611a1f088b51fccb36d3e7710e28c3d2b8.
Now I'm gonna shoot some beasties, and come back on Monday!
Having talked things over a lot, I'm much happier about Freenet: We have a good handle on many of the larger problems, we have enough time to deal with them, we have a limited set of plausible theories, we have ways to find out what the problem is, and if it did turn out to be an attack (which IMHO is unlikely), getting the engineering right will make it much harder (and ultimately darknet may make it dramatically harder - stay tuned for big progress on darknet usability after the current round of problems is sorted out; on security, the big question is whether we can fix the Pitch Black attack convincingly, Oskar has a solution, Evan doubts it but thinks there probably is a solution, snark may be getting involved; we'll see). I really do care about Freenet, and I'm not just doing things randomly, but I think we have a reasonable chance of going from well behind where it was a year ago to significantly faster, at least on some metrics, as well as being more secure and a better architecture.
Now I'm going to rest (involving sleep, gaming and perhaps the odd walk), see you all on Monday! Health is annoying but not serious, and getting sorted, at present. As for my religious-philosophical struggles, see below; I'm not always entirely happy one way or the other, and do need to find more social contact, and there are a great many things I am uncertain about, but that's where I'm at, at least intellectually.
Egypt is fascinating, lets hope it turns out the right way... I don't believe those who say if the Middle East has democracy it will all turn into radical islamists who will then attack Israel; I just don't believe suicide is all that popular. More importantly, the demonstrators are going to need to keep it up, and increase their numbers, although the government is clearly on the back foot at the moment... And all without telecommunications, but maybe that won't be a big problem given the intensity it's already reached. Communication tools are needed that can work when the government pulls the plug; whether Freenet, or something using some Freenet tech, has anything to contribute to that is a long-term question.
Freenet has a serious problem with users leaving, apparently since roughly December. It is not immediately clear why people are leaving. The old uninstall survey on Google Spreadsheets never worked and we abandoned it, and we haven't got around to installing anything similar since.
My working hypothesis is it is related to poor performance. There has been talk of severe breakage of USK polling and possibly SSK issues for a while, although the problems with bootstrapping on Freetalk should be somewhat improved. More broadly, I've been trying to get the last critical features fixed before working towards a feature freeze and an alpha release, unfortunately some of this has been fairly disruptive. The important features:
So, apologies for the problems we've been having recently. Some of the poor performance has just been down to having so many updates in quick succession - I hope that we can reach a stable one-update-a-week pattern again very soon. But it's likely to be a bit rough for a while still as new load management is deployed. However, it will be worth it, new load management solves many problems we've been struggling with since the 0.4/0.5 era.
More broadly, development has been under a lot of pressure for a while. I originally agreed to a January 31st deadline for a feature complete alpha (I have since been told that such a thing is a first beta). I wasn't aware at the time that we would need the darknet enhancements, that Freetalk would take so long (although this has bought time for me since it's p0s's project), or that new load management would take so long (but we're definitely getting there). I am of the view that we can probably reach feature complete status by the end of February however - hopefully.
So I try to prioritise. And everyone knows I'm not very good at it. And sometimes this means things other people regard as important get postponed, marginalised, ignored or forgotten about. Or saying stupid things to valued volunteers. I will try to do better but please bear in mind that we need a release soon, and that may well mean your code can't be merged until after the end of the release process. We may need to branch if there are big, hard to review contributions coming in - but at the moment there aren't any.
We may have missed our ideal publicity window (arguably shortly after the wikileaks fun), but on the other hand a lot more people recognise now that censorship on the internet is a serious problem, and that the solutions that people have always assumed will deal with it are precarious at best. DNS can be forced off the internet. Google can be got at (not in the case of wikileaks but court orders have hit them in the past). Names can be demanded from twitter just as from anyone else (even if there's no practical censorship of such services just yet). Any site on a small hoster can be forced off the internet relatively cheaply via DDoS, and any site on a big hoster can be forced off via political interference (both of these happened to wikileaks). And modern, developed, industrialised, democratic nations are preparing to comprehensively block websites which contain copyrighted materials. Such powers will inevitably be misused to block places like wikileaks (which hosts commercial leaks, and the MPAA asked about blocking it under ACTA), or xenu.org (scientology claims copyright on the Operating Thetan documents, which accuse Jesus of being a paedophile amongst other things). Many such documents are legal under most countries' law because they are in the overwhelming public interest, in spite of being copyrighted. But what if you're not a big organisation with deep funds for fighting legal battles to high courts in many different countries? Freedom of speech online is hanging off a precipice, and more people recognise that now than have done for a long time. However, most nightmare scenarios which might kill Freenet, such as blocking all peer to peer communications, are unlikely to be anywhere near the political consciousness for many years.
So, on some levels, work is very depressing right now: Performance is well below what it has been, and users are leaving at an alarming rate. On other levels, we have a good handle on several of the fundamental long-term problems, and we have a golden opportunity in the public consciousness. Releasing immediately at any price is not a good idea, but nor is waiting too long. And we still have $14K in the bank, so we're not in an imminent funding crisis. So there's everything to play for, despite it looking pretty cloudy at the moment.
We finally got the floors sorted out, so they no longer go up and down when you walk on them. This kicked up a lot of dust, but doesn't seem to have been catastrophic for my health (we had planned to be out of the way but that didn't happen). Another chest infection (probably pre-existent), another load of antibiotics, and finally getting my ears sorted out. Slowly coming to terms with the fact that I can no longer consider myself a Christian. And, going further back, the CCC demonstration for zero carbon britain by 2030 went remarkably well considering the abysmal weather (which cleared up on the day) and other reasons to be gloomy.
Skip this post-religion rant to the previous post which is Freenet-relevant
The following has been an increasingly big issue for something approaching 6 months now. The rather long text below mostly was already written but not posted - I haven't taken a lot of time out from Freenet to write it today! If you are a Christian and have a weak faith and mean to keep it, I suggest you look away; I don't bear any malice towards Christians, and respect many of them as great people. But I no longer consider myself, with honesty and intellectual integrity, to be one. Perhaps one day I may find that I can be again, but for now I don't see conservative (in the sense of respecting scripture) Christianity as viable, nor do I see liberal (in the opposite sense) as being viable, and I'm not convinced there is any viable middle ground either.
At this moment, I am, as far as I can tell, a weak atheist agnostic. I grew up in a Christian family and considered myself a Christian since a very early age, thanks in part to my great parents. I was never a straight down the line fundamentalist: I also have a scientific outlook, and have never been a creationist though mostly I have been a conservative. Mostly I was able to reconcile those two threads. One of the best reasons for being a Christian was dad: His extensive, if anecdotal experience of healings, answered prayer and so on, his scholarly approach (see his website, he is working on a commentary on 2 Corinthians), his consistently walking the walk, have been an inspiration, especially with his having a background in hard science and maths. Unfortunately his testimony is somewhat undermined by his belief in homoeopathy and telepathy, also on the basis of anecdotal experience: This is not a reason to not believe in Christianity but it does tend to suggest that his evaluation of anecdotal evidence and personal experience is suspect. Mum is more of a liberal, but nonetheless a firm believer. Both of them, and other strong believers I have talked to, have dramatic conversion experiences they can point to. These are great for them, but doubtful from anyone else's point of view.
I give below most of the issues that have resulted in me reaching this point:
Right now, the only reasons I can come up with for continuing to believe in God are bad reasons: Pascal's wager (you can always get the right answer by putting an infinity on one side of the inequality!), social aspects, making others happy about you (biblically "fear of man"), stubbornness (Puddleglum's defence in The Silver Chair!). And the intellectual framework just doesn't work, even ignoring the long-term sci/tech issues that admittedly may or may not become severe in my lifetime - but in principle are well within the horizon at least in theory (and in some cases are potentially career-relevant - is it immoral to work with computational neuroscientists?). Liberal Christianity does not seem to be a viable solution to me intellectually. It's true that not all liberals are resurrection deniers (and thus not Christians at all according to Paul), and it's true that the core of the gospel is not biblical inerrancy but the incarnation, death and resurrection of Christ. But to throw out the Law, even as an authoritative albeit compromised ("hardness of hearts", Matthew 19) exploration of God's moral values, brings me into conflict with Jesus himself, and probably also with Paul: Both of them valued it very highly. Throwing out Matthew might be an option, but then exactly where does one drop one's anchor when one is ignoring substantial parts of scripture whenever they conflict with one's worldly biases? Clearly the worldly biases are authoritative and not scripture! I would accept the intellectual cost of a liberal outlook if I had any real credible evidence, but right now I don't see any - other people's highly subjective and anecdotal experiences, and unresolvable debates over things that happened millennia ago, seem to be the best that is available. It might be worth reading up a bit more on the latter eventually...
If God wants me he knows where to find me. My grandad was a militant atheist up to his remarkable deathbed conversion; I am, I hope, open to answers and experience - certainly more open than he was! I'm not sure what exactly would convince me, but if God exists I'm sure he's far more creative than I am! Some of the weakest arguments for atheism are based on the principle that God is a giant - big and stupid. If there were a creator it is absurd to assume that he must have all our limitations (as the Greeks did), and therefore only care about big stuff. And if I was saved in the first place, he will probably reach me again one way or another. But for now, I will get on with my life, which is pretty full even without reference to God. And find other friends and places...
I do not at present regard humanism, transhumanism or any other life philosophy as necessary. As I mentioned, I believe (along with pretty much all the world's religions) that the Golden Rule is self-evident, if hard to apply in detail, and harder to stick to. Also, scientific skepticism has been a part of my life for a long time, if not a strong part (I'm no scientist, I'm an engineer or just a hacker - but I generally respect empiricism). I specifically reject transhumanism, because why should I fear death if it is no more than the end? Better to end than to become an agent of darkness - Darth Vader is far more to be pitied than Obi-Wan Kenobe - and no great loss if you have contributed what you can. Of course when faced with it doubts strike at everything you thought you'd achieved, and there is no doubt hideous pain... I also reject humanism, as while I recognise much that is bad resulting from faith, I am not (yet?) openly hostile - much good has also come from Christianity, and the same mixed picture is true of many worldviews (perhaps not on the same scale). On the other hand I am certainly hostile enough that I would be a disruptive influence in church... social contact is a problem having left.
On some levels, it feels good to have closure, what is hard is not having a decision one way or another. Emotions are not a reliable guide to anything - emotions follow the decision, and solidify it, and if you make them your goal you drive yourself mad. This may or may not be a scriptural principle but it is also scientific - those with no emotions are incapable of making any decisions.
Wikileaks is in big trouble. Julian Assange has been arrested, him and the organisation have lost $100,000, and the main website is down due to government pressure. Some of the others may go down soon too, and governments are of course busily (but slowly) granting themselves the power to block sites. They promise that this will be a judicial process but IMHO that's just not how the judicial system works in the US or UK - IMHO we will be seeing notice-and-takedown which you can dispute if you have deep pockets, just like with the DMCA.
People are talking about distributed DNS as a possible solution (which of course it isn't, IP-level and proxy-level blocking is both feasible and actually already implemented in most countries). IMHO it will suffer from the same problems that plague similar systems everywhere else: There is no good way to prevent spam. CAPTCHAs can be solved, by humans, for something like $1 for 1000, and there are companies that offer this service. Hashcash is vastly more expensive for a real user with a slow computer than for an attacker with a fast system and GPUs. And anything that handles real money will be centralised and vulnerable. Which leaves trust based systems, such as CAcert.
Of course these issues also affect Freenet - particularly Freetalk. In the long term we will need to get rid of CAPTCHAs. IMHO the best realistic solution is the Scarce KSKs proposal, which involved allowing every darknet link to send a single scarce token every X period. Of course, this requires pure darknet. But as you can see from recent discussions on the devl list, opennet is IMHO impossible to really secure, short of bolting on a third party onion network (which is also difficult to secure, but not really our problem).
Anyway, you might want to visit the Reddit post.
Of warming, cooling and divine sovereignty
Apparently I'm not the first person to come up with the theory that God is determined to prevent a climate deal happening any time soon. Monbiot did miss a link though: 2010 had a lot of hurricanes, yet very few of them hit the USA. And arguably this is the tip of the iceberg: Not only is it colder in the West (especially in winter) despite being warmer almost everywhere else, all the political circumstances are pointing in the wrong direction too. It's true that China may be starting to take the issue seriously (there is a good case to be made that China was set up as the bad guy at Copenhagen), but without the US the situation internationally is pretty hopeless until some time in 2013 - and that's if Obama is reelected and has a legislature that isn't completely riddled with climate change deniers. One of the most fun things about this is that a lot of the climate deniers are also evolution deniers - and which way does cause and effect run? You train people to treat scientists as immoral atheists part of a great international conspiracy to suppress the truth, what do you think happens?
A scientific explanation may be available: This year, the global average temperature has been somewhere between the hottest on record and the third hottest on record, there have been extraordinary events involving record high temperatures in Russia and Pakistan, yet the West is remarkably cool (Africa is hammered as usual, of course). This is believed to the the result of a major shift in the jet stream, the global wind currents. For a while this was thought to be the result of low solar output, but here and here suggest that maybe the rapidly shrinking arctic ice cover is to blame - local cooling in one place, while major heating in others, resulting from global warming (more properly termed "climate change"!). Unfortunately this coincided with the last best chance to get a US climate law on the books before COP17 let alone COP16 - and the rise of deficit politics in the US, and the Tea Party scum.
What can I conclude? Well...
Scarborough Big Climate Connection
Nine of us went to see Robert Goodwill MP about climate change. That being me, my mum Evelyn (a concerned citizen), Jane (an enthusiastic local climate activist and prop maker, formerly the driving force behind Scarborough 10:10, a campaign to get people to sign the 10:10 pledge and raising awareness by showing The Age of Stupid with question and answer sessions), with Paul and Chow from 10:10, Mark and somebody with a name starting with a G from Greenpeace (I'm not good at names and faces!), Andy from the emissions partnership, and Kevin from the Green Party and private tenants rights group. Most of these people I hadn't met before, and it was great to make contact; many of them are deeply involved in local sustainability, some of them wearing at least 4 different hats: 8 organisations represented by 9 people!
It went really well. Goodwill is a front-bencher, assistant government whip and formerly shadow roads minister, and a farmer, and has a remarkably positive voting record on climate (a lot of tory backbenchers are climate skeptics unfortunately). We discussed the main demands: Energy efficiency in housing, including the Green Deal (aka Green Mortgages, a petition originally proposed this many years back, Labour rejected it only to introduce it in their election campaign) and particularly a minimum standard for rented property. It was pointed out that most tenants are on short leases and the government's limited proposal to let tenants demand reasonable energy improvements is insufficient: we need a specific standard, that all rented properties must be band E or better by 2016. There should be help with finance for smaller landlords, possibly via some version of the Green Deal (Goodwill and Paul had experience as landlords), but while many have difficulties and demand is only going to increase, most of the fuel-poor live in rented property, and this needs to be dealt with.
We briefly discussed the proposed Emissions Performance Standard - a legal maximum grams of CO2 per MWh of electricity generated for all newly built power stations. This was a manifesto pledge but has been delayed; we are asking for enabling powers to be passed so it can be set quickly in secondary legislation once the consultations are finished and the details worked out. Goodwill told us that since this is not in the bill it's unlikely to be introduced in the first reading but it might happen in the second. We then discussed recycling and waste incineration, a topic that was raised by somebody and which Robert had some experience with while an MEP.
Running low on time, having been sidetracked a few times (which is probably a good thing, I don't want to be doing all the talking!), we briefly mentioned the international negotiations - as I understand it, while it is highly unlikely that there will be a complete, binding deal, there is a good chance of some progress on adaptation funding. Our demand in this area is, in jargon, that the government support a single UN fund for adaptation, mitigation, and forestry, funded by internationally agreed taxes on shipping, aviation and banking. In English, this means that the rich countries need to pay into a pot of money controlled by both rich and poor countries, under the UN (not the World Bank!), which will be used to deal with the enormous impact of climate change on currently poor countries (for instance, some parts of Africa seem to have climate related food crises every few years - this was not the case 30 years ago, talk to any NGO active in the area), by for instance building flood defences, planting drought resistant crops, and so on; to help them to develop cleanly (by building wind farms instead of coal plants, for instance), and to help them to preserve their forests (which are a massive store of carbon, as well as providing many other important global benefits, which we largely get for free). We created this mess, and the global poor will be hit hardest both because of geography and because they have the least capacity to cope: It's a matter of justice, as well as being absolutely essential for any binding deal to eventually be signed when America comes to its senses. Goodwill said he'd write to Chris Huhne and get back to me, so we'll see. He seems very positive, he took us seriously, and everything I have seen suggests he is very capable. He's always answered my emails, and if I ever voted for a Tory it'd be him. It is great to have the government on roughly the same page on so many of the green issues, even if they do do some really stupid things like the planned forestry sell-off; we are largely concerned with details in the main demand above. Something similar to the picture will be in the Scarborough Evening News next week hopefully.
The banner, chopped off in the photo, originally said "Scarborough Says: People and Planet Before Profit". We originally had decided not to use it but ended up doing so anyway. It provoked a useful discussion, both in the pre-meeting in the cafe nearby beforehand and with Goodwill. The short-term, front message is that right now, action on climate change is a win on all levels: It will create jobs, expand industries, improve public health, and have a great many benefits. It will help to put our economy on a firm footing now that the fountain of youth that is the financial services sector is looking frankly shaky, and it should help to get the Northern economy going, with some of the ports upgrade investment likely to be spent in Whitby; this money will be used to break the engineering bottleneck which means we were unlikely to build enough offshore wind to meet our 2020 renewables target.
As an aside that is definitely not part of what we discussed with Robert, I am skeptical that exponential economic growth can continue forever: It is possible to reduce the carbon emissions for every unit of growth, but as long as we want more physical things, and recycling is imperfect, it will be very difficult to shrink carbon intensity fast enough to account for both growth and the radical cuts needed real soon now. On the other hand, who knows what the future may bring; technology has a habit of surprising us. Unfortunately waiting for better technology is not acceptable at this stage as we are already far into the red with nature and on the verge of sheer catastrophe - we need to deploy what we have, because the time has run out.
The traditional answer to this is the service sector and the knowledge based economy. The latter is usually accompanied by ever-increasing demands for intellectual property laws bordering on protofascism and often on innovation-impeding and growth-impeding corporatism, as we've seen in both copyrights and patents in recent years (most recently Cameron and Google on UK copyright law). Given that I work on censorship resistant peer to peer software for a living, clearly I don't support this sort of nonsense - but there are many ways to run a "knowledge based economy", as we've seen.
To get back to the issue at hand, we did talk about nuclear briefly - the policy is there will be no subsidy, the only way the economics makes sense is if there is a strong carbon price, but as our MP pointed out, if it is too strong, it might result in offshoring in heavy industry; but you have to consider that that sort of argument comes out of the steel industry now, when according to Sandbag the current scheme, the EU ETS, actually contributes significant profit to most heavy industries (effectively subsidising them out of consumers' electricity bills).
So all in all, a successful meeting. Will be interesting to see if we get any press coverage. I'm off to Leeds tomorrow with Jane for a mini-conference on building the climate movement, although I'm skeptical that I'll have time to do much in the near future...
I will post a Freenet-related update next time, I promise! I may also elaborate on the doubts issue I mentioned last time, which is partly resolved. Those who came for the climate stuff use the direct link. I will be migrating this to a proper blog in a while, with comments (sorry for the lack of comment support, feel free to email me, or reply on your own blog and send me a link), although whether that is Freenet-only remains to be seen; this site will remain up in any case.
The world gets what America and China deserve...
I have, perhaps foolishly, volunteered myself to organise a meeting with our local MP as part of The Big Climate Connection. Hopefully that will go well. We are asking for more, additional, UN-controlled adaptation funding (probably from taxes on shipping, aviation, banking and carbon auctions ideally with a minimum price), for domestic energy efficiency (especially for rented homes; the Green Deal should ensure owned homes are upgraded), and probably planning issues too (it can take nearly as long to get approval for a small onshore windfarm as for a nuclear reactor; offshore is vastly more expensive and has engineering capacity issues, in the short to medium term wind is a vital part of the future, and as much as possible of it IMHO; and in North Yorkshire there is virtually none). Meanwhile, America votes on Tuesday. Until America passes meaningful climate legislation there is little hope of progress internationally - and without a global deal, there will always be countries who just go and burn the last drop of oil and the last gram of coal.
From a climate point of view, it is very straightforward: Every single Republican Senate candidate either denies climate change exists or opposes any effective action to deal with the problem. Even with 59 seats (including the two independants), the Democrats weren't able to get a hideous compromise of a carbon trading law through the senate - partly because they need 60 to beat the inevitable filibuster, and partly because of coal state democrats who vote against it. The predictions look like they will lose control of the House and lose many seats in the Senate, making it even more hopeless... As an aside, just because the proposed bill was a hopeless compromise does not mean it was necessarily of no value; without a bill to limit the total emissions across the US, there will always be leakage. Internationally, this is also true - domestic legislation in the US enables a global deal, China won't move without the US moving first. Carbon trading, offsets, free handouts of carbon permits, and the maximum price would all weaken it to the point where it won't achieve anything like what's needed in actually reducing emissions, at least in the short term - this is pretty much the case in the EU already (we don't have a maximum price, but then we don't have a minimum price either, which is actually a gain for the US proposal) - but just having a target would likely unlock the negotiations.
Another issue arguably at stake is Obama's plans to require all peer to peer systems, web services etc to comply with intercept warrants - even if this requires major technological redesigns. This would probably affect Freenet, were it passed. It is the latest in a long line of unconstitutional attempts to make strong encryption illegal, going back decades; there is a good chance it will not happen, and even if it does happen it will be struck down sooner or later. However, given the fact that this time there is money at stake (eventually the copyright mafia will need to deal with secure peer to peer) and not merely national security (terrorists could chat over Facebook/Second Life/Freemail/Skype), it might pass - and if it's struck down they can always pass it again, like they do with some other laws... The rhetoric from the Tea Party folks is often about defending the constitution, it will be interesting to see whether that principle survives the combination of corporate lobbying and national security...
Just as with the climate law, if this passes it will have consequences beyond the US. It would have severe consequences for me personally, for instance, if my largest client, Freenet Project Inc, was put out of business. Much software is developed in the US, and while many horrible things are in progress in Europe, these things are harder to fight if they are already established elsewhere.
Which of these issues is more important? The cynic in me says they're both hopeless. The optimist says on climate we will find miraculous technology (e.g. the claims that solar PV will be competitive with coal soon, breakthroughs in battery technology etc) that we can deploy instantly at no cost and will solve the climate crisis without any major government intervention (having already blown the opportunity to create millions of jobs, put the economy on a more sustainable footing, improve public health, greatly reduce risk, and encourage technological development by using what we already have), or more realistically, that the peak oil crisis due real soon now will have such a devastating effect that governments will finally get their act together (and will miraculously choose not to go for the easy short-term option, coal) - although whether it will be possible to deal with the problem in 2020 having made minimal progress before that, is an open question. And on encryption, the optimist says it won't pass regardless - none of the previous attempts did. And even if it does it will be struck down, and FPI will find a way to perpetuate itself - until it becomes impossible for me to continue here, where we have a much weaker constitutional defence; efforts to "modernise" wiretapping are happening here too, and we already have (at least the primary legislation for, if not actually implemented) what amounts to a three strikes law, and a blacklist of copyright infringing sites...
So am I, as a Brit, saying you should all vote democrat? Personally I would, in this specific election, if I lived in the US. Fortunately I don't: I'm not convinced I'd ever move to the US even if I could, for many good reasons. My point is that what you do affects everyone else in a way that is almost unique to the USA. Think before you vote. There's an old saying that democracy means the people get what they deserve. Unfortunately here it looks like the world gets what the US and China deserve (not that China is a democracy by any stretch), at least on climate. Such is the nature of empire, and we're some way from being rid of the US empire, despite all the problems recently...
As explained in the last update, which didn't get inserted to Freenet (sorry folks), but is visible below, I am gradually merging the new load management system into the main fred source code. This means, roughly speaking, merging a big chunk of code, doing lots of bugfixes, letting it run for a week while fixing minor bugs and catching up with email, and once it's mandatory, repeating the whole process. Arguably we shouldn't be adding big new features at this point in the release cycle, but IMHO it is justified for load management and for zidel's new packet format. It is already well under way, although far from finished (I need to deploy most of it to debug the rest properly); many of the intermediary stages have large side-benefits, such as the recent work on block transfer and fairness between peers; it is essential to prevent relatively easy DoS attacks on the network that have been possible ever since 0.7 but remain serious and will sooner or later be exploited (although fair sharing between peers, merged in 1296, greatly increases the cost); and it is a fundamental issue which Freenet has struggled with more or less forever, yet thanks to the effort of many people, particularly evanbd and some folk on Frost some years ago, and the Content Centric Networking paper, we finally have a good idea what we're doing. Further, there is a good chance that this will significantly improve both data retention and performance in general; IMHO the first is probably more important.
The next step will be to deploy the bulk vs realtime flag, although it may make sense to do some work on the request layer first - the bulk flag will significantly increase the number of requests in flight, so it may be necessary to make RequestSender fully asynchronous first (block transfers are already asynchronous, and the async version of the code is, after some refactoring, relatively simple and fits well with the rest of new-load-management). After that, most of it is ready; turning on the load status messages, various changes relating to timeouts and counting how many requests are on the next node, and finishing off the core changes. Once we have new load management, zidel's work will be deployed as well; it would be good to deploy it immediately but it's probably best for the network to only have one huge disruptive change at once.
Other important stuff remaining include:
Critical stuff that I'm only peripherally involved in includes Freetalk/WoT, which hopefully will reach some sort of release ready status in the coming weeks, enabling us to ship FlogHelper as well (and making life easier for testers of e.g. Sone), and getting the new wininstaller deployed, which afaics only requires a new update script now. See the roadmap on the (new) wiki.
We are getting really close to 0.8.0, and we need the publicity; and there are quite a few things going on in the world that might help us, or we might help. Which reminds me, if you can translate Freenet into a new language, or update an existing translation, please contact us! It would be really awesome to have an up to date mandarin translation, or even Farsi (I am prepared to add dir=rtl's etc where needed), although the lack of broadband in most of Iran makes Freenet of only limited usefulness. There are also many European languages which various people have started translations for but are not maintained any more; the more the better!
More broadly, Freenet is working a lot better than it has been (especially in terms of working fast out of the box), and we are very near to acceptably feature complete for 0.8.0; we don't want to keep adding features indefinitely. There are big clouds on the horizon legislatively, and there is the unfortunate fact that opennet is hideously insecure; but we have a real opportunity here, and IMHO if we can get Freenet big, we can bootstrap a darknet. There is a need for further work on security even on darknet (e.g. tunnels), but if we solve some of the fundamental problems with performance and functionality, and get a lot more users, we can make some serious progress, and maybe even persist without having to constantly rely on Google's generosity!
My health is reasonable; it's not great, but it's not bad. I'm not sure I'm entirely rid of the chest infection, and my other issues aren't entirely resolved, but are probably purely a matter of diet at this point. Finances aren't great but FPI (in the form of Ian) have generously agreed to another pay rise, and it should all be sorted some time next year... Servalan (my server/workstation/myth backend/everything-box) is still managing to struggle with 8GB of RAM, apparently a big part of the reason why is my habit of having 100+ firefox tabs open, I'm playing with memory compression which may help; and of course closing some of those tabs! On the other hand I've had some major faith issues lately, which I'm at something of a stalemate with right now; but if God wants me to remain in his Kingdom with peace and clarity of thought he will eventually reveal a way forward.
Load management and other things
The current network continues to have a lot of problems with backoff. Some people report it's improving but it's taking a long time. I've managed to convince Ian that rewriting load management makes sense at this point. As I explained in the previous post, this is something we've been pondering for a long time, and somebody with a better grasp of network issues might have come up with it earlier. Anyway, we have a fairly good idea of what needs to be done, and I've made a start on implementing it...
Basically the principle is 1) each node figures out what its capacity is (based on how long it will take to transfer everything if every request succeeds), 2) we never route to the wrong node unless the right node is severely broken, and 3) we queue, and to some degree retry, until the request is routed to the right node. Hence we can have accurate routing, load management that doesn't care whether requests are local or remote and so leaks no information, and that runs as fast as possible without causing major misrouting.
One cost of this is relatively high latency on each hop. It can take many seconds to get a slot on the next node, because we have limited capacity, largish blocks, and many peers. Reducing the number of peers is unlikely to help much either, and reducing the size of the blocks will increase other costs considerably. The solution is to divide requests into either bulk mode or realtime mode. The latter will be used for fproxy, the former for long term downloads. Fproxy requests have higher transfer priority but lower capacity, and are expected to be very bursty, enabling us to have relatively low latency. Bulk requests should be constant, high throughput, high latency; realtime requests are bursty, low throughput, low latency. I'm not sure who first suggested this, maybe Thelema. Anyway, it appears that the new load management scheme won't work without it. An interesting consequence of this is that we can run more bulk requests at once, improving throughput and security, but we will need threadless block transfers to make this work. I have another branch with this but it needs updating and debugging.
There are also some security issues. We need to give our peers enough information to predict whether we will reject their requests based on bandwidth liability. Currently the information used internally is rather too high resolution to be given out to peers. IMHO reducing it to just the number of transfers in each direction won't impact performance much (there is plenty of vagueness elsewhere), but will make it safe to share this. Note that the current load management system has much bigger security issues: It is much more vulnerable to DoS (although fair sharing between peers fixes this), and the originator rate control (AIMD) thing is horrible.
In other news, the new windows installer seems to be working well, although it is not yet finished, and Freetalk continues to make promising progress. Hopefully both will be ready soon; you've heard that before, of course. Help with either would be appreciated!
Another interesting if scary development is some back of the envelope calculations done recently on what it would cost to connect to all opennet nodes, in order to e.g. do correlation attacks. I'm not quoting numbers but lets say they are low enough, even on a large network, to make the long-term viability of opennet rather doubtful. They may even explain why we haven't seen any serious DoS attacks - the bad guys find it more effective to watch what everyone is doing than try to prevent them from doing it. The long term solution has always been darknet (assuming oskar solves the pitch black attack), and the chicken and egg problem has always been that because Freenet is small, slow and has a relatively high proportion of paedophiles etc, nobody can convince their friends to use it. Any input welcomed, this and other things hopefully will be discussed at an IRC meeting sometime in the near future that nextgens has convinced me to organise. Another big issue is 0.8 timing. We have funding for some time now, it would be great to get 0.8 out, on the other hand you can always add more features - and some of them are really important, e.g. load management changes, data persistence, filesharing searching, ...
I am on holiday in Wales from Saturday to Thursday, and only to be contacted in case of emergency. I'm not going to merge the load management code until I get back! Health wise, I'm fairly sure I know what the problem is, but I won't know for sure until some time after I get back. Mum and Dad have gone up already via Bristol because of a friend's PhD.
Chaos good and bad
Much has happened since last time I posted. We are now up to build 1271. The metadata changes in 1255 needed many more builds for bugfixes, but should now be more or less stable; see the previous post for details on why they were worthwhile. There were many more optimisations and bugfixes in the period up to ~ 1266, including:
By this point it was clear that the main problem people were complaining about was downloads stalling, and that fixing this would be nontrivial; I chased up various leads and got nowhere. It also became clear that the problem was probably in the request queue / scheduling / cooldown logic. What happens here was we have a tree of requests which need to be run (we have a separate structure for which requests want which key, which is what the splitfile bloom filters are for, due to the need to recognise when a key is offered by a node or is wanted by two separate downloads). This tree starts with priority, and then retry count, and then random between clients (e.g. the global downloads queue is only one client; Frost is another), and then random between high level requests (e.g. a queued download), and then the individual low level requests (e.g. a single splitfile segment), each of which might have more than one block to fetch and so may start more than one network-level request. There was also a separate structure, the "cooldown queue": Persistent downloads are generally queued with max retries = -1 i.e. we will keep on requesting failing blocks forever, but we do not want the network to be flooded with requests which are unlikely to succeed. So we impose a limit: A single key may only be requested 3 times every half hour. Hence after that, we unregistered the request, and put it on the cooldown queue. The problem with this was that even if we were not making any progress, the database (in which all this is stored) was constantly changing, resulting in random writes / disk seeks, fragmentation of the node.db4o[.crypt] file, slowdowns to the whole system, code complexity and so on. So the change made initially in 1267 was:
This should radically reduce the amount of disk I/O needed, simplify the code, circumvent any bugs in the code responsible for going into cooldown, coming out of cooldown etc, avoid fragmentation, reduce opportunities for database corruption, and generally improve performance, and hopefully in the medium term stability too.
Of course, this did involve quite large changes, and as a result, there were many bugs, which needed to be fixed over the next few builds. One of the biggest issues was a change to when we inserted healing blocks. When we download a splitfile, each segment (chunk of 4MB) is decoded, and the original 128 data blocks plus 128 check blocks (for redundancy) are recovered. Those blocks that we were unable to fetch and had to decode are "healed", that is reinserted. However, not having accurate information on how many times we tried to fetch each block, this was a bit tricky with the new code. Initially I made it just heal everything that had to be decoded, even if we had not tried to fetch it. This resulted in a lot more inserts, which caused the network to go a bit mad. Load management takes time to adjust to a big change such as a lot more inserts, and I released a new build (two new builds, doh) to fix the underlying problem; I am fairly sure that the major problem at the moment is still backoff caused by load management still being in the process of readjusting. Sorry folks...
If it doesn't settle in a week or so there may be a deeper problem, but I doubt it. It does seem to be impacting overall performance - the success rates by HTL has dropped somewhat for example. I would appreciate any input people can give, anyway. I am chatting on FMS and Freetalk lately, and also have read only access to the freenet board on Frost.
It is of course tempting to rewrite the whole load management system. Most of the changes we've made recently are things that have been "cooking" for quite a long time (years in some cases) but we've only put all the pieces together and been confident about what needs to happen relatively recently. I think we are at that point with load management.
Our current load management scheme has three components: First, the node estimates its capacity for requests, and rejects those it can't deal with. Second, if a request is rejected or if it times out or fails in transfer, the node routing to it "backs off" that node - it goes into the backed off state for a short period, and if it keeps on causing problems that period increases exponentially up to a limit; while a node is backed off we only route to it as a last resort. Third, each node maintains a set of additive-increase-multiplicative-decrease numbers, which function as a window estimating the number of requests that can run on the network at any given time. We use these to control the rate at which we originate new, local requests. This is intended to be similar to what happens on TCP/IP over Ethernet, but IMHO it's not a good match: We can make it react more quickly or more slowly, but it is never going to be able to deal with local network conditions (e.g. small numbers of fast darknet peers) efficiently, and makes tuning fairness between request types and peers tricky; but above all, it can cause a great deal of misrouting. This affects data persistence: A leading theory as to why data persistence is so poor is that inserts are rejected by the nodes that would have been the ideal locations. If you insert a block 3 times it greatly increases its chances of success - even though it should in theory go down exactly the same set of nodes! Our current load management system, in the worst case, degenerates to routing load to the least overloaded node, just as 0.5's did; this is however a temporary state while the AIMD's adapt, but it is built on a significant amount of misrouting and randomly rejecting perfectly good requests. Also, it is far from ideal in terms of security, as the originator runs the AIMD algorithm to decide how fast it can start its own requests.
Also, it is unnecessary. Discussions years ago on Frost, and more recently, the CCN paper (which is well worth a read) have shown that this sort of load management by guessing rates is unnecessary on a request/response system; what we need to do is send requests as long as we have available capacity to do so. A new load management scheme is proposed on this basis:
Anyway, this is probably something for 0.9. Hopefully the network will settle, and we can improve data persistence significantly by simple tweaks such as making nodes more likely to accept inserts. In the meantime, assuming we stick to the original plan, there are relatively few features left before 0.8.0:
Unfortunately, our third Summer of Code student, working on distributed searching and file indexes, dropped out. That would have been very interesting. The top item on uservoice, now by some distance, is "Write a killer file-sharing application"... IMHO a reliable built in forums system and significant improvements to data persistence would go a long way towards this, but easy to use spam resistant distributed file searching and index maintenance tools (similar to Thaw but done right) probably won't be part of 0.8.0.
In 0.9 I hope we will include major changes to load management, full selective reinsertion, basic time-based insert protection, distributed filesharing/file searching, web-pushing, hopefully a solution to the Pitch Black attack, and hopefully much more. More work on data persistence, and possibly bloom filter sharing, are also very important. Bloom filter sharing will make a significant gain, although it may only be worthwhile on darknet or peers we have a very long term relationship with; other network level tweaks may improve performance and persistence more, we need to do a lot more analysis of the network. We had planned to do Bloom filter sharing with Google's last payment, but what we've actually done is IMHO better. Passive requests and secure bursting are probably 0.10 stuff; both fix critical design flaws, so ideally they would be moved up a bit... We'd also like basic PDF and VP8 filters, and transport plugins; these might be Summer of Code projects. Sorting out multi-container freesites is also essential some time before 0.10 or so.
What else is going on? Well, Google decided, without being prompted, to give us another $18K (the third such donation so far!). We were down to our last $100, although I didn't have an exact figure so the website showed $600 - but only for a few days! I have no idea what their agenda is, especially given some of their high ups' recent comments about anonymity, but this means we can continue, so it's clearly a good thing! Certainly they're as close to "the good guys" as multinational corporations get, although of course I am skeptical about such single points of control/failure/censorship - he who can compel Google to block a keyword effectively erases it from the Internet...
Evan quoted my long term roadmap, most of it I stand by; depending on demand, we may move features up or down (e.g. partial reinsert support). Then he disappeared, and I had to take over his GSoC student, and apologise to Google for missing the mid-term evaluation deadline. He's publishing his stats again now though, although I haven't actually talked to him recently; maybe some emergency.
So that's it for Freenet. What else is happening? Well, I've seen a consultant, who was even more thorough than the senior GP when I eventually got to see him (having repeatedly seen locum's and not got very far). I have a bunch of tests scheduled, and hopefully we can confirm what the problem is that has caused me to lose far too much weight. I finally got around to upgrading my computer (having bought the parts ages ago), setting up wifi, and getting mum's downstairs Linux system working again (so dad can get some work done and not have to put up with mum's constant chattering with friends on Skype!), and am looking forward to a holiday in Wales some time in September.
I'm still theoretically looking for more contract work to diversify my business and earn more than I get from Freenet, but this is largely on hold until I am feeling a good deal better. If and when I stop working for Freenet, I may go to university; working for Google or similar is at least somewhat interesting, but I don't think I'd want to work in London, I don't think I could handle the pollution levels long-term; all my family have asthma and we can feel the difference when going out of a major city; it was one of the biggest reasons for us moving from Bristol up to Scarborough.
In recent ponderings about Ripple, IMHO when people start finding things to buy through online social networks, routable friend-to-friend currency will become a whole lot easier to bootstrap, and maybe inevitable. Technically speaking the problems are surmountable; the real difficulty is how to bootstrap it, but IMHO that may not be a big problem for much longer... My view is that full p2p is unlikely at least in the short term because of difficulties with finding good routes and liability issues on downtime, but a few thousand servers (run by different people) that exchange a lot of data with each other could likely find perfect routes very efficiently. Privacy wouldn't be perfect but it never is with money; Ripple isn't about privacy, it's about decentralising value creation and making money map more closely to what people (as opposed to corporations, governments and economists) want and trust. Of course it's a ludicrous experiment, and it may fall flat on its head - but it'd be fascinating to try. Not sure what I think about the circular barter proposal on the ripple list, it'd make sense to try it out in a computer game perhaps...
Meanwhile the world continues to march enthusiastically into the abyss. The latest situation:
We could do the right thing. Or we could wait for some magical new technology that is cleaner and cheaper than fossil fuels and takes no time, no investment and no subsidy to be deployed everywhere. Or maybe the next phase of the oil shock will save us. We'll see. I believe in divine sovereignty - but some terrible things have happened, and the good that came out of them only became clear a long time after.
Build 1255 etc
A lot of work has gone into build 1255. Most of that work has been related to the metadata layer: When we changed it slightly in 1251, a lot of users got upset about reinserts no longer working. 1255 will provide the ability to reinsert files exactly as they were originally inserted. It also implements a great number of metadata changes all at once, in order to minimise the disruption resulting from CHKs changing in multiple builds.
One big change is that anything inserted under an SSK will use randomised splitfile encryption. This means that it will not be possible to guess which CHKs will be used, even if you know the content that is going to be inserted. The upload form (which is now on a separate page) explains this and allows you to insert as a CHK or an SSK (with it generating a random SSK automatically for you), or specify your own key, or will soon have a reinsert option. This should greatly improve security on opennet, particularly against the mobile attacker source tracing attack that we have discussed before. Random splitfile encryption was actually agreed at the freenet mini-summit in 2007...
The other big change, which is arguably even more important, is cross-segment splitfile redundancy: For any file of at least 80MB, instead of just chopping it up into independant segments of 128 data and 128 check blocks (the size varies a bit according to even segment splitting), we add a second layer of redundancy which operates between segments in a similar way to that used on data/audio CD's. This greatly improves the reliability of very big files, assuming the same probability of a single block being found (or lost). The simulated figures are really quite remarkable, but we'll see what difference it makes in practice...
Other big changes include:
Various other changes include making it clear that the only way to get HIGH security level is to use darknet (usability input solicited), a new testing mode for inserts that only inserts to the datastore, various bugfixes, some nice optimisations to threads and temp files, etc.
Evan and FreenetUser have recently brought up the topic of reinserts. Reinserts are dangerous but seem to be a necessary evil at this point. On one level, 1255 significantly improves support for reinserts. You can download a file, it will detect the correct reinsert settings, and then you can specify them on your reinsert. On the other hand, we don't yet have a reinsert option on the upload a file page (we will eventually, it is pretty easy), and the option of inserting to a random SSK with all the keys randomised makes reinserts a little harder. But the real goal is to reduce the need for reinserts: Both better segment splitting and especially cross-segment redundancy will help significantly with this for larger splitfiles. These are both client layer changes: The chances of a single block succeeding are not changed, but better redundancy lets us dramatically improve success rates for (some) whole files.
But there seems to be considerable scope to improve the block level success rate too: We now have 3 flags on inserts which affect how hard we try to send the data to the right nodes. Previous test data suggests we may be able to dramatically improve block success rates, but we have to try this out to see what happens... That would be the big prize though, dramatically improved data persistence would make filesharing on Freenet much more feasible, and may make it a realistic option for relatively unpopular content which is often not available from bittorrent due to no seeds, which is an area Freenet should be good at IMHO.
So the first task after 1255 ships is to write a tester which inserts single blocks, tracks their retrievability after a week, and tries various combinations of flags. The data suggests we should be able to move from a mid-70s% success rate to a mid-90s% success rate, which would have a huge impact. And we can do this without having any practical impact on resource usage, just by making it more likely that inserts will be accepted on the nodes where they should be accepted. We might need to change the actual implementation once we know what the underlying mechanisms are. Arguably this links in very nicely with rewriting the load management (based on the two principles of severely limiting misrouting and only limiting based on the local capacity rather than trying to estimate the capacity of the whole network), but that is likely to be post-0.8; most likely we can come up with an acceptable set of hacks, maybe the ones that the flags implement, maybe not. I need to point out at this point that it was Evan who suggested, and did much of the theoretical work behind, even segment splitting, cross segment redundancy and the new insert flags...
Capacity based load management opens up some fascinating long term options however. Have a look at my post "[freenet-dev] Fast *and* secure Freenet: Secure bursting and long term requests was Re: Planned changes to keys and UI", for a proposal that is half way to long-term/passive requests, that works best on darknet but quantifies what can be done on opennet, and that in ideal circumstances (darknet), not only beats the mobile-attacker source tracing attack but does so while transferring data very quickly, and providing for offline data transfer - all the requests go out at once (so an attacker cannot use the stream to gradually approach the originator), and then the data returns as quickly or as slowly as possible, possibly taking alternate routes when nodes go offline and possibly waiting for nodes (even the originator) to come back online. This is not true passive requests because it only persists after the data has been found, but it goes a long way in that direction and would need the same sort of load management.
In the nearer future, the two things that we absolutely must have for 0.8.0 continue to depend on specific individual volunteeers: Freetalk, which p0s seems to be making strong progress on, and a working Windows installer, which is much less certain, but also hopefully easier. Other important (but not essential) stuff from the roadmap:
Another important point is we need a new format index to include as the default index. Please let me know if you run a Spider (you will also need to run Library).
A very belated answer to FreenetUser's post about OneToOne and private messaging. Freemail actually uses most of the tricks he talks about - in particular it establishes a private channel between two identities so that traffic analysis is impossible after the first contact. However, Freemail is buggy and more or less unmaintained. There have been some patches recently though, hopefully we can have a new maintainer... With regards to the content filter, we can whitelist plugins, we already do this with Library so you can have a search box on your flog (FlogHelper does this). Bombe's fast templating engine might be useful if you're still looking for one; it's faster than HTMLNode's so will probably end up used by Freenet eventually. You will need to deal with trust and introduction in WoT, otherwise you will get spam; there isn't much point using WoT if you don't plan to reuse identities between different applications and manage trust lists IMHO. You can reuse the existing interfaces though. But also it would be awesome to be able to click on a poster in Freetalk, go to his identity's page, click on a button and send him a private message; integration is good. If on the other hand you don't like WoT, you could maybe use hashcash per introduction or something... With regards to logging, as I'm sure you've discovered by now, you can use the log level details setting to show exactly what you want and not what you don't want.
On the subject of removing EXIF, the JPEG filter in the Freenet source can do this, and we (sajack) will soon introduce the option to filter files on insert to strip out incriminating data.
On databases, mentioned by FreenetUser but mainly by In The Raw's Gestalt, I would just like to point out that for any on-freenet database that needs to scale, a solution can surely be constructed based on infinity0's COW btrees structure that backs new format indexes. Have a look at the wiki page on the subject. IMHO we can provide forms of atomicity, consistency and durability; isolation is closely related to multi-user support, and it's likely that multi-user support will consist in merging changes from one fork of a database to another, so maybe a little different to traditional databases - even traditional distributed databases.
Then there's Project Evergreen, a distributed social network based on Freenet that currently has a GUI that doesn't filter content. IMHO something like this is a good idea, I'm just not sure exactly what; traditional social networking functionality is hardly compatible with hardcore privacy, for example, but on the other hand it is essential that users be able to produce content really really easily, and we really have two separate social networks in Freenet - the darknet, and the network of anonymous identities that is the Web of Trust.
Uservoice makes interesting reading as always:
What can I say about this? Improving persistence should help reduce the need for insert on demand and greatly increase the amount of data that can be found and downloaded with no hassle. That doesn't solve the problem of how to find it though. Forums work, but they work a lot better if you can fetch files that people posted a while ago. Once Freetalk is bundled, we can have a message be posted automatically as soon as an upload finishes, making it somewhat easier to post content. Pages with links to files ought to work but have never been very popular on Freenet, probably because of poor data persistence. Hopefully nikotyan's work on distributed indexing will provide a big step towards an easy to use, spam resistant file search system, although due mostly to exams etc we haven't seen much yet.
All the folk who voted for "One GUI for all" will presumably be a good deal happier once Freetalk is official and bundled. The third most popular item is essentially transport plugins; zidel's work will make this a lot easier to implement (as well as greatly improving the efficiency of the transport layer), but transport plugins won't happen any time soon. I'm surprised how popular the pause feature is still, given that reconnecting to opennet is very fast nowadays; maybe it's worth looking into anyway, but I'm not convinced.
And for non-Freenet stuff...
For the record, Nestle gave in; kitkats no longer cause deforestation. (I inserted and referred to a short video about this some time ago). Of course this doesn't help those of us with gluten intolerance...
Hardware wise, I have narrowed down the problems to the Velociraptor and RMA'ed it; the replacement should come tomorrow. Hopefully this will "just work", rather than stalling every few hours as the old drive did; it's a nice fast disk, although of course all these SSD's are tempting. But rumour is Intel will introduce 600GB models this year, in which case it's a really bad time to buy; and anyway, I don't have the money. Then there is the question of a CPU upgrade; I've acquired an X6 very cheaply, but I need a motherboard for it, and possibly to switch RAM types... Mostly CPU isn't an issue, but on the other hand I had zgrep running all night on logs recently and it still hadn't finished, apparently being CPU-bound...
In terms of wetware, struggling on; food, fibre and gastrointestinal tract issues are far from dealt with; I may have to go back to talk about chest issues, and spirometry still hasn't happened; and mucus is a big problem, which seems only to be partially seasonal, and seems related to both other issues, but we have some ideas on how to manage it (if it's not purely due to pollen then dust is probably relevant). One upside is I'm able to get a decent amount of work done lately, especially last week, although it was rather tiring.
Reading wise, I'm not reading as much as I was (which is a good thing, it means I'm feeling a bit better), and am still chewing through various short stories by John Wyndham, and a few technical magazines...
News wise, people are starting to realise that 25% budget cuts are going to be really hard - and that the tax/benefits measures in the budget were pretty much as regressive as you'd expect from a Conservative government. It has been said that modern government is a theocracy of the rather vague science of economics... Lets hope they're right, because if they're wrong and Obama (and the keynesians he listens to) is right, Europe could go the way of Japan... Also it's interesting that the recovery will supposedly depend on strong demand, while demand depends on reckless spending, reckless credit and not increasing VAT (doh!). Of course, I'm not really sure what to hope for here: Long term, consumerism will kill us all... Increasingly I subscribe to the view that this is just one of a series of shocks that are going to turn the world economy upside down. One of the big ones will be the next stage of the oil shock around 2018, which probably won't be cushioned at all by the current pathetically weak efforts to kick the oil habit before it kills us (especially if the rumours are true and the tories scrap the battery subsidy), but there are plenty of opportunities to blow it in the meantime...
Build 1251, 1252 and 1253 are all because of the same bug, which technically was discovered on Saturday, but because of my not immediately recognising the implications fixes were not fully deployed until today. Plus I was out shopping in the morning. Thanks to everyone who has reported bugs about this and nagged me about this - particularly to the original reporter on Saturday who I got into a verbal fight with after he didn't want to leave his contact address for future testing. Sorry about that, you did us a great favour and if I'd been more graceful I might have recognised what was going on more quickly.
The bad news is that the bug is potentially rather serious, and also goes way back: Odd-shaped splitfiles could cause the native FEC code to cause segfaults, or glibc memory errors. For all I know this is exploitable for remote code execution, although there is no evidence anyone has found out how to do this yet. It's a wierd bug: Using more than 100% redundancy is known to work with the onion FEC codec if you use the C API, so it must be something to do with the JNI wrapper (all this has been discussed on the chat today so is public knowledge regardless, as well as being mentioned in the commit logs).
There is however evidence that somebody was watching the #freenet chat logs, or the commit logs, with a fine toothcomb on Saturday; he has inserted a splitfile which triggers this bug in pre-1251 nodes as a Frost message. The reason we picked it up on Saturday in the first place was that 1) 1251 inserts splitfiles with >100% redundancy (it adds an extra check block to each segment of less than 128 blocks i.e. to most segments given even splitting), and 2) aforementioned person on #freenet bugged me about it when his system kept segfaulting consistently on fetching a specific key. Unfortunately I didn't realise then that splitfiles deliberately inserted with >100% redundancy would still cause older nodes to crash, and worse that segfaults can usually be turned into remote code execution vulnerabilities with a certain amount of work. I guess I'm still not 100%, although I'm certainly getting more done lately. Anyhow, by Monday, the attacker had inserted this key, which was consistently causing segfaults for some users (and another user reported a glibc memory error). Fortunately one of aforesaid users chased us up on #freenet. One lesson from this is if something is important nag us until we fix it! As regards potential compromises, only nodes which use the native FEC code are (were) vulnerable, so only 32-bit x86 linux and windows systems (mine is AMD64). It is possible that the few nodes testing freenet-ext.jar version 27 might have been vulnerable on other platforms, but this is uncertain, the newer code might fix the bug entirely. Ximin (infinity0) will have a look at this later this week hopefully.
Of course, there is a significant amount of other code in 1251/2/3, and I may as well explain it here. Firstly, splitfiles are now split up into segments evenly: rather than all but the last segment being exactly 128 data blocks and 128 check blocks (so the last segment sometimes has 1 data block and 1 check block, meaning it will get lost way too quickly), the segments are all almost the same size (the last one is only a few blocks smaller). And various other tweaks. This should help quite a bit with data retrievability for those files where the last segment is significantly smaller than the rest. And it's what ultimately kicked off the FEC errors...
The other big change is the content filter. A big chunk of sajack's work has been merged, filtering is now in the client layer rather than fproxy, it operates on streams rather than buckets (which will be used to significantly reduce the amount of disk I/O needed soon), and so on. And you can filter downloads to disk from fproxy, and downloads via FCP. Spencer (sajack)'s future work will include further optimisations and filters for more formats (e.g. mp3).
One other thing going on: A new version of Library. Previous versions were inserting new-format indexes in whatever the local character set was, and parsing them in it too. This was resulting in the indexes being unusable. We now force UTF-8. If you have been running the Spider you may have to dump its progress (shut down and delete library*, some of which are directories). A workaround was apparently found on Frost; those using this workaround should be able to continue with the new code.
I had begun to start thinking about cross-segment FEC, but it turns out that we need binding content hashes first. So I have a branch for content hashes in metadata. We will take the SHA-256 hash of the original, uncompressed content, and put it in the metadata. This will then be verified when the download completes, providing an extra level of integrity checking in case of bugs (useful for e.g. auto-updates, although they won't use this for some time). It's also useful for cross-network stuff - a file with the same hash is the same file, whichever network it is on. We will include other hashes on larger files and/or on demand, since other networks often use different hashes (e.g. sha1, sha512, md5). Also, we can use the hash to generate a single key for all the blocks in a splitfile, and thus we only need to store the routing key, not the crypto key, saving 32 bytes per block and halving the size of splitfile metadata (at the cost of some coalescing). And once we're doing that, we can offer the option to randomize that key for the paranoid, because predictable content (reinserts, but also e.g. inserting a specific leaked document or video) makes life a lot easier for an attacker trying to trace somebody inserting content. Of course we would allow reinserts with the same random crypto key for those willing to accept the risk, and we might eventually end up doing some healing between splitfiles with the same hash but different crypto keys, or even elaborate schemes where the insertor uses a random key and then some of the downloaders (who are presumably less vulnerable) reinsert it with a known key... But the immediate application will be to use the overall content hash to generate "random" numbers needed for cross-segment FEC (we will need to use a different hash to seed this than we use for the crypto keys, probably will use hashing the hash with a constant, like JFK uses).
1249 and 1250 both fix serious bugs:
1249: freenet.ini doesn't exist: 1248 seems to have triggered many Windows nodes to update their config. Unfortunately, because of an older bug, when we do this we create freenet.ini.tmp, write the data to it, forget to close the file, can't rename it over the old one because we didn't close it, remove the old freenet.ini, try to copy over, that fails too, so we end up with no freenet.ini. And then the launcher refuses to start because there's no freenet.ini. This only affects Windows but is pretty bad. If you're affected (if you're reading this over freenet, as opposed to over the web, you probably aren't), rename freenet.ini.tmp to freenet.ini manually.
1250: Writing inserts to the client-cache (and possibly the datastore): Inserts were written to the client-cache. They were not supposed to be. This means if you insert controversial data, it will be in your client cache, and if your computer is seized, the bad guys may be able to see it. Note however that if you view the content you've inserted (after the 30 minute period immediately after) it will go into the client-cache anyway; the client-cache is pretty dangerous, you should secure it properly (set physical seclevel to maximum and restart when you want to clear it, set it to high and don't give the bad guys the password etc). Also, in the rare case of an insert via FCP where the client software explicitly asked for the data to be cached in the client cache, it might have been written to the datastore - and hence be remotely probeable.
Sorry folks! However, 1250 also has major improvements to the USK code, as I have described in the announcement mail, which I quote here:
Freenet 0.7.5 build 1250 is now available. It will be mandatory on the 15th. Changes:
The next step is to merge sajack's content filtering infrastructure work. After that I will probably tackle even segment splitting, probably the new MHK tester, and then maybe cross-segment redundancy. There haven't been any major objections to the roadmap (or even much in the way of comments on it) so far, not even Ian's usual emphasis on a prompt release, but I guess without Freetalk there's no immediate rush...
1248 and the Spider
Freenet build 1248 is now available. But more importantly, the new Spider plugin and major changes to the Library plugin are now released. These depend on 1248, so a brief changelog to encourage you to upgrade:
Thanks to toad, hungrid, p01air3, xor, sajack, zidel and nikotyan.
The new version of Library and new Spider plugin together allow creating new format indexes by spidering Freenet, in much the same way as XMLSpider allows creating old format indexes by spidering Freenet. The new index format is far more scalable than the old format, and should improve performance both of the spider and the client (Library) doing the search. It has other benefits which I have discussed elsewhere e.g. it is possible to fork a new format index, merging your data to somebody else's index to create a whole new index without having to reinsert everything.
New format indexes are automatically inserted by Library on behalf of Spider. Data from Spider is first written to library.index.data.<number>, then merged into an on-disk index in library-temp-index-<different number>, then merged from disk to Freenet. Only the data that has changed (and the nodes above that data in the tree) are uploaded, and once the index has been updated, a USK is inserted, which is logged in wrapper.log (you may have to grep for it, it's rather chatty).
New format indexes include ranking data, and are functional now in much the same way as old format indexes are, although they should be faster and the user interface during loading isn't quite the same.
Both new and old format indexes now index numbers of 3 characters or more. Also as of the work from the previous build we have reasonable Chinese support.
To use the new Spider, you need to load the Spider plugin (not XMLSpider); Library will already have been upgraded by installing 1248. Then configure it. The only essential options are the number of requests to run, which are two options right at the top. You can also configure keyword blacklists if you want; these are useful both for shameless censorship (or being able to sleep at night depending on your point of view) but also for excluding known spider traps and spam sites. The code will do pretty much all the rest of the work, as you can see by watching wrapper.log. It will spider Freenet, write the data to disk, merge it to big on-disk trees, then merge the big on-disk trees to Freenet, then upload a USK for each one. All of this happens in parallel. Your first USK should be uploaded within a day or two. Expect the process to slow down significantly over time as the index gets bigger (currently my test indexes are taking 7 hours to finish uploading but I expect it will be several days when it gets a bit bigger) - but then the spider itself slows down significantly over time as it runs out of easily findable stuff. I have been testing with max memory set to 1024 (1GB) in wrapper.conf so that should be enough. Note also that we cache everything we upload in library-spider-pushed-data-cache/ , which is currently not garbage collected; you can remove it but then you are relying on the data being retrievable when we need to update it.
A note to would-be standard index authors:
YOU MUST BE ANONYMOUS. I have been running the spider for testing purposes but I will NOT publish any produced indexes. You should publish anonymously, and take your anonymity very seriously, as you are a clear single point of attack/vulnerability; if you are taken down this affects the experience of Freenet for everyone but especially newbies. Note also that if you censor anything, and are traceable, you may be legally compelled to censor everything that is illegal in your jurisdiction. Create an identity (in Frost, FMS or Freetalk) solely for announcing your index and post the USK there. When an index is popular enough and complete and consistent enough, we will add it to the default indexes list.
Historically we haven't been very good at doing things in priority order, and it can sometimes be difficult to tell what's going on, or what needs to be done next. Financially speaking we have a brief breathing room, and there is much to be done on Freenet: the developers have a better grasp of the outstanding challenges than we have had for quite some time, and it is tempting to just get on with it, implement lots of important features because if Freenet works, people will use it, release or no release. However, 0.8.0 is significant for publicity reasons, and sooner or later - hopefully sooner - Freetalk will be ready and we'll have much less of an excuse for further delays. And our finances are far from secure, we have around 15 weeks including my last (in advance) payment. So lets talk about priorities. I list below most of the important things that need to be done in the near to intermediate future. Many of these may be dropped in the name of getting a quick 0.8.0 release, but many of them will be so transformative that we should seriously consider getting them out before 0.8.0. This is an expanded version of a mail I sent to the devl list today.
Critical stuff depending on other people
Important stuff depending on other people
Critical ongoing stuff
Important easy feature work
Fairly important security work
Important ongoing stuff: data persistence
When we have enough data from the new MHKs tester, we will probably need to make further changes to routing, storage etc to improve persistence. If it is still true that triple inserting data drastically improves its longevity, we need to know why, and we may be able to adapt inserts to achieve the same result with less than the cost of inserting the same block three times.
As part of this effort we will need random-routed tester requests at some point. These gather data such as average quantised datastore size. The data they gather is separated from any other data so is more or less useless to an attacker. Also, a variant will do displaced fetches/inserts, which are useful for testing, but also offers an "is my site retrievable" feature.
Larger features that are worth seriously considering
Key usability stuff
Summary of short-to-mid-term roadmap
I reluctantly propose to postpone binding splitfile hashes and many other things as not being vital for 0.8. There are many other bugs and important features which should be implemented after 0.8. Most of them are on the bug tracker.
Summer of Code
nikotyan (who is currently busy with exams, as his mentor, infinity0, was until very recently), will be working on distributed indexing. This could and should result in a good spam-proof file searching system.
sajack (mentored by nextgens) is working on content filtering. He has posted a pull request for architectural changes moving filtering into the client layer and out of fproxy, which will allow some important optimisations for big files when he gets to work on audio/video filters later on, as well as allowing us to filter data downloaded direct to disk. He has also given us the HTML changes mentioned above. If he has any time left at the end of summer he may start a minimal PDF filter (the PDF spec is huge, but a great many PDFs use very few features).
zidel (mentored by evanbd) is working on a new low-level packet format which will use padding to transmit data, thus greatly improving payload percentages, work well on small MTUs, and pave the way for transport plugins (especially with small packet sizes). So far there have been some questions about refactoring and some code I haven't looked at yet. I'd appreciate a status report if zidel has got far enough to make one.
In non-Freenet news, feeling very slightly better, particularly digestive system. Chest isn't great but might just be a cold; if it's a chest infection it will sabotage the spirometry, but they scheduled it in 6 weeks to try to avoid that. Got over 31 hours of work done last week - admittedly only 27 of it billable to FPI. Nonetheless, looking up. Looking for contract work semi-seriously, partly because I'm gonna have a load of hardware costs in the near future and financially iffy even without that, partly because FPI is precarious as always, etc; contact me if you are interested in a cheapish java programmer with extensive experience in Freenet (networking, crypto, web interfaces, db4o, etc, but sadly no SQL/J2EE, which seems to be what most people are looking for), see my homepage.
With a great deal of luck AMD will RMA thomas's X6, in which case I will have a new CPU, twice as fast as the current; even if they don't it may be worth risking a cheap motherboard in an expendable box as there aren't that many bent pins and as far as I know the reason it doesn't work is simply that he put it in a motherboard that doesn't support it... We'll see how generous they are feeling (as I've explained the situation is their fault - it should be possible to remove the fan without pulling the CPU out, as it is with Intel's better-designed slots). In either case I'll probably upgrade to 12GB of RAM, and T's two old motherboards will find homes in other systems around here (family of geeks; he's upgrading to a Core i7). I've also been wondering about getting an SSD, it would rather be overkill though... Hopefully when I'm not running the spider any more Servalan will behave a bit better, although of course the disk I/O optimisation work I mentioned above should also help significantly - not only with my system but with everyone else's too!
Politics and technology
Meanwhile the politicians are rumbling about sharing the pain, buy-in and fairness... while talking openly about "tough decisions" on pay, pensions and benefits. Expect drastic cuts in benefits, generalised tax rises (although we may see some semblence of fairness), and of course "difficult" spending cuts... Meanwhile, an EU carbon cut of 30% by 2020 seems to be dead, with France and Germany opposed, presumably joined by Poland etc. Miracles can happen; but no sign of one at Bonn so far, from the minimal coverage I've seen. Exclusively developed world professors talking about voluntary action to create jobs and improve energy security, while america's easy option for energy security continues to pour vast quantities of crude into the coastline; the most basic, essential projects using current technology continue to be blocked because they spoil the view, and so on ... Voluntary action is code for no action, business as usual, deploying technologies only when they are perfect, or at least cheaper than fossil fuels (carefully avoiding accounting for the delayed costs of carbon, the immediate human costs of pollution, and anything else that can be safely thrown out the window in the quest for a short-term profit). When I was in Copenhagen there were posters up arguing to develop tech now and cut later; the fact of the matter is (at least the consensus view) that if we in the developed world don't deploy the technologies we have now (which cost a little more than the fossil-fuel equivalents, but create jobs in an industry that won't disappear overnight in a puff of red numbers), the cuts we will need later will be of such magnitude as to be infeasible without major pain, and likely require large scale geoengineering as well. This is the Last Parliament: If we don't start to make major progress by the next UK election in 2015, life gets a whole lot harder. Meanwhile, the oil shock will be on us (again) in less than 10 years, and proactive action now will likely save us a lot of trouble.
A few technologies that may eventually change the game by being cheaper than the alternatives are electric cars (assuming lithium-air batteries, 10 years+, and widespread infrastructure, more than that; car companies estimate 10% of sales could be electric by 2020, it's just way too slow; there are tens of thousands of deaths from air pollution per year just in the UK, even ignoring climate), and kite power (which according to estimates published in New Scientist should eventually be cheaper than coal, while providing reliable power because it reaches into the jet stream), and the Green Grid (when it blows in russia it's not blowing in england etc). Meanwhile, those who invest in BP for their retirement get exactly what they deserve; ah well, not quite.
Books, technology, justice, sin and society
Reading-wise, just finished The Chrysalids, which is an interesting Wyndham tale featuring a post-nuclear-holocaust world, mutants, telepathy, inter-species ethics, and more; many of these being topics that come up in his other works, which are well worth reading (albeit perhaps at arm's length!). IMHO a human subspecies isn't going to emerge spontaneously any time soon, and I don't believe radioactive fallout would necessarily result in one either; in any case that particular horror has receded slightly (but not by as much as people assume, it's gonna be a rough century). However, technology presents many such issues; I strongly recommend Rainbow's End (Vinge, 2006), which is an interesting tale which manages both to be largely ideas which are either inevitable or are being talked about very seriously by scientists, think tanks etc, and to push forward his singularity agenda (singularity here not being artificial intelligence as such, but enhanced intelligence, including telepathy). Of course many such ideas (e.g. really cheap biotech and its consequences both very positive and very negative) are likely to be further off than he envisages, and some of them may turn out quite different: Re bioterror, the good guys benefit too, and have way more resources than the aggressors, and re telepathy, much comes down to how fast you can compose text in future eye-tracking, brainwave or direct neural interfaces. Smartphones seem to be an important development, which may result in useful augmented reality and the aforesaid ubiquitous computing interfaces; one day I'll get one, maybe an N900, the tradeoff between the principle of open root access versus the possible employability benefits of Android is tricky... Of course, setting up wifi locally would be a first step. One thing to note is that reading in bed from a laptop is not conducive to a good night's sleep; an e-reader is far preferable physiologically. And all these gadgets - and the upgrades that seem essential to make Servalan behave itself - cost mucho money. That can be reduced significantly by time, Moore's law, and so on; but there will certainly be various forms of digital divide for some time to come.
And that's before we even get onto the topic of enhancement drugs (ritalin, modafinil, etc). Personally I don't smoke (duh), drink (matter of taste mostly, maybe genetic), or take illegal drugs (quality control), and I consume relatively small amounts of caffeine, but well, Jesus took drugs, and helped others to take drugs, so I don't have any moral problem with it (and don't try to argue that alcohol isn't a drug, it's ranked #4 by the UK advisory committee on aggregate societal and individual harm, after only cocaine, heroin and one other). I'm also rather skeptical of any claims about chemicals being safe (despite being a geek); for quite some time we bought organic food, partly because dad's guts seem to be able to tell the difference, until finances made that impossible to sustain. The point is even before we get into issues of human genetic selection and in the long term human genetic engineering, there are a great many things that divide us within "what it is to be human". On a more practical level, 2 billion people don't have access to decent sanitation, nearly a billion don't have access to clean drinking water; the global poor are far more vulnerable to natural disasters, frequently have no access to capital, and are stepped on by the powerful in ways that we middle class citizens see much less often and much less blatently. We think we're just hanging on, because of our perception of what we need, based on (mostly) artificial scarcity resulting from the market - they really are just hanging on. And frequently it's our fault - broken promises, outright exploitation, discredited ideologies, and so on. Often we can make a difference as citizens and consumers.
I guess the point, if there is one, is that 1) the powerful, who have all the advantages (access to technology, capital, hygiene, etc), don't need to exterminate the weak, because they can either ignore them, exploit them, or blithely destroy their future, and 2) human nature is the fundamental problem. Can human nature be changed? Lewis said no (more or less; see The Abolition of Man and to a lesser but more entertaining degree That Hideous Strength); Egan says yes (see e.g. Chaff in Luminous, another Egan collection of short stories). An individual human can change himself, although there are limits; technology may change what those limits are, but ultimately the difference isn't all that fundamental, it still comes down to your choices as a moral actor. If you choose to be purely self-interested in a short-term and cynical sense, technology may be able to help you; if you choose to care about other people, again technology can help you. Even if what it is that is making the choice is itself malleable (which of course it is to a degree, even without technology), humans are responsible for their actions, and they frequently choose very badly.
The other book I read recently was Greg Egan's Oceanic. This contains some fairly blatent attacks on religion, and some equally blatent techno-utopian content, but as an honest ideas writer, he also talks about the problems of technology. There is a spectrum of ideas from post-scarcity economics (essentially Star Trek economics, meaning that everyone can afford the basics of life because the basics of life are ridiculously cheap; arguably this is practical now, nobody really wants to sit on the couch watching TV all their life unless they have bigger psychological problems, see the Citizen's Income) up to people as software, with backups all over the place, living as long as you like (often, but not always, conveniently glossing over the issue of fertility in a finite universe), invulnerable to any form of interpersonal harm. IMHO again this does not change the fundamentals. It might be centuries before we can upload consciousnesses, if the Penrose/quantum tubules lot are right, but fundamentally if people interact with each other there is the potential to hurt one another. In a society with a legal system, the forms of harm are supposedly limited by deterrence (although in practice deterrence is never enough). In a world where nobody need starve or be enslaved (for example being forced to be exposed to hazardous chemicals), these are further limited; in a purely virtual world (as in e.g. Diaspora), these are again limited. But a great deal of harm in the real world comes at the more refined level: Betrayal, social exclusion, unkind words, psychological bullying, and so on. Of course if physical harm was impossible the edge is taken off much of this. But does that mean that if we can provide our needs for practically free and back up our consciousness so that even if we could be physically harmed we could always come back, concepts such as sin are outmoded and salvation is unnecessary? Not in my book.
Freenet: New format index Library/Spider progress
Lately I have been working on the new index format for Freenet keyword searches (the format has many other Freenet applications). As I've explained, this is a massive improvement on the old format: It is scalable, it can be forked easily, it supports selective uploading when merging new data into an on-Freenet index, and various optimisations promise to make it (at least for popular indexes), really fast. It is also the topic of one of our Summer of Code students, who is supposed to be working on distributed searching - using the Web of Trust to find indexes, and searching all of them (with lots of scalability optimisations), and making it easy to publish and maintain your own file indexes within this so that we can start to have a decent filesharing search system. The new index format is needed for its current primary application as soon as possible, it will have a big impact on the newbie end-user, because it needs to fetch a lot less data to answer a search and hence is faster, and it's a lot easier to maintain a new format index; so I am working on it now as it's a big and relatively easy win, infinity0 having done most of the work for me.
Getting it right has been a lot of work and an awful lot of waiting for computers to run tests. Probably the biggest area has been making it work well within a limited memory budget. Recently this involved reorganising update() to process the data in order in such a way that the queues didn't explode, a neat bit of optimisation by queueing theory (and no I've never studied queueing theory). Anyway, right now, if you load an up to date (git) version of Library, the new Spider plugin, configure it (set the number of jobs options, the buffer size, and possibly the bad keywords list if you want to exclude specific sites e.g. spider traps), the Spider will happily spider Freenet, feeding the data to Library in large chunks (containing a series of keywords and a bunch of sites and word locations where they occur), which will upload a freesite index, updating it with each such bundle. It automatically inserts a USK for each such upload - we don't display this in the UI at the moment but it is in the wrapper.log / standard output.
The problem, of course, is that this is slow, and it gets slower the bigger the index gets, so Spider ends up waiting most of the time for Library to do its uploads (even though we allow quite a bit of slack). If we assume the on-Freenet index we are updating is very large, we'd expect it to have to update one subtree for each term, plus one leaf of the main tree, plus all the nodes above that leaf. Making the leaves or nodes smaller won't help much because it undermines compression - the nodes are several megabytes of programmer-readable text, but they compress really well down to a relatively small number of keys. So what we will have to do is merge the data from Spider into an on-disk tree, and then periodically merge that (much larger) tree into the on-Freenet tree (creating a new temporary on-disk tree at the same time). The resulting uploads to Freenet will take a long time but should be reasonably efficient due to e.g. overlapping terms from many smaller updates.
Freenet: USKs and Freetalk
Freetalk is rapidly approaching a new jar and hopefully semi-official status (i.e. inclusion on the plugins page with a warning). It has around 22 critical FIXME's left for p0s (xor) to fix before that can happen; many of these represent serious bugs that could be exploited for a denial of service attack, so they have to be fixed before we can make Freetalk easily available from the plugins page, even with a "caveat emptor" warning.
Both Freetalk and FMS use "outbox polling" to deal with spam: Frost, an earlier chat system, used a single global outbox, writable by anyone, to post messages; this was implemented as a KSK queue, so a message might be KSK@frost-boardname-date-index - where the last part, "index", is just a number incremented for each post that day. As has been demonstrated by some helpful person, who on Freenet can remain anonymous forever, and whose motives could be anything from benign to malign to sheer curiosity, it is very easy to break this system, precisely because anyone can post to the global queue. This is why Freetalk and FMS use a different system: Each identity has a separate outbox queue on which it publishes messages, identities trust or distrust each other in a Web of Trust, and your Freetalk will watch the outboxes of all the identities it trusts. Of course this cannot possibly scale - but neither can Frost. In practice there are several reasons to think we can make it work up to a reasonable size - we propagate "hints" about recent messages, Freenet can be reasonably efficient if lots of nodes are polling the same key, we don't necessarily have to poll every identity that we indirectly trust at the same frequency, and so on. Anyway, FMS's outbox queues are per-day, that is, you have something like SSK@blah,blah,blah/messages-date-index. Freetalk's messages are not per-day, we have SSK@blah,blah,blah/messages-index. FMS's approach has the advantage that we can immediately find the first message for a given day; the drawback is we need to figure out which message is the last for that day. Freetalk's approach desperately needs date-based edition hints - something like the index number for the first message in a day inserted to SSK@blah,blah,blah/messages-date. Freetalk does not yet implement this (although the other hints I've mentioned mean it's not quite as bad as you'd think), and it relies quite heavily on USKs. For a long time, it has been planned for USKs to have a similar mechanism...
USKs are a kind of updateable key implemented in the client layer. CHKs and SSKs (and KSKs, which are a kind of SSK which everyone can write to and thus is not very useful as shown by Frost), are implemented by the node. USKs are essentially a sequence of SSKs. So USK@blah,blah,blah/sitename/number is actually SSK@blah,blah,blah/sitename-number. But when you fetch a freesite by a USK, the node will check whether it knows of a more recent version of that USK, and if so, redirect you to the new version. It will also start a fetch in the background for later versions of that USK; in future (when web-pushing is fixed and enabled by default), Freenet will provide a pop-up notification if it finds a later version of a site while you are browsing it. You can also force Freenet to find the latest version, by making the number negative (note this will take some time); or you can add a site to your bookmarks, and your node will "subscribe" to it, checking for updates frequently for as long as it is bookmarked. This subscription mechanism is what Freetalk does so it doesn't have to poll the slots itself.
The problem is, USKs are not terribly reliable. When you click on one, unless it is bookmarked, there is a good chance it is way out of date. Even if you subscribe to a USK, it can take the node a long time to catch up if you started at an old edition. There are nasty bugs related to multiple clients subscribing to the same USK at different editions, and so on.
So while waiting for the Library/Spider combination to run out of memory, I've been making a start on long-overdue serious improvements to USKs. The most critical thing will be date based hints: Inserting a hint containing the latest edition to a key determined by the date. These will be hierarchical, and based on human-readable dates, so on inserting a USK update USK@blah,blah,blah/toad/23 we would also insert to USK@blah,blah,blah/toad-USKDATE-2010, USK@blah,blah,blah/toad-USKDATE-2010-05, and USK@blah,blah,blah/toad-USKDATE-2010-05-27 (the exact patterns have not been finalised, IMHO it's important to have a week number one as well). Of course, if any of these have already been inserted, we can't change them. So when we fetch a USK, we can fetch the year, month and day hints (and if we are having difficulty, maybe try last year, last month etc), and quickly get a rough idea of the latest edition - and then fetch from there. This will prevent us from getting so far out of date that we can't find the latest edition, or can't find it quickly enough. It is also exactly what Freetalk is missing for its polling. I have not yet started implementing hierarchical date based hints yet, as there is much refactoring and bug fixing to do first on USKs, but it will be done soon as it is essential for Freetalk to work efficiently, and would significantly improve Freenet for end users. I am already very close to fixing the problems with multiple subscribers hinting at different editions, which took quite a lot of refactoring to deal with.
I am still waiting for the NHS to contact me to organise a date for my next lot of chest tests (spirometry); guess they can't be that urgent. Remaining health problems seem to be improving; I've learned a lot more about such things than I've ever really wanted to, but seem to be getting there.
Computing wise, Servalan continues to randomly pause the whole UI for annoyingly long periods (a minute or so?). This doesn't appear to be a swapping problem. I've added the linux Compressed Cache driver, which is a fascinating piece of code; I've always been of the view that correctly applied, compression should be able to improve performance rather than just available space (IMHO this is true of filesystems as well, for stuff that doesn't need a lot of within-a-huge-file seeking; see Reiser4 cryptcompress plugin, BTRFS compression support). In any case I've had pauses even when I'm not in swap, or only marginally in swap, although now I have this fairly scary picture:
OrigDataSize: 3507080 kB ComprDataSize: 820801 kB MemUsedTotal: 1111584 kB
Considering that the system has 8GB of RAM, a fast disk for freenet logs, source code, root and swap, and RAID1 on slow big disks for the rest, this sort of thing really shouldn't happen or be necessary. Aggravating factors include a MythTV backend (and the requisite mysql), my rather excessive use of tabbed browsing in Firefox, Eclipse, Freenet with a 1GB memory allocation for aforementioned library/spider indexing tests, KDE 3.5, and so on. As I understand it, the pausing is caused by heavy backlogs of disk I/O; the vast bulk of disk writes, and a significant proportion of total disk I/O, is down to Freenet logfiles, so it may help to move them so that only logfiles are on the fast disk... Although RAM appears not to be the immediate problem, I have repeatedly pondered getting 16GB of RAM ... but it really isn't affordable, certainly not at the moment. The new AMD Phenom 2 X6's look very impressive too, but I don't think that CPU horsepower is the major problem here at the moment. My brother recently bought one, discovered it won't work with his AM2+ motherboard, and got the somewhat familiar I-can't-unlock-the-chip-without-taking-the-fan-off-and-I-can't-take-the-fan-off-without-pulling-the-chip-out RMA generator design flaw...
Had an interesting discussion with saces (who is working on making Freenet work with OSGi so we can have proper plugin dependancies, and who has a reprap), featuring various interesting hardware; he was hoping to use Powered USB to connect to his sealed rooftop freifunk hub, but it looks like the cable length wouldn't be acceptable even if he could find the connectors. Which brings me to a few pieces of hardware that if mass produced cheaply could have a major disruptive impact. I've talked about this before, just as much of what I've said above partially duplicates past musings, but ideas evolve over time, I'm not only repeating myself. Anyway, my 3 favourites at the moment:
I have recently been conversing with this blogger, whom I will call Ted. Particularly on this post. Initially Ted's argument was that science is useless to religious people (because any observations that don't make sense must be miracles) and that religion is useless to scientifically minded people (for all the usual anti-miraculous reasons). IMHO I have debunked the first point adequately: Just because occasionally God does miracles does not mean that the reason your experiment isn't doing what you think it should is because of miraculous intervention! The second point goes straight into apologetics: Is it possible to argue rationally from evidence to the divine? The main objection in principle as I see it is that miracles are so extraordinary that you can require any evidential threshold you like, taking the view that any implausible unsubstantiated absurdity is more likely than the proposed miracle. But extraordinary things do happen - many have happened in the last 10 years, the last 100 years. How do we deal with this? Of course this is without actually investigating any alleged evidence. I know far less about historical apologetics than I should, although I have some idea about scientific apologetics - universal parameter tweaking versus inaccessible multiverse etc, but I have recently started to have a look at a rather thorough book on the topic (The Resurrection of the Son of God by NT Wright)... Have a look anyway! Most of the debate centers on interpretation of the consistency principle.
It has recently become clear that joining the ACM as a professional member has some interesting benefits and is relatively cheap - especially as I'm not a graduate and may not be for some time. However, their Code of Ethics implies respect for and adherence to software patents:
"Violation of copyrights, patents, trade secrets and the terms of license agreements is prohibited by law in most circumstances. Even when software is not so protected, such violations are contrary to professional behavior. Copies of software should be made only with proper authorization. Unauthorized duplication of materials must not be condoned."
I despise copyright infringement, in virtually all cases (there are exceptional cases where there is an overwhelming public interest, see e.g. scientology). Casual copyright infringement undermines attempts to build a sustainable future - free software, free culture, truly free music (e.g. jamendo, magnatune is better than half way), and so on. However, software patents are entirely another matter. As the FFII has extensively documented (some of the more ludicrous examples), even those patents which are likely to be held as valid, being technical (which lately means "can be implemented with technical implements such as pen and paper", see the Hitachi decision), new and non-obvious (both of which are interpreted by the EPO as weasel words), are frequently either unavoidable or the inevitable solution to a small everyday problem which is discovered in parallel every time some programmer tries to solve that particular problem. It is not practical for me to read every patent, especially as they are written in legalese and many of them are so broad as to be invalid, or there is prior art; and only implementing algorithms I have created myself is no defence.
Practical, shipping software will inevitably violate many patents. No small company which ships software (as opposed to a patent troll) can ever successfully sue a large company owning a lot of patents: they are forced to cross-license, and the big companies usually cross-license voluntarily to avoid mutually assured destruction. And open source software has no per-copy revenue from which to pay license fees, even if that were reasonably possible (which it isn't due to the sheer number of patents a real product will infringe on). Software patents must end.
However, ACM has continued to send me nagging mails to try to get me to join even after I explained this to them. I am interpreting that as they still want me to join even though I disagree with a minor point of their code of ethics. Since it appears to be good value (I occasionally buy papers anyway) and important to professional development, I have joined, and emailed them again with my concerns post-joining and my membership number. I am also documenting this here, and linking it from my home page, to make it as public as possible.
Of indexes, illnesses, and elections
I've added Now What? to my flog links, as well as evan's freenet stats page. His latest post (concerning aggregation) suggests we really need to merge kurmi's ATOM filter from last year's Summer of Code, and ideally write an RSS filter (the problem with RSS is there are many different formats and they are all rather chaotic; ATOM solves these problems but is less well known). With regards to Freetalk, xor has recently discovered a major bug (lots of stuff that should have been indexed wasn't), and new Freetalk builds should be significantly faster. These will depend on 1248 due to internal API changes. Progress is being made, Freetalk will become semi-official (= listed on the plugins page drop-down) soon, but then we've been saying that for a long time ... At least plugins are shut down properly on node shutdown now, xor has been bugging me about that for ages. As for many simultaneous USK subscriptions, this is certainly somewhere we could improve things; the most immediate priority with USKs is date based edition hints (see bug 150; important for Freetalk in medium term), but we may want to e.g. make sure that if possible we always poll the next few slots sufficiently frequently that we should get notified by ULPRs on a new version; at the moment polling is based on exponential backoff (with limits), so we gradually try more and more future editions and poll the obvious next few less frequently - using a lot of request slots for little gain, if we are sure we are already fairly close to the current head; this is less than perfect for this sort of application. Also, exponential probing (as opposed to increasing the number of slots probed slowly) might help, but given that Freenet is unreliable, it's far from a simple binary search. See bug 4139 and 4046. Oh, and yes, embedded Freetalk forums in flogs is planned. You can already embed a search engine, FlogHelper does this.
With regards to default bookmark indexes, our policy officially is that we decide which indexes to link to on the basis of how useful they are to a user (probably a newbie) trying to find stuff on Freenet. IMHO good labelling and categorisation make an index much more useful to most newbie users, who are not familiar with the endless acronyms for "excrement". This was why we dropped Another Index - there was a child porn site under the Indexes category with no warning about its content, no text, absolutely nothing to indicate what it was, except the name, which newbies would not have been able to decode. We cannot refuse to link to an index just because it contains links to objectionable content - however, we can refuse to link to it if 1) it contains objectionable content (e.g. grossly offensive activelinks), 2) it is publicly writable (e.g. The Public Index) and therefore vulnerable to vandalism, or 3) it doesn't adequately label content and therefore is likely to scare away and put off new users.
Evan's work on LDPC codes or cross-segment redundancy should radically improve the reliability of big files. This is well up the todo list, although it may involve significant source code and data structure changes, and evan doesn't seem to be making rapid progress on a good LDPC algorithm. Selective reinserts at a much lower and more precise level than suggested are of course possible and will eventually be implemented as many filesharing tools will want them; however, all predictable inserts are inherently very dangerous, so I am generally rather skeptical about insert-on-demand and reinsert-on-demand solving all our filesharing problems. IMHO there is room for radical improvements to Freenet's current rather poor data persistence - at the FEC redundancy level, but also at the routing/network/block storage level. Much work has been done on this and much more needs to be done; I will be implementing some tester code soon that should throw a little more light on it (specifically, whether bug 3639 has made much difference, and whether triple-inserted blocks still have vastly higher persistence rates). Of course my strategic point of view doesn't mean I have anything against higher level filesharing tools based on kludgy solutions and reinsert on demand, although IMHO they should make the risks clear to the user; if such tools are plugins they could well be official plugins, on the plugins page and maybe even bundled if they have a sensible architecture and good code. As always, a serious strategic issue is churn: If we have 100,000 new users but they have on average very low uptime, most data won't be readily available unless we have a lot of redundancy. However, current statistics aren't as bad as feared, and opennet bootstrapping is much faster than it used to be so new opennet nodes become useful much more quickly.
Two other serious problems with big file filesharing at the moment: Firstly, if you have a big queue, Freenet really thrashes your disk etc, often slowing down your whole computer. We have done some optimisation work this year, and there is a lot more we could do; this is IMHO a reasonably high priority. Secondly, how exactly do you find content? We need a good, scalable, spam-proof search system. One of our Summer of Code students should be implementing Web-of-Trust based distributed searching. This will be based on the new index format - a big piece of infrastructure developed in last year's Summer of Code, which I am now deploying.
My current focus is to get the new index format working with XMLSpider. The old index format is not scalable, not forkable, and not incrementally updatable. The new index format is all of these things and with some small changes should be really fast as well as fairly robust against losing keys. Without the planned changes it should still be a lot faster on the client side than the old format, and as I mentioned it will be inserted incrementally, elimating the current week-long disk-thrashing index generation.
In detail, if you load recent versions (not yet published jars pending ongoing debugging) of XMLSpider and Library, and configure a maximum buffer size for new format indexes, XMLSpider will periodically send a load of data to Library, which it will merge into an on-Freenet new format index. At the moment it's a bit slow, but it is likely that once the spider has got through your datastore, index updating won't be the bottleneck. I've also written about this on the mailing list. Note that while I am of course testing this, I cannot release an index myself, for legal and practical reasons; I suggest that index inserters, people like wAnnA who should take their anonymity very seriously, should start to look into the new code soon - at least once I've got enough bugs out to put out an official release. One other change - XMLSpider now supports a keyword blacklist as well as an extensions blacklist. It defaults to blocking no keywords, but you can specify keywords to block if you so desire. This is keywords in freenet URI's, not in the content of the pages, and thus is a very crude instrument; I have no intention to "improve" it, it's just a quick hack so I don't see my node pulling sites that are obviously Really Bad Stuff while testing. Note also that legally speaking by blocking anything you may make yourself liable for what you don't block; read the EFF's advice to peer to peer devs if interested.
One other issue with bittorrent is that many torrents don't have any seeds; if we can improve Freenet's data persistence enough we can outperform it for rare stuff IMHO. Of course realistically Freenet is going to be relatively slow for some time to come, even if we implement e.g. the proposed changes to load management to make it more bursty and adapt to local conditions better. On the upside, upstream bandwidth (at least in rich countries) is rising relatively rapidly - I finally went from 8/0.5 to 20/1.2 recently and it makes a big difference (note that Scarborough is a small tourist-oriented town and hub for nearby semi-rural and rural areas), and there is work going on nationwide to deploy FTTC with up to 40/10, which will likely eventually be upgraded to FTTH at 100/100 or more. How Freenet will work in such an environment is something I look forward to exploring; but it is important that it scales down to basic broadband for hostile environments.
With regards to the legal and strategic issues, the principle of peer to peer filesharing is sound, and not inherently illegal or immoral. Personally I find a wide range of content objectionable: Illegally sharing copyrighted material (apart from exceptional cases of corporate, cultist etc abuse) is objectionable because it undermines attempts to build a better alternative; and I have my reasons for objecting to pornography in general (but of course child porn in particular). However, filesharing in general is not used only for these things, any more than Freenet is. And it's not so uncommon nowadays that politically sensitive materials can be quite large - email archives, suppressed video, wikileaks archives, and so on, can be largish. Thus it is important that Freenet work well for such things. One of our Summer of Code projects will implement a filter for Ogg Theora so that videos can be downloaded safely through Freenet, for example, and then there's the work above about making big files work better.
Legally speaking, even the (previous) UK government admitted that peer to peer is a legitimate technology and area of innovation; in a white paper it explained that sacrificing the telecomms industry to save the publishing industry is pointless as they're about the same size. Of course, then it went and implemented a three strikes law, which will take effect in approximately one year. In France, Freenet is much larger than it was before Hadopi (another three strikes law), so it's quite possible that Freenet will gain significantly from such measures in the UK. Hopefully it will be quite some time before anyone tries to attack Freenet (or anonymous peer to peer in general) specifically, but if and when they do, darknet provides a reasonably robust solution, provided that 1) the density and distribution of would-be users is such that you can actually find people to peer with, and 2) the government doesn't implement politically or technically expensive options such as traffic flow analysis, blocking all peer to peer comms etc. Even in the latter case, e.g. in Cuba (no legal internet connectivity), Iran (willing to tolerate massive disruption to everyday users' lives, mostly dial-up rather than broadband), etc, sneakernet-based routed peer to peer (which may draw significant chunks of technology from Freenet) should have a future; IMHO Haggle is great, but that sort of public, opportunistic model is not going to work in many hostile environments.
Near future priorities (for me and Freenet development more broadly):
When will we release 0.8.0? I don't know. At the moment the financial situation isn't critical - FPI just sent me $6000 (international transfers are expensive so I get paid infrequently), and have about $3400 left after the 2.5 months (modulo me feeling well) it will take me to use that up. IMHO improving Freenet is probably more important than declaring a specific version as stable and getting some press for it. I do think that we should if possible solve several of the above issues before any stabilisation-and-publicity-release period. But it's up to Ian. IMHO the only things that are absolute blockers are Freetalk and fixing the wininstaller, but several of the other things I've mentioned above would bring such an improvement in Freenet's usability that IMHO they take precedence over last minute bug fixing.
Meanwhile, the UK, where I live, has a new government, a liberal-conservative coalition with an impressive list of agreed policies. It appears that much of the liberal agenda on fairness and climate will be implemented, and it preserves the best bits from the tory manifesto (two interesting tory environment policies which if done right could make a big difference: carbon floor price and bribes for local authorities allowing wind farms). Of course, the long overdue referendum on AV probably isn't winnable, and the big question will be where does the axe fall - Tory instincts are naturally to tax the majority and squeeze the poor, but we'll start to get some idea what the priorities of the new government are soon. Spending matters, no matter how good the other policies. For example, only the liberals committed to climate adaptation funding being additional to aid - a commitment that will surely fall by the wayside now, despite their controlling Energy and Climate Change, but that is a probable deal-breaker globally as well as being a key issue of global fairness - if you believe there is any chance of a deal in the first place. One interesting question is whether the blatent upper middle class subsidy of higher rate pensions relief will survive - it's quite a lot of money and it's not the rich who need to be encouraged to save. It's not mentioned one way or another in the coalition document, but posting all the bad news at once is probably a bad idea. Capital gains tax reform is mentioned but the wording may allow for a massive discount for shares, so it may not mean CEOs paid in options paying their fair share after all (relatively small amount of money involved here either way). Anyway, VAT increases etc would be very unpopular (whereas higher rate taxpayers are unlikely to defect to Labour), and real terms spending cuts will be very difficult. I don't see why the coalition can't last for a reasonable time, given that neither party can easily force an election at a moment of weakness for the other - but it will be difficult to keep the back-benchers in line. Whereas if it had been a weak tory majority government, this would have been just as bad, except all the back-benchers would have been tories, and most of them (according to various polls) climate-denying pro-hunting anti-poor nasties. The liberal-tory pact may well be the best realistic outcome we could have hoped for; I voted Labour in the hopes of such a result.
In other political news, there is an outside chance of the EU adopting the 30% cuts by 2020 target. This is essential - the 20% target combined with the recession and the "flexibility mechanisms" (offsetting, over-generous free industrial allocations subsidised by the harder-to-decarbonise electricity generators' customers, banking of unused permits) mean very little meaningful action will be taken if we don't move to the higher target, and the 30% target would in fact cost less now than the original forecasts for the 20% target. Leadership sometimes means acting even though others aren't, or in anticipation of others acting. It is high time we showed some - Europe are supposed to be the good guys relatively speaking, but that is rapidly being eroded by their own inaction. The events of the recent election show that sometimes, surprising things do happen; so again, we'll see.
Health-wise, I've just had two courses of antibiotics and a chest X-ray, and have problems with both my respiratory and digestive systems; whether the current issues are related to the problems at Copenhagen is unclear, but for any brothers reading this, praying for me to be well and able to work at full speed would be helpful. I haven't been working at full speed for much of this year - not that I haven't done any work, but between illness, worry about illness, and other distractions, I've not worked as much as I should.
Meanwhile my computer (named Servalan after a certain female sci-fi baddie) is periodically stalling for no apparent reason ... except that it's nearly 4GB into swap, partly because of testing indexing (but it was 2G into swap and still sticking occasionally even before that). 8GB ought to be enough for almost anything... apparently it isn't any more.
Have a break, and opennet
This video was taken down from YouTube due to a copyright complaint from Nestle. Recently I have eaten a few kitkats after they became fairtrade, but sadly no more, at least for now, as they are apparently buying palm oil from deforested areas. The video is available on Vimeo and now on Freenet.
In other news, there has been a lot of work on opennet recently. This divides into three categories:
The intention is to dramatically speed up bootstrapping, and so far it seems to be working, bootstrapping to 6 peers in 0:46 today. Faster reintegration will mean that data goes where it should and therefore should persist longer, it will mean happier users, and so on. On the other hand it might mean more casual users, in which case we will have to increase redundancy to maintain data persistence. We don't maintain very much state for connections, and connection setup isn't that heavy (3K or so), so the connection churn shouldn't be a problem, but I have emailed the senior theorists - but when I've asked them before connection churn hasn't been a big issue.
The next step is to investigate data retention further. Recently we discovered that blocks inserted 3 times have 90%+ retention after a week versus blocks inserted once have 70%+. This is unexpected as it is likely that the data has fallen out of the cache after a week, but not out of the stores, which should be the same for all 3 inserts. It may be that this bug (not caching for a few hops resulting in missing the places where the data should be stored) is part of the reason why. So having fixed the bug, we can now test with and without the fix, with single inserts and triple inserts (as it is configurable per-request), to see what the difference is. If there is still a big difference between single insert and triple insert more work will be needed to figure out why and whether it can be reproduced without actually inserting stuff 3 times.
Meanwhile it is just possible that the infamous Windows installer Service did not respond to signal" bug has been fixed with the latest installers, by a simple fix relating to detecting where Java is installed. If you can reproduce this bug then please contact us, we really need help with it - primarily people who are able to reproduce the bug at will and test fixes, contact me or Zero3. We may yet have to take the extreme solution of installing Freenet to run on login rather than on startup as a service, which would be a fair amount of work, would cost us a little time on login, and wouldn't work well for servers, but has a number of advantages.
A new version of WoT is available, and Freetalk will soon follow, although in the near future both will be reset i.e. the context name will change so all old identities, boards and threads will become inaccessible - but the upside of this is WoT and hopefully Freetalk will be added (as betas) to the official plugins list and easily loadable from the plugins page. There is a bug related to leaking USK handlers that affects Freetalk quite badly, which I will investigate soon.
Evan is still working on making large splitfiles more reliable. At the moment, we divide data into 32KB blocks, and then we group those blocks into segments of 128 (= 4MB of data). We encode each segment with a Vandermonde FEC code, creating 128 "check blocks", so that we can reconstruct the original data blocks (and all the check blocks too) with any 128 data or check blocks. The big problem is that big splitfiles may consist of a lot of such segments, and since there is no redundancy between them the odds are that one or more of these segments will fail. The less big problem is that Vandermonde codes, while being "perfect" in that they don't need any more blocks than the original count of data blocks, use a lot of CPU and memory and/or disk seeks, and that this gets a lot worse if we increase the segment size beyond 128 data + 128 check blocks.
The first solution Evan investigated was two level segments: On top of the ordinary "outer" segments we would add a layer of "inner" segments, which involve randomly chosen blocks from across the outer segments. This would dramatically improve retrievability for a given block success rate. This still uses Vandermonde codes, and has its own eventual limitations, however it would not be difficult to implement. The second solution is LDPC codes. These would, for larger files, give an even better success rate, although they need a few extra blocks so for small files it may be better to continue to use regular segments. Plus the CPU usage is very low, and blocks are decoded in small groups of no more than 5 (so no heavy memory usage or seeking), which can happen partially progressively, although most of the decoding will still have to happen near the end. Evan is still searching for a good algorithm for selecting the blocks so he can simulate this and compare it to the other schemes. If LDPC codes are the big improvement that it looks like they will be, they will very likely be implemented before 0.8, because either scheme could make a huge difference to the retrievability of big files.
Sashee has been trying to get the web-pushing branch merged for a while, and hopefully it will go in soon (disabled by a config option by default, as it is still quite buggy). This will make it easier to maintain and allow sashee to reintroduce the stuff he took out to make merging easier. There has been some interest in this recently from benaroia on IRC, hopefully he will help us to further improve it. The web-pushing branch essentially uses AJAX to update various elements of the web interface in-place, rather than having to refresh or reload pages: The progress bar when loading a page, the notices at the top of the various generated pages, the content of various generated pages (e.g. stats, connections, etc), and most impressive (when it works), loading lots of inline images on a freesite.
Oh and we got slashdotted on Saturday. A PC Pro article very similar to the Guardian article last year. This seems to have got us a few more users.
Status update (February 2010)
Time for a status update...
Our last stable build, 1239, was in November. We have just released a new one, 1240. This has many changes (opennet stuff, optimisations, all sorts of stuff), which I list in the mail about it. One of the most important is that there are several new seednodes, and many dead ones have been removed. I have tested it 3 times today and it's bootstrapped fast each time, although yesterday it bootstrapped very slowly one time.
NETWORK STATUS AND NETWORK STATISTICS
Evan Daniel has been doing some useful work analysing the network. Amongst other things, he has discovered that:
EMU IS DEAD, LONG LIVE OSPREY
We have finally gotten rid of emu! Our faithful and powerful dedicated server supplied at a discount by Bytemark is no more. We now have a virtual machine called Osprey, which does most of the same job, for a much lower cost, and has a much simplified setup so should be easier to maintain. We have tried to outsource services, for example we use Google Code for our downloads, but some things will have to stay under our direct control for some time to come e.g. mailing lists and the bug tracker.
You may have some difficulty with the update scripts, if you use update.sh / update.cmd. If it doesn't work, try updating the script manually from https://checksums.freenetproject.org/latest/update.cmd (or update.sh)
WOT, FREETALK, RELATED THINGS AND OTHER PLUGINS
Xor (also known as p0s) continues to work on the Web of Trust and Freetalk plugins. These are approaching the point where we can make them loadable from the plugins page, and then bundle them, enabled by default.
WoT is the backend system which implements a pseudonymous web of trust, which functions in a similar way to that in FMS. You can create identities, assign trust to other identities, announce your identity via CAPTCHAs and so on. This is the Community menu, from which you can see your identities and other people's, and the trust relationships between them. WoT is used by Freetalk, FlogHelper, and probably soon by distributed searching, real time chat and other things.
Freetalk is a spam-resistant chat system based on WoT. This is similar to FMS, but it will eventually be bundled with Freenet, and will be a part of it by default. You will be able to embed a Freetalk board on your freesite. FlogHelper is a WoT-based plugin for writing a flog (freenet blog), which is very easy to use, but uses WoT to manage identities. I would have bundled FlogHelper months ago, but WoT isn't ready yet and FlogHelper needs it.
WoT should be ready soon. Recently a major issue has been discovered with the trust calculation algorithm, after that is fixed and some minor issues, WoT will become a semi-official plugin, which will sadly require flushing the existing "testing" web of trust, so sadly all old messages and identities will go away. Freetalk needs more work, about 50% of the bugs marked for 0.1 on the roadmap are fixed at the moment.
In build 1240, we pull in a new version of Library. This is a great improvement over the old version, it is faster, it supports embedding a search on a freesite, and has many bugs fixed. However searching for common terms can still cause out of memory crashes.
There is another issue with Library: infinity0 spent last summer creating a scalable index format for Library, which should make it a lot easier to insert and maintain big indexes. We will soon change the spider to use this new format, and in the process we expect to greatly improve performance for writing indexes, so it doesn't take a week any more and is done incrementally. I realise this has been promised before, but it is important, so it will happen sooner or later, hopefully sooner.
Full Web of Trust-based distributed searching, with a focus on filesharing, is on the distant horizon at the moment. infinity0 might be able to do some work on it as part of his studies, we'll see. It won't be in 0.8.0.
PRIORITIES AND RELEASES
We would like to get 0.8 out soon, or at least a beta of 0.8. Several major issues:
Please test Freenet, and report any bugs and usability issues you find on the bug tracker or via Freetalk board en.freenet (note that this will be wiped soon so if after a new Freetalk release it is wiped you may need to resend).
We have many ideas on how to improve opennet bootstrapping (make nodes assimilate into the network more quickly), and to improve opennet generally. Some of these are implemented in 1240, including many bugfixes. More will be put out over time so we can see their impact. Improving opennet should improve performance for the majority of users who don't run 24x7 and it should improve performance for everyone else too, as those nodes will get connected and start doing useful work more quickly.
We have many ideas on how to improve data persistence. There is a lot of capacity on the network, yet data seems to become inaccessible quite quickly (stats below). I am convinced that improving data persistence will improve Freenet's usability and perceived performance immensely. The continued popularity of insert on demand on uservoice demonstrates this as much as anything: People want a system that works! IMHO we can greatly improve things without resorting to insert on demand, although filesharing clients based on distributed searching may eventually offer it (but there are serious security issues with insert on demand).
Evan is convinced that mostly poor data persistence is not due to data falling out of stores, but due to the small number of nodes that stored the data (as opposed to caching it) going offline or becoming unreachable. We have increased the number of nodes that store data, we have made the node use the store for caching if there is free space, we have done various things aimed at improving data persistence, and there is much more we can do. An immediate question is whether the security improvements gained last year by not caching at high HTL have broken many inserts by making them not get cached on the right nodes; we will test this in 1241. A related question is why inserting the same key 3 times gives such a huge performance gain relative to inserting it once; we will investigate this soon after. We will probably triple-insert the top blocks of splitfiles soonish, but the bigger prize is to achieve the 90%+ success after a week that we see with triple-insertion of a single block, and this may well be possible with some changes to how inserts work...
Finally, the redundancy in the client layer could be a lot smarter: We divide files up into groups of 128 blocks, called segments, and then add another 128 "check blocks" for redundancy. Unfortunately this means that sometimes the last segment only has 1 block and 1 check block, and so is much less reliable than the rest of the splitfile. We will fix this.
We have been collecting statistics on data retrievability over time. The below are "worst case" in that they relate to single CHK blocks, with no retries. Real life, with many retries (at least 2 for a direct fetch and more if the file is queued), and with large, redundant splitfiles, should be substantially better than these numbers. Every day we insert 32 blocks and fetch a bunch of 32 blocks from 1 day ago, 3 days ago, 7 days ago, etc. There are two of these running to get more data, so I am just showing both results here. The percentages are the proportion of the original insert that is still retrievable:
Now, here's an interesting one. In each case we insert a 64KB CHK splitfile - that is, one block at the top and four underneath it. We insert one three times, and we insert three different ones once each. We then pull them after a week. We can therefore compare success rates for a single block inserted once, a single block inserted 3 times, and a simulated MHK, that is, a block which has been re-encoded into 3 blocks so that we fetch all of them and if any of them succeeds we can regenerate the others.
Total attempts where insert succeeded and fetch executed: 63
Single keys succeeded: 61
MHKs succeeded: 58
Single key individual fetches: 189
Single key individual fetches succeeded: 141
Success rate for individual keys (from MHK inserts): 0.746031746031746
Success rate for the single key triple inserted: 0.9682539682539683
Success rate for the MHK (success = any of the 3 different keys worked): 0.9206349206349206
USER INTERFACE AND USABILITY
Ian's friend pupok is working on a new AJAXy user interface mockup for Freenet. sashee's web-pushing branch, which makes the user interface a lot more dynamic without making it look much difference, should be merged soon, but turned off by default, since it has some nasty bugs. When it is turned on, it solves the age-old parallel connections bug, showing individual progress for each image without hogging your browser's limited number of connections (6 or 8 on modern browsers). Both of these may miss 0.8.
More broadly on usability, usability testing is always welcome: Persuade a friend to install Freenet, watch them do it, don't help them unless they get really stuck, report any problems they have or any comments they make about how it could be better.
Copenhagen and Christmas
Merry Christmas! I have taken a good deal of time off lately because of being in London and Copenhagen, and because of getting a nasty stomach bug while there. I am taking some more time off for Christmas; more on Copenhagen below, the rest of this paragraph is work stuff. It will thus be a while before there is a new build out, but when there is hopefully it will include the WoT plugin at least as an easily loadable option on the plugins page, and maybe Freetalk as well. It will also include the first stage of some important opennet improvements. Evanbd has been doing some impressive work which for the first time means we are not completely in the dark: We know the network has around 15k users over a week, excluding all those who try it once and then leave; we know that the latter group is relatively small, as is the group of 24x7 users, and we know that due to the Guardian article and all the echoes of it across the world, the network size has more than doubled. Online at any given time are around 5-7k users. We seem to be losing some users now, but it is not yet time to panic, or to decide that we gained nothing from the Guardian article; it's possible some of the decline is a Christmas effect (holidays, work computers switched off etc). Anyway, have a look at evanbd's flog or his graphs.
Now I will post what I wrote up while waiting for the train home, about my misadventures in Copenhagen, with lots of related rants. It's rather long for a blog post but if you want to read it then read it. I will post pictures later. Since I got back I have read most everything that has been written about the outcome of COP15; basically, it wasn't fair ($100B is a lot, possibly not enough, but will it be predictable, will it all be offsets, will it be redistributed from aid budgets?), wasn't ambitious (targets are way behind where they should be), and wasn't binding (well duh). Others will go into much more detail on this.
First, New Life Copenhagen, who found people to stay with for 5000 activists in Copenhagen, including me (see below), wants some links. Highlights (I didn't take part in any of these events):
Friday December 4th:
Saturday December 5th:
Eurostar to Brussels (2 hours). They claim 7.6kg of CO2, against 100kg or so flying the same route (IIRC). There are various disputes over this; the main inputs are concrete (tunnels, stations etc, but see here), and electricity (motive power, lighting and heating stations etc; I guess heating might be gas; Eurostar may not be affected so much by heating but Cologne station has a lot of heating). Eurostar scores particularly well as most of the route is in France or Belgium and therefore nuclear powered, and of course the trains are electric (half the emissions of diesel or better). Sleep in a private room in a hostel in Brussels ("Sleep Well").
Sunday December 6th:
Go up the Dome (cathedral). Unfortunately they don't let you store baggage at the bottom, so this takes quite a while with my enormous rucksack! The view from the top is worth it though.
When I get back down, I discover a tiny climate demo/rally. Get talking to people and go with them to some sort of spanish center which has been partially co-opted for climate folk. Meet some freegans, who are in my humble opinion completely nuts. Freegan means you never pay for food: you get food from shops where it would have gone to waste, legally or otherwise; a lot of packaged, safe, slightly out of date food gets dumped. Apparently they are strictly vegan, and almost entirely freegan: they buy beer and cigarettes and very occasionally food. Apparently they squat in a forest and have one job between six of them. They are part of the Climate Caravan, coaches/vans travelling across Europe to Copenhagen. Right now coaches are actually cleaner than trains in most cases, although once our power grid and cement production are made "green", trains might beat them.
Not being a big fan of long road journeys, perhaps hypocritically, I get the train. This is a big sleeper which takes 12 hours to get to Copenhagen, overnight. Other parts of it split off and go to places further afield. Lots of electric trains in the station, we are lagging behind on this, although the sleeper itself appears to be diesel.
Monday December 7th:
When I arrive at the metro station nearest my host, I discover a bunch of wicker people and tree stumps. The poster says this is a place for praying for an end to deforestation - kneel on the tree stumps or between the wicker people, or come back at 6PM every day for a larger group. Personally I haven't prayed anywhere near enough over the period of COP15...
Met my host, Soren, and his flatmate, Nils. I'm sleeping on their couch. New Life Copenhagen have made an art project out of finding people to stay with for 5000 activists.
Copenhagen is an interesting city. Flocks of bicycles, with their own traffic lights, everywhere. Wind turbines (and a presumably CHP coal plant) clearly visible from the center, along with the big metropolitan buildings. And as with all capital cities, a good public transport system. District heating (based on waste heat from biomass electricity generation) and soon district cooling as well.
But one of the most curious parts is Christiania: An autonomous suburb featuring its own quasi-anarchist self-government, open drug dealing (hence no pictures), many dubious buildings, and a 24 hour bakery. Buy some meat pasty thing from the bakery, and then discover the Climate Bottom Forum, an alternative event in a bunch of tents with a giant woman built of trash (PHOTO), focusing on transition towns, eco-villages, spiritual dimensions, indigenous knowledge, the end of capitalism and all sorts of other less than mainstream aspects to the crisis. Start throwing up before I leave.
Tuesday December 8th:
Wednesday December 9th:
Thursday December 10th:
Friday December 11th:
False solutions include carbon trading, which has some very serious problems and at best is an unproven attempt to minimise the costs to western companies, offsetting, nuclear energy (the Green Grid proposals make it unnecessary, since the wind always blows somewhere in europe), and geoengineering (a distraction, they say, from radical cuts in carbon right now). A core part of the radical argument is that endless economic growth is unsustainable: The planet has finite resources, and buying more and more things, with their built-in obsolescence, is unsustainable, unless we have perfect recycling; and historically, major reductions in carbon output in parallel with economic growth have happened very rarely if at all. In any case I decide not to sign the declaration, mostly because I believe we will probably need geoengineering and nuclear, two major technologies regarded as "false solutions" by klimaforum. In theory we shouldn't need nuclear, as it is very rare that the wind doesn't blow anywhere in europe - link up the electricity grids, build a vast amount of wind, throw in some saharan solar and the norwegian hydro and a tiny amount of biomass and you have a completely renewable and reasonably cheap Green Grid. However this requires that everyone builds wind at the same rate, and there are serious worries about biomass as well as not having the ships we need for deploying offshore wind (of course onshore wind is much cheaper, but we need both), so IMHO we will probably need some nuclear - although it's not a long-term solution as we will eventually run out of uranium.
Geoengineering is deliberately influencing the climate by either sucking carbon out of the atmosphere (e.g. via artificial trees), or by reducing the amount of sunlight that reaches the earth (e.g. via dumping chemicals in the upper atmosphere, similar to a "nuclear winter"). It's a last resort technology, but the latest science and the gross lack of progress in international negotiations mean we will probably need it. Unfortunately it will be very expensive and very risky, according to a recent Royal Society report. But we need to research it now as we will probably need it to keep a habitable planet, with scientists talking about the real possibility of 6 degrees by 2100 and 4 degrees by 2060 if we don't take sufficient action now (which we appear not to be doing, sadly). Engineering an artificial nuclear winter is not an acceptable alternative to cutting emissions now, but we are probably going to need both. And of course, as it's expensive, it will be deployed at the last minute, and not in time to save the poor countries; and it will affect different countries differently, with possibly severe side-effects...
Attend a few talks at Klimaforum. One on why interest bearing banking is bad, contributes to short-termism and hence to climate chaos (e.g. deforestation), makes the poor poorer and the rich richer, and so on. The interest free banking folks apparently have a functional bank in Sweden which allows its members to borrow and save, charges a minimal fee, and seems fairly awesome. Next is a session on the Ghanaian perspective which seems to be cancelled. The interest free folks say we can pop in to the after-party but the address leads me to an S&M shop next to a gay bar with no immediate evidence of klimaforum people, so I go home to sleep!
Saturday December 12th:
Didn't get any photos on the demo itself, but you can find plenty on the net, the following is of the place where the big demo started from but was on another day:
Sunday December 13th:
I misread the Klimaforum schedule and so think there is nothing on; there wasn't much on Saturday, but there's tons today, which I miss. I wander around the center of town, visit the Hopenhagen center in the middle of town, which according to the radical folk is a bastion of corporate greenwash (it's sponsored by Coca-Cola!). Certainly there are substantial corporate presences here, most notably Siemens, who have surprisingly frank set of plastic cubes representing different aspects of your carbon footprint, lots of video on all the amazing climate friendly things they do, and a magazine with a lot more. Watched the video on the High Voltage Direct Current line between Australia and Tasmania, which enables both countries to avoid building more power stations. The same technology will enable europe to supply all its own energy with virtually no non-renewable output. Now, as a big engineering company, Siemens stands to make a lot of money out of decarbonising the economy (which is a good thing), but on the other hand it is involved in fossil fuel energy as well as evidenced by their stuff about slightly better fossil fuel turbines. Ultimately it comes down to what they are saying at the Bella Center. Businesses are lobbying really heavily for their various agendas, often with huge delegations and side-events compared to the NGOs (groups such as Greenpeace or Oxfam campaigning for strong and fair action). I hope that Siemens is in favour of strong action, certainly from the material they showed it makes a lot of sense, but I don't personally know one way or the other.
Another exhibit is for a low-carbon ship based on natural gas fuel cells. Another is for the local state-owned energy provider, which provides not only electricity but also piped heat (and from the same biomass power plants), making a massive saving in carbon (although there are serious worries about being able to grow enough sustainable biomass without displacing food production and thus causing deforestation; it probably wouldn't work in the UK, although in Denmark wood grown on marginal land is supposedly a big part of the 40% of their electricity they get from biomass). They are getting into district cooling from March, which also looks pretty awesome, with a 70% reduction in carbon relative to conventional air conditioning (as well as less noise and equipment, and cheaper). I continue to wander, around the museum and other places. There is a big display of low-carbon cars and vans at Hopenhagen. Some of them just burn biomass (with the problems I mentioned above) but most are practical electric cars, with good performance but usually limited range. We really need better batteries; air-breathing lithium ion batteries will have the same energy density (=range) as petrol, but the current prototypes ignite on contact with water vapour, so it's likely to be 10 years before that's sorted - and eventually we will run out of lithium. In the meantime, current electric smart-cars are very cheap per mile, although they have limited range and performance; the new electrics (from 2011) will have much better range and very good performance, ranging from plug-in hybrids to improved smartcars to expensive but very good stuff such as the Tesla Model S, which is fast and spacious and options for range from 100 to 300 miles. Infrastructure will be a problem, but hopefully the power companies will pay for most of the charging points, and eventually you'll be able to battery swap at service station. Also have a brief look in one of the museums, and watch a few films by indigenous people affected by the changing weather.
All in all, a day of rest. That's okay then, although I missed some very interesting talks.
Monday December 14th:
Go back home to collect my stuff and say bye to Nils. Get lost on the way back, taking the wrong bus, but get the sleeper train on time.
Tuesday December 15th:
Wednesday December 16th:
All in all a bit of a battle, and I spent a lot less time following the negotiations, and listening and chatting about the solutions at Klimaforum, than I had expected. Ironically one of the reasons I went straight from The Wave to Copenhagen was to avoid bringing swine flu from The Wave back to family some of whom have health issues; it wasn't swine flu but I'm glad it's over! But the most important thing was the demonstration, and that went really well. 900 arrests out of 100,000 people - and most of those innocent bystanders. People from all over europe, and beyond, telling the politicians that the time for bickering over short-term national self-interest is over, that climate change is the biggest threat facing mankind, and that if the politics does not change then the politicians will have to change!
Not that I trust David Cameron on the climate (who turns down most wind applications? tory councils, of course!), although today's announcement that the tories will finally implement green mortgages is very welcome; in theory all three parties are in favour of this, but Labour has dragged its feet for far too long. Green mortgages are a tool whereby you get insulation and then pay for it out of your reduced energy bills over a period. Most people would benefit from insulation, solar water heating etc, but most people don't have the capital, and don't want to borrow large amounts to cut future bills, with the loan following them if they move but the reductions in bills not doing so. Green mortgages change this, making it easy and risk-free. Hopefully in the long term this will apply to solar water heating, solar electricity panels, etc, as well as insulation, but insulation alone will make a huge difference to our consumption of gas, oil and electricity for heating - and most people don't qualify for the various free insulation schemes available. I'm not just mentioning this because it's in the news; at Copenhagen there was a lot of talk of Eco-villages and Transition Towns. Eco-villages are green and increasingly self-sufficient rural communities. An important part of this is electricity. 70% of wind turbine applications in the UK are turned down, mostly because it spoils the view and thus reduces house prices (although a recent american study casts doubt on the reducing house prices bit). One way to get support for a wind turbine is for the people who live locally to own it. You get together, start a small company, raise a small amount of funds, apply for planning permission and borrow a largish amount from the bank to buy a proper turbine (not the little ones, the big 2MW ones). For the next 25+ years you have green energy, some of which you use and some of which you sell, plus the subsidy for wind. Some of this money goes to repaying the loan, but a lot of it can be spent on improving the local environment - which of course the village gets together to decide on. Eco-villages also often grow their own biofuel for heating (e.g. wood on sloping land), and generally try to be self-sufficient as well as green. There were some impressive case studies shown - towns and villages where almost all their electricity and heating is produced locally. Transition Towns are the urban equivalent, and often don't do much in energy, as they usually don't have good sites for wind; Scarborough's two transition groups work on food and transport, contact Pete Redwood for more information and to get involved, although they are both undergoing changes at the moment. I have no idea whether we could have a wind turbine in Scarborough, but it might be worth thinking about. With the new feed-in tariff subsidies for solar (which are actually surprisingly generous, especially if you apply before April), it may even be worth thinking about solar, on individual, commercial or council buildings (solar is dependant on sunlight, and doesn't like to be too warm, so it's not out of the question here in the North; also you often don't need planning permission now) - but as always the question is the capital outlay; hopefully eventually small scale solar will be covered by green mortgages, making it easy and essentially risk free.
Meanwhile, protesters have got into the Bella Center. I know no more about it than you do, but personally one reason I left Copenhagen on Monday was to avoid the big direct actions. I have a lot of sympathy for their position, but on the other hand sabotaging the conference on the day when the ministers arrive is probably not productive. Unless you believe that nothing good will come out of the conference, and want to make a point for the global media. We'll see; there won't be a legally binding agreement, there probably won't be an agreement on adaptation money (the vast sums needed by the third world to develop cleanly and adapt to the impacts of climate change that already kills 300,000 of them annually, e.g. by building sea defences), and the forest protection stuff is based on offsetting (thus locking in polluting western infrastructure) and seems to be compromised in some big ways. And even if we get an agreement in the new year, the rich country targets are way below what the science requires - 25-40% by 2020 for a 50% chance of avoiding 2 degrees or more, but would you get on a plane with a 50% chance of crashing? Anything over 2 degrees is likely to be enough for most of the positive feedback systems to kick in, leading to 3, then a terrible 4, and (without large scale geoengineering) up to an apocalyptic 6 degrees. More recent science, and a prudent desire for a larger safety margin, suggests at least 40%, with global emissions peaking as soon as possible - if emissions peak in 2020, as currently expected, we will need to cut carbon emissions by 5% globally after that every year. This sounds easy, but it is in fact very hard, especially when you consider that major fast-developing nations won't peak until the 2030s (after all, China has hundreds of millions of rural and urban poor as well as its middle class almost the size of the USA) - so it might be as much as 9% a year for the developed nations, and also require substantial geo-engineering. This will be very hard, and it's never happened before. It will dominate our economy for decades if we are serious about it, and given the risks the sooner the better I say! UPDATE: Here's an account of what happened on the Wednesday from the protesters' point of view.
Campaign against Climate Change are calling for a climate emergency response. They have 45 MPs signed up to their EDM 2057 (Climate Change 2), which calls for a declaration of a climate emergency, 1 million green jobs (e.g. in insulation, improving public transport), a 55mph speed limit, banning domestic flights, a strong public information campaign (nobody complained about war coverage on TV in WW2, and that's exactly the situation we're in, an existential threat). They were behind a rally I went to just before The Wave, which made essentially these demands. Treating the problem as the serious issue that it is will create substantial numbers of jobs, increase bills in the short term, stop most of us from flying, show global leadership instead of the current ridiculous I-might-if-you-will games, and greatly reduce the amount of trouble that we are storing up for ourselves.
What's trouble? Crop failures, droughts, erratic weather, hurricanes, floods, poor countries losing huge amounts of land (e.g. bangladesh stands to lose 18% of its land mass, many small island states stand to be submerged completely), serious problems with water supply, many horrible things, which will likely have knock-on effects such as wars over the remaining fertile land or water resources, and huge numbers of refugees who will as always be demonised as "economic migrants". Most of these things are already happening, in India or Africa, or even sometimes the rich world, hence the figure that climate change kills 300,000 people a year already. Since the sea is generally affected a lot less than the land, 4 degrees globally means 14 degrees in the arctic, 6 in Spain and much of Australia, 5-7 in most of Africa, 6 or 7 in much of the US, and accelerating loss of ice and permafrost (two notable feedback mechanisms). Exactly how bad it will be is not clear yet but it will be bad. The interactive map on the Met Office's website talks about 20-30% loss of agricultural yield in mid to low latitudes - meaning that poor countries will be hit hard. But this is based solely on temperature, not taking into account the increasingly regular droughts and floods. And some say the temperature effects on yields could be vastly worse - a recent study talking about 60-80% loss of US agricultural productivity in a 4 degree scenario. Hopefully that study is wrong; many are! Even if it isn't, solutions may be found (e.g. with genetic engineering; the current efforts to that end are largely corporates cashing in on offsets with crops that require less tilling but more chemicals, but future efforts may be more successful, this is another reason I didn't sign the Klimaforum declaration) - but erratic weather may limit what we can do cheaply.
However, if we let it reach 6 degrees, we are looking at a situation not seen for 100 million years; last time it happened most life was concentrated around the poles, a truly apocalyptic scenario. As a placard said at the demonstration, "The planet's fine. We're f*cked.". Although there are a few scientists, including the very notable James Hansen, who argue for a "Venus syndrome" i.e. terminal climate change, the literal end of the world as a biosphere (Venus has runaway global warming, scorching temperatures, high pressures and sulphuric acid clouds). Hopefully we won't let it get that far - at some point the people and the politicians will realise what is at stake and will do what it takes to prevent it, even if that means a lot of collateral damage from geoengineering (it probably will), and vast expense next to which current complaints are irrelevant. But if we reduce emissions now, we can avoid most such problems, prevent most of the positive feedbacks ("tipping points") from kicking in, as well as moving to a more sustainable society.
So now we wait. In the new year there will be more battles. Scarborough 10:10 is taking a well-earned rest after showing Age of Stupid 6 times to 150 people, but will be active in the new year. Contact me if you're a local and interested in helping. If you are interested in Sustainable Scarborough / Scarborough as a Transition Town, contact Pete Redwood. Note to Freenetters: Scarborough 10:10 is the local branch of the 10:10 campaign, a small group of people running showings of Age of Stupid, stalls etc to try to get people to sign the 10:10 pledge to cut their carbon emissions 10% in 2010. The objective is to use this to influence government policy for more ambitious action - and there has been some small success in this, see the website. I've taken over co-ordinating the local group since Jane stepped down, but we are resting until January 9th, and we will likely be making a much less intense effort for most of 2010; this was another demand on my time pre-Copenhagen.
More holiday photos are here (on Freenet). I have stripped out anything containing anyone's visible face since I can't remove them from Freenet once uploaded. Sorry I didn't get much in Copenhagen, and nothing on the main demo, but there are plenty of places you can find such pics on the net.
1238 is now available. Normally I wouldn't blog on every build, but this one is worth talking about. The most important change is that this build has fixed a whole bunch of bugs in the client layer (persistent downloads and uploads), including stalling uploads and downloads, and it has done it in such a way that old, broken uploads should fail rather than just hang forever, old downloads should complete, and new uploads should complete. Some of this may have been a relatively recent bug according to the people I've been working with (notably p0s, although evanbd has helpfully nagged me about the problem), however there are definitely some longer term issues here too, so if you have stalling downloads please try 1238!
Another notable change in 1238 is the new CSS filter, kurmi's SoC project, has been merged. This is much more detailed and comprehensive than the old filter, and hopefully much more secure as a result: It parses everything and lets through only what it can understand, just like the HTML filter. This is the only way to be sure that a filter is secure. The old CSS filter only tokenised and didn't parse, so might (at least in principle) have let dangerous content through, and in order to prevent this it was more strict with strings etc than the new filter is. Also, the new filter supports > selectors, although not CSS3 selectors (which the old filter did, because it didn't parse them). It also has an extensive set of unit tests. Kurmi's mentor was nextgens and I finished it off, resolved parsing issues, added unit tests, and merged it. So it is a significant improvement, something that we would have had to implement sooner or later, and it might just be a security improvement.
Another change is that the first-time wizard is now considerably more concise, while hopefully being clearer and having less FUD. If you can do usability testing, please do - that is, find somebody who has never installed Freenet, ask them to install it, and note down wherever they get stuck, what questions they ask etc. Don't help them, at least not until after they have got stuck. If they get stuck, that is a usability bug. Please send the results to the devl list, or to me via Freetalk on the en.freenet board, so that we can solve these bugs.
Also, we have finally merged the new wininstaller, thanks to Zero3 and Juiceman, which does not create a custom user for Freenet, so should run into far fewer anti-virus/system policy/Vista compatibility issues, which were affecting very many users. Also, it includes a system tray icon from which you can conveniently start and stop Freenet if you need to for e.g. online gaming. So a bunch of important improvements, coming from various people. Worth a quick blog post IMHO.
Next up, the new Mac installer, which will also include a system tray icon, thanks to mrsteveman1. p0s is continuing to work on Freetalk, currently working on a login system (you might have to wait a day or two for me to catch up with and deploy Freetalk as the current version is incompatible with 1238); hopefully it will be ready to be official before Christmas.
Recent performance testing data shows that as long as you fetch immediately, Freenet can perform well, around 59 seconds to fetch a 1MB test key from a freshly bootstrapped node. Anecdotally very popular or very new keys can achieve very high speeds, maybe 30-80KB/sec. However, if you wait a few days, many of the keys will have fallen out, so it may fail and will probably be slower. Now, the long-term push/pull data (waiting e.g. 3 days between inserting and fetching) strongly suggests that some proportion of inserts are "getting lost" on the initial insert, pushed to the wrong part of the network due to backoff or some other reason. I am going to implement some test code to verify this theory; if it turns out to be correct, MHKs (duplicate the top block) and some tweaks to splitfile redundancy should be a fairly easy workaround, and improve persistence considerably.
We are also planning some more work on network level diagnostics. Currently, we have probe requests. Probe requests are a kind of request that does not return data, is routed to a location rather than a key, and returns information about the nodes along the way - their locations, UIDs (which are only used for probe requests) and whether they are backed off. This is conceivably somewhat useful to an attacker, although swap requests (which are a vital part of routing) reveal most of this information without probe requests. Probe requests help us to understand what is going on on the network, whether routing is working, whether churn is a problem, and how big the network is. But we are planning to get rid of probe requests in favour of more specific, more useful mechanisms:
Obviously these are temporary measures to help us to understand what is going on on the network and thus improve performance. One of the best ways to improve Freenet's security and survivability is to get a bigger network IMHO, and we will achieve this by improving performance. Probe requests suggest Freenet is around 2500 nodes at any given time, up slightly from a few months ago, while performance has improved.
Bloom filter sharing is still planned for 0.8, but may be postponed, it is more important to deal with the other network issues first as they may be easier to fix now. Most of the hostile environment / security stuff we have planned for 0.8 has already been implemented, there is more work to do on usability and integrated functionality, probably more work to be done on persistent downloads (e.g. backing up the database, less disk access for downloads), fixing XMLSpider, fixing the uninstall survey, etc. Sashee's web-pushing branch will be merged when it is ready, this might be after 0.8.
Build 1237 and related things
1237 is now available, please upgrade. This fixes a bug in the client layer which was causing downloads not to complete (probably not the only one, sadly), fixes a minor exploit in the content filter, makes Library official and auto-loads it over Freenet, has more work on evanbd's hourly by-HTL stats logging, and some changes to the FCP feeds API.
Library is the new search plugin built by infinity0 and mikeb. It is integrated into the fproxy user interface, supports phrase searching, boolean operators, basic page ranking, and is a big improvement. But the real story is that Library supports the new Interdex search file format, which is an on-Freenet b-tree, which should be much more scalable than the current index format. In the medium term hopefully it will also support distributed WoT-based searching, which could form the basis for a really interesting filesharing system. The reason it's called Library is that it is not just the user interface: XMLSpider will be adapted to talk to Library to do its index writing. Hopefully in the not too distant future, the spider will use Library, and will write indexes much more quickly by bunching writes together in memory and then linearly rewriting all the indexes. We can do this with the old format, which would make it much easier to run a spider (as currently it can take 4 days to write a large index if you don't have a pair of Cheetah's or an Intel SSD), but it is even more interesting with the new, scalable format.
In the short term, evanbd's work on stats could turn out to be very important. Recent tests show that if you insert data on a well established node and fetch it immediately from another well established node, you can see transfer rates in the region of 40-80KB/sec, with one report of rather more. However if you wait 3 days, there is a disturbingly high failure rate - although that seems to have improved a bit since the last bunch of routing fixes. evanbd is working consistently on stats and simulations, and this has been very helpful, with the recent routing fixes coming out of it. Soon I will implement "tracer requests"; these will insert data using a special key type, record an identifier for the nodes it is inserted to, and then when we fetch them we will see where the request went to likewise. We will use a new node identifier that doesn't correlate with anything, and IMHO this is no more dangerous than probe requests (which we will be turning off), especially as it doesn't use "real" keys (we will create a very small datastore for such test keys). We think routing works pretty well, a popular theory for why data retention is so bad is that we don't handle opennet churn very well, but there are other possibilities, hopefully tracer requests will help us to understand what is going on. Given that Freenet can be fast, with some fixes and probably Bloom filter sharing, hopefully we can have a substantial performance improvement.
Other stuff since last post, which you probably already know about: the routing changes I mentioned above: for a while now we have taken into account not only our peers' locations but also our peers' peers' locations, but we were not handling this properly. The seednodes file is now auto-updated, and the Add a Friend page now serves the latest Windows and Java (OS/X and Linux) installers. We now have a Dutch translation, updates to French, German, Italian and Chinese, and more of the web interface can be translated. 1234 included some important work on memory usage, 1235 included some optimisations to the client layer (persistent downloads and uploads), trying to reduce database accesses and therefore disk i/o, cpu usage and memory. 1236 included some important work on ULPRs. Ultra-Lightweight Passive Requests are an optimisation introduced some time ago which is designed to make polling apps such as chat clients more efficient and faster, by remembering for an hour which nodes have requested a specific key, or we have requested the key from, and telling them if we find it. In 1236, we propagate ULPRs more effectively, more securely, and more accurately, as well as fixing a bug that may have been preventing it from working since July. Other recent work includes plugins, the datastore, the client layer, bandwidth limiting, node shutdown, automated tests and simulations, and more.
However, I have taken some time off work lately - partly because of minor ailments, but mostly because of getting involved in climate change events (all over until December for me now). Yesterday was the Blog Action Day, an attempt to have a co-ordinated campaign of blog entries about climate change; obviously I missed it! Unless strong action is taken to prevent severe climate change (defined as more than 2 degrees warming), we may find ourselves without "a planet similar to that on which civilisation developed", in James Hansen's words. Of course, something will be done - the question is whether enough will be done (Hansen and the small island states are arguing for a much tougher position than developed and rapidly developing nations are willing to commit to right now). The Met Office, for example, is saying that 4 degrees by 2060 is possible. Have a look at the New Scientist coverage if four degrees doesn't scare you. It won't be evenly distributed: some areas would have no warming, some would have 10 degrees or more. And as always, the poor are hit hardest, because they live in places which are already seeing severe changes in the weather, because they live off the land, and because they often have no resources to adapt with. Christian Aid estimates there may be a billion climate refugees by 2050 (others estimate 150-250 million), although some argue that those who are worst affected are probably also least able to migrate... The global poor are paying for our pollution, and the situation is going to get a lot worse even if we do take strong action. If we don't, vast areas of the world will likely become unfarmable due to drought, floods (often directly following drought), desertification, more natural disasters and so on. All of these are happening now, and have gotten significantly worse over the last 50 years, with only a 0.8 degree rise in temperatures since the industrial revolution. Maybe we'll be able to geo-engineer our way out of the most severe consequences, but it will be much cheaper and less painful to deal with the problem now. What must happen is drastic cuts in emissions - 40% from 1990 levels by 2020 in the rich countries, and some meaningful progress from the fast developing nations - and large amounts of cash to help the developing countries adapt and to grow in a low-carbon way (at least $150B). Neither of these demands is likely to be met fully in Copenhagen, with the US position constrained by domestic politics, India unwilling to take on any binding targets, and the Chinese so far only have vague mumblings about intensity targets, and demanding the right to keep on increasing emissions until 2050, by which time they will occupy the entire carbon budget for the world and then some (recent research shows they could peak by 2030 without any serious economic/poverty impact if they wanted to).
Sorting out this mess will be painful, energy prices will have to rise considerably and flying will probably in the long term become the preserve of the rich once again (but we'll have better rail), but in the near future a great many jobs could be created. Unfortunately many countries - particularly Britain - have squandered obvious opportunities for green recovery, preferring to spend vast sums on propping up retailers and car companies. Nonetheless there is a short-term economic case for action. The long-term case is rather more stark: the Stern review, years ago, said the economic cost of taking action would be around 1% of global GDP, and the cost of not taking action would be around 5%, but he also gave a worse-case estimate of costing 2% to take action and 20% to not take action. He has recently said that in the light of recent science the latter is probably more accurate. 20% of GDP is equivalent to two world wars and the Great Depression, and we need to start taking it that seriously!
So if you have not already contacted your representatives (national MPs and if appropriate MEPs), and told them that you care about the future of the planet, that Copenhagen must deliver, and that your nation must take drastic action both to lead in negotiations (40% by 2020, as Norway recently pledged, and their share of $150B+ of adaptation funding) and to build green infrastructure instead of brown infrastructure (Kingsnorth and Heathrow 3rd runway cancelled in the last month, yay!), please do so. There are many resources on the web and involved organisations e.g. 350.org, Stop Climate Chaos, Campaign against Climate Change, Operation Noah, Christian Aid. Read their resources and then email your representatives (UK) in your own words, because this is far more effective than clicking a standard form letter. After that, think about attending a relevant national demonstration - in the UK, The Wave is planned in London on the 5th of December. The talks themselves are on the 7th to the 18th of December in Copenhagen. The EU negotiating position will be finalised on 29-30th of October, so if you are in Europe you should write to your MEPs before then and maybe consider demonstrating during the session (although I'm not sure anything's organised yet, Climat et Justice Sociale Belgium may have something). The 24th of October is a global day of action for 350.org, there is probably an event near you, unless you are in the UK. In the UK, a colossal scheduling cock-up means there is a major anti-war protest (Iraq killed around a million people, climate change already kills that many every 3-4 years), the Anarchist Bookfair, and other local events going on... good luck, I'll probably attend the York Wave rather than try to organise anything in Scarborough. Recent events I've been involved in include Operation Noah's Climate Change Day of Prayer, for which I printed out far too many pages of printed resources (I suspect I'll be able to unload some of them though), and Tearfund's climate justice evenings (an equipping meeting that was interesting and encouraging), for which I had to sleep over in Leeds (one small disadvantage of living in Scarborough is transport is mediocre). I'll be in London for The Wave, and in Copenhagen for some of the period of the negotiations (if you are coming talk to New Life about accomodation, you probably won't find a hotel or a hostel). How much difference demonstrations outside international negotiating events make, when nobody can tell who are their constituents, is an open question, but it's good to have a presence, and the alternative summit should be fun. You can get an idea what is going on from the Adopt a Negotiator blog (on Freenet via Freerunner, RSS). Oh, and tomorrow Climate Camp and others will be shutting down Radcliffe on Soar, a major UK coal power station which produces more CO2 than a large group of least developed countries put together...
Build 1233 and related things
Much work since last time! The XML bug, scaling max peers with bandwidth, plugin loading/update over Freenet, the minimalist theme and the new status bar, some minor filter changes with more to come, some important datastore fixes, more internal documentation (javadocs) and lots of bug fixes!
First off, the XML bug. This has been extensively covered elsewhere so I'll give you the basics and the current status: a remote code execution bug in a widely used XML parsing library is present in Sun Java up to 1.5.0_20 and 1.6.0_15 (amongst other things). Freenet will therefore refuse to load official plugins which parse XML on older versions of Java. 32-bit Windows users should be largely unaffected as the auto-updater will pick it up. 64-bit may have a problem as the auto-update only works for the 64-bit JVM and we need a 32-bit one (we are considering solutions given the increasing prevalence of Vista/64); download the latest 32-bit JVM and you'll be fine. Linux users may have to obtain the packages directly from repositories (e.g. the pool directory for debian), if your distro hasn't pushed the updated versions yet. OS/X 1.6 has the new JVMs, 1.5 update 5 will have it when it is out; currently Mac users have a big problem, but this will go away soon, so it is unclear that it is worth importing a separate library for it just for Mac users for a short period...
Scaling max peers with bandwidth: Freenet will now use between 10 and 35 peers depending on your bandwidth usage. This should improve speed on both slow and fast nodes (lower overheads for slow nodes, more routes for fast nodes), and it was the number one uservoice request for quite some time. Please let us know whether Freenet has improved performance-wise since 1231! Again this has been covered elsewhere. The number one item on uservoice is now "one GUI for all", which I interpret as a plea for more functionality integrated into the web interface...
Plugin updating over Freenet: We now load plugins over Freenet by default! Previously we were loading official plugins from emu directly, which is very hazardous and may not be possible at all in anywhere vaguely hostile, not to mention giving away your IP if you are on darknet. We didn't do this automatically - if a plugin was out of date, we'd ask the user whether to fetch it from emu - but it meant anyone unable to access checksums.freenetproject.org couldn't keep their plugins up to date - and the old versions frequently would stop working because of changes in the core. In 1233 for example there are core changes affecting 6 plugins, all of which will now be automatically pulled over Freenet on starting up; these fetches should work, but if they don't, Freenet will allow to either give up, let the node continue to try to fetch the plugins, or fetch them from emu. This is important now, especially for hostile regimes, but it is essential with Freetalk and Artefact2's upcoming blogging plugin!
Recently I've been doing a good deal of code reviewing and other miscellaneous stuff. This is great, because it means we have more volunteer work going on! Artefact2 has recently started contributing to the web interface (new theme, status bar), statistics (persistent uptime and bytes transferred stats), plugin infrastructure (localisation and data storage), and is working on a blogging plugin, which is something we should have had years ago. evanbd has suggested we implement a breakdown of request success rates by HTL, which has produced some remarkable results (have a look!), and promise to reduce the amount of alchemy that has crept in, largely because of the considerable difficulty of any objective measurements, since Ian declared the end of the age of alchemy all those years ago. He has analysed the data and helped with theory, and is working on a nano-blogging plugin and some changes to splitfiles. kurmi's XHTML and BMP filter changes have been merged, and his CSS, Thaw and ATOM filters will be merged soon. infinity0, Artefact2 and others have been working on a new freenet-ext.jar (please test and report back!) with non-segfaulting 64-bit FEC support, FreeBSD support, etc, please test it! Hopefully infinity0 and mikeb will be able to fix any remaining issues with Library so it can be the default new awesome search plugin (even if it may not have full distributed search yet), and more importantly, update XMLSpider to produce indexes (in the new format, supported by Library alongside the old one) much more quickly. Zero3's (and Juiceman's) new windows installer (please test and report back, no update.cmd at present, and not fully compatible with the old one) is nearly ready, but it has been nearly ready for some time so we'll see; it features a system tray icon, and doesn't create a custom user, so should have less of a problem with antivirus, system password policies etc. mrsteveman1 has had some trouble getting a systray icon to work on OS/X, but hopefully there will be one at least for some versions. On Linux, we will have to use a Java version, but I have been provided with some example code. saces has been working on various internal things and plugins, hopefully his multi-container freesite insert code will be made default before 0.8, it would be a huge improvement for big freesites. ljb's persistent node to node file transfers haven't hit fred-staging yet but we passed him (and the others) so I am confident they will. Our translators continue to be helpful, but as always we need more languages. There have been hints about Russian and Farsi but so far not much has actually happened. A possible Farsi translator said something about RTL support, we need to look into this but it might be easier with some help from the would-be translator; if you want to translate Freenet, please let us know. sashee's web-pushing branch is also approaching being merge-ready, and hopefully will be merged within the next month; loading the Activelink Index with pushing is quite impressive. Work on Freetalk (mostly by p0s aka xor) is slowly progressing, it should work in the current build but expect incompatible updates, bugs and so on. However I do read the Freetalk boards, so it may be the easiest way to contact me anonymously (Freemail is very temperamental for me). I've probably missed out a bunch of people and I'm sorry, I'm just trying to give a flavour of the current situation: There is a lot going on, most of it by new or re-engaged volunteers!
In the reasonably near future, changes to splitfiles (reinsert the top block when it takes more than a little while to fetch, split data evenly among a splitfile, make smallest segments a bit bigger) should improve data persistence somewhat, and if necessary MHKs will improve it further, without involving major changes. There are a few security and hostile environment matters that need to be dealt with, or that we would very much like to deal with: use more rounds in AES, encrypt plugins' on-disk data (and completed download list files), keep a copy of the latest installer on disk or at least link to it on the Add a Friend page, possibly RSKs (revocable SSKs, so if the private key is leaked you can still tell people this), some means to prioritise a modem-based darknet peer so it has some chance of its requests being answered, and possibly a Friend-to-Friend web proxy system to get around national firewalls with no frills attached (this is actually a lot easier than it sounds!). Freetalk is essential for 0.8, and hopefully p0s can get there with minimal help; an RSS filter would be nice to go with the ATOM filter, there are bugs in USKs but also there is the potential for major improvements, db4o auto-backups are essential, as is better usage of the datastore, and we also expect a few more usability/UI improvements (and I mentioned a few things mentioned already). Most of this is relatively easy. However, I still plan to implement Bloom filter sharing. All the indications are this would be a significant performance and data retrievability gain, and the initial implementation shouldn't take more than a couple of weeks, now that we have the prerequisites (caching changes) sorted.
Many other features have been postponed until 0.9, and many of the ones mentioned here may also be. Fortunately we have a reasonable amount of breathing space, as we have spent around a third of Google's $18K so far. So 0.8 should be really great! There have been a number of positive comments lately from users trying Freenet again after some years, and reviewing Content of Evil was quite educational, showing how far we have come. While on the one hand we will run out of funding sometime early next year, 0.8 could be a breakthrough in many ways, and I believe we will find additional funding - whether it be through paypal micro-payments from many concerned individuals (and I have been entirely paid off these at various points in the past, although I had lower costs then), from groups interested in hostile environment deployments (in some cases Freenet may be the best solution), or from some other source.
Build 1226 is now out. A lot has happened in this build. Firstly, we have finally fixed The Register's attack. Basically the problem is freesites you browse, files you download etc were cached in your node's datastore, speeding up future accesses but making it possible either to look at your store and find out what you've been browsing, or to remotely probe your node's datastore (assuming the attacker is connected), and see how long it takes to fetch blocks to try to work out whether you have fetched particular files. What we do in 1226 is only cache in the datastore when the HTL is over some level, so your request will not be cached unless it is at least 2 hops away from the originator (often more).
Why has it taken so long? Well, we had assumed it would be fixed when we implemented premix routing or encrypted tunnels. These would also make a lot of other attacks a lot harder. Recently it has become clear that these will likely be a lot of work and may have a significant performance cost. One reason for these changes (which may reduce performance slightly) is that they are essential prerequisites for Bloom filter sharing, which should greatly improve performance, and will be implemented in the coming months (see more below). But another reason is that in many hostile environments, seizing a node and checking what people have been downloading is a very real threat, and up to 1226 Freenet left far too much evidence...
1226 allows you to configure the tradeoff between convenience, performance and security. Physical security levels now range from LOW to MAXIMUM. At LOW, nothing is encrypted by Freenet (but you might have whole drive encryption). At NORMAL, your node.db4o (downloads/uploads database) and your client cache (a new kind of datastore used to avoid fetching the same freesites repeatedly) are encrypted, with the keys stored in a file called master.keys. Securely delete that file and both are useless, and so are all the other temporary files as their keys are kept in node.db4o. The panic button has been fixed and reinstated, and will do this for you (but watch out for data-journaling filesystems and flash devices). On HIGH, you can set a password for master.keys, and you cannot access the download/upload queue until you enter this. On MAXIMUM, the keys are random on every startup, so there are no persistent requests, but you can still fetch big files as long as they complete before restarting.
Freenet migrates your node.db4o automatically on startup if it needs to be encrypted, it supports defragmenting node.db4o on restart (mine shrunk by a factor of 10), which it will do when you upgrade, there are security and performance improvements to ULPRs (ultra-lightweight persistent requests, used to speed up polling e.g. Frost, FMS), ljb's mostly internal work on user-events (did you know your node has an RSS feed on /feed/ ?), and bugfixes and optimisations. So that's 1226, please go get it! There may be a small performance hit in the short term...
The big performance feature for 0.8 is Bloom filter sharing. I have had a lot of help with working this out from evanbd, but originally it was Oskar's idea. Basically, we tell our peers what keys we have in our datastore (using a highly compressed data structure called a Bloom filter). This should cost around 300 bytes/second on a node with 30KB/sec upstream, but it enables us to short-cut and route directly to the node with the data when we happen to be nearby. For really popular data this means we can probably fetch it from one of our direct peers, and bypass load management. For less popular data it means we check 20 times more nodes' datastores, so once we are in the rough area where the data should be, we should find it efficiently. Hence it should improve performance, in terms of transfer rates, latency and how much data can actually be found, by a considerable amount, but we don't know by how much yet. Unfortunately we can't just deploy this for security reasons - hence the work above to ensure that the only stuff in our datastore is data which other peers at least 2 hops away have requested.
A big problem with Freenet is that it is relatively difficult to communicate with other users. This is partly because of the lack of a good integrated chat system, which p0s is working on with Freetalk. But it is also a fair amount of work, especially for less technical users, to set up jSite, build a site in HTML and upload it. It appears a basic blog engine can be written in 800 lines of php including templates ... so it might make sense to spend a few days on this as a relatively high priority, to make it easier for new users to contribute content. Freetalk would handle both comments and announcing a site. Granted, Thingamablog does something similar, but I don't think this has been maintained lately, and it is a separate application which you have to download; it should be a web interface, just like Blogger. This is not just a question of what will get us the most users (and hence funding) in the short term: If Freenet were to be used in a hostile environment (e.g. China), people using it would probably not be very technically literate, would be used to using web services to create content, and so on. Security matters, but usability matters too.
Chat and other matters
Freenet 0.7 is coming along nicely, although the tentative feature freeze is by no means absolute. Ian wants to release in March; a release candidate towards the end of March seems feasible at this point. ULPRs and related code (the cooldown queue) are pretty much finished, and there is a feature freeze, although some HTL changes have been necessary and one more may be in the next few days. Bugfixing is in full swing; the spider and the transport layer have been recent targets. We still have way too many timeouts, but many nodes in one VM simulations (freenet.node.simulator) have helped to find the cause of some...
However, one big piece isn't ready yet: a working, user friendly, hard to spam, bundleable chat client. The spammer seems to have taken a day off today, but until today he has relentlessly spammed all the default boards and many more popular boards, making them unusable. IMHO Freenet is a community, and for any community to function there must be a usable means for chat. Right now FMS is apparently working and under heavy development, but in C, and using an NNTP interface; seagull is apparently working on porting it to java, but we haven't seen his work yet. Ian is of the view that Frost and FMS are separate projects to the node and we shouldn't wait for them. What do you think? Answers to an appropriate forum, should we:
Note that because FMS is in C and therefore not bundleable, I don't currently run FMS. I may run the java port of FMS when it becomes available - hurry up seagull! I've seen test posts, so it can't be too far away...
In other news, we should seriously consider whether to take part in Google Summer of Code this year. I was approached on Wednesday by an enthusiastic and apparently competent would-be SoC student, who now has an SVN account and is working on a bug in the transport layer. Last year SoC was a lot of effort and a partial success - a lot of the code was of poor quality, but some of the devs have reappeared or not disappeared. If we do do SoC this year we will probably take fewer students, and we'll certainly want to be more careful in selecting the student (and not the proposal; proposals can be renegotiated).
A standard criticism of the Christian gospel is that the various resurrection accounts (bibles on freenet here or on the web here) are inconsistent. And at first glance they disagree on many important matters. I just found this gem (from the web), which makes a persuasive case that they can be pieced together into a coherent whole. Why does this matter? Well, if Jesus did rise from the dead, he probably is the Son of God. If he didn't, then Christianity (with all of its widely acknowledged moral contribution to the world) is nonsense. You can of course claim that the whole thing was made up, so it doesn't matter that they are consistent; but they were clearly written by different authors, and reasonably early although probably not direct eyewitness accounts; and Matthew's gospel was written to jews so claiming that the guards' tale was widely known among the jews probably would have been a bad idea if it wasn't true. Have a look!
I can't write a flog entry without a Freenet-related update. And I should probably explain what's going on locally too. Firstly, emu (the freenet project web/mail/etc server) has been down for a few hours, it will be back up soon, due to its being moved from London to Manchester to cut costs for bytemark; we're getting a free memory upgrade in the process. Secondly, we (family) are moving to Scarborough (wikipedia) on Thursday and may not be around for a while after (hopefully we will have a phone line when we arrive, so we should have dial-up, but there will obviously be lots of stuff going on e.g. unpacking). Reasons for the move include health (Bristol's air is really unhealthy), and closeness to relatives, and a much bigger house. Working from home has certain advantages!
Now for developments in Freenet itself: Build 1069 features a major security fix for our connection setup code: It was possible for an attacker to do a man-in-the-middle attack using weak diffie-hellman keys. Tor had a similar vulnerability in 2005, and Freenet 0.5 still has this problem (we are not going to fix it, we will apply a patch if you send in one, but Freenet 0.5 is unmaintained and unsupported). 1069 is mandatory on Tuesday, so upgrade! Hopefully the auto-updater will upgrade your node automatically, or is even now awaiting your clicking the "Update Now" button.
In other news, slow progress continues to be made towards full opennet. Path folding is working, UP&P (automatic port forwarding and IP detection if your router supports it) is working, reconnecting after reasonable downtime is working (I dropped my darknet peers, shut down my node for 12 hours, and it got back on its feet with opennet soon after starting back up; admittedly I'm not NATed...). 1070 will have a fairly major security fix relating to path folding (traffic analysis related to packet size ... sorry), and the major items remaining before we can implement automatic bootstrapping are automatic detection of port forwards (to get rid of the annoying connectivity messages and automatically detect if you are eligible to be a seednode), and anonymous connect crypto (the current link setup assumes both sides already know the other, which is great for darknet, and even for opennet, but not for bootstrapping opennet). So hopefully well before Christmas we will have full opennet support - full meaning you don't need to exchange noderefs with strangers, ever, even when you first connect to Freenet.
That's no excuse for running Freenet in its most insecure mode though! If you know somebody who has a freenet node, it is a really good idea to connect to them rather than to a total stranger chosen by the opennet logic. For several reasons: Firstly, your friend is likely to be more trustworthy than a random stranger, and Freenet is not perfect: those you are connected to can try to analyse your requests (see also the wiki). Secondly, it's not a random stranger: an attacker can choose to be connected to you relatively easily. Thirdly, your node is not invisible, an opennet node can be detected relatively easily, and then blocked on e.g. the Golden Shield (many western nations have some form of ISP-level blocking even now). And it is likely that one day Freenet will be illegal, even where you live, whether by the IPRED2 (pending, EU), the DADVSI (passed, France), or some other legislation targeting copyright infringers, terrorists and paedophiles (but most likely the first). Oh and you americans, you're not safe either: they've already tried for mandatory trusted computing once. Opennet is a transitional phase, a means to an end: Your node will automatically reduce the number of opennet peers (Strangers) it uses as you gain new darknet peers (Friends) (thanks to somebody on Frost for the idea). In the long run there will be no opennet. But in the short run it should allow much faster growth and a much easier install for most people trying Freenet.
Apart from that, there has been some other progress: JFK has been merged (new connection crypto setup, strongly resistant to denial of service attacks), there have been many bugfixes including some fairly juicy ones (continually polling USKs, node stopping sending requests due to stuck inserts, etc), better IP detection (we run the plugins in parallel as they can be slow), translation updates, major bugfixes and improvements to jSite (it is now able to insert large freesites without silently dropping files after the 1000th), XMLLibrarian, the freesite search engine, is working and has a default index (type XMLLibrarian# or XMLLibrarian* into the plugins box to load it via HTTP; the latter will reload it on every startup), you can now tell a darknet peer to drop you when you remove it, datastore performance fixes, faster HTML filtering, cruft removal, and much more, all since the last update.
But on the whole, progress is relatively slow. Why? Partly that life keeps getting in the way for me (don't worry about your donations, I charge FPI by the hour!), and largely because we have relatively few volunteer devs at the moment, although we just gained an anonymous freemail dev from Frost (Freemail is now a plugin, and appears to work at least sometimes: install it by loading Freemail* from the plugins page [ will reload from HTTP on every startup ], and send me a freemail!). Freenet is really a much bigger undertaking than the current team can handle in a reasonable time. If you want to help build a critical piece of a free online future, please contact us on IRC, the devl lists, or Frost. If you're interested in becoming our second full-time paid dev for Freenet, that might just be possible (no promises, sorry...), I advise you to start coding and contact us once you've established a reputation for being a capable coder (after say a month); that's what I did.
And the Google Summer of Code 2007 is well and truly over, I've sent a final round-up to the development list; we learned a lot from this year, and that was cemented by and added to at the conference Google hosted for us (which originally I went to because I didn't want nextgens to be the sole voice representing us! It was great though, much better than last year). Many thanks to Google for SoC, even though many of our students were disappointing; we got some useful code out of it, and at least two of the devs will hopefully stay on. Next year, if we are accepted, we will do things a little different!
Expect the next update when you see it!
Google Summer of Code is finally over. Yay! Several of our students are likely to stay on as devs, and some useful code has come of it, although most of the projects were not fully achieved. Rough summary:
Apologies for the limited progress with Freenet recently: I never seem to finish anything, life has been rather mad lately, it'll be sorted out soon (after we move)... Current priority items:
Finally, I wasn't sure whether to post the following: some folk might get the wrong idea, and it does break my quasi-anonymous persona somewhat (not that I'm actually anonymous, of course..). But I promised to host it, so you'd have found it eventually, and more importantly, it might do some good: ritual humiliation (completely unrelated to Freenet (on the web).
Upgrade to 1037! Really! The long-awaited Update over Mandatory support will make life much easier for the developers when debugging the network, because nobody will be left behind by a mandatory build ever again (well at least until the next incompatible crypto/packet level change), and the new probe request code will allow us to test a new backtracking/HTL algorithm (which should help significantly on darknets and hybrid darknet/opennets) without breaking the network. Lots of other goodies too (less short packets, possibly less timeout, f2f file transfer...). And it'll be mandatory on tuesday. So go get it!
One reason for the short mandatory is that work on opennet will start in the very near future... We need opennet, because of topology issues (#freenet-refs sucks), and because we need more users, more developers, more money, more content. And we need it to grow in order to have enough users to be able to build a global darknet during the limited time remaining to us. So watch this space!
One last thing. On the resources page, you will find a full set of debian AMD64 ISO's (all 21 CD's). Let me know if you manage to fetch them! (Or even one of them...)
The original plan was to capture a load of data about the network topology and use it to determine what is wrong with the network and whether opennet is The Ultimate Solution™ (or just a necessary evil). In the process we discovered a routing bug, which was probably causing the topology data to be corrupt, as well as various more serious issues, and is fixed in 1033. It will be mandatory on June 5th, so that gives a window to fix some more bugs and implement some long-planned features in the meantime (not necessarily all of these todo items):
PS: Move your freesites! Build 1035, out Real Soon Now, will by default disable access to pre-1010 insecure SSKs (you can still enable access manually). In a month or two we will drop the back compatibility code which allows these (and old CHKs too) to be stored. What this means is you need to move your freesite now if it is still using an insecure SSK/USK key (one which ends in BAAE rather than CAAE, and for CHKs AAEC* rather than AAIC*). Or move your favourite unmaintained site (suggest coordination via Frost). I have moved the Greater Toad Pictures Stash.
Looking up for once... mostly!
For once things seem to be looking up. The network is, compared to recent history, zooming along, inserting a CD image in around 2 days, routing is highly specialised even on low bandwidth nodes, link crypto is secure (thanks to STS), we have 6 SoC students and I only have to mentor two of them, and FPI is in reasonable health due to Ian's newfound talent for finding rich donors. The next build will feature a full localisation infrastructure (this is 99% done, I just need to put a few more l10n keys in), so hopefully in the near future Freenet's web interface will work in your native language (contact us if you are a native speaker of a non-english language but also speak good english and want to translate for us, or validate other people's translations). As regards opennet, we are currently collecting data on the network topology, which should tell us whether #freenet-refs is having a major negative impact; if it is, we will commence work on opennet in the near future.
Now for the bad news. IPRED2 passed the European Parliament. This piece of legislation makes inciting, aiding and abetting, attempting or doing any intellectual property infringement intentionally (not defined) on a commercial scale (loosely defined) a criminal offence subject to full criminal sanctions (prison time, judicial winding up, denial of legal aid etc). In its original form this applied to all "intellectual property", which would probably have criminalised the entire EU software industry; fortunately the passed version excludes patents. Unfortunately, it would probably still make Freenet illegal under incitement or aiding and abetting. It is likely to be some time before this completes the EU process, and once it does it can be up to 18 months for it to be enacted in each state's laws. So we are on borrowed time. If you want to help Freenet out, now's the time!
In unrelated news, check out Ripple, it's a friend-to-friend darknet-style currency system (there is an old version which is fairly centralised but there is slow progress towards a true f2f system, I've had interesting discussions about routing with Ryan). The basic principle is that money is a system of IOUs: it's information plus trust. So make that system explicit, and instead of trusting the bank all the time, keep it as close to your social network as possible. It would be quite interesting for LETS style hour currencies, but even for hard cash it would be useful. Have a look.
Backups are important. Not only that, but backups of the right files are important! I lost a hard disk... it turns out that my (weekly) backups didn't include my most recent GPG key (but I have the old ones), nor my Frost identity. They do now! On the upside, Servalan (aka amphibian.dyndns.org) now runs Debian Etch on amd64 (the workstation still runs 32-bit, to minimize disruption and because of openoffice and gaming complications). Now, how do you know I am still me?:
My new public key is E43DA450 in my keyring (4096-bit, yay!). The crash message is signed here with the new key. My new Frost ID is toad@zceUWxlSaHLmvEMnbr4RHnVfehA and my new freemail address is firstname.lastname@example.org (long version), although this isn't necessarily fully working/integrated yet; don't send any important correspondance over it yet, but if you want to send me a test message I'd appreciate it.
Geek computers are like the TARDIS. They are immensely powerful, but they're constantly breaking, because we're constantly tinkering with them! (Well in this case it was a case of abusing commodity hardware; why should a hard disk fail? I know even Seagate drives have a limited lifespan, but they shouldn't! :)).
On the other hand, last week's holiday (Monday to Thursday) was pretty cool. Me and mum went to see granny in Harrogate. Spent some time with granny, visited some nice green places, got lots of cheap videos from the charity shop, watched all four Harry Potter films, discovered that a cousin of mine is a Red Dwarf fan. Normal service (normal chaos?) will be resumed shortly; most of the TODO list on the last but one item still stands :(.
PS, here is the text of the Pope's original speech, if anyone cares; it's primarily about religion and rationality, but does contain a subtle attack on Islam even if we accept his apology with respect to the most blunt quote.
PPS, the below item (Hereticnet) has a Frost board called Hereticnet.
There are a number of persecuted groups who would greatly benefit from freenet's technology, but who cannot use it for moral or political reasons. For example, persecuted churches. Even if you are an atheist I hope you accept that freedom of thought, and therefore of religion, is important: You have the right to sincerely believe in the Jesus, Buddha or the Flying Spaghetti Monster, if you want to. Others have the right to ignore you and think you are crazy. So there may be room for a darknet variant which uses a lot of freenet's code, but has different goals. Note that I am saying nothing against Freenet itself: I like Freenet, I am morally happy with it, but I think there may be room for something else as well, once Freenet has reached a reasonable level of stability.
Such a network would be resistant to external censorship, but provide for internal censorship. In other words, it would be a high standards darknet: A community with its own standards for content, which it could enforce through expulsions and schisms, but which is not necessarily the same as the outside world's standard. On such a network, content inserts would be tagged with a cryptographic structure allowing the insert to be traced back one hop at a time, but only with the consent of (for example) 2/3rds of the nodes surrounding each hop. If somebody found some content they object to, they could file a complaint. This would be discussed on the chat system, and ultimately people on the network would inspect the disputed content (hence the need for a fairly 'high' standard), and decide whether to vote to trace the author, to trace the complainant, or to do nothing. If enough nodes vote to trace the author at each hop, he would be traced. He would then be identified to his direct peers, and everyone else would know his topological position. The network must then decide what to do with him. His direct peers may simply disconnect from him. Or they may choose to protect him, (either after the trace or during it), in which case they themselves may be disconnected from. Irreconcilable differences will have to be dealt with by a larger network split: What was one community is now two.
This is by no means an easy way out of the conundrum that is freedom of speech. It requires significant effort on the part of the users, and it also requires a fairly high standard; "anything but child porn", for example, is likely to result in permanent brain damage (or at least a need for counselling) to active participants on the network, since disputed content will normally be close to the border between what is allowed and what is not. A persecuted church would have a much higher standard, while most of its content would still be illegal by the local laws. And it is likely that such networks would have major problems with splits and schisms, as any other community does. It would closely represent the underlying community. It seems to me that this would be an interesting experiment, and it might be useful to somebody. Comments? Contact me!
Probably a good idea to update my freesite once in a blue moon, right? Well there have been lots of goings on in Freenet land... Minor stuff first. Build 950 somewhat experimentally makes the node treat backoff as advisory rather than mandatory, so your node will send requests to backed off nodes if it can't send them to nodes which aren't. This seems reasonable to me, (because we have a separate load limiting system), Anyway it's not a long term solution; the current mechanism has major security issues (flooding and possibly some local attacks too). Tell me what you think of performance on Frost. There have been many other improvements, a good deal of which is due to our volunteers and Google SoC students (students Google pays to hack on freenet over the summer); upgrade! As you've probably heard, John Gilmore (who remarkably enough owns toad.com) has donated $15,000 to Freenet, so I'm definitely going to be paid for the next six months. Obviously this is a good thing!
One of the big questions in 0.7 now is opennet. Here's the deal. On darknet, you only connect to your friends (in theory; in practice most links are established via #freenet-refs etc between strangers). On opennet, the network assigns you connections. Either way, you are vulnerable to your peers: they can do correlation attacks, and probably lots more fun things. On opennet, you are probably connected to the Bad Guys, because they probably harvest the network (impossible on darknet), pretend to be a thousand nodes (hard on darknet), and connect to everyone they can - preferably more than once under different identities. In other words, Opennet is vulnerable to harvesting (quickly finding large numbers of nodes) and Sybil attacks. For the time being there isn't much we can do about correlation attacks; I have a few ideas which may be implemented soon but they won't provide a very large anonymity set; expect major progress in 0.8, but it's quite likely that premix routing won't work on opennet. So as you can see, we need opennet for bootstrapping: for getting slashdotters onto the network. In the long term, we need as many true darknet connections as possible, because Freenet will be illegal or attacked. One thing which is important is not to jump in too soon; many problems are easier to solve on darknet, which has lower connection churn etc. But we do need opennet, and once we have it we need people to move from all opennet to some opennet and some true darknet connections, and eventually to pure darknet, or the network will be unsustainable. What incentives we can provide is the subject of some debate.
So when will Freenet be illegal? The DADVSI in France arguably makes it illegal there. We may have a constitutional case, or we may not, but from what I've heard we are in trouble. People who distribute or develop it may be liable to prison time and a €300,000 fine. What's more fun than the DADVSI then? The IPRED2 is a directive going through the European Parliament which makes it a crime, punishable by all manner of criminal sanctions including prison time, judicial winding up, large fines, and lots of other fun things, to intentionally commit, attempt, aid, abet, or incite (!) the violation of any intellectual property on a commercial scale. "On a commercial scale" is undefined, but caselaw suggests it's not a real protection for anything either Freenet or the free software movement does; "intentional" may not require knowledge of the specific IP according to the FFII. But even if it does, knowingly violating a software patent on the grounds that it's a load of nonsense would now become a criminal offence! The application to patents alone will seriously harm the free software movement, of which Freenet is a part, and fighting it on the patents issue is the best way to attack the directive (as we won the fight last year). But the application to copyrights might well ban Freenet EU-wide. Please don't get depressed and cynical about this. Join the FFII, read as much as you can, and write to your MEPs. Last year we fought the proponents of software patents (including IBM, which is either schizophrenic or trying to have its cake and eat it, or both) to a stalemate: All that is required for evil to triumph is that good men do nothing, but there is much that we can do (before we reach the "we must all hang together..." stage!). Here's a letter (HTML) I sent recently (addressed to Lord Sainsbury, although you should generally write to your MEPs on this); please don't send exactly the same letter, but you're more than welcome to use it in writing your own.
Finally, the government expects all UK broadband ISPs to be blocking child porn sites using Cleanfeed by the end of 2007. They have assured me that it will only be used for this purpose, but I still worry that ISPs will be forced by litigation to block specific copyrighted content (e.g. xenu.org) - certainly they will if the IPRED2 passes. We shall see...
TODO LIST: Current relatively high priority items:
Shadows of shadows
A freesite with a few links to stuff downloaded from a certain p2p program recently came to my attention. It contains at least one file which is pure copyright infringement, so I won't link it here. But it also contains this file, labelled on the original site "US ARMY 3 - top secret defense - Biological Weapons Technology.pdf". Sorry to burst your bubble, but it's on the web. Just google for "Those delivering BW could be". That gets you to www.fas.org/irp/threat/mctl98-2/p2sec03.pdf. Go up a few levels and you find all sorts of wonderful stuff about WMD and intelligence threats and so on. Another few clicks and you get to stinet.dtic.mil. An amazing amount of this stuff really is just out there, and there is absolutely no shame in pruning through it and finding the good stuff. Or even calling attention to it. I honestly don't know what the legal situation is with this sort of stuff, but frankly I don't care; it's almost certainly covered by copyright exceptions in my country. Have fun, and insert anything you find especially interesting!2006/05/03
Welcome to Freenet 0.7!
I have finally updated my Freesite, two years after the last post - and this time it's on the long-promised Freenet 0.7 alpha darknet! Yay! here is a page about Freenet 0.7's current security or lack thereof, from the wiki. Also, we are looking for student coders to work with us for money over the summer, thanks to Google's summer of code. You will however need to blow your anonymity by visiting the web site to find out about this.
The excellent (if small as yet) new index called FreeHoo carries in-depth reviews of Freesites. His second site listed was Naphtala, a site which I have never visited because I believed it to contain child porn. It doesn't. It contains a fair amount of interesting content written by a paedophile, none of which qualifies as pornographic or illegal as far as I can see. He claims at one point to use non-violent child porn so that he doesn't have sex with real girls, while consistently defending his practice of "child love". Definitely worth a read, as Freesites go, but I won't link it here directly for purely paranoid reasons. Obviously I do not endorse his views: any sexual relationship with a child is evil, illegal, a sin, and likely to damage them for life (I go much further than most people here by endorsing the biblical view of sex, which is that it is a good thing, within marriage). However it is interesting to hear from the other side, and I rather pity Naphtala.2004/05/28
Brain Dump - why do I do Freenet?
I was recently bothered by the whole thing of why I do Freenet, and whether it is compatible with my ethical basis. I include the resulting braindump in case anyone cares. You may find it interesting. I rather hope this won't result in me becoming even more of a cult figure, but perhaps it will spread some useful insight.
Why do I think it is right for me to do Freenet?
As an evangelical Christian, I'm not the stereotypical Freenet user. I don't surf the porn sites, I accept an ultimate (although not human) Authority, and so on. I don't define myself as a Libertarian or Anarchist. Many of my brothers and sisters in Christ would probably not approve of what I am doing. So, why do I think it is appropriate for me to do Freenet?
Firstly, Why is this difficult? Well, firstly, my income comes from working on Freenet. I don't have any real financial commitments to worry about, but it is better that I am able to contribute to the family running costs. And I enjoy giving to both Tear Fund (an explicitly Christian overseas charity operating in Africa, Columbia and other "nice" places; they help people to work their way out of poverty, provide emergency relief, and witness the gospel in both their words and actions) and my local church. Thus I have a vested interest here, which makes objective ethical decisions difficult.
So, now the meat. I do Freenet because I believe that freedom of speech is, despite not being a biblical concept, something worth fighting for in a democracy. Does this mean I idolize it? If the state has the right to taxes ("give to caesar what is caesar's...") and even to wield the sword (one of Paul's letters), then isn't it legitimate for it to say what people can and can't say? It is only a small step from here to "Isn't it legitimate for the state to say what people can and can't believe? Isn't it within the bounds of statehood for Rome to persecute the early Christians?". If we look at it from this perspective... Rome persecuted the early church, and failed. Badly. The church did not fight Rome (it was not a good idea historically, nor was it compatible with the sermon on the mount). But the Church prevailed, and those martyred (in horrible ways) will be rewarded in heaven for their perseverence.
So how does this relate to Freenet? Well, Freenet can be used for good or evil; it could be used by Chinese student groups to keep in contact with each other and the West (for example to organize prayer when they are particularly persecuted), by urban Saudi churches to keep in contact with current Western christian literature, by Western whistleblowers to publish things such as the Diebold files, or to publish criticism of the Church of Scientology, by paedophiles to exchange pictures of child abuse, by others to publish racist hate speech, and by yet others to publish lists of people accused (but never proved.. or who have served their time) of paedophilia and related offences, for future ostracisation and lynchings. Surely it is the state's right to prevent the last 3 cases?
Firstly, even if there was no Freenet, there would still be files that while technically illegal are clearly in the public interest, and should be hosted by as many people as possible. For example, the various sites on the Church of Scientology (read them, they're on Freenet). These are of clear benefit to the kingdom of heaven. The state's regulatory role is trumped by the clear good that is being done in keeping people away from this vile cult.
I have always thought that what Paul said about the state's right to wield arms, and so on, are guidelines - true in general, but occasionally it may be necessary to go up against the state; to disobey, in extremis to actively work against the state, perhaps using violence as a last resort. The resistance against the Nazis in WWII is one clear example of this. If you have Jews in a cellar, and the gestapo come and ask you about them, it is perfectly okay to lie to them. If a higher cause is being served it may sometimes be necessary to steal or even to kill, in such extreme circumstances. However on the whole, people lie to cover up sin, or to "fit in" (fear of man"), people steal to feed their sinful desires, and people kill for much the same reason.
So where does that leave us? We have to fend for ourselves, and work out exactly what the costs and benefits are. Even if the state has a legitimate right to suppress lists of people accused of crimes they were never prosecuted for (for obvious reasons), that does not mean that it will be successful. Very often such people are persecuted - often in error - because somebody in the police themselves tipped off the locals, or the paper. Once this information is available, it will spread, because it is believed to be in the public interest. Sometimes perhaps it is. If such data is published on Freenet, and the author is believed (which may be difficult - he needs to earn some trust - but this is certainly not impossible - and in this particular case it may not be hard :( ), then it is difficult for the state to get rid of it. To that extent, our building of Freenet has helped this to happen, even if we do not ourselves approve of such content's publication. Similarly in the case of paedophilia: we hide the people distributing the images, and although the police may well be able to investigate in other ways (e.g. matching the children involved), they are not able to simply trace the distributor, unless he does something stupid to give away his identity (trapping him on a message board may work). They are therefore available to a slightly larger audience at a somewhat reduced risk (although theoretically if they are not accessed they SHOULD fall out of Freenet).
So we are clearly building something that CAN be abused. As can a kitchen knife, or just about anything else, but more specifically, weaponry. We are building a weapon here, of a sort. Like all weapons it is not tactically neutral; it cannot be used to suppress information. It can be used to prevent such suppression. My intent is that it be used for the positive purposes listed above, and more. But some will inevitably use it for evil. How do I know that the good will outweigh the evil? Lets look at the good first:
Freenet can probably be shut down relatively easily in its current state by a moderately well funded attacker. For example, a half time Chinese technician (of some skill), with access to the firewall rules, could set up some nodes, harvest some references, and also the seednodes, and block every node found by IP address. However, compare it with what they have now (or did before recent developments): a mailing list on which open proxies are announced. Every 24 hours or less, a new open proxy is found and announced. The government is subscribed to that list, and within 24 hours, the new proxy has been blocked. Using Freenet (or I2P, which has similar attack issues) instead makes the cost to the Chinese government higher. If Freenet gets bigger, it becomes harder for them to block, but it remains reasonably feasible. However, should Freenet routing start to work, we may be able to provide new features that make it substantially more difficult: trusted mesh routing, which would severely limit harvesting in hostile environments, and thus limit the damage from busting one node; new (steganographic) transports, which slow down and make more difficult attacks, and so on. In fact, right now, you can identify all Freenet traffic by some identifier bytes. But we will get rid of this. And even if Freenet is never really THAT useful, there may well be future anonymous P2Ps that are. We will have laid the groundwork, even if they don't use any of our code, through our extensive practical research. Little of it is formally published, but I still think a lot of that knowledge would be out there.
In terms of impact: revolutions are always (in China anyway) led by the students. Even in China, the students have internet access. Bloodless revolutions have happened on many occasions in the last century. But the people need to know that another world is possible in order to demand change. I do think that such things *ARE* God-blessed. Velvet revolutions as we saw in the Soviet Union, in Serbia, in Georgia, are usually surrounded by a delicate chain of coincidences that is the signature of divine intervention; God *is* interested, and he *does* care. And the stakes are pretty high: there is a good chance of China and the USA having another cold war, quite apart from human rights issues, and the persecution of the Church in China.
In the West, files such as the Diebold files (leaked info on voting machines with major security issues), the Church of Scientology files (which the Co$ has consistently abused copyright law to pursue, even if they are readily available *NOW*), and so on, demonstrate that freedom of publication is vital to protecting democracy. And democracy has achieved great things for us, even if it is constantly under attack. At its best it can be a great means for change in a positive direction. At its worst, it devolves into plutocracy, as do all other governmental systems. The basic difference is the attitude of the electorate. If they are apathetic, evil triumphs; "All that is required for evil to triumph is that good men do nothing", as a wise american once said. So as always, true progress comes from the renewing of minds; but nonetheless democracy is valuable and worthwhile. And without freedom of speech, democracies devolve, as the public do not know what is going on, even if they DO care.
In the future, the West may become a very dark place. I cannot know for certain but I suspect we are in the twilight of democracy here. The War on Terror expands, and if it is not over quickly, it risks destroying democracy. If there is another serious attack, we will take another major step into the abyss. The intelligence war may be going well, however, there is plenty that the terrorists could do, and the war for hearts and minds is largely ignored (as we see with the reluctance of the US to bring real pressure to bear on Israel). If it continues (and it may, as there is considerable money to be made on both sides), we are going down the tubes - and this can't easily be fought with the prospect of terrorists crashing jets into nuclear reactors on the horizon. The other big influcence is in cyberspace, which increasingly dominates meatspace as a communications medium. It is quite possible that the little war between open source software and Microsoft et al will come to an end. If Microsoft loses, its stock price will collapse, and this will have substantial effects on the US economy (IIRC Microsoft is 20% of the S&P index, which a lot of pensions are linked to; not sure about the NASDAQ). If Microsoft wins, we will have increasingly unreasonable DRM regimes, and the War on Piracy will start to move. Because there are always bugs, automatic updates of trusted operating systems are inevitable. Because you cannot prevent people from aiming a camera at a TV screen, we may well be looking at a means of remotely deleting files from all trusted PCs everywhere that match a particular checksum. Once this is technically available, even if judicially supervised, it will be abused. Eventually it will be abused for overt political goals, but where there is a blur between copyright and freedom of speech, the former will prevail, as it usually has in the past, and increasingly will in the future as further legislation passes which makes it ever more obvious which side the legislators' bread is buttered on. Freenet will not be the only key piece in this fight; black-market hardware may well be an issue, and there will be many more; and I personally will not be involved, once OSS (or Freenet) is made illegal, because of my prominence beforehand; but this is another area where Freenet and similar technologies may well be of some practical benefit. And the Church will not be unharmed either. Prohibiting radical islam would be politically incorrect, so what is likely to happen would be prohibiting religious intolerance (there are already laws about this, we will have to wait and see how they are interpreted). Sermons and literature on why other religions are wrong, or why homosexuality is wrong, and so on, would become risky. If the WoT continues to escalate, or if there is a regional nuclear war (as the world continues to "tool up"), we may end up with explicit regulation of religion.
So, in conclusion: I do not know whether Freenet will be a net benefit. I do know that I am writing a tool that can be used for good or evil, that the good it can be used for is considerable, and the evil that it can be used for is there. God, however, knows. So it all comes down to whether God is happy about it. Which is expressed in my conscience; the indwelling Holy Spirit informs a believer's conscience. And I don't really have a problem here, most of the time, despite having a better relationship with God than I had some years ago. It is also expressed practically. I pray to God for success, and I have success, often from unexpected quarters. I have had huge last minute donations; I have been led to enormous bugs by coincidence and hunches after prayer. I have been blessed in many ways. I also honestly believe that the project we are undertaking is massively difficult, and a lot comes down to luck; we do not have the resources for it to be otherwise. I don't believe in luck; I believe in providence. I have met several Christians through Freenet, despite the natural expectation that everyone involved in such a project would be either very shallow or a committed Libertarian atheist. And so I continue. I believe, and trust, that if it is wrong for me to do Freenet, God will reveal this, in time. And even if he does not, or if he makes it abundantly clear and I ignore it for years, I will be forgiven at the end of time, because He has paid all my debts, and gone where I could not return from, and come back, and given me the promise of eternal life, secured by the only true authority in the Universe. Glory be to the Lamb forever and ever!2004/01/29
Hutton Craziness An interesting puzzle...
The Hutton Report (Freenet) was published today. I haven't read it (been far too busy with Freenet), but the general gist of it, according to the press, is that the government has been completely exonerated. A lot of other interesting stuff came out during the process of the inquiry, but the conclusion is interesting, given the fact that we still haven't found any WMD in Iraq, and Hutton says the dossier (Freenet) was _not_ 'sexed up', we have a few options:
The other interesting part is that no British government is going to go to launch a preemptive war on the basis of intelligence for the foreseeable future. This is probably the real payoff here. America loses its biggest ally in the fight against States We Don't Like. How sad. Sidenote: I actually think Iraq is better off for the war for humanitarian reasons. That's not really the issue here though.
And now for something completely different: Freenet Update Freenet is currently having some major problems. Both branches are running NGRouting, which could probably be improved significantly; m0davis is working on that, when he's around, which is rare. The unstable network probably has 100 nodes, and seems to work remarkably well for new content, with insert/retrieve tests succeeding immediately even at relatively low HTLs, most of the time. Stable however isn't. Stable is estimated to have on the order of 10,000 nodes, as of a few months ago: Iakin ran an ubernode for 4 days and had 16,000 unique IPs contact it. The maximum HTL has been reduced to 10 on the stable network to try to reduce load. At least on stable, nodes are in overload almost all the time, and it's almost all the time caused by bandwidth usage. However, this is expected; we now reject queries based on outbound bandwidth usage, with the result that when we come out of overload, if we get a few queries for files that are actually retrievable (and yes, this does happen), our bandwidth gets used up and we go back into overload. After a somewhat paranoid conversation with an old friend, I discovered a possible attack that could have caused the current symptoms, you can read the thread here, which was fixed in 5064, but not completely resolved, as you can read in the thread. This may be happening a little by accident, but I doubt that it's happening a lot by accident, because the log messages that would show it are relatively rare ("Got really late DataReply" is the obvious one). Anyway, 5064 fixes it, but it also makes probing the network for a given key a bit easier, so there will need to be further action on it. I am currently engaged in implementing a new load balancing system, based on the idea of enforced, explicit maximum request rates, which was originally proposed by Ian. The result should be that the load is moved back to the flooder - if a node makes too many requests, it will get RNFs rather than overwhelming the network. This should help to balance load and heal routing... but we've all heard that before. It also gives us some interesting possibilities w.r.t. fairness - we can change the minimum request interval on a per node basis, so that we can "punish" nodes for bad behaviour (such as the attack above), or accept more requests from nodes whose queries are most likely to be successful. This should not take more than a week to implement, and it will require a network reset. After that, we are going to attempt to set up a "testnet", an expanded form of the old watchme network - a completely separate, non-anonymous network for testing routing, debugging, and so on. Unfortunately for it to be of any use we will need several hundred nodes to run it. But eventually it should give us a much better idea of what is going on.2003/07/31
Rants go here! I replied to jrand0m's last post on the list IIRC, but future ones will be replied to in both places. I'm sure I'll have other things to rant about occasionally.
Since jrand0m asked nicely, and since I knew he wouldn't mind me borrowing his template, I've decided to set up a rant site of my own.
I actually got a mail from a Chinese citizen on the support list today. He can't get to the web site, as it is blocked. Hopefully I can send him a ZIP. I still think anyone using Freenet in China is crazier than I am though - but as Ian often points out, what they used before was even worse.