Beautiful BytecodesJekyll2022-05-19T04:06:37-04:00/Ian Preston/ianopolous@gmail.com
/privacy/Snoopers
/privacy/Snoopers2016-03-16T00:00:00-04:002016-03-16T00:00:00-04:00Ian Prestonianopolous@gmail.com<p>An MP for the Scottish National Party, Joanna Cherry, gave an excellent speech during the debate in parliament yesterday of the Investigatory Powers Bill (Snoopers Charter). She said most of the things I wanted someone to say, and in an eloquent way. I’m quoting it here for future reference.</p>
<p><b>Joanna Cherry (Edinburgh South West) (SNP):</b>
Before I begin my speech, on behalf of the Scottish National Party I want to associate myself with the comments of the Home Secretary and shadow Home Secretary regarding the death of the prison officer in Northern Ireland and extend my party’s heartfelt condolences and sympathies to his family, colleagues and friends.</p>
<p>The SNP joins the MPs from all parties in the House who have grave concerns about many aspects of the Bill. We do not doubt that that the law needs a thorough overhaul and welcome attempts to consolidate a number of statutes in order to have a modern, comprehensive law. We also recognise that the security services and police require adequate powers to fight terrorism and serious crime. However, such powers must always be shown to be necessary, proportionate and in accordance with the law. In particular, powers must not impinge unduly on the right to privacy or the security of private data. We feel that many of the Bill’s powers do not currently pass those tests. For that reason, the SNP cannot give its full support to the Bill in its current form. We intend to join others in the House to ensure that the Bill is as extensively amended as possible. We shall be abstaining today, but if the Bill is not amended to our satisfaction, we reserve the right to vote against it at a later stage.</p>
<p>The Bill is a rushed job that comes on the back of a draft Bill that lacked clarity and did not go far enough to protect civil liberties. In recent weeks, three parliamentary Committees have expressed significant misgivings about many aspects of the draft Bill and made extensive recommendations for its revisal. The Bill was published barely two weeks after the ink was dry on the last of those three reports, leaving insufficient time for the Government to go back to the drawing board to deal adequately with the concerns expressed by the three Committees. Like others in the House, SNP Members were concerned to read last week that the United Nations special rapporteur on the right to privacy concluded that some of the Bill’s proposals fail the benchmarks set in recent judgments of the European Court of Justice and the European Court of Human Rights. [Interruption.] Government Members may scoff, but I invite them to read his report as it contains a careful exploration of recent case law and should not be dismissed lightly.</p>
<p>The benchmarks suggest that surveillance should be targeted by means of warrants that are focused, specific and based on reasonable suspicion. Under the Bill, however, targeted interception warrants may apply to groups of persons or more than one organisation or premises. Bulk interception warrants lack specificity and lack any requirement for reasonable suspicion, giving licence for speculative surveillance. The shadow Home Secretary questioned whether we should be using the term “mass surveillance” in relation to this Bill, and I wonder whether it would be more accurate to say that aspects of the Bill permit “suspicionless surveillance”, which leads to civil liberties concerns. Another aspect of the Bill that concerns us is that an actual threat to national security is not required.</p>
<p>The powers to retain internet connection records and the bulk powers go beyond what is currently authorised in other western democracies and thus could set a dangerous precedent and a bad example internationally. The only other western democracy to authorise the retention of material similar to internet connection records was Denmark, which subsequently abandoned its experiment having found that it did not yield significant benefits for law enforcement. I see the Home Secretary looking at me and I am sure that she will argue that her proposed scheme differs from Denmark’s, but the devil is in the detail, which we will need to consider closely in Committee. The USA is rolling back from bulk data collection having found it to be unconstitutional in some cases and of questionable value in fighting terrorism. It is for this Government to justify why they alone are required to go so much further than other Governments in western democracies. Such operational cases as have been produced are anecdotal and hypothetical and do not constitute independent evaluation of the utility of bulk powers.</p>
<p><b>Lucy Frazer:</b>
If the hon. and learned Lady thinks that international comparisons are important, does she agree that the judicial authorisation procedure proposed by the Home Secretary goes further than in other European examples, such as Germany, the Netherlands and France?</p>
<p><b>Joanna Cherry:</b>
We need to compare apples with apples and oranges with oranges. A more correct comparison is with jurisdictions such as Canada and America, the systems of which are more similar to ours than the continental European jurisdictions that the hon. and learned Lady describes, but I will come back to that when I get to authorisation.</p>
<p>I am sure everyone in this House wants to get the balance right between protecting civil liberties, and giving the security services and the police the necessary and proportionate powers to fight serious crime and terrorism. However, we in the Scottish National party believe that the Government’s attempt has not got that important balance right and we are looking forward to working with other parliamentarians to try to get it right. We are worried that the Government are not giving sufficient time for the consideration of this enormous Bill. The 14 Home Office documents relating to the Bill that were released to Parliament on 1 March, including the Bill itself, extend to 1,182 pages, which is almost treble the amount of material released with the draft Bill last November. There is a suspicion that the amount of material being released in large tranches, coupled with relatively short timescales within which to consider and amend proposals, is an indication that the Government do not really want proper parliamentary scrutiny of this. We are determined to do our best to make sure that sufficient parliamentary scrutiny is provided.</p>
<p><b>Mrs May:</b>
Let me be absolutely clear about this. I have been in this House long enough to see Bills go through the House where parliamentarians have complained when the Government have failed to bring codes of practice that should sit alongside the Bill to the House at the very first stage of the debate. This Government have brought those codes of practice to the House more than several days before Second Reading, precisely so that Members of this House have an opportunity to see them and consider them alongside the Bill.</p>
<p><b>Joanna Cherry:</b>
But the Home Secretary misunderstands my complaint—it is not about the fact that the material has been produced. My complaint is that the material has been produced with a timescale following thereon that is not sufficient for us to scrutinise it properly. I must make something crystal clear before I go any further: the SNP will not be morally blackmailed or bullied by Conservative Members into blind support for a Bill of dubious legality in some respects, which seeks powers that go beyond those of other western democracies. We are not going to tolerate any suggestion that by seeking proper scrutiny of the Bill and full justification for the far-reaching powers sought, we are being soft on terrorism and serious crime. I would associate myself with the other main Opposition party in that respect.</p>
<p>Let me give hon. Members an example of why they can be assured that the SNP is not soft on terrorism or serious crime. We have been in government in Scotland for nine years and we have shown ourselves to be a responsible Government. Although issues of national security are reserved, we have always co-operated closely with the UK Government, for example, when Glasgow airport was attacked by terrorists in 2007. Our record in fighting crime in Scotland is second to none. The Scottish Government have got recorded crime down to a 41-year low and we are committed to a progressive justice policy. We will not, therefore, stand accused of being “soft” on serious crime or terrorism, because that is simply not a fair statement to make.</p>
<p>In the coming years, we confidently expect to be devising the security policy of an independent Scotland, and it will be a responsible security policy that will not only seek to work closely with near neighbours on these islands, but will look to international models from other democracies and strive to take proper cognisance of international human rights norms and the rule of law. That is all we are about in our opposition and in our scrutiny of this Bill.</p>
<p>Our concerns about the Bill are not just our concerns. They are shared by: the parties sitting around me; many Conservative Members sitting opposite me; many of the members of three parliamentary Committees; non-governmental organisations; the technical sector; eminent legal commentators—more than 200 senior lawyers signed that letter in The Guardian today; communications service providers; and the UN special rapporteur on the right to privacy. [Interruption.] I hear somebody shout confidently from the Government Benches that the 200 lawyers who signed that letter are wrong. I suggest that he or she—I think it was probably a he—looks at the list of those who signed it and perhaps accords them a bit more respect; there is room for a difference of opinion here.</p>
<p><b>Sammy Wilson (East Antrim) (DUP):</b>
For clarification, so that the hon. and learned Lady is not seen to be speaking for my party, may I ask whether she accepts that the balances in the Bill that the Secretary of State has outlined are, by and large, supported by people in Northern Ireland, simply because we have gone through the experience of terrorism and know how important such safeguards are for the general public?</p>
<p><b>Joanna Cherry:</b>
I always listen carefully to what the hon. Gentleman and his colleagues have to say because, as he says, they have experienced terrorism—indeed, they are, sadly, still experiencing it as a result of the tragic news we heard today. I apologise if I in any way included him in a sweeping statement, but I do not agree with him that the Government have got the balance right, and that is the whole purpose of my speech today.</p>
<p>The point I am seeking to make is that it is the job of a responsible Opposition not only to oppose responsibly and to scrutinise, but to articulate and inform public concerns. The public are concerned about this, and there is greater public knowledge about this Bill than perhaps there was last time around. A survey commissioned by Open-Xchange found that only 12% of the public believe that the Home Secretary has adequately explained the impact of the Bill to the UK public and presented a balanced argument for its introduction. I suspect that it is possibly a little unfair, pinning it all on the Home Secretary, because it is the responsibility of all of us in this House to inform our constituents about this Bill and where it is going.</p>
<p><b>Simon Hoare:</b>
Will the hon. and learned Lady give way?</p>
<p><b>Joanna Cherry:</b>
I hope the hon. Gentleman will not mind if I make some progress for the time being and possibly give way later. I mentioned the letter to The Guardian. I am conscious that the right hon. and learned Member for Beaconsfield (Mr Grieve), the former Attorney General, has expressed his view on the matter. I would always accord that the respect it deserves, but I respectfully disagree with him. The letter to The Guardian from the lawyers today was focused initially on the problem of bulk intercept. Even the Interception of Communications Commissioner’s Office, the independent watchdog, has said that bulk intercept provides “generalised initial interception”, and that is the issue here—it is the generality, and the lack of focus and specificity, that the lawyers are worried about.</p>
<p><b>Mr Grieve:</b>
I should emphasise that I take the letter seriously, because I regard it as a serious matter. If what was happening was what was set out in the first objection by those writing it, it would be a very serious matter indeed: the House would be sanctioning a system by which there was generalised access to electronic communications, in bulk. The point at issue is that that is not what actually goes on at all. Not only that, but if one looks at the Bill, one sees that it is clear that that should not be able to go on and that we will prevent it from happening if there is any possible risk of it. We have been round this issue on many occasions, and this is why there is a difficulty of communication and understanding on something that is fundamental to the way in which the agencies go about this work.</p>
<p><b>Joanna Cherry:</b>
I can only reiterate that I and many others, including more than 200 lawyers who signed this letter, disagree with the right hon. and learned Gentleman on this occasion and about this point. One thing that this issue illustrates is the importance of having very focused language in Bills dealing with such major matters of constitutional importance, rather than having vague language, which is not properly understood and which can on a later day be twisted by those it suits, to expand to cover powers that were not envisaged at the time. We are all well aware that that has happened in the past.</p>
<p>We should not dismiss too lightly the importance of the notion of the rule of law overarching this Bill. If the Government really want this legislation to be world-leading, they cannot have legislation that potentially violates international standards. As things stand, the UK is still bound by the jurisdiction of the European Court of Justice; there were no proposals to withdraw from the charter of fundamental rights in the agreement negotiated by the Prime Minister over Europe last month. We are still awaiting proposals for the repeal of the Human Rights Act, but the Government have recently been moving to reassure us that we will not be withdrawing as a signatory from the Council of Europe. We are therefore still going to be bound by the Court in Luxembourg and the Court in Strasbourg. Many distinguished lawyers believe that if this Bill is not significantly amended, the law of the UK will be on a collision course with those European Courts. I remind the Government that an unamended Bill could result in unnecessary and expensive litigation. It could require Parliament to revise the law all over again at some point in the future. That should not happen, provided that we ensure that the law meets international standards. [Interruption.] I hear Government Members shouting at me, “Which parts?” I will come to that when I get into the meat of my speech. [Interruption.] I suggest that they read the report that has come from the UN rapporteur on the right to privacy, and consider the law here. They may prefer to follow in the footsteps of Russia, which last December passed a law allowing its constitutional court to decide whether to comply with international human rights courts, but I would suggest that, on these matters at the very least, Russia is perhaps not the best role model for the United Kingdom.</p>
<p>I want to challenge the premise that the more privacy we sacrifice, the more security we gain, because that is not backed up by the evidence. Indeed, some of this House’s Committees have heard evidence that swamping analysts with data can impede investigation, because they are unable to find the crucial needles in the haystack of information before them. We should be looking at how to achieve security in a really intelligent way, not blanket data retention and suspicionless surveillance.</p>
<p>The Home Office responded to the Intelligence and Security Committee’s recommendations by simply adding one word to the start of the Bill so that the first part now refers to “privacy”. It has not, however, added any detail relating to any overarching principles of privacy. Its response to the ISC seems somewhat cynical.</p>
<p>I have indicated that the SNP is concerned about a number of aspects of the Bill. Time does not permit me to tackle all of them, but I am concerned about four in particular. I will endeavour to keep my comments to a minimum, bearing in mind that I speak on behalf of the third party in the House.</p>
<p>Our first issue with the Bill is the legal thresholds for surveillance; the second is the authorisation process, which the shadow Home Secretary has already talked about; the third is the provision for the collection of internet connection records; and the fourth is bulk powers, which I have already mentioned.</p>
<p>On the legal thresholds for surveillance, the Government essentially want to re-legislate on RIPA’s three broad statutory grounds. The SNP is not alone in its concern that those grounds are unnecessarily broad and vague and dangerously undefined. The Joint Committee on the draft Bill recommended that it should include definitions of national security and economic wellbeing, but that has not been done. The ISC recommended that economic wellbeing should be subsumed within a national security definition, finding it “unnecessarily confusing and complicated”. Those recommendations have been dismissed and the core purposes for which extraordinary powers can be used remain undefined and dangerously flexible.</p>
<p>On the authorisation of warrants, we welcome the move towards greater judicial involvement, and we acknowledge the fact that the Government have moved considerably towards the double lock. However, I agree with the shadow Home Secretary, because we also want an equal lock. Judicial review is not the same as judicial authorisation. Judicial review creates the illusion of judicial control over surveillance, and it does not achieve enough movement away from the status quo.</p>
<p>I want to give some concrete examples of that. The case law of the United Kingdom Supreme Court shows that, in civil proceedings that do not relate to deprivation of liberty, a less intensive standard of judicial review is applied—more Wednesbury reasonableness than strict necessity and proportionality—and that is why many fear that that is what will happen if the Bill is passed unamended. There will be little or no scope for review on the merits.</p>
<p><b>Suella Fernandes:</b>
Will the hon. and learned Lady accept that she is simply wrong? In their evidence to the Joint Committee, of which I was a member, Sir Stanley Burnton, senior judicial commissioner, and Lord Judge, senior surveillance commissioner, were clear that the Wednesbury unreasonableness standards had no place in this context. The wording of the Bill is clear, importing a clear judicial review standard involving necessity and proportionality.</p>
<p><b>Joanna Cherry:</b>
The hon. Lady will no doubt be unsurprised to hear that I do not accept that I am wrong. She is cherry-picking her way through the evidence that was heard. There was evidence contrary to the position that she has stated. I accept that there is a debate about this point, but I take the side that the review of judicial review principles does not go far enough. Why not go as far as other countries? Why not have one stage of judicial authorisation? That is the norm in comparable jurisdictions, by which I mean the United States, Australia and Canada. Judicial authorisation would help us, because it would encourage co-operation from US technology firms.</p>
<p>On a practical note, a two-stage process—whereby the issue goes to a Minister first and then to a judicial commissioner—risks delay. There is a huge volume of surveillance warrants, and it looks like there will be an awful lot more as a result of this Bill. It is unsuitable for a small number of Cabinet Ministers to deal with them.</p>
<p>I want to deal with another false premise that is often used to justify ministerial involvement in the issuance of warrants. Some people seek to argue that Ministers are democratically or politically accountable to this House on the issue of surveillance warrants, but that is a misconceived argument. Ministers are not really democratically accountable for their role in issuing warrants, because, first, the disclosure of the existence of a warrant has been criminalised and it will remain as such under the Bill. Secondly, all of us know—even those such as me who have been in this House for only nine months—that requests for information concerning such matters in this House are routinely parried with claims about national security. I do not accept that Ministers are practically, politically or democratically accountable to this House on the issuance of warrants. To return to the jurisprudence of the Strasbourg Courts, they have made it very clear that it is important to have effective supervision by an independent judiciary. We query whether the double lock mechanism meets that test.</p>
<p>We agree with many others that the case for collecting internet connection records, including the claimed benefit for law enforcement, is flawed. That is not just my say-so: there are many concerns across the industry. People who understand the technicalities far better than I do have explained the problem to me. I again associate myself with what the shadow Home Secretary said: the internet is not like the telephone system. An internet connection record cannot be compared to a telephone bill. The phone system consists of a set of records relating to when A calls B. If we collect phone system records, we will see at what time A called B and the duration of the call. As I understand it, the internet is more like a mailbox that collects packets of information and then takes them from A to B.</p>
<p>To take a rather middle-aged example, if somebody uses the Facebook messenger service, all the internet connection record will show is that he or she has connected to Facebook messenger. It will not show with whom he or she then communicated, because that occurs at a higher or lower level or in another unreachable packet. The internet connection record will not show the when, where and who that the Government say they want, and which they already get from phone records.</p>
<p>What the internet connection records will show is a detailed record of all of the internet connections of every person in the United Kingdom. There would be a 12-month log of websites visited, communication software used, system updates downloaded, desktop widgets, every mobile app used and logs of any other devices connected to the internet. I am advised that that includes baby monitors, games consoles, digital cameras and e-book readers. That is fantastically intrusive. As has been said, many public authorities will have access to these internet connection records, including Her Majesty’s Revenue and Customs, and the Department for Work and Pensions, and it will be access without a warrant. Do we really want to go that far? There is no other “Five Eyes” country that has gone as far. David Anderson QC said:</p>
<p>“Such obligations were not considered politically conceivable by my interlocutors in Germany, Canada or the US”</p>
<p>and therefore, he said, “a high degree of caution” should be in order.</p>
<p>Finally, let me turn to bulk powers. I have already made the point that even the Interception of Communications Commissioner’s Office says that bulk provides at the outset generalised initial intercept. We became aware of these bulk interception programmes only when they were disclosed by Edward Snowden in June 2013—whatever Members think about those disclosures and whether they were appropriate, that is how we became aware of the matter. This House has never before debated or voted on bulk powers, so we are being asked to do something very novel and very challenging, and we must do it properly.</p>
<p>The power to conduct mass interception has been inferred from the vaguely worded power in section 8(4) of RIPA, which illustrates the danger of vaguely worded legislation. Targeting bulk warrants at a telecommunications system or at entire populations rather than at specific individuals is a radical departure from both the common law and human rights law, yet that is the approach that will be maintained in this Bill. In many respects, that is the most worrying part of the Bill. Indeed, it is the part of the Bill about which the UN special rapporteur on privacy is most concerned. Let me read what he said, because it is very respectful of the tradition of the United Kingdom and it makes some very good points. He said:</p>
<p>“It would appear that the serious and possibly unintended consequences of legitimising bulk interception and bulk hacking are not being fully appreciated by the UK Government. Bearing in mind the huge influence that UK legislation still has in over 25% of the UN’s member states that still form part of the Commonwealth, as well as its proud tradition as a democracy which was one of the founders of leading regional human rights bodies such as the Council of Europe, the SRP encourages the UK Government to take this golden opportunity to set a good example and step back from taking disproportionate measures which may have negative ramifications far beyond the shores of the UK. More specifically, the SRP invites the UK Government to show greater commitment to protecting the fundamental right to privacy of its own citizens and those of others and also to desist from setting a bad example to other states by continuing to propose measures, especially bulk interception and bulk hacking, which prima facie fail the standards of several UK parliamentary Committees, run counter to the most recent judgements of the European Court of Justice and the European Court of Human Rights, and undermine the spirit of the very right to privacy.”</p>
<p>The rapporteur is appealing to the better tradition in this country, and saying that we should look at this Bill very carefully. He is suggesting not that we should throw it out, but that we scrutinise it very carefully, bearing in mind how far it intends to go in comparison with other countries and with existing international case law.</p>
<p><b>Andy Burnham:</b>
The hon. and learned Lady has made a very good speech this afternoon. Government Members should be working a little harder to reach out and build consensus. Before she finishes, may I invite her to say whether she will be supporting our call in Committee and on Report to make internet connection records accessible only through a warrant based on serious crime, not any crime, to give protection, and also for a clear definition of national security?</p>
<p><b>Joanna Cherry:</b>
Those are both issues on which we will work with the Labour party. I have already indicated that we intend to attempt to amend the Bill extensively in Committee. We are very concerned about internet connection records. We query whether their retention is necessary or appropriate at all, but we will look seriously at proposals put forward by other parties and will work with them.</p>
<p>The SNP is in favour of targeted surveillance. We welcome the double lock on judicial authorisation as an improvement, but it does not go far enough. Our concern is, quite clearly, that many of the powers sought in this Bill are of dubious legality and go further than other western democracies without sufficient justification. It is for that reason that we cannot give this Bill, in its current form, our full support. We will work with others to attempt to amend it extensively. Today, we shall abstain, but if the Bill is not amended to our satisfaction, we reserve the right to vote it down at a later stage.</p>
<p><a href="/privacy/Snoopers">Joanna Cherry's speech during the IPB debate in UK parliament</a> was originally published by Ian Preston at <a href="">Beautiful Bytecodes</a> on March 16, 2016.</p>
/java/IPFS
/java/IPFS2015-09-27T00:00:00-04:002015-09-27T00:00:00-04:00Ian Prestonianopolous@gmail.com<p>The InterPlanetary File System (<a href="https://ipfs.io/">IPFS</a>) has ambitious goals. It aims to decentralise the internet, make it more secure, and enable the creation of purely distributed applications. It combines some of the best features of the peer-to-peer protocol BitTorrent, the version control system Git and distributed hash tables. There is no central node, and nodes do not need to trust each other.</p>
<p>When you add a file to IPFS your node broadcasts to the network the hash of the file (This is the location of the file as far as the network is concerned). This enables any node in the network to ask your node for the file. When a node asks the network for a certain hash, it receives a list of nodes serving that hash and can download from any of them. The result is hashed locally to prove that you got what you asked for. No such thing as a man-in-the-middle attack!</p>
<p>I have recently implemented a <a href="https://github.com/ipfs/java-ipfs-api">Java API to IPFS</a>. This enables easy, type safe access to a local IPFS node over HTTP. It is really simple to use! Just add the (40 KiB!) IPFS.jar to your classpath. We hope to use IPFS to replace the distributed hash table in <a href="https://peergos.org">Peergos</a>, our secure, distributed file storage, sharing and social network.</p>
<p>Ensure that IPFS is <a href="https://ipfs.io/docs/install/">installed</a> and running (IPFS is written in Go):</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="c"># We will build from source, we're</span>
<span class="c"># developers after all</span>
<span class="nb">sudo </span>apt-gt <span class="nb">install </span>golang
<span class="c"># Git is used to download dependencies</span>
<span class="nb">sudo </span>apt-get <span class="nb">install </span>git
<span class="c"># download and build IPFS</span>
go get <span class="nt">-u</span> github.com/ipfs/go-ipfs/cmd/ipfs
<span class="c"># run the IPFS daemon</span>
ipfs daemon </code></pre></figure>
<p>You can then create an accessor in Java with:</p>
<figure class="highlight"><pre><code class="language-java" data-lang="java"><span class="no">IPFS</span> <span class="n">ipfs</span> <span class="o">=</span> <span class="k">new</span> <span class="no">IPFS</span><span class="o">(</span><span class="k">new</span> <span class="nc">MultiAddress</span><span class="o">(</span><span class="s">"/ip4/127.0.0.1/tcp/5001"</span><span class="o">));</span></code></pre></figure>
<p>To add a file to IPFS simply,</p>
<figure class="highlight"><pre><code class="language-java" data-lang="java"><span class="nc">NamedStreamable</span> <span class="n">file</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">NamedStreamable</span><span class="o">.</span><span class="na">FileWrapper</span><span class="o">(</span><span class="k">new</span> <span class="nc">File</span><span class="o">(</span><span class="s">"hello.txt"</span><span class="o">));</span>
<span class="nc">MerkleNode</span> <span class="n">addResult</span> <span class="o">=</span> <span class="n">ipfs</span><span class="o">.</span><span class="na">add</span><span class="o">(</span><span class="n">file</span><span class="o">);</span>
<span class="nc">Multihash</span> <span class="n">pointer</span> <span class="o">=</span> <span class="n">addResult</span><span class="o">.</span><span class="na">hash</span><span class="o">;</span></code></pre></figure>
<p>or if you prefer to work with raw byte arrays:</p>
<figure class="highlight"><pre><code class="language-java" data-lang="java"><span class="kt">byte</span><span class="o">[]</span> <span class="n">contents</span> <span class="o">=</span> <span class="s">"G'day world! IPFS rocks!"</span><span class="o">.</span><span class="na">getBytes</span><span class="o">();</span>
<span class="nc">String</span> <span class="n">name</span> <span class="o">=</span> <span class="s">"hello.txt"</span><span class="o">;</span>
<span class="nc">NamedStreamable</span> <span class="n">raw</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">NamedStreamable</span><span class="o">.</span><span class="na">ByteArrayWrapper</span><span class="o">(</span><span class="n">name</span><span class="o">,</span> <span class="n">contents</span><span class="o">);</span>
<span class="nc">MerkleNode</span> <span class="n">addResult</span> <span class="o">=</span> <span class="n">ipfs</span><span class="o">.</span><span class="na">add</span><span class="o">(</span><span class="n">raw</span><span class="o">);</span>
<span class="nc">Multihash</span> <span class="n">pointer</span> <span class="o">=</span> <span class="n">addResult</span><span class="o">.</span><span class="na">hash</span><span class="o">;</span></code></pre></figure>
<p>To get the contents of your file back,</p>
<figure class="highlight"><pre><code class="language-java" data-lang="java"><span class="nc">Multihash</span> <span class="n">filePointer</span> <span class="o">=</span> <span class="nc">Multihash</span><span class="o">.</span><span class="na">fromBase58</span><span class="o">(</span><span class="s">"QmPZ9gcCEpqKTo6aq61g2nXGUhM4iCL3ewB6LDXZCtioEB"</span><span class="o">);</span>
<span class="kt">byte</span><span class="o">[]</span> <span class="n">contents</span> <span class="o">=</span> <span class="n">ipfs</span><span class="o">.</span><span class="na">cat</span><span class="o">(</span><span class="n">filePointer</span><span class="o">);</span></code></pre></figure>
<p>If you want to keep a copy of something someone else has added, you can “pin” it. This means that you will keep a local copy of the file, and also act as a server for it. (Files you add are automatically pinned)</p>
<figure class="highlight"><pre><code class="language-java" data-lang="java"><span class="nc">Multihash</span> <span class="n">file</span> <span class="o">=</span> <span class="nc">Multihash</span><span class="o">.</span><span class="na">fromBase58</span><span class="o">(</span><span class="s">"QmPZ9gcCEpqKTo6aq61g2nXGUhM4iCL3ewB6LDXZCtioEB"</span><span class="o">);</span>
<span class="nc">List</span><span class="o"><</span><span class="nc">Multihash</span><span class="o">></span> <span class="n">pinned</span> <span class="o">=</span> <span class="n">ipfs</span><span class="o">.</span><span class="na">pin</span><span class="o">.</span><span class="na">add</span><span class="o">(</span><span class="n">file</span><span class="o">,</span> <span class="kc">true</span><span class="o">);</span></code></pre></figure>
<p>Any files you retrieve are temporarily cached, and served. Periodically, the local files that aren’t pinned are garbage collected. You can force this clean up with,</p>
<figure class="highlight"><pre><code class="language-java" data-lang="java"><span class="n">ipfs</span><span class="o">.</span><span class="na">repo</span><span class="o">.</span><span class="na">gc</span><span class="o">();</span></code></pre></figure>
<p>If you want to explicitly remove a file from the local cache, use:</p>
<figure class="highlight"><pre><code class="language-java" data-lang="java"><span class="nc">Multihash</span> <span class="n">file</span> <span class="o">=</span> <span class="nc">Multihash</span><span class="o">.</span><span class="na">fromBase58</span><span class="o">(</span><span class="s">"QmPZ9gcCEpqKTo6aq61g2nXGUhM4iCL3ewB6LDXZCtioEB"</span><span class="o">);</span>
<span class="nc">List</span><span class="o"><</span><span class="nc">Multihash</span><span class="o">></span> <span class="n">removed</span> <span class="o">=</span> <span class="n">ipfs</span><span class="o">.</span><span class="na">pin</span><span class="o">.</span><span class="na">rm</span><span class="o">(</span><span class="n">file</span><span class="o">);</span></code></pre></figure>
<p>Unleash the power of IPFS on your JVM!</p>
<p><a href="/java/IPFS">IPFS comes to Java</a> was originally published by Ian Preston at <a href="">Beautiful Bytecodes</a> on September 27, 2015.</p>
/javascript/innerHTML
/javascript/innerHTML2014-08-06T00:00:00-04:002014-08-06T00:00:00-04:00Ian Prestonianopolous@gmail.com<p>The other day I was writing a web user interface in Javascript to view a dataset and I discovered that some calls to set innerHTML on a div element were taking over a minute (in Chrome). This line was the culprit:</p>
<figure class="highlight"><pre><code class="language-js" data-lang="js"><span class="nx">someDiv</span><span class="p">.</span><span class="nx">innerHTML</span> <span class="o">=</span> <span class="nx">newHTML</span><span class="p">;</span></code></pre></figure>
<p>Looks innocuous right? I dug a little deeper and discovered that the slow calls were only when the existing html in the div was large (~6M characters). The new html could be trivially small, and still the call was taking 1 minute. The existing html had a large number of elements with their onclick function set to some function on the document. It seemed like Chrome was taking a long time removing the exisiting elements from the div. I tried out an alternative to directly setting innerHTML.</p>
<figure class="highlight"><pre><code class="language-js" data-lang="js"><span class="kd">function</span> <span class="nx">replaceInnerHTML</span><span class="p">(</span><span class="nx">oldDiv</span><span class="p">,</span> <span class="nx">html</span><span class="p">)</span> <span class="p">{</span>
<span class="kd">var</span> <span class="nx">newDiv</span> <span class="o">=</span> <span class="nx">oldDiv</span><span class="p">.</span><span class="nx">cloneNode</span><span class="p">(</span><span class="kc">false</span><span class="p">);</span>
<span class="nx">newDiv</span><span class="p">.</span><span class="nx">innerHTML</span> <span class="o">=</span> <span class="nx">html</span><span class="p">;</span>
<span class="nx">oldDiv</span><span class="p">.</span><span class="nx">parentNode</span><span class="p">.</span><span class="nx">replaceChild</span><span class="p">(</span><span class="nx">newDiv</span><span class="p">,</span> <span class="nx">oldDiv</span><span class="p">);</span>
<span class="p">};</span></code></pre></figure>
<p>The original node is cloned to preserve all its properties except for it child nodes. Then we set the new node’s innerHTML and replace references to the old node in its parent with the new node. This was just as fast as the direct method for setting a large html value, but up to 3000 times faster subsequently setting the html to something else! You can try it out below. The time taken to replace the contents of a div directly using innerHTML increases with the number of child elements currently in the div. The ratio shows just how much faster my method is. In Chrome I’ve seen it vary from 4 to 3000X faster, and up to 600X faster in Firefox.
<br />
<br />Current large html size (characters): <span id="size">0</span></p>
<p><a class="btn btn-success btn-large" onclick="timeAndCompare();">Time setting innerHTML directly and my replacement method</a>
<a class="btn btn-danger btn-large" onclick="largeHtml += largeHtml;updateSize();">Double large html size</a></p>
<h3>Times taken (S)</h3>
<table>
<col width="200" />
<col width="100" />
<col width="200" />
<col width="100" />
<tr>
<td>Direct (large):</td>
<td><span id="directL">0</span>
</td>
<td>Direct (small):</td>
<td><span id="directS">0</span>
</td>
</tr>
<tr>
<td>Custom (large):</td>
<td><span id="customL">0</span>
</td>
<td>Custom (small):</td>
<td><span id="customS">0</span>
</td>
</tr>
</table>
<p><br />
<span id="speedup" style="color:red"></span>
<br />
<span style="width:900px; height:200px; overflow-y:auto; word-wrap:break-word;">
<span id="box"></span>
</span></p>
<script>
var smallHtml = "Some text for the small html.";
var largeHtml = "";
for (var i = 0; i < 10000; i++) {
largeHtml += "<span style='cursor:pointer; color:blue;'>name" + i + "</span> ";
}
updateSize();
function updateSize() {
document.getElementById("size").innerHTML = largeHtml.length;
};
function someFunction(stuff) {};
function timeAndCompare() {
var t1 = time(setter(direct, largeHtml))();
var t2 = time(setter(direct, smallHtml))();
var t3 = time(setter(fast, largeHtml))();
var t4 = time(setter(fast, smallHtml))();
document.getElementById("directL").innerHTML = (t1) / 1000;
document.getElementById("directS").innerHTML = (t2) / 1000;
document.getElementById("customL").innerHTML = (t3) / 1000;
document.getElementById("customS").innerHTML = (t4) / 1000;
var ratio = Math.floor(t2/t4*10)/10;
document.getElementById("speedup").innerHTML = "Speed up: " + ratio + ((ratio > 100) ? "!!!! Holy cow!" : ((ratio > 5) ? "!" : ""));
};
function time(f) {
return function () {
var t1 = Date.now();
f();
var t2 = Date.now();
return t2 - t1;
};
};
var setter = function (f, html) {
return function () {
f(document.getElementById("box"), html);
};
};
var direct = function (element, html) {
element.innerHTML = html;
};
var fast = function (oldElement, html) {
var newElement = oldElement.cloneNode(false);
newElement.innerHTML = html;
oldElement.parentNode.replaceChild(newElement, oldElement);
};
</script>
<p><a href="/javascript/innerHTML">A faster way to replace an element's contents in a web page</a> was originally published by Ian Preston at <a href="">Beautiful Bytecodes</a> on August 06, 2014.</p>
/maths/surreal
/maths/surreal2014-06-30T00:00:00-04:002014-06-30T00:00:00-04:00Ian Prestonianopolous@gmail.com<figure>
<a href="/images/surreal.gif"><img src="/images/surreal.gif" /></a>
<figcaption><a href="http://ianopolous.github.io/" data-toggle="tooltip" title="Surreal!">The construction of the Surreal numbers.</a></figcaption>
</figure>
<p>We’ve all heard of integers, rationals, reals, even complex
numbers, but what on earth are surreal numbers? They are a
beautiful way of defining a class of numbers which includes
all reals, but also ordinal numbers; i.e. all the different
infinities and even infinitesimal numbers. Not only this but
we get a full system of arithmetic for all these numbers.
Ever wondered what ($\infty-1$) is, or $\sqrt{\infty}$ ? Before
we get stuck into that, let’s learn some history.</p>
<p>It all started a long, long time ago in a galaxy far, far away
(Cambridge in the 1970s). A man by the name of John H. Conway
was playing Go, an ancient Chinese game that’s very elegant in
itself. After much thought, he realised that the later stages
of the game could be thought of as the sum of many smaller
games. Conway then applied his ideas to other games like
Checkers and Dominoes. It seemed that these games were behaving
as if they were numbers.</p>
<p>Conway’s ideas led him to define a new family of numbers from
sets, which were contructed essentially by a series of binary
choices. It turned out that this wonderfully simple new system
included all the real numbers and more.</p>
<p>In 1972, Conway described
his system of numbers to computer scientist Donald Knuth at Stanford.
Knuth (creator of the $\TeX$ typesetting system) then went away and wrote a short novelette
introducing these numbers. It was the first time
for a major mathematical discovery to be published in a work of
fiction first. Knuth coined the term <i>surreal numbers</i>; taking “Sur” from the French for “above”. The surreal
numbers satisfy the axioms for a field (but the question of whether or not they constitute a field is
complicated by the fact that, collectively, they are too large to form a set).</p>
<p>Only two axioms are needed to give you all the surreal numbers. Three more and you get
additon and subtraction. One further axiom gives you multiplication.
So not only can you get the full structure of the reals from six axioms, but
we also get the full power of this arithmetic for the rest of the surreal
numbers! So without further ado let’s jump into this beautiful landscape.</p>
<p><b>Rule 1</b> - Defining what a number is</p>
<p>A number $x$ is an ordered pair of sets of numbers created previously
\(x = (X_L,X_R)\text{, where }X_L \not\geq X_R\)
I’m using notation where $ A \not\geq B $ means that, for any a in A
and b in B,
\(a \not\geq b\)</p>
<p>“But what is $\not\geq$?” I hear you say. Well that’s the beauty of it –
we don’t have to define it! On the 1st day there was nothing and so
the guy in charge said “Let there be $0$”. We haven’t created any numbers
yet, so the left and right sets of zero have to be empty! Yes that’s it!
\(0=(\emptyset,\emptyset) \text{, where $\emptyset$ is the empty set}\)
We have to check that $0$ is a number. To do this we must check that each
element of the left set of zero is $\not\geq$ each element of the right
set. This is trivial because neither set has any elements!</p>
<p>So zero was created on the first day. Now on the second day we can use
$0$ as an element in the left and right sets. So we get two new numbers,</p>
\[\begin{eqnarray*}
1=(\{0\},\emptyset) \\
-1=(\emptyset,\{0\}) \\
\end{eqnarray*}\]
<p>It is easy to check that these are both numbers also, because of the
empty set again. Now you ask me why the first one is called 1 and the
second -1. Here’s where we need our second rule.</p>
<p><b>Rule 2</b> - defining “$x$ is less than or like $y$”</p>
\[x \leq y \text{ means } X_L \not\geq y \text{ and } x \not\geq Y_R\]
<p>The first thing to prove is that $0$ is less than or like $0$. This
also falls through because of the empty set. I bet you’ve never proved
that $0 \leq 0$ before!</p>
<p>We can go on and prove that $-1\leq 0 $ in the same way. If we try
and prove that $-1=0$ by then showing $0 \leq -1$ we hit a brick wall
(thankfully). As it is equivalent to $0 \not\geq 0$ which we already
know is false:
\(0 \leq -1 \text{ by definition, means } \emptyset \not\geq -1\text{ and }
0 \not\geq 0\)
Thus -1 is strictly less than $0$. Well, it all makes
good sense so far. We can carry on this way and on the 3rd day it turns
out we get 17 new numbers, although most of them are <i>like</i> each
other and we are left with only 4 new ones: 2, -2, $\frac{1}{2}$ and
$-\frac{1}{2}$.</p>
<p>Another thing we need – and which is normally <i>assumed</i> when dealing with the reals – is
the transitive law. We can prove (yes, I said <i>prove</i>) the
transitive law.</p>
<p>To do this we assume the opposite; that we have 3 numbers satisfying
the following:</p>
\[x \leq y\text{, } y \leq z \text{ and } x \not\leq z\]
<p>We will call such a triple of numbers, $(x,y,z)$ a <i>naughty</i> triple of numbers.
The first two of these mean,</p>
\[\begin{eqnarray*}
X_L \not\geq y, x \not\geq Y_R\\
Y_L \not\geq z, y \not\geq Z_R\\
\end{eqnarray*}\]
<p>The last condition for $(x,y,z)$ to be <i>naughty</i> means that at least
one of the following is true.</p>
\[\begin{eqnarray*}
\text{There is some } x_L \in X_L\text{ such that }x_L \geq z \\
\text{There is some } z_R \in Z_R\text{ such that }x \geq z_R \\
\end{eqnarray*}\]
<p>Case 1: ($x_L \geq z$)
This means that $(y,z,x_L)$ are <i>naughty</i> numbers.
This is because we have, ($y \leq z$) and ($z \leq x_L$). Then using the initial
assumptions about $x$, $y$ and $z$ we also have ($X_L \not\geq y$). Therefore ($y \not\leq x_L$)</p>
<p>Case 2: ($x \geq z_R$)
This means that $(z_R,x,y)$ are <i>naughty</i> numbers for similar reasons to case 1. Can you see them?</p>
<p>If we call the day on which a number is created its <i>day number</i>, then
in both cases the sum of the three day numbers has decreased, because a number
can only be made up of sets of numbers that were previously made. This
can’t keep going forever because the smallest possible day sum is 3.
Basically it’s proof by induction on the day sum number: We prove that it is true
for a day sum of 3 ($x=y=z=0$). Then we have shown that if it is true for all
previous day sum numbers, then it is true for the next one. Now, look at the
diagram at the beginning of this article. It makes a little more sense now
doesn’t it!</p>
<p>We can go on and prove some important things, like the fact that all numbers are related:
\(x \not\geq y \rightarrow x \leq y\)
and that a number is between its two sets,
\(X_L < x < X_R\)</p>
<p>Addition is defined in a similar recursive way to $\leq$.</p>
<p><b>Rule 3</b> - Defining addition</p>
\[x+y = ((X_L+y)\cup(Y_L+x),(Y_R+x)\cup(X_R+y))\]
<p>What does this mean? It means that to get the left set of $x+y$,
you take every element $x_L$ of $X_L$ and compute $x_L+y$, then do
the same with x and y swapped. Then just take all the results of
these additions and put them in the left set of $x+y$. For the
right set of $x+y$ you just do exactly the same thing but with R’s
instead of L’s.</p>
<p>For example,</p>
\[\begin{eqnarray*}
0+0 = (\emptyset,\emptyset)=0\\
0+1 = (\{0+0\},\emptyset)=(\{0\},\emptyset)=1\\
\end{eqnarray*}\]
<p>The other two rules you need for addition are really to do with subtraction.</p>
<p><b>Rule 4</b> - Defining negative</p>
\[-x = (-X_R,-X_L)\]
<p><b>Rule 5</b> - Defining subtraction</p>
<p>\(x-y=x+(-y)\)
With these the surreals become a group under addition. However, there’s one more
subtlety we’ve so far forgotten about: we have to prove that $x+y$ and $-x$ are
both numbers; i.e. that they satisfy Rule 1. For this I’ll refer you to Knuth’s book.</p>
<p>So far we’ve only created numbers that have a finite binary representation (and
every number with a finite binary representation gets created eventually). To get
the remaining (real) numbers we need to keep going until we get to $\aleph_0$ day (the limit as the day number tends to infinity)!
On this day all the remaining real numbers were created. To see this we just
have to realise that now $X_L$ and $X_R$ can be <i>infinite</i> sets. So any
remaining real number $r$ can just be made with a set of binary numbers below $r$ in $R_L$
that converge to $r$, and similarly for $R_H$. Now the rest of the diagram
is beginning to make sense… We also get infinity!
\(\omega = (\{1,2,3...\},\emptyset)\)</p>
<p>Something else intriguing is created on $\aleph_0$ day. Consider this number,
\(\epsilon = (\{0\},\{\frac{1}{2},\frac{1}{4},\frac{1}{8}...\})\)
This is greater than $0$ but smaller than every positive fraction (using the result
that a number is between each of its two sets)! In fact we get this kind of
number around every number that was created before $\aleph_0$ day. On the next
day the rest of the reals get numbers like this close to them (i.e. that are
closer to them than any other real number).</p>
<p>There is one important subtlety I’ve glossed over. The day sum argument doesn’t
work once we pass $\aleph_0$ day. We need a new way of handling induction.
Basically, we always showed that if a theorem fails for $x$
then it fails for some element $x_L$ in $X_L$ and then it fails for an element in one
of $x_L$’s sets and so on. This gives us a sequence of <i>ancestors</i> of $x$. All
we need to do is realise that this sequence cannot be infinite. To see this we could,
whenever we create a new number $x$, prove simultaneously that there is no
infinite sequence of <i>ancestors</i> of $x$. NICE!</p>
<p>With the final rule, which defines multiplication, we get the full tapestry of the
surreal numbers and their arithmetic. We can then prove all kinds of interesting
things. Like,
\(\epsilon \omega =1\)</p>
<p>Now, as promised,
\(\omega-1=(\{1,2,3,...\},\{\omega\})\)
and
\(\sqrt{\omega}=(\{1,2,3,...\},\{\frac{\omega}{1},\frac{\omega}{2},\frac{\omega}{3},...\})\)</p>
<p>For more information, here are a few very good books:<br /></p>
<ul>
<li>
<a href="http://www-cs-faculty.stanford.edu/\~knuth/sn.html">Donald Ervin Knuth, <i>Surreal Numbers: How Two Ex-Students Turned on to Pure Mathematics and Found Total Happiness</i></a>
<li>John Horton Conway, <i>On Numbers and Games</i>
<li>David Wolfe and Elwyn R Berlekamp, <i>Mathematical Go: Chilling
gets the last point</i>
</ul>
</li></li></li></ul>
<p><a href="/maths/surreal">Surreal numbers</a> was originally published by Ian Preston at <a href="">Beautiful Bytecodes</a> on June 30, 2014.</p>