tag:nothingbutsnark.svbtle.com,2014:/feedTall, Snarky Canadian2015-08-11T11:03:51-07:00Brett Cannonhttps://nothingbutsnark.svbtle.comSvbtle.comtag:nothingbutsnark.svbtle.com,2014:Post/moving-the-blog-to-silvrback2015-08-11T11:03:51-07:002015-08-11T11:03:51-07:00Moving my blog to Silvrback<p>I have been using <a href="http://www.svbtle.com">Svbtle</a> since it opened up to the public and in general I have enjoyed it. It’s has a very clean design and you get to blog in Markdown which is great when you typically talk about code.</p>
<p>But I have been wanting to set up a custom domain for my blog which typically requires paying for blog hosting (which I’m fine with). And if I’m going to be putting money into something I want to make sure it’s the best I can get for the money I’m spending. And that’s when I found <a href="http://www.silvrback.com">Silvrback</a>.</p>
<p>Basically Silvrback does everything that Svbtle does in terms of features from custom domain to letting people subscribe by email. I still do prefer Svbtle’s overall look, Silvrback isn’t far off. And then there is the fact that Silvrback is USD 30/year while Svtble works out to USD 72/year. Look quibbles aside, I just couldn’t ignore the difference, especially when I have Damian of Silvrback so responsive to questions/suggestions I had.</p>
<p>And so my blog/bio page is now at <a href="http://www.snarky.ca/">http://www.snarky.ca/</a> and is hosted by Silvrback (if you prefer HTTPS access you can get it through <a href="https://nothingbutsnark.silvrback.com/">https://nothingbutsnark.silvrback.com/</a> but obviously that is not as stable as the custom domain).</p>
tag:nothingbutsnark.svbtle.com,2014:Post/python-3-support-on-pypi2015-06-12T08:11:14-07:002015-06-12T08:11:14-07:00Python 3 support on PyPI<p>At (and since) <a href="https://us.pycon.org/2015/">PyCon 2015</a>, there has been interest in trying to get quantified numbers in relation to Python 3 adoption (see <a href="https://caremad.io/2015/04/a-year-of-pypi-downloads/">PyPI download numbers</a> and <a href="http://nothingbutsnark.svbtle.com/looking-out-python-usage-in-the-astronomy-community">uptake in the astronomy community</a>). One number I am personally interested in is per-project adoption of Python 3. While the <a href="http://python3wos.appspot.com/">Python 3 Wall of Superpowers</a> shows wide support for Python 3 with the top projects by download, I wanted to take a larger look at PyPI as a whole to measure project adoption. So I wrote a <a href="https://gist.github.com/brettcannon/d03fbcf365a9c76d4aaa">script</a> that downloads the JSON data for every project on PyPI and then analyzes the data in various ways.</p>
<h2 id="methodology_2">Methodology <a class="head_anchor" href="#methodology_2">#</a>
</h2>
<p>The methodology I used was to classify a project as supporting Python 3 if its trove classifier said it did (in other words, putting a project in the “Python 3” bin means it <em>at least</em> supports Python 3 and <em>may</em> support Python 2 as well), the same goes for Python 2 classifying (although if the project supports Python 3 it goes in that bin instead), and all other cases being bucketed as “unknown”. Unfortunately you will discover that the majority of projects don’t have a trove classifier specifying what versions of Python they support. Because of this and knowing that Python 2 is still the dominant version of Python it might be tempting to automatically clump all unknown projects into the Python 2 bucket, but as <a href="https://github.com/brettcannon/caniusepython3/blob/master/caniusepython3/overrides.json">caniusepython3’s overrides file</a> shows, Python 3 projects also don’t always specify their trove classifiers properly. So while you can group all unknown projects into Python 2 as a simple worst-case scenario, it is also conservative and thus not totally accurate (as is anything where human beings are relied upon to provide the data). I just unfortunately don’t know what the possible rate is for people leaving the trove classifier off of Python 3 projects without doing a random sample, so I have erred on the side of conservative and just clumped all unknown projects into the Python 2 bucket in any adoption rate numbers cited below (which are calculated by simply doing <code class="prettyprint">Python 3 / (Python 2 + unknown)</code> so it’s a ratio more than an overall percentage which means 100% would indicate half of all projects are in Python 3 and the other half are in Python 3).</p>
<p>Another thing to realize about these numbers is that when one project switches to Python 3 it doesn’t necessarily lead to only one other project porting, but possibly many. Think of when Django added Python 3 support: it didn’t open the possibility of just one other project to port to Python 3 but hundreds. That means the impact of these numbers can be a little misleading since it is not going to be a linear porting rate due to network effects.</p>
<p>And finally, Python 3 will never hit 100%. This isn’t just because there will be people who never port to Python 3 (because there will be), but also because some people have created separate projects for their Python 3 ports. I don’t know how widespread this practice is, but it does mean in some instances the numbers actually cancel each other out because one project counts once for Python 2 support and then again for Python 3 support under a different name. In other words whatever number you want to consider meaning Python 3 has been adopted by the community, you need to make sure it’s below 100%.</p>
<h2 id="some-numbers_2">Some numbers <a class="head_anchor" href="#some-numbers_2">#</a>
</h2><h3 id="by-release-date_3">By release date <a class="head_anchor" href="#by-release-date_3">#</a>
</h3>
<p>First I looked at every project on PyPI that has ever uploaded a release (i.e., created a project and uploaded some file):</p>
<ul>
<li>Unknown: 34,447</li>
<li>Python 2: 8,064</li>
<li>Python 3: 11,377</li>
</ul>
<p>This is a rather junk number since Python 2 is heavily weighted thanks to simply existing longer and thus having more abandoned projects. But even in this instance, Python 3 is supported by over 26% of projects.</p>
<p>The next number I looked at is the number of projects which released a version in the last 2 years:</p>
<ul>
<li>Unknown: 19,760</li>
<li>Python 2: 5,898</li>
<li>Python 3: 10,295</li>
</ul>
<p>40% of projects support Python 3 with this filter. I would argue this is the bare minimum release cadence to consider a project not fully abandoned.</p>
<p>Now it’s probably more reasonable to claim that a project that has been updated in the last year is a better metric:</p>
<ul>
<li>Unknown: 12,864</li>
<li>Python 2: 4,091</li>
<li>Python 3: 8,329</li>
</ul>
<p>At about 49%, there is a clear increase in Python 3 support over the past year. This might correspond with <a href="https://www.python.org/download/releases/3.4.0/">Python 3.4.0</a> being released on 2014-03-16 which falls closer to being a year ago than two years ago.</p>
<p>What happens if you make the cut-off at 6 months?</p>
<ul>
<li>Unknown: 8,183</li>
<li>Python 2: 2,809</li>
<li>Python 3: 6,134</li>
</ul>
<p>That’s over 55% for Python 3. This seems to suggest that more and more people are using Python 3 and that the growth from 2 years ago versus this past year is simply increased Python 3 support.</p>
<p>Now looking at just the latest release still runs into the issue of new projects that were simply uploaded and then never touched again, which is a similar issue as just looking at all PyPI projects without any sort of time horizon. So what does the numbers look like if 2 releases have to be made within the time horizon? For instance, what if a project has to make 2 releases within the past year?</p>
<ul>
<li>Unknown: 8,033</li>
<li>Python 2: 2,779</li>
<li>Python 3: 5,889</li>
</ul>
<p>That puts Python 3 adoption at over 54% compared to Python 2. That’s a measurable increase compared to projects that had to have only a single release within the past year.</p>
<p>How about 2 releases in the last six months?</p>
<ul>
<li>Unknown: 4,747</li>
<li>Python 2: 1,732</li>
<li>Python 3: 3,920</li>
</ul>
<p>This gets us to 60% adoption for Python 3 compared to Python 2. Once again, an increase compared to the single release case. This continues to suggest that a decent of active projects are supporting Python 3.</p>
<h2 id="by-downloads_2">By downloads <a class="head_anchor" href="#by-downloads_2">#</a>
</h2>
<p>Having looked at release dates and how recently two releases have occurred, you might have noticed that popularity has still not come into the picture. What happens if you take monthly download counts into consideration? Take for instance all projects that are downloaded more than 1,440 time a month (which is equal to if the project was downloaded twice an hour):</p>
<ul>
<li>Unknown: 4,115</li>
<li>Python 2: 1,215</li>
<li>Python 3: 3,332</li>
</ul>
<p>Over 62%. Now download numbers are really unreliable as continuous integration can muck with the numbers, so this should be taken with a grain of salt. And to reiterate, most people who tell me they have a dependency blocking them from moving to Python 3 is more of a long tail issue and not major projects.</p>
<p>Now what if you use all of the measurement angles that I have suggested? What if you only look at projects that have had 2 releases in the past year which were downloaded at least 1,440 times in the past month?</p>
<ul>
<li>Unknown: 2,241</li>
<li>Python 2: 868</li>
<li>Python 3: 2,465</li>
</ul>
<p>Over 79% in this instance. Obviously the same issues with the other download-based numbers apply here, especially in the instance of people saying the long tail is their blocking dependency and not major projects. This also aligns with the Python 3 Wall of Superpowers suggesting that very active, popular projects have mostly added Python 3 support.</p>
<h3 id="by-some-definition-of-quotactively-maintained_3">By some definition of “actively maintained” <a class="head_anchor" href="#by-some-definition-of-quotactively-maintained_3">#</a>
</h3>
<p>All these numbers are nice, but the definition of an “actively maintained” project is still rather loose. What if we tightened it up a bit more? One could argue that a project that has made two releases over the past year that were at least 90 days apart seems like an active project. This suggests a rough average of a release at least every six months and weeds out any quick releases due to bugs being found immediately after release. Those numbers look like:</p>
<ul>
<li>Unknown: 3,158</li>
<li>Python 2: 1,188</li>
<li>Python 3: 2,786</li>
</ul>
<p>That’s 64% and more in the range of measurements based on downloads rather than the strict latest release measurements. This suggests to me that if a good number of fairly active projects have been ported (which probably also tend to be popular, either because of their activity or their activity is because of their popularity).</p>
<h3 id="new-projects_3">New projects <a class="head_anchor" href="#new-projects_3">#</a>
</h3>
<p>And finally, the last way to slice the PyPI data is to look at new projects. Way back when Python 3.0 was released, Guido said that he hoped a majority of new projects would be using Python 3 within 5 years. It’s now over 6 years since Python 3.0’s release in December 2008 (which based on how I’m calculating ratios means hitting 100%). How has that hope panned out?</p>
<p>Let’s first look at projects created over the past year:</p>
<ul>
<li>Unknown: 8,242</li>
<li>Python 2: 2,589</li>
<li>Python 3: 5,050</li>
</ul>
<p>That’s over 46%. It should also be mentioned that since there is no historical data to compare against this is entirely based on the oldest release for a project. If a project deletes old releases – which would be bad as it would break people using older versions for some reason – then that could cause them to look newer than they actually are (you also used to be able to re-upload a file under the same release number, but PyPI no longer supports that). But using this definition of project creation does allow for calculating the rate throughout the past which may be a nice way to graph the adoption rate (it also makes it easier to re-calculate values in case more robust adoption measurements are used, e.g., checking whether any uploaded wheel files support Python 3).</p>
<p>What happens if we limit it to projects created in the past 6 months?</p>
<ul>
<li>Unknown: 4,260</li>
<li>Python 2: 1,401</li>
<li>Python 3: 2,952</li>
</ul>
<p>52%. Is this increasing rate of new projects supporting Python 3? Let’s just consider projects created in the past 30 days (which might be too noisy due to the number of projects under consideration):</p>
<ul>
<li>Unknown: 761</li>
<li>Python 2: 247</li>
<li>Python 3: 534</li>
</ul>
<p>Over 52% still. So at the moment the percentage is close to constant, but we are also diving into numbers that are small enough that little fluctuations will throw things off.</p>
<h2 id="conclusion_2">Conclusion <a class="head_anchor" href="#conclusion_2">#</a>
</h2>
<p>A whole bunch of numbers that vary from 26% to 79% comparing Python 3 to Python 2 (remember: 100% would mean a 50/50 split between the two versions).</p>
<p>Based on all of this, I think there are a couple of statistics we can use to measure Python 3 adoption as a community. For new members of the Python community, the percentage of projects created within the last year seems like a good measure. For long tail acceptance, you can either measure from a set date like the release of Python 3.1.0 or a sliding window like any project that has released in the past e.g., 5 years; I’m not sure which way to measure will be the best nor what values to use to detect long tail adoption (feel free to hit me up on social media if you have any input on this). Major project support is mostly there so I don’t think there is a need to be measuring that separate from the other two measurements. Maybe toss in one or two other metrics – downloads from python.org? – and we should have the metrics suite we want to measure Python 3 adoption as an overall community.</p>
tag:nothingbutsnark.svbtle.com,2014:Post/looking-out-python-usage-in-the-astronomy-community2015-05-15T06:45:49-07:002015-05-15T06:45:49-07:00Looking at Python 3 usage in the astronomy community<p>Thomas Robitaille ran a survey on Python usage in the astronomy community and wrote up <a href="http://astrofrog.github.io/blog/2015/05/09/2015-survey-results/">a great blog post on the results</a> (a big thanks from me to Thomas for taking the time do to this!). The highlights of the survey results are:</p>
<ul>
<li>17% - 20% of respondents are using Python 3 depending on how you want to count</li>
<li>No one really uses any version other than Python 2.7 and 3.4</li>
<li>Windows users are better about using Python 3 than Linux and OS X users</li>
<li>Seasoned Python programmers are using Python 3 more than new users</li>
<li>The biggest reason people give for not using Python 3 is lack of incentives</li>
</ul>
<p>Now this data is very much skewed towards astronomers using Python, but I still think it’s worth looking at the results as a microcosm of the general community.</p>
<h1 id="the-amount-of-python-3-adoption_1">The amount of Python 3 adoption <a class="head_anchor" href="#the-amount-of-python-3-adoption_1">#</a>
</h1>
<p>I have seen <a href="https://caremad.io/2015/04/a-year-of-pypi-downloads/">PyPI numbers that provide a lower bound of 5% adoption</a> across the community, while I have seen other numbers like the one Thomas found where in parts of the Python community that have had their popular libraries and frameworks ported the uptake is closer to 20% (I have primarily seen these kind of adoption numbers for the scientific and web communities). And while I would like to see the overall number be higher than 5% (which might be a little low due to how the PyPI numbers are calculated), I’m quite happy with the 20% penetration in communities who have decided to embrace Python 3.</p>
<h1 id="you-should-probably-drop-support-for-python-2_1">You should probably drop support for Python 2.6 and 3.3 <a class="head_anchor" href="#you-should-probably-drop-support-for-python-2_1">#</a>
</h1>
<p>If you want an idea of how much skew there might be in the PyPI numbers due to caching and what version of Python the initial project fetch was done for, look at the fact that the PyPI numbers suggest that just under 30% of the community overall is on Python 2.6. Anecdotally from people I have talked to at PyCon suggests that number is a bit high, and Thomas’ survey shows that at least in astronomy circles it’s <strong>very</strong> high. This year I have begun to advocate that people drop Python 2.6 support as it was <a href="https://www.python.org/download/releases/2.6/">released on October 1, 2008</a> which is over 7 years ago and thus past the 5 year support period for security fixes from the Python development team. Python 2.7 also has nearly 5 years of bugfixes on Python 2.6 since 2.6.6’s release on August 8, 2010. That means Python 2.7 has significantly less bugs, making it not only nicer to use but also easier to port to Python 3.</p>
<p>As for Python 3.3, the percentage of users from an absolute perspective is just too small. Both the PyPI numbers and Thomas’ numbers suggest only about 1% of people use Python 3.3 directly. And with Python 3.5 due out in a few months you will want to only be supporting Python 3.4 and 3.5 anyway.</p>
<h1 id="windows-users-are-the-most-uptodate_1">Windows users are the most up-to-date <a class="head_anchor" href="#windows-users-are-the-most-uptodate_1">#</a>
</h1>
<p>Thomas’ survey showed that Windows users are using Python 3 at a rate of about 2x to 3x the rate of Linux and OS X users, respectively. I bet this is a side-effect of Windows users having to install Python themselves, and thus grabbing the newest release. The implication of this is people do typically use the version of Python that comes installed on their system.</p>
<p>For OS X 10.10 users, they get Python 2.4, 2.6, and 2.7 installed by default. Now I have no clue when Apple will include Python 3 by default with OS X, so who knows how long the lack of Python 3 coming with OS X could go on for.</p>
<p>But for Linux users, this will start changing in 2016. <a href="http://www.phoronix.com/scan.php?page=news_item&px=Ubuntu-16.04-Python-Plans">Ubuntu plans to only ship Python 3 in 16.04</a> which also happens to be an LTS release. <a href="https://fedoraproject.org/wiki/F22_Beta_release_announcement#Roadmap">Fedora plans to only ship Python 3 in Fedora 23</a>. That means the two most popular Linux distributions are going to make Python 3 what you get automatically and force you to install Python 2 manually if you want it in 2016.</p>
<h1 id="getting-new-users-into-python-3_1">Getting new users into Python 3 <a class="head_anchor" href="#getting-new-users-into-python-3_1">#</a>
</h1>
<p>Thomas’ data suggests that the people using Python 3 are not the people new to Python, but more seasoned users who know what Python 3 gets you. I suspect part of this ties into the system installation of Python not being Python 3 yet, and for some they just use whatever is already installed.</p>
<p>But for others it’s possible they are being taught Python 2 over 3. In that instance I think it’s a mistake. In my experience it’s easier to teach Python 3 since it’s a more consistent language and then teach exceptions to the rules for Python 2 use. Going the other way from Python 2 to learn what rough edges have been softened in Python 3 typically isn’t as smooth for the new user.</p>
<h1 id="incentives-for-switching-to-python-3_1">Incentives for switching to Python 3 <a class="head_anchor" href="#incentives-for-switching-to-python-3_1">#</a>
</h1>
<p>Communicating the benefits of Python 3 has always been difficult if you don’t use Unicode. I have always said that once you have used Python 3 and had the sharp corners of Python 2 removed you simply won’t want to use Python 2 unless you absolutely have to. Literally <strong>everyone</strong> I have ever talked to who gets to write code in pure Python 3 agrees with me; once you have fully switched to Python 3 you simply don’t want to go back to Python 2.</p>
<p>Unfortunately because this is an instance of a lot of little things adding up to an overall improvement it’s hard to point to any one single feature that makes people go, “ooh, I really want that” which causes them to make the switch. It’s hard to say that keyword-only arguments, <code class="prettyprint">__pycache__</code> directories, or enhanced exceptions are good enough on their own to cause people to make the switch compared to Python 2. This is especially true for things from the stdlib which people have backported to Python 2, thus minimizing what is exclusive to Python 3 to only that which is based on syntax.</p>
<p>But features unique to Python 3 which can’t be backported and can be motivation on their own started in Python 3.4 with <a href="https://docs.python.org/3/library/asyncio.html#module-asyncio"><code class="prettyprint">asyncio</code></a> combined with <a href="https://docs.python.org/3.5/whatsnew/3.3.html#pep-380"><code class="prettyprint">yield from</code></a>. That was the first instance where there was a very clear benefit to a group of people – those writing asynchronous code – where switching to Python 3 would be truly useful, no questions asked (yes, <a href="https://pypi.python.org/pypi/trollius">trollius</a> does exist but <code class="prettyprint">yield from</code> makes <code class="prettyprint">asyncio</code> nicer to work with).</p>
<p>But Python 3.5 is going to bring two big changes which simply won’t work in Python 2. The first is matrix multiplication thanks to <a href="https://www.python.org/dev/peps/pep-0465/">PEP 465</a>. The <a href="http://www.numpy.org/">numpy</a> users will now have a syntax available to them for when they matrix multiply to arrays. This should make a large number of scientists very happy.</p>
<p>The second feature landing in Python 3.5 is <code class="prettyprint">async</code>/<code class="prettyprint">await</code> syntax thanks to <a href="https://www.python.org/dev/peps/pep-0492/">PEP 492</a>. This will essentially do away with the need to use <code class="prettyprint">yield from</code> with <code class="prettyprint">asyncio</code> or any other event loop for asynchronous programming and instead use syntax similar to that which is planned for/used in C#, Dart, ES7, etc. This should lead to a bunch more asynchronous programming in Python since the syntax makes it very explicit and easy to understand asynchronous programming.</p>
<h1 id="i-think-2016-is-going-to-be-a-milestone-in-py_1">I think 2016 is going to be a milestone in Python 3 adoption <a class="head_anchor" href="#i-think-2016-is-going-to-be-a-milestone-in-py_1">#</a>
</h1>
<p>I think the confluence of Linux distributions switching over to Python 3, syntactic features in Python 3.5 that won’t be available in Python 2, all of the tooling now available to port to Python 3 (<a href="https://docs.python.org/3/howto/pyporting.html">HOWTO</a> and <a href="https://www.youtube.com/watch?v=KPzDX5TX5HE">PyCon presentation</a>), the end-of-life for Python 2.7 being less than 5 years away, and just simply time passing is going to lead to a nice bump in Python 3 adoption during 2016 that will hopefully be sustained going forward.</p>
tag:nothingbutsnark.svbtle.com,2014:Post/porting-to-python-3-is-like-eating-your-vegetables2015-04-23T12:14:19-07:002015-04-23T12:14:19-07:00Porting to Python 3 is like "eating your vegetables"<p>While at PyCon this year, someone pointed out that the hope/goal/expectation when Python 3 was released was to have over 50% of new projects using Python 3 within five years, and that had not come to fruition (while I have not personally crunched the numbers and this was told to me anecdotally, I will just assume they are right). Immediately after pointing this out they said that Python 3 is thus a failure at this point and we need to start figuring out when we call it quits on Python 3. I obviously disagree with this view.</p>
<h1 id="we-can39t-predict-the-future_1">We can’t predict the future <a class="head_anchor" href="#we-can39t-predict-the-future_1">#</a>
</h1>
<p>Just because some prediction didn’t come true does not mean we have lost hope in Python 3. All it means is that the community’s uptake of Python 3 has been slower than we initially thought, and honestly that’s fine. The extension of Python 2.7’s end-of-life to 2020 is an acknowledgment of this slower pace. There are plenty of companies and projects which have switched to Python 3 fully or are at least supporting it and seem happy with Python 3. The polling of the audience by Guido during his keynote along with some uptake numbers shared in confidence with the core developers shows that the level of use is actually too high to consider abandoning Python 3 without screwing over a sizable chunk of the community (Donald Stufft’s post on <a href="https://caremad.io/2015/04/a-year-of-pypi-downloads/">PyPI download statistics</a> shows this).</p>
<p>So why are we as a community, more than 6 years after Python 3 was released, not transitioning to Python 3 faster? At the language summit, Larry Hastings said porting to Python 3 is like “eating your vegetables”; you know you should, but that doesn’t make it enjoyable. I think that is a very fair analogy as to why most people don’t finding porting fun.</p>
<h1 id="why-the-hell-we-separated-text-and-binary-dat_1">Why the hell we separated text and binary data in the language <a class="head_anchor" href="#why-the-hell-we-separated-text-and-binary-dat_1">#</a>
</h1>
<p>I have a sneaking suspicion that some people think we decided to clearly separate text and binary data “just because” or it seemed like the proper thing to do from some theoretical language design standpoint or something; all of those reasons are wrong. The reason we made the separation was to provide language level support/restrictions to prevent people from making subtle bugs when they mixed text and binary data. Talk to a Django core developer and ask them how much code and work it is to properly handle the separation of text and binary data in Python 2 and I’m sure you will be told it’s a lot and it’s a pain. I think Jacob Kaplan-Moss tells me almost annually at PyCon that the day Django stops supporting Python 2 they will be able to rip out a ton of code that exists purely because it was so easy to mix text and binary data and get it wrong. In other words <a href="https://youtu.be/hIJdFxYlEKE">most of us are mediocre programmers</a> and this text/binary separation helps us not to screw up (also making sure code works with Unicode for text is also a benefit of this separation).</p>
<p>This means dealing with the text/binary separation in Python 3 is part of eating your vegetables. It was done to help people in the long term. People also seem to forget that the core developers have to eat the exact same vegetables as everyone else. The core developers are not some magical group of people who get to code in Python 3 all day long; plenty of us core developers still work with Python 2 as part of our job. That means decisions like the text/binary separation are not taken lightly as it impacts <em>our</em> lives just like everyone else’s.</p>
<h1 id="how-to-make-eating-your-vegetables-easier_1">How to make eating your vegetables easier <a class="head_anchor" href="#how-to-make-eating-your-vegetables-easier_1">#</a>
</h1>
<p>So you should eat your vegetables, but that doesn’t mean we can’t try to make it easier. I gave a talk at PyCon 2015 on <a href="https://youtu.be/KPzDX5TX5HE">making Python 2 code be 2/3 compatible</a>. You should be able to get all the same information from the <a href="https://docs.python.org/3/howto/pyporting.html">Python 2/3 porting HOWTO</a>. The tooling and such should actually be to the point that if you run your code against Python 2.7 and Python 3.5 (when it’s released) that any porting issue shouldn’t be silent but instead raise a warning, exception, message, etc. In other words the only bit that should require a good amount of thinking is the text/binary separation for your APIs and making that separation happen.</p>
<p>You can also view the various steps to support Python 3 as part of your code’s health if that helps motivate you (or your manager). Since the changes in Python 3 are to make your code better, you could argue that modernizing your code and cleaning it up in Python 2 has a side-effect of supporting Python 3. And much like code maintenance, you can make porting a piecemeal process where you don’t have to do all of it at once but in pieces to gain different things (e.g., moving to iterator versions of <code class="prettyprint">map</code> and other built-ins can lead to better performance and more use of iterators which just happens to align with what Python 3’s built-ins do).</p>
<p>But please don’t assume porting is free. It can be fairly straightforward for some projects, but harder for others. Either way there is <em>some</em> work involved. There have been reports of people in the community being somewhat mean to project maintainers for not porting which isn’t right. You can always ask someone to port their code, but you should always be cordial about it and be understanding if they choose not to.</p>
<h1 id="getting-treats-for-eating-your-vegetables_1">Getting treats for eating your vegetables <a class="head_anchor" href="#getting-treats-for-eating-your-vegetables_1">#</a>
</h1>
<p>Like many of you, when I was younger and I didn’t want to eat my vegetables my parents would tell me that if I did eat my vegetables I would get dessert. The Python development team is constantly trying to toss in more desserts into Python 3.</p>
<p>For some it may be <a href="https://docs.python.org/3/library/asyncio.html#module-asyncio">asyncio</a> that they get to look forward to when they port (and I’m willing to bet the first person to create a library which really harnesses asyncio is going to have a popular project on their hands). Others may be looking forward to <a href="https://www.python.org/dev/peps/pep-0484/">type hints</a> inlined in their code (for Python 2 you can put the type hints in stub files). Assuming things go the way I think they will, the <a href="https://www.python.org/dev/peps/pep-0492/"><code class="prettyprint">async</code>/<code class="prettyprint">await</code> syntax for co-routines</a> will be landing in Python 3.5 and that seems to be exciting people. For others it might be performance improvements that we have landed over the years which help with things like memory usage from strings, etc. But the point is that we are always trying to make your effort in porting to Python 3 worth that much more.</p>
<h1 id="we-are-one-community_1">We are one community <a class="head_anchor" href="#we-are-one-community_1">#</a>
</h1>
<p>Regardless of what version of Python you run, we are still a single community. While I think Python 3.4 is currently the best version of Python available, you’re welcome to disagree with me or agree but not think it’s worth your time and effort to use it; all I ask is you be informed in your opinion. Honestly, if some people never switch to Python 3 and make Python 2 the COBOL of dynamic programming languages I’m totally fine with it (just as long as you don’t make me maintain it past 2020 ☺). The one thing I really want to make sure never goes away is how awesome the Python community is.</p>
tag:nothingbutsnark.svbtle.com,2014:Post/going-allin-on-the-mobile-web2015-03-06T10:44:03-08:002015-03-06T10:44:03-08:00Going all-in on the mobile web<p>In this world of Android vs. iOS – with a smattering of Windows Mobile and Blackberry – I find native apps somewhat annoying. In the beginning, <a href="http://www.cultofmac.com/125180/steve-jobs-was-originally-dead-set-against-third-party-apps-for-the-iphone/">iOS was actually not going have any third-party apps on the phone</a> and everything would run through Safari on your iPhone. Developer backlash due to worries about performance, though, helped lead to Apple changing its position on native apps on iOS.</p>
<p>I think this is a shame. I appreciate that web apps embody <strong>SLICE</strong> (<strong>S</strong>ecure, <strong>L</strong>inkable, <strong>I</strong>ndexable, <strong>C</strong>omposable, <strong>E</strong>phemeral, and I have heard it suggested that Updatable get tossed in). I also appreciate that the web embodies a common denominator platform that is cross-platform (I’m not naïve enough to think the entire web platform is cross-platform thanks to differences in what APIs are implemented at any given time by the various browsers). Why do developers need to choose whether to launch on iOS or Android first or exclusively on one platform? Why can’t developers simply launch simultaneously and instantaneously on all platforms through the web?</p>
<p>The answer is that many can, but they choose not to for various reasons (some legitimate, some not).</p>
<h2 id="what-leads-to-requiring-a-native-app_2">What leads to requiring a native app? <a class="head_anchor" href="#what-leads-to-requiring-a-native-app_2">#</a>
</h2>
<p>What has traditionally set native apps apart from web apps on mobile phones? I would argue it’s the following:</p>
<ol>
<li>Performance (varies between browser releases and phone generations, but low-level stuff like controlling socket connections outside of <a href="https://developer.mozilla.org/en/docs/WebSockets">WebSockets</a> or SIMD for GPU calculations isn’t possible)</li>
<li>Offline access (being solved thanks to <a href="http://www.w3.org/TR/service-workers/">Service Workers</a> as <a href="http://alistapart.com/article/application-cache-is-a-douchebag">AppCache is not pleasant to work with</a>; actual <a href="https://developers.google.com/chrome/whitepapers/storage">storage space can also be an issue</a>)</li>
<li>Periodic background processing (Service Workers could do it if the <a href="http://www.w3.org/TR/task-scheduler/">task scheduler API</a> gets accepted)</li>
<li>Notifications (<a href="http://www.w3.org/TR/notifications/">w3c spec for text-based notifications</a>, plus there is a new <a href="https://w3c.github.io/push-api/">Push API</a> for Service Workers to work in the background to push notifications to the user)</li>
<li>Sensors (stuff like geolocation are available, some other things are not)</li>
<li>OS-specific features (e.g., intents on Android)</li>
</ol>
<p>As you can see, a good amount of features have either just landed in browsers – <a href="http://blog.chromium.org/2014/12/chrome-40-beta-powerful-offline-and.html">Service Workers arrived in Chrome 40</a> – or are coming very soon – the <a href="https://www.chromestatus.com/feature/5416033485586432">Push API for Service Workers is coming in Chrome 42</a>. Unfortunately not everything is actively scheduled to land in a browser – scheduling a Service Worker to run isn’t planned for any browser yet – and OS-specific features like intents are typically off the table since they are not OS-agnostic and thus won’t work in a browser no matter what OS it’s running on. In terms of raw performance, it’s a constantly fluctuating thing that’s always being discussed, e.g. <a href="https://hacks.mozilla.org/2014/10/introducing-simd-js/">Mozilla, Google, and Intel looking into SIMD in JavaScript</a>. In other words claiming the browser is “slow” doesn’t hold a-priori.</p>
<h2 id="making-myself-a-guinea-pig_2">Making myself a guinea pig <a class="head_anchor" href="#making-myself-a-guinea-pig_2">#</a>
</h2>
<p>How many apps do people use that lack a mobile web version but which actually could get away with having one (either full-featured or with some degraded UX)? Taking stock of what I have on the homescreen of my Android phone, I have the following list of applications grouped by category which I use almost daily and looked at whether they could have a web app experience of some form that was still useful. Anything in <strong>italic</strong> means that a web experience of some sort is possible today, and if something is <strong>bold</strong> then someone actually implemented a mobile-friendly browser experience.</p>
<ul>
<li>Alarm (<a href="https://play.google.com/store/apps/details?id=ch.bitspin.timely">Timely</a>): not possible as a web app due to lack of background scheduling and the ability to force a web page to appear to play some alarm constantly</li>
<li>
<em>Podcast player</em> (<a href="https://play.google.com/store/apps/details?id=au.com.shiftyjelly.pocketcasts">Pocket Casts</a>): they have a <a href="https://play.pocketcasts.com">web app</a>, but it isn’t mobile-optimized</li>
<li>
<em>Navigation</em> (<a href="https://play.google.com/store/apps/details?id=com.waze">Waze</a>): would require keeping the app upfront, but there is no reason a navigation app couldn’t exist through a browser</li>
<li>
<strong>Email</strong> (<a href="https://play.google.com/store/apps/details?id=com.google.android.apps.inbox">Inbox</a>): Inbox doesn’t have a mobile web app, but <a href="https://gmail.com">Gmail</a> does</li>
<li>
<strong>Local search</strong> (<a href="https://play.google.com/store/apps/details?id=com.google.android.apps.maps">Maps</a>): <a href="https://maps.google.com">Maps</a> has a full-featured mobile experience</li>
<li>
<strong>Messaging</strong> (<a href="https://play.google.com/store/apps/details?id=com.google.android.talk">Hangouts</a>, <a href="https://play.google.com/store/apps/details?id=com.whatsapp">WhatsApp</a>, <a href="https://play.google.com/store/apps/details?id=cordproject.cord">Cord</a>, and <a href="https://play.google.com/store/apps/details?id=com.facebook.orca">Facebook Messenger</a>): a degraded experience until the Push API is available is possible when the web app is running upfront; <a href="https://www.facebook.com">Facebook</a> has Messenger as part of their mobile web app while Hangouts does not and the way WhatsApp is designed simply won’t allow a web experience that doesn’t require the native app without better storage guarantees in the browser</li>
<li>
<em>Phone calls</em> (<a href="https://play.google.com/store/apps/details?id=com.google.android.apps.hangoutsdialer">Hangouts Dialer</a>): WebRTC shows that making phone calls doesn’t require anything special from the browser on your phone</li>
<li>
<em>Check-in</em> (<a href="https://play.google.com/store/apps/details?id=com.foursquare.robin">Swarm</a>): A degraded experience that doesn’t tell you about when your friends are nearby is totally doable in the browser</li>
<li>
<em>Learning a language</em> (<a href="https://play.google.com/store/apps/details?id=com.duolingo">Duolingo</a>): already <a href="http://www.duolingo.com">works on the desktop</a>, so no reason not to work on mobile</li>
<li>
<strong>Lists</strong> (<a href="https://play.google.com/store/apps/details?id=com.google.android.keep">Keep</a>): <a href="https://keep.google.com">Keep</a> has a mobile web app</li>
<li>
<em>Membership cards</em> (<a href="https://play.google.com/store/apps/details?id=de.stocard.stocard">Stocard</a>): Geofencing would let you get deals pushed to you, but scanning barcodes in a browser and showing them later is totally doable</li>
<li>
<em>Streaming</em> (<a href="https://play.google.com/store/apps/details?id=com.netflix.mediaclient">Netflix</a> and <a href="https://play.google.com/store/apps/details?id=com.google.android.videos">Play Movies</a>): <a href="http://netflix.com">Netflix</a> is obviously on the desktop, so as long as mobile browsers have the DRM support necessary it should work in mobile browsers</li>
<li>
<strong>Movie/TV info</strong> (<a href="https://play.google.com/store/apps/details?id=com.imdb.mobile">IMDb</a> and <a href="https://play.google.com/store/apps/details?id=com.battlelancer.seriesguide">Series Guide</a>): <a href="http://imdb.com">IMDb</a> works fine on mobile – albeit with some missing features, like custom lists – and Series Guide can store its data in <a href="http://trakt.tv">Trakt</a> which has a mobile web site</li>
<li>
<em>Music</em> (<a href="https://play.google.com/store/apps/details?id=com.google.android.music">Play Music</a> and <a href="https://play.google.com/store/apps/details?id=com.sonos.acr">Sonos</a>): Sonos requires special socket access so that’s not doable, but <a href="https://music.google.com">Play Music</a> could totally be done in the browser</li>
<li>
<strong>News</strong> (<a href="https://play.google.com/store/apps/details?id=com.google.android.apps.genie.geniewidget">News & Weather</a>): <a href="https://news.google.com">Google News</a> has a mobile website</li>
<li>
<strong>Social media</strong> (<a href="https://play.google.com/store/apps/details?id=com.google.android.apps.plus">Google+</a> and <a href="https://play.google.com/store/apps/details?id=com.facebook.katana">Facebook</a>): <a href="https://plus.google.com">Google+</a> and <a href="https://www.facebook.com">Facebook</a> both have good mobile websites</li>
<li>
<strong>Feeds</strong> (<a href="https://play.google.com/store/apps/details?id=com.noinnion.android.greader.readerpro">gReader Pro</a> backed by <a href="https://feedly.com">Feedly</a>): <a href="http://feedly.com">Feedly</a> lacks a mobile site, but <a href="https://theoldreader.com">The Old Reader</a> does not and gReader can read from either as a backend service.</li>
<li>
<strong>Read later</strong> (<a href="https://play.google.com/store/apps/details?id=com.ideashower.readitlater.pro">Pocket</a>): <a href="https://getpocket.com">Pocket</a> has a desktop site, there is no mobile site; <a href="http://instapaper.com">Instapaper</a> does have one, though</li>
<li>
<strong>Calendar</strong> (<a href="https://play.google.com/store/apps/details?id=com.google.android.calendar">Calendar</a>): might not be able to raise event reminders, but <a href="https://calendar.google.com">Google Calendar</a> is available on mobile</li>
<li>
<strong>Passwords</strong> (<a href="https://play.google.com/store/apps/details?id=com.github.brettcannon.oplop">Oplop</a> and <a href="https://play.google.com/store/apps/details?id=com.dashlane">Dashlane</a>): <a href="https://oplop.appspot.com">Oplop</a> has been mobile-friendly for years while Dashlane is not (nor is LastPass)</li>
</ul>
<p>Out of 20 categories, 19 could have a useful web experience but only 11 do (and 3 of them would require changing who provided me the service to get a web experience). I would be really curious to see a study done that evaluated if doing a mobile web app for Android and iOS – or even just one of the platforms – led to more or less work than doing a native app. But my point remains that just because someone provides a native app doesn’t mean a web app wouldn’t also work for the same use-case.</p>
tag:nothingbutsnark.svbtle.com,2014:Post/alternative-voting-systems-for-canada2015-01-26T14:56:56-08:002015-01-26T14:56:56-08:00Alternative voting systems for Canada<p>With 2015 bringing a federal election where <a href="http://www.threehundredeight.com/p/canada.html">current polling</a> suggests that either the Liberals or Conservatives could have a minority government based on how people vote, I’m rather interested in how election systems could influence the outcome. Last month there was an <a href="http://www.threehundredeight.com/2014/12/the-seat-implications-of-electoral.html">article about two alternative voting systems</a> and how they would influence the elections. While reading into these systems I also found out that voting system reform is not a new thing in Canada and that both the NDP and Liberals have actually discussed changing how federal elections work after the 2015 election. With the importance of this election being so high – the Conservatives have been the ruling party since 2006 with a majority since 2011 and so it is expected they will at minimum lose their majority stance – I decided to try and learn what alternative voting systems are being proposed and how they would influence the political landscape in Canada.</p>
<h1 id="quick-primer-on-canada39s-federal-government_1">Quick primer on Canada’s federal government <a class="head_anchor" href="#quick-primer-on-canada39s-federal-government_1">#</a>
</h1>
<p>Canada’s government is based on the <a href="http://en.wikipedia.org/wiki/Westminster_system">Westminster system</a> where there is Parliament made up of representatives elected by the public. Each member of Parliament – known as an MP – is elected by <a href="http://www.elections.ca/content.aspx?section=res&dir=cir/list&document=index&lang=e">a riding</a> of which the largest is just over 180,000 people but hovers more around 100,000 nationally (which I have been told is one of the best ratios of people to representative in the world). The political party with the most number of representatives appoint the prime minister – who is always the party leader – who is known at the PM. Federal elections are called whenever <a href="http://en.wikipedia.org/wiki/Dissolution_of_parliament#Canada">Parliament is dissolved</a> which typically occurs when a majority of MPs vote for elections to occur.</p>
<p>One key thing to realize about this kind of system is that whenever a party has a majority government – which cannot last longer than 5 years by law and what Canada currently has – they can basically pass any law they want as long as the <a href="http://en.wikipedia.org/wiki/Senate_of_Canada">Senate</a> signs off on it (which they almost always do). The only check on Parliament then comes from the Queen of England and the <a href="http://www.gg.ca/">Governor General</a> who is the Queen’s representative in Canada. Either of these two people can dissolve Parliament and are the only people who can do so (when Parliament votes to dissolve Parliament it’s just a suggestion to the Governor General).</p>
<p>We also have five national parties: Conservatives, Liberals, NDP, Greens, and Bloc Quebecois. The first three are the big ones while the Greens are fairly new and only have 2 MPs and the Bloc is exclusively from Quebec and only have 2 MPs. In other words fairness in voting is a big deal here since voting is not a binary choice.</p>
<h1 id="firstpastthepost-voting-what-the-conservative_1">First-past-the-post voting (what the Conservatives want and what the current system is) <a class="head_anchor" href="#firstpastthepost-voting-what-the-conservative_1">#</a>
</h1>
<p>The voting system currently used in Canada is called <a href="http://en.wikipedia.org/wiki/First-past-the-post_voting">first-past-the-post</a> (abbreviated FPTP). In this system you get a single vote and you cast it for the person who you want to win. The person with the largest number of votes wins the election. Very simple.</p>
<p>Unfortunately it also leads to disproportionate representation. Let’s pretend we have a total of 3 MPs named Brett, Andrea and Gidget. Each MP represents a riding made up of 10 people. Brett is a Liberal and gets elected by 6 votes with the other 4 going to the Greens. Andrea is also elected by a 6-4 vote along Liberal/Greens party lines. Gidget, though, gets all 10 votes in her riding and she is a Green representative. So while Parliament would be split 2-1 Liberal/Greens, the 30 voters actually voted 12-18 Liberal/Greens. In other words while people as a region voted more for Greens than Liberals, because of the way first-past-the-post works the Liberals are actually the party in power.</p>
<p>This example shows why sacrificial voting is caused by first-past-the-post. In our example, it obviously is not worth voting for the Greens if you know they won’t win your riding, regardless of whether you think they are the best party to run Canada. And so people end up voting for the best party they think can <em>win</em> instead of the best party <em>period</em>. While a two-party system like they have in the United States doesn’t really need to worry about this so much, in Canada where there are five officially recognized political parties at the federal level, sacrificial voting is definitely something one has to take into consideration, e.g. I might prefer the Greens but if I would rather the Liberals win out in a close race against the Conservatives I would be compelled to vote for the Liberals to help make sure the least bad result which actually has a chance of occurring.</p>
<h1 id="alternative-voting-what-the-liberals-want_1">Alternative voting (what the Liberals want) <a class="head_anchor" href="#alternative-voting-what-the-liberals-want_1">#</a>
</h1>
<p>Also known as <a href="http://en.wikipedia.org/wiki/Instant-runoff_voting">instant-runoff voting</a>, alternative voting – abbreviated AV – is a voting system where you rank the candidates in order of preference. Once all votes are cast each ballot counts as a single vote for the person’s top choice. If no one got a majority of votes, then the candidate with the fewest votes is eliminated and then the ballots are counted again. In this next round, though, any ballot that cast the now-eliminated candidate as their primary choice shift to having their second pick count as their vote. This process of elimination and shifting who gets the vote from a ballot continues until someone gets a majority of votes. Or another way of looking at it is imagine everyone casts their votes for their top choice, if there is no clear winner then the least popular person is eliminated, and then there is an instant-runoff where everyone votes again with one less person to consider (this is where the name of the system comes from).</p>
<p>In Canada, AV is used to elect the leaders of the Liberal and Conservative parties. It’s also used in Australia to elect members of their House of Representatives.</p>
<p>AV does a better job of proportional representation than FPTP which is obviously good. It also makes key ridings less important as you may not win a riding simply by having the most #1 votes if it’s less than a majority while someone else has a majority of #2 votes.</p>
<p>FPTP is not flawless though, as it does lead to collusion between parties in telling people to vote specific parties lower than another one. The idea is that I will tell my constituents to put you second if you tell your constituents to put me second. So there is room to influence votes with AV.</p>
<h1 id="mixedmember-proportional-representation-what_1">Mixed-member proportional representation (what the NDP and Greens want) <a class="head_anchor" href="#mixedmember-proportional-representation-what_1">#</a>
</h1>
<p>Abbreviated MMP, <a href="http://en.wikipedia.org/wiki/Mixed-member_proportional_representation">mixed-member proportional representation</a> has a somewhat interesting history in Canada. Back in 2004, the Law Commission of Canada – de-funded since 2011 – released a report <a href="http://www.elections.ca/res/eim/article_search/article.asp?id=129&lang=e">suggesting MMP</a> would be a good voting system for Canada (<a href="http://publications.gc.ca/collections/Collection/J31-61-2004E.pdf">PDF</a>; relevant discussion of MMP is in Section 4.4 starting on page 83 of the printed text, page 105 of the PDF). So what is MMP and why was it suggested for Canada (and is already used in New Zealand and Germany among others for their federal elections)?</p>
<p>In MMP you vote for <strong>two</strong> separate things. First you vote for the party you want to lead the federal government. Second, you vote for the person you want to represent you. In both FPTP and AV this is one and the same vote which means if you don’t like the person running in your riding but they represent the party you want to run the country then you have to decide which is more important to you. But in MMP who you think can represent your riding the best is not conflated with what political party should be setting the agenda in Ottawa. Think of it as you’re voting for the party you want running the federal government and you’re separately suggesting the individual you want to personally represent you regardless of what party they represent.</p>
<p>The other interesting aspect of MMP is how these two seemingly separate votes consolidate with each other so that you end up with a representative government that is proportional to what your region wants. For each riding, the representative MP that you directly voted for wins based on first-past-the-post. These elected riding MPs make up 2/3 of all MPs representing a region (which can be as big as a province). The remaining 1/3 of MPs are filled based on how many MPs a party would need to top up their MP count to more accurate represent the party vote in the region.</p>
<p>These “filler” MPs are called list MPs because they are essentially pulled from a list provided by the party (the Law Commission of Canada suggested letting people either vote for specific list MPs that they preferred or simply vote for a party and let that party choose which list MP would get that vote, that way people can choose how involved they want to be in choosing their list MPs; pulling from the list exclusively is called closed list MPs, always voting for the list MP is called open list, and allowing either is called mixed list).</p>
<p>Consider a region with 10 MPs: 6 riding MPs and 4 list MPs. Let’s also say that the Liberals got 50% of the party vote along with 3 riding MPs elected. That would mean the Liberals get 2 list MPs since the region said they should make up 50% of the government but fell short in the riding MP votes by 2 representatives. Notice how voting for a riding MP does not mean you can vote for your party twice; if you voted in a Liberal MP <strong>and</strong> voted for the Liberal party you don’t get to make 2 MPs suddenly appear for the Liberals, just that you got to help select one of the Liberal MPs directly because you liked them personally. A good example of how this whole thing works can be found in the <a href="http://publications.gc.ca/collections/Collection/J31-61-2004E.pdf">Law Commission of Canada’s findings</a> on printed page 94.</p>
<p>A huge effect of MMP is that minority parties can get much larger representation in government. Since your party vote directly correlates to proportional representation for your region it means you can vote for e.g., the Greens knowing that your votes actually counts towards them representing you in Parliament. This more favourable representation is why the <a href="http://www.ndp.ca/news/ndp-wants-to-implement-mixed-proportional-system">NDP actually brought a vote on MMP</a> this past December and have a <a href="http://petition.ndp.ca/demand-that-your-vote-count">petition</a> to try and change the voting system in Canada after the 2015 election.</p>
<p>MMP is not perfect, though. Your vote can still not count the same as someone in another region based on how the number of MPs a region represents is divided. In some places that use MMP they solve this by having the list MPs be filled in nationally so that everyone’s vote is equal (this wouldn’t work in Canada for historical reasons, but basically certain provinces and all territories have minimum representation guarantees in Parliament and so having overflow representation would break those guarantees). But compared to both FPTP and AV, MMP usually has the better representation of voter intent.</p>
<p>The other issue is the size of the ballot. With a mixed list vote you would have a ballot listing all of the riding MPs you have to choose from and then you would have the mixed list vote where you either vote for a party generally or vote for a list MP specifically which could be as long as 15 people. I have looked at the example ballot and it isn’t complex, but it definitely isn’t small if you have up to 5 parties listing 15 people each for their listing MP votes (this is more of a technicality and is a solvable problem, but it is something to consider).</p>
<h1 id="my-personal-opinion_1">My personal opinion <a class="head_anchor" href="#my-personal-opinion_1">#</a>
</h1>
<p>After reading about all of these voting systems, I have to admit that MMP appeals to me the most. It comes the closest to making sure my vote for a party actually leads to representation by that party in Parliament. I also appreciate that my personal representation in Parliament by an MP is a separate vote and does not impact which party I want to represent me.</p>
<p>It also has the side-effect that majority governments would become a rarity. With small parties able to win more seats it leads to more compromise and teamwork between parties in order to get legislation passed. MMP just seems the most fair, and as a Canadian I like being fair.</p>
tag:nothingbutsnark.svbtle.com,2014:Post/why-whatsapp-made-its-web-app-use-your-phone-as-a-server2015-01-23T09:27:18-08:002015-01-23T09:27:18-08:00Why WhatsApp made its web app use your phone as a server<p>Ever since WhatsApp announced their web app, I have seen various people complain about having to keep your phone on to send messages. But in all of these posts people seem to be overlooking two key points about the design of WhatsApp which either necessitate this design or at least facilitate WhatsApp in keeping their service lean and fast.</p>
<h1 id="encryption_1">Encryption <a class="head_anchor" href="#encryption_1">#</a>
</h1>
<p>In case you didn’t hear, <a href="https://whispersystems.org/blog/whatsapp/">WhatsApp (supposedly) does end-to-end encryption</a>. That means WhatsApp can’t read what people say in their messages. What this also means, though, is that if someone sends you a message and you have your phone and web app, there is no way to read that message independently of both devices without sharing the decryption key, which is not good security practice. You could have separate keys for your phone and web browser, but then you would then need to make sure to share keys between your phone, your browser, and all of your contacts so they could receive messages from any of your devices.</p>
<p>But if you treat your phone as your personal WhatsApp server, then your phone can continue to have the master keys for your account and then you only need to manage keys between your phone and your other devices, making it a hub-and-spoke system where you phone is the hub and your other devices are the spokes. This keeps WhatsApp out of the key management business and allows you to easily revoke keys for your other devices without WhatsApp having to store anything for you. So tying everything through your phone keeps everything encrypted and simplifies key sharing between users to just their phones and keeps security simple which is how you want it to be.</p>
<h1 id="storage_1">Storage <a class="head_anchor" href="#storage_1">#</a>
</h1>
<p>Someone pointed out to me that WhatsApp is not in the storage business, and when you stop and look at how the service was structured before the web app came along you will realize that WhatsApp tried to minimize what data it had to store from its inception. For instance, if you get a new phone you actually have to <a href="http://www.whatsapp.com/faq/en/general/21002842">migrate your messages over <strong>yourself</strong></a>. Since WhatsApp doesn’t keep messages on their servers, your phone ends up being the keeper of truth when it comes to what messages you sent and received.</p>
<p>But the key thing about WhatsApp not storing messages on their servers is how much it simplifies their service. Consider their last <a href="http://blog.whatsapp.com/613/500000000">publicly stated user count of 500,000,000</a>. Since WhatsApp doesn’t store messages for you, they really only need to store messages that have yet to be delivered to a user’s phone and your account’s configuration data. So let’s assume <strong>every</strong> user suddenly sent a bunch of photos that came to a total of 1 MB of pending messages (remember that WhatsApp is only going to show you a small version of a photo and so they can compress them such that they don’t take up much space; 1 MB should go a long way). That’s 1 MB * 500,000,000 = 500 TB of storage. OK, not a puny number for most services.</p>
<p>But let’s look at this from a cloud perspective. Let’s say you wanted an extremely fast service, so you would want to use <a href="https://cloud.google.com/compute/docs/local-ssd">local SSD which Google Compute Engine offers</a>. As I write this, a local SSD on GCE is 375 GB and you can have up to 4 per instance. At 375 GB that gets means you would need 1,334 SSDs to store 500 TB of data (it would also require at minimum 334 instances and <a href="http://blog.whatsapp.com/196/1-million-is-so-2011">probably enough for the service to run</a>, but I’m not pricing out computation or bandwidth costs to keep this simple). Now according to GCE’s pricing, a <a href="https://cloud.google.com/compute/pricing#localssdpricing">local SSD costs</a> USD 81.75/month. That means it would cost you USD 81.75 * 1,334 SSDs = USD 109,054.50/month or USD 1,308,654/year. Now let’s be totally extravagant and say you want the data replicated near the sender, near the receiver, and on one other continent for safekeeping until delivery (when the sender and receiver are on the same continent the data could go to two separate clusters). So we are talking about 3N data replication. That works out to USD 3,925,962/year in local SSD storage costs if you used Google Compute Engine at full retail and always maintained this level of storage constantly. In the current startup climate, USD 4 million/year is pittance (and unnecessary as I bet WhatsApp could get away with 2N data replication if that since this is only for buffering purposes). And for a company like Facebook who have their own clusters? Storing 1.5 PB of data would not be difficult at all, so WhatsApp could go as far as backing up the data on every populated continent if they wanted to for 3 PB or less than USD 8 million/year.</p>
<p>In other words by relying on your phone as the storage mechanism for messages and not having to keep anything on their own servers, WhatsApp can run very cheaply and efficiently (and only cost you USD 1/year as a user). This makes WhatsApp basically nothing more than a fast routing service with some buffering between phones, much like the telcos (which might be what inspired WhatsApp to use Erlang). It’s rather ingenious and probably why the service seems so fast and has such great uptime.</p>
<p>And I bet users don’t care about the potential loss of messages either; when was the last time you scrolled back into the history on your phone to a point that preceded you getting that phone or having catastrophic data loss on the device? Plus you have to consider how many users of WhatsApp are really going to use the web app; I bet a large portion of their users don’t even have their own computers beyond their phones so this web app service probably isn’t critical to WhatsApp’s success.</p>
tag:nothingbutsnark.svbtle.com,2014:Post/how-to-get-your-grandma-off-of-skype-using-webrtc2015-01-07T11:50:33-08:002015-01-07T11:50:33-08:00How to get your parents off of Skype using WebRTC<p>My mother-in-law has a cousin she likes to talk to. Because I like to simplify my tech support requirements by standardizing my entire family on a single platform when possible, all of my immediate family uses <a href="https://www.google.com/+/learnmore/hangouts/">Google Hangouts</a> (promoting saving money through <a href="https://support.google.com/hangouts/answer/3187125?hl=en">free phone calls to the US and Canada</a> also helps). Unfortunately my mother-in-law’s cousin is on Yahoo and Skype and is not that tech-savvy. That means I want a video chat solution that is dead-simple and doesn’t require a common account (if she had a Google account then my mother-in-law could just <a href="https://plus.google.com/+MarkChang1/posts/cPGsX7LZvwy">email a link to a hangout</a> and just reuse that hangout every time).</p>
<p>Luckily it turns out that you can overcome these obstacles thanks to <a href="http://www.webrtc.org/">WebRTC</a>. If you don’t happen to know what WebRTC is, think of it as providing real-time voice and video in the browser without plug-ins. On a more technical level, think of it as browser-supported session and signaling support for voice and video communication. It’s <a href="http://caniuse.com/#search=webrtc">currently supported</a> in Chrome, Firefox, and Opera (including Chrome and Firefox for Android); Safari and IE don’t support WebRTC natively at the moment, but do have plugins (<a href="http://blogs.msdn.com/b/ie/archive/2014/10/27/bringing-interoperable-real-time-communications-to-the-web.aspx">IE has announced plans</a> to add support in the future). All of this means that you can now do video chats through the browser without installing a plug-in.</p>
<p>The common format is a website that will launch a video chat room at some URL (sometimes of your own choosing by picking a chat room name). You can then send that URL to other people that they visit in a supported browser. At that point all they have to do is allow the web page to access their microphone and camera (probably saying “look for something asking if the web page can use the camera and microphone at the top or bottom of the page” should be enough instruction for most people). The sites that I found which worked in some form from Android on Chrome are <a href="https://appear.in/">appear.in</a>, <a href="https://talky.io/">Talky</a>, and <a href="https://vline.com/">vLine</a>, although none of them are mobile or tablet-optimized (Talky is my personal favourite). Mozilla is also adding support through <a href="https://support.mozilla.org/en-US/kb/firefox-hello-make-and-receive-calls-guest-mode">Hello</a> which requires Firefox to initiate the video chat but which can be joined through an WebRTC-supported browser.</p>
<p>And for you developers out there, a lot of open source code exists to facilitate using WebRTC, so if you want to add audio/video support to your own web page it shouldn’t be too difficult.</p>
<p>So for those instances where you and someone you want to video chat with don’t have a common messaging platform, WebRTC works out well as a common denominator.</p>
tag:nothingbutsnark.svbtle.com,2014:Post/commentary-on-getting-your-code-to-run-on-python-232014-12-05T13:00:08-08:002014-12-05T13:00:08-08:00Commentary on getting your code to run on Python 2/3<p>Today I committed a heavily updated version of the <a href="https://docs.python.org/3.5/howto/pyporting.html">Python 2/3 porting HOWTO</a>. Basically the doc has shifted from suggesting you use <a href="https://docs.python.org/3/library/2to3.html">2to3</a> for gaining Python 3 compatibility for your Python 2 code and instead you aim for a Python 2/3 source-compatible code base using various tools. Read the HOWTO for the details.</p>
<p>This blog post is about how the shift in approach for Python 3 support came about, general timelines on what will (not) be coming in the future to help with porting, and to emphasize that you should start porting your code <strong>today</strong> no matter what your dependencies say.</p>
<h1 id="the-story-behind-the-shift-in-direction-for-t_1">The story behind the shift in direction for the HOWTO <a class="head_anchor" href="#the-story-behind-the-shift-in-direction-for-t_1">#</a>
</h1>
<p>When Python 3 was being developed, python-dev thought a transpiler from Python 2 code to Python 3 code would be the best solution. So we created <a href="https://docs.python.org/3/library/2to3.html">2to3</a> to transpile Python 2 code to Python 3 and structured it so that distutils would do the transpiling at install-time. That way people could continue to write in idiomatic Python 2 code that could then also run under Python 3 with an extra step.</p>
<p>But opinions had changed by the language summit at PyCon 2014. By then several years had passed since Python 3 came out in December 2008. It had become clear that the transpile step was not the way the community had decided they preferred to make Python 2 code support Python 3. Instead, the community had noticed that the differences to get Python 2 code to run under Python 3 were not really that drastic, and so they had begun writing source-compatible Python 2/3 code, completely cutting out 2to3 and its transpiling step. And so at the summit we talked about what Python the language could do in Python 3.5 to help with this source-compatibility approach and realized that more tooling was necessary to facilitate supporting Python 3.</p>
<p>Being a staunch supporter of Python 3, I decided to take it upon myself to get the tooling to where it needed to be so that source-compatible Python 2/3 was as automated as possible. I had already created <a href="https://pypi.python.org/pypi/caniusepython3">caniusepython3</a> to help people track their dependencies and their Python 3 status, so that part of the puzzle was solved. To actually help transition code I looked around and there was Armin Ronacher’s <a href="https://pypi.python.org/pypi/modernize">Modernize</a> and python-future’s <a href="http://python-future.org/automatic_conversion.html">Futurize</a>. Since Futurize extended Modernize I figured I would help on the lower-level tool.</p>
<p>Right when I decided to start helping out, Armin let the project be spun out by Thomas Kluyver and Daira Hopwood. I looked at what Modernize did and what it could potentially do, and then set out to make the discrepancy between those two lists disappear. In the end I was actually made a project owner by Thomas and Daira and managed to get all of my desired changes in.</p>
<p>With that out of the way, I realized that just because someone updated their code to be Python 2/3 source-compatible it didn’t mean their dependencies had caught up. How were projects to stay Python 3 compatible if they weren’t able to run under Python 3 yet due to their dependencies? That’s when I realized I could update <a href="https://pypi.python.org/pypi/pylint">Pylint</a> to have checkers for things that would not work under Python 3 (either syntactically or semanitcally). This would allow projects to basically make Python 2/3 source-compatibility part of their style guide and have Pylint help enforce it. In the end a bunch of checks got added by me and Pylint added a <code class="prettyprint">--py3k</code> to run Pylint with just the Python 3 checkers so people weren’t forced to buy into all of Pylint’s other checkers.</p>
<p>I would like to point out that when I started this endeavour I didn’t announce it ahead of time, nor was I in some special position with either Modernize or Pylint. I was just another developer out there contributing to open source. And yet I was able to affect both projects and (hopefully) improve them for the better by contributing code. Open source is quite an amazing process when it works.</p>
<h1 id="what-is-not-coming-in-the-near-future-to-help_1">What is (not) coming in the near future to help with porting? <a class="head_anchor" href="#what-is-not-coming-in-the-near-future-to-help_1">#</a>
</h1>
<p>So with the tooling all there now for porting to Python 2/3 code, what exactly is coming in the future that might influence when you start porting? Basically the only thing is modulo/<code class="prettyprint">%</code> operator support for the <code class="prettyprint">bytes</code> type in Python 3.5 thanks to <a href="https://www.python.org/dev/peps/pep-0461/">PEP 461</a>. That should help with the usual text/binary data separation that can trip people up when they do a lot of binary manipulation.</p>
<p>But otherwise from a language perspective I wouldn’t expect anymore backporting of Python 2 features to close the gap between Python 2 & 3. Basically once Python 3.5 comes out you should expect to have the language support you are going to have until Python 2.7 support ends in 2020 in order to facilitate supporting Python 2 and 3 simultaneously.</p>
<h1 id="start-porting-today_1">Start porting TODAY <a class="head_anchor" href="#start-porting-today_1">#</a>
</h1>
<p>You have Modernize and Futurize to automate a large portion of the transition. You have Pylint to help make sure you don’t regress in your Python 3 support. You have caniusepython3 to let you know when your dependences are no longer the hold-up. The tooling is all there, and so there is no reason to wait to port. Even if PEP 461 would help you, you can port now and simply wait to run on Python 3.5.</p>
<p>And you shouldn’t wait for your dependencies either because transitioning now can be helpful. You can start using new practices that Python 3 introduced and get used to them sooner rather than later. And Python 2/3 code is actually very readable; the only major differences from straight Python 2 code is you have to import some functions that used to be built-ins and you use functions to access iterators over dictionaries instead of methods.</p>
<p>Please don’t postpone porting your code. You might as well enjoy the newer idioms you get when you start using all the <code class="prettyprint">__future__</code> statements in Python 3 along with other idiomatic changes. It also means that once your dependencies get ported you will be able to hit the ground running on Python 3 instead making yourself be the hold-up.</p>
tag:nothingbutsnark.svbtle.com,2014:Post/the-longterm-view-of-python-272014-11-28T11:02:49-08:002014-11-28T11:02:49-08:00The long-term view of Python 2.7<p>When Python 2.7.9rc1 was released, I <a href="https://plus.google.com/+Python/posts/hCqdZbrTZw8">shared the news</a> through the <a href="https://plus.google.com/+Python/">+Python</a> Google+ account. Comments on the post ranged from “thanks for keeping Python 2.7 alive!” to “why haven’t you just killed off Python 2.7?” To help frame these discussions, realize that Python 2.7.9 has two big themes:</p>
<ol>
<li>Getting HTTPS/SSL support and security more in line with Python 3</li>
<li>Adding <code class="prettyprint">ensurepip</code> so that <a href="https://pypi.python.org/pypi/pip">pip</a> will be installed alongside Python 2.7.9</li>
</ol>
<p>If you don’t know the reasoning behind these changes it might seem like we are suddenly adding features to Python 2.7 to help extend its life and keep it relevant. The truth, though, is more nuanced.</p>
<p>We as the Python development team have decided to <a href="https://www.python.org/dev/peps/pep-0373/#update">support Python 2.7 until 2020</a> (that was announced at PyCon 2014 so hopefully this isn’t news to you). As with other releases of Python, “support” means bugfixes and not new features. So why the heck have we backported the <code class="prettyprint">ssl</code> module from Python 3.4 back to Python 2.7 and made these changes to the default security setup?</p>
<p>It boils down to security and backwards-compatibility. The security part is hopefully pretty obvious: you want HTTPS and SSL to be up-to-date and secure whenever possible. Since Python 2.7 didn’t secure HTTPS by default, this was an issue. And with Python – both overall and specifically Python 2 – being so popular, having insecure software running a large chunk of the internet isn’t good. And so the decision was made through PEPs <a href="https://www.python.org/dev/peps/pep-0466/">466</a> and <a href="https://www.python.org/dev/peps/pep-0476/">476</a> to make Python 2.7 secure by default when it comes to networking.</p>
<p>The backwards-compatibility part comes into play since this is a change in defaults in the name of security. Since Python 2.7 has been essentially insecure for so long we wanted to give users who want that specific setup a way to opt-out of the new defaults and get back to the way things were.</p>
<p>As for the addition of <code class="prettyprint">ensurepip</code>, since that is a side benefit of the installer and not the language or standard library it was not viewed as backwards-incompatible. Plus we want to help move the community towards switching to pip for their project installation needs to unify the community around a single tool (it also helps move the community towards using <a href="https://pypi.python.org/pypi/wheel">wheels</a> more).</p>
<p>In other words, these changes are in line with the Python development team’s normal dedication to security and backwards-compatibility. This doesn’t mean we don’t want people to be switching to Python 3 actively (we do and I should have news about that before the year is out), nor does it mean we are going to let Python 2.7 be a security hazard simply because we want people to move on to Python 3. Python 2.7 is still in bugfix-only mode and support <strong>will</strong> end in 2020 as planned, we just happened to fix a long-standing security issue after we decided we were not going to drop Python 2.7 support in 2015 as originally planned.</p>