Commentary on getting your code to run on Python 2/3  

Today I committed a heavily updated version of the Python 2/3 porting HOWTO. Basically the doc has shifted from suggesting you use 2to3 for gaining Python 3 compatibility for your Python 2 code and instead you aim for a Python 2/3 source-compatible code base using various tools. Read the HOWTO for the details.

This blog post is about how the shift in approach for Python 3 support came about, general timelines on what will (not) be coming in the future to help with porting, and to emphasize that you should start porting your code today no matter what your dependencies say.

The story behind the shift in direction for the HOWTO #

When Python 3 was being developed, python-dev thought a transpiler from Python 2 code to Python 3 code would be the best solution. So we created 2to3 to transpile Python 2 code to Python 3 and structured it so that distutils would do the transpiling at install-time. That way people could continue to write in idiomatic Python 2 code that could then also run under Python 3 with an extra step.

But opinions had changed by the language summit at PyCon 2014. By then several years had passed since Python 3 came out in December 2008. It had become clear that the transpile step was not the way the community had decided they preferred to make Python 2 code support Python 3. Instead, the community had noticed that the differences to get Python 2 code to run under Python 3 were not really that drastic, and so they had begun writing source-compatible Python 2/3 code, completely cutting out 2to3 and its transpiling step. And so at the summit we talked about what Python the language could do in Python 3.5 to help with this source-compatibility approach and realized that more tooling was necessary to facilitate supporting Python 3.

Being a staunch supporter of Python 3, I decided to take it upon myself to get the tooling to where it needed to be so that source-compatible Python 2/3 was as automated as possible. I had already created caniusepython3 to help people track their dependencies and their Python 3 status, so that part of the puzzle was solved. To actually help transition code I looked around and there was Armin Ronacher’s Modernize and python-future’s Futurize. Since Futurize extended Modernize I figured I would help on the lower-level tool.

Right when I decided to start helping out, Armin let the project be spun out by Thomas Kluyver and Daira Hopwood. I looked at what Modernize did and what it could potentially do, and then set out to make the discrepancy between those two lists disappear. In the end I was actually made a project owner by Thomas and Daira and managed to get all of my desired changes in.

With that out of the way, I realized that just because someone updated their code to be Python 2/3 source-compatible it didn’t mean their dependencies had caught up. How were projects to stay Python 3 compatible if they weren’t able to run under Python 3 yet due to their dependencies? That’s when I realized I could update Pylint to have checkers for things that would not work under Python 3 (either syntactically or semanitcally). This would allow projects to basically make Python 2/3 source-compatibility part of their style guide and have Pylint help enforce it. In the end a bunch of checks got added by me and Pylint added a --py3k to run Pylint with just the Python 3 checkers so people weren’t forced to buy into all of Pylint’s other checkers.

I would like to point out that when I started this endeavour I didn’t announce it ahead of time, nor was I in some special position with either Modernize or Pylint. I was just another developer out there contributing to open source. And yet I was able to affect both projects and (hopefully) improve them for the better by contributing code. Open source is quite an amazing process when it works.

What is (not) coming in the near future to help with porting? #

So with the tooling all there now for porting to Python 2/3 code, what exactly is coming in the future that might influence when you start porting? Basically the only thing is modulo/% operator support for the bytes type in Python 3.5 thanks to PEP 461. That should help with the usual text/binary data separation that can trip people up when they do a lot of binary manipulation.

But otherwise from a language perspective I wouldn’t expect anymore backporting of Python 2 features to close the gap between Python 2 & 3. Basically once Python 3.5 comes out you should expect to have the language support you are going to have until Python 2.7 support ends in 2020 in order to facilitate supporting Python 2 and 3 simultaneously.

Start porting TODAY #

You have Modernize and Futurize to automate a large portion of the transition. You have Pylint to help make sure you don’t regress in your Python 3 support. You have caniusepython3 to let you know when your dependences are no longer the hold-up. The tooling is all there, and so there is no reason to wait to port. Even if PEP 461 would help you, you can port now and simply wait to run on Python 3.5.

And you shouldn’t wait for your dependencies either because transitioning now can be helpful. You can start using new practices that Python 3 introduced and get used to them sooner rather than later. And Python 2/3 code is actually very readable; the only major differences from straight Python 2 code is you have to import some functions that used to be built-ins and you use functions to access iterators over dictionaries instead of methods.

Please don’t postpone porting your code. You might as well enjoy the newer idioms you get when you start using all the __future__ statements in Python 3 along with other idiomatic changes. It also means that once your dependencies get ported you will be able to hit the ground running on Python 3 instead making yourself be the hold-up.