Supporting old Python versions

[dgelessus]

This topic came up in multiple recent issues (#675, kaitai-io/kaitai_struct_python_runtime#25) - which old Python versions should we support for how long?

Which versions does KS currently support?

kaitai_struct_python_runtime is compatible with a lot of Python versions, because it doesn't use many advanced Python features. The minimum required version seems to be Python 2.6, which added support for byte-string literals (b"...") and the io module. Python 3 versions before 3.3 are not supported: the runtime uses setup.cfg-based declarative configuration, which was added in setuptools 30.3.0, and setuptools 30.0.0 dropped support for Python 3.2. (This just applies to the package metadata though - the actual source code might even be compatible with Python 3.0-3.2.)

The Python code generated by ksc is also compatible with Python 2.6 and later (and possibly even older, if it wasn't limited by kaitai_struct_python_runtime's requirements). There is one exception: if the spec uses enums, the generated code uses the enum module, which was only added to the standard library in Python 3.4 (and doesn't exist in Python 2). However, the enum34 library provides a backport of the enum module to older Python versions, down to 3.1 and 2.4.

In short, we currently support Python 3.4 and later, and if enum34 is installed also 3.3, 2.7 and 2.6.

Note: the CI only tests on Python 2.7 and 3.6, so it's possible that there are subtle bugs on versions older than that.

Which Python versions are still supported, and which are EOL?

The question of which Python versions are "supported" or "EOL" is a little complicated.

Most of the time, we're talking about what is still supported by the CPython developers. (CPython is the reference implementation of Python and is used by the vast majority of Python users.) By that definition, Python 2.6 has been EOL since 2013-10-29, Python 3.3 since 2017-09-29, Python 3.4 since 2019-03-18, and Python 2.7 since 2020-01-01 (although the final Python 2.7 version will be released in "mid-April, 2020").

However, this is only what the CPython devs themselves support - there are third parties that provide longer support for CPython versions that the upstream devs no longer support. Most notably this includes Linux distributors who provide long-term support releases of their distros (e. g. Ubuntu, RHEL, CentOS) - they support the Python versions that shipped with each release for the entire support duration of that OS release. This is especially relevant for Python 2.7, which is included in Ubuntu 18.04 LTS (supported until 2023-04, and paid support until 2028-04) and RHEL and CentOS 8 (supported until 2024-05, security fixes until 2029).

PyPy, which is perhaps the biggest alternative implementation of Python, plans to support its Python 2 version "forever".

Which versions should we support in the future?

I think this should depend on what the Kaitai Struct userbase uses - unfortunately I don't think we have a lot of statistics for that. According to PyPI stats for the kaitaistruct package, the majority of users use Python 3.6 and 3.7. Usage of 3.8, 3.5 and 2.7 is very small, and other versions are basically nonexistant. I don't know how reliable these statistics are though - PyPI download statistics are known to be inaccurate.

IMHO it would be safe to drop support for Python 2.6, 3.3 and probably also 3.4 - these versions are relatively old by now, and all of them have compatible newer versions that users can upgrade to. Python 2.7 is a different story - although it's technically no longer supported by the CPython devs, it's still used a lot, because migration from Python 2 to 3 can be difficult.

[KOLANICH]

There is one exception: if the spec uses enums, the generated code uses the enum module, which was only added to the standard library in Python 3.4

stuff like enums and some other stdlib are usually easily transplanted to older versions. I remember that enum.py from 3.6 worked on 3.4 without any changes. And some other modules too.

Python 3 versions before 3.3 are not supported: the runtime uses setup.cfg-based declarative configuration, which was added in setuptools 30.3.0,

It is needed only to build a wheel (pip also builds wheels in process of installing).

they support the Python versions that shipped with each release for the entire support duration of that OS release.

In fact they have old packages with extreme shitcode, like debdelta.

the majority of users use Python 3.6 and 3.7.

Debian use it as a default interpreter and most of binary packages for python it distributed via apt are built for it. Though if I do some dirty hacks with symlinking some of them also work fine in 3.8.

Python 2.7 is a different story - although it's technically no longer supported by the CPython devs, it's still used a lot, because migration from Python 2 to 3 can be difficult.

It really can for some badly-written projects.

[GreyCat]

No strong opinions here (as I don't really use Python), but considering what I hear from time to time, this seems to be pretty much on the spot:

the majority of users use Python 3.6 and 3.7. Usage of 3.8, 3.5 and 2.7 is very small, and other versions are basically nonexistant.

i.e. people slowly switch to Python 3, but lots of older projects either are Python 2 only, or support both for some reason. Nobody really cares much about compatiblity within versions, and Python folks seem to be relatively agile and fast to upgrade.

OS-wise, I would pay attention to the fact that:

• Ubuntu 16.04LTS ships Python 3.5.1
• Ubuntu 18.04LTS ships Python 3.6.5
• Debian 8 jessie from 2015 ships 3.4.2
• Debian 9 stretch (=oldstable) from 2017 ships 3.5.3
• Debian 10 buster (=stable) from 2019 ships 3.7.3

I would say that we can wait for ~1-2 weeks, and unless anyone can jump in with solid arguments, we can easily say "we support 2.7 + 3.5 + 3.6 + 3.7 + 3.8", and cement that in CI and documentation.

[dgelessus]

Whatever versions we decide on, we should set the python_requires option in setup.cfg accordingly, so that pip doesn't try to install the library on an incompatible Python version. For the version range suggested above (3.5 and newer plus 2.7), this would look like this:

[options]
# ...
python_requires = >=2.7,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*

Yes, this doesn't look nice, but it's the only way to specify a version range with "holes" in it. When we drop 2.7 support at some point in the future, this can be simplified to just >=3.5 (or whatever minimum version we require by then).

[KOLANICH]

I vote for supporting 3.4 (with a hack of replacing enum.py from later versions) because it is the latest ver available for XP (they have dropped it because they wanted a new Visual Studio and didn't want to spend time on making it buildable with free toolchains and linkable to free runtimes). Enforcing users to upgrade is good only when the upgrade is free, for XP PCs it is definitely not free (and Linux is completely unsuitable for everyone who has exotic hardware (s.a. AFM) or just don't have large excess of physical RAM until 12309 is really fixed, the only feature of Linux-based desktop OSes that makes me tolerate 12309 is that they don't have M$ disgusting attitute to users of their OSes).

[dgelessus]

Supporting 3.4 should be fine, there aren't a lot of additions/changes from 3.4 to 3.5 that are relevant to KS. In general supporting older Python versions should be much easier for KS than for normal Python projects, because most code is generated, so you don't care as much about convenience features in the language (like f-strings).

[GreyCat]

Ok, so if everyone's happy with it, action items are:

• Document this in Python notes
• Ensure that CI lists 2.7, 3.4, 3.8
• Fix python_requires line in setup.cfg in runtime

[dgelessus]

Add 3.4, 3.5, 3.7 to CI

There's also Python 3.8.

I don't know if we should include every single supported Python 3 version in the CI matrix, especially considering that most other target languages only have one or two versions in the matrix. The changes between Python 3 versions are usually relatively minor, so it might be enough to test just Python 2.7, 3.4, and 3.8 (and maybe 3.6), in the interest of keeping the CI matrix at a manageable size.

[GreyCat]

Ok, let's stick to the idea that we're doing special things individually (2.7) and for normal things (3.x) we only test boundaries (3.4 and 3.8). And we review it once a year.

[GreyCat]

Ok, so can I ask you guys to review it: kaitai-io/kaitai_struct_python_runtime#38 ?

[generalmimon]

Is there anything left to solve here?

[dgelessus]

Quoting from #695 (comment):

• Document this in Python notes

Other than that I think this issue is finished, as we've decided on a version range to support officially. A remaining question might be if/when to drop support for Python 2.7, but that can be discussed in a separate issue.

[generalmimon]

Document this in Python notes

@dgelessus Any chance I can ask you to do it so we can push the 0.9 milestone forward? As I see it, a single sentence "Kaitai Struct officially supports Python versions ..." somewhere at the beginning of the Python notes would be enough, but you can add additional info as you see fit.