Drop support for end-of-life Python versions for KS 0.10?

[dgelessus]

Revisiting the discussion in #695 a bit. Now that Kaitai Struct 0.9 is released, should we drop support for Python versions that are no longer supported by the CPython developers? To be exact, those versions are Python 2.7 (EOL since 2020-01-01, final version 2.7.18 released on 2020-04-20), Python 3.4 (final version 3.4.10 released on 2019-03-18) and Python 3.5 (final version 3.5.10 released on 2020-09-05).

I'm suggesting this not just because these versions are no longer supported, but because dropping them would allow simplifying some aspects of the KS internals for Python, and would allow using some (relatively) modern Python features in the future. In particular:

• Dropping Python 2.7 would allow:

  • Removing all of the Python 2/3 compatibility logic in the kaitaistruct module, which would make many byte array operations much more straightforward. Many of the related helper methods (int_from_byte, byte_from_int, byte_array_index, etc.) would become trivial enough that they could simply be inlined in the compiler-generated and runtime code. (Some of the methods would need to stay for backwards compatibility with code generated by older compiler versions, but newer compilers don't have to use them anymore and can generate the same code inline instead.)
  • Using various features added in Python 3 (before 3.4), such as int.from_bytes (added in Python 3.2) for reading packed integers from bytes with less boilerplate than the struct module.
  • Making the version situation a bit less confusing. We can just say "Python 3.4 or later" instead of "Python 3.4 and later as well as Python 2.7". We also don't have to explain anymore how to install enum34, because it's not needed on Python 3.4 and later.

• Dropping Python 3.4 would allow using the typing module to provide static type annotations. (The syntax for type annotations has been present since Python 3.0, but the typing module, which provides many definitions needed to use type annotations effectively, has only been included in the standard library since Python 3.5.)

• Dropping Python 3.5 would allow:

  • Using f-strings for string formatting, which can be used to replace str.format and %-based formatting in almost all cases, and are much more convenient to use.
  • Adding type annotations for variables. (Python 3.5 and older only support annotating function parameters and return values.)

Like in the previous discussion, I don't really have any good statistics on what Python versions are used with Kaitai Struct. According to the PyPI statistics for the Python runtime, the majority of downloads come from Python 3.6 through 3.8, with 3.7 being the most common. There are still a small number of downloads from Python 2.7 and 3.5, but Python 3.4 users seem to be basically nonexistant.

This means that we can probably safely drop Python 3.4 without causing trouble for many users, and perhaps also 3.5. Unfortunately we can't benefit from the new features in 3.5 and 3.6 unless we also drop Python 2.7, and that's the version I worry the most about - because it's the last version in the Python 2 series, it's still used/required by some older systems and software that haven't made the jump to Python 3. On the other hand, the people still running Python 2 probably also won't update their Kaitai Struct compiler and runtime very often, so they might not really be affected much if a future Kaitai Struct version drops Python 2.7 support - they can just keep using the last KS version still supporting 2.7.

TLDR: Should we drop support for Python 2.7, 3.4, and 3.5? If anyone still uses Kaitai Struct with these Python versions, please leave a comment.

[KOLANICH]

Dropping Python 3.4 would allow using the typing module to provide static type annotations.

Not correct. typing in python 3.4 works for me.

Using f-strings for string formatting

IMHO not needed.

Adding type annotations for variables.

It is a tasty feature, but it is incompatible to 3.4. For 3.4-compatihility the official approach is to use so-called type comments, there is even some support for them in ast module.

You know my position about 3.4: since 3.5 has dropped Win XP, upgrading to 3.5+ means buying a new PC with NSA and Chinese backdoors, and either with tonns of RAM to mitigate 12309, or a license to Windows 10 or 7, and maybe other equipment, which has no drivers for Linux or modern Windows. It is inacceptable.

Or maybe rewriting modern version of python to work on XP, and rebuilding all the binary packages to work with it, and maintaining it, since it is unlikely that python devs want this stinking shit in their code again. Of course it is a sign of a design flaw (lack of proper abstraction layer) in python itself, but we don't have another python and maybe maintaining a properly-designed python would be more burden, rather than maintaining it in its current state, assumming that all the old shit is dropped.

[dgelessus]

Not correct. typing in python 3.4 works for me.

Yes, it can be installed from PyPI, but it's only been included in the stdlib since 3.5. I should have said "would allow using the typing module without having to install extra dependencies".

Win XP

Python 3.5.3 (3f6eaa010fce, Jan 11 2018, 04:46:12)
[PyPy 5.10.1 with MSC v.1500 32 bit] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>>> import sys
>>>> sys.getwindowsversion()
sys.getwindowsversion(major=5, minor=1, build=2600, platform=2, service_pack='Service Pack 3')

Also I'm sure you can build a more recent version of CPython under the last Cygwin version that still supports XP.

[KOLANICH]

Python 3.5.3 (3f6eaa010fce, Jan 11 2018, 04:46:12) [PyPy 5.10.1 with MSC v.1500 32 bit] on win32

Thanks for the info, should definitely try it. Though I'm not sure that cexts like numpy, scipy, pandas, and cython, compression libraries (i.e. zstd and brotli), etc are available for pypy and work fine. Also pypy has a drawback of overconsumption of memory. About cygwin ... I cannot say anything about it, I have not used it.

[generalmimon]

The topic of dropping Python 2 came up again in kaitai-io/kaitai_struct_python_runtime#78 (comment) by @armijnhemel:

Of course, if you are want to keep Python 2 support, then you would need to keep the current solution. I don't know if other languages support something like this, but if so, then dropping Python 2 support would be a small price to pay.

I don't think we have to keep supporting Python 2 for much longer. It's EOL for almost 5 years now, and as I mentioned in kaitai-io/kaitai_struct_python_runtime#72 (comment), according to https://pypistats.org/packages/kaitaistruct, the download stats for our kaitaistruct PyPI package on Python 2 look negligible, so it seems unlikely that anyone still depends on this combination to work.

I'd say let's keep Python 2 support in the upcoming release (0.11), announce in the release notes that we'll drop it soon, and if no one complains, drop it in the next release after 0.11.

[armijnhemel]

I'd say let's keep Python 2 support in the upcoming release (0.11), announce in the release notes that we'll drop it soon, and if no one complains, drop it in the next release after 0.11.

This makes sense to me. The only places where there is Python 2 support at the moment is in the LTS releases of various distributions, but I think that these typically don't ship any kaitai struct related things.