Unable to install pytest-bdd in Debian 11 Bullseye

[Jamie-]

Describe the bug
Pip released version 25.1 4 days ago: https://pip.pypa.io/en/stable/news/#v25-1

It seems that Gherkin 29.0.0 is not installable on Debian Bullseye (Python 3.9) anymore starting from Pip 25.1 - see fail below.

pytest-bdd 8.1.0 pins gherkin-official to ^29.0.0 - https://github.com/pytest-dev/pytest-bdd/blob/8.1.0/pyproject.toml#L44

Strangely this doesn't reproduce in the python:3.9-bullseye Docker image using the Python version it supplies, but it does using a vanilla version of Bullseye and installing Python from Debian's package repos.

To Reproduce
I'm using Docker to give me a Debian Bullseye environment

docker run --rm -it debian:bullseye

From within this container:

apt update
apt install -y --no-install-recommends python3 python3-pip python3-venv
python3 -m venv tmp
. ./tmp/bin/activate
pip install pip==25.1
pip install pytest-bdd

Produces:

# pip install pytest-bdd
Collecting pytest-bdd
  Downloading pytest_bdd-8.1.0-py3-none-any.whl.metadata (53 kB)
Collecting Mako (from pytest-bdd)
  Downloading mako-1.3.10-py3-none-any.whl.metadata (2.9 kB)
Collecting gherkin-official<30.0.0,>=29.0.0 (from pytest-bdd)
  Downloading gherkin_official-29.0.0-py3-none-any.whl.metadata (563 bytes)
Collecting packaging (from pytest-bdd)
  Downloading packaging-25.0-py3-none-any.whl.metadata (3.3 kB)
Collecting parse (from pytest-bdd)
  Downloading parse-1.20.2-py2.py3-none-any.whl.metadata (22 kB)
Collecting parse-type (from pytest-bdd)
  Downloading parse_type-0.6.4-py2.py3-none-any.whl.metadata (12 kB)
Collecting pytest>=7.0.0 (from pytest-bdd)
  Downloading pytest-8.3.5-py3-none-any.whl.metadata (7.6 kB)
Collecting typing-extensions (from pytest-bdd)
  Downloading typing_extensions-4.13.2-py3-none-any.whl.metadata (3.0 kB)
Collecting exceptiongroup>=1.0.0rc8 (from pytest>=7.0.0->pytest-bdd)
  Downloading exceptiongroup-1.2.2-py3-none-any.whl.metadata (6.6 kB)
Collecting iniconfig (from pytest>=7.0.0->pytest-bdd)
  Downloading iniconfig-2.1.0-py3-none-any.whl.metadata (2.7 kB)
Collecting pluggy<2,>=1.5 (from pytest>=7.0.0->pytest-bdd)
  Downloading pluggy-1.5.0-py3-none-any.whl.metadata (4.8 kB)
Collecting tomli>=1 (from pytest>=7.0.0->pytest-bdd)
  Downloading tomli-2.2.1-py3-none-any.whl.metadata (10 kB)
Collecting MarkupSafe>=0.9.2 (from Mako->pytest-bdd)
  Downloading MarkupSafe-3.0.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (4.0 kB)
Collecting six>=1.15 (from parse-type->pytest-bdd)
  Downloading six-1.17.0-py2.py3-none-any.whl.metadata (1.7 kB)
Downloading pytest_bdd-8.1.0-py3-none-any.whl (49 kB)
Downloading gherkin_official-29.0.0-py3-none-any.whl (37 kB)
Downloading pytest-8.3.5-py3-none-any.whl (343 kB)
Downloading pluggy-1.5.0-py3-none-any.whl (20 kB)
Downloading exceptiongroup-1.2.2-py3-none-any.whl (16 kB)
Downloading tomli-2.2.1-py3-none-any.whl (14 kB)
Downloading iniconfig-2.1.0-py3-none-any.whl (6.0 kB)
Downloading mako-1.3.10-py3-none-any.whl (78 kB)
Downloading MarkupSafe-3.0.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (20 kB)
Downloading packaging-25.0-py3-none-any.whl (66 kB)
Downloading parse-1.20.2-py2.py3-none-any.whl (20 kB)
Downloading parse_type-0.6.4-py2.py3-none-any.whl (27 kB)
Downloading six-1.17.0-py2.py3-none-any.whl (11 kB)
Downloading typing_extensions-4.13.2-py3-none-any.whl (45 kB)
Installing collected packages: parse, gherkin-official, typing-extensions, tomli, six, pluggy, packaging, MarkupSafe, iniconfig, exceptiongroup, pytest, parse-type, Mako, pytest-bdd
ERROR: Exception:
Traceback (most recent call last):
  File "/usr/lib/python3.9/py_compile.py", line 144, in compile
    code = loader.source_to_code(source_bytes, dfile or file,
  File "<frozen importlib._bootstrap_external>", line 853, in source_to_code
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/tmp/lib/python3.9/site-packages/gherkin/count_symbols_py2.py", line 3
    REGEX_ASTRAL_SYMBOLS = re.compile(ur'[\uD800-\uDBFF][\uDC00-\uDFFF]', re.UNICODE)
                                        ^
SyntaxError: invalid syntax

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.9/compileall.py", line 238, in compile_file
    ok = py_compile.compile(fullname, cfile, dfile, True,
  File "/usr/lib/python3.9/py_compile.py", line 150, in compile
    raise py_exc
py_compile.PyCompileError:   File "/tmp/lib/python3.9/site-packages/gherkin/count_symbols_py2.py", line 3
    REGEX_ASTRAL_SYMBOLS = re.compile(ur'[\uD800-\uDBFF][\uDC00-\uDFFF]', re.UNICODE)
                                        ^
SyntaxError: invalid syntax


During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/tmp/lib/python3.9/site-packages/pip/_internal/cli/base_command.py", line 105, in _run_wrapper
    status = _inner_run()
  File "/tmp/lib/python3.9/site-packages/pip/_internal/cli/base_command.py", line 96, in _inner_run
    return self.run(options, args)
  File "/tmp/lib/python3.9/site-packages/pip/_internal/cli/req_command.py", line 68, in wrapper
    return func(self, options, args)
  File "/tmp/lib/python3.9/site-packages/pip/_internal/commands/install.py", line 459, in run
    installed = install_given_reqs(
  File "/tmp/lib/python3.9/site-packages/pip/_internal/req/__init__.py", line 83, in install_given_reqs
    requirement.install(
  File "/tmp/lib/python3.9/site-packages/pip/_internal/req/req_install.py", line 867, in install
    install_wheel(
  File "/tmp/lib/python3.9/site-packages/pip/_internal/operations/install/wheel.py", line 728, in install_wheel
    _install_wheel(
  File "/tmp/lib/python3.9/site-packages/pip/_internal/operations/install/wheel.py", line 614, in _install_wheel
    success = compileall.compile_file(path, force=True, quiet=True)
  File "/usr/lib/python3.9/compileall.py", line 255, in compile_file
    msg = err.msg.encode(sys.stdout.encoding,
TypeError: encode() argument 'encoding' must be str, not None

Note that you can reproduce this with pip install gherkin-official\<30 instead of pip install pytest-bdd but as the latest release version of gherkin works (pip install gherkin-official) it seems like a pytest-bdd issue.

Also worth adding, which I find very strange, this only seems to reproduce on first install, subsequent runs of pip install pytest-bdd complete without issue. Even though the first attempt fails, and with exit code 2.

Expected behavior
pytest-bdd to successfully install

Version

• pytest version: 8.3.5
• pytest-bdd version: 8.1.0
• OS: Debian Bullseye

[webknjaz]

I think sys.stdout.encoding is derived from the env vars. Check the difference in the mentioned envs for the LANG/LC_* env vars. You could also try forcing UTF-8 by invoking it like python -Xutf8 -m pip. Although, it's probably better to set an equivalent env var.

OTOH, it's None which probably means running w/o TTY or something like a pipe.

Though, if another release works, the deps should be bumped provided that it's compatible.

[Jamie-]

debian:bullseye

# env
HOSTNAME=a61b586e95bf
PWD=/
HOME=/root
TERM=xterm
SHLVL=1
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
_=/usr/bin/env

python:3.9-bullseye

# env
PYTHON_SHA256=8c136d199d3637a1fce98a16adc809c1d83c922d02d41f3614b34f8b6e7d38ec
HOSTNAME=b04afe71bf20
PYTHON_VERSION=3.9.22
PWD=/
HOME=/root
LANG=C.UTF-8
GPG_KEY=E3FF2839C048B25C084DEBE9B26995E310250568
TERM=xterm
SHLVL=1
PATH=/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
_=/usr/bin/env

The python:3.9-bullseye, which works, does have LANG=C.UTF-8. However adding that doesn't seem to help, and neither does python -Xutf8 -m pip :( - see below

Which brings us to your other suggestion that it's None, but there is a TTY and I'm not piping anything.

# python -Xutf8 -m pip install pytest-bdd
Collecting pytest-bdd
  Using cached pytest_bdd-8.1.0-py3-none-any.whl.metadata (53 kB)
Collecting Mako (from pytest-bdd)
  Using cached mako-1.3.10-py3-none-any.whl.metadata (2.9 kB)
Collecting gherkin-official<30.0.0,>=29.0.0 (from pytest-bdd)
  Using cached gherkin_official-29.0.0-py3-none-any.whl.metadata (563 bytes)
Collecting packaging (from pytest-bdd)
  Using cached packaging-25.0-py3-none-any.whl.metadata (3.3 kB)
Collecting parse (from pytest-bdd)
  Using cached parse-1.20.2-py2.py3-none-any.whl.metadata (22 kB)
Collecting parse-type (from pytest-bdd)
  Using cached parse_type-0.6.4-py2.py3-none-any.whl.metadata (12 kB)
Collecting pytest>=7.0.0 (from pytest-bdd)
  Using cached pytest-8.3.5-py3-none-any.whl.metadata (7.6 kB)
Collecting typing-extensions (from pytest-bdd)
  Using cached typing_extensions-4.13.2-py3-none-any.whl.metadata (3.0 kB)
Collecting exceptiongroup>=1.0.0rc8 (from pytest>=7.0.0->pytest-bdd)
  Using cached exceptiongroup-1.2.2-py3-none-any.whl.metadata (6.6 kB)
Collecting iniconfig (from pytest>=7.0.0->pytest-bdd)
  Using cached iniconfig-2.1.0-py3-none-any.whl.metadata (2.7 kB)
Collecting pluggy<2,>=1.5 (from pytest>=7.0.0->pytest-bdd)
  Using cached pluggy-1.5.0-py3-none-any.whl.metadata (4.8 kB)
Collecting tomli>=1 (from pytest>=7.0.0->pytest-bdd)
  Using cached tomli-2.2.1-py3-none-any.whl.metadata (10 kB)
Collecting MarkupSafe>=0.9.2 (from Mako->pytest-bdd)
  Using cached MarkupSafe-3.0.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (4.0 kB)
Collecting six>=1.15 (from parse-type->pytest-bdd)
  Using cached six-1.17.0-py2.py3-none-any.whl.metadata (1.7 kB)
Using cached pytest_bdd-8.1.0-py3-none-any.whl (49 kB)
Using cached gherkin_official-29.0.0-py3-none-any.whl (37 kB)
Using cached pytest-8.3.5-py3-none-any.whl (343 kB)
Using cached pluggy-1.5.0-py3-none-any.whl (20 kB)
Using cached exceptiongroup-1.2.2-py3-none-any.whl (16 kB)
Using cached tomli-2.2.1-py3-none-any.whl (14 kB)
Using cached iniconfig-2.1.0-py3-none-any.whl (6.0 kB)
Using cached mako-1.3.10-py3-none-any.whl (78 kB)
Using cached MarkupSafe-3.0.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (20 kB)
Using cached packaging-25.0-py3-none-any.whl (66 kB)
Using cached parse-1.20.2-py2.py3-none-any.whl (20 kB)
Using cached parse_type-0.6.4-py2.py3-none-any.whl (27 kB)
Using cached six-1.17.0-py2.py3-none-any.whl (11 kB)
Using cached typing_extensions-4.13.2-py3-none-any.whl (45 kB)
Installing collected packages: parse, gherkin-official, typing-extensions, tomli, six, pluggy, packaging, MarkupSafe, iniconfig, exceptiongroup, pytest, parse-type, Mako, pytest-bdd
ERROR: Exception:
Traceback (most recent call last):
  File "/usr/lib/python3.9/py_compile.py", line 144, in compile
    code = loader.source_to_code(source_bytes, dfile or file,
  File "<frozen importlib._bootstrap_external>", line 853, in source_to_code
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/tmp/lib/python3.9/site-packages/gherkin/count_symbols_py2.py", line 3
    REGEX_ASTRAL_SYMBOLS = re.compile(ur'[\uD800-\uDBFF][\uDC00-\uDFFF]', re.UNICODE)
                                        ^
SyntaxError: invalid syntax

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.9/compileall.py", line 238, in compile_file
    ok = py_compile.compile(fullname, cfile, dfile, True,
  File "/usr/lib/python3.9/py_compile.py", line 150, in compile
    raise py_exc
py_compile.PyCompileError:   File "/tmp/lib/python3.9/site-packages/gherkin/count_symbols_py2.py", line 3
    REGEX_ASTRAL_SYMBOLS = re.compile(ur'[\uD800-\uDBFF][\uDC00-\uDFFF]', re.UNICODE)
                                        ^
SyntaxError: invalid syntax


During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/tmp/lib/python3.9/site-packages/pip/_internal/cli/base_command.py", line 105, in _run_wrapper
    status = _inner_run()
  File "/tmp/lib/python3.9/site-packages/pip/_internal/cli/base_command.py", line 96, in _inner_run
    return self.run(options, args)
  File "/tmp/lib/python3.9/site-packages/pip/_internal/cli/req_command.py", line 68, in wrapper
    return func(self, options, args)
  File "/tmp/lib/python3.9/site-packages/pip/_internal/commands/install.py", line 459, in run
    installed = install_given_reqs(
  File "/tmp/lib/python3.9/site-packages/pip/_internal/req/__init__.py", line 83, in install_given_reqs
    requirement.install(
  File "/tmp/lib/python3.9/site-packages/pip/_internal/req/req_install.py", line 867, in install
    install_wheel(
  File "/tmp/lib/python3.9/site-packages/pip/_internal/operations/install/wheel.py", line 728, in install_wheel
    _install_wheel(
  File "/tmp/lib/python3.9/site-packages/pip/_internal/operations/install/wheel.py", line 614, in _install_wheel
    success = compileall.compile_file(path, force=True, quiet=True)
  File "/usr/lib/python3.9/compileall.py", line 255, in compile_file
    msg = err.msg.encode(sys.stdout.encoding,
TypeError: encode() argument 'encoding' must be str, not None

[Jamie-]

Ahh, I've just found pypa/pip#13359

Which contains:

That means, according to the Python 3.9 release schedule, pip will crash if compileall raises an error while compiling a Python module on Python 3.9.6 or older.

$ docker run --rm -it python:3.9-bullseye bash
root@52ae87692cf2:/# python3 -V
Python 3.9.22

vs

$ docker run --rm -it debian:bullseye
root@5cfa485f3386:/# apt update && apt install -y python3
...
root@5cfa485f3386:/# python3 -V
Python 3.9.2