pip can't handle MSVC warnings containing special characters

[Cutter]

BPO	32350
Nosy	@pfmoore, @ncoghlan, @tjguk, @zware, @zooba, @dstufft
Files	console.txt: Console log of pylint installation

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/pfmoore'
closed_at = <Date 2017-12-17.12:17:26.155>
created_at = <Date 2017-12-17.11:59:06.945>
labels = ['OS-windows', 'type-crash']
title = "pip can't handle MSVC warnings containing special characters"
updated_at = <Date 2017-12-17.12:17:26.154>
user = 'https://bugs.python.org/Cutter'

bugs.python.org fields:

activity = <Date 2017-12-17.12:17:26.154>
actor = 'paul.moore'
assignee = 'paul.moore'
closed = True
closed_date = <Date 2017-12-17.12:17:26.155>
closer = 'paul.moore'
components = ['Windows']
creation = <Date 2017-12-17.11:59:06.945>
creator = 'Cutter'
dependencies = []
files = ['47334']
hgrepos = []
issue_num = 32350
keywords = []
message_count = 2.0
messages = ['308485', '308486']
nosy_count = 8.0
nosy_names = ['paul.moore', 'ncoghlan', 'tim.golden', 'zach.ware', 'steve.dower', 'dstufft', 'Marcus.Smith', 'Cutter']
pr_nums = []
priority = 'normal'
resolution = 'third party'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'crash'
url = 'https://bugs.python.org/issue32350'
versions = ['Python 3.6']

Linked PRs

• gh-76351: Support custom extractor for ZipFile.extract and ZipFile.extractall #154704

[Cutter]

When trying to install pylint using pip on Windows 10, the installation of wrapt (a dependency of pylint) fails because a special character in an MSVC warning can't be decoded to utf-8.

Below is the relevant part of the console output:
---

Exception:
Traceback (most recent call last):
  File "C:\Users\(...)\AppData\Local\Programs\Python\Python36\lib\site-packages\pip\compat\__init__.py", line 73, in console_to_str
    return s.decode(sys.__stdout__.encoding)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 51: invalid start byte

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\(...)\AppData\Local\Programs\Python\Python36\lib\site-packages\pip\basecommand.py", line 215, in main
    status = self.run(options, args)
  File "C:\Users\(...)\AppData\Local\Programs\Python\Python36\lib\site-packages\pip\commands\install.py", line 342, in run
    prefix=options.prefix_path,
  File "C:\Users\(...)\AppData\Local\Programs\Python\Python36\lib\site-packages\pip\req\req_set.py", line 784, in install
    **kwargs
  File "C:\Users\(...)\AppData\Local\Programs\Python\Python36\lib\site-packages\pip\req\req_install.py", line 878, in install
    spinner=spinner,
  File "C:\Users\(...)\AppData\Local\Programs\Python\Python36\lib\site-packages\pip\utils\__init__.py", line 676, in call_subprocess
    line = console_to_str(proc.stdout.readline())
  File "C:\Users\(...)\AppData\Local\Programs\Python\Python36\lib\site-packages\pip\compat\__init__.py", line 75, in console_to_str
    return s.decode('utf_8')
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 51: invalid start byte

---

I changed line 73 in \pip\compat\init.py to: print("!! TEST !! : ", s). The full console output after that change is in the attachment. The relevant part is:

 !! TEST !! :  b'C:\\Program Files (x86)\\Microsoft Visual Studio\\2017\\Community\\VC\\Tools\\MSVC\\14.11.25503\\bin\\HostX86\\x64\\cl.exe /c /nologo /Ox /W3 /GL /DNDEBUG /MD -IC:\\Users\\(...)\\AppData\\Local\\Programs\\Python\\Python36\\include -IC:\\Users\\(...)\\AppData\\Local\\Programs\\Python\\Python36\\include "-IC:\\Program Files (x86)\\Microsoft Visual Studio\\2017\\Community\\VC\\Tools\\MSVC\\14.11.25503\\ATLMFC\\include" "-IC:\\Program Files (x86)\\Microsoft Visual Studio\\2017\\Community\\VC\\Tools\\MSVC\\14.11.25503\\include" "-IC:\\Program Files (x86)\\Windows Kits\\10\\include\\10.0.16299.0\\ucrt" "-IC:\\Program Files (x86)\\Windows Kits\\10\\include\\10.0.16299.0\\shared" "-IC:\\Program Files (x86)\\Windows Kits\\10\\include\\10.0.16299.0\\um" "-IC:\\Program Files (x86)\\Windows Kits\\10\\include\\10.0.16299.0\\winrt" /Tcsrc/wrapt/_wrappers.c /Fobuild\\temp.win-amd64-3.6\\Release\\src/wrapt/_wrappers.obj\r\n'
 !! TEST !! :  b'_wrappers.c\r\n'
 !! TEST !! :  b"src/wrapt/_wrappers.c(195): warning C4244: 'return'\xff: conversion de 'Py_hash_t' en 'long', perte possible de donn\x82es\r\n"
 error
 Exception:
 Traceback (most recent call last):
   File "C:\Users\(...)\AppData\Local\Programs\Python\Python36\Lib\site-packages\pip\compat\__init__.py", line 74, in console_to_str
     return s.decode(sys.__stdout__.encoding)
 UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 51: invalid start byte

As a workaround I've changed the original line 73 in \pip\compat\init.py :

return s.decode(sys.__stdout__.encoding)

to:

return s.decode(sys.__stdout__.encoding, "replace")

(thanks to dieter on comp.lang.python for his help). I don't have the knowledge to propose a patch.

[pfmoore]

This is a pip issue, not a Python issue. It's a known issue in the current version of pip, and should be fixed in the development version (via pull request pypa/pip#4486).

If you like, you can try the development version of pip. At the moment there is no planned date for the next release of pip, but we're hoping to put together a release in the not too distant future.