bad docstring for sign functions

[vEpiphyte]

In multiple places in docs, the sign() functions are noted to take bytes objects as the digest. However, this does not work since in many places pyopenssl is doing .encode() calls on what it expects to be str objects.

xcsr = crypto.X509Req()
xcsr.set_pubkey(pkey)  # assuming pkey is valid
xcsr.sign(pkey, b'sha256')
.... results in the following exception
  File "/home/user/.pyenv/versions/3.6.1/envs/someenv/lib/python3.6/site-packages/OpenSSL/crypto.py", line 990, in sign
    digest_obj = _lib.EVP_get_digestbyname(_byte_string(digest))
  File "/home/user/.pyenv/versions/3.6.1/envs/someenv/lib/python3.6/site-packages/OpenSSL/_util.py", line 112, in byte_string
    return s.encode("charmap")
AttributeError: 'bytes' object has no attribute 'encode'

The example above works with the str object 'sha256'.

A side effect of this is that tools which are type-hint aware (such as PyCharm) will report type errors for correct code.

Link to one invalid doc example: https://pyopenssl.org/en/stable/api/crypto.html#OpenSSL.crypto.X509Req.sign

This was encountered using python 3.6.1 and pyOpenSSL 17.3.0.

[jlaine]

OK indeed judging from the unit tests, a string is indeed expected. I'm willing to submit a pull request but I'm not sure in what style we want to document this: :class:str (py3) or :class:unicode (py2) ? There seem to be both in existing docstrings..

[hynek]

That’s a good question as 2020 is looming. But I’m gonna say let’s use unicode and bytes for now because it’s unequivocal and pyOpenSSL is currently downloaded by 88% on Python 2.

[ilons]

Since it's now March 2020, and Python 2 are now sunsetted, can we please go ahead and make this change?