cryptography
cryptography copied to clipboard
pyca
Implement (or document) streaming API
I'm looking at https://cryptography.io/en/latest/fernet/, and I'm wondering how to use this API if I want to encrypt a lot of data.
I hope that I can simply call encrypt in a loop:
with open('secret.txt', 'rb') as ifh:
with open('encrypted.txt', 'wb') as ofh:
f = Fernet(key)
while True:
buf = ifh.read(BUFSIZE)
if not buf:
break
ofh.write(f.encrypt(buf)))
..but it would be nice to have this offically blessed (and implemented, if it doesn't work like that at the moment).
Fernet is unfortunately unsuitable for streaming encryption due to the requirement that the entirety of the payload be processed by HMAC before you know if any data can be used. We have some ideas for an authenticated encryption recipe that allows streaming (utilizing some concepts from Tahoe-LAFS's merkle tree authentication of ciphertext) but no spec has yet been written.
Ah, I see. I also just noticed that the data returned by Fernet is base64 encoded - probably also not a good idea if there's a lot of it :-).
In case you are interested (just throwing it out there): I was interested in the cryptography module because I was thinking it could replace my own stream-encryption code in https://bitbucket.org/nikratio/s3ql/src/407ece4bfb515c98f973c2ecb0eab1d3a13fb766/src/s3ql/backends/comprenc.py?at=default&fileviewer=file-view-default#comprenc.py-453. Maybe this can give you some ideas for API design (or what mistakes people do when implementing it themselves from the primitives) :-).
Another streaming AES implementation I wrote over the last days: https://github.com/nils-werner/zget/blob/crypto/zget/crypto.py
- Implemented as a simple generator
- hashes password using PBKDF2 and salt
- uses AES CTR (so we can accept arbitrarily sized blocks to encrypt)
- uses SHA256 HMAC for message Encrypt-then-MAC authentication
- prepends IV and salt to message
- automatically appends HMAC signature
- automatically appends
cipher.finalize()call - Python 2 and 3 compatible and tested
Because it is implemented as a generator you can use it very easily, e.g.
from zget import crypto
cipher = crypto.aes.encrypt("secret_key")
with open("README.md", "r") as fh:
for line in cipher(fh):
print line
We have some ideas
FTR: https://mail.python.org/pipermail/cryptography-dev/2015-January/000371.html
And also, an implementation that does AES-CTR+HMAC+SPAKE2. None of this code has been reviewed yet (but is about 130 lines long, plus a couple of lines for SPAKE), so any feedback very welcome. It can be used like
from zget import crypto
cipher = crypto.aes_spake.encrypt("secret_key")
key_a = cipher.start() # send key_a to the recipient
cipher.finish(key_b) # received key_b from recipient
with open("README.md", "r") as fh:
for line in cipher(fh):
print line
I'm still interested in this and have a potential spec but haven't had the bandwidth to get it reviewed by enough cryptographers to feel comfortable implementing it in a public API yet. It's always useful to know about more people who could use an authenticated encryption streaming API though!
