ion-python icon indicating copy to clipboard operation
ion-python copied to clipboard

Reading a literal (non-escaped) emoji doesn't work when reading from a file with mode 'rb'

Open pbcornell opened this issue 5 years ago • 0 comments

For a file containing the following symbol:

'🏳0🌈️'

the following behavior is observed:

>>> 
>>> with open('<filename>', 'r') as f:
...   print(ion.load(f, single_value=False))
... 
[IonPySymbol(text='🏳0🌈️', sid=None, location=None)]    # correct
>>> with open('<filename>', 'rb') as f:
...   print(ion.load(f, single_value=False))
... 
[IonPySymbol(text='ð\x9f\x8f³0ð\x9f\x8c\x88ï¸\x8f', sid=None, location=None)]    # incorrect

Note that ion-python tests appear to be cheating: https://github.com/amzn/ion-python/blob/master/tests/test_vectors.py#L87

I believe the same symbol should be constructed whether the file is opened with mode 'r' or 'rb'.

pbcornell avatar Jan 10 '20 02:01 pbcornell