UTF-8 error with Python 3.6 (related with issue #27)

[snb2013]

Config:

\Connections\Connection_RSSL1\rsslPort = "14002"
\Connections\Connection_RSSL1\ServerList = "REPLACE_TO_SERVER_NAME"
\Connections\Connection_RSSL1\connectionType = "RSSL"
\Connections\Connection_RSSL1\UserName = "REPLACE_TO_USER_NAME"
\Connections\Connection_RSSL1\Position = "127.0.0.1"
\Connections\Connection_RSSL1\ServiceName = "ELEKTRON_CONNECT"
\Sessions\Session1\connectionList = "Connection_RSSL1"

Example:

import pyrfa

p = pyrfa.Pyrfa()
p.createConfigDb("./pyrfa.cfg")
p.acquireSession("Session1")
p.createOMMConsumer()
p.login()
p.directoryRequest()
p.dictionaryRequest()

print(p.getTimeSeries("USDAM3L1Y="))

I've tried Python3.5 and Python 3.6. Pyrfa version is 8.5.2. Usage: PYTHONIOENCODING=utf-8 python3.6 timeseries.py Error:

Traceback (most recent call last):
  File "/SOME_ROOT/timeseries.py", line 11, in <module>
    print(p.getTimeSeries("USDAM3L1Y="))
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xed in position 0: unexpected end of data

Originally posted by @snb2013 in https://github.com/devcartel/pyrfa/issues/27#issuecomment-435367702

[wiwat-tharateeraparb]

Did you tried export an env for Python 3 by running export PYTHONIOENCODING=utf-8 on Linux shell before running this PyRFA script?

Or try

print(str(p.getTimeSeries("USDAM3L1Y="), 'utf-8'))

[snb2013]

I tried both methods.

print('PYTHONIOENCODING=%s' % os.environ.get('PYTHONIOENCODING'))
print(str(p.getTimeSeries("USDAM3L1Y="), 'utf-8'))

Output is:

PYTHONIOENCODING=utf-8
Traceback (most recent call last):
  File "/home/sergey-ubuntu/git/pyrfa-master/examples/timeseries.py", line 13, in <module>
    print(str(p.getTimeSeries("USDAM3L1Y="), 'utf-8'))
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xed in position 0: unexpected end of data

Process finished with exit code 1

[wiwat-tharateeraparb]

Can you try:

...
ts = p.getTimeSeries("USDAM3L1Y=")
for record in ts:
    print(record)

[wiwat-tharateeraparb]

Another idea is to set locale on your Ubuntu terminal like:

export LC_ALL=en_US.UTF-8
export LANG=en_US.UTF-8

See if this helps.

[snb2013]

It doesn't help. (((

print('PYTHONIOENCODING=%s' % os.environ.get('PYTHONIOENCODING'))
print('LC_ALL=%s' % os.environ.get('LC_ALL'))
print('LANG=%s' % os.environ.get('LANG'))
ts = p.getTimeSeries("USDAM3L1Y=")
for record in ts:
    print(record)

PYTHONIOENCODING=utf-8
LC_ALL=en_US.UTF-8
LANG=en_US.UTF-8
Traceback (most recent call last):
  File "/home/sergey-ubuntu/git/pyrfa-master/examples/timeseries.py", line 18, in <module>
    ts = p.getTimeSeries("USDAM3L1Y=")
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xed in position 0: unexpected end of data

[snb2013]

Any more ideas?

[wiwat-tharateeraparb]

Can you turn debug on and send the log file to our [email protected] to take a look? We couldn't replicate the UnicodeDecodeError?

Also not sure if this is only specific to timeseries. Can you try with normal subscription?

[snb2013]

I've sent the log. Normal subscription is OK.

p.marketPriceRequest("USDAM3L1Y=")
end = False
while not end:
    try:
        for data in p.dispatchEventQueue(10):
            print(data)
    except KeyboardInterrupt:
        end = True

exit(0)

No exceptions, data is received correctly.

[vmarunov]

Hi, @wiwat-tharateeraparb . Any updates?

[wiwat-tharateeraparb]

I sent below email but got no response. Here is some questions I need for finding the cause:

Hi Sergey,

Good to hear from you.

1. First please set \Logger\AppLogger\useInternalLogStrings = true in pyrfa.cfg in order to get rid of logger errors.

2. Is USDAM3L1Y= the only one the cause this exception or it happens with other TS1 RICs as well? (We don’t have access to this data set anymore)

3. What if you limit returned time-series data with p.setTimeSeriesMaxRecords(N)?

4. What if you turn debug off and doe not print anything out to the screen, can the script finish?

5. Any chance to run it with Python2.7? Would it give same error?

[vmarunov]

Hi Wiwat,

1. First please set \Logger\AppLogger\useInternalLogStrings = true in pyrfa.cfg in order to get rid of logger errors.

Value already set.

2. Is USDAM3L1Y= the only one the cause this exception or it happens with other TS1 RICs as well? (We don’t have access to this data set anymore)

No. Any RICs raised error.

3. What if you limit returned time-series data with p.setTimeSeriesMaxRecords(N)?

No effect. I try with p.setTimeSeriesMaxRecords(1)

4. What if you turn debug off and doe not print anything out to the screen, can the script finish?

No effect. Error raised on code line: data = p.getTimeSeries("EUR=")

Any chance to run it with Python2.7? Would it give same error?

With python 2.7.16 all work fine.

PS. After second call getTimeSeries thrown error "RuntimeError: unidentifiable C++ exception":

>>> p.getTimeSeries("USDAM3L1Y=")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xed in position 0: unexpected end of data
>>> p.getTimeSeries("USDAM3L1Y=")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
RuntimeError: unidentifiable C++ exception

[wiwat-tharateeraparb]

We still couldn't reproduce this issue. We saw Python3 encoding error on Linux in the past and it's due to print() not PyRFA which was resolved by running export PYTHONIOENCODING=utf-8 (on Linux) or chcp 65001 on your terminal application.

Is it possible we can access this time-series TS1 data? Let us know at [email protected].

[manochasunilgithub]

Same issue on windows as well, getting same error when trying to use with Python 3.6, someone in my team tried with python2.7 which is using ascii encoding and it worked fine. I would expect pyrfa to handle encoding correctly rather than clients dealing with it .

[manochasunilgithub]

timeseries = p.getTimeSeries("AAPL.OQ") Above line throws below error UnicodeDecodeError: 'utf-8' codec can't decode byte 0xf7 in position 0: invalid start byte .. not sure, how can I pass encoding to use in pyrfa Api or PyRfa should use UTF-8

[wiwat-tharateeraparb]

To be totally honest, as we don't have enough sponsors to fund our PyRFA development. So we are not in the position to afford Refinitiv feeds to fix this issue. Since we are not able to fix for users, we open-sourced PyRFA so anyone who has access to the such data can fix it.

[manochasunilgithub]

Totally understandable, could you please do one favor? Could you please compile the code with Visual Studio 2015 or Visual Studio 2017 and update the code? There are missing headers as well, would be great if all files are there.. as I tried to compile code by upgrading to 2017 version ... it couldn't find dependent header files. Thanks Sunil // TODO: reference additional headers your program requires here