Python API breaks when evaluating certain lowercase tags

[ModestMC]

Affiliation General Atomics / DIII-D

Version(s) Affected Client : 7.139.59, 6.1.84 Server : 7.1.[can't remember offhand], 7.139.59

Platform RHEL8 / RHEL6

Describe the bug / To Reproduce The Python API has some kind of bug (or potentially bugs) in the evaluation layer when dealing with lower case characters related to escape sequences such as '\a... ' , '\b... ' , '\f... ' , '\r... ' , '\t... ' eg. '\top...' can fail like so:

import MDSplus

c = MDSplus.Connection([server])
c.openTree("TEST", 350)
good_str="\TOP.NUMBA"
bad_str="\top.numba"
print("GOOD: ",c.get(good_str))
print("BRICKS:",c.get(bad_str))

Expected behavior

GOOD:  350

whereas:

Traceback (most recent call last):
  File "mdsbugs.py", line 8, in <module>
    print("BRICKS:",c.get(bad_str))
  File "/fusion/usc/c8/opt/mdsplus/alpha/7.139.59/python/MDSplus/connection.py", line 319, in get
    return self.conn.get(exp, *args, **kwargs)
  File "/fusion/usc/c8/opt/mdsplus/alpha/7.139.59/python/MDSplus/connection.py", line 213, in get
    return self._get_answer(timeout)
  File "/fusion/usc/c8/opt/mdsplus/alpha/7.139.59/python/MDSplus/connection.py", line 143, in _get_answer
    dims.ctypes.data, numbytes, ctypes.byref(ans), ctypes.byref(mem), int(to_msec)))
  File "/fusion/usc/c8/opt/mdsplus/alpha/7.139.59/python/MDSplus/mdsExceptions.py", line 94, in checkStatus
    raise exception
MDSplus.mdsExceptions.TreeNNF: %TREE-W-NNF, Node Not Found

Or when using a tag like '\a...' the error turns into

line 143, in _get_answer
    dims.ctypes.data, numbytes, ctypes.byref(ans), ctypes.byref(mem), int(to_msec)))
  File "/fusion/usc/c8/opt/mdsplus/alpha/7.139.59/python/MDSplus/mdsExceptions.py", line 94, in checkStatus
    raise exception
MDSplus.mdsExceptions.TdiSYNTAX: %TDI-E-SYNTAX, Bad punctuation or misspelled word or number

When using an additional escape character or a python raw string, these resolve correctly.

Additional context Our hunch is this is an issue with the text processing on the client side, but we couldn't easily confirm it. We also expected '\e...' and '\v...' to fail, however they seemed to work fine.

[mwinkel-dev]

The escape character processing is an inherent feature of Python strings. As is illustrated by this example.

>>> str1="\Tz"
>>> len(str1)
3
>>> 
>>> str2="\tz"
>>> len(str2)
2
>>> 
>>> str3=r"\tz"
>>> len(str3)
3
>>> 
>>> str4="\\tz"
>>> len(str4)
3

And the solution is as described at the end of the initial bug report, namely to use an additional backslash or to use a raw string.

[mwinkel-dev]

Some ideas for consideration . . .

• Update the documentation on the mdsplus.org web site regarding use of tabs (\t) and other special characters in node names / paths passed to the Python API.

• Improve error handling to check for the common special characters in node paths. Because MDSplus uses a backslash to indicate a tag (e.g., \TOP, \NEW_CALC), when users type those tags in lower case it will be easy to unintentionally create typos (e.g. \top starts with a tab, and \new_calc starts with a new-line).

• Not recommended, but it would also be possible for the Python API to detect special characters and convert them back to the two-character mnemonic (e.g, tab is converted to \t).

[ModestMC]

For a tool we've been developing internally I just have a line that converts the input string to uppercase (it wants the treenames to be capitalized for some reason, might actually be a related bug come to think of it).

[mwinkel-dev]

The issue is that uppercasing <tab>something merely makes it <tab>SOMETHING. As is shown in the following example.

>>> str5 = '\tz'
>>> len(str5)
2

>>> str6 = str5.upper()
>>> str6
'\tZ'
>>> len(str6)
2

To avoid having a <tab> in a string, one must either . . .

• not type Python strings that contain tabs (i.e., the \t sequence), or
• have code that scans input strings for special characters and replaces them with two characters (i.e., building up the "raw" string that is equivalent to the user's supposed intent)

[ModestMC]

Ahh, I should have thought of that. Didn't realize the string object got created in memory when it got passed into a function, but it makes sense. My hope was to make it simpler to potentially do preprocessing, but it looks like realistically the outcome here is that while normal MDSplus is not case sensitive, the python API just needs all caps.

[mwinkel-dev]

Hi @ModestMC,

Correct. Users should either use uppercase, raw strings, or double-backslash of special characters (such as \\t instead of \t) when entering Python strings that are passed to the Python API for MDSplus.

We could probably add some documentation to the mdsplus.org Wiki if you wish. However, adding a feature to the Python API to scan for special characters would likely be a low priority (e.g., expanding the coverage of the test suite likely is more important).

Should I close this issue?

[ModestMC]

I think the issue is fine to close.

If you can point me at where in the API the preprocessing would need to live, I can see about potentially adding some string preprocessing to fix the input (at least when TDI is attempting to be evaluated). Even then, this issue could be closed as some kind of tech debt/won't fix/etc. Would that be helpful?

[mwinkel-dev]

Hi @ModestMC,

Thanks for your input. I'll look at the code and get back to you. (And I am also inclined to add a few sentences to the mdsplus.org web site explaining the issue with Python strings.)

[ModestMC]

It appears that string handling errors crop up in other places as well.