How to debug Python part (gyp itself) of project?

[owl-from-hogvarts]

Hello! Thanks for waiting) I'd like to debug the Python part of node-gyp (which is located in gyp folder). Within my previouse contribuiton (#2254 and #2428, which still is not applyed) i have modified some lines of code in gyp folder. For preciouse versions of gyp-next these modifications worked well but after certain update (don't know which exectly) the things have went wrong. So on Windows with Python 3.6 error occures:

Error message (path to file `gyp\pylib\gyp\easy_xml.py`):

Traceback (most recent call last):
  File "D:\a\node-gyp\node-gyp\gyp\gyp_main.py", line 45, in <module>
    sys.exit(gyp.script_main())
  File "D:\a\node-gyp\node-gyp\gyp\pylib\gyp\__init__.py", line 662, in script_main
    return main(sys.argv[1:])
  File "D:\a\node-gyp\node-gyp\gyp\pylib\gyp\__init__.py", line 654, in main
    return gyp_main(args)
  File "D:\a\node-gyp\node-gyp\gyp\pylib\gyp\__init__.py", line 639, in gyp_main
    generator.GenerateOutput(flat_list, targets, data, params)
  File "D:\a\node-gyp\node-gyp\gyp\pylib\gyp\generator\msvs.py", line 2129, in GenerateOutput
    _GenerateProject(project, options, msvs_version, generator_flags, spec)
  File "D:\a\node-gyp\node-gyp\gyp\pylib\gyp\generator\msvs.py", line 1016, in _GenerateProject
    return _GenerateMSBuildProject(project, options, version, generator_flags, spec)
  File "D:\a\node-gyp\node-gyp\gyp\pylib\gyp\generator\msvs.py", line 3727, in _GenerateMSBuildProject
    toolset,
  File "D:\a\node-gyp\node-gyp\gyp\pylib\gyp\generator\msvs.py", line 2208, in _GenerateMSBuildFiltersFile
    easy_xml.WriteXmlIfChanged(content, filters_path, pretty=True, win32=True)
  File "D:\a\node-gyp\node-gyp\gyp\pylib\gyp\easy_xml.py", line 127, in WriteXmlIfChanged
    xml_string = xml_string.decode("cp1251").encode(encoding)
AttributeError: 'str' object has no attribute 'decode'

From which we can conclude that there is a type mismatch. I want to figure out, do we still need to tweak in that line of code? The best way to figure this out is to see the args and return values of the function. And that is where the debugger comes into the game. BUT! I am JS programmer, not a Python, then I don't know how to use Python debugger. And there is one more issue. Python process is spawned by the js program, not by me/user. Then how I can guide Python to enter the debug mode?

[cclauss]

Can you please provide more details?

[owl-from-hogvarts]

I will but a little bit later) i am quite busy now

[owl-from-hogvarts]

@cclauss thanks for waiting. Check out description of the issue)

[cclauss]

The key insights here are:

1. python2 -c "print(bytes is str)" # --> True (the wild west!)
2. python3 -c "print(bytes is str)" # --> False (a clear distinction)
3. bytes.decode() is the opposite operation of str.encode()

So, in Python 3, str never had a .decode() method and bytes never had a .encode() method. Decoding a str in Python 3 does not make sense because it is already decoded. Encoding bytes in Python 3 does not make sense because it is already decoded. If in doubt, the Python builtin function isinstance() can help and you can always write:

x_as_bytes = x.encode() if isinstance(x, str) else x

[owl-from-hogvarts]

Does this mean that problem with encoding in that place specific only to Python 2? I guess i should test without that line on Windows with Python 3.6 to ensure that things work well?

Btw, i does not really understand to what we decode string? To internal python representation or to some default encoding like utf-8? Or it is like we putting some numbers as bytes and then telling python to treat them as string (i.e. decode numbers to string)? And encode is the opposit proccess which represents string as array of bytes in some encoding (e.g. utf-8)? Am i understand things right?

According to this what we are doing in that line:

1. (Implicit) treat string as array of bytes
2. convert the array of bytes to internal representation (or default encoding)
3. convert back to array of bytes encoded with desired encoding
4. (Implicit) treat result as string and return it

So the line was necessary due to reason that string was treated as already decoded but actually it was not. But in pyhton 3 decoding is done somewhere before?

[cclauss]

Given that Python 3.6 is EOL, is this still a problem?

• https://devguide.python.org/versions

I would try to fix this as:

xml_string = xml_string.decode("cp1251").encode(encoding)
# -->
if isinstance(xml_string, str):
    xml_string = xml_string.decode("cp1251")  # str --> bytes
xml_string = xml_string.encode(encoding)  # bytes --> str

[owl-from-hogvarts]

Hello! Excuse me for such a long response time. This change seems good to me. I guess it won't cause any disruption. It is already included in #2554. Still need to be updated on gyp-next. I will take care of that. Thank you)

[owl-from-hogvarts]

Oops! This does not work. I need some time to do investigation. And for that purpose I need to how to launch gyp in debug mode. My thought is that I can just add debugger flag to args string just before starting gyp from js code.

[StefanStojanovic]

This issue seems no longer relevant as it uses a no longer supported version of Python and has been stale for a long time. If there are no objections, I will close this issue next Mon-Tue. In case something similar happens again please open a new issue.