`--to-empty` still leaves unnecessary whitespace

[ghost]

Running strip-hints --inplace --to-empty getnonce.py on https://github.com/nyuszika7h/getnonce has the following result:

diff --git a/getnonce.py b/getnonce.py
index 14a6e98..0ae9cc8 100755
--- a/getnonce.py
+++ b/getnonce.py
@@ -26,7 +26,7 @@ def finish():
     input()
 
 
-def run_process(command: str, *args: str, silence_errors: bool = False, timeout: Optional[int] = None) -> Optional[str]:
+def run_process(command , *args , silence_errors  = False, timeout  = None)  :
     """Run a command with the specified arguments."""
     try:
         p = subprocess.run([command, *args], stdout=subprocess.PIPE, stderr=subprocess.STDOUT, encoding='utf-8', timeout=timeout)
@@ -43,7 +43,7 @@ def run_process(command: str, *args: str, silence_errors: bool = False, timeout:
     return p.stdout.strip()
 
 
-def wait_for_device(mode: str) -> None:
+def wait_for_device(mode )  :
     """Wait for a device to be connected over USB and unlocked."""
 
     if mode == 'normal':
@@ -61,7 +61,7 @@ def wait_for_device(mode: str) -> None:
     print()
 
 
-def mobilegestalt_read_int(key: str) -> Optional[str]:
+def mobilegestalt_read_int(key )  :
     """Read an integer from MobileGestalt and return it as a hex string."""
 
     plist = plistlib.loads(run_process('idevicediagnostics', 'mobilegestalt', key).encode('utf-8'))
@@ -74,7 +74,7 @@ def mobilegestalt_read_int(key: str) -> Optional[str]:
         return '{:X}'.format(value)
 
 
-def mobilegestalt_read_bytes(key: str, endianness: str) -> Optional[str]:
+def mobilegestalt_read_bytes(key , endianness )  :
     """Read bytes with the specified endianness from MobileGestalt and return it as a hex string."""
:

I would expect all redundant whitespace before commas, closing parentheses, colons, etc. to be removed.

Environment:

• Debian 11
• Python 3.9.2
• strip-hints 0.1.10

[abarker]

The program should remove all the whitespace, but the problem is that it works at the level of the Python tokenizer/untokenizer. The tokens for type hints have their strings converted to empty strings, but the space before the hints is still restored by untokenize. I briefly looked into the issue a while back, but I didn't come up with a quick fix. The Python untokenize function does not guarantee round-trip untokenizing for whitespace between tokens.

[gsingh93]

Running into the same problem. I haven't read the code, but why can't we just check if we're leaving behind an extra space and skip it in that case?

[abarker]

The program works at the tokenizer level, by changing or setting empty the strings associated with certain tokens in order to strip them. When the Python untokenizer is called to piece the tokens back together into text it sometimes adds whitespace. It's done by the algorithm in the Python library, and cannot be changed.

One option might be to simply drop certain tokens rather than changing the string associated with them, but strip-hints isn't currently designed that way. This might also lead to edge cases where the untokenizer would do other unexpected things or raise exceptions.