gaubert
Standard output gets stuck on Windows when redirected
Hi Guillaume,
I'd like to build a simplistic Windows GUI on top of this great tool, however, I'm facing an issue. When my program executes gmvault sync [email protected] in a separate Process, it has to use RedirectStandardOutput to be able to show the user what is going on, but when we get to the message that the browser needs to be opened, the output from gmvault stops flowing. Once I press ENTER and supply the verification code, I receive all the instructions, but it's too late (users won't know that the program is waiting for them to press ENTER and then to supply the verification code).
Simple demonstration by using gmvault directly versus piping its output through more (I marked the interaction points with NEED_TO_PRESS_ENTER and SUPPLY_VERIFICATION_CODE):
D:\GMail\gmvault>gmvault sync [email protected]
Use gmvault-db located in D:\GMail/gmvault-db.
Activate Gmvault db cleaning.
Create defaults in D:\GMail/.gmvault/gmvault_defaults.conf. Please touch this file only if you know what to do.
Authentication performed with Gmail OAuth2 access token.
Initiate interactive session to get OAuth2 token from Gmail.
gmvault will now open a web browser page in order for you to grant gmvault access to your Gmail.
Please make sure you're logged into the correct Gmail account ([email protected]) before granting access.
Press ENTER to open the browser.NEED_TO_PRESS_ENTER
You should now see the web page on your browser now.
If you don't, you can manually open:
https://accounts.google.com/o/oauth2/auth?client_id=1070918343777-0eecradokiu8i77qfo8e3stbi0mkrtog.apps.googleusercontent.com&redirect_uri=urn%3Aietf%3Awg%3Aoauth%3A2.0%3Aoob&response_type=code&scope=https%3A%2F%2Fmail.google.com%2F
Once you've granted gmvault access, enter the verification code and press enter:
SUPPLY_VERIFICATION_CODE
Error: Problems when trying to connect to Google oauth2 endpoint: https://accounts.google.com/o/oauth2/token.
Error: HTTP Error 400: Bad Request.
=== Exception traceback ===
Traceback (most recent call last):
File "c:\Users\toto\Dev\gmvault\build\gmv_runner\out00-PYZ.pyz\gmv.gmv_cmd", line 743, in run
File "c:\Users\toto\Dev\gmvault\build\gmv_runner\out00-PYZ.pyz\gmv.credential_utils", line 235, in get_credential
File "c:\Users\toto\Dev\gmvault\build\gmv_runner\out00-PYZ.pyz\gmv.credential_utils", line 387, in get_oauth2_credential
File "c:\Users\toto\Dev\gmvault\build\gmv_runner\out00-PYZ.pyz\gmv.credential_utils", line 342, in _get_oauth2_tokens
File "c:\Users\toto\Dev\gmvault\build\gmv_runner\out00-PYZ.pyz\gmv.credential_utils", line 306, in _get_authorization_tokens
HTTPError: HTTP Error 400: Bad Request
=== End of Exception traceback ===
D:\GMail\gmvault>gmvault sync [email protected] | more
Use gmvault-db located in D:\GMail/gmvault-db.
Activate Gmvault db cleaning.
Create defaults in D:\GMail/.gmvault/gmvault_defaults.conf. Please touch this file only if you know what to do.
Authentication performed with Gmail OAuth2 access token.
Initiate interactive session to get OAuth2 token from Gmail.
NEED_TO_PRESS_ENTER
SUPPLY_VERIFICATION_CODE
gmvault will now open a web browser page in order for you to grant gmvault access to your Gmail.
Please make sure you're logged into the correct Gmail account ([email protected]) before granting access.
Press ENTER to open the browser.You should now see the web page on your browser now.
If you don't, you can manually open:
https://accounts.google.com/o/oauth2/auth?client_id=1070918343777-0eecradokiu8i77qfo8e3stbi0mkrtog.apps.googleusercontent.com&redirect_uri=urn%3Aietf%3Awg%3Aoauth%3A2.0%3Aoob&response_type=code&scope=https%3A%2F%2Fmail.google.com%2F
Once you've granted gmvault access, enter the verification code and press enter:
Error: Problems when trying to connect to Google oauth2 endpoint: https://accounts.google.com/o/oauth2/token.
Error: HTTP Error 400: Bad Request.
=== Exception traceback ===
Traceback (most recent call last):
File "c:\Users\toto\Dev\gmvault\build\gmv_runner\out00-PYZ.pyz\gmv.gmv_cmd", line 743, in run
File "c:\Users\toto\Dev\gmvault\build\gmv_runner\out00-PYZ.pyz\gmv.credential_utils", line 235, in get_credential
File "c:\Users\toto\Dev\gmvault\build\gmv_runner\out00-PYZ.pyz\gmv.credential_utils", line 387, in get_oauth2_credential
File "c:\Users\toto\Dev\gmvault\build\gmv_runner\out00-PYZ.pyz\gmv.credential_utils", line 342, in _get_oauth2_tokens
File "c:\Users\toto\Dev\gmvault\build\gmv_runner\out00-PYZ.pyz\gmv.credential_utils", line 306, in _get_authorization_tokens
HTTPError: HTTP Error 400: Bad Request
=== End of Exception traceback ===
My guess is that gmv_runner.exe turns on output buffering because of the redirection, and after a day of Googling, I still can't find a way to make that stop from the outside (I wanted to avoid asking for a change within Gmvault). I saw that Python has a way to turn off this buffering globally, which would probably be the easiest fix.
Thanks for the help, Denes
@PantherDD Which version have you tried ?
version 1.9.2-beta-1 available for test and fixing many additional problems (especially in the restore). Please test it and let me know if this is fine within 6 days. Many thanks.
https://www.dropbox.com/sh/d5ceo77juacr03y/AACUGcTt6Um-6j6JmBizGPA2a?dl=0
@gaubert: I was using the latest stable version (1.9.1) when filing the issue. I'm installing the version you linked right now. (Minor issue: Windows' "Programs and Features" item in Control Panel lists the new version az 1.5-beta which is a little confusing.)
I tried the same thing with 1.9.2-beta-1, the results are better:
C:\Users\MyUser\AppData\Local\gmvault>gmvault sync [email protected] | more
Use gmvault-db located in C:\Users\MyUser/gmvault-db.
Activate Gmvault db cleaning.
Authentication performed with Gmail OAuth2 access token.
Initiate interactive session to get OAuth2 token from Gmail.
gmvault will now open a web browser page in order for you to grant gmvault access to your Gmail.
Please make sure you're logged into the correct Gmail account ([email protected]) before granting access.
NEED_TO_PRESS_ENTER
Press ENTER to open the browser.You should now see the web page on your browser now.
If you don't, you can manually open:
https://accounts.google.com/o/oauth2/auth?client_id=1070918343777-0eecradokiu8i77qfo8e3stbi0mkrtog.apps.googleusercontent.com&redirect_uri=urn%3Aietf%3Awg%3Aoauth%3A2.0%3Aoob&response_type=code&scope=https%3A%2F%2Fmail.google.com%2F
Once you've granted gmvault access, enter the verification code and press enter:
SUPPLY_VERIFICATION_CODE
Use Default certificate context.
Error: Problems when trying to connect to Google oauth2 endpoint: https://accounts.google.com/o/oauth2/token.
Error: HTTP Error 400: Bad Request.
=== Exception traceback ===
Traceback (most recent call last):
File "gmv\gmv_cmd.py", line 743, in run
File "gmv\credential_utils.py", line 254, in get_credential
File "gmv\credential_utils.py", line 412, in get_oauth2_credential
File "gmv\credential_utils.py", line 367, in _get_oauth2_tokens
File "gmv\credential_utils.py", line 331, in _get_authorization_tokens
HTTPError: HTTP Error 400: Bad Request
=== End of Exception traceback ===
It's a lot better, but Press ENTER to open the browser. is still stuck in a buffer somewhere I assume. And of course I don't know whether there are other places in the code that would still exhibit the same behavior. Not sure what you did to fix most of the issue, but I'd probably try to suppress buffering on a global level (even just optionally via a flag, or rather an env var) to make the fix permanent. If it's indeed a buffering issue, that is. :)
Either way, thanks! Denes
@PantherDD I suppressed buffering at the python level. I did a test on win 7 and win 10 and now it is working ? What version are you using ? I don't get it
@gaubert I'm on Windows 10 x64 with all updates applied (v1607, build 14393.693). As for Gmvault, I was using the binary you linked.
So on your machine, when you issue the command gmvault sync [email protected] | more, you see the Press ENTER to open the browser. message before you press ENTER? Because that one is still stuck for me (as you can see above). Looking at the code, my guess would be it's doing this because of the lack of a terminal \n in that raw_input statement.
This could possibly be a more complete solution than the one you are using right now, although I'm not sure as I've never used Python before, it's just something that I found and has a lot of upvotes.
Another suggestion from a long time ago is to just avoid using raw_input and do a flush manually after printing the prompt message. If you make a global method for this and start using that instead of raw_input, it shouldn't be too painful. But it's all up to you.
And again, thanks for looking into this.
@PantherDD I was hoping this would work. What I do not explain it that it is working for me and not for you ? Why is it different ? I am puzzled as before I could reproduce the issue. I was hoping that I would avoid having to implement my own raw input but well I will add it and let you know when it is ready for testing.
@gaubert It is strange indeed. I have no idea why it'd work on your machine... maybe you have Python installed and that makes a difference somehow? Or some magic environment variable? I don't know.
Anyway, a custom raw_input implementation will hopefully fix this everywhere. As long as there are no other output statements in the code that write to stdout without a terminal \n... 😀
The Unbuffered(sys.stdout) solution could possibly cover even those cases, but I don't know how much of a performance hit the constant flushing would be. If you go for that one, you could get around that potential issue by adding a command line flag, e.g. --allow-output-buffering, that would make the code skip over the line that adds the Unbuffered wrapper to stdout. Then the user can choose whether they want performance or real-time output.