Automate desktop apps? - visual automation + keyboard() / mouse() / clipboard()

[splunkitwo]

This product is very good, but the examples currently provided are all about operating websites. How can I operate desktop applications? Can you give some examples

[kensoh]

Thank you very much for your kind words.. For desktop apps, you primarily use

1. visual automation https://github.com/tebelorg/RPA-Python#visual-automationsee_no_evil2.
2. combined with keyboard() https://github.com/tebelorg/RPA-Python#keyboard-automationmusical_keyboard
3. and sometimes OCR https://github.com/tebelorg/RPA-Python#ocr-automation

Basically using visual automation to click or type into desktop UI elements. For copying out data, you can use keyboard() to do things like [ctrl]a and then [ctrl]c and then clipboard() to read the values. But first go to or tab to the right element. Or perhaps use visual automation or hover() and mouse() to 'select' the element before copying out.

[kensoh]

For starting desktop apps, it could be doing dclick() on the desktop icon, using keyboard('[win]') to search and run from Start menu, keyboard('[win]r') to run command from the Run popup etc. Essentially try to replicate what you do manually.

[splunkitwo]

got it, thank you very much.

[kensoh]

My pleasure, do post here on GitHub issue or at the Telegram group chat, if you run into blockers! https://t.me/pythonrpa

[Peter-YAN]

@kensoh Hi kensoh, i try to use compose keys to delelet all the chars of a textbox, but only one char is deleted. Would you please tell me the reason?

python code: r.type('E:\project\RPA\RPAforPython\idata\export_file_name.png',' ') r.click('E:\project\RPA\RPAforPython\idata\export_file_name.png') r.keyboard('[end][shift][home][delete]')

Thank you very much for your kind words.. For desktop apps, you primarily use

1. visual automation https://github.com/tebelorg/RPA-Python#visual-automationsee_no_evil2.
2. combined with keyboard() https://github.com/tebelorg/RPA-Python#keyboard-automationmusical_keyboard
3. and sometimes OCR https://github.com/tebelorg/RPA-Python#ocr-automation

Basically using visual automation to click or type into desktop UI elements. For copying out data, you can use keyboard() to do things like [ctrl]a and then [ctrl]c and then clipboard() to read the values. But first go to or tab to the right element. Or perhaps use visual automation or hover() and mouse() to 'select' the element before copying out.

[kensoh]

@Peter-YAN, maybe something like below works?

Will need to break up modifier keys so the automation knows key to apply the modifiers like [ctrl], [shift], [alt] etc:

r.click('E:\\project\\RPA\\RPAforPython\\idata\\export_file_name.png')
r.keyboard('[ctrl]a')
r.keyboard('[delete]')

r.click('E:\\project\\RPA\\RPAforPython\\idata\\export_file_name.png')
r.keyboard('[end]')
r.keyboard('[shift][home]')
r.keyboard('[delete]')

[uditi-ymsli]

Hi, i came across this thread of yours where youve mentioned how to read desktop application elements with ctrl{a}, umm im using the same approach of like pressing tab and landing on the right element but doing ctrl[a] after that selects all elements. i want to select the right element only. how can i achieve that any ideas

[kensoh]

Hi @uditijain for using keyboard() you have to first do click('icon.png') on the browser icon or something on the webpage to make it active, then do click('element.png') to make the element really active, not the whole browser window.