OpenAdapt
OpenAdapt copied to clipboard
OpenAdaptAI
Implement Chrome Extension
Certain use cases (e.g. SalesForce, LinkedIn) would greatly benefit from reading and writing website markup and the DOM (i.e. by simplifying tokenization of screenshots).
-
Implement a Chrome Plugin that sends and receives messages from PuterBot via native messaging: https://developer.chrome.com/docs/extensions/mv3/nativeMessaging/ . On every change to the Document Object Model (DOM) of the currently opened page, the extension should send a message to PuterBot.
-
Implement an adapter in PuterBot to receive and store the messages with https://github.com/Rayquaza01/nativemessaging. To start with, this can look something like this:
browser.py:
import nativemessaging
def main():
# based on https://github.com/Rayquaza01/nativemessaging#sample
reply_num = 0
while True:
message = nativemessaging.get_message()
print(message)
nativemessaging.send_message(nativemessaging.encode_message(str(reply_num))
reply_num += 1
if __name__ == "__main__":
main()
In the extension:
// from https://github.com/Rayquaza01/nativemessaging#sample
function onReceived(response) {
console.log(response);
}
// runtime.connectNative
var port = browser.runtime.connectNative("application_name");
port.onMessage.addListener(onReceived);
port.postMessage("hello");
// runtime.sendNativeMessage
browser.runtime.sendNativeMessage("puterbot", document).then(onReceived);
// based on https://stackoverflow.com/questions/8882502/how-to-track-dom-change-in-chrome-extension
let observer = new MutationObserver(mutations => {
for(let mutation of mutations) {
browser.runtime.sendNativeMessage("puterbot", mutation).then(onReceived);
}
});
observer.observe(document, { childList: true, subtree: true });
Once this is working, we will want to add the necessary functionality to record.py.
Regarding MutationObserver: https://stackoverflow.com/questions/20383356/mutation-observer-is-undefined It is only supported in Internet Explorer 11.
Currently Figuring out an alternative for Chrome.
It is only supported in Internet Explorer 11.
Are you sure? https://developer.chrome.com/blog/detect-dom-changes-with-mutation-observers/
Some Notes regarding Chrome Extension: https://docs.google.com/presentation/d/106AXW3sBe7-7E-zIggnMnaUKUXWAj_aAuSxBspTDcGk/edit?usp=sharing
Link to information on publishing extension: https://developer.chrome.com/docs/webstore/publish/