Add Chrome browser event in database during recording

[abrichr]

What kind of change does this PR introduce? This PR implements the chrome extension to write the DOM changes (documents' outer HTML of Head and Body and the url) in database.

Summary Motivation: Related Issue: https://github.com/OpenAdaptAI/OpenAdapt/issues/51

Checklist

• [x] My code follows the style guidelines of OpenAdapt
• [x] I have performed a self-review of my code
• [x] If applicable, I have added tests to prove my fix is functional/effective
• [x] I have linted my code locally prior to submission
• [x] I have commented my code, particularly in hard-to-understand areas
• [x] I have made corresponding changes to the documentation (e.g. README.md, requirements.txt)
• [x] New and existing unit tests pass locally with my changes
• [x] How can your code be run and tested? -> https://github.com/OpenAdaptAI/OpenAdapt/pull/744#issuecomment-2212523232

Other information Supercedes https://github.com/OpenAdaptAI/OpenAdapt/pull/364

[KrishPatel13]

An update for Chrome Extension Issue:

After spending about 4-5 hours debugging nativeMessaging, I was not able to fix the issue. I dropped the idea of using nativeMessaging. Reasons:

• Setup is complex, and it is different for all types of Os.
• APIs and example are about 6-7 years old and no debugging features with it, which prevent to make progress.
• Very less resources to find help about nativeMessaging.

Hence, After an hour of research, I recalled that I had used WebSockets in my assignment in CSC309 to communicate between an web server and my java running process. The task there was that whenever the user click submit I had to sent that api request to my java running client and then, connect to db and save some info about the user (registering).

I thought this might relevant for the task that we are working on too.

Hence, I quickly made a web_socket_extension example to demo this idea for constant communication between extension and a desktop running process (in our case a .py file). I made a dumpy record.py that just listening for incoming messages fromt the websocket and prints them.

Commit for the Web Socket "Ping" Example: https://github.com/OpenAdaptAI/OpenAdapt/pull/744/commits/9c749caef0f0ba9f6e8b79fd57d00309b822d114

Some Screenshots for the output:

[KrishPatel13]

My next step would be to log the dom changes instead of "ping" message.

[KrishPatel13]

My goal is to at least, make a prototype by tonight:

record.py (dummy): that runs and save all the dom changes in a sqlite db and once this is done, I will then manipulate the actual record.py next week so that it records the dom changes in our actual db. (our main goal). I hope this sounds like a good plan to tackle this issue :-)

[KrishPatel13]

Update: Now I am able to log DOM changes

Commit: d9f9792cd9416ea2370d499af14d9a9e774d14ca

[KrishPatel13]

Now, I can save all the DOM messages sent by the extension into a db synchronously.

Commit: 1b606fb17fd38056392b82952be472882b1f0aab

Some Screenshots for the work:

and

This commit proves a working prototype for the chrome extension issue.

Now my next steps shall be to integrate this into our actual application.

[KrishPatel13]

Finally fixed all checks and test :-)

[abrichr]

@KrishPatel13 is this ready for re-review?

[KrishPatel13]

@KrishPatel13 is this ready for re-review?

I have not completed this yet. I will let you know. I have add WebSockets creation and connection in openadapt package now and then test it by starting recording. I suspect this will take a bit of time and debugging :)

[KrishPatel13]

@abrichr Just wanted to clarify that we want to include the chrome browser events in the main db (openadapt.db) right ?

Or Is it fine we have a seperate db: say, chrome.db that has its own data ?

How do we want to design it ?

Thank you.

[abrichr]

@KrishPatel13 same db, via crud.py 🙏

[KrishPatel13]

Alright on it right now! Thank you clarifying.

[KrishPatel13]

@abrichr Do you know how to fix this error:

There are many int type variable in config.py, there should be something where I am going wrong:

Thank you

[abrichr]

num_reveal = int(len(str(val))) * pct_reveal

Not sure why it's trying to obfuscate DASHBOARD_CLIENT_PORT, but at least it helped identify this bug 😄

[KrishPatel13]

Ready for Review @abrichr !

[KrishPatel13]

To test overall follow these steps:

1. Go To: Chrome Extension Page

2. Turn on the Developer mode to on (near top right):

3. Click on Load Unpacked (near top left).

4. Select chrome_extension directory

5. You shall see this:

Meaning that the extension is loaded !

Now. start recording, once Everything is started, then go to chrome browser and search for some page, do a few click and then stop the recording. Let it finish successfully.

Once done, you can check the openadapt.db -> table browser_event it should have all of your browser activity logs in it.

You can check for correctness by using sqlite3 cli or using an extension like SQLite Viewer in VS Code to open openadapt.db.

[KrishPatel13]

Excellent work @KrishPatel13 ! I left a few comments, happy to chat about any of it if you like 😄

Thank you !!

Edit: can you please also add these instructions to the README: #744 (comment)

Done Here: dffe40bba4d5c50ba83ca6dac0547c712754511c

[KrishPatel13]

For this comment: https://github.com/OpenAdaptAI/OpenAdapt/pull/744#discussion_r1668785377,

Resolved/done here: fa9e458a6214cd16fc1d5f1848a5c72355e48d70

[KrishPatel13]

@abrichr I have addresses all of your comments.

Before merging, it will be great if we could test this on MacOS.

I will test this again later this week on windows for another time on a frest clone of the repo and checkout this repo and see if this works.

Thereafter, we can think of merging this :-)

Thank you!!

[KrishPatel13]

@abrichr Resolved all of the comments :-)

Ready for review ! :)

[KrishPatel13]

I retested on Windows my recording is here: https://utoronto.zoom.us/rec/share/Oz08EcFFl1eEaF6Jj_iRnR6IvylFMSj9kmXfVMgWrskBFYOicUgNSR5C9j-l_yA1._ObjKzWQ-VRaxQ0I

[KrishPatel13]

Also, we want to update visualize.py and the dashboard to display these events. (https://github.com/OpenAdaptAI/OpenAdapt/pull/744#discussion_r1691404444)

FIxed Visualization here: b3d54eb

I just ran visualize on a browser_event record recording. Here are some screenshots to support:

,