OmniParser
OmniParser copied to clipboard
Published
20 hours ago •
microsoft
microsoft
Add Dockerfile and client.py; deploy to EC2 on AWS via Github Actions
Summary
This PR implements functionality to automatically deploy OmniParser to EC2 on AWS via Docker, Github Actions, and boto3.
It adds a Dockerfile which calls a download.py script for downloading the model weights, a deploy.py which automates deployment to AWS EC2 via Github Actions, and a working example client.py that supports command line usage:
python3.10 -m venv venv && source venv/bin/activate && pip install -r deploy_requirements.txt
python deploy.py start
python client.py "http://<server_ip>:7861"
python deploy.py status
python deploy.py ssh
python deploy.py stop
And API usage:
>>> from deploy import Deploy
>>> from client import predict
>>>
>>> # Start the deployment process
>>> Deploy.start()
>>> instance_id, instance_ip = Deploy.configure_ec2_instance()
>>> assert instance_ip, "Failed to retrieve instance IP address."
>>> print(f"Instance deployed at IP: {instance_ip}")
>>>
>>> # Use the deployed API programmatically with `client.py`
>>> api_url = f"http://{instance_ip}:7861"
>>> result = predict(api_url, "~/Desktop/screenshot.png")
Loaded as API: http://<instance_ip>:7861/ ✔
>>> result_data = result["result_data"]
>>> output_image_path = result['output_image']
>>> parsed_content_list = result_data["parsed_content_list"]
>>> label_coordinates = result_data["label_coordinates"]
>>> len(label_coordinates), len(parsed_content_list)
(80, 80)
It updates the README to include the following section, and another for deploying:
:rocket: Docker Quick Start
Prerequisites:
- CUDA-enabled GPU
- Recent version of Docker
# Build the image (requires CUDA)
sudo docker build -t omniparser .
# Run the image
sudo docker run -d -p 7861:7861 --gpus all --name omniparser-container omniparser
Example successful Github Actions:
- https://github.com/OpenAdaptAI/OmniParser/actions/runs/11805112211/job/32886840898
- https://github.com/OpenAdaptAI/OmniParser/actions/runs/11576276825/job/32224839056
Based on https://github.com/microsoft/SoM/pull/19.
Usage Instructions
- Fork https://github.com/OpenAdaptAI/OmniParser, then clone your fork to your local machine and check out the feat/deploy branch:
git clone https://github.com/<your-username>/OmniParser.git
cd OmniParser
git checkout feat/deploy # not required once PR#52 is merged
Or, until this is merged, you can add a new remote to your existing fork:
git remote add openadapt https://github.com/OpenAdaptAI/OmniParser
git fetch openadapt
git checkout feat/deploy
- Follow the instructions at the top of deploy.py.
Requires the following values:
AWS_ACCESS_KEY_ID=<your aws access key id (required)>
AWS_SECRET_ACCESS_KEY=<your aws secret access key (required)>
AWS_REGION=<your aws region (required)>
GITHUB_OWNER=<your github owner (required)> # e.g. abrichr
GITHUB_REPO=<your github repo (required)> # e.g. OmniParse
GITHUB_TOKEN=<your github token (required)>
PROJECT_NAME=<your project name (required)> # for tagging AWS resources
Example output
Deployment:
(venv) abrichr@MacBook-Pro-5 OmniParser % python deploy.py start
2024-10-29 11:35:55.759 | INFO | __main__:set_github_secret:205 - set secret_name='AWS_ACCESS_KEY_ID'
2024-10-29 11:35:56.303 | INFO | __main__:set_github_secret:205 - set secret_name='AWS_SECRET_ACCESS_KEY'
2024-10-29 11:35:56.849 | INFO | __main__:set_github_secret:205 - set secret_name='SSH_PRIVATE_KEY'
2024-10-29 11:35:58.240 | INFO | __main__:get_or_create_security_group_id:307 - Created security group 'omniparser-SecurityGroup' with ID: sg-046ff0f92bfcf9e6f
2024-10-29 11:35:58.550 | INFO | __main__:get_or_create_security_group_id:311 - Added inbound rules to allow access on ports=[22, 7861]
2024-10-29 11:36:15.800 | INFO | __main__:deploy_ec2_instance:414 - New instance created: ID - i-04075f30de683eee7, IP - 44.198.58.162
2024-10-29 11:37:30.893 | ERROR | __main__:configure_ec2_instance:456 - SSH connection attempt 1 failed: [Errno 60] Operation timed out
2024-10-29 11:37:30.894 | INFO | __main__:configure_ec2_instance:458 - Retrying SSH connection in 10 seconds...
2024-10-29 11:37:41.516 | INFO | __main__:configure_ec2_instance:482 - Executing command: sudo apt-get update
2024-10-29 11:38:36.778 | INFO | __main__:configure_ec2_instance:489 - Command executed successfully
2024-10-29 11:38:36.779 | INFO | __main__:configure_ec2_instance:482 - Executing command: sudo apt-get install -y docker.io
2024-10-29 11:39:58.567 | INFO | __main__:configure_ec2_instance:489 - Command executed successfully
2024-10-29 11:39:58.568 | INFO | __main__:configure_ec2_instance:482 - Executing command: sudo systemctl start docker
2024-10-29 11:39:58.876 | INFO | __main__:configure_ec2_instance:489 - Command executed successfully
2024-10-29 11:39:58.876 | INFO | __main__:configure_ec2_instance:482 - Executing command: sudo systemctl enable docker
2024-10-29 11:39:59.636 | INFO | __main__:configure_ec2_instance:489 - Command executed successfully
2024-10-29 11:39:59.636 | INFO | __main__:configure_ec2_instance:482 - Executing command: sudo usermod -a -G docker ${USER}
2024-10-29 11:39:59.881 | INFO | __main__:configure_ec2_instance:489 - Command executed successfully
2024-10-29 11:39:59.881 | INFO | __main__:configure_ec2_instance:482 - Executing command: sudo curl -L "https://github.com/docker/compose/releases/download/1.29.2/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose ]
2024-10-29 11:40:00.235 | INFO | __main__:configure_ec2_instance:489 - Command executed successfully
2024-10-29 11:40:00.235 | INFO | __main__:configure_ec2_instance:482 - Executing command: sudo chmod +x /usr/local/bin/docker-compose
2024-10-29 11:40:00.405 | INFO | __main__:configure_ec2_instance:489 - Command executed successfully
2024-10-29 11:40:00.406 | INFO | __main__:configure_ec2_instance:482 - Executing command: sudo ln -s /usr/local/bin/docker-compose /usr/bin/docker-compose
2024-10-29 11:40:00.560 | INFO | __main__:configure_ec2_instance:489 - Command executed successfully
2024-10-29 11:40:00.560 | INFO | __main__:configure_ec2_instance:482 - Executing command: sudo docker system prune -af --volumes
2024-10-29 11:40:00.886 | INFO | __main__:configure_ec2_instance:489 - Command executed successfully
2024-10-29 11:40:01.795 | INFO | __main__:get_or_create_security_group_id:276 - Security group 'omniparser-SecurityGroup' already exists: sg-046ff0f92bfcf9e6f
2024-10-29 11:40:02.030 | INFO | __main__:get_or_create_security_group_id:287 - Rule for port 22 already exists
2024-10-29 11:40:02.462 | INFO | __main__:get_or_create_security_group_id:287 - Rule for port 7861 already exists
2024-10-29 11:40:02.824 | INFO | __main__:deploy_ec2_instance:366 - Instance already running: ID - i-04075f30de683eee7, IP - 44.198.58.162
2024-10-29 11:40:02.838 | INFO | __main__:generate_github_actions_workflow:535 - GitHub Actions EC2 workflow file generated successfully.
2024-10-29 11:40:02.928 | INFO | __main__:update_git_remote_with_pat:611 - Git remote 'origin' updated successfully.
diff --git a/.github/workflows/docker-build-ec2.yml b/.github/workflows/docker-build-ec2.yml
new file mode 100644
index 0000000..4669196
--- /dev/null
+++ b/.github/workflows/docker-build-ec2.yml
@@ -0,0 +1,41 @@
+# Autogenerated via deploy.py, do not edit!
+
+name: Docker Build on EC2 Instance for OmniParser
+
+on:
+ push:
+ branches:
+ - feat/deploy
+
+jobs:
+ build:
+ runs-on: ubuntu-latest
+ steps:
+ - name: Checkout code
+ uses: actions/checkout@v2
+
+ - name: SSH and Execute Build on EC2
+ uses: appleboy/ssh-action@master
+ with:
+ command_timeout: "60m"
+ host: 44.198.58.162
+ username: ubuntu
+
+ key: ${{ secrets.SSH_PRIVATE_KEY }}
+
+ script: |
+ rm -rf OmniParser || true
+ git clone https://github.com/OpenAdaptAI/OmniParser
+ cd OmniParser
+ git checkout feat/deploy
+ git pull
+
+ # Stop and remove any existing containers
+ sudo docker stop omniparser-container || true
+ sudo docker rm omniparser-container || true
+
+ # Build the Docker image
+ sudo nvidia-docker build -t omniparser .
+
+ # Run the Docker container on the specified port
+ sudo docker run -d -p 7861:7861 --gpus all --name omniparser-container omniparser
[feat/deploy 7b8d20c] add workflow file
1 file changed, 41 insertions(+)
create mode 100644 .github/workflows/docker-build-ec2.yml
2024-10-29 11:40:03.054 | INFO | __main__:start:645 - Changes committed successfully.
warning: current Git remote contains credentials
Enumerating objects: 6, done.
Counting objects: 100% (6/6), done.
Delta compression using up to 12 threads
Compressing objects: 100% (3/3), done.
Writing objects: 100% (5/5), 891 bytes | 891.00 KiB/s, done.
Total 5 (delta 1), reused 0 (delta 0), pack-reused 0
remote: Resolving deltas: 100% (1/1), completed with 1 local object.
To https://github.com/OpenAdaptAI/OmniParser.git
0657932..7b8d20c feat/deploy -> feat/deploy
branch 'feat/deploy' set up to track 'origin/feat/deploy'.
2024-10-29 11:40:04.649 | INFO | __main__:git_push_set_upstream:582 - Branch 'feat/deploy' pushed and set up to track 'origin/feat/deploy'.
2024-10-29 11:40:04.651 | INFO | __main__:start:660 - Deployment process completed.
2024-10-29 11:40:04.651 | INFO | __main__:start:661 - Check the GitHub Actions at https://github.com/OpenAdaptAI/OmniParser/actions.
2024-10-29 11:40:04.651 | INFO | __main__:start:662 - Once the action is complete, run:
2024-10-29 11:40:04.651 | INFO | __main__:start:663 - python client.py http://44.198.58.162:7861
(venv) abrichr@MacBook-Pro-5 OmniParser % git status
Redeployment:
(venv) abrichr@MacBook-Pro-5 OmniParser % python deploy.py start
2024-10-29 11:16:39.718 | INFO | __main__:set_github_secret:205 - set secret_name='AWS_ACCESS_KEY_ID'
2024-10-29 11:16:40.346 | INFO | __main__:set_github_secret:205 - set secret_name='AWS_SECRET_ACCESS_KEY'
2024-10-29 11:16:41.268 | INFO | __main__:set_github_secret:205 - set secret_name='SSH_PRIVATE_KEY'
2024-10-29 11:16:41.952 | INFO | __main__:get_or_create_security_group_id:276 - Security group 'omniparser-SecurityGroup' already exists: sg-0f89ae01d0689defd
2024-10-29 11:16:42.158 | INFO | __main__:get_or_create_security_group_id:287 - Rule for port 22 already exists
2024-10-29 11:16:42.512 | INFO | __main__:get_or_create_security_group_id:287 - Rule for port 7861 already exists
2024-10-29 11:16:42.895 | INFO | __main__:deploy_ec2_instance:366 - Instance already running: ID - i-09c48e242dba6fa58, IP - 34.206.53.77
2024-10-29 11:16:43.429 | INFO | __main__:configure_ec2_instance:482 - Executing command: sudo apt-get update
2024-10-29 11:16:45.848 | INFO | __main__:configure_ec2_instance:489 - Command executed successfully
2024-10-29 11:16:45.848 | INFO | __main__:configure_ec2_instance:482 - Executing command: sudo apt-get install -y docker.io
2024-10-29 11:16:46.476 | INFO | __main__:configure_ec2_instance:489 - Command executed successfully
2024-10-29 11:16:46.476 | INFO | __main__:configure_ec2_instance:482 - Executing command: sudo systemctl start docker
2024-10-29 11:16:46.625 | INFO | __main__:configure_ec2_instance:489 - Command executed successfully
2024-10-29 11:16:46.626 | INFO | __main__:configure_ec2_instance:482 - Executing command: sudo systemctl enable docker
2024-10-29 11:16:47.557 | INFO | __main__:configure_ec2_instance:489 - Command executed successfully
2024-10-29 11:16:47.558 | INFO | __main__:configure_ec2_instance:482 - Executing command: sudo usermod -a -G docker ${USER}
2024-10-29 11:16:47.871 | INFO | __main__:configure_ec2_instance:489 - Command executed successfully
2024-10-29 11:16:47.872 | INFO | __main__:configure_ec2_instance:482 - Executing command: sudo curl -L "https://github.com/docker/compose/releases/download/1.29.2/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
2024-10-29 11:16:48.156 | INFO | __main__:configure_ec2_instance:489 - Command executed successfully
2024-10-29 11:16:48.156 | INFO | __main__:configure_ec2_instance:482 - Executing command: sudo chmod +x /usr/local/bin/docker-compose
2024-10-29 11:16:48.432 | INFO | __main__:configure_ec2_instance:489 - Command executed successfully
2024-10-29 11:16:48.433 | INFO | __main__:configure_ec2_instance:482 - Executing command: sudo ln -s /usr/local/bin/docker-compose /usr/bin/docker-compose
2024-10-29 11:16:48.624 | ERROR | __main__:configure_ec2_instance:498 - Error in command: sudo ln -s /usr/local/bin/docker-compose /usr/bin/docker-compose, Exit Status: 1, Error: b"ln: failed to create symbolic link '/usr/bin/docker-compose': File exists\n"
2024-10-29 11:16:48.625 | INFO | __main__:configure_ec2_instance:482 - Executing command: sudo docker system prune -af --volumes
2024-10-29 11:16:55.334 | INFO | __main__:configure_ec2_instance:489 - Command executed successfully
2024-10-29 11:16:56.012 | INFO | __main__:get_or_create_security_group_id:276 - Security group 'omniparser-SecurityGroup' already exists: sg-0f89ae01d0689defd
2024-10-29 11:16:56.272 | INFO | __main__:get_or_create_security_group_id:287 - Rule for port 22 already exists
2024-10-29 11:16:56.622 | INFO | __main__:get_or_create_security_group_id:287 - Rule for port 7861 already exists
2024-10-29 11:16:56.959 | INFO | __main__:deploy_ec2_instance:366 - Instance already running: ID - i-09c48e242dba6fa58, IP - 34.206.53.77
2024-10-29 11:16:56.968 | INFO | __main__:generate_github_actions_workflow:535 - GitHub Actions EC2 workflow file generated successfully.
2024-10-29 11:16:57.015 | INFO | __main__:update_git_remote_with_pat:611 - Git remote 'origin' updated successfully.
2024-10-29 11:16:57.044 | INFO | __main__:start:647 - No changes to commit.
branch 'feat/deploy' set up to track 'origin/feat/deploy'.
Everything up-to-date
2024-10-29 11:16:57.968 | INFO | __main__:git_push_set_upstream:582 - Branch 'feat/deploy' pushed and set up to track 'origin/feat/deploy'.
2024-10-29 11:16:57.969 | INFO | __main__:start:660 - Deployment process completed.
2024-10-29 11:16:57.969 | INFO | __main__:start:661 - Check the GitHub Actions at https://github.com/OpenAdaptAI/OmniParser/actions.
2024-10-29 11:16:57.969 | INFO | __main__:start:662 - Once the action is complete, run:
2024-10-29 11:16:57.969 | INFO | __main__:start:663 - python client.py http://34.206.53.77:7861 <path>/<to>/<image.png>
Command line client:
(venv) abrichr@MacBook-Pro-5 OmniParser % python client.py http://44.198.58.162:7861 ~/oa/OpenAdapt/tests/assets/excel.png
Loaded as API: http://44.198.58.162:7861/ ✔
2024-10-29 18:55:51.336 | INFO | __main__:predict:61 - Prediction completed successfully
2024-10-29 18:55:51.337 | INFO | __main__:predict:70 - label_coordinates={'0': [0.12276943773031235, 0.002296212362125516, 0.045681655406951904, 0.028702640905976295], '1': [0.48429691791534424, 0.008610793389379978, 0.030335474759340286, 0.01607
3478385806084], '2': [0.0060670948587358, 0.043628014624118805, 0.02890792302787304, 0.016073478385806084], '3': [0.049250528216362, 0.043628014624118805, 0.02855103462934494, 0.016073478385806084], '4': [0.0935046374797821, 0.043628014624118805, 0.0
242683794349432, 0.016073478385806084], '5': [0.13276232779026031, 0.04247990995645523, 0.057102058082818985, 0.02238805964589119], '6': [0.20628122985363007, 0.043628014624118805, 0.04282655194401741, 0.016073478385806084], '7': [0.26552459597587585
, 0.043628014624118805, 0.021770164370536804, 0.016073478385806084], '8': [0.3029978573322296, 0.04247990995645523, 0.03283369168639183, 0.017221584916114807], '9': [0.3508208394050598, 0.043628014624118805, 0.02319771610200405, 0.016073478385806084]
, '10': [0.39971449971199036, 0.043628014624118805, 0.03354746475815773, 0.016073478385806084], '11': [0.9625267386436462, 0.04649827629327774, 0.026409707963466644, 0.016073478385806084], '12': [0.08422555029392242, 0.08553387224674225, 0.0578158460
5574608, 0.01836968958377838], '13': [0.4057815968990326, 0.08610792458057404, 0.03854389861226082, 0.017221584916114807], '14': [0.4921484589576721, 0.08553387224674225, 0.03354746475815773, 0.016073478385806084], '15': [0.012491078115999699, 0.1194
0297484397888, 0.021770164370536804, 0.014925390481948853], '16': [0.08529621362686157, 0.11997702717781067, 0.06495360285043716, 0.02238805964589119], '17': [0.40613847970962524, 0.1228473111987114, 0.0239114910364151, 0.016073478385806084], '18': [
0.4382583796977997, 0.1228473111987114, 0.026409707963466644, 0.014925372786819935], '19': [0.6145610213279724, 0.11710677295923233, 0.04318344220519066, 0.017221584916114807], '20': [0.6623840928077698, 0.11940297484397888, 0.027837172150611877, 0.0
14925390481948853], '21': [0.7041398882865906, 0.11997703462839127, 0.015703069046139717, 0.01205510925501585], '22': [0.7494646906852722, 0.11997703462839127, 0.022483939304947853, 0.01205510925501585], '23': [0.7847965955734253, 0.11997703462839127
, 0.0239114910364151, 0.01205510925501585], '24': [0.8183440566062927, 0.1188289225101471, 0.02855103462934494, 0.015499443747103214], '25': [0.8954318165779114, 0.11997703462839127, 0.017130620777606964, 0.01205510925501585], '26': [0.93076366186141
97, 0.11997703462839127, 0.016416845843195915, 0.01205510925501585], '27': [0.6145610213279724, 0.13318024575710297, 0.04139900207519531, 0.016073478385806084], '28': [0.6602426767349243, 0.13375429809093475, 0.031406138092279434, 0.01205510925501585
], '29': [0.699857234954834, 0.13318024575710297, 0.0242683794349432, 0.014925372786819935], '30': [0.8979300260543823, 0.13375429809093475, 0.01927194930613041, 0.01205510925501585], '31': [0.9304068684577942, 0.13318024575710297, 0.0249821562319993
97, 0.014925372786819935], '32': [0.02105638198554516, 0.16590125858783722, 0.059957172721624374, 0.01836968958377838], '33': [0.08422557264566422, 0.16532720625400543, 0.3875803053379059, 0.01894374191761017], '34': [0.9107779264450073, 0.1659012585
8783722, 0.07708779722452164, 0.017221584916114807], '35': [0.050321198999881744, 0.19517795741558075, 0.07566024363040924, 0.024684270843863487], '36': [0.18022841215133667, 0.30022960901260376, 0.10706637054681778, 0.051090698689222336], '37': [0.3
1334760785102844, 0.3053961396217346, 0.12312633544206619, 0.04477611929178238], '38': [0.4464668035507202, 0.3071182668209076, 0.055317629128694534, 0.03731343150138855], '39': [0.581013560295105, 0.305970162153244, 0.06745181977748871, 0.0373134315
0138855], '40': [0.014989293180406094, 0.3639494776725769, 0.013918629847466946, 0.029276693239808083], '41': [0.09671662747859955, 0.3559127151966095, 0.08065667748451233, 0.0482204370200634], '42': [0.29407569766044617, 0.3639494776725769, 0.013918
586075305939, 0.029276693239808083], '43': [0.4275517165660858, 0.3639494776725769, 0.013204853981733322, 0.029276693239808083], '44': [0.014989293180406094, 0.4161882996559143, 0.013918629847466946, 0.02985074557363987], '45': [0.09743039309978485,
0.4087255895137787, 0.07994289696216583, 0.0482204370200634], '46': [0.29407569766044617, 0.4161882996559143, 0.013918586075305939, 0.02985074557363987], '47': [0.4275517463684082, 0.4161882996559143, 0.013918629847466946, 0.02985074557363987], '48':
[0.0970735102891922, 0.4609643816947937, 0.08065667748451233, 0.04879448935389519], '49': [0.29407569766044617, 0.46785303950309753, 0.013918586075305939, 0.030998852103948593], '50': [0.4275517463684082, 0.4690011441707611, 0.013918629847466946, 0.
02985074557363987], '51': [0.5613847374916077, 0.4690011441707611, 0.013561741448938847, 0.02985074557363987], '52': [0.6948608160018921, 0.46785303950309753, 0.013918629847466946, 0.030998852103948593], '53': [0.09671661257743835, 0.5143513679504395
, 0.08137047290802002, 0.0482204370200634], '54': [0.29407569766044617, 0.5206659436225891, 0.013918586075305939, 0.030998852103948593], '55': [0.4275517463684082, 0.5206659436225891, 0.013918629847466946, 0.030998852103948593], '56': [0.561384737491
6077, 0.5206659436225891, 0.013561741448938847, 0.030998852103948593], '57': [0.6948608160018921, 0.5206659436225891, 0.013918629847466946, 0.030998852103948593], '58': [0.008922198787331581, 0.7824339866638184, 0.02783725969493389, 0.036739379167556
76], '59': [0.008208423852920532, 0.8363949656486511, 0.027123482897877693, 0.034443169832229614], '60': [0.008922199718654156, 0.8892077803611755, 0.027123482897877693, 0.034443169832229614], '61': [0.055317629128694534, 0.9500574469566345, 0.029621
697962284088, 0.014925372786819935], '62': [0.015346181578934193, 0.9787600636482239, 0.027480371296405792, 0.017221584916114807], '63': [0.06780871003866196, 0.9787600636482239, 0.10135617107152939, 0.017221584916114807], '64': [0.9586009979248047,
0.9787600636482239, 0.024625267833471298, 0.014925372786819935], '65': [0.6651304364204407, 0.08138538151979446, 0.022024760022759438, 0.03468729183077812], '66': [0.7468552589416504, 0.07959043234586716, 0.023525850847363472, 0.028781019151210785],
'67': [0.6991221308708191, 0.08180632442235947, 0.018437061458826065, 0.03362924978137016], '68': [0.7826605439186096, 0.0806342288851738, 0.02190661057829857, 0.027261780574917793], '69': [0.6214930415153503, 0.08046738058328629, 0.01830035261809826
, 0.031031927093863487], '70': [0.8183373212814331, 0.07944951951503754, 0.021337995305657387, 0.03104429505765438], '71': [0.09984152764081955, 0.9439650177955627, 0.02320893481373787, 0.028721420094370842], '72': [0.38827988505363464, 0.11905978620
052338, 0.01550249382853508, 0.02412971667945385], '73': [0.866210401058197, 0.08966667205095291, 0.015609142370522022, 0.03238759562373161], '74': [0.3112151622772217, 0.07979363203048706, 0.01715545356273651, 0.030124405398964882], '75': [0.0003453
716926742345, 0.9426923394203186, 0.044026125222444534, 0.029541436582803726], '76': [0.6377907395362854, 0.2446022927761078, 0.015758832916617393, 0.03145069256424904], '77': [0.3110503554344177, 0.12188976258039474, 0.017567103728652, 0.02876365743
5774803], '78': [0.8960394263267517, 0.08187582343816757, 0.020451003685593605, 0.02726871706545353], '79': [0.9309751987457275, 0.0796903744339943, 0.02361646667122841, 0.029221685603260994]}
2024-10-29 18:55:51.339 | INFO | __main__:predict:71 - parsed_content_list=['Text Box ID 0: 5 v G', 'Text Box ID 1: Book1', 'Text Box ID 2: Home', 'Text Box ID 3: Insert', 'Text Box ID 4: Draw', 'Text Box ID 5: Page Layout', 'Text Box ID 6: Formu
las', 'Text Box ID 7: Data', 'Text Box ID 8: Review', 'Text Box ID 9: View', 'Text Box ID 10: Tell me', 'Text Box ID 11: Share', 'Text Box ID 12: Calibri (Body)', 'Text Box ID 13: Wrap Text', 'Text Box ID 14: General', 'Text Box ID 15: Paste', 'Text
Box ID 16: B I W ~ [', 'Text Box ID 17: Merge', 'Text Box ID 18: Center', 'Text Box ID 19: Conditional', 'Text Box ID 20: Format', 'Text Box ID 21: Cell', 'Text Box ID 22: Insert', 'Text Box ID 23: Delete', 'Text Box ID 24: Format', 'Text Box ID 25
: Sort', 'Text Box ID 26: Find', 'Text Box ID 27: Formatting', 'Text Box ID 28: as Table', 'Text Box ID 29: Styles', 'Text Box ID 30: Filter', 'Text Box ID 31: Select', 'Text Box ID 32: Office Update', 'Text Box ID 33: To keep up-to-date with securit
y updates, fixes, and improvements, choose Check for Updates:', 'Text Box ID 34: Check for Updates', 'Text Box ID 35: 4* v fx', 'Text Box ID 36: Marketing', 'Text Box ID 37: Engineering', 'Text Box ID 38: Sales', 'Text Box ID 39: Admin', 'Text Box ID
40: 2', 'Text Box ID 41: 13-May', 'Text Box ID 42: 2', 'Text Box ID 43: 3', 'Text Box ID 44: 3', 'Text Box ID 45: 14-May', 'Text Box ID 46: 3', 'Text Box ID 47: 2', 'Text Box ID 48: 15-May', 'Text Box ID 49: 3', 'Text Box ID 50: 3', 'Text Box ID 51:
3', 'Text Box ID 52: 3', 'Text Box ID 53: 16-May', 'Text Box ID 54: 2', 'Text Box ID 55: 2', 'Text Box ID 56: 2', 'Text Box ID 57: 1', 'Text Box ID 58: 10', 'Text Box ID 59: 11', 'Text Box ID 60: 12', 'Text Box ID 61: Sheet1', 'Text Box ID 62: Ready
', 'Text Box ID 63: Accessibility: Good to go', 'Text Box ID 64: 287%', 'Icon Box ID 65: a drawing or painting tool.', 'Icon Box ID 66: a layout or table layout.', 'Icon Box ID 67: a drawing or painting tool.', 'Icon Box ID 68: expanding or opening a
window or view.', 'Icon Box ID 69: a table or spreadsheet.', 'Icon Box ID 70: a table or spreadsheet.', 'Icon Box ID 71: adding or creating a new item.', 'Icon Box ID 72: Decrease', 'Icon Box ID 73: Download', 'Icon Box ID 74: a menu or list.', 'Ico
n Box ID 75: navigating to the previous item or screen.', 'Icon Box ID 76: the "E" function.', 'Icon Box ID 77: Justified', 'Icon Box ID 78: Text formatting options.', 'Icon Box ID 79: a search function.']
2024-10-29 18:55:51.341 | INFO | __main__:predict:77 - Parsed content saved to: result_data_20241029_185545.json
2024-10-29 18:55:51.343 | INFO | __main__:predict:83 - Output image saved to: output_image_20241029_185545.png
(venv) abrichr@MacBook-Pro-5 OmniParser % open output_image.png
(venv) abrichr@MacBook-Pro-5 OmniParser % head result_data_20241029_185545.json
{
"label_coordinates": {
"0": [
0.12276943773031235,
0.002296212362125516,
0.045681655406951904,
0.028702640905976295
],
"1": [
0.48429691791534424,
(venv) abrichr@MacBook-Pro-5 OmniParser % tail result_data_20241029_185545.json
"Icon Box ID 72: Decrease",
"Icon Box ID 73: Download",
"Icon Box ID 74: a menu or list.",
"Icon Box ID 75: navigating to the previous item or screen.",
"Icon Box ID 76: the \"E\" function.",
"Icon Box ID 77: Justified",
"Icon Box ID 78: Text formatting options.",
"Icon Box ID 79: a search function."
]
}%
Client API:
>>> from client import predict
>>> from PIL import Image
>>>
>>> result = predict("http://98.80.191.41:7861", "~/Desktop/screenshot.png")
Loaded as API: http://98.80.191.41:7861/ ✔
>>> result_data = result["result_data"]
>>> label_coordinates = result_data["label_coordinates"]
>>> parsed_content_list = result_data["parsed_content_list"]
>>> len(label_coordinates), len(parsed_content_list)
(80, 80)
>>> output_image_path = result['output_image']
>>> Image.open(output_image_path).show()
Status:
(venv) abrichr@MacBook-Pro-5 OmniParser % python deploy.py status
2024-10-29 18:04:25.726 | INFO | __main__:status:748 - Instance ID: i-04075f30de683eee7, State: running, HTTP URL: http://44.198.58.162:7861
Termination:
(venv) abrichr@MacBook-Pro-5 OmniParser % python deploy.py stop
2024-10-29 11:25:03.330 | INFO | __main__:stop:716 - Terminating instance: ID - i-09c48e242dba6fa58
2024-10-29 11:31:24.090 | INFO | __main__:stop:719 - Instance i-09c48e242dba6fa58 terminated successfully.
2024-10-29 11:31:24.604 | INFO | __main__:stop:724 - Deleted security group: omniparser-SecurityGroup
(venv) abrichr@MacBook-Pro-5 OmniParser %
(Note that all AWS configuration is automated.)
