OpenAdapt icon indicating copy to clipboard operation
OpenAdapt copied to clipboard

Published 20 hours ago • verified publisher icon OpenAdaptAI

feat: CompletionProvider API

Open FFFiend opened this issue 2 years ago • 20 comments

What kind of change does this PR introduce? Added a rough sketch of what the infrastructure surrounding the models we use might look like. Addresses #252 , #69 and #368 and may also be tangentially/directly relevant to other issues

Checklist

  • [x] My code follows the style guidelines of OpenAdapt
  • [ ] I have performed a self-review of my code
  • [ ] If applicable, I have added tests to prove my fix is functional/effective
  • [ ] I have linted my code locally prior to submission
  • [ ] I have commented my code, particularly in hard-to-understand areas
  • [ ] I have made corresponding changes to the documentation (e.g. README.md, requirements.txt)
  • [ ] New and existing unit tests pass locally with my changes

How can your code be run and tested?

Other information

FFFiend avatar Jul 12 '23 02:07 FFFiend

Thanks for getting started on this @FFFiend !

What do you think about:

class Modality:
    TEXT = "TEXT"
    IMAGE = "IMAGE"

MODALITY_BY_DB_MODEL = {
    models.ActionEvent: Modality.TEXT,
    models.Screenshot: Modality.IMAGE,
    models.WindowEvent: Modality.TEXT,
}

ML_MODELS = [
    {
        "name": "gpt-4",
        "provider": openai.ml.openai,
        # "tune_func": openadapt.ml.openai.gpt4.tune,
        # "infer_func": openadapt.ml.openai.gpt4.infer,
        "modalities": openadapt.ml.openai.gpt4.MODALITIES,  #  gpt4/__init__.py: MODALITIES =  [Modality.TEXT],
    },
     {
        "name": "BLIP2",
        "provider": openai.ml.huggingface,
        # "tune_func": openadapt.ml.huggingface.blip2.tune,
        # "infer_func": openadapt.ml.huggingface.blip2.infer,
        "modalities": openadapt.ml.huggingface.blip2.MODALITIES,  #  huggingface/__init__.py: MODALITIES =  [Modality.TEXT, Modality.IMAGE],
    },
    ...
]

    
def select_ml_model(recording_ids):
    recording_data = get_data_by_modality(recording_ids)
    for ml_model in ML_MODELS:
        # evaluate the performance of the given models autoregressively on an entire recording
        # (optionally with start_timestamp/end_timestamp parameters for evaluating on a section of a recording)

abrichr avatar Jul 18 '23 00:07 abrichr

@FFFiend please see:

https://www.youtube.com/watch?v=DxInMGIvkp0

See also: https://mldsai.slack.com/archives/C050869D3K4/p1689640271500029

https://twitter.com/Yampeleg/status/1668337702440165376

abrichr avatar Jul 18 '23 01:07 abrichr

Some notes for myself:

It is possible for there to be window states with no action events associated with them, i.e. we can have

recording_window_events = get_window_events(recording)
recording_action_events = get_action_events(recording)
len(recording_window_events) != len(recording_action_events)

so we only concern ourselves with recording_action_events since the window pairing already exists in that object. Next up:

Assuming the length of the recording is sufficiently small, TODO: Distill recording_action_events to obtain pairing as discussed in Option 1 in 1-on-1 notes and minimize information loss

Afterwards, create {action:.., window: ... } JSON from recording_action_events and feed to model.

@abrichr Have I laid out the proper approach above? I also had a question, for GPT3.5-Davinci we feed strings in the form of {prompt: prompt_str, completion: completion_str} and I apologize if I didn't get clarification on this during the meeting but, given a parsed JSON as above (i.e v_{i} = {action:_{i}.., window_{i}:...}) where v_{i} is the i-th action/window pairing inside recorded_action_events , would this v_{i} be our prompt_str? And if so, what would the completion_str be?

Edit: Also, is it okay if we set the 'data' key's value within each action's paired window event to {} ? For reference, this is what the 'data' key contains for some Window event:

sanitize(test_acts[i])[window_event]['state']['data']={'AXTitle': 'OpenAdapt — poetry shell ▸ Python — 80×24', 'AXChildrenInNavigationOrder': [{'AXEnabled': True, 'AXFrame': {'x': 562.0, 'y': 176.0, 'w': 14.0, 'h': 16.0, 'type': 'kAXValueCGRectType'}, 'AXSize': {'x': None, 'y': None, 'w': 14.0, 'h': 16.0, 'type': 'kAXValueCGSizeType'}, 'AXRole': 'AXButton', 'AXPosition': {'x': 562.0, 'y': 176.0, 'w': None, 'h': None, 'type': 'kAXValueCGPointType'}, 'AXRoleDescription': 'close button', 'AXSubrole': 'AXCloseButton'}, {'AXFrame': {'x': 582.0, 'y': 176.0, 'w': 14.0, 'h': 16.0, 'type': 'kAXValueCGRectType'}, 'AXPosition': {'x': 582.0, 'y': 176.0, 'w': None, 'h': None, 'type': 'kAXValueCGPointType'}, 'AXRoleDescription': 'minimize button', 'AXSubrole': 'AXMinimizeButton'}, {'AXFrame': {'x': 602.0, 'y': 176.0, 'w': 14.0, 'h': 16.0, 'type': 'kAXValueCGRectType'}, 'AXHelp': 'this button also has an action to zoom the window', 'AXPosition': {'x': 602.0, 'y': 176.0, 'w': None, 'h': None, 'type': 'kAXValueCGPointType'}, 'AXRoleDescription': 'full screen button', 'AXSubrole': 'AXFullScreenButton'}, {'AXFrame': {'x': 684.0, 'y': 175.0, 'w': 16.0, 'h': 16.0, 'type': 'kAXValueCGRectType'}, 'AXSize': {'x': None, 'y': None, 'w': 16.0, 'h': 16.0, 'type': 'kAXValueCGSizeType'}, 'AXRole': 'AXImage', 'AXPosition': {'x': 684.0, 'y': 175.0, 'w': None, 'h': None, 'type': 'kAXValueCGPointType'}, 'AXTitle': 'OpenAdapt', 'AXRoleDescription': 'image'}, {'AXFrame': {'x': 702.0, 'y': 175.0, 'w': 293.0, 'h': 16.0, 'type': 'kAXValueCGRectType'}, 'AXPosition': {'x': 702.0, 'y': 175.0, 'w': None, 'h': None, 'type': 'kAXValueCGPointType'}, 'AXNumberOfCharacters': 41, 'AXSelectedTextRange': {'x': None, 'y': None, 'w': None, 'h': 0.0, 'type': 'kAXValueCFRangeType'}, 'AXRole': 'AXStaticText', 'AXVisibleCharacterRange': {'x': None, 'y': None, 'w': None, 'h': 41.0, 'type': 'kAXValueCFRangeType'}, 'AXSize': {'x': None, 'y': None, 'w': 293.0, 'h': 16.0, 'type': 'kAXValueCGSizeType'}, 'AXRoleDescription': 'text'}, {'AXFrame': {'x': 555.0, 'y': 198.0, 'w': 570.0, 'h': 343.0, 'type': 'kAXValueCGRectType'}, 'AXChildren': [{'AXChildren': [{'AXSelectedTextRange': {'x': None, 'y': None, 'w': None, 'h': 0.0, 'type': 'kAXValueCFRangeType'}, 'AXNumberOfCharacters': 110317, 'AXFrame': {'x': 555.0, 'y': None, 'w': 570.0, 'h': 32907.34668, 'type': 'kAXValueCGRectType'}, 'AXDescription': 'shell', 'AXSize': {'x': None, 'y': None, 'w': 570.0, 'h': 32907.34668, 'type': 'kAXValueCGSizeType'}, 'AXRole': 'AXTextArea', 'AXInsertionPointLineNumber': 2349, 'AXPosition': {'x': 555.0, 'y': None, 'w': None, 'h': None, 'type': 'kAXValueCGPointType'}, 'AXRoleDescription': 'text entry area', 'AXValue': 'Last login: Wed Jul 19 19:20:24 on ttys000\nowaiszahid@Owaiss-MacBook-Pro modelapi % git clone https://github.com/Owais-Enhancement-Org/OpenAdapt.git\nCloning into \'OpenAdapt\'...\nremote: Enumerating objects: 3311, done.\nremote: Counting objects: 100% (1505/1505), done.\nremote: Compressing objects: 100% (345/345), done.\nremote: Total 3311 (delta 1313), reused 1215 (delta 1141), pack-reused 1806\nReceiving objects: 100% (3311/3311), 24.74 MiB | 29.08 MiB/s, done.\nResolving deltas: 100% (2127/2127), done.\nowaiszahid@Owaiss-MacBook-Pro modelapi % ls\nOpenAdapt\nowaiszahid@Owaiss-MacBook-Pro modelapi % cd openadapt\nowaiszahid@Owaiss-MacBook-Pro openadapt % git checkout models_branch\nBranch \'models_branch\' set up to track remote branch \'models_branch\' from \'origin\'.\nSwitched to a new branch \'models_branch\'\nowaiszahid@Owaiss-MacBook-Pro openadapt % ls\nCHANGELOG.md\t\talembic\t\t\tpermissions_in_macOS.md\nCONTRIBUTING.md\t\talembic.ini\t\tpoetry.lock\nLICENSE\t\t\tassets\t\t\tpyproject.toml\nREADME.md\t\tinstall\t\t\tsetup.py\nSETUP.md\t\topenadapt\t\ttests\nowaiszahid@Owaiss-MacBook-Pro openadapt % ls\nCHANGELOG.md\t\talembic\t\t\tpermissions_in_macOS.md\nCONTRIBUTING.md\t\talembic.ini\t\tpoetry.lock\nLICENSE\t\t\tassets\t\t\tpyproject.toml\nREADME.md\t\tinstall\t\t\tsetup.py\nSETUP.md\t\topenadapt\t\ttests\nowaiszahid@Owaiss-MacBook-Pro openadapt % cd openadapt\nowaiszahid@Owaiss-MacBook-Pro openadapt % ls\n__init__.py\tcrud.py\t\tmodels.py\tscrub.py\twindow\napp\t\tdb.py\t\tplayback.py\tstart.py\ncache.py\tevents.py\trecord.py\tstrategies\ncommon.py\textensions\treplay.py\tutils.py\nconfig.py\tml_models\tscripts\t\tvisualize.py\nowaiszahid@Owaiss-MacBook-Pro openadapt % cd ml_models\nowaiszahid@Owaiss-MacBook-Pro ml_models % ls\n__init__.py\thuggingface\tml_models.py\topenai\nowaiszahid@Owaiss-MacBook-Pro ml_models % black ml_models.py\nerror: cannot format ml_models.py: Cannot parse: 21:0: ]\n\nOh no! 💥 💔 💥\n1 file failed to reformat.\nowaiszahid@Owaiss-MacBook-Pro ml_models % black ml_models.py\nreformatted ml_models.py\n\nAll done! ✨ 🍰 ✨\n1 file reformatted.\nowaiszahid@Owaiss-MacBook-Pro ml_models % cd ..\nowaiszahid@Owaiss-MacBook-Pro openadapt % cd ..\nowaiszahid@Owaiss-MacBook-Pro openadapt % ls\nCHANGELOG.md\t\talembic\t\t\tpermissions_in_macOS.md\nCONTRIBUTING.md\t\talembic.ini\t\tpoetry.lock\nLICENSE\t\t\tassets\t\t\tpyproject.toml\nREADME.md\t\tinstall\t\t\tsetup.py\nSETUP.md\t\topenadapt\t\ttests\nowaiszahid@Owaiss-MacBook-Pro openadapt % cd ..\nowaiszahid@Owaiss-MacBook-Pro modelapi % ls\nOpenAdapt\nowaiszahid@Owaiss-MacBook-Pro modelapi % cd openadapt\nowaiszahid@Owaiss-MacBook-Pro openadapt % git branch\n  main\n* models_branch\nowaiszahid@Owaiss-MacBook-Pro openadapt % poetry shell\nCreating virtualenv openadapt-AV4LSnQD-py3.10 in /Users/owaiszahid/Library/Caches/pypoetry/virtualenvs\nSpawning shell within /Users/owaiszahid/Library/Caches/pypoetry/virtualenvs/openadapt-AV4LSnQD-py3.10\nowaiszahid@Owaiss-MacBook-Pro openadapt % emulate bash -c \'. /Users/owaiszahid/Library/Caches/pypoetry/virtualenvs/openadapt-AV4LSnQD-py3.10/bin/activate\'\n(openadapt-py3.10) owaiszahid@Owaiss-MacBook-Pro openadapt % poetry install\nInstalling dependencies from lock file\n\nPackage operations: 226 installs, 1 update, 0 removals\n\n  • Installing catalogue (2.0.8)\n  • Installing certifi (2023.5.7)\n  • Installing charset-normalizer (3.1.0)\n  • Installing click (8.1.3)\n  • Installing cymem (2.0.7)\n  • Installing exceptiongroup (1.1.1)\n  • Installing idna (3.4)\n  • Installing murmurhash (1.0.9)\n  • Installing numpy (1.25.0)\n  • Installing pydantic (1.7.4)\n  • Installing sniffio (1.3.0)\n  • Installing srsly (2.4.6)\n  • Installing typing-extensions (4.7.0)\n  • Installing urllib3 (2.0.3)\n  • Installing anyio (3.7.0)\n  • Installing blis (0.7.9)\n  • Installing confection (0.1.0)\n  • Installing filelock (3.12.2)\n  • Installing fsspec (2023.6.0)\n  • Installing markupsafe (2.1.3)\n  • Installing mpmath (1.3.0)\n  • Installing packaging (23.1)\n  • Installing preshed (3.0.8)\n  • Installing pyobjc-core (9.2)\n  • Installing pyyaml (6.0)\n  • Installing requests (2.31.0)\n  • Updating setuptools (67.8.0 -> 68.0.0)\n  • Installing six (1.16.0)\n  • Installing smart-open (6.3.0)\n  • Installing tqdm (4.64.0)\n  • Installing typer (0.9.0)\n  • Installing wasabi (1.1.2)\n  • Installing asttokens (2.2.1)\n  • Installing bidict (0.22.1)\n  • Installing executing (1.2.0)\n  • Installing frozenlist (1.3.3)\n  • Installing huggingface-hub (0.15.1)\n  • Installing humanfriendly (10.0)\n  • Installing jinja2 (3.1.2)\n  • Installing langcodes (3.3.0)\n  • Installing lazy-object-proxy (1.9.0)\n  • Installing multidict (6.0.4)\n  • Installing networkx (3.1)\n  • Installing parso (0.8.3)\n  • Installing pathy (0.10.2)\n  • Installing pillow (9.5.0)\n  • Installing ptyprocess (0.7.0)\n  • Installing pure-eval (0.2.2)\n  • Installing pyobjc-framework-cocoa (9.2)\n  • Installing pyrect (0.2.0)\n  • Installing python-engineio (4.4.1)\n  • Installing regex (2023.6.3)\n  • Installing requests-file (1.5.1)\n  • Installing spacy-legacy (3.0.12)\n  • Installing spacy-loggers (1.0.4)\n  • Installing starlette (0.27.0)\n  • Installing sympy (1.12)\n  • Installing thinc (8.1.10)\n  • Installing tokenizers (0.13.3)\n  • Installing traitlets (5.9.0)\n  • Installing wcwidth (0.2.6)\n  • Installing wrapt (1.15.0)\n  • Installing aiosignal (1.3.1)\n  • Installing altgraph (0.17.3)\n  • Installing appnope (0.1.3)\n  • Installing astroid (2.15.5)\n  • Installing async-timeout (4.0.2)\n  • Installing attrs (23.1.0)\n  • Installing backcall (0.2.0)\n  • Installing bottle (0.12.25)\n  • Installing chardet (5.1.0)\n  • Installing coloredlogs (15.0.1)\n  • Installing contourpy (1.1.0)\n  • Installing cycler (0.11.0)\n  • Installing decorator (4.4.2)\n  • Installing dill (0.3.6)\n  • Installing distlib (0.3.6)\n  • Installing docopt (0.6.2)\n  • Installing fastapi (0.98.0)\n  • Installing flatbuffers (23.5.26)\n  • Installing fonttools (4.40.0): Installing...\n  • Installing h11 (0.14.0)\n  • Installing httptools (0.5.0)\n  • Installing isort (5.12.0)\n  • Installing jedi (0.18.2): Installing...\n  • Installing joblib (1.3.1)\n  • Installing kiwisolver (1.4.4)\n  • Installing lxml (4.9.2): Installing...\n  • Installing lxml (4.9.2)\n  • Installing matplotlib-inline (0.1.6)\n  • Installing mccabe (0.7.0)\n  • Installing pexpect (4.8.0)\n  • Installing phonenumbers (8.13.15): Installing...\n  • Installing pickleshare (0.7.5)\n  • Installing platformdirs (3.8.0)\n  • Installing prompt-toolkit (3.0.38)\n  • Installing protobuf (4.23.3)\n  • Installing proxy-tools (0.1.0)\n  • Installing pscript (0.7.7)\n  • Installing pygetwindow (0.0.4)\n  • Installing pygments (2.15.1): Installing...\n  • Installing pymsgbox (1.0.9)\n  • Installing pyobjc-framework-quartz (9.2)\n  • Installing h11 (0.14.0)\n  • Installing httptools (0.5.0)\n  • Installing isort (5.12.0)\n  • Installing jedi (0.18.2): Installing...\n  • Installing joblib (1.3.1)\n  • Installing kiwisolver (1.4.4)\n  • Installing lxml (4.9.2)\n  • Installing matplotlib-inline (0.1.6)\n  • Installing mccabe (0.7.0)\n  • Installing pexpect (4.8.0)\n  • Installing phonenumbers (8.13.15): Installing...\n  • Installing pickleshare (0.7.5)\n  • Installing platformdirs (3.8.0)\n  • Installing prompt-toolkit (3.0.38)\n  • Installing protobuf (4.23.3)\n  • Installing proxy-tools (0.1.0)\n  • Installing pscript (0.7.7)\n  • Installing pygetwindow (0.0.4)\n  • Installing pygments (2.15.1): Installing...\n  • Installing pymsgbox (1.0.9)\n  • Installing pyobjc-framework-quartz (9.2)\n  • Installing fonttools (4.40.0)\n  • Installing h11 (0.14.0)\n  • Installing httptools (0.5.0)\n  • Installing isort (5.12.0)\n  • Installing jedi (0.18.2): Installing...\n  • Installing joblib (1.3.1)\n  • Installing kiwisolver (1.4.4)\n  • Installing lxml (4.9.2)\n  • Installing matplotlib-inline (0.1.6)\n  • Installing mccabe (0.7.0)\n  • Installing pexpect (4.8.0)\n  • Installing phonenumbers (8.13.15): Installing...\n  • Installing pickleshare (0.7.5)\n  • Installing platformdirs (3.8.0)\n  • Installing prompt-toolkit (3.0.38)\n  • Installing protobuf (4.23.3)\n  • Installing proxy-tools (0.1.0)\n  • Installing pscript (0.7.7)\n  • Installing pygetwindow (0.0.4)\n  • Installing pygments (2.15.1): Installing...\n  • Installing pymsgbox (1.0.9)\n  • Installing pyobjc-framework-quartz (9.2)\n  • Installing pymsgbox (1.0.9)\n  • Installing pyobjc-framework-quartz (9.2)\n  • Installing pygments (2.15.1)\n  • Installing pymsgbox (1.0.9)\n  • Installing pyobjc-framework-quartz (9.2)\n  • Installing pickleshare (0.7.5)\n  • Installing platformdirs (3.8.0)\n  • Installing prompt-toolkit (3.0.38)\n  • Installing protobuf (4.23.3)\n  • Installing proxy-tools (0.1.0)\n  • Installing pscript (0.7.7)\n  • Installing pygetwindow (0.0.4)\n  • Installing pygments (2.15.1)\n  • Installing pymsgbox (1.0.9)\n  • Installing pyobjc-framework-quartz (9.2)\n  • Installing phonenumbers (8.13.15)\n  • Installing pickleshare (0.7.5)\n  • Installing platformdirs (3.8.0)\n  • Installing prompt-toolkit (3.0.38)\n  • Installing protobuf (4.23.3)\n  • Installing proxy-tools (0.1.0)\n  • Installing pscript (0.7.7)\n  • Installing pygetwindow (0.0.4)\n  • Installing pygments (2.15.1)\n  • Installing pymsgbox (1.0.9)\n  • Installing pyobjc-framework-quartz (9.2)\n  • Installing joblib (1.3.1)\n  • Installing kiwisolver (1.4.4)\n  • Installing lxml (4.9.2)\n  • Installing matplotlib-inline (0.1.6)\n  • Installing mccabe (0.7.0)\n  • Installing pexpect (4.8.0)\n  • Installing phonenumbers (8.13.15)\n  • Installing pickleshare (0.7.5)\n  • Installing platformdirs (3.8.0)\n  • Installing prompt-toolkit (3.0.38)\n  • Installing protobuf (4.23.3)\n  • Installing proxy-tools (0.1.0)\n  • Installing pscript (0.7.7)\n  • Installing pygetwindow (0.0.4)\n  • Installing pygments (2.15.1)\n  • Installing pymsgbox (1.0.9)\n  • Installing pyobjc-framework-quartz (9.2)\n  • Installing jedi (0.18.2)\n  • Installing joblib (1.3.1)\n  • Installing kiwisolver (1.4.4)\n  • Installing lxml (4.9.2)\n  • Installing matplotlib-inline (0.1.6)\n  • Installing mccabe (0.7.0)\n  • Installing pexpect (4.8.0)\n  • Installing phonenumbers (8.13.15)\n  • Installing pickleshare (0.7.5)\n  • Installing platformdirs (3.8.0)\n  • Installing prompt-toolkit (3.0.38)\n  • Installing protobuf (4.23.3)\n  • Installing proxy-tools (0.1.0)\n  • Installing pscript (0.7.7)\n  • Installing pygetwindow (0.0.4)\n  • Installing pygments (2.15.1)\n  • Installing pymsgbox (1.0.9)\n  • Installing pyobjc-framework-quartz (9.2)\n  • Installing pyobjc-framework-security (9.2)\n  • Installing pyobjc-framework-webkit (9.2)\n  • Installing pyparsing (3.1.0)\n  • Installing pyscreeze (0.1.19)\n  • Installing python-dateutil (2.8.2)\n  • Installing python-dotenv (1.0.0)\n 
.
..
...
....

FFFiend avatar Jul 20 '23 20:07 FFFiend

Please review https://github.com/OpenAdaptAI/OpenAdapt/pull/420 and let's discuss 1-1

abrichr avatar Jul 24 '23 14:07 abrichr

TODO: create gpt 4 class, will add more provider classes gradually today

FFFiend avatar Jul 25 '23 16:07 FFFiend

@FFFiend notes from our conversation:

[
{ “action”: {“name”: … }, “window”: {“...”: …, }},
{ “action”: {“name”: … }, “window”: {“...”: …, }},
…
]

For GPT: require a prompt and completion

Autoregressive on a single recording?

prompt:
[
{ “action”: {“name”: … }, “window”: {“...”: …, }}_{1},
{ “action”: {“name”: … }, “window”: {“...”: …, }}_{2},
…
{ “action”: {“name”: … }, “window”: {“...”: …, }}_{k}   # number of reference actions should be configurable, goal is to use diffs to remove unnecessary information, in order to maximize number actions that can be in a single prompt for a fixed context window
# this also involves knowing/calculating the context size of a model (this should also be in the API), and subtracting config.NUM_REF_WINDOW_IN_COMPLETION in order to fit the maximum possible number)

]

Completion:
{ “action”: {“name”: … }, “window”: {“...”: …, }}_{k+1},
{ “action”: {“name”: … }, “window”: {“...”: …, }}_{k+2},  # number of window/actions in completion should be configurable, default to 1, config.NUM_REF_WINDOW_IN_COMPLETION
.
Where k is the timestamp of each action, window pairing 


Def finetune(**kwargs):
     


For action, window pairing in processed_recording:
	finetune((action,window))

For reference_window, reference_action, target_window, target_action in get_finetune_pairs(processed_recording):
    finetune(reference_window, reference_action, target_window, target_action)

get_finetune_pairs(processed_recording):
    “””
    Loop through recording actions in order, and yield tuples of (ref_win, ref_act, trg_win, trg_act)
         
    “””

    For prev_action, next_action in zip(processed_recording.actions, processed_recording.actions[1:]):
        Prev_window = prev_action.window
        Next_window = next_action.window

        Ref_win = prev_window
        Ref_act = prev_action
        Trg_win = next_window
        Trg_act = next_action

        Yield ref_win, ref_act, trg_win, trg_act


.
.

**

Llama says you can’t use it for training another model
Freewilly is only for research

Define licenses for models inside the API.
Define  context length for each model inside API.
Class License:
    #APACHE2: “APACHE2”
    #LLAMA2: “LLAMA2”
    COMMERCIAL = “COMMERCIAL”
    NONCOMMERCIAL = “NONCOMMERCIAL”

Class Capability:
    TRAINING
    TUNING
    MODEL_TRAINING
    INFERENCE

ML_MODELS = [
    {
        "name": "gpt-4",
        "provider": openai.ml.openai,
        # "tune_func": openadapt.ml.openai.gpt4.tune,
        # "infer_func": openadapt.ml.openai.gpt4.infer,
        "modalities": openadapt.ml.openai.gpt4.MODALITIES,  #  gpt4/__init__.py: MODALITIES =  [Modality.TEXT],
        "license": License.FOO,
        "capabilities": [Capability.TRAINING, Capability.TUNING, …],
    },
     {
        "name": "BLIP2",
        "provider": openai.ml.huggingface,
        # "tune_func": openadapt.ml.huggingface.blip2.tune,
        # "infer_func": openadapt.ml.huggingface.blip2.infer,
        "modalities": openadapt.ml.huggingface.blip2.MODALITIES,  #  huggingface/__init__.py: MODALITIES =  [Modality.TEXT, Modality.IMAGE],
    },
    ...
]


config.ML_MODEL_LICENSES

abrichr avatar Jul 25 '23 21:07 abrichr

We need to filter this out:

'AXRoleDescription': 'text entry area', 'AXValue': 'Last login: Wed Jul 19 19:20:24 on ttys000\nowaiszahid@Owaiss-MacBook-Pro modelapi % git clone https://github.com/Owais-Enhancement-Org/OpenAdapt.git\nCloning into \'OpenAdapt\'...\nremote: Enumerating objects: 3311, done.\nremote: Counting objects: 100% (1505/1505), done.\nremote: Compressing objects: 100% (345/345), done.\nremote: Total 3311 (delta 1313), reused 1215 (delta 1141), pack-reused 1806\nReceiving objects: 100% (3311/3311), 24.74 MiB | 29.08 MiB/s, done.\nResolving deltas: 100% (2127/2127), done.\nowaiszahid@Owaiss-MacBook-Pro modelapi %

Options:

  1. Explicitly remove textareas over some predefined length
  2. Explicitly remove any attribute whose length is over some predefined length
  3. Measure the rate of change of attributes and remove those whose rate of change is greater than some threshold

etc.

abrichr avatar Jul 25 '23 21:07 abrichr

We also want to add the recording.task_description to the prompts/finetune pairs.

To clarify, fine tune pairs should be generated by replay strategies. These are what generate the prompts, and these prompts are what we need to finetune on.

Related: https://github.com/OpenAdaptAI/OpenAdapt/issues/413

abrichr avatar Jul 25 '23 21:07 abrichr

Related: https://github.com/OpenAdaptAI/OpenAdapt/issues/430

abrichr avatar Jul 25 '23 21:07 abrichr

Potential things to add: Docker image generation file for models that cant be run on CPU?

FFFiend avatar Aug 02 '23 08:08 FFFiend

Thanks @FFFiend ! Can you please review https://github.com/CambioML/pykoi to see if we might be able to re-use some of their primitives?

abrichr avatar Aug 14 '23 17:08 abrichr

Added a simple Base HuggingFace provider class and an exceptions.py file, feel free to check it out.

Next TODO: flesh out model config API (quantization configs, temperature, max token size, etc) and dive deeper into CPU, GPU workflow (perhaps a GGML model API for CPU for different RAM types)

@abrichr @LaPetiteSouris thoughts on the TODO?

FFFiend avatar Aug 16 '23 19:08 FFFiend

Added tests. Multiple files have changed because they didn't adhere to black formatting.

Edit: I still have to reformat docstrings to pass flake8 checks, and checks are pending fixes on styling inside main. Other than that, ready to merge.

Formatting, styling completed and checks have passed. Ready to merge @abrichr 👍

FFFiend avatar Aug 26 '23 00:08 FFFiend

@FFFiend it looks like this PR includes several unrelated changes (e.g. alembic/versions/5139d7df38f6_add_recording_task_description.py).

Can you please remove all unrelated code? 🙏

abrichr avatar Aug 29 '23 22:08 abrichr

@FFFiend it looks like this PR includes several unrelated changes (e.g. alembic/versions/5139d7df38f6_add_recording_task_description.py).

Can you please remove all unrelated code? 🙏

Those changes are purely stylistic (converting single quotes to double, fixing formatting) and were done using pre-commit run --all-files to fix black formatting across the entire codebase, my checks wouldn't pass without it.

FFFiend avatar Aug 29 '23 22:08 FFFiend

@FFFiend it looks like this PR includes several unrelated changes (e.g. alembic/versions/5139d7df38f6_add_recording_task_description.py).

Can you please remove all unrelated code? 🙏

@abrichr removed all unrelated code, however privacy provider tests (fixed in #488 ) are failing, so just waiting on that to get merged 👍

FFFiend avatar Aug 30 '23 17:08 FFFiend

@FFFiend #488 is merged now. Please update the branch then continue :-)

KrishPatel13 avatar Aug 30 '23 18:08 KrishPatel13

@abrichr Ready to merge.

FFFiend avatar Aug 30 '23 19:08 FFFiend

This is now stale and needs to be updated to support images.

Related: https://github.com/OpenAdaptAI/OpenAdapt/pull/527

@FFFiend do you have any bandwidth? 🙏

abrichr avatar Dec 13 '23 06:12 abrichr

@FFFiend would love your thoughts on https://github.com/OpenAdaptAI/OpenAdapt/issues/534 🙏 😄

abrichr avatar Dec 14 '23 04:12 abrichr