Open-Assistant
Open-Assistant copied to clipboard
LAION-AI
Conversion of Data agumentation Notebooks to python script functions
As discussed in the discord I'd like to convert the Notebooks inside of the data argumentation to a runable python script with argument parsing.
I will take the functionalities in
- https://github.com/LAION-AI/Open-Assistant/tree/main/notebooks/data-argumentation
- https://github.com/LAION-AI/Open-Assistant/tree/main/notebooks/code-bugger
- https://colab.research.google.com/drive/1nZx5LRjO61fYprFyqtrwPDLOis6ctR4p#scrollTo=1EE8CriiaCXj and convert them into a single python file.
The goal is to be able to run them via command line, and enabling to pass tsv file pointing to the essay-txts or similar.
Those newly generated question - answer pairs can than be passed to the model for fine tuning.
Unfortunately I have to work during the week I wanted to add to the code-bugger at least one more error type and some internal checks to guarantee that the error has been applied and to also return a dictionary with type of errors injected, the locations and which strings have been substituted to what. That should be a good enough starting point to generate conversations. If we have future plans to actually run code that the assistant is writing we should add some sort of flags to code that can't be running (e.g. bugged methods that overflow memory or do infinite loops).
Is there some draft PR about this? As I think I could finish this one if needed.
Is there some draft PR about this? As I think I could finish this one if needed.
Sure, I will do a draft PR rn. Wasn't able to test the Codebugger, because of some installation issues i couldnt resolve yet. Other classes should work fine however
@finitearth were you on windows? I did from mac and Indeed the pip install was not working in windows, it now works on my gaming pc. I anyway will add a new notebook with a big copypasta of the methods to simplify your life.
@finitearth were you on windows? I did from mac and Indeed the pip install was not working in windows, it now works on my gaming pc. I anyway will add a new notebook with a big copypasta of the methods to simplify your life.
Yes indeed I'm working on windows. It appears to be an issue with the pip installation.
Feel free to send me the notebook on discord (also finitearth)
can you try if it works now? it should be working, I tried on w10 locally, I updated the setup.py 20 minutes ago on openasisstant github so by re-cloning it should get the correct one, also might be because it is calliing python3 pip and you want python pip. Opening a pull-request here with the new notebook anyway.
From my side I think I fixed the bug for pip install on windows, also added in the 546 a notebook that requires no install. I am working this week but in the weekend I will update the bugging code.
I forgot to ask, but could you keep my copyright notice in the section of the code that I wrote?
# coding=utf-8
# Copyright 2021-2023, Ontocord, LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.