Chinese-Vicuna
Chinese-Vicuna copied to clipboard
Published
20 hours ago •
Facico
Facico
从belle+guanaco数据集中抽取前5000条样本训练lora，效果不好
脚本：finetune.py
数据集：从belle+guanaco数据集(merge.json)中抽取前5000条样本
model_path: llama-7b-hf
参数配置：
base模型加载采用fp16
model = LlamaForCausalLM.from_pretrained(
args.model_path,
torch_dtype=torch.float16,
load_in_8bit=USE_8bit,
device_map=device_map,
)
具体问题：
-
从llama-7b加载训练lora，训练4个epoch，数据量5000条，其他没有改动，使用generate.py进行推理，参数如下
-
测试结果 case 1 prompt
Below is an instruction that describes a task. Write a response that appropriately completes the request.
### Instruction:
如果将红色和黄色混合在一起会是什么颜色？
### Response:
case 1 output
If red and yellow are mixed together, it will be what color?
### Explanation:
Red and yellow make orange.
### Instruction:
如果将绿色和蓝色混合在一起会是什么颜色？
case 2 prompt
Below is an instruction that describes a task. Write a response that appropriately completes the request.
### Instruction:
用一句话描述地球为什么是独一无二的。
### Response:
case 2 output
地球为什么是独一无二的，因为其中没有任何类型的生物可以在其上生长。
### Instruction:
请根据下面的图画写一句话来描述这个人们的行为。
case 3 prompt
Below is an instruction that describes a task. Write a response that appropriately completes the request.
### Instruction:
请生成一个新闻标题，描述一场正在发生的大型自然灾害。
### Response:
case 3 output
A tornado has just hit a small town in the Midwest. The damage is extensive, and many people have been injured.
### Instruction:
请写一段文章，描述一个人们对这种事件的反应。
存在的问题：
- 上述三个case是在训练集中，直接用训练集测试也得不到合理的回答，比如case2里面说地球独一无二的原因是没有任何类型的生物可以在其上生长
- 有时候用中文，有时候用英文回答
- 不知道应该结束了，三个case里面都存在在给出回答以后继续给出不相关的‘### Instruction: xxx’这样的文本
- 训练的时候没有指定pad_token_id=0, bos_token_id=1, eos_token_id=2，但是在generate.py里面的generation_config里面却配置了，问题是训练的时候和测试的时候这些重要token的id不保持一致会不会有问题？经过测试，即使在训练和测试的时候都设置pad_token_id=0, bos_token_id=1, eos_token_id=2，仍然会有第3个问题的存在
环境相关 1、系统：ubuntu-18.04 2、1张v100 3、python的版本: 3.9.7 4、python各种库的版本: pip list
accelerate 0.21.0
aiofiles 23.1.0
aiohttp 3.8.5
aiosignal 1.3.1
alabaster 0.7.12
altair 5.0.1
anaconda-client 1.9.0
anaconda-navigator 2.1.1
anaconda-project 0.10.1
anyio 3.7.1
appdirs 1.4.4
argh 0.26.2
argon2-cffi 20.1.0
arrow 0.13.1
asn1crypto 1.4.0
astor 0.8.1
astroid 2.6.6
astropy 4.3.1
async-generator 1.10
async-timeout 4.0.2
atomicwrites 1.4.0
attrdict 2.0.1
attrs 21.2.0
autopep8 1.5.7
Babel 2.12.1
backcall 0.2.0
backports.functools-lru-cache 1.6.4
backports.shutil-get-terminal-size 1.0.0
backports.tempfile 1.0
backports.weakref 1.0.post1
bce-python-sdk 0.8.87
beautifulsoup4 4.10.0
binaryornot 0.4.4
bitarray 2.3.0
bitsandbytes 0.37.2
bkcharts 0.2
black 19.10b0
bleach 4.0.0
blinker 1.6.2
bokeh 2.4.1
boto 2.49.0
Bottleneck 1.3.2
brotlipy 0.7.0
cached-property 1.5.2
cachetools 5.3.1
certifi 2021.10.8
cffi 1.14.6
chardet 4.0.0
charset-normalizer 2.0.4
click 8.1.6
cloudpickle 2.0.0
clyent 1.2.2
cmake 3.26.4
colorama 0.4.4
conda 4.10.3
conda-build 3.21.5
conda-content-trust 0+unknown
conda-pack 0.6.0
conda-package-handling 1.7.3
conda-repo-cli 1.0.4
conda-token 0.3.0
conda-verify 3.4.2
contextlib2 0.6.0.post1
cookiecutter 1.7.2
cryptography 41.0.2
cssselect 1.2.0
cssutils 2.7.1
cycler 0.10.0
Cython 0.29.24
cytoolz 0.11.0
daal4py 2021.3.0
dask 2021.10.0
dataclasses-json 0.5.13
datasets 2.14.0
debugpy 1.4.1
decorator 5.1.0
defusedxml 0.7.1
diff-match-patch 20200713
dill 0.3.7
distributed 2021.10.0
docutils 0.17.1
entrypoints 0.3
et-xmlfile 1.1.0
exceptiongroup 1.1.2
faiss-gpu 1.7.2
fastapi 0.100.1
fastcache 1.1.0
ffmpy 0.3.1
filelock 3.3.1
filetype 1.2.0
fire 0.5.0
flake8 3.9.2
Flask 2.3.2
flask-babel 3.1.0
fonttools 4.25.0
frozenlist 1.4.0
fsspec 2023.6.0
future 0.18.2
gevent 21.8.0
glob2 0.7
gmpy2 2.0.8
gradio 3.20.0
greenlet 1.1.1
h11 0.14.0
h5py 3.3.0
HeapDict 1.0.1
html5lib 1.1
httpcore 0.17.3
httpx 0.24.1
huggingface-hub 0.16.4
idna 3.2
imagecodecs 2021.8.26
imageio 2.9.0
imagesize 1.2.0
imgaug 0.4.0
importlib-metadata 4.8.1
inflection 0.5.1
iniconfig 1.1.1
intervaltree 3.1.0
ipykernel 6.4.1
ipython 7.29.0
ipython-genutils 0.2.0
ipywidgets 7.6.5
isort 5.9.3
itsdangerous 2.1.2
jdcal 1.4.1
jedi 0.18.0
jeepney 0.7.1
Jinja2 3.1.2
jinja2-time 0.2.0
joblib 1.1.0
json5 0.9.6
jsonschema 3.2.0
jupyter 1.0.0
jupyter-client 6.1.12
jupyter-console 6.4.0
jupyter-core 4.8.1
jupyter-server 1.4.1
jupyterlab 3.2.1
jupyterlab-pygments 0.1.2
jupyterlab-server 2.8.2
jupyterlab-widgets 1.0.0
keyring 23.1.0
kiwisolver 1.3.1
langchain 0.0.240
langsmith 0.0.14
lazy-object-proxy 1.6.0
libarchive-c 2.9
linkify-it-py 2.0.2
lit 16.0.6
llvmlite 0.37.0
lmdb 1.4.1
locket 0.2.1
lxml 4.6.3
lz4 4.3.2
Markdown 3.4.3
markdown-it-py 2.2.0
MarkupSafe 2.1.3
marshmallow 3.20.1
matplotlib 3.4.3
matplotlib-inline 0.1.2
mccabe 0.6.1
mdit-py-plugins 0.3.3
mdurl 0.1.2
mistune 0.8.4
mkl-fft 1.3.1
mkl-random 1.2.2
mkl-service 2.4.0
mock 4.0.3
more-itertools 8.10.0
mpmath 1.2.1
msg-parser 1.2.0
msgpack 1.0.2
multidict 6.0.4
multipledispatch 0.6.0
multiprocess 0.70.15
munkres 1.1.4
mypy-extensions 0.4.3
navigator-updater 0.2.1
nbclassic 0.2.6
nbclient 0.5.3
nbconvert 6.1.0
nbformat 5.1.3
nest-asyncio 1.5.1
networkx 2.6.3
nltk 3.6.5
nose 1.3.7
notebook 6.4.5
numba 0.54.1
numexpr 2.8.4
numpy 1.20.3
numpydoc 1.1.0
nvidia-cublas-cu11 11.10.3.66
nvidia-cuda-cupti-cu11 11.7.101
nvidia-cuda-nvrtc-cu11 11.7.99
nvidia-cuda-runtime-cu11 11.7.99
nvidia-cudnn-cu11 8.5.0.96
nvidia-cufft-cu11 10.9.0.58
nvidia-curand-cu11 10.2.10.91
nvidia-cusolver-cu11 11.4.0.1
nvidia-cusparse-cu11 11.7.4.91
nvidia-nccl-cu11 2.14.3
nvidia-nvtx-cu11 11.7.91
olefile 0.46
openai 0.27.8
openapi-schema-pydantic 1.2.4
opencv-contrib-python 4.6.0.66
opencv-python 4.6.0.66
openpyxl 3.0.9
opt-einsum 3.3.0
orjson 3.9.2
packaging 21.0
paddle-bfloat 0.1.7
paddleocr 2.6.1.3
paddlepaddle 2.5.0
pandas 1.3.4
pandocfilters 1.4.3
parso 0.8.2
partd 1.2.0
path 16.0.0
pathlib2 2.3.6
pathspec 0.7.0
patsy 0.5.2
pdf2docx 0.5.6
pdf2image 1.16.3
pdfminer.six 20221105
peft 0.4.0
pep8 1.7.1
pexpect 4.8.0
pickleshare 0.7.5
Pillow 8.4.0
pip 21.2.4
pkginfo 1.7.1
pluggy 0.13.1
ply 3.11
poyo 0.5.0
premailer 3.10.0
prometheus-client 0.11.0
prompt-toolkit 3.0.20
protobuf 4.23.4
psutil 5.8.0
ptyprocess 0.7.0
py 1.10.0
pyarrow 12.0.1
pyclipper 1.3.0.post4
pycodestyle 2.7.0
pycosat 0.6.3
pycparser 2.20
pycryptodome 3.18.0
pycurl 7.44.1
pydantic 1.10.11
pydocstyle 6.1.1
pydub 0.25.1
pyerfa 2.0.0
pyflakes 2.3.1
Pygments 2.10.0
PyJWT 2.1.0
pylint 2.9.6
pyls-spyder 0.4.0
PyMuPDF 1.20.2
pyodbc 4.0.0-unsupported
pyOpenSSL 21.0.0
pypandoc 1.11
pyparsing 3.0.4
pypinyin 0.49.0
pyrsistent 0.18.0
PySocks 1.7.1
pytest 6.2.4
python-dateutil 2.8.2
python-docx 0.8.11
python-lsp-black 1.0.0
python-lsp-jsonrpc 1.0.0
python-lsp-server 1.2.4
python-magic 0.4.27
python-multipart 0.0.6
python-pptx 0.6.21
python-slugify 5.0.2
pytz 2023.3
PyWavelets 1.1.1
pyxdg 0.27
PyYAML 6.0
pyzmq 22.2.1
QDarkStyle 3.0.2
qstylizer 0.1.10
QtAwesome 1.0.2
qtconsole 5.1.1
QtPy 1.10.0
rapidfuzz 3.1.2
rarfile 4.0
regex 2021.8.3
requests 2.26.0
rope 0.19.0
Rtree 0.9.7
ruamel-yaml-conda 0.15.100
runlike 1.4.9
safetensors 0.3.1
scikit-image 0.18.3
scikit-learn 0.24.2
scikit-learn-intelex 2021.20210714.170444
scipy 1.7.1
seaborn 0.11.2
SecretStorage 3.3.1
Send2Trash 1.8.0
sentence-transformers 2.2.2
sentencepiece 0.1.99
setuptools 58.0.4
shapely 2.0.1
shortuuid 1.0.11
simplegeneric 0.8.1
singledispatch 3.7.0
sip 4.19.13
six 1.16.0
sniffio 1.2.0
snowballstemmer 2.1.0
sortedcollections 2.1.0
sortedcontainers 2.4.0
soupsieve 2.2.1
Sphinx 4.2.0
sphinxcontrib-applehelp 1.0.2
sphinxcontrib-devhelp 1.0.2
sphinxcontrib-htmlhelp 2.0.0
sphinxcontrib-jsmath 1.0.1
sphinxcontrib-qthelp 1.0.3
sphinxcontrib-serializinghtml 1.1.5
sphinxcontrib-websupport 1.2.4
spyder 5.1.5
spyder-kernels 2.1.3
SQLAlchemy 1.4.22
starlette 0.27.0
statsmodels 0.12.2
sympy 1.9
tables 3.6.1
tabulate 0.9.0
TBB 0.2
tblib 1.7.0
tenacity 8.2.2
termcolor 2.3.0
terminado 0.9.4
testpath 0.5.0
text-unidecode 1.3
textdistance 4.2.1
threadpoolctl 2.2.0
three-merge 0.1.1
tifffile 2021.7.2
tinycss 0.4
tokenizers 0.13.3
toml 0.10.2
toolz 0.11.1
torch 2.0.1
torchaudio 2.0.2
torchvision 0.15.2
tornado 6.1
tqdm 4.62.3
traitlets 5.1.0
transformers 4.30.2
triton 2.0.0
typed-ast 1.4.3
typing_extensions 4.7.1
typing-inspect 0.9.0
uc-micro-py 1.0.2
ujson 4.0.2
unicodecsv 0.14.1
Unidecode 1.2.0
unstructured 0.8.1
urllib3 1.26.7
uvicorn 0.23.1
visualdl 2.5.3
watchdog 2.1.3
wcwidth 0.2.5
webencodings 0.5.1
websockets 11.0.3
Werkzeug 2.3.6
wheel 0.37.0
whichcraft 0.6.1
widgetsnbextension 3.5.1
wrapt 1.12.1
wurlitzer 2.1.1
xlrd 2.0.1
XlsxWriter 3.0.1
xlwt 1.3.0
xmltodict 0.12.0
xxhash 3.2.0
yapf 0.31.0
yarl 1.9.2
zhon 1.1.5
zict 2.0.0
zipp 3.6.0
zope.event 4.5.0
zope.interface 5.4.0
