BentoML icon indicating copy to clipboard operation
BentoML copied to clipboard

Published 20 hours ago • verified publisher icon bentoml

supports export to GCS

Open stephenleo opened this issue 2 years ago • 1 comments

Describe the bug

fs-gcsfs is a required dependency to export models to GCS. Without it, running bentoml export results in an error as below

ValueError: Unknown or unsupported protocol gs. Some supported protocols are 'ftp', 's3', and 'osfs'.

To reproduce

  1. pip install "bentoml[all]" scikit-learn pandas

  2. Run the sample code trying to export to any GCS bucket

    import bentoml

from sklearn import svm
from sklearn import datasets

# Load training data set
iris = datasets.load_iris()
X, y = iris.data, iris.target

# Train the model
clf = svm.SVC(gamma='scale')
clf.fit(X, y)

# Save model to the BentoML local model store
saved_model = bentoml.sklearn.save_model("iris_clf", clf)

# Export model to GCS bucket
bentoml.models.export_model("iris_clf:latest", "gs://any-gcs-bucket") # Error happens here!

    ValueError: Unknown or unsupported protocol gs. Some supported protocols are 'ftp', 's3', and 'osfs'.

Expected behavior

Once fs-gcsfs is installed there are no errors and the model is pushed to GCS successfully!

Environment

Environment variable

BENTOML_DEBUG=''
BENTOML_QUIET=''
BENTOML_BUNDLE_LOCAL_BUILD=''
BENTOML_DO_NOT_TRACK=''
BENTOML_CONFIG=''
BENTOML_CONFIG_OPTIONS=''
BENTOML_PORT=''
BENTOML_HOST=''
BENTOML_API_WORKERS=''

System information

bentoml: 1.0.12 python: 3.9.15 platform: macOS-13.0-arm64-arm-64bit uid_gid: 282443551:1192903136 conda: 4.14.0 in_conda_env: True

conda_packages 
name: test
channels:
  - conda-forge
dependencies:
  - bzip2=1.0.8=h3422bc3_4
  - ca-certificates=2022.12.7=h4653dfc_0
  - libffi=3.4.2=h3422bc3_5
  - libsqlite=3.40.0=h76d750c_0
  - libzlib=1.2.13=h03a7124_4
  - ncurses=6.3=h07bb92c_1
  - openssl=3.0.7=h03a7124_1
  - pip=22.3.1=pyhd8ed1ab_0
  - python=3.9.15=hea58f1e_0_cpython
  - readline=8.1.2=h46ed386_0
  - setuptools=65.5.1=pyhd8ed1ab_0
  - tk=8.6.12=he1e0b03_0
  - tzdata=2022g=h191b570_0
  - wheel=0.38.4=pyhd8ed1ab_0
  - xz=5.2.6=h57fd34a_0
  - pip:
    - aiohttp==3.8.3
    - aiosignal==1.3.1
    - anyio==3.6.2
    - appdirs==1.4.4
    - appnope==0.1.3
    - asgiref==3.5.2
    - asttokens==2.2.1
    - async-timeout==4.0.2
    - attrs==22.1.0
    - backcall==0.2.0
    - backoff==2.2.1
    - bentoml==1.0.12
    - boto3==1.26.30
    - botocore==1.29.30
    - build==0.9.0
    - cattrs==22.2.0
    - certifi==2022.12.7
    - charset-normalizer==2.1.1
    - circus==0.18.0
    - click==8.1.3
    - click-option-group==0.5.5
    - cloudpickle==2.2.0
    - comm==0.1.2
    - commonmark==0.9.1
    - contextlib2==21.6.0
    - debugpy==1.6.4
    - decorator==5.1.1
    - deepmerge==1.1.0
    - deprecated==1.2.13
    - entrypoints==0.4
    - exceptiongroup==1.0.4
    - executing==1.2.0
    - filetype==1.2.0
    - frozenlist==1.3.3
    - fs==2.4.16
    - fs-s3fs==1.1.1
    - googleapis-common-protos==1.56.2
    - grpcio==1.51.1
    - grpcio-channelz==1.48.1
    - grpcio-health-checking==1.48.1
    - grpcio-reflection==1.48.1
    - h11==0.14.0
    - idna==3.4
    - ipykernel==6.19.2
    - ipython==8.7.0
    - jedi==0.18.2
    - jinja2==3.1.2
    - jmespath==1.0.1
    - joblib==1.2.0
    - jupyter-client==7.4.8
    - jupyter-core==5.1.0
    - markupsafe==2.1.1
    - matplotlib-inline==0.1.6
    - multidict==6.0.3
    - nest-asyncio==1.5.6
    - numpy==1.23.5
    - opentelemetry-api==1.14.0
    - opentelemetry-exporter-jaeger==1.14.0
    - opentelemetry-exporter-jaeger-proto-grpc==1.14.0
    - opentelemetry-exporter-jaeger-thrift==1.14.0
    - opentelemetry-exporter-otlp==1.14.0
    - opentelemetry-exporter-otlp-proto-grpc==1.14.0
    - opentelemetry-exporter-otlp-proto-http==1.14.0
    - opentelemetry-exporter-zipkin==1.14.0
    - opentelemetry-exporter-zipkin-json==1.14.0
    - opentelemetry-exporter-zipkin-proto-http==1.14.0
    - opentelemetry-instrumentation==0.35b0
    - opentelemetry-instrumentation-aiohttp-client==0.35b0
    - opentelemetry-instrumentation-asgi==0.35b0
    - opentelemetry-instrumentation-grpc==0.35b0
    - opentelemetry-proto==1.14.0
    - opentelemetry-sdk==1.14.0
    - opentelemetry-semantic-conventions==0.35b0
    - opentelemetry-util-http==0.35b0
    - packaging==21.3
    - pandas==1.5.2
    - parso==0.8.3
    - pathspec==0.10.3
    - pep517==0.13.0
    - pexpect==4.8.0
    - pickleshare==0.7.5
    - pillow==9.3.0
    - pip-requirements-parser==32.0.0
    - pip-tools==6.12.0
    - platformdirs==2.6.0
    - prometheus-client==0.15.0
    - prompt-toolkit==3.0.36
    - protobuf==3.19.6
    - psutil==5.9.4
    - ptyprocess==0.7.0
    - pure-eval==0.2.2
    - pyarrow==10.0.1
    - pydantic==1.10.2
    - pygments==2.13.0
    - pynvml==11.4.1
    - pyparsing==3.0.9
    - python-dateutil==2.8.2
    - python-json-logger==2.0.4
    - python-multipart==0.0.5
    - pytz==2022.6
    - pyyaml==6.0
    - pyzmq==24.0.1
    - requests==2.28.1
    - rich==12.6.0
    - s3transfer==0.6.0
    - schema==0.7.5
    - scikit-learn==1.2.0
    - scipy==1.9.3
    - simple-di==0.1.5
    - six==1.16.0
    - sniffio==1.3.0
    - stack-data==0.6.2
    - starlette==0.23.1
    - threadpoolctl==3.1.0
    - thrift==0.16.0
    - tomli==2.0.1
    - tornado==6.2
    - traitlets==5.7.1
    - typing-extensions==4.4.0
    - urllib3==1.26.13
    - uvicorn==0.20.0
    - watchfiles==0.18.1
    - wcwidth==0.2.5
    - wrapt==1.14.1
    - yarl==1.8.2
prefix: /Users/sleo/mambaforge/envs/test
pip_packages 
aiohttp==3.8.3
aiosignal==1.3.1
anyio==3.6.2
appdirs==1.4.4
appnope==0.1.3
asgiref==3.5.2
asttokens==2.2.1
async-timeout==4.0.2
attrs==22.1.0
backcall==0.2.0
backoff==2.2.1
bentoml==1.0.12
boto3==1.26.30
botocore==1.29.30
build==0.9.0
cattrs==22.2.0
certifi==2022.12.7
charset-normalizer==2.1.1
circus==0.18.0
click==8.1.3
click-option-group==0.5.5
cloudpickle==2.2.0
comm==0.1.2
commonmark==0.9.1
contextlib2==21.6.0
debugpy==1.6.4
decorator==5.1.1
deepmerge==1.1.0
Deprecated==1.2.13
entrypoints==0.4
exceptiongroup==1.0.4
executing==1.2.0
filetype==1.2.0
frozenlist==1.3.3
fs==2.4.16
fs-s3fs==1.1.1
googleapis-common-protos==1.56.2
grpcio==1.51.1
grpcio-channelz==1.48.1
grpcio-health-checking==1.48.1
grpcio-reflection==1.48.1
h11==0.14.0
idna==3.4
ipykernel==6.19.2
ipython==8.7.0
jedi==0.18.2
Jinja2==3.1.2
jmespath==1.0.1
joblib==1.2.0
jupyter_client==7.4.8
jupyter_core==5.1.0
MarkupSafe==2.1.1
matplotlib-inline==0.1.6
multidict==6.0.3
nest-asyncio==1.5.6
numpy==1.23.5
opentelemetry-api==1.14.0
opentelemetry-exporter-jaeger==1.14.0
opentelemetry-exporter-jaeger-proto-grpc==1.14.0
opentelemetry-exporter-jaeger-thrift==1.14.0
opentelemetry-exporter-otlp==1.14.0
opentelemetry-exporter-otlp-proto-grpc==1.14.0
opentelemetry-exporter-otlp-proto-http==1.14.0
opentelemetry-exporter-zipkin==1.14.0
opentelemetry-exporter-zipkin-json==1.14.0
opentelemetry-exporter-zipkin-proto-http==1.14.0
opentelemetry-instrumentation==0.35b0
opentelemetry-instrumentation-aiohttp-client==0.35b0
opentelemetry-instrumentation-asgi==0.35b0
opentelemetry-instrumentation-grpc==0.35b0
opentelemetry-proto==1.14.0
opentelemetry-sdk==1.14.0
opentelemetry-semantic-conventions==0.35b0
opentelemetry-util-http==0.35b0
packaging==21.3
pandas==1.5.2
parso==0.8.3
pathspec==0.10.3
pep517==0.13.0
pexpect==4.8.0
pickleshare==0.7.5
Pillow==9.3.0
pip-requirements-parser==32.0.0
pip-tools==6.12.0
platformdirs==2.6.0
prometheus-client==0.15.0
prompt-toolkit==3.0.36
protobuf==3.19.6
psutil==5.9.4
ptyprocess==0.7.0
pure-eval==0.2.2
pyarrow==10.0.1
pydantic==1.10.2
Pygments==2.13.0
pynvml==11.4.1
pyparsing==3.0.9
python-dateutil==2.8.2
python-json-logger==2.0.4
python-multipart==0.0.5
pytz==2022.6
PyYAML==6.0
pyzmq==24.0.1
requests==2.28.1
rich==12.6.0
s3transfer==0.6.0
schema==0.7.5
scikit-learn==1.2.0
scipy==1.9.3
simple-di==0.1.5
six==1.16.0
sniffio==1.3.0
stack-data==0.6.2
starlette==0.23.1
threadpoolctl==3.1.0
thrift==0.16.0
tomli==2.0.1
tornado==6.2
traitlets==5.7.1
typing_extensions==4.4.0
urllib3==1.26.13
uvicorn==0.20.0
watchfiles==0.18.1
wcwidth==0.2.5
wrapt==1.14.1
yarl==1.8.2

stephenleo avatar Dec 15 '22 01:12 stephenleo