vespa Vespa Deploy fails for application package with model files

Describe the bug When models are included in the application package, the vespa deploy fails with either of the below two errors:

Uploading application package... failed
Error: invalid application package (400 Bad Request)
{
    "error-code": "BAD_REQUEST",
    "message": "Unable to decompress application stream: Truncated ZIP file"
}

OR

Uploading application package... failed
Error: error from deploy API at 127.0.0.1:19071 (503 Idle timeout expired: 180005/180000 ms):
<html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
<title>Error 503</title>
</head>
<body>
<h2>HTTP ERROR: 503</h2>
<p>Problem accessing /application/v2/tenant/default/prepareandactivate. Reason:
<pre>    Idle timeout expired: 180005/180000 ms</pre></p>
<hr/>
</body>
</html>

To Reproduce The issue can be reproduced via before / after introducing model files in application package. Before //

Clone and deploy sample app: $vespa clone multi-vector-indexing my-app && cd my-app - this works because it seems the model files are being read from Vespa hosted URLs (see service.xml supplied in the sample app) and not manually created using optimum-cli

After // We will continue to use the same sample app, but introduce model files in it.

Create model files and delete unnecessary files from the command $ optimum-cli export onnx --task sentence-similarity -m intfloat/multilingual-e5-small multilingual-e5-small-onnx
Move the model files under my-app\models and make refer to the model files in service.xml
Retry vespa deploy --wait 600 -- this is where the error occurs

Expected behavior Vespa Deploy to succeed.

Screenshots If applicable, add screenshots to help explain your problem. Adding Vespa Logs snippet:

INFO    configserver     Container.com.yahoo.vespa.config.server.http.HttpErrorResponse	Returning response with response code 400, error-code:BAD_REQUEST, message=Unable to decompress application stream: Truncated ZIP file

Environment (please complete the following information):

OS: MacOS 14.0 (23A344)
Infrastructure: self-hosted
Docker: Container memory usage: 3.95GB / 7.48GB
Docker: Container CPU usage: 11.13% / 1200% (12 CPUs available)

% docker version
Client:
 Cloud integration: v1.0.35+desktop.11
 Version:           25.0.3
 API version:       1.44
 Go version:        go1.21.6
 Git commit:        4debf41
 Built:             Tue Feb  6 21:13:26 2024
 OS/Arch:           darwin/arm64
 Context:           desktop-linux

Server: Docker Desktop 4.28.0 (139021)
 Engine:
  Version:          25.0.3
  API version:      1.44 (minimum version 1.24)
  Go version:       go1.21.6
  Git commit:       f417435
  Built:            Tue Feb  6 21:14:22 2024
  OS/Arch:          linux/arm64
  Experimental:     false
 containerd:
  Version:          1.6.28
  GitCommit:        ae07eda36dd25f8a1b98dfbf587313b99c0190bb
 runc:
  Version:          1.1.12
  GitCommit:        v1.1.12-0-g51d5e94
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Vespa version Include Vespa version(s) used.

Vespa version: 8.325.46
Vespa Docker Image aee351eb05b1b2f54c6541b9cd38eff68ba30a0d1d290e996a290049c2942992
Vespa CLI version 8.323.45 compiled with go1.22.1 on darwin/arm64

Additional context Add any other context about the problem here.

Apr 04 '24 12:04 rharige

Hi! I was able to (almost) reproduce your issue, encountering

Uploading application package... failed
Error: invalid application package (400 Bad Request)
Invalid application:
Could not import model 'sentencepiece_bpe':
Could not import a Vespa model from '/opt/vespa/var/db/vespa/config_server/serverdb/tenants/default/sessions/13/.preprocessed/models/sentencepiece.bpe.model'

It seems like the issue stems from the file sentencepiece.bpe.model, which you probably have located in the models-directory. The naming of this file likely causes some unexpected behavior, and removing or renaming it will hopefully solve your problem. We will follow up internally for potential fix/doc improvement.

Apr 05 '24 10:04 thomasht86

See also https://docs.vespa.ai/en/reference/stateless-model-reference.html

Apr 05 '24 10:04 thomasht86

Hi @thomasht86 - As noted in the steps to reproduce:

Create model files and delete unnecessary files from the command

The sentencepiece.bpe.model file is already removed from my my-app\models directory, it contains only onnx file and tokenizer.json. The issue still occurs, and I can reproduce this issue consistently - hence I raised the bug here.

Apr 05 '24 10:04 rharige

vespa vespa copied to clipboard

Vespa Deploy fails for application package with model files

vespa
vespa copied to clipboard