spin spin deploy - internal server error when trying to push

I have successfully built my app and it's running with spin build --up. However, deployment is not working:

I tried creating a database - it does work locally, but not remotely.

-- schema.sql

CREATE TABLE IF NOT EXISTS users (
    id TEXT PRIMARY KEY NOT NULL,
    company_name TEXT,
    company_vat_id TEXT,
    name TEXT NOT NULL,
    address_country TEXT NOT NULL,
    email TEXT UNIQUE NOT NULL,
    email_confirmed BOOLEAN NOT NULL,
    language TEXT NOT NULL,
    password_hash BLOB NOT NULL,
    authority TEXT NOT NULL
);

CREATE TABLE IF NOT EXISTS orders (
    id TEXT NOT NULL,
    revision INTEGER NOT NULL,
    is_example_order BOOLEAN NOT NULL,
    map_name TEXT NOT NULL,
    user_id TEXT NOT NULL,
    style_id TEXT NOT NULL,
    latitude REAL NOT NULL,
    longitude REAL NOT NULL,
    projection TEXT NOT NULL,
    width_mm INTEGER NOT NULL,
    height_mm INTEGER NOT NULL,
    scale INTEGER NOT NULL,
    date_ordered TEXT NOT NULL, -- Stored as RFC3339 string
    map_price REAL NOT NULL,
    street_index_price REAL,
    hillshade_price REAL,
    contours_price REAL,
    vat REAL NOT NULL,
    vat_percent REAL NOT NULL,
    transaction_fee REAL NOT NULL,
    payment_method TEXT NOT NULL,
    is_test_transaction BOOLEAN NOT NULL,
    invoice_id TEXT UNIQUE NOT NULL,
    txn_id TEXT,
    language TEXT NOT NULL,
    status TEXT NOT NULL,
    currently_edited_by TEXT,
    archived BOOLEAN NOT NULL,
    finished_by TEXT,
    finished_date TEXT, -- Stored as RFC3339 string
    last_download_date TEXT, -- Stored as RFC3339 string
    lines_price REAL,
    points_price REAL,
    ip_hash TEXT,
    extra_lines TEXT, -- Storing complex types as JSON string
    extra_points TEXT, -- Storing complex types as JSON string
    extra_graphic_design TEXT, -- Storing complex types as JSON string
    pdf_file_data BLOB, -- Storing as BLOB
    ai_file_data BLOB, -- Storing as BLOB
    street_index_file_data TEXT, -- Storing as TEXT
    PRIMARY KEY (id, revision)
);

CREATE TABLE IF NOT EXISTS unconfirmed_emails (
    confirm_hash TEXT PRIMARY KEY NOT NULL,
    email TEXT NOT NULL,
    expiry_date TEXT NOT NULL -- Stored as RFC3339 string
);

CREATE TABLE IF NOT EXISTS open_sessions (
    session_id TEXT PRIMARY KEY NOT NULL,
    user_id TEXT NOT NULL
);

CREATE TABLE IF NOT EXISTS id_2fa (
    token TEXT PRIMARY KEY NOT NULL,
    session_id TEXT NOT NULL,
    expiry_date TEXT NOT NULL -- Stored as RFC3339 string
);

CREATE TABLE IF NOT EXISTS forgotten_passwords (
    reset_hash TEXT PRIMARY KEY NOT NULL,
    user_id TEXT NOT NULL,
    expiry_date TEXT NOT NULL -- Stored as RFC3339 string
);

CREATE TABLE IF NOT EXISTS quotes (
    id TEXT PRIMARY KEY NOT NULL,
    created_date TEXT NOT NULL, -- Stored as RFC3339 string
    accepted_date TEXT, -- Stored as RFC3339 string
    lang TEXT NOT NULL,
    map TEXT NOT NULL, -- Storing complex types as JSON string
    graphic_design TEXT, -- Storing complex types as JSON string
    chat_id TEXT
);

CREATE TABLE IF NOT EXISTS user_quotes (
    id TEXT PRIMARY KEY NOT NULL,
    created_date TEXT NOT NULL, -- Stored as RFC3339 string
    user_id TEXT NOT NULL,
    map TEXT NOT NULL, -- Storing complex types as JSON string
    graphic_design TEXT NOT NULL, -- Storing complex types as JSON string
    quote_id TEXT,
    lang TEXT NOT NULL
);

CREATE TABLE IF NOT EXISTS online_users (
    id_or_email TEXT PRIMARY KEY NOT NULL,
    last_seen TEXT NOT NULL, -- Stored as RFC3339 string
    online BOOLEAN NOT NULL
);

CREATE TABLE IF NOT EXISTS chats (
    id TEXT PRIMARY KEY NOT NULL,
    started_date TEXT NOT NULL, -- Stored as RFC3339 string
    language TEXT NOT NULL,
    user_id TEXT NOT NULL,
    with_email TEXT NOT NULL,
    user_email TEXT NOT NULL,
    user_typing BOOLEAN NOT NULL,
    admin_typing BOOLEAN NOT NULL
);

CREATE TABLE IF NOT EXISTS chat_msgs (
    id TEXT PRIMARY KEY NOT NULL,
    chat_id TEXT NOT NULL,
    is_reply BOOLEAN NOT NULL,
    text TEXT NOT NULL,
    sent TEXT NOT NULL, -- Stored as RFC3339 string
    status TEXT NOT NULL, -- MessageStatus as string
    attached_docs TEXT -- Storing complex types as JSON string
);

CREATE TABLE IF NOT EXISTS documents (
    id TEXT PRIMARY KEY NOT NULL,
    name TEXT NOT NULL,
    created_by TEXT NOT NULL,
    created TEXT NOT NULL, -- Stored as RFC3339 string
    mime_type TEXT NOT NULL,
    size INTEGER NOT NULL,
    bytes BLOB NOT NULL
);

CREATE TABLE IF NOT EXISTS chat_stream_msgs (
    id TEXT PRIMARY KEY NOT NULL,
    date TEXT NOT NULL, -- Stored as RFC3339 string
    sender TEXT NOT NULL,
    receiver TEXT NOT NULL,
    message TEXT NOT NULL -- Storing complex types (WsResponseMsg) as JSON string
);

CREATE TABLE IF NOT EXISTS extra_data_not_yet_ordered (
    hash TEXT PRIMARY KEY NOT NULL,
    extra_points_json TEXT NOT NULL,
    extra_lines_json TEXT NOT NULL,
    expiry_date TEXT NOT NULL -- Stored as RFC3339 string
);

CREATE TABLE IF NOT EXISTS campaigns (
    campaign_id TEXT NOT NULL,
    ip_hash TEXT NOT NULL,
    visit_date TEXT NOT NULL, -- Stored as RFC3339 string
    lang TEXT NOT NULL,
    referrer_url TEXT,
    register_date TEXT, -- Stored as RFC3339 string
    checkout_date TEXT, -- Stored as RFC3339 string
    chat_date TEXT, -- Stored as RFC3339 string
    user_id TEXT,
    chat_id TEXT,
    order_id TEXT,
    PRIMARY KEY (campaign_id, ip_hash, visit_date) -- Composite primary key to allow multiple visits
);

CREATE TABLE IF NOT EXISTS mail (
    id TEXT PRIMARY KEY NOT NULL,
    date TEXT NOT NULL, -- Stored as RFC3339 string
    sender TEXT NOT NULL,
    recipients TEXT NOT NULL,
    data TEXT NOT NULL
);

CREATE TABLE IF NOT EXISTS ipn_requests (
    id TEXT PRIMARY KEY NOT NULL,
    original_form_url_data TEXT NOT NULL,
    test_ipn BOOLEAN NOT NULL,
    receiver_email TEXT NOT NULL,
    txn_id TEXT NOT NULL,
    invoice_id TEXT NOT NULL,
    payment_status TEXT NOT NULL,
    currency TEXT NOT NULL,
    payment_gross REAL NOT NULL,
    received TEXT NOT NULL, -- Stored as RFC3339 string
    status TEXT NOT NULL -- IpnStatus as string
);

CREATE TABLE IF NOT EXISTS temp_chunks (
    id TEXT PRIMARY KEY NOT NULL,
    doc_id TEXT NOT NULL,
    chunk_index INTEGER NOT NULL,
    chunk_data BLOB NOT NULL
);

CREATE TABLE IF NOT EXISTS temp_documents (
    id TEXT PRIMARY KEY NOT NULL,
    user_id TEXT NOT NULL,
    name TEXT NOT NULL,
    size INTEGER NOT NULL,
    created_at TEXT NOT NULL
);

Creating a database and executing this works locally:

#!/bin/bash

# --- Local Development Database Setup Script ---
# This script sets up the SQLite database for local testing with `spin up`.
# It reads the schema from `schema.sql` in the project root.

# Exit immediately if a command exits with a non-zero status.
set -e

# --- Configuration ---
# This MUST match the database name in spin.toml (e.g., sqlite_databases = ["default"])
DB_NAME="default"
# This is the directory where Spin will store local data.
# We'll create it in the project root for clarity.
SPIN_DATA_DIR=".spin"
DB_DIR="$SPIN_DATA_DIR/sqlite"
DB_FILE="$DB_DIR/$DB_NAME.db"
SCHEMA_FILE="schema.sql"

# --- Pre-flight Checks ---
# Check if the sqlite3 command is available.
if ! command -v sqlite3 &> /dev/null; then
    echo "Error: sqlite3 command could not be found." >&2
    echo "Please install it. On macOS, use: brew install sqlite" >&2
    exit 1
fi

# Check if the schema file exists.
if [ ! -f "$SCHEMA_FILE" ]; then
    echo "Error: Schema file not found at '$SCHEMA_FILE'" >&2
    exit 1
fi

# --- Database Setup ---
echo "--- Setting up local database for Spin ---"

echo "Ensuring data directory exists at '$DB_DIR'..."
mkdir -p "$DB_DIR"

# If the database file already exists, remove it for a clean start.
if [ -f "$DB_FILE" ]; then
    echo "Removing existing database file: $DB_FILE"
    rm "$DB_FILE"
fi

echo "Creating new database and executing schema from '$SCHEMA_FILE'..."
# This command creates the DB file and executes the SQL script against it.
sqlite3 "$DB_FILE" < "$SCHEMA_FILE"

echo "Successfully created local database at '$DB_FILE'."
echo "You can now run 'spin up' to start your application."
echo "---"

Now I am trying to create this schema in the "cloud", however:

I have only one "default" database? How do I connect to the default database of that project?
In the code I use Connection::open_default().

My spin.toml looks like this:

spin_manifest_version = 2

[application]
name = "m4p"
version = "0.1.7"
authors = ["Felix Schütt <[email protected]>"]
description = ""

[[trigger.http]]
route = "/..."
component = "m4p"

[component.m4p]
source = "target/wasm32-wasip1/release/m4p.wasm"
allowed_outbound_hosts = ["https://github.com", "https://fonts.gstatic.com", "https://raw.githubusercontent.com", "https://maps4print.com", "https://*.paypal.com"]
sqlite_databases = ["default"]

[component.m4p.build]
command = "cargo build --release --package m4p --target wasm32-wasip1 --offline"
watch = ["src/**/*.rs", "Cargo.toml"]

So, before running spin deploy, I try to create a database:

#!/bin/bash

# --- Fermyon Cloud Database Deployment Script ---
# This script is designed for CI/CD environments. It ensures a database
# exists on Fermyon Cloud and applies the schema from `schema.sql`.

# Exit immediately if a command exits with a non-zero status.
set -e

# --- Configuration ---
# This MUST match the database name you want to use in Fermyon Cloud.
DB_NAME="default"
SCHEMA_FILE="schema.sql"

# --- Pre-flight Checks ---
# In a CI environment, you would have a step to install `spin` first.
if ! command -v spin &> /dev/null; then
    echo "Error: spin command could not be found." >&2
    echo "Please ensure the Fermyon Spin CLI is installed and in the PATH." >&2
    exit 1
fi

# Check if the schema file exists.
if [ ! -f "$SCHEMA_FILE" ]; then
    echo "Error: Schema file not found at '$SCHEMA_FILE'" >&2
    exit 1
fi

# In a CI environment, you must be logged in. This would typically be
# done with a token, e.g., `spin login --auth-method token --token ${{ secrets.FERMYON_TOKEN }}`
echo "Checking Spin login status..."
if ! spin cloud login --status &> /dev/null; then
    echo "Error: Not logged into Fermyon Cloud. Please run 'spin login'." >&2
    exit 1
fi

# --- Cloud Database Deployment ---
echo "--- Deploying database schema to Fermyon Cloud ---"

echo "Checking for existing cloud database '$DB_NAME'..."
# The `spin cloud sqlite create` command will fail if the DB already exists.
# We capture this to provide a clear message, but we don't treat it as a script-stopping error.
if spin cloud sqlite create "$DB_NAME"; then
  echo "Database '$DB_NAME' did not exist and has been created."
else
  echo "Database '$DB_NAME' already exists. Proceeding to apply schema."
fi

echo "Executing schema migrations for '$DB_NAME' from '$SCHEMA_FILE'..."
# The `-d` flag is for older spin versions. The modern syntax is just `spin cloud sqlite execute <DB_NAME>`.
# This is more robust for different CLI versions.
spin cloud sqlite execute --database "$DB_NAME" "$(cat $SCHEMA_FILE)"

echo "Successfully applied schema to Fermyon Cloud database '$DB_NAME'."
echo "---"

Result:

./setup_db_remote.sh 
Checking Spin login status...
--- Deploying database schema to Fermyon Cloud ---
Checking for existing cloud database 'default'...
Error: Problem creating database default

Caused by:
    {"":["AppId is required for 'default' databases."]}
Database 'default' already exists. Proceeding to apply schema.
Executing schema migrations for 'default' from 'schema.sql'...
error: Found argument '-- schema.sql

CREATE TABLE IF NOT EXISTS users (
    id TEXT PRIMARY KEY NOT NULL,
    company_name TEXT,
    company_vat_id TEXT,
    name TEXT NOT NULL,
    address_country TEXT NOT NULL,
    email TEXT UNIQUE NOT NULL,
    email_confirmed BOOLEAN NOT NULL,
    language TEXT NOT NULL,
    password_hash BLOB NOT NULL,
    authority TEXT NOT NULL
); ...

What / how do I specify an AppId? How do I execute SQL to create the database schema? I can't do "cat schema.sql" and specifying "schema.sql" (the filename) doesn't work either.

Now, since that didn't work, maybe I can work out the database stuff later and just get it to deploy. But spin deploy doesn't work either:

$ spin deploy
Uploading m4p version 0.1.7 to Fermyon Cloud...
Error: cannot push Spin application

Caused by:
    Server error: url https://registry.cloud.fermyon.com/v2/cloud-user-6abdf1dd-ea43-4eb7-b4bf-2e8f691a3a4a/blobs/uploads/3a2ba4ed-6555-344f-b2da-5643c3f42930?digest=sha256%3A092b82fd273b49b0a5a596de0355030f186fe4e082697fa652d1e792fd15da6c, code: 524, message: error code: 524

Learn more at https://developer.fermyon.com/cloud/faq

$ spin deploy         
Uploading m4p version 0.1.7 to Fermyon Cloud...
Error: cannot push Spin application

Caused by:
    Server error: url https://registry.cloud.fermyon.com/v2/cloud-user-6abdf1dd-ea43-4eb7-b4bf-2e8f691a3a4a/blobs/uploads/4ccc282d-4c5b-3607-b83d-9333c52a2a59?digest=sha256%3A092b82fd273b49b0a5a596de0355030f186fe4e082697fa652d1e792fd15da6c, code: 524, message: error code: 524

Learn more at https://developer.fermyon.com/cloud/faq

My .wasm file is 99.9 MB large, which is to be expected since I included a lot of binary static data. But it's not misconfigured. I am using the 4.0.0 version of the Rust spin-sdk. My potential recommendation would be namespaced databases, i.e. spin cloud sqlite execute project/component/database

Spin version (spin --version): spin 3.3.1 (6fd46d4 2025-06-17)
Installed plugin versions (spin plugins list --installed): cloud 0.11.0 [installed]

Aug 18 '25 21:08 fschutt

If anyone could test and get it to run in a cloud deployment, I'd be very happy. I built it with the wasip1 and it does work when running spin build --up, tests are running fine, etc. Only deployment / cloud database stuff doesn't work.

Aug 18 '25 22:08 fschutt

Setting up application databases

There's a distinction between database names (what the physical database is called in Fermyon Cloud) and database labels (how applications refer to the database). Labels abstract the database reference away from the app logic, so that default can refer to one database locally and one in Fermyon Cloud.

Database names are account scoped. Database labels are application scoped.

default is a label, and as such makes sense only in the context of a particular application.

It seems like what you want to do is create and set up an actual database, and then share it across applications. The way to do this is:

Create the database by database name (not label): spin cloud sqlite create my-db
Run the setup script by database name (not label): spin cloud sqlite execute -d my-db
Deploy the application, mapping its default label to the database name: spin cloud deploy --link sqlite:default=my-db

Note that you do not need an application ID for the first two steps if you use a database name rather than a label. Once an application has been created and its labels linked, you can execute against it using spin cloud execute -a <app> -l <label>. But -d will still work as well.

524 errors

I can't reproduce this here, but my guess is it's a size issue - I believe that Fermyon has a default app size limit of 100MB. I'll ask someone to look into it and get back to you.

Aug 18 '25 22:08 itowlson

By the way, would you mind sharing where you found this:

# The `-d` flag is for older spin versions. The modern syntax is just `spin cloud sqlite execute <DB_NAME>

The spin cloud sqlite execute <DB_NAME> syntax was removed some two years ago. The modern syntax is the -d syntax which you use. If there's docs that still advise the old syntax then I'll get them updated. Thanks!

Aug 18 '25 22:08 itowlson

524 errors

The default application size in Fermyon Cloud is indeed 100 MB, though as you've pointed out, this application should be (just) under. However, a different error would be received if this limit was exceeded (a not-very-helpful 500 error, but different nonetheless).

Here, we're seeing a 524 which is when a timeout is encountered by CloudFlare when waiting for the origin server to respond (default is 100s). It's possible that the large layer upload leads to this behavior. Is this failure mode consistent or intermittent? Does the deploy command in fact hang for ~100 seconds before returning the error? If consistent, we have some options to consider (increase timeout or look at spin client fixes, etc), so this will be helpful to know. Thank you.

Aug 19 '25 13:08 vdice

524 errors

The default application size in Fermyon Cloud is indeed 100 MB, though as you've pointed out, this application should be (just) under. However, a different error would be received if this limit was exceeded (a not-very-helpful 500 error, but different nonetheless).

Here, we're seeing a 524 which is when a timeout is encountered by CloudFlare when waiting for the origin server to respond (default is 100s). It's possible that the large layer upload leads to this behavior. Is this failure mode consistent or intermittent? Does the deploy command in fact hang for ~100 seconds before returning the error? If consistent, we have some options to consider (increase timeout or look at spin client fixes, etc), so this will be helpful to know. Thank you.

Okay @vdice it's likely a timeout error. If possible, please upload the package in chunks, that would terminally fix it (ideally with a progress bar). Upload speeds are not that great around the world. I've optimized the .wasm file to 60MB, it still may need > 1 minute for uploading, depending on internet connection.

Aug 20 '25 13:08 fschutt

Run the setup script by database name (not label): spin cloud sqlite execute -d my-db

@itowlson - yes, this step is failing, even if I do it this way. It seems that this part isn't tested well wrt. multiline SQL syntax scripts: If I remove all comments and newlines from the "schema.sql", THEN it works. But not with my multiline / comment-included schema.sql

Aug 20 '25 13:08 fschutt

Okay, it crashed once again, because the now half-uploaded 0.1.7 version was blocking any new packages.

spin cloud deploy --link sqlite:default=prod
Uploading m4p version 0.1.7 to Fermyon Cloud...
Error: cannot push Spin application

Caused by:
    Server error: url https://registry.cloud.fermyon.com/v2/m4p/blobs/uploads/, code: 520, message: error code: 520

Learn more at https://developer.fermyon.com/cloud/faq

The error code isn't really helpful and also, it should check whether the package version is already published BEFORE doing the package upload. I.e. do "preflight checks".

What would also be sensible, but not really required is to include git information (current tag) in the deployment, so that, for every live spin version, I can instantly correcate it with the correct repo+hash - so I don't have to do manual tagging with git. This could be an optional feature on "spin cloud deploy", but it would be helpful on errors.

Aug 20 '25 13:08 fschutt

I did now manage to deploy the 0.1.8 version:

spin cloud deploy --link sqlite:default=prod
Uploading m4p version 0.1.8 to Fermyon Cloud...
Deploying...
Waiting for application to become ready................... ready

View application:   https://m4p-bayqtigw.fermyon.app/
Manage application: https://cloud.fermyon.com/app/m4p

But the process on "how do I run spin cloud execute " against the default database of the application isn't really that well documented.

The process for "spin cloud sqlite isn't really documented well here: https://spinframework.dev/v3/sqlite-api-guide - so I would need to know how to actually finally deploy databases.

Also, visual feedback would be nice:

I only see my applications, but where do I see my databased, my running AI models, my ... ?

My "prod" database that I created is only visible after I figured out how to deploy. I can't inspect the database, export / restore, etc. A simple "spin cloud sqlite download" or something like that would be great for weekly backups.

The "logs" page is very unwieldy if the logs get long, no filtering, searching, piping the output to other log tools (Sentry, etc.)

I like the service and idea, but for real production use there's still a long way to go.

Kind regards, Felix

Aug 20 '25 14:08 fschutt

Thanks for persisting through various failure modes and taking the time to provide thoughtful and helpful feedback @fschutt. I see a handful of items to track.

Before I log off for the day I wanted to mention some items specifically around the deploy/upload errors. I spent some time today revisiting the oci client library and was reminded that it currently defaults to monolithic (eg not chunked) uploads for blobs during spin cloud deploy, just as you correctly suspected. I was able to adjust logic to force chunked uploads and pushed a similar app (with at least one large ~99MB layer) successfully. However, at least in my testing, I'm seeing overall upload times increase for this app when using chunked uploads (perhaps the additional req/resp overhead of the 99MB layer split into ~20 ~5MB upload requests?) So I'd like to do a little more investigation here.

There are also some configuration updates we can try on the registry server side to either avoid the 520 and 524 timeouts or at the least improve the error messages. (Though, hopefully, if chunked uploads are used or can be opted-into, those should disappear altogether, at least when timeout-related).

Aug 20 '25 16:08 vdice

spin cloud sqlite execute -d my-db ... is failing, even if I do it this way. It seems that this part isn't tested well wrt. multiline SQL syntax scripts: If I remove all comments and newlines from the "schema.sql", THEN it works. But not with my multiline / comment-included schema.sql

You can do spin cloud sqlite execute -d my-db @schema.sql (the @ prefix) to read the SQL from a file, similar to the Spin CLI. I see that's not covered in the command help - I'll get that fixed.

Aug 20 '25 20:08 itowlson

The process for "spin cloud sqlite isn't really documented well here: https://spinframework.dev/v3/sqlite-api-guide - so I would need to know how to actually finally deploy databases.

spinframework.dev covers only the open core CNCF project. Fermyon vendor content, including Fermyon Cloud, is at developer.fermyon.com. E.g. DB stuff at https://developer.fermyon.com/cloud/noops-sql-db#tables-and-data-in-fermyon-cloud (and the links/labels stuff at https://developer.fermyon.com/cloud/linking-applications-to-resources-using-labels).

Aug 20 '25 23:08 itowlson