[yugabyted] yb-tserver fails to start on Docker for Mac with host bind mount: initdb failed with exit code 1 preceded by Failed to pre-allocate space: Operation not supported (system error 95)

[gsantopaolo]

Jira Link: DB-16961

Description

When running YugabyteDB (version 2024.1.0.0-b129) in Docker Desktop on macOS Sequoia 15.5 using a host bind mount for the data directory, the yb-tserver process fails to start. The primary FATAL error in the yb-tserver log is Runtime error (yb/yql/pgwrapper/pg_wrapper.cc:781): /home/yugabyte/postgres/bin/initdb failed with exit code 1.

This initdb failure appears to be preceded by warnings/errors in the yb-tserver.INFO log related to filesystem operations on the host bind mount, specifically:

W0529 18:57:57.825258 1063 file_system.cc:85] Failed to pre-allocate space for a file: Not implemented (yb/util/file_system_posix.cc:425): ... MANIFEST-000020: Operation not supported (system error 95).

The issue does not occur if an incorrect, non-persistent volume mapping is used (e.g., mapping the host directory to an unused path like /root/var/data inside the container), in which case YugabyteDB starts using ephemeral storage but data is lost on restart. This indicates the problem is specific to the interaction with the host bind mount.

To Reproduce (code can be downloaded from: https://github.com/gsantopaolo/YugabyteDB_sample

Steps to reproduce the behavior:

Environment:

macOS Version: Sequoia 15.5

Docker Desktop Version: [Your Docker Desktop Version, e.g., 4.20.1]

YugabyteDB Image: yugabytedb/yugabyte:2024.1.0.0-b129

docker-compose.yml:

networks:
  yugabyte_net:
    name: yugabyte_net

services:
  yugabytedb:
    container_name: yugabytedb
    image: yugabytedb/yugabyte:2024.1.0.0-b129
    ports:
      - "7000:7000"
      - "9000:9000"
      - "15433:15433"
      - "5433:5433"
      - "9042:9042"
    command: [
      "/home/yugabyte/bin/yugabyted",
      "start",
      "--base_dir=/home/yugabyte/yb_data",
      "--background=false"
    ]
    environment:
      - LANG=en_US.UTF-8
      - LANGUAGE=en_US.UTF-8 # Or en_US:en
      - LC_ALL=en_US.UTF-8
    volumes:
      # Path on macOS host mapped to YB data directory
      - "../data/yugabytedb/single-node:/home/yugabyte/yb_data"
    restart: unless-stopped
    healthcheck:
      test: ["CMD-SHELL", "/home/yugabyte/postgres/bin/pg_isready -h localhost -p 5433 -U yugabyte || exit 1"]
      interval: 20s
      timeout: 10s
      retries: 5
      start_period: 30s
    networks:
      - yugabyte_net

Host Directory Preparation (on macOS):

# (Assuming docker-compose.yml is in a 'deployment' subdirectory)
mkdir -p ../data/yugabytedb/single-node
sudo chmod -R 777 ../data/yugabytedb/single-node
# Ensure the directory is empty before starting
sudo rm -rf ../data/yugabytedb/single-node/*

Start YugabyteDB:

sudo docker-compose -f docker-compose.yml up

Expected behavior The yugabytedb container should start successfully, initdb should complete without error, and the YSQL/YCQL endpoints should become available for connections, with data persisting on the macOS host directory.

Actual behavior The yb-tserver process within the container fails to start. docker-compose logs yugabytedb (or yb_startup_logs.txt from previous attempts) shows yugabyted starting, but connections to YSQL (port 5433) or YCQL (9042) fail.

Screenshots/Logs

yb-tserver.FATAL log snippet:

Log file created at: 2025/05/29 18:34:44
Current UTC time: 2025/05/29 18:34:44
Running on machine: b3385cf7a28a
Application fingerprint: version 2024.1.0.0 build 129 revision 03545b054cb8f3ab09e15f46bc26373fa6cca053 build_type RELEASE built at 30 May 2024 13:25:22 UTC
Node information: { hostname: 'b3385cf7a28a', rpc_ip: '172.26.0.2', webserver_ip: '172.26.0.2', uuid: '3fdc873435c144fe9442b8b2f9c52cef' }
Running duration (h:mm:ss): 0:00:00
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
F0529 18:34:44.412140   467 tablet_server_main_impl.cc:287] Runtime error (yb/yql/pgwrapper/pg_wrapper.cc:781): /home/yugabyte/postgres/bin/initdb failed with exit code 1
F20250529 18:34:44 ../../src/yb/tserver/tablet_server_main_impl.cc:287] Runtime error (yb/yql/pgwrapper/pg_wrapper.cc:781): /home/yugabyte/postgres/bin/initdb failed with exit code 1
    @     0xaaaae316b9bc  google::LogMessage::SendToLog()
    @     0xaaaae316c860  google::LogMessage::Flush()
    @     0xaaaae316cefc  google::LogMessageFatal::~LogMessageFatal()
    @     0xaaaae491b0fc  yb::tserver::TabletServerMain()
    @     0xaaaae311ce50  main
    @     0xffff8340e384  __libc_start_main
    @     0xaaaae303f034  (unknown)

Relevant yb-tserver.INFO log snippets:

I0529 18:57:57.621583  1011 pg_wrapper.cc:767] Launching initdb: [/home/yugabyte/postgres/bin/initdb, -D, /home/yugabyte/yb_data/data/pg_data_11, -U, postgres]
...
W0529 18:57:57.825258  1063 file_system.cc:85] Failed to pre-allocate space for a file: Not implemented (yb/util/file_system_posix.cc:425): /home/yugabyte/yb_data/data/yb-data/tserver/data/rocksdb/table-4b24297b16be4f8c98d28dd2e8b2c07b/tablet-a125f441e4b5454db796b54d888db70e/MANIFEST-000020: Operation not supported (system error 95)
...

(Followed by the FATAL initdb error as above)

Additional context

The issue seems related to filesystem operations (like pre-allocation via fallocate) on the macOS host bind mount not being fully supported or behaving as expected by YugabyteDB/RocksDB, leading to ENOTSUP (error 95). This, in turn, likely causes the subsequent initdb failure for the YSQL layer.

Setting broad permissions (chmod 777) on the macOS host directory and ensuring correct locale environment variables (LANG, LC_ALL) in the container does not resolve the issue.

The UID of the process inside the container appears to be root (UID 0).

(Optional, if you tested this: Mention if Docker Named Volumes work as a workaround, as this would strongly confirm the host bind mount is the issue source.)

Possible Questions/Areas for Investigation:

Is there a known incompatibility or specific configuration needed for YugabyteDB's RocksDB usage (particularly fallocate or other pre-allocation/fsync behavior) when using Docker for Mac with VirtioFS (or other sharing mechanisms) on macOS host bind mounts?

Are there any YugabyteDB t-server gflags that can alter filesystem pre-allocation behavior or make initdb more tolerant of filesystem idiosyncrasies on such shared volumes?

Could the actual error output from the initdb command itself be logged by pg_wrapper.cc to provide more specific reasons for its failure beyond exit code 1?

Warning: Please confirm that this issue does not contain any sensitive information

• [x] I confirm this issue does not contain any sensitive information.

[gsantopaolo]

@yugabyte-ci anyone?

[ddorian]

Hi @gsantopaolo

Can you fix this issue:

Failed to pre-allocate space for a file: Not implemented

You're using a filesystem that we don't support. See this another example & detail: https://stackoverflow.com/questions/78842948/failed-to-pre-allocate-space-when-creating-yugabytedb-cluster

[gsantopaolo]

hi @ddorian, thank you for the link, even though not being a file system expert, it's hard for me to understand.

Do you, or anyone else know, if there is a way to make yuga to store data on a volume on a mac?

[ddorian]

@gsantopaolo

Can you try running it without container https://download.yugabyte.com/#macos ? Does it work?

[gsantopaolo]

Hi @ddorian, I'm sure it will work, but for me and my team it's mandatory to make it work in a container so that our local dev env is the mnost similar the possible to our stg/test/prod envs. And today I spent another whole day on it, but nothing works. I'm wondering if MacOS + Docker is supported at alll

[ddorian]

I'm sure it will work,

Please try it. It's part of the troubleshooting process. It needs to be the same path that you were using for docker mount.

[gsantopaolo]

@ddorian I made it work, but it was a trial-and-error process; I was missing:

init: true
environment:
  - TINI_SUBREAPER=true

On my repo, the working compose (single and multi-node) https://github.com/gsantopaolo/YugabyteDB_sample/tree/main/deployment

So, to recap the issues:

1. bind mounts (like ./data/yugabyte:/home/yugabyte/yb_data) are served from the Mac host via VirtioFS (or, if you hadn’t enabled VirtioFS, via gRPC FUSE). Neither of these is ext4, which is the only file system compatible with yougabylte. The only way I found to serve a file system ext4 on a Mac was via named volumes. When you use a named volume, Docker stores its data inside the Linux VM’s disk image, which is formatted as an ext4 filesystem.

So I did something like:

volumes:
  yugabytedb-data:

And then in the service definition:

volumes:
      - yugabytedb-data:/home/yugabyte/yb_data

2. This way, data on Yuga is persisted on the disk. However, there is a caveat: if you stop and restart your container, when it starts up, it wipes all the data. To avoid that behaviour, I added:

    init: true
    environment:
      - TINI_SUBREAPER=true

This prevents the data from being wiped out when the container restarts.

I'm now adding mongo and postgres, then I'll do some perf comparison, and then hopefully a post on my blog

Thanks for the support, also many thanks to LexLuthr who gave me the last bit of the puzzle in this thread