server
server copied to clipboard
wandb
Fail to sign up due to 'entity already exists for the user local' ERROR
I set up wandb local docker on server and attempt to sign up in http://server_ip:8080 . After signing up, 'entity already exists for the user local' ERROR always appears, so that I cannot get API KEY.
Does anyone have any ideas?
Hey @lostork not sure how you got into that state. If you don't have any data in the instance, the simplest fix is to remove the persistent volume and start over:
docker stop wandb-local
docker volume rm wandb
wandb local
Then when you first access the instance you'll be prompted to create your admin account.
This happened to me as well when trying to created the user again after a failure due to mysql stopping itself (https://github.com/wandb/local/issues/40).
@carloalbertobarbano if you remove the volume, be sure to visit http://localhost:8080/logout or access the instance from an incognito window to ensure the initial account creation process is properly triggered.
These are the steps i followed:
-
I created a folder
/data/wandb-temp -
I started wandb by running
docker run --rm -d -u root -v /data/wandb-tmp:/vol -p 8080:8080 --name wandb-local-test wandb/local:latest -
This is the full output i get:
$ docker logs -f wandb-local-test
*** Running /etc/my_init.d/00_regen_ssh_host_keys.sh...
*** Running /etc/my_init.d/01_enable-services.sh...
*** Copying services to runit
*** Copying jobber template
*** Enabling production mode
*** Running /etc/my_init.d/02_load-settings.sh...
*** Loading settings...
2021/10/12 08:09:58 Created default user
2021/10/12 08:09:58 Generating new session key for auth...
2021/10/12 08:09:58 Generating new certificate and key for auth...
*** Booting runit daemon...
*** Runit started as PID 59
*** Setting up mysql database...
*** Starting wandb servers...
*** Configuring minio...
Bucket created successfully `local/local-files`.
Successfully added arn:minio:sqs:wandb-local:_:redis
*** Migrating database...
[mysql] 2021/10/12 08:11:14 packets.go:36: read tcp 127.0.0.1:52748->127.0.0.1:3306: i/o timeout
[mysql] 2021/10/12 08:11:14 connection.go:310: invalid connection
panic: migration failed in line 0: ALTER TABLE runs ADD COLUMN deleted_at datetime DEFAULT NULL;
(details: invalid connection) and driver: bad connection in line 0: SELECT RELEASE_LOCK(?)
goroutine 1 [running]:
main.main()
/mnt/ramdisk/core/services/gorilla/cmd/migrate/main.go:116 +0xb05
*** All services started
*** Access W&B at http://localhost:8080
*** Starting wandb servers...
*** All services started
*** Access W&B at http://localhost:8080
!!! Unable to start backend, see logs at /var/logs. Trying again in 1 minute
!!! Set the environment variable LOCAL_RESTORE=true if you're unable to login to the admin console
*** Starting wandb servers...
*** All services started
*** Access W&B at http://localhost:8080
*** Starting wandb servers...
*** All services started
*** Access W&B at http://localhost:8080
!!! Unable to start backend, see logs at /var/logs. Trying again in 1 minute
!!! Set the environment variable LOCAL_RESTORE=true if you're unable to login to the admin console
What I've tried so far, all of which resulted in the same log/error:
- I've completely nuked the wandb-tmp folder i created (tried multiple times)
- I've played with the permission of the wandb-tmp folder (tried many things, from chmod 777, to setting uid/gid based on the container running processes (999/0), to using -u root when launching docker)
- I tried accessing the instance using an incognito window
- I tried using the
wandb localmultiple times, deleting the volume created by docker.
Here are some of the relevant logs (i checked which files were modified) grabbed inside the container from /var/log:
From gorilla-filemeta.log (output similar to gorilla.log):
goroutine 1 [running]:
main.main()
/mnt/ramdisk/core/services/gorilla/cmd/gorilla-filemeta/main.go:115 +0x1055
{"level":"INFO","time":"2021-10-12T08:20:42.65090004Z","info":{"program":"gorilla-filemeta","source":"gorilla-filemeta/main.go:96","pid":15100},"data":{"config":"{MetadataStore:[REDACTED],FileMetadataSource:redis://127.0.0.1:6379/filemetadata,FileStore:s3://qvi4eO6wUuW59E2y5NlvnA==:[email protected]:9000/local-files,FileStoreIsProxied:true,FileHost:http://localhost:8080,DataFrameStore:noop://,TaskQueue:noop://,Onprem:true,BatchMDS:true,StorageBucket:s3://local-files,Tracer:noop://,Statsd:{Host: Port:0},SentryDSN:,SentryEnvironment:onprem-local,PProfAddr::8080,GCPProject:dev~wandb-local,GoogleApplicationCredentials:,AzureAccountKey:[REDACTED],MySQL:{DialTimeout:0s ReadTimeout:0s WriteTimeout:0s MaxIdleConns:0 MaxOpenConns:0 ConnMaxLifetime:0s}}"},"message":"Running with config {MetadataStore:[REDACTED],FileMetadataSource:redis://127.0.0.1:6379/filemetadata,FileStore:s3://qvi4eO6wUuW59E2y5NlvnA==:[email protected]:9000/local-files,FileStoreIsProxied:true,FileHost:http://localhost:8080,DataFrameStore:noop://,TaskQueue:noop://,Onprem:true,BatchMDS:true,StorageBucket:s3://local-files,Tracer:noop://,Statsd:{Host: Port:0},SentryDSN:,SentryEnvironment:onprem-local,PProfAddr::8080,GCPProject:dev~wandb-local,GoogleApplicationCredentials:,AzureAccountKey:[REDACTED],MySQL:{DialTimeout:0s ReadTimeout:0s WriteTimeout:0s MaxIdleConns:0 MaxOpenConns:0 ConnMaxLifetime:0s}}"}
panic: github.com/wandb/core/services/gorilla/mysql.(*metadataStore).prepareCountSweeps-fm: Error 1146: Table 'wandb_local.sweeps' doesn't exist
goroutine 1 [running]:
main.main()
/mnt/ramdisk/core/services/gorilla/cmd/gorilla-filemeta/main.go:115 +0x1055
From mysql.log:
2021-10-12T08:10:00.041756Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2021-10-12T08:10:01.010319Z 0 [Warning] InnoDB: New log files created, LSN=45790
2021-10-12T08:10:01.317132Z 0 [Warning] InnoDB: Creating foreign key constraint system tables.
2021-10-12T08:10:01.455292Z 0 [Warning] No existing UUID has been found, so we assume that this is the first time that this server has been started. Generating a new UUID: cc6b4b1c-2b33-11ec-bfe7-0242ac110005.
2021-10-12T08:10:01.486188Z 0 [Warning] Gtid table is not ready to be used. Table 'mysql.gtid_executed' cannot be opened.
2021-10-12T08:10:01.599882Z 0 [Warning] A deprecated TLS version TLSv1 is enabled. Please use TLSv1.2 or higher.
2021-10-12T08:10:01.599895Z 0 [Warning] A deprecated TLS version TLSv1.1 is enabled. Please use TLSv1.2 or higher.
2021-10-12T08:10:01.600310Z 0 [Warning] CA certificate ca.pem is self signed.
2021-10-12T08:10:01.659316Z 1 [Warning] root@localhost is created with an empty password ! Please consider switching off the --initialize-insecure option.
2021-10-12T08:10:16.636334Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2021-10-12T08:10:16.637245Z 0 [Note] mysqld (mysqld 5.7.35-0ubuntu0.18.04.2) starting as process 435 ...
2021-10-12T08:10:16.639580Z 0 [Note] InnoDB: PUNCH HOLE support available
2021-10-12T08:10:16.639592Z 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2021-10-12T08:10:16.639595Z 0 [Note] InnoDB: Uses event mutexes
2021-10-12T08:10:16.639597Z 0 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
2021-10-12T08:10:16.639599Z 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2021-10-12T08:10:16.639602Z 0 [Note] InnoDB: Using Linux native AIO
2021-10-12T08:10:16.639797Z 0 [Note] InnoDB: Number of pools: 1
2021-10-12T08:10:16.639852Z 0 [Note] InnoDB: Using CPU crc32 instructions
2021-10-12T08:10:16.640678Z 0 [Note] InnoDB: Initializing buffer pool, total size = 128M, instances = 1, chunk size = 128M
2021-10-12T08:10:16.645084Z 0 [Note] InnoDB: Completed initialization of buffer pool
2021-10-12T08:10:16.646269Z 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
2021-10-12T08:10:16.658860Z 0 [Note] InnoDB: Highest supported file format is Barracuda.
2021-10-12T08:10:16.736527Z 0 [Note] InnoDB: Creating shared tablespace for temporary tables
2021-10-12T08:10:16.736607Z 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
2021-10-12T08:10:16.845357Z 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
2021-10-12T08:10:16.846123Z 0 [Note] InnoDB: 96 redo rollback segment(s) found. 96 redo rollback segment(s) are active.
2021-10-12T08:10:16.846137Z 0 [Note] InnoDB: 32 non-redo rollback segment(s) are active.
2021-10-12T08:10:16.846424Z 0 [Note] InnoDB: Waiting for purge to start
2021-10-12T08:10:16.896579Z 0 [Note] InnoDB: 5.7.35 started; log sequence number 2749533
2021-10-12T08:10:16.896787Z 0 [Note] InnoDB: Loading buffer pool(s) from /vol/mysql/ib_buffer_pool
2021-10-12T08:10:16.897089Z 0 [Note] Plugin 'FEDERATED' is disabled.
2021-10-12T08:10:16.898389Z 0 [Note] InnoDB: Buffer pool(s) load completed at 211012 8:10:16
2021-10-12T08:10:16.907550Z 0 [Note] Found ca.pem, server-cert.pem and server-key.pem in data directory. Trying to enable SSL support using them.
2021-10-12T08:10:16.907582Z 0 [Note] Skipping generation of SSL certificates as certificate files are present in data directory.
2021-10-12T08:10:16.907591Z 0 [Warning] A deprecated TLS version TLSv1 is enabled. Please use TLSv1.2 or higher.
2021-10-12T08:10:16.907597Z 0 [Warning] A deprecated TLS version TLSv1.1 is enabled. Please use TLSv1.2 or higher.
2021-10-12T08:10:16.908709Z 0 [Warning] CA certificate ca.pem is self signed.
2021-10-12T08:10:16.908764Z 0 [Note] Skipping generation of RSA key pair as key files are present in data directory.
2021-10-12T08:10:16.908903Z 0 [Note] Server hostname (bind-address): '*'; port: 3306
2021-10-12T08:10:16.908943Z 0 [Note] IPv6 is available.
2021-10-12T08:10:16.908952Z 0 [Note] - '::' resolves to '::';
2021-10-12T08:10:16.908971Z 0 [Note] Server socket created on IP: '::'.
2021-10-12T08:10:16.926231Z 0 [Note] Event Scheduler: Loaded 0 events
2021-10-12T08:10:16.926510Z 0 [Note] mysqld: ready for connections.
Version: '5.7.35-0ubuntu0.18.04.2' socket: '/var/run/mysqld/mysqld.sock' port: 3306 (Ubuntu)
2021-10-12T08:11:14.085420Z 4 [Note] Aborted connection 4 to db: 'wandb_local' user: 'wandb_local' host: '127.0.0.1' (Got an error reading communication packets)
exec mysqld >> /var/log/mysql.log 2>&1
@carloalbertobarbano these errors are being caused by the initial migration of the database failing. This is likely due to filesystem driver issues or other docker specific issues with the filesystem being mounted in. The most important error is right after "*** Migrating database":
*** Migrating database...
[mysql] 2021/10/12 08:11:14 packets.go:36: read tcp 127.0.0.1:52748->127.0.0.1:3306: i/o timeout
[mysql] 2021/10/12 08:11:14 connection.go:310: invalid connection
panic: migration failed in line 0: ALTER TABLE runs ADD COLUMN deleted_at datetime DEFAULT NULL;
(details: invalid connection) and driver: bad connection in line 0: SELECT RELEASE_LOCK(?)
goroutine 1 [running]:
main.main()
/mnt/ramdisk/core/services/gorilla/cmd/migrate/main.go:116 +0xb05
This makes me think that the file system you're mounting in is a networked file system. During the initial migration the creation of all of the database files is causing a timeout. I would suggest trying to run the container with a local volume attached to it.
At the end of the day running MySQL inside of the container is not a good idea. We allow it for quick trial configurations, but it has poor performance and you'll risk dataloss. You'll want to get a license for the instance and connect a managed database with backups for any production workloads.
Actually, the file system is a locally mounted ext4 partition.. What is weird is that older versions of wandb/local (e.g.0.9.38) work without issues
Same, i got the same error for a locally mounted ext4 partition for newest version of wandb/local (0.9.44) . Its not a networked file system. Plus, its a clean install of wandb. Another information I can add is that this partition has "nosuid", not sure if that affects. But it affected me when I exec bash into the docker and tried to run "sudo mysql". I couldn't sudo because my partition is mounted with "nosuid".
The error is:
(details: invalid connection) and driver: bad connection in line 0: SELECT RELEASE_LOCK(?)
Actually I tried every recent version and discovered that 0.9.44, 0.9.43 and 0.9.42 has this issue.
However, for 0.9.39, it works well. For 0.9.40, i had to wait more than 30 minutes, but it loaded in the end. 0.9.41 also loaded successfully. Therefore it should be a change between 0.9.41 and 0.9.42 that caused this issue.
heres the full error from docker logs wandb-local:
*** Migrating database...
[mysql] 2021/10/20 04:58:03 packets.go:36: read tcp 127.0.0.1:37330->127.0.0.1:3306: i/o timeout
[mysql] 2021/10/20 04:58:03 connection.go:310: invalid connection
panic: migration failed in line 0: CREATE TABLE subscriptions (
created_at datetime NOT NULL,
updated_at datetime DEFAULT NULL,
id int(11) NOT NULL AUTO_INCREMENT,
name varchar(64) DEFAULT NULL,
limits json DEFAULT NULL,
`usage` json DEFAULT NULL,
start_date datetime DEFAULT NULL,
end_date datetime DEFAULT NULL,
PRIMARY KEY (id)
)
CHARACTER SET utf8mb4
COLLATE utf8mb4_general_ci;
CREATE TABLE entities (
created_at datetime NOT NULL,
updated_at datetime DEFAULT NULL,
id int(11) NOT NULL AUTO_INCREMENT,
name varchar(64) DEFAULT NULL,
photo_url varchar(256) DEFAULT NULL,
subscription_id int(11) DEFAULT NULL,
PRIMARY KEY (id),
UNIQUE KEY ix_entities_name (name),
KEY subscription_id (subscription_id),
CONSTRAINT entities_ibfk_1 FOREIGN KEY (subscription_id) REFERENCES subscriptions (id)
)
CHARACTER SET utf8mb4
COLLATE utf8mb4_general_ci;
CREATE TABLE users (
created_at datetime NOT NULL,
updated_at datetime DEFAULT NULL,
id int(11) NOT NULL AUTO_INCREMENT,
email varchar(64) DEFAULT NULL,
auth_id varchar(64) DEFAULT NULL,
name varchar(64) DEFAULT NULL,
username varchar(64) DEFAULT NULL,
photo_url varchar(256) DEFAULT NULL,
admin tinyint(1) DEFAULT NULL,
default_entity_id int(11) DEFAULT NULL,
account_type varchar(64) DEFAULT NULL,
logged_in_at datetime DEFAULT NULL,
analytics json DEFAULT NULL,
PRIMARY KEY (id),
UNIQUE KEY ix_users_auth_id (auth_id),
UNIQUE KEY ix_users_email (email),
UNIQUE KEY ix_users_username (username),
KEY default_entity_id (default_entity_id),
CONSTRAINT users_ibfk_1 FOREIGN KEY (default_entity_id) REFERENCES entities (id)
)
CHARACTER SET utf8mb4
COLLATE utf8mb4_general_ci;
CREATE TABLE projects (
created_at datetime NOT NULL,
updated_at datetime DEFAULT NULL,
id int(11) NOT NULL AUTO_INCREMENT,
entity_id int(11) DEFAULT NULL,
user_id int(11) DEFAULT NULL,
name varchar(128) DEFAULT NULL,
access varchar(64) DEFAULT NULL,
repo varchar(256) DEFAULT NULL,
views json DEFAULT NULL,
description text,
group_path varchar(256) DEFAULT NULL,
sub_group_path varchar(256) DEFAULT NULL,
featured int(11) DEFAULT NULL,
analytics json DEFAULT NULL,
PRIMARY KEY (id),
UNIQUE KEY ix_projects_name_entity_id (name,entity_id),
KEY entity_id (entity_id),
KEY user_id (user_id),
CONSTRAINT projects_ibfk_1 FOREIGN KEY (entity_id) REFERENCES entities (id),
CONSTRAINT projects_ibfk_2 FOREIGN KEY (user_id) REFERENCES users (id)
)
CHARACTER SET utf8mb4
COLLATE utf8mb4_general_ci;
CREATE TABLE api_keys (
`key` varchar(128) NOT NULL,
user_id int(11) DEFAULT NULL,
description varchar(512) DEFAULT NULL,
ips json DEFAULT NULL,
rate_limit varchar(128) DEFAULT NULL,
PRIMARY KEY (`key`),
KEY ix_api_keys_user_id (user_id),
CONSTRAINT api_keys_ibfk_1 FOREIGN KEY (user_id) REFERENCES users (id)
)
CHARACTER SET utf8mb4
COLLATE utf8mb4_general_ci;
CREATE TABLE files (
created_at datetime NOT NULL,
updated_at datetime DEFAULT NULL,
id int(11) NOT NULL AUTO_INCREMENT,
name varchar(512) DEFAULT NULL,
mimetype varchar(64) DEFAULT NULL,
project_id int(11) DEFAULT NULL,
run_name varchar(128) DEFAULT NULL,
meta json DEFAULT NULL,
storage_bucket varchar(128) DEFAULT NULL,
special tinyint(1) DEFAULT NULL,
archived tinyint(1) DEFAULT NULL,
deleted_at datetime DEFAULT NULL,
PRIMARY KEY (id),
UNIQUE KEY ix_files_name_run_name_project_id (name,run_name,project_id),
UNIQUE KEY ix_files_project_id_run_name_name (project_id,run_name,name),
CONSTRAINT files_ibfk_1 FOREIGN KEY (project_id) REFERENCES projects (id)
)
CHARACTER SET utf8mb4
COLLATE utf8mb4_general_ci;
CREATE TABLE invites (
created_at datetime NOT NULL,
updated_at datetime DEFAULT NULL,
`key` varchar(64) NOT NULL,
from_user_id int(11) DEFAULT NULL,
to_user_id int(11) DEFAULT NULL,
entity_id int(11) NOT NULL,
email varchar(64) DEFAULT NULL,
accepted tinyint(1) NOT NULL,
expires_at datetime DEFAULT NULL,
PRIMARY KEY (`key`),
KEY from_user_id (from_user_id),
KEY to_user_id (to_user_id),
KEY ix_invites_entity (entity_id),
CONSTRAINT invites_ibfk_1 FOREIGN KEY (entity_id) REFERENCES entities (id),
CONSTRAINT invites_ibfk_2 FOREIGN KEY (from_user_id) REFERENCES users (id),
CONSTRAINT invites_ibfk_3 FOREIGN KEY (to_user_id) REFERENCES users (id)
)
CHARACTER SET utf8mb4
COLLATE utf8mb4_general_ci;
CREATE TABLE jobs (
created_at datetime NOT NULL,
updated_at datetime DEFAULT NULL,
id int(11) NOT NULL AUTO_INCREMENT,
type varchar(64) DEFAULT NULL,
description text,
program varchar(256) DEFAULT NULL,
repo varchar(256) DEFAULT NULL,
PRIMARY KEY (id)
)
CHARACTER SET utf8mb4
COLLATE utf8mb4_general_ci;
CREATE TABLE run_fields (
project_id int(11) NOT NULL,
`column` varchar(64) NOT NULL,
field varchar(256) NOT NULL,
count int(11) DEFAULT NULL,
PRIMARY KEY (project_id,`column`,field),
KEY ix_run_fields_count (count)
)
CHARACTER SET utf8mb4
COLLATE utf8mb4_general_ci;
CREATE TABLE tags (
id int(11) NOT NULL AUTO_INCREMENT,
name varchar(64) NOT NULL,
PRIMARY KEY (id),
UNIQUE KEY name (name)
)
CHARACTER SET utf8mb4
COLLATE utf8mb4_general_ci;
CREATE TABLE run_tags (
project_id int(11) NOT NULL,
run_name varchar(64) NOT NULL,
tag_id int(11) NOT NULL,
PRIMARY KEY (project_id,run_name,tag_id),
KEY run_tags_ibfk_2 (tag_id),
CONSTRAINT run_tags_ibfk_1 FOREIGN KEY (project_id) REFERENCES projects (id),
CONSTRAINT run_tags_ibfk_2 FOREIGN KEY (tag_id) REFERENCES tags (id)
)
CHARACTER SET utf8mb4
COLLATE utf8mb4_general_ci;
CREATE TABLE runs (
created_at datetime NOT NULL,
updated_at datetime DEFAULT NULL,
project_id int(11) NOT NULL,
name varchar(128) NOT NULL,
job_id int(11) DEFAULT NULL,
user_id int(11) DEFAULT NULL,
sweep_name varchar(64) DEFAULT NULL,
config json DEFAULT NULL,
summary_metrics json DEFAULT NULL,
system_metrics json DEFAULT NULL,
state varchar(64) DEFAULT NULL,
commit varchar(64) DEFAULT NULL,
host varchar(64) DEFAULT NULL,
exitcode bigint(20) DEFAULT NULL,
description text,
heartbeat_at datetime DEFAULT NULL,
group_name varchar(128) DEFAULT NULL,
subgroup_name varchar(128) DEFAULT NULL,
PRIMARY KEY (project_id,name),
KEY ix_runs_user_id (user_id)
)
CHARACTER SET utf8mb4
COLLATE utf8mb4_general_ci
PARTITION BY HASH (project_id)
PARTITIONS 101;
CREATE TABLE teams (
user_id int(11) NOT NULL,
entity_id int(11) NOT NULL,
type varchar(64) DEFAULT NULL,
name varchar(64) DEFAULT NULL,
PRIMARY KEY (user_id,entity_id),
KEY entity_id (entity_id),
CONSTRAINT teams_ibfk_1 FOREIGN KEY (entity_id) REFERENCES entities (id),
CONSTRAINT teams_ibfk_2 FOREIGN KEY (user_id) REFERENCES users (id)
)
CHARACTER SET utf8mb4
COLLATE utf8mb4_general_ci;
CREATE TABLE users_auth0 (
user_id int(11) NOT NULL,
auth_id varchar(64) NOT NULL,
PRIMARY KEY (user_id,auth_id),
CONSTRAINT users_auth0_ibfk_1 FOREIGN KEY (user_id) REFERENCES users (id)
)
CHARACTER SET utf8mb4
COLLATE utf8mb4_general_ci;
CREATE TABLE views (
created_at datetime NOT NULL,
updated_at datetime DEFAULT NULL,
id int(11) NOT NULL AUTO_INCREMENT,
name varchar(128) DEFAULT NULL,
type varchar(128) DEFAULT NULL,
user_id int(11) DEFAULT NULL,
project_id int(11) DEFAULT NULL,
description text,
spec text,
PRIMARY KEY (id),
UNIQUE KEY uniq_view__project_user_name (project_id,user_id,name,type),
KEY user_id (user_id),
CONSTRAINT views_ibfk_1 FOREIGN KEY (project_id) REFERENCES projects (id),
CONSTRAINT views_ibfk_2 FOREIGN KEY (user_id) REFERENCES users (id)
)
CHARACTER SET utf8mb4
COLLATE utf8mb4_general_ci;
(details: invalid connection) and driver: bad connection in line 0: SELECT RELEASE_LOCK(?)
goroutine 1 [running]:
main.main()
/mnt/ramdisk/core/services/gorilla/cmd/migrate/main.go:116 +0xb05
@leeyh20 I'm not sure this is related to a specific version of wandb or simply the amount of MySQL migrations that are required in later versions. Each release generally contains some additional schema information that needs to be updated in the database. I'm curious if the migration is able to recover after it fails. To test this, you can stop the failed container and start it back up with the same volume mounted in it.
docker stop wandb-local
wandb local
If the migrations are taking more than a minute or two there must be something misconfigured with the docker file system driver that is negatively impacting IO. In general wandb/local should only be run with an external MySQL database for production setups. Are you planning to use this with teamates, or are you just running this for yourself?
Just tried the latest versions..still not working on a fresh ubuntu install
@carloalbertobarbano are you getting the "RELEASE LOCK" error, or the "entity already exists for user local" error? Can you share the output of docker logs wandb-local?
I'm still getting errors with the database migration
*** Migrating database...
panic: dial tcp 127.0.0.1:3306: connect: connection refused
goroutine 1 [running]:
main.main()
/mnt/ramdisk/core/services/gorilla/cmd/migrate/main.go:50 +0x8d4
This is the content:
2021-11-10T21:10:33.908497Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2021-11-10T21:10:35.104233Z 0 [Warning] InnoDB: New log files created, LSN=45790
2021-11-10T21:10:35.481787Z 0 [Warning] InnoDB: Creating foreign key constraint system tables.
2021-11-10T21:10:35.629346Z 0 [Warning] No existing UUID has been found, so we assume that this is the first time that this server has been started. Generating a new UUID: a5bdde90-426a-11ec-af1c-0242ac110002.
2021-11-10T21:10:35.664306Z 0 [Warning] Gtid table is not ready to be used. Table 'mysql.gtid_executed' cannot be opened.
2021-11-10T21:10:36.458295Z 0 [Warning] A deprecated TLS version TLSv1 is enabled. Please use TLSv1.2 or higher.
2021-11-10T21:10:36.458330Z 0 [Warning] A deprecated TLS version TLSv1.1 is enabled. Please use TLSv1.2 or higher.
2021-11-10T21:10:36.458778Z 0 [Warning] CA certificate ca.pem is self signed.
2021-11-10T21:10:36.654130Z 1 [Warning] root@localhost is created with an empty password ! Please consider switching off the --initialize-insecure option.
2021-11-10T21:11:04.285548Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2021-11-10T21:11:04.287591Z 0 [Note] mysqld (mysqld 5.7.36) starting as process 346 ...
2021-11-10T21:11:04.288761Z 0 [ERROR] Fatal error: Please read "Security" section of the manual to find out how to run mysqld as root!
2021-11-10T21:11:04.288792Z 0 [ERROR] Aborting
2021-11-10T21:11:04.288810Z 0 [Note] Binlog end
2021-11-10T21:11:04.289873Z 0 [Note] mysqld: Shutdown complete
./run: line 35: fg: job has terminated
After the first block it is repeated from "TIMESTAMP [...]" on
The command I'm issuing is:
docker run --rm -d -u root -v /data/wandb-storage:/vol -p 8080:8080 --name wandb-local wandb/local:latest
This is helpful, MySQL is not allowing you to start it as root. Our system is designed to be started as an untrusted user as long as it's started with the root group. Can you try running the command with -u 1001 -g 0 instead of -u root?
-g is unrecognized, did you mean -u 1001:0 ?
With docker run --rm -d -u 1001:0 -v /data/wandb-storage2:/vol -p 8080:8080 --name wandb-local wandb/local:latest I'm getting:
docker logs -f wandb-local
*** Running /etc/my_init.d/00_regen_ssh_host_keys.sh...
*** Running /etc/my_init.d/01_enable-services.sh...
*** Copying services to runit
*** Copying jobber template
*** Enabling production mode
*** Running /etc/my_init.d/02_load-settings.sh...
*** Loading settings...
2021/11/10 21:53:36 Created default user
2021/11/10 21:53:36 Generating new session key for auth...
2021/11/10 21:53:36 Generating new certificate and key for auth...
*** Booting runit daemon...
*** Runit started as PID 49
*** Setting up mysql database...
mc: <ERROR> Unable to get mcConfigDir. exit status 2.
*** Starting wandb servers...
*** Minio still starting up...
*** Still waiting...
mc: <ERROR> Unable to get mcConfigDir. exit status 2.
!!! Something's not right
ERROR config-dir option must be provided: Invalid arguments specified
*** Configuring minio...
mc: <ERROR> Unable to get mcConfigDir. exit status 2.
mc: <ERROR> Unable to get mcConfigDir. exit status 2.
mc: <ERROR> Unable to get mcConfigDir. exit status 2.
*** Minio still starting up...
*** Still waiting...
mc: <ERROR> Unable to get mcConfigDir. exit status 2.
!!! Something's not right
ERROR config-dir option must be provided: Invalid arguments specified
ERROR config-dir option must be provided: Invalid arguments specified
mc: <ERROR> Unable to get mcConfigDir. exit status 2.
find: ‘//.mc’: No such file or directory
cp: cannot stat '//.mc/config.json': No such file or directory
find: ‘/vol/env/minio-config.json’: No such file or directory
cp: cannot stat '/vol/env/minio-config.json': No such file or directory
./run: line 1: kill: (274) - No such process
mc: <ERROR> Unable to get mcConfigDir. exit status 2.
*** Minio still starting up...
*** Still waiting...
mc: <ERROR> Unable to get mcConfigDir. exit status 2.
!!! Something's not right
ERROR config-dir option must be provided: Invalid arguments specified
ERROR config-dir option must be provided: Invalid arguments specified
ERROR config-dir option must be provided: Invalid arguments specified
*** Configuring minio...
mc: <ERROR> Unable to get mcConfigDir. exit status 2.
mc: <ERROR> Unable to get mcConfigDir. exit status 2.
mc: <ERROR> Unable to get mcConfigDir. exit status 2.
*** Minio still starting up...
*** Still waiting...
and so on
If I create a new storage folder, chmod it to 777 and start the container with: docker run --rm -it -v /data/wandb-storage2:/vol -p 8080:8080 --name wandb-local wandb/local:latest
I get the following:
docker run --rm -it -v /data/wandb-storage2:/vol -p 8080:8080 --name wandb-local wandb/local:latest
*** Running /etc/my_init.d/00_regen_ssh_host_keys.sh...
*** Running /etc/my_init.d/01_enable-services.sh...
*** Copying services to runit
*** Copying jobber template
*** Enabling production mode
*** Running /etc/my_init.d/02_load-settings.sh...
*** Loading settings...
2021/11/10 21:58:44 Created default user
2021/11/10 21:58:44 Generating new session key for auth...
2021/11/10 21:58:44 Generating new certificate and key for auth...
*** Booting runit daemon...
*** Runit started as PID 48
*** Setting up mysql database...
*** Starting wandb servers...
*** Configuring minio...
Bucket created successfully `local/local-files`.
Successfully added arn:minio:sqs:wandb-local:_:redis
*** Migrating database...
[mysql] 2021/11/10 22:00:27 packets.go:36: read tcp 127.0.0.1:43520->127.0.0.1:3306: i/o timeout
[mysql] 2021/11/10 22:00:27 connection.go:310: invalid connection
panic: migration failed in line 0: CREATE TABLE subscriptions (
created_at datetime NOT NULL,
updated_at datetime DEFAULT NULL,
id int(11) NOT NULL AUTO_INCREMENT,
name varchar(64) DEFAULT NULL,
limits json DEFAULT NULL,
`usage` json DEFAULT NULL,
start_date datetime DEFAULT NULL,
end_date datetime DEFAULT NULL,
PRIMARY KEY (id)
)
CHARACTER SET utf8mb4
COLLATE utf8mb4_general_ci;
CREATE TABLE entities (
created_at datetime NOT NULL,
updated_at datetime DEFAULT NULL,
id int(11) NOT NULL AUTO_INCREMENT,
name varchar(64) DEFAULT NULL,
photo_url varchar(256) DEFAULT NULL,
subscription_id int(11) DEFAULT NULL,
PRIMARY KEY (id),
UNIQUE KEY ix_entities_name (name),
KEY subscription_id (subscription_id),
CONSTRAINT entities_ibfk_1 FOREIGN KEY (subscription_id) REFERENCES subscriptions (id)
)
CHARACTER SET utf8mb4
COLLATE utf8mb4_general_ci;
CREATE TABLE users (
created_at datetime NOT NULL,
updated_at datetime DEFAULT NULL,
id int(11) NOT NULL AUTO_INCREMENT,
email varchar(64) DEFAULT NULL,
auth_id varchar(64) DEFAULT NULL,
name varchar(64) DEFAULT NULL,
username varchar(64) DEFAULT NULL,
photo_url varchar(256) DEFAULT NULL,
admin tinyint(1) DEFAULT NULL,
default_entity_id int(11) DEFAULT NULL,
account_type varchar(64) DEFAULT NULL,
logged_in_at datetime DEFAULT NULL,
analytics json DEFAULT NULL,
PRIMARY KEY (id),
UNIQUE KEY ix_users_auth_id (auth_id),
UNIQUE KEY ix_users_email (email),
UNIQUE KEY ix_users_username (username),
KEY default_entity_id (default_entity_id),
CONSTRAINT users_ibfk_1 FOREIGN KEY (default_entity_id) REFERENCES entities (id)
)
CHARACTER SET utf8mb4
COLLATE utf8mb4_general_ci;
CREATE TABLE projects (
created_at datetime NOT NULL,
updated_at datetime DEFAULT NULL,
id int(11) NOT NULL AUTO_INCREMENT,
entity_id int(11) DEFAULT NULL,
user_id int(11) DEFAULT NULL,
name varchar(128) DEFAULT NULL,
access varchar(64) DEFAULT NULL,
repo varchar(256) DEFAULT NULL,
views json DEFAULT NULL,
description text,
group_path varchar(256) DEFAULT NULL,
sub_group_path varchar(256) DEFAULT NULL,
featured int(11) DEFAULT NULL,
analytics json DEFAULT NULL,
PRIMARY KEY (id),
UNIQUE KEY ix_projects_name_entity_id (name,entity_id),
KEY entity_id (entity_id),
KEY user_id (user_id),
CONSTRAINT projects_ibfk_1 FOREIGN KEY (entity_id) REFERENCES entities (id),
CONSTRAINT projects_ibfk_2 FOREIGN KEY (user_id) REFERENCES users (id)
)
CHARACTER SET utf8mb4
COLLATE utf8mb4_general_ci;
CREATE TABLE api_keys (
`key` varchar(128) NOT NULL,
user_id int(11) DEFAULT NULL,
description varchar(512) DEFAULT NULL,
ips json DEFAULT NULL,
rate_limit varchar(128) DEFAULT NULL,
PRIMARY KEY (`key`),
KEY ix_api_keys_user_id (user_id),
CONSTRAINT api_keys_ibfk_1 FOREIGN KEY (user_id) REFERENCES users (id)
)
CHARACTER SET utf8mb4
COLLATE utf8mb4_general_ci;
CREATE TABLE files (
created_at datetime NOT NULL,
updated_at datetime DEFAULT NULL,
id int(11) NOT NULL AUTO_INCREMENT,
name varchar(512) DEFAULT NULL,
mimetype varchar(64) DEFAULT NULL,
project_id int(11) DEFAULT NULL,
run_name varchar(128) DEFAULT NULL,
meta json DEFAULT NULL,
storage_bucket varchar(128) DEFAULT NULL,
special tinyint(1) DEFAULT NULL,
archived tinyint(1) DEFAULT NULL,
deleted_at datetime DEFAULT NULL,
PRIMARY KEY (id),
UNIQUE KEY ix_files_name_run_name_project_id (name,run_name,project_id),
UNIQUE KEY ix_files_project_id_run_name_name (project_id,run_name,name),
CONSTRAINT files_ibfk_1 FOREIGN KEY (project_id) REFERENCES projects (id)
)
CHARACTER SET utf8mb4
COLLATE utf8mb4_general_ci;
CREATE TABLE invites (
created_at datetime NOT NULL,
updated_at datetime DEFAULT NULL,
`key` varchar(64) NOT NULL,
from_user_id int(11) DEFAULT NULL,
to_user_id int(11) DEFAULT NULL,
entity_id int(11) NOT NULL,
email varchar(64) DEFAULT NULL,
accepted tinyint(1) NOT NULL,
expires_at datetime DEFAULT NULL,
PRIMARY KEY (`key`),
KEY from_user_id (from_user_id),
KEY to_user_id (to_user_id),
KEY ix_invites_entity (entity_id),
CONSTRAINT invites_ibfk_1 FOREIGN KEY (entity_id) REFERENCES entities (id),
CONSTRAINT invites_ibfk_2 FOREIGN KEY (from_user_id) REFERENCES users (id),
CONSTRAINT invites_ibfk_3 FOREIGN KEY (to_user_id) REFERENCES users (id)
)
CHARACTER SET utf8mb4
COLLATE utf8mb4_general_ci;
CREATE TABLE jobs (
created_at datetime NOT NULL,
updated_at datetime DEFAULT NULL,
id int(11) NOT NULL AUTO_INCREMENT,
type varchar(64) DEFAULT NULL,
description text,
program varchar(256) DEFAULT NULL,
repo varchar(256) DEFAULT NULL,
PRIMARY KEY (id)
)
CHARACTER SET utf8mb4
COLLATE utf8mb4_general_ci;
CREATE TABLE run_fields (
project_id int(11) NOT NULL,
`column` varchar(64) NOT NULL,
field varchar(256) NOT NULL,
count int(11) DEFAULT NULL,
PRIMARY KEY (project_id,`column`,field),
KEY ix_run_fields_count (count)
)
CHARACTER SET utf8mb4
COLLATE utf8mb4_general_ci;
CREATE TABLE tags (
id int(11) NOT NULL AUTO_INCREMENT,
name varchar(64) NOT NULL,
PRIMARY KEY (id),
UNIQUE KEY name (name)
)
CHARACTER SET utf8mb4
COLLATE utf8mb4_general_ci;
CREATE TABLE run_tags (
project_id int(11) NOT NULL,
run_name varchar(64) NOT NULL,
tag_id int(11) NOT NULL,
PRIMARY KEY (project_id,run_name,tag_id),
KEY run_tags_ibfk_2 (tag_id),
CONSTRAINT run_tags_ibfk_1 FOREIGN KEY (project_id) REFERENCES projects (id),
CONSTRAINT run_tags_ibfk_2 FOREIGN KEY (tag_id) REFERENCES tags (id)
)
CHARACTER SET utf8mb4
COLLATE utf8mb4_general_ci;
CREATE TABLE runs (
created_at datetime NOT NULL,
updated_at datetime DEFAULT NULL,
project_id int(11) NOT NULL,
name varchar(128) NOT NULL,
job_id int(11) DEFAULT NULL,
user_id int(11) DEFAULT NULL,
sweep_name varchar(64) DEFAULT NULL,
config json DEFAULT NULL,
summary_metrics json DEFAULT NULL,
system_metrics json DEFAULT NULL,
state varchar(64) DEFAULT NULL,
commit varchar(64) DEFAULT NULL,
host varchar(64) DEFAULT NULL,
exitcode bigint(20) DEFAULT NULL,
description text,
heartbeat_at datetime DEFAULT NULL,
group_name varchar(128) DEFAULT NULL,
subgroup_name varchar(128) DEFAULT NULL,
PRIMARY KEY (project_id,name),
KEY ix_runs_user_id (user_id)
)
CHARACTER SET utf8mb4
COLLATE utf8mb4_general_ci
PARTITION BY HASH (project_id)
PARTITIONS 101;
CREATE TABLE teams (
user_id int(11) NOT NULL,
entity_id int(11) NOT NULL,
type varchar(64) DEFAULT NULL,
name varchar(64) DEFAULT NULL,
PRIMARY KEY (user_id,entity_id),
KEY entity_id (entity_id),
CONSTRAINT teams_ibfk_1 FOREIGN KEY (entity_id) REFERENCES entities (id),
CONSTRAINT teams_ibfk_2 FOREIGN KEY (user_id) REFERENCES users (id)
)
CHARACTER SET utf8mb4
COLLATE utf8mb4_general_ci;
CREATE TABLE users_auth0 (
user_id int(11) NOT NULL,
auth_id varchar(64) NOT NULL,
PRIMARY KEY (user_id,auth_id),
CONSTRAINT users_auth0_ibfk_1 FOREIGN KEY (user_id) REFERENCES users (id)
)
CHARACTER SET utf8mb4
COLLATE utf8mb4_general_ci;
CREATE TABLE views (
created_at datetime NOT NULL,
updated_at datetime DEFAULT NULL,
id int(11) NOT NULL AUTO_INCREMENT,
name varchar(128) DEFAULT NULL,
type varchar(128) DEFAULT NULL,
user_id int(11) DEFAULT NULL,
project_id int(11) DEFAULT NULL,
description text,
spec text,
PRIMARY KEY (id),
UNIQUE KEY uniq_view__project_user_name (project_id,user_id,name,type),
KEY user_id (user_id),
CONSTRAINT views_ibfk_1 FOREIGN KEY (project_id) REFERENCES projects (id),
CONSTRAINT views_ibfk_2 FOREIGN KEY (user_id) REFERENCES users (id)
)
CHARACTER SET utf8mb4
COLLATE utf8mb4_general_ci;
(details: invalid connection) and driver: bad connection in line 0: SELECT RELEASE_LOCK(?)
goroutine 1 [running]:
main.main()
/mnt/ramdisk/core/services/gorilla/cmd/migrate/main.go:116 +0x856
*** All services started
*** Access W&B at http://localhost:8080
*** Logging disabled, find logs at /var/log/gorilla.log or enable by setting LOGGING_ENABLED=true
*** Starting wandb servers...
*** All services started
*** Access W&B at http://localhost:8080
*** Logging disabled, find logs at /var/log/gorilla.log or enable by setting LOGGING_ENABLED=true
!!! Unable to start backend, see logs at /var/logs. Trying again in 1 minute
!!! Set the environment variable LOCAL_RESTORE=true if you're unable to login to the admin console
*** Starting wandb servers...
*** All services started
*** Access W&B at http://localhost:8080
*** Logging disabled, find logs at /var/log/gorilla.log or enable by setting LOGGING_ENABLED=true
*** Starting wandb servers...
*** All services started
*** Access W&B at http://localhost:8080
*** Logging disabled, find logs at /var/log/gorilla.log or enable by setting LOGGING_ENABLED=true
!!! Unable to start backend, see logs at /var/logs. Trying again in 1 minute
!!! Set the environment variable LOCAL_RESTORE=true if you're unable to login to the admin console
The /var/log/mysql.log file now contains:
021-11-10T21:58:45.294076Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2021-11-10T21:58:46.945256Z 0 [Warning] InnoDB: New log files created, LSN=45790
2021-11-10T21:58:47.828044Z 0 [Warning] InnoDB: Creating foreign key constraint system tables.
2021-11-10T21:58:48.122890Z 0 [Warning] No existing UUID has been found, so we assume that this is the first time that this server has been started. Generating a new UUID: 61cd5ba1-4271-11ec-a972-0242ac110002.
2021-11-10T21:58:48.285890Z 0 [Warning] Gtid table is not ready to be used. Table 'mysql.gtid_executed' cannot be opened.
2021-11-10T21:58:48.687262Z 0 [Warning] A deprecated TLS version TLSv1 is enabled. Please use TLSv1.2 or higher.
2021-11-10T21:58:48.687291Z 0 [Warning] A deprecated TLS version TLSv1.1 is enabled. Please use TLSv1.2 or higher.
2021-11-10T21:58:48.687821Z 0 [Warning] CA certificate ca.pem is self signed.
2021-11-10T21:58:48.906615Z 1 [Warning] root@localhost is created with an empty password ! Please consider switching off the --initialize-insecure option.
2021-11-10T21:59:50.716548Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2021-11-10T21:59:50.718452Z 0 [Note] mysqld (mysqld 5.7.36) starting as process 350 ...
2021-11-10T21:59:50.721280Z 0 [Note] InnoDB: PUNCH HOLE support available
2021-11-10T21:59:50.721297Z 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2021-11-10T21:59:50.721302Z 0 [Note] InnoDB: Uses event mutexes
2021-11-10T21:59:50.721306Z 0 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
2021-11-10T21:59:50.721310Z 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2021-11-10T21:59:50.721315Z 0 [Note] InnoDB: Using Linux native AIO
2021-11-10T21:59:50.721671Z 0 [Note] InnoDB: Number of pools: 1
2021-11-10T21:59:50.721786Z 0 [Note] InnoDB: Using CPU crc32 instructions
2021-11-10T21:59:50.723373Z 0 [Note] InnoDB: Initializing buffer pool, total size = 128M, instances = 1, chunk size = 128M
2021-11-10T21:59:50.731241Z 0 [Note] InnoDB: Completed initialization of buffer pool
2021-11-10T21:59:50.733478Z 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
2021-11-10T21:59:50.745222Z 0 [Note] InnoDB: Highest supported file format is Barracuda.
2021-11-10T21:59:50.799665Z 0 [Note] InnoDB: Creating shared tablespace for temporary tables
2021-11-10T21:59:50.799801Z 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
2021-11-10T21:59:50.907759Z 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
2021-11-10T21:59:50.909871Z 0 [Note] InnoDB: 96 redo rollback segment(s) found. 96 redo rollback segment(s) are active.
2021-11-10T21:59:50.909910Z 0 [Note] InnoDB: 32 non-redo rollback segment(s) are active.
2021-11-10T21:59:50.910830Z 0 [Note] InnoDB: Waiting for purge to start
2021-11-10T21:59:50.961230Z 0 [Note] InnoDB: 5.7.36 started; log sequence number 2751061
2021-11-10T21:59:50.961668Z 0 [Note] InnoDB: Loading buffer pool(s) from /vol/mysql/ib_buffer_pool
2021-11-10T21:59:50.962390Z 0 [Note] Plugin 'FEDERATED' is disabled.
2021-11-10T21:59:50.967697Z 0 [Note] InnoDB: Buffer pool(s) load completed at 211110 21:59:50
2021-11-10T21:59:50.981030Z 0 [Note] Found ca.pem, server-cert.pem and server-key.pem in data directory. Trying to enable SSL support using them.
2021-11-10T21:59:50.981067Z 0 [Note] Skipping generation of SSL certificates as certificate files are present in data directory.
2021-11-10T21:59:50.981080Z 0 [Warning] A deprecated TLS version TLSv1 is enabled. Please use TLSv1.2 or higher.
2021-11-10T21:59:50.981089Z 0 [Warning] A deprecated TLS version TLSv1.1 is enabled. Please use TLSv1.2 or higher.
2021-11-10T21:59:50.982036Z 0 [Warning] CA certificate ca.pem is self signed.
2021-11-10T21:59:50.982090Z 0 [Note] Skipping generation of RSA key pair as key files are present in data directory.
2021-11-10T21:59:50.983207Z 0 [Note] Server hostname (bind-address): '*'; port: 3306
2021-11-10T21:59:50.983255Z 0 [Note] IPv6 is available.
2021-11-10T21:59:50.983267Z 0 [Note] - '::' resolves to '::';
2021-11-10T21:59:50.983295Z 0 [Note] Server socket created on IP: '::'.
2021-11-10T21:59:51.013171Z 0 [Note] Event Scheduler: Loaded 0 events
2021-11-10T21:59:51.013628Z 0 [Note] mysqld: ready for connections.
Version: '5.7.36' socket: '/var/run/mysqld/mysqld.sock' port: 3306 MySQL Community Server (GPL)
2021-11-10T22:00:27.342794Z 4 [Note] Aborted connection 4 to db: 'wandb_local' user: 'wandb_local' host: '127.0.0.1' (Got an error reading communication packets)
exec mysqld >> /var/log/mysql.log 2>&1
2021-11-10T22:00:35.460728Z 5 [Note] Aborted connection 5 to db: 'wandb_local' user: 'wandb_local' host: '127.0.0.1' (Unknown error)
...last two lines?
The new error is caused by a very slow filesystem. Can you try modifying the storage driver being used and starting with fresh volumes to see if you can find one that works on your machine?
https://docs.docker.com/storage/storagedriver/select-storage-driver/
On Wed, Nov 10, 2021 at 2:05 PM Carlo Alberto Barbano < @.***> wrote:
If I create a new storage folder, chmod it to 777 and start the container with: docker run --rm -it -v /data/wandb-storage2:/vol -p 8080:8080 --name wandb-local wandb/local:latest
I get the following:
docker run --rm -it -v /data/wandb-storage2:/vol -p 8080:8080 --name wandb-local wandb/local:latest *** Running /etc/my_init.d/00_regen_ssh_host_keys.sh... *** Running /etc/my_init.d/01_enable-services.sh... *** Copying services to runit *** Copying jobber template *** Enabling production mode *** Running /etc/my_init.d/02_load-settings.sh... *** Loading settings... 2021/11/10 21:58:44 Created default user 2021/11/10 21:58:44 Generating new session key for auth... 2021/11/10 21:58:44 Generating new certificate and key for auth... *** Booting runit daemon... *** Runit started as PID 48 *** Setting up mysql database... *** Starting wandb servers... *** Configuring minio... Bucket created successfully
local/local-files. Successfully added arn:minio:sqs:wandb-local:_:redis *** Migrating database... [mysql] 2021/11/10 22:00:27 packets.go:36: read tcp 127.0.0.1:43520->127.0.0.1:3306: i/o timeout [mysql] 2021/11/10 22:00:27 connection.go:310: invalid connection panic: migration failed in line 0: CREATE TABLE subscriptions ( created_at datetime NOT NULL, updated_at datetime DEFAULT NULL, id int(11) NOT NULL AUTO_INCREMENT, name varchar(64) DEFAULT NULL, limits json DEFAULT NULL,usagejson DEFAULT NULL, start_date datetime DEFAULT NULL, end_date datetime DEFAULT NULL, PRIMARY KEY (id) ) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;CREATE TABLE entities ( created_at datetime NOT NULL, updated_at datetime DEFAULT NULL, id int(11) NOT NULL AUTO_INCREMENT, name varchar(64) DEFAULT NULL, photo_url varchar(256) DEFAULT NULL, subscription_id int(11) DEFAULT NULL, PRIMARY KEY (id), UNIQUE KEY ix_entities_name (name), KEY subscription_id (subscription_id), CONSTRAINT entities_ibfk_1 FOREIGN KEY (subscription_id) REFERENCES subscriptions (id) ) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
CREATE TABLE users ( created_at datetime NOT NULL, updated_at datetime DEFAULT NULL, id int(11) NOT NULL AUTO_INCREMENT, email varchar(64) DEFAULT NULL, auth_id varchar(64) DEFAULT NULL, name varchar(64) DEFAULT NULL, username varchar(64) DEFAULT NULL, photo_url varchar(256) DEFAULT NULL, admin tinyint(1) DEFAULT NULL, default_entity_id int(11) DEFAULT NULL, account_type varchar(64) DEFAULT NULL, logged_in_at datetime DEFAULT NULL, analytics json DEFAULT NULL, PRIMARY KEY (id), UNIQUE KEY ix_users_auth_id (auth_id), UNIQUE KEY ix_users_email (email), UNIQUE KEY ix_users_username (username), KEY default_entity_id (default_entity_id), CONSTRAINT users_ibfk_1 FOREIGN KEY (default_entity_id) REFERENCES entities (id) ) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
CREATE TABLE projects ( created_at datetime NOT NULL, updated_at datetime DEFAULT NULL, id int(11) NOT NULL AUTO_INCREMENT, entity_id int(11) DEFAULT NULL, user_id int(11) DEFAULT NULL, name varchar(128) DEFAULT NULL, access varchar(64) DEFAULT NULL, repo varchar(256) DEFAULT NULL, views json DEFAULT NULL, description text, group_path varchar(256) DEFAULT NULL, sub_group_path varchar(256) DEFAULT NULL, featured int(11) DEFAULT NULL, analytics json DEFAULT NULL, PRIMARY KEY (id), UNIQUE KEY ix_projects_name_entity_id (name,entity_id), KEY entity_id (entity_id), KEY user_id (user_id), CONSTRAINT projects_ibfk_1 FOREIGN KEY (entity_id) REFERENCES entities (id), CONSTRAINT projects_ibfk_2 FOREIGN KEY (user_id) REFERENCES users (id) ) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
CREATE TABLE api_keys (
keyvarchar(128) NOT NULL, user_id int(11) DEFAULT NULL, description varchar(512) DEFAULT NULL, ips json DEFAULT NULL, rate_limit varchar(128) DEFAULT NULL, PRIMARY KEY (key), KEY ix_api_keys_user_id (user_id), CONSTRAINT api_keys_ibfk_1 FOREIGN KEY (user_id) REFERENCES users (id) ) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;CREATE TABLE files ( created_at datetime NOT NULL, updated_at datetime DEFAULT NULL, id int(11) NOT NULL AUTO_INCREMENT, name varchar(512) DEFAULT NULL, mimetype varchar(64) DEFAULT NULL, project_id int(11) DEFAULT NULL, run_name varchar(128) DEFAULT NULL, meta json DEFAULT NULL, storage_bucket varchar(128) DEFAULT NULL, special tinyint(1) DEFAULT NULL, archived tinyint(1) DEFAULT NULL, deleted_at datetime DEFAULT NULL, PRIMARY KEY (id), UNIQUE KEY ix_files_name_run_name_project_id (name,run_name,project_id), UNIQUE KEY ix_files_project_id_run_name_name (project_id,run_name,name), CONSTRAINT files_ibfk_1 FOREIGN KEY (project_id) REFERENCES projects (id) ) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
CREATE TABLE invites ( created_at datetime NOT NULL, updated_at datetime DEFAULT NULL,
keyvarchar(64) NOT NULL, from_user_id int(11) DEFAULT NULL, to_user_id int(11) DEFAULT NULL, entity_id int(11) NOT NULL, email varchar(64) DEFAULT NULL, accepted tinyint(1) NOT NULL, expires_at datetime DEFAULT NULL, PRIMARY KEY (key), KEY from_user_id (from_user_id), KEY to_user_id (to_user_id), KEY ix_invites_entity (entity_id), CONSTRAINT invites_ibfk_1 FOREIGN KEY (entity_id) REFERENCES entities (id), CONSTRAINT invites_ibfk_2 FOREIGN KEY (from_user_id) REFERENCES users (id), CONSTRAINT invites_ibfk_3 FOREIGN KEY (to_user_id) REFERENCES users (id) ) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;CREATE TABLE jobs ( created_at datetime NOT NULL, updated_at datetime DEFAULT NULL, id int(11) NOT NULL AUTO_INCREMENT, type varchar(64) DEFAULT NULL, description text, program varchar(256) DEFAULT NULL, repo varchar(256) DEFAULT NULL, PRIMARY KEY (id) ) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
CREATE TABLE run_fields ( project_id int(11) NOT NULL,
columnvarchar(64) NOT NULL, field varchar(256) NOT NULL, count int(11) DEFAULT NULL, PRIMARY KEY (project_id,column,field), KEY ix_run_fields_count (count) ) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;CREATE TABLE tags ( id int(11) NOT NULL AUTO_INCREMENT, name varchar(64) NOT NULL, PRIMARY KEY (id), UNIQUE KEY name (name) ) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
CREATE TABLE run_tags ( project_id int(11) NOT NULL, run_name varchar(64) NOT NULL, tag_id int(11) NOT NULL, PRIMARY KEY (project_id,run_name,tag_id), KEY run_tags_ibfk_2 (tag_id), CONSTRAINT run_tags_ibfk_1 FOREIGN KEY (project_id) REFERENCES projects (id), CONSTRAINT run_tags_ibfk_2 FOREIGN KEY (tag_id) REFERENCES tags (id) ) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
CREATE TABLE runs ( created_at datetime NOT NULL, updated_at datetime DEFAULT NULL, project_id int(11) NOT NULL, name varchar(128) NOT NULL, job_id int(11) DEFAULT NULL, user_id int(11) DEFAULT NULL, sweep_name varchar(64) DEFAULT NULL, config json DEFAULT NULL, summary_metrics json DEFAULT NULL, system_metrics json DEFAULT NULL, state varchar(64) DEFAULT NULL, commit varchar(64) DEFAULT NULL, host varchar(64) DEFAULT NULL, exitcode bigint(20) DEFAULT NULL, description text, heartbeat_at datetime DEFAULT NULL, group_name varchar(128) DEFAULT NULL, subgroup_name varchar(128) DEFAULT NULL, PRIMARY KEY (project_id,name), KEY ix_runs_user_id (user_id) ) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci PARTITION BY HASH (project_id) PARTITIONS 101;
CREATE TABLE teams ( user_id int(11) NOT NULL, entity_id int(11) NOT NULL, type varchar(64) DEFAULT NULL, name varchar(64) DEFAULT NULL, PRIMARY KEY (user_id,entity_id), KEY entity_id (entity_id), CONSTRAINT teams_ibfk_1 FOREIGN KEY (entity_id) REFERENCES entities (id), CONSTRAINT teams_ibfk_2 FOREIGN KEY (user_id) REFERENCES users (id) ) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
CREATE TABLE users_auth0 ( user_id int(11) NOT NULL, auth_id varchar(64) NOT NULL, PRIMARY KEY (user_id,auth_id), CONSTRAINT users_auth0_ibfk_1 FOREIGN KEY (user_id) REFERENCES users (id) ) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
CREATE TABLE views ( created_at datetime NOT NULL, updated_at datetime DEFAULT NULL, id int(11) NOT NULL AUTO_INCREMENT, name varchar(128) DEFAULT NULL, type varchar(128) DEFAULT NULL, user_id int(11) DEFAULT NULL, project_id int(11) DEFAULT NULL, description text, spec text, PRIMARY KEY (id), UNIQUE KEY uniq_view__project_user_name (project_id,user_id,name,type), KEY user_id (user_id), CONSTRAINT views_ibfk_1 FOREIGN KEY (project_id) REFERENCES projects (id), CONSTRAINT views_ibfk_2 FOREIGN KEY (user_id) REFERENCES users (id) ) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci; (details: invalid connection) and driver: bad connection in line 0: SELECT RELEASE_LOCK(?)
goroutine 1 [running]: main.main() /mnt/ramdisk/core/services/gorilla/cmd/migrate/main.go:116 +0x856 *** All services started *** Access W&B at http://localhost:8080 *** Logging disabled, find logs at /var/log/gorilla.log or enable by setting LOGGING_ENABLED=true *** Starting wandb servers... *** All services started *** Access W&B at http://localhost:8080 *** Logging disabled, find logs at /var/log/gorilla.log or enable by setting LOGGING_ENABLED=true !!! Unable to start backend, see logs at /var/logs. Trying again in 1 minute !!! Set the environment variable LOCAL_RESTORE=true if you're unable to login to the admin console *** Starting wandb servers... *** All services started *** Access W&B at http://localhost:8080 *** Logging disabled, find logs at /var/log/gorilla.log or enable by setting LOGGING_ENABLED=true *** Starting wandb servers... *** All services started *** Access W&B at http://localhost:8080 *** Logging disabled, find logs at /var/log/gorilla.log or enable by setting LOGGING_ENABLED=true !!! Unable to start backend, see logs at /var/logs. Trying again in 1 minute !!! Set the environment variable LOCAL_RESTORE=true if you're unable to login to the admin console
The /var/log/mysql.log file now contains:
021-11-10T21:58:45.294076Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details). 2021-11-10T21:58:46.945256Z 0 [Warning] InnoDB: New log files created, LSN=45790 2021-11-10T21:58:47.828044Z 0 [Warning] InnoDB: Creating foreign key constraint system tables. 2021-11-10T21:58:48.122890Z 0 [Warning] No existing UUID has been found, so we assume that this is the first time that this server has been started. Generating a new UUID: 61cd5ba1-4271-11ec-a972-0242ac110002. 2021-11-10T21:58:48.285890Z 0 [Warning] Gtid table is not ready to be used. Table 'mysql.gtid_executed' cannot be opened. 2021-11-10T21:58:48.687262Z 0 [Warning] A deprecated TLS version TLSv1 is enabled. Please use TLSv1.2 or higher. 2021-11-10T21:58:48.687291Z 0 [Warning] A deprecated TLS version TLSv1.1 is enabled. Please use TLSv1.2 or higher. 2021-11-10T21:58:48.687821Z 0 [Warning] CA certificate ca.pem is self signed. 2021-11-10T21:58:48.906615Z 1 [Warning] @.*** is created with an empty password ! Please consider switching off the --initialize-insecure option. 2021-11-10T21:59:50.716548Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details). 2021-11-10T21:59:50.718452Z 0 [Note] mysqld (mysqld 5.7.36) starting as process 350 ... 2021-11-10T21:59:50.721280Z 0 [Note] InnoDB: PUNCH HOLE support available 2021-11-10T21:59:50.721297Z 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2021-11-10T21:59:50.721302Z 0 [Note] InnoDB: Uses event mutexes 2021-11-10T21:59:50.721306Z 0 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier 2021-11-10T21:59:50.721310Z 0 [Note] InnoDB: Compressed tables use zlib 1.2.11 2021-11-10T21:59:50.721315Z 0 [Note] InnoDB: Using Linux native AIO 2021-11-10T21:59:50.721671Z 0 [Note] InnoDB: Number of pools: 1 2021-11-10T21:59:50.721786Z 0 [Note] InnoDB: Using CPU crc32 instructions 2021-11-10T21:59:50.723373Z 0 [Note] InnoDB: Initializing buffer pool, total size = 128M, instances = 1, chunk size = 128M 2021-11-10T21:59:50.731241Z 0 [Note] InnoDB: Completed initialization of buffer pool 2021-11-10T21:59:50.733478Z 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2021-11-10T21:59:50.745222Z 0 [Note] InnoDB: Highest supported file format is Barracuda. 2021-11-10T21:59:50.799665Z 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2021-11-10T21:59:50.799801Z 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2021-11-10T21:59:50.907759Z 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2021-11-10T21:59:50.909871Z 0 [Note] InnoDB: 96 redo rollback segment(s) found. 96 redo rollback segment(s) are active. 2021-11-10T21:59:50.909910Z 0 [Note] InnoDB: 32 non-redo rollback segment(s) are active. 2021-11-10T21:59:50.910830Z 0 [Note] InnoDB: Waiting for purge to start 2021-11-10T21:59:50.961230Z 0 [Note] InnoDB: 5.7.36 started; log sequence number 2751061 2021-11-10T21:59:50.961668Z 0 [Note] InnoDB: Loading buffer pool(s) from /vol/mysql/ib_buffer_pool 2021-11-10T21:59:50.962390Z 0 [Note] Plugin 'FEDERATED' is disabled. 2021-11-10T21:59:50.967697Z 0 [Note] InnoDB: Buffer pool(s) load completed at 211110 21:59:50 2021-11-10T21:59:50.981030Z 0 [Note] Found ca.pem, server-cert.pem and server-key.pem in data directory. Trying to enable SSL support using them. 2021-11-10T21:59:50.981067Z 0 [Note] Skipping generation of SSL certificates as certificate files are present in data directory. 2021-11-10T21:59:50.981080Z 0 [Warning] A deprecated TLS version TLSv1 is enabled. Please use TLSv1.2 or higher. 2021-11-10T21:59:50.981089Z 0 [Warning] A deprecated TLS version TLSv1.1 is enabled. Please use TLSv1.2 or higher. 2021-11-10T21:59:50.982036Z 0 [Warning] CA certificate ca.pem is self signed. 2021-11-10T21:59:50.982090Z 0 [Note] Skipping generation of RSA key pair as key files are present in data directory. 2021-11-10T21:59:50.983207Z 0 [Note] Server hostname (bind-address): '*'; port: 3306 2021-11-10T21:59:50.983255Z 0 [Note] IPv6 is available. 2021-11-10T21:59:50.983267Z 0 [Note] - '::' resolves to '::'; 2021-11-10T21:59:50.983295Z 0 [Note] Server socket created on IP: '::'. 2021-11-10T21:59:51.013171Z 0 [Note] Event Scheduler: Loaded 0 events 2021-11-10T21:59:51.013628Z 0 [Note] mysqld: ready for connections. Version: '5.7.36' socket: '/var/run/mysqld/mysqld.sock' port: 3306 MySQL Community Server (GPL) 2021-11-10T22:00:27.342794Z 4 [Note] Aborted connection 4 to db: 'wandb_local' user: 'wandb_local' host: '127.0.0.1' (Got an error reading communication packets) exec mysqld >> /var/log/mysql.log 2>&1 2021-11-10T22:00:35.460728Z 5 [Note] Aborted connection 5 to db: 'wandb_local' user: 'wandb_local' host: '127.0.0.1' (Unknown error)
...last two lines?
— You are receiving this because you commented.
Reply to this email directly, view it on GitHub https://github.com/wandb/local/issues/39#issuecomment-965790129, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAAAEN5W4JBB2CRETSLT3DULLUBVANCNFSM5FBSHB5Q .
Changing the storage driver does not solve the issue..
PS I am getting the same errors even if using a nvme ssd storage backend
@carloalbertobarbano can you try starting the server without a volume mounted in so that we can verify this is a docker storage driver related issue? Just know that the server will lose all data if you restart it, we just want to see if this is driver related or something else is happening.
Sure thing. The host specs are as follows:
- OS:
Ubuntu 24.04.03 LTS - Kernel version:
Linux alpina 5.10.0-1045-oem #47-Ubuntu SMP Wed Aug 18 10:41:03 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux - CPU: Intel(R) Xeon(R) W-2255 CPU @ 3.70GHz
- RAM: 64GB
- 1TB SSD NVME (/var/lib/docker is mounted on this device)
The command I've given: docker run --rm -it -p 8080:8080 --name wandb-latest wandb/local:latest.
For every config, I opened the wandb webpage with a new incognito browser sessions.
Config 1: overlay2
Migrating database takes a bit (1-2 minutes).
$ docker info
[...]
Server Version: 20.10.8
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
userxattr: false
Ouput:
*** Running /etc/my_init.d/01_enable-services.sh...
*** Copying services to runit
*** Copying jobber template
*** Enabling production mode
*** Running /etc/my_init.d/02_load-settings.sh...
*** Loading settings...
2021/11/12 09:47:18 Created default user
2021/11/12 09:47:18 Generating new session key for auth...
2021/11/12 09:47:18 Generating new certificate and key for auth...
*** Booting runit daemon...
*** Runit started as PID 54
*** Setting up mysql database...
*** Starting wandb servers...
*** Configuring minio...
Bucket created successfully `local/local-files`.
Successfully added arn:minio:sqs:wandb-local:_:redis
*** Migrating database...
Loading custom chart definitions...
*** All services started
*** Access W&B at http://localhost:8080
*** Logging disabled, find logs at /var/log/gorilla.log or enable by setting LOGGING_ENABLED=true
Creating an account results in "Unable to create account, contact your administrator. An application error occured". No further stdout from the docker. MySQL logs attached as config1_mysql.log
Config 2: overlay
Migrating database takes a bit here as well.
$ docker info
[..]
Server Version: 20.10.8
Storage Driver: overlay
Backing Filesystem: extfs
Supports d_type: true
Output (looks the same):
*** Running /etc/my_init.d/00_regen_ssh_host_keys.sh...
*** Running /etc/my_init.d/01_enable-services.sh...
*** Copying services to runit
*** Copying jobber template
*** Enabling production mode
*** Running /etc/my_init.d/02_load-settings.sh...
*** Loading settings...
2021/11/12 09:57:56 Created default user
2021/11/12 09:57:56 Generating new session key for auth...
2021/11/12 09:57:56 Generating new certificate and key for auth...
*** Booting runit daemon...
*** Runit started as PID 56
*** Setting up mysql database...
*** Starting wandb servers...
*** Configuring minio...
Bucket created successfully `local/local-files`.
Successfully added arn:minio:sqs:wandb-local:_:redis
*** Migrating database...
Loading custom chart definitions...
*** All services started
*** Access W&B at http://localhost:8080
*** Logging disabled, find logs at /var/log/gorilla.log or enable by setting LOGGING_ENABLED=true
Here as well, creating an account results in "Unable to create account, contact your administrator. An application error occured". No further stdout from the docker. MySQL logs attached as config2_mysql.log (looks the same)
Config 3: aufs
Migrating database takes a bit here as well.
$ docker info
[..]
Server Version: 20.10.8
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 12
Dirperm1 Supported: true
Output: same as above Account creation leads to the same error, mysqlog is the same as above
Config 4: fuse-overlayfs
Docker not starting.
I am starting to suspect this is a CPU related issue. If you need any more files from any of the container just let me know
same issue here, any one know which docker tag is the last working version?
@Gccxeon @carloalbertobarbano can you try with the latest version of local wandb/local:0.9.46 on a fresh volume? If you're still getting errors, please share the output of docker logs wandb-local and any screenshot of errors in the application itself.
Hi, thanks for your response. Unfortunately I am getting the same errors I attached above.. :(
@carloalbertobarbano there are a number of different errors in this thread, please share the output of docker logs wandb-local, the command you're running, and any screenshots that are relevant.
I am getting the same exact erros and logs I attached in my last comment where I tried all of the possible docker configurations: https://github.com/wandb/local/issues/39#issuecomment-966985030
Same error (as in screenshot), same mysql log, same docker output.
This is the command docker run --rm -it -p 8080:8080 --name wandb-latest wandb/local:0.9.46
@carloalbertobarbano apologies for the delayed response, I was on vacation. You should be mounting a volume into the container, otherwise you'll lose all data when it restarts. You can add -v wandb:/vol to your docker run command. If you're still getting that error message and the output of docker logs wandb-local contains:
*** Migrating database...
Loading custom chart definitions...
*** All services started
*** Access W&B at http://localhost:8080
Then you may need to manually remove and add a user via the cli. That can be done with:
docker exec -it wandb-local bash
cd /vol/env
vi users.htpasswd # remove any line that contains [email protected] and save the file
/usr/local/bin/local password [email protected]
Then you should be able to login. If you don't see the *** All services started in the logs, please share what you see in the logs here.
@carloalbertobarbano apologies for the delayed response, I was on vacation. You should be mounting a volume into the container, otherwise you'll lose all data when it restarts. You can add
-v wandb:/volto your docker run command. If you're still getting that error message and the output ofdocker logs wandb-localcontains:*** Migrating database... Loading custom chart definitions... *** All services started *** Access W&B at http://localhost:8080Then you may need to manually remove and add a user via the cli. That can be done with:
docker exec -it wandb-local bash cd /vol/env vi users.htpasswd # remove any line that contains [email protected] and save the file /usr/local/bin/local password [email protected]Then you should be able to login. If you don't see the
*** All services startedin the logs, please share what you see in the logs here.
Hi, I followed the steps but got an "Invalid password" error when i tried to login. Could u please clarify the format of users.htpasswd. Thanks!
@daihengming you should not manually add any lines to users.htpasswd that's what the /usr/local/bin/local password [email protected] command is for. You should replace the emails address with the address you use to login and then you'll be prompted for your password. That commands writes a BCrypted version of the password to the file.
Hello, I'm trying to run wandb local with docker but unfortunately it won't start. I'm using wandb-local v0.9.48 (v0.9.38 works fine on the same machine)
The machine I'm running the docker on has the following specs:
- OS: Ubuntu 20.04.3 LTS
- Kernel version: Linux eidos-service 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
- CPU: Intel Xeon Processor (Skylake, IBRS) - 2099.996 MHz
- RAM: 16 GB
I used the command docker run --rm -it -p 8081:8080 -v /data/wandb-storage-0.9.48:/vol --name wandb-local-0.9.48 wandb/local:0.9.48 and get the following logs:
*** Running /etc/my_init.d/00_regen_ssh_host_keys.sh...
*** Running /etc/my_init.d/01_enable-services.sh...
*** Copying services to runit
*** Copying jobber template
*** Enabling production mode
*** Running /etc/my_init.d/02_load-settings.sh...
*** Loading settings...
2022/02/23 07:38:15 Created default user
2022/02/23 07:38:15 Generating new session key for auth...
2022/02/23 07:38:15 Generating new certificate and key for auth...
*** Booting runit daemon...
*** Runit started as PID 49
*** Setting up mysql database...
*** Starting wandb servers...
*** Configuring minio...
Bucket created successfully `local/local-files`.
Successfully added arn:minio:sqs:wandb-local:_:redis
*** Migrating database...
[mysql] 2022/02/23 07:47:40 packets.go:36: read tcp 127.0.0.1:49070->127.0.0.1:3306: i/o timeout
[mysql] 2022/02/23 07:47:40 connection.go:310: invalid connection
panic: migration failed in line 0: ALTER TABLE runs
ADD COLUMN tags_json json DEFAULT NULL;
(details: invalid connection) and driver: bad connection in line 0: SELECT RELEASE_LOCK(?)
goroutine 1 [running]:
main.main()
/mnt/ramdisk/core/services/gorilla/cmd/migrate/main.go:125 +0x8b6
*** All services started
*** Access W&B at http://localhost:8080
*** Logging disabled, find logs at /var/log/gorilla.log or enable by setting LOGGING_ENABLED=true
*** Starting wandb servers...
*** All services started
*** Access W&B at http://localhost:8080
*** Logging disabled, find logs at /var/log/gorilla.log or enable by setting LOGGING_ENABLED=true
*** Starting wandb servers...
*** All services started
*** Access W&B at http://localhost:8080
*** Logging disabled, find logs at /var/log/gorilla.log or enable by setting LOGGING_ENABLED=true
!!! Unable to start backend, see logs at /var/logs. Trying again in 1 minute
!!! Set the environment variable LOCAL_RESTORE=true if you're unable to login to the admin console
*** Starting wandb servers...
*** All services started
*** Access W&B at http://localhost:8080
*** Logging disabled, find logs at /var/log/gorilla.log or enable by setting LOGGING_ENABLED=true
*** Starting wandb servers...
*** All services started
*** Access W&B at http://localhost:8080
*** Logging disabled, find logs at /var/log/gorilla.log or enable by setting LOGGING_ENABLED=true
!!! Unable to start backend, see logs at /var/logs. Trying again in 1 minute
!!! Set the environment variable LOCAL_RESTORE=true if you're unable to login to the admin console
looping the last 10 lines over and over.
And the log of /var/log/mysql.log
2022-02-23T07:38:16.754872Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2022-02-23T07:38:19.756031Z 0 [Warning] InnoDB: New log files created, LSN=45790
2022-02-23T07:38:21.049262Z 0 [Warning] InnoDB: Creating foreign key constraint system tables.
2022-02-23T07:38:22.005493Z 0 [Warning] No existing UUID has been found, so we assume that this is the first time that this server has been started. Generating a new UUID: 939cc0ca-947b-11ec-b2d6-0242ac110002.
2022-02-23T07:38:22.058236Z 0 [Warning] Gtid table is not ready to be used. Table 'mysql.gtid_executed' cannot be opened.
2022-02-23T07:38:22.717850Z 0 [Warning] A deprecated TLS version TLSv1 is enabled. Please use TLSv1.2 or higher.
2022-02-23T07:38:22.717885Z 0 [Warning] A deprecated TLS version TLSv1.1 is enabled. Please use TLSv1.2 or higher.
2022-02-23T07:38:22.718395Z 0 [Warning] CA certificate ca.pem is self signed.
2022-02-23T07:38:22.927808Z 1 [Warning] root@localhost is created with an empty password ! Please consider switching off the --initialize-insecure option.
2022-02-23T07:38:54.521077Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2022-02-23T07:38:54.523642Z 0 [Note] mysqld (mysqld 5.7.37) starting as process 300 ...
2022-02-23T07:38:54.526796Z 0 [Note] InnoDB: PUNCH HOLE support available
2022-02-23T07:38:54.526821Z 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2022-02-23T07:38:54.526826Z 0 [Note] InnoDB: Uses event mutexes
2022-02-23T07:38:54.526831Z 0 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
2022-02-23T07:38:54.526835Z 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2022-02-23T07:38:54.526839Z 0 [Note] InnoDB: Using Linux native AIO
2022-02-23T07:38:54.527306Z 0 [Note] InnoDB: Number of pools: 1
2022-02-23T07:38:54.527476Z 0 [Note] InnoDB: Using CPU crc32 instructions
2022-02-23T07:38:54.529585Z 0 [Note] InnoDB: Initializing buffer pool, total size = 128M, instances = 1, chunk size = 128M
2022-02-23T07:38:54.538805Z 0 [Note] InnoDB: Completed initialization of buffer pool
2022-02-23T07:38:54.542035Z 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
2022-02-23T07:38:54.554762Z 0 [Note] InnoDB: Highest supported file format is Barracuda.
2022-02-23T07:38:54.738119Z 0 [Note] InnoDB: Creating shared tablespace for temporary tables
2022-02-23T07:38:54.738255Z 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
2022-02-23T07:38:55.036475Z 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
2022-02-23T07:38:55.039097Z 0 [Note] InnoDB: 96 redo rollback segment(s) found. 96 redo rollback segment(s) are active.
2022-02-23T07:38:55.039154Z 0 [Note] InnoDB: 32 non-redo rollback segment(s) are active.
2022-02-23T07:38:55.040279Z 0 [Note] InnoDB: Waiting for purge to start
2022-02-23T07:38:55.090758Z 0 [Note] InnoDB: 5.7.37 started; log sequence number 2750533
2022-02-23T07:38:55.091545Z 0 [Note] InnoDB: Loading buffer pool(s) from /vol/mysql/ib_buffer_pool
2022-02-23T07:38:55.091912Z 0 [Note] Plugin 'FEDERATED' is disabled.
2022-02-23T07:38:55.094091Z 0 [Note] InnoDB: Buffer pool(s) load completed at 220223 7:38:55
2022-02-23T07:38:55.111984Z 0 [Note] Found ca.pem, server-cert.pem and server-key.pem in data directory. Trying to enable SSL support using them.
2022-02-23T07:38:55.112026Z 0 [Note] Skipping generation of SSL certificates as certificate files are present in data directory.
2022-02-23T07:38:55.112041Z 0 [Warning] A deprecated TLS version TLSv1 is enabled. Please use TLSv1.2 or higher.
2022-02-23T07:38:55.112051Z 0 [Warning] A deprecated TLS version TLSv1.1 is enabled. Please use TLSv1.2 or higher.
2022-02-23T07:38:55.113149Z 0 [Warning] CA certificate ca.pem is self signed.
2022-02-23T07:38:55.113216Z 0 [Note] Skipping generation of RSA key pair as key files are present in data directory.
2022-02-23T07:38:55.114515Z 0 [Note] Server hostname (bind-address): '*'; port: 3306
2022-02-23T07:38:55.114571Z 0 [Note] IPv6 is available.
2022-02-23T07:38:55.114594Z 0 [Note] - '::' resolves to '::';
2022-02-23T07:38:55.114629Z 0 [Note] Server socket created on IP: '::'.
2022-02-23T07:38:55.153375Z 0 [Note] Event Scheduler: Loaded 0 events
2022-02-23T07:38:55.153879Z 0 [Note] mysqld: ready for connections.
Version: '5.7.37' socket: '/var/run/mysqld/mysqld.sock' port: 3306 MySQL Community Server (GPL)
2022-02-23T07:44:47.800227Z 0 [Note] InnoDB: page_cleaner: 1000ms intended loop took 7801ms. The settings might not be optimal. (flushed=200 and evicted=0, during the time.)
2022-02-23T07:45:01.712284Z 0 [Note] InnoDB: page_cleaner: 1000ms intended loop took 6675ms. The settings might not be optimal. (flushed=200 and evicted=0, during the time.)
2022-02-23T07:47:40.548372Z 4 [Note] Aborted connection 4 to db: 'wandb_local' user: 'wandb_local' host: '127.0.0.1' (Got an error reading communication packets)
exec mysqld >> /var/log/mysql.log 2>&1
2022-02-23T07:48:13.269296Z 9 [Note] Aborted connection 9 to db: 'wandb_local' user: 'wandb_local' host: '127.0.0.1' (Got an error reading communication packets)
2022-02-23T07:48:14.392937Z 8 [Note] Aborted connection 8 to db: 'wandb_local' user: 'wandb_local' host: '127.0.0.1' (Got an error reading communication packets)
Any suggestions?
