Finally I got in(cannot login and met wired login/signup error)

[ericxsun]

Through https://github.com/wandb/local/issues/64#issuecomment-1106030481, I got in

---

Pre:

• Docker version 20.10.12, build e91ed57
• wandb, version 0.12.14
• wandb local image latest

With the following actions:

# start a new docker volume

docker stop wandb-local
docker volume rm wandb

docker run --rm -d -v wandb:/vol -p 8080:8080 --name wandb-local wandb/local

0. then, trigger the localhost:8080/logout

and then I can see thelogin and add license :

1. goto: Add license -> finally get http://localhost:8080/system-admin/

the Login links to http://localhost:8080/system-admin/login, a invalid link address (403 Forbidden)

2. goto: Login in 0) -> finally get http://localhost:8080/signup

click continue, I get the error:

3. set user and password through wandb bash cmd

docker exec -it wandb-local bash

wandb@66de73cf1e02:~$ cat /vol/env/users.htpasswd

only one line: [email protected]:.....

## create a user:
wandb@66de73cf1e02:~$ /usr/local/bin/local password [email protected]
Enter password for [email protected]:
xxxxx
User updated

and then goto http://localhost:8080/logout, and Login, it automatically jumps to the signup screen, and I get the same as described in 2) above.

4. try to restart after creation of the [email protected] in 3) and try again

docker stop wandb-local
docker run --rm -d -v wandb:/vol -p 8080:8080 --name wandb-local wandb/local

and then goto http://localhost:8080/logout, and Login, it automatically jumps to the signup screen, and I get the same as described in 2) above.

5. goto wandb shell and set something:

# rm [email protected]:.....   in /vol/env/users.htpasswd, then only one line left:
[email protected]:....

# restart mysql and outside the wandb docker, restart docker wandb local and re-login wandb shell, dive into mysql:
mysql -u wandb_local -p

select * from wandb_local.users;

the username of [email protected] is empty, then update it:
update wandb_local.users set username='test' where email = 'test@gmail';

# restart mysql and outside the wandb docker, restart docker wandb local
/etc/initd/mysql restart

docker run -e LOCAL_RESTORE=true --rm -d -v wandb:/vol -p 8080:8080 --name wandb-local wandb/local

re-goto http://localhost:8080/logout, and login, failed into signup page(it is different from the page in 2)

and re signup a new user test2, a wired error ocurred. signup username is test2 , but the error is for test, see the image: wandb_local

Attached users in mysql (the user local is deleted

---

It's really boring, for so many tries, I still cannot login successfully.

ref:

• https://github.com/wandb/local/issues/59
• https://github.com/wandb/local/issues/39

[ericxsun]

One more try:

step 1: create and start a new volume

docker stop wandb-local
docker volume rm wandb

docker run -e LOCAL_RESTORE=true --rm -d -v wandb:/vol -p 8080:8080 --name wandb-local wandb/local

step 2: login wandb docker shell and remove line [email protected]... in /vol/env/users.htpasswd

docker exec -it wandb-local bash

vi /vol/env/users.htpasswd  # remove line: [email protected]... 
cat /vol/env/users.htpasswd

NB: I guess the param -e LOCAL_RESTORE=true led to the line [email protected]:....

step 3: create new user [email protected]

/usr/local/bin/local password [email protected]

cat /vol/env/users.htpasswd 

and login mysql through mysql -u wandb_local -p to see some infos:

step 4: restart mysql and docker wandb,

# in wandb docker shell
/etc/init.d/mysql restart

# in host
docker stop wandb-local
docker run -e LOCAL_RESTORE=true --rm -d -v wandb:/vol -p 8080:8080 --name wandb-local wandb/local

and trigger the login or signup page http://localhost:8080/logout and click Login, input the created user in step 3, and it will bring me to the signup page. Now, input the Fullname and Username (same as created before test), click Continue, I can login. Wow, wow...

[vanpelt]

Hey @ericxsun we're working on a fix for this in the next release. You should never under any circumstances manually alter database records or create entities, that will potentially get you into an even more funky state but I'm glad you got in and apologies for this bug.

[better629]

@ericxsun follow your steps, but still can't login in.sign up with pre-create account, but still occurs with entity already exists for the user xxxx. But can login in with restore account.

[ericxsun]

@better629 with the latest try(https://github.com/wandb/local/issues/64#issuecomment-1106030481), I do not meet such same error(entity already exists for the user xxxx). Does the username (xxxx) in signup page is same as the pre-create account xxxx@email-address? If same, I think you can show the infos in wandb_local.entities, remove it and retry(or re-do from the begining). Otherwise, try with the same username. Hope you could login.

[better629]

@ericxsun I use the email with other username which I update the sql table. If it should be the same, I will try it. After login in, do u meet the backend api problem when save the license?

[ericxsun]

@ericxsun I use the email with other username which I update the sql table. If it should be the same, I will try it. After login in, do u meet the backend api problem when save the license?

@better629

1）Yes, I think the username and email-name should be same.

2）No, I didn't meet the error(the backend api problem). All is okay:

Or you can start the wandb-local with LICENSE (I mean, do not update LICENSE through Add LICENSE): docker run -e LICENSE=your-LICENSE --rm -d -v wandb:/vol -p 8080:8080 --name wandb-local wandb/local. It's okay.

[Jeffrey28]

There has been no change in this interface. Is there any method to tackle it?

[ericxsun]

There has been no change in this interface. Is there any method to tackle it?

It'll cost much time to init the environment

Waiting!!!

or try again(if it cost too much time) :

docker stop wandb-local
docker volume rm wandb

docker run -e LOCAL_RESTORE=true --rm -d -v wandb:/vol -p 8080:8080 --name wandb-local wandb/local

Wish you good luck

[Jeffrey28]

I have tried and waited for a long time almost one day. It still doesn't work.

[vanpelt]

@Jeffrey28 can you share the output of docker logs wandb-local

[Jeffrey28]

Here it comes!

*** Running /etc/my_init.d/00_regen_ssh_host_keys.sh...
*** Running /etc/my_init.d/01_enable-services.sh...
*** Copying services to runit
*** Copying jobber template
*** Enabling restore mode

Login at http://localhost:8080/api/login
  username: [email protected]
  password: Ay56WFwuEPXTBA==

*** Enabling production mode
*** Running /etc/my_init.d/02_load-settings.sh...
*** Loading settings...
*** Booting runit daemon...
*** Runit started as PID 46
*** Starting wandb servers...
*** Minio still starting up...
*** Migrating database...
panic: Dirty database version 2. Fix and force version.

goroutine 1 [running]:
main.main()
	/mnt/ramdisk/core/services/gorilla/cmd/migrate/main.go:125 +0x8d6
*** All services started
*** Access W&B at http://localhost:8080
!!! Unable to start backend, see logs at /var/logs.  Trying again in 1 minute
!!! Set the environment variable LOCAL_RESTORE=true if you're unable to login to the admin console
*** Starting wandb servers...
*** All services started
*** Access W&B at http://localhost:8080
*** Starting wandb servers...
*** All services started
*** Access W&B at http://localhost:8080
!!! Unable to start backend, see logs at /var/logs.  Trying again in 1 minute
!!! Set the environment variable LOCAL_RESTORE=true if you're unable to login to the admin console

[Jeffrey28]

And there is no a file named logs in the folder /var.

[Jeffrey28]

*** Running /etc/my_init.d/00_regen_ssh_host_keys.sh...
*** Running /etc/my_init.d/01_enable-services.sh...
*** Copying services to runit
*** Copying jobber template
*** Enabling restore mode

Login at http://localhost:8080/api/login
  username: [email protected]
  password: fJqL0uj1fewDqg==

*** Enabling production mode
*** Running /etc/my_init.d/02_load-settings.sh...
*** Loading settings...
2022/05/18 02:59:40 Created default user
2022/05/18 02:59:40 Generating new session key for auth...
2022/05/18 02:59:40 Generating new certificate and key for auth...
*** Booting runit daemon...
*** Runit started as PID 49
*** Setting up mysql database...
*** Starting wandb servers...
*** Minio still starting up...
*** Configuring minio...
Bucket created successfully `local/local-files`.
*** Minio still starting up...
Successfully added arn:minio:sqs:wandb-local:_:redis
*** Migrating database...
[mysql] 2022/05/18 03:04:00 packets.go:37: read tcp 127.0.0.1:60638->127.0.0.1:3306: i/o timeout
[mysql] 2022/05/18 03:04:00 connection.go:299: invalid connection
panic: 2 errors occurred:
	* migration failed in line 0: ALTER TABLE runs ADD COLUMN deleted_at datetime DEFAULT NULL;
 (details: invalid connection)
	* driver: bad connection in line 0: SELECT RELEASE_LOCK(?)



goroutine 1 [running]:
main.main()
	/mnt/ramdisk/core/services/gorilla/cmd/migrate/main.go:125 +0x8d6
*** All services started
*** Access W&B at http://localhost:8080
*** Starting wandb servers...
*** All services started
*** Access W&B at http://localhost:8080
!!! Unable to start backend, see logs at /var/logs.  Trying again in 1 minute
!!! Set the environment variable LOCAL_RESTORE=true if you're unable to login to the admin console

[vanpelt]

The database is timing out when migrating. There's an issue with your docker filesystem driver. Can you provide answers to:

1. Why are you running a local server?
2. What OS and docker version are you using?

[Jeffrey28]

1. Running a local server is useful when the network is poor. I often encounter data cannot be uploaded and network errors.
2. Ubuntu 20.04, Docker version 20.10.16

[vanpelt]

Hmm, let's try the following and see if things come up:

docker stop wandb-local docker rm volume wandb docker run -d --rm -v wandb:/vol/env:delegated -p 8080:8080 --name wandb-local wandb/local:latest docker logs -f wandb-local

Hopefully we don't see any errors in those logs and everything comes up after ~5 minutes or so. The main change here is adding the :delegated flag to the volume mount. That should improve disk performance.

[Jeffrey28]

Hmm, let's try the following and see if things come up:

docker stop wandb-local docker rm volume wandb docker run -d --rm -v wandb:/vol/env:delegated -p 8080:8080 --name wandb-local wandb/local:latest docker logs -f wandb-local

Hopefully we don't see any errors in those logs and everything comes up after ~5 minutes or so. The main change here is adding the :delegated flag to the volume mount. That should improve disk performance.

Still same errors. *** Running /etc/my_init.d/00_regen_ssh_host_keys.sh... *** Running /etc/my_init.d/01_enable-services.sh... *** Copying services to runit *** Copying jobber template *** Enabling production mode *** Running /etc/my_init.d/02_load-settings.sh... *** Loading settings... 2022/05/19 00:19:09 Created default user 2022/05/19 00:19:09 Generating new session key for auth... 2022/05/19 00:19:09 Generating new certificate and key for auth... *** Booting runit daemon... *** Runit started as PID 987 *** Setting up mysql database... *** Starting wandb servers... *** Configuring minio... Bucket created successfully local/local-files. Successfully added arn:minio:sqs:wandb-local:_:redis *** Migrating database... [mysql] 2022/05/19 00:23:30 packets.go:37: read tcp 127.0.0.1:60560->127.0.0.1:3306: i/o timeout [mysql] 2022/05/19 00:23:30 connection.go:299: invalid connection panic: 2 errors occurred: * migration failed in line 0: ALTER TABLE runs ADD COLUMN deleted_at datetime DEFAULT NULL; (details: invalid connection) * driver: bad connection in line 0: SELECT RELEASE_LOCK(?)

goroutine 1 [running]: main.main() /mnt/ramdisk/core/services/gorilla/cmd/migrate/main.go:125 +0x8d6 *** All services started *** Access W&B at http://localhost:8080 *** Starting wandb servers... *** All services started *** Access W&B at http://localhost:8080 !!! Unable to start backend, see logs at /var/logs. Trying again in 1 minute !!! Set the environment variable LOCAL_RESTORE=true if you're unable to login to the admin console *** Starting wandb servers... *** All services started *** Access W&B at http://localhost:8080 *** Starting wandb servers... *** All services started *** Access W&B at http://localhost:8080 !!! Unable to start backend, see logs at /var/logs. Trying again in 1 minute !!! Set the environment variable LOCAL_RESTORE=true if you're unable to login to the admin console *** Starting wandb servers... *** All services started *** Access W&B at http://localhost:8080 *** Starting wandb servers... *** All services started *** Access W&B at http://localhost:8080 !!! Unable to start backend, see logs at /var/logs. Trying again in 1 minute !!! Set the environment variable LOCAL_RESTORE=true if you're unable to login to the admin console

[vanpelt]

Can you share the mysql logs?

docker exec -it wandb-local bash cat /var/logs/mysql.log

[Jeffrey28]

wandb@e37bc2627b9b:~$ cat /var/logs/mysql.log
cat: /var/logs/mysql.log: No such file or directory
wandb@e37bc2627b9b:~$

[exalate-issue-sync[bot]]

WandB Internal User commented: better629 commented: @ericxsun follow your steps, but still can't login in.sign up with pre-create account, but still occurs with entity already exists for the user xxxx. But can login in with restore account.

[exalate-issue-sync[bot]]

WandB Internal User commented: ericxsun commented:

@better629 with the latest try(https://github.com/wandb/local/issues/64#issuecomment-1106030481), I do not meet such same error(entity already exists for the user xxxx). Does the username (xxxx) in signup page is same as the pre-create account xxxx@email-address? If same, I think you can show the infos in wandb_local.entities, remove it and retry(or re-do from the begining). Otherwise, try with the same username. Hope you could login.

[exalate-issue-sync[bot]]

WandB Internal User commented: better629 commented: @ericxsun I use the email with other username which I update the sql table. If it should be the same, I will try it. After login in, do u meet the backend api problem when save the license?

[exalate-issue-sync[bot]]

WandB Internal User commented: Jeffrey28 commented: There has been no change in this interface. Is there any method to tackle it?

[exalate-issue-sync[bot]]

WandB Internal User commented: Jeffrey28 commented: And there is no a file named logs in the folder /var.

[exalate-issue-sync[bot]]

WandB Internal User commented: Jeffrey28 commented:

*** Running /etc/my_init.d/00_regen_ssh_host_keys.sh...
*** Running /etc/my_init.d/01_enable-services.sh...
*** Copying services to runit
*** Copying jobber template
*** Enabling restore mode

Login at http://localhost:8080/api/login
  username: [email protected]
  password: fJqL0uj1fewDqg==

*** Enabling production mode
*** Running /etc/my_init.d/02_load-settings.sh...
*** Loading settings...
2022/05/18 02:59:40 Created default user
2022/05/18 02:59:40 Generating new session key for auth...
2022/05/18 02:59:40 Generating new certificate and key for auth...
*** Booting runit daemon...
*** Runit started as PID 49
*** Setting up mysql database...
*** Starting wandb servers...
*** Minio still starting up...
*** Configuring minio...
Bucket created successfully `local/local-files`.
*** Minio still starting up...
Successfully added arn:minio:sqs:wandb-local:_:redis
*** Migrating database...
[mysql] 2022/05/18 03:04:00 packets.go:37: read tcp 127.0.0.1:60638->127.0.0.1:3306: i/o timeout
[mysql] 2022/05/18 03:04:00 connection.go:299: invalid connection
panic: 2 errors occurred:
	* migration failed in line 0: ALTER TABLE runs ADD COLUMN deleted_at datetime DEFAULT NULL;
 (details: invalid connection)
	* driver: bad connection in line 0: SELECT RELEASE_LOCK(?)



goroutine 1 [running]:
main.main()
	/mnt/ramdisk/core/services/gorilla/cmd/migrate/main.go:125 +0x8d6
*** All services started
*** Access W&B at http://localhost:8080
*** Starting wandb servers...
*** All services started
*** Access W&B at http://localhost:8080
!!! Unable to start backend, see logs at /var/logs.  Trying again in 1 minute
!!! Set the environment variable LOCAL_RESTORE=true if you're unable to login to the admin console

[exalate-issue-sync[bot]]

WandB Internal User commented: vanpelt commented: The database is timing out when migrating. There's an issue with your docker filesystem driver. Can you provide answers to:

1. Why are you running a local server?
2. What OS and docker version are you using?

[exalate-issue-sync[bot]]

WandB Internal User commented: vanpelt commented: Hmm, let's try the following and see if things come up:

docker stop wandb-local docker rm volume wandb docker run -d --rm -v wandb:/vol/env:delegated -p 8080:8080 --name wandb-local wandb/local:latest docker logs -f wandb-local

Hopefully we don't see any errors in those logs and everything comes up after ~5 minutes or so. The main change here is adding the :delegated flag to the volume mount. That should improve disk performance.

[exalate-issue-sync[bot]]

WandB Internal User commented: Jeffrey28 commented:

Hmm, let's try the following and see if things come up:

docker stop wandb-local docker rm volume wandb docker run -d --rm -v wandb:/vol/env:delegated -p 8080:8080 --name wandb-local wandb/local:latest docker logs -f wandb-local

Hopefully we don't see any errors in those logs and everything comes up after ~5 minutes or so. The main change here is adding the :delegated flag to the volume mount. That should improve disk performance.

Still same errors. *** Running /etc/my_init.d/00_regen_ssh_host_keys.sh... *** Running /etc/my_init.d/01_enable-services.sh... *** Copying services to runit *** Copying jobber template *** Enabling production mode *** Running /etc/my_init.d/02_load-settings.sh... *** Loading settings... 2022/05/19 00:19:09 Created default user 2022/05/19 00:19:09 Generating new session key for auth... 2022/05/19 00:19:09 Generating new certificate and key for auth... *** Booting runit daemon... *** Runit started as PID 987 *** Setting up mysql database... *** Starting wandb servers... *** Configuring minio... Bucket created successfully local/local-files. Successfully added arn:minio:sqs:wandb-local:_:redis *** Migrating database... [mysql] 2022/05/19 00:23:30 packets.go:37: read tcp 127.0.0.1:60560->127.0.0.1:3306: i/o timeout [mysql] 2022/05/19 00:23:30 connection.go:299: invalid connection panic: 2 errors occurred: * migration failed in line 0: ALTER TABLE runs ADD COLUMN deleted_at datetime DEFAULT NULL; (details: invalid connection) * driver: bad connection in line 0: SELECT RELEASE_LOCK(?)

goroutine 1 [running]: main.main() /mnt/ramdisk/core/services/gorilla/cmd/migrate/main.go:125 +0x8d6 *** All services started *** Access W&B at http://localhost:8080 *** Starting wandb servers... *** All services started *** Access W&B at http://localhost:8080 !!! Unable to start backend, see logs at /var/logs. Trying again in 1 minute !!! Set the environment variable LOCAL_RESTORE=true if you're unable to login to the admin console *** Starting wandb servers... *** All services started *** Access W&B at http://localhost:8080 *** Starting wandb servers... *** All services started *** Access W&B at http://localhost:8080 !!! Unable to start backend, see logs at /var/logs. Trying again in 1 minute !!! Set the environment variable LOCAL_RESTORE=true if you're unable to login to the admin console *** Starting wandb servers... *** All services started *** Access W&B at http://localhost:8080 *** Starting wandb servers... *** All services started *** Access W&B at http://localhost:8080 !!! Unable to start backend, see logs at /var/logs. Trying again in 1 minute !!! Set the environment variable LOCAL_RESTORE=true if you're unable to login to the admin console

[exalate-issue-sync[bot]]

WandB Internal User commented: Jeffrey28 commented:

wandb@e37bc2627b9b:~$ cat /var/logs/mysql.log
cat: /var/logs/mysql.log: No such file or directory
wandb@e37bc2627b9b:~$

[nate-wandb]

Hi @ericxsun are there any log files within /var/logs/ and if so could you send the contents of those?