Unable to push to doltlab when sql-server is running

[kde99]

Hello everyone!

Today I installed a local doltlab instance and wanted to upload my current work repository, so create a repo on doltlab and ran the commands it displayed to push. Namely:

dolt remote add origin http://10.0.1.117:50051/admin/PartnerDEV
dolt push origin main

Which fails on the second command with:

PS D:\db\partner> dolt push origin main
- Uploading...Error 1105 (HY000): fatal: remote 'origin' not found.
Please make sure the remote exists.

Then I tried adding the remove again:

PS D:\db\partner> dolt remote add origin http://10.0.1.117:50051/admin/PartnerDEV
error: a remote named 'origin' already exists.
remove it before running this command again

which didn't work. So then I tried to remove and add again:

PS D:\db\partner> dolt remote remove origin
PS D:\db\partner> dolt remote add origin http://10.0.1.117:50051/admin/PartnerDEV
PS D:\db\partner> dolt push origin main
- Uploading...Error 1105 (HY000): fatal: remote 'origin' not found.
Please make sure the remote exists.

Then I checked if it exists:

PS D:\db> dolt version
dolt version 1.34.2
Warning: you are on an old version of Dolt. The newest version is 1.34.3.
To disable this warning, run 'dolt config --global --add versioncheck.disabled true'

PS D:\db\partner> dolt remote
origin
PS D:\db\partner> dolt branch
* main

It indeed exists. After this I did other work and I had the idea of what if I stopped the running sql-server instance (the instance log shows the same error message printed). So I stopped my local dolt server and now I could push the repo.

My dolt comes from chocolatey, but I don't think that influences things. Doltlab is running on a linux host.

Thank you for your help!

[macneale4]

Based on the commands above, I believe the issue is that you have a dolt sql-server instances which is running. The dolt remote command is unfortunately not handling that situation well.

The work around is to not use the dolt remote command, and instead add the remote using sql operations:

$ dolt sql
# Welcome to the DoltSQL shell.
# Statements must be terminated with ';'.
# "exit" or "quit" (or Ctrl-D) to exit.
db/main> CALL DOLT_REMOTE('add','origin','http://10.0.1.117:50051/admin/PartnerDEV');

Documentation for the procedure can be found here: https://docs.dolthub.com/sql-reference/version-control/dolt-sql-procedures#dolt_remote

Getting the dolt remote command corrected to do the right thing will take a few days to complete.

[verdverm]

I'll add that while dolt sql-server is running

1. I can dolt table rm
2. I cannot dolt table import

[verdverm]

dolt docs upload README.md README.md 
cannot update manifest: database is read only

[verdverm]

Also happens with dolt sql -q '...' -s <name>

[macneale4]

Hmm. You are supposed to get an error message which tells you to stop sql-server in cases like this. I'll test.

[macneale4]

There are a couple bugs here, probably not the answer you wanted. First, the table and docs commands should simply tell you what you are attempting is impossible and requires you to stop the sql-server. The table command in particular should never run with sql-server. It's worth calling out that after you dolt table rm, the sql-server thinks the table still exists. This is super broken, and I'll prevent this shortly. Thanks for bringing it to our attention.

That said, I know you want this to actually work. The dolt table command is going to need a significant amount of work to co-operate with a running sql-server, and that will need to be scheduled. Fixing the error message and preventing the update is pretty straight forward so I'll do that today.

I get the following error with -s:

$ dolt sql -q "select * from blah" -s select_star
error: --save must be used in a dolt database directory.

Is that the error you are getting? Either way, that is a bug which I'll fix. Different error though.

[macneale4]

Actually, the dolt table rm command does go through the SQL interface, which is correct. It's in a round about way though and I'm going to disable it since the table subcommand in general should not be used with sql-server at this time.

[verdverm]

re: -s error, it's the same error as all the others, database is read-only. All of my small comments have this same error, I omitted it for most of them.

Definitely I want this feature. It's the biggest pain-point I have personally. Should I open a separate issue?

[timsehn]

Should I open a separate issue?

https://github.com/dolthub/dolt/issues/3922

@macneale4 Maybe we should open individual issues for the commands that don't work with a running server?

[macneale4]

Having a seperate ticket for each command wouldn't make this much better. If @verdverm is seeing the same error for all of them, then it's really one bug.

That said, I'm not seeing this behavior. Can you be very specific about where you are running commands in relation to your server? Full details with the complete output would be very helpful.

Today I spent a fair amount of time trying to simply get the right error message to print in this scenario, and it turns out this broke a ton of tests which are depending on this behavior. And, honestly, it would drop the existing table commands which are working for you.

When you say this is your biggest pain point, I want to be clear that migrating each command to SQL so that they work seamlessly with sql-server is non-trivial. docs and remote are straight forward, but table import is definitely not. It can all be done with sufficient development effort.

[verdverm]

My largest Dolt use-case is as data cleaning and iteration as part of doing data science. This has me wanting to both look at the data to see where it currently is at and then iterating on the cleanup and redoing the import. Ideally I can always leave the server running and just run my code that process & loads the data.

As of now, each iteration requires me to:

1. stop the server
2. run import
3. start the server
4. reconnect the sql browser or the python cell in a jupyter notebook

This is much more cumbersome that pushing <the up arrow> + <enter> after each code change

---

I will follow up with later with the steps to reproduce

iirc, I have

data/
  foo/
    .dolt/
    ...
  bar/
    .dolt/
    ...

and I run

• dolt sql-server from data/ to see all my data sets at once and run cross-database queries
• dolt table ... from subdirs like foo/ to run the code & data I'm iterating on

The error is database is read only