upgrading from finn

[prologic]

Hi Josh :D

I'd like to upgrade bitraft from finn. I need some help though... Mostly it would be really nice if you could document a few more internals. The examples look good, README is great, but some of the API docs are a bit sparse.

For example; I don't know what Initialdata should be, the type is interface{} of course, but I get very "urky" when seeing such open types :)

Also many of the options on the Config struct are a bit undocumented and I'm not sure what some of them mean.

Thanks!

[tidwall]

Hi James :)

Sorry about the docs. It's a work in progress. I'm still in the process of polishing the api.

The machine data is just an interface{}. Uhaha doesn't care what it is, just that it's the same thing though the life of the process. I know that the interface{} type isn't pretty, but it's the most explicitly Go way to say "this can be anything".

The InitialData should be assigned to your main data in its initial empty state. For example, this could be as simple as a make(map[string]string) for a key/value store, or it could an empty boltdb database, or a sqlite database with it's initial schema, etc. Along with the Config.InitialData, there's also the optional Config.Restore function, which "restores" a database from a snapshot and also returns an interface{} that, like InitialData, is you machine data. These are the only two times where you need to be concerned with creating and assigning your data.

Each example in the examples directory uses a slightly different way of dealing with InitialData, Snapshot, and Restore. It also may be worthwhile to take a peek at the Uhasql code, which uses sqlite.

The Config type has a bunch of options, and I agree that it needs more comments. 😔

[prologic]

Hi James :)

Hi :D

Sorry about the docs. It's a work in progress. I'm still in the process of polishing the api.

No worries!

The machine data is just an interface{}. Uhaha doesn't care what it is, just that it's the same thing though the life of the process. I know that the interface{} type isn't pretty, but it's the most explicitly Go way to say "this can be anything".

What if we didn't make it an interface{} but something bit more concrete? I coudl live with it being a bit more explicit, like "Data must implement blah".

The InitialData should be assigned to your main data in its initial empty state. For example, this could be as simple as a make(map[string]string) for a key/value store, or it could an empty boltdb database, or a sqlite database with it's initial schema, etc. Along with the Config.InitialData, there's also the optional Config.Restore function, which "restores" a database from a snapshot and also returns an interface{} that, like InitialData, is you machine data. These are the only two times where you need to be concerned with creating and assigning your data.

Yeah okay fair enough... Does this work though with say for example the following:

db, _ := bitcask.Open(dbPath)
conf.InitialData = db

There is no guarentee that db here is a new Bitcask DB in it's initial empty state?

Does this matter?

Each example in the examples directory uses a slightly different way of dealing with InitialData, Snapshot, and Restore. It also may be worthwhile to take a peek at the Uhasql code, which uses sqlite.

Ahh good I'll check that out, might give me a few more ideas.

The Config type has a bunch of options, and I agree that it needs more comments. 😔

Yeah some of the options look interesting, but I'm not sure what they do without reading the full code behind them :D

Thanks for your time!

[tidwall]

Actually, The Uhasql source doesn't use the InitalData but rather a global variable and a mutex.

It also uses the Config.DataDirReady function, which fires when the directory that store all machine related data is created.

Then its initial data is just a blank database created from scratch. It removes older data if needed.

conf.DataDirReady = func(dir string) {
	os.RemoveAll(filepath.Join(dir, "db"))
	os.Mkdir(filepath.Join(dir, "db"), 0777)
	dbPath = filepath.Join(dir, "db", "sqlite.db")
	wdb = must(openSQLDatabase(dbPath, false)).(*sqlDatabase)
}

[tidwall]

For your initial data you shouldn't reopen an existing database. You need to open a new database.

Something like

os.RemoveAll(dbPath)
db, _ := bitcask.Open(dbPath)
conf.InitialData = db

[prologic]

Gotcha! That kind of makes sense!

If we wrap up what you just described as proper interface it will makes things more clear I think :D Not sure what to call the methods on the interface though?

[prologic]

Having trouble joining a node here:

$ ./bitraft -p ./data2 -a :5921 -b :4921 -j localhost:5920
INFO[0000] 66757:S 19 Dec 2020 02:11:03.904 # starting bitraft version 0.0.1@HEAD
INFO[0000] 66757:S 19 Dec 2020 02:11:04.161 * synchronized time
INFO[0000] 66757:S 19 Dec 2020 02:11:04.161 * server listening at [::]:4921
INFO[0000] 66757:S 19 Dec 2020 02:11:04.161 * server advertising as :5921
INFO[0000] 66757:S 19 Dec 2020 02:11:04.217 * initial configuration: index=0 servers=[]
INFO[0000] 66757:S 19 Dec 2020 02:11:04.217 * joining existing cluster at localhost:5920
INFO[0000] 66757:F 19 Dec 2020 02:11:04.217 * entering follower state: follower="Node at [::]:4921 [Follower]" leader=
INFO[0000] 66757:F 19 Dec 2020 02:11:04.218 # raft server add: dial tcp [::1]:5920: connect: connection refused

Not really sure what I'm doing wrong :/

Code here

[prologic]

Note a couple of TODO and XXX notes too. I need your help with some of those, there might be things to improve with the framework here to address these?

[tidwall]

The server that is being launched at :5921 is trying to join to an existing cluster at localhost:5920. Is there a server running at localhost:5920?

[prologic]

Yup there is! The problem is the advertise socket isn't listening? Or is it the same socket?