neon icon indicating copy to clipboard operation
neon copied to clipboard

Wrap more neon_base commands with python code

Open SomeoneToIgnore opened this issue 3 years ago • 1 comments

Based on https://github.com/neondatabase/neon/pull/1648#discussion_r868949772 and related comments

Python tests are now managing pageservers and safekeepers

  • via neon_base in some cases like init
  • via manual impls (etcd, pageserver, safekeeper management)

We should aim to reduce the code duplicaiton and work on control_plane/neon_base code more to enable the missing features and use them in python wrappers instead own reimpls

SomeoneToIgnore avatar May 16 '22 21:05 SomeoneToIgnore

I wonder if instead we should add python bindings for the neon/control_plane crate, and use those directly in fixtures. Some reasons why I find this appealing (assuming it's not difficult):

  1. Less plumbing, no need to go through a CLI
  2. Test fixtures would spawn child processes without conflicting with any locally running neon, and without daemonizing, which would make setup/cleanup more reliable
  3. The same python bindings can be reused in non-test code, if we need python scripts to manage production services (detach a tenant, migrate a tenant, wait for remote_consistent_lsn, see if page with given key exists, etc.) Would have been nice to have a toolset like this during the storage format migration. Our best option was to copy code from pytest fixtures :S
  4. We can experiment with writing tests in rust if most of the fixture code is shared
  5. We get experience with FFI. Would be very useful if we want to ship neon clients for users to interact with neon from different languages. Would also be useful to see if a rust rewrite of the pageserver client we have in postgres would be worth it. The c/rust barrier limits experimentation

Not sure

bojanserafimov avatar Aug 15 '22 18:08 bojanserafimov