hse
hse copied to clipboard
hse-project
Reimplement REST support with libevent
Previously HSE used to use libmicrohttpd for its REST server. For various reason that may or may not have been accurate, it was determined that libmicrohttpd was not a good fit for what we were tring to accomplish with a REST server in HSE.
Enter libevent. libevent is primarily an event loop library. It just so happens that libevent also has support for an HTTP server. libevent is a little bit more lighterweight than libmicrohttpd when it comes to dependencies. libevent is also more useful than just an HTTP server. At some point in time, we may want to integrate an event loop within HSE. Who knows? But it is a possibility, and we wouldn't even need another dependency for one.
I evaluated an alternative like libuv, which I happed to think might be more maintained due to its use in technologies like node.js, but libuv doesn't have an HTTP server, and instead only implements TCP. While the HTTP 1.1 spec is fairly minimal, I don't have the time to implement it, nor do I want other people to have to read it to contribute in this area of the code.
I have created a libhse-rest.a for use within the rest of HSE. This library provides the interfaces to register HTTP routes, create HTTP request handlers, and lifecycle functions for an HTTP server. Many of the interfaces are inspired by Go's HTTP support. libevent's structures and APIs are hidden away from the users of the library should we ever come to the determination that libevent no longer suits our needs. It would be a fairly easy swap out of code contained with libhse-rest.a.
The hse_util/rest_client files have also been removed. These files made HSE take a dependency on libcurl when the interfaces in hse_util/rest_client weren't even used within the library. They were used only in the CLI and maybe some tools. In order to replace the functionality that was removed, I added a REST client into libhse-cli.a in addition to REST API wrappers for C. The REST client is modeled a little bit off of JavaScript's fetch() API.
The previous REST server was extremely under-documented. Routes, methods, and parameters were fairly unknown. I believe you could query a running REST server for some help documentation, but it just wasn't cutting it. With this commit, I have added a docs/openapi.yaml file that is a full description (or at least aims to be) of what the REST server does and how to interact with it. OpenAPI is a fairly popular standard for documenting HTTP APIs, so it was a fairly easy choice. In addition, also added a GitHub Actions workflow for validating it in the event it ever changes.
Many of the REST interfaces along with being re-implemented were cleaned up. For instance, many used YAML. They all use JSON now. We set the "Content-Type" header as appropriate. We return status codes which describe the operations and their potential errors. This was previously impossible in the previous iteration of the REST server. Some of the output format changed, but all REST output has been thoroughly documented with the OpenAPI description I have provided.
Signed-off-by: Tristan Partin [email protected]
This is pretty much everything except, replacing kvdb_rest_test, replacing rest_api_test, and some of the routes that get registered in platform.c. I didn't want to look at those code paths anymore...Greg, want to help? :)
I created a static library libhse-rest, that doesn't leak anything about libevent to the rest of HSE. I have wrapped libevent through the use of opaque pointers where needed. This means we could in theory sub out libevent and revert back to libmicrohttpd, or some other HTTP sevrer library, at a moment's notice. Pretty simple.
I like where this has gone quite a bit, and I think it is a big step up over the design of the previous REST interfaces. I took a lot of inspiration from Go's HTTP support.
I am not entirely sure how confident I am in the project management of libevent considering there hasn't been a release in 2 years and pull requests seem to stall out quite a bit. But for what we are currently doing, libevent should be fine.
I also evaluated libuv, but it doesn't implement HTTP support, only TCP support, and I don't really feel like reading and implementing the HTTP spec. I think libuv is probably the better event loop library however since more professional projects probably rely on it.
I evaluated libsoup from the GNOME ecosystem, but didn't want to bring GLib as a transitive dep. Would much rather use GLib for testing infrastructure though.
I evaluated libwebsockets, but it felt like it was going to drag in a lot of unnecessary stuff.
I did find some C++ solutions like https://github.com/facebook/proxygen, but they are C++, although with the libhse-rest.a interface, C++ would basically be an implementation detail if you expose a C API.
I have also been toying with the idea of making the REST support optional to build in HSE.
I kept the notion of non-exact routes around even though I don't like them. I think using a regex engine like PCRE2 would be a lot better, but I didn't feel like adding another dependency.
Oh and you'll notice, I pretty much finished the docs/openapi.yaml file minus the aforementioned unimplemented routes. What this gets you is a document that can be rendered like so:
I added linting to the CI for this document too. You can drill down to read more about these routes like:
Overall, pretty cool. Getting to pull a lot on my previous web development experience.
Edit: Just another thought while we are thinking about HSE's dependencies, we should remove cURL from a dependency of libhse, and make it a dependency of the CLI and any tools that need it. I think rest_client_test as it currently exists is not really worth maintaining.
So as it stands right now, there is a race condition when rest_start_server() is called, which spawns the thread to start the event loop, and rest_stop_server() is called right after. We may call out to exit the event loop prior to the event loop being started. This is noticeable in the tests I have been working on. Only solution is to throw a usleep() around from what I can tell.
I don't think libevent allows you to know when the event loops has started. I opened an issue on the libevent repo, but overall I don't have much faith in libevent if I can't do this workflow properly. Maybe other people have ideas.
Update. Happy with libevent for now. Currently writing unit tests. I have written the equivalent of the rest_api_test, REST binding wrappers for usage in C, and currently writing the equivalents of kvdb_rest_test, which I am splitting out into global_rest_test, kvdb_rest_test, and kvs_rest_test. Still to do are catch up with Greg's cn tree REST route changes and remove the data tree for well-known REST routes.
Finished kvdb_rest_test, finished global_rest_test, finished kvs_rest_test except for the cN tree route. That is next on my plate in addition to data tree stuff... :)
/ps renamed to /workqueues, but I think it has been implemented. Next is /kmc/vmstat. I think I will remove /kmc/test since it seems like it just testing allocators. I'll just let someone yell at me if this is wrong 😄! From there, onto the data tree (kill me) + cn tree stats!
What is your plan with the data tree? It has some nice properties that I'd like not to lose. In particular: automatic hierarchical name space, easy to add counters to tree, easy to list all counters in a subtree.
Great question. I don't think I am going to remove it, just expose it differently in the REST interface.
Re: data tree - I agree completely with Alex on the flexibility it provides. It wouldn’t be a surprise if the REST exposure could stand a fresh look. Tom Blamer might have thoughts in that area.
One thing that used to be odd and may still be … the root of the tree sort of contained itself from a naming standpoint so that at least internally you would say the equivalent of /root/root/subtreeA/… vs. /root/subtreeA/…
If that’s still around it would be nice to clean that up.
- db
On Jul 19, 2022, at 10:14, Alex Tomlinson @.***> wrote:
What is your plan with the data tree? It has some nice properties that I'd like not to lose. In particular: automatic hierarchical name space, easy to add counters to tree, easy to list all counters in a subtree.
— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because your review was requested.
/kmc/vmstat route is done. cn tree route and data tree are all that are left.
All routes are implemented. Tests are pretty much implemented. I was kind of lazy and didn't mock the information returned in the routes. If someone pokes me, I will do it :).
If you want to view the OpenAPI description, you can download the OpenAPI extension for VSCode and run the command "show preview using swagger UI", or copy and paste the file into https://editor.swagger.io/. The cool thing about this API description is that we could in theory generate REST bindings from the YAML description, thought I am not sure how good they would actually be.
https://github.com/OpenAPITools/openapi-generator
In the routes, kvdbs is plural but kvs is not.
/kvdbs/{alias}/params
/kvdbs/{alias}/kvs/{kvsName}/params
I prefer singular: /kvdb/{alias}/...
That was purposeful, since I find words ending in s awkward to make plural in REST interfaces. I changed it to /kvdbs from /kvdb because REST APIs are hierarchical, meaning you go from a group to a specific resource. In this case, ideally its a group of KVDBs related to a process, and after viewing the list, you can select the specific KVDB, and get the information pertaining to it.
Here is an example within the GitHub API.
Here is a conventions guide from Google which uses the same style.
https://cloud.google.com/apis/design/resource_names
I am definitely down to make mclass and kvs plural in the API routes if people have good ways to do it.
I will concede that this topic is talked about frequently in web dev circles.
cleaned up some clang-tidy warnings that weren't manifesting as errors :/. Why would clang-tidy not have an equivalent to -Werror, sigh.
What command lines do I run to see metrics output and jobs output?
./build/tools/hsettp -h # lists operations
./build/tools/hsettp kvs-cn-tree-get 0 myKvs --format tab
./build/tools/hsettp workqueues-get --format tab
If workqueues is what you mean by jobs. If you are referring to the scheduler, then you will see I have that code commented out.
If you are wondering why I have the buffer.c/.h code duplicated, that is because the util code is not composed into a static library, so I can't easily consume code within it.
I would like to organize the util code to do that in the future.
$ /opt/hse/bin/hsettp -s $(sudo jq -r '.socket.path' /mnt/kvdbs/kvdb1/kvdb.pid) Segmentation fault (core dumped)
$ /opt/hse/bin/hsettp Segmentation fault (core dumped) (139)
Is -s always required, or are there operations that don't need it?
$ /opt/hse/bin/hsettp -s $(sudo jq -r '.socket.path' /mnt/kvdbs/kvdb1/kvdb.pid) kmc-vmstat-get [{"name":"kvdb_pagecache","used_chunks":0,"huge_pages":0,"used_slabs":0,"used_slabs_size_kb":0,"empty_slabs":0,"allocated_slabs":0,"free_slabs":0,"item_size":4096,"item_alignment":4096,"item_aligned_size":4096,"total_items":0,"used_items":0,"allocations":0,"deallocations":0,"huge":true,"packed":false,"hardware_cache_aligned":false,"descriptor_convertible":false},{"name":"c0_cursor","used_chunks":0,"huge_pages":0,"used_slabs":0,"used_slabs_size_kb":0,"empty_slabs":0,"allocated_slabs":0,"free_slabs":0,"item_size":272,"item_alignment":64,"item_aligned_size":320,"total_items":0,"used_items":0,"allocations":0,"deallocations":0,"huge":false,"packed":true,"hardware_cache_aligned":true,"descriptor_convertible":false},{"name":"c0kvms","used_chunks":2,"huge_pages":0,"used_slabs":2,"used_slabs_size_kb":512,"empty_slabs":0,"allocated_slabs":2,"free_slabs":0,"item_size":1024,"item_alignment":128,"item_aligned_size":1024,"total_items":510,"used_items":3,"allocations":89,"deallocations":86,"huge":false,"packed":true,"hardware_cache_aligned":false,"descriptor_convertible":false},{"name":"ctxn_locks_impl","used_chunks":0,"huge_pages":0,"used_slabs":0,"used_slabs_size_kb":0,"empty_slabs":0,"allocated_slabs":0,"free_slabs":0,"item_size":1152,"item_alignment":16,"item_aligned_size":1152,"total_items":0,"used_items":0,"allocations":0,"deallocations":0,"huge":true,"packed":false,"hardware_cache_aligned":false,"descriptor_convertible":true},{"name":"ctxn_locks_slab","used_chunks":0,"huge_pages":0,"used_slabs":0,"used_slabs_size_kb":0,"empty_slabs":0,"allocated_slabs":0,"free_slabs":0,"item_size":16256,"item_alignment":16,"item_aligned_size":16256,"total_items":0,"used_items":0,"allocations":0,"deallocations":0,"huge":true,"packed":false,"hardware_cache_aligned":false,"descriptor_convertible":false},{"name":"ctxn_pfxlock","used_chunks":0,"huge_pages":0,"used_slabs":0,"used_slabs_size_kb":0,"empty_slabs":0,"allocated_slabs":0,"free_slabs":0,"item_size":256,"item_alignment":128,"item_aligned_size":256,"total_items":0,"used_items":0,"allocations":0,"deallocations":0,"huge":false,"packed":false,"hardware_cache_aligned":false,"descriptor_convertible":false},{"name":"ctxn_pfxlock_entry","used_chunks":0,"huge_pages":0,"used_slabs":0,"used_slabs_size_kb":0,"empty_slabs":0,"allocated_slabs":0,"free_slabs":0,"item_size":64,"item_alignment":16,"item_aligned_size":64,"total_items":0,"used_items":0,"allocations":0,"deallocations":0,"huge":false,"packed":false,"hardware_cache_aligned":false,"descriptor_convertible":false},{"name":"lc_cursor","used_chunks":0,"huge_pages":0,"used_slabs":0,"used_slabs_size_kb":0,"empty_slabs":0,"allocated_slabs":0,"free_slabs":0,"item_size":576,"item_alignment":16,"item_aligned_size":576,"total_items":0,"used_items":0,"allocations":0,"deallocations":0,"huge":false,"packed":true,"hardware_cache_aligned":false,"descriptor_convertible":false},{"name":"wbti","used_chunks":0,"huge_pages":0,"used_slabs":0,"used_slabs_size_kb":0,"empty_slabs":0,"allocated_slabs":17,"free_slabs":17,"item_size":48,"item_alignment":64,"item_aligned_size":64,"total_items":0,"used_items":0,"allocations":24,"deallocations":24,"huge":false,"packed":false,"hardware_cache_aligned":true,"descriptor_convertible":false},{"name":"ibnode","used_chunks":0,"huge_pages":0,"used_slabs":0,"used_slabs_size_kb":0,"empty_slabs":0,"allocated_slabs":0,"free_slabs":0,"item_size":16,"item_alignment":16,"item_aligned_size":16,"total_items":0,"used_items":0,"allocations":0,"deallocations":0,"huge":false,"packed":false,"hardware_cache_aligned":false,"descriptor_convertible":false},{"name":"ibldr","used_chunks":4,"huge_pages":4,"used_slabs":11,"used_slabs_size_kb":2816,"empty_slabs":7,"allocated_slabs":370,"free_slabs":359,"item_size":8180,"item_alignment":16,"item_aligned_size":8192,"total_items":351,"used_items":8,"allocations":4294,"deallocations":4286,"huge":true,"packed":false,"hardware_cache_aligned":false,"descriptor_convertible":false},{"name":"cntreenode","used_chunks":1,"huge_pages":0,"used_slabs":1,"used_slabs_size_kb":256,"empty_slabs":0,"allocated_slabs":1,"free_slabs":0,"item_size":832,"item_alignment":128,"item_aligned_size":896,"total_items":292,"used_items":13,"allocations":25,"deallocations":12,"huge":false,"packed":true,"hardware_cache_aligned":false,"descriptor_convertible":false},{"name":"kvsiter","used_chunks":4,"huge_pages":4,"used_slabs":8,"used_slabs_size_kb":2048,"empty_slabs":5,"allocated_slabs":260,"free_slabs":252,"item_size":1200,"item_alignment":128,"item_aligned_size":1280,"total_items":1632,"used_items":10,"allocations":2011,"deallocations":2001,"huge":true,"packed":false,"hardware_cache_aligned":false,"descriptor_convertible":false},{"name":"kvset32k","used_chunks":2,"huge_pages":2,"used_slabs":2,"used_slabs_size_kb":512,"empty_slabs":0,"allocated_slabs":5,"free_slabs":3,"item_size":32704,"item_alignment":64,"item_aligned_size":32704,"total_items":16,"used_items":7,"allocations":39,"deallocations":32,"huge":true,"packed":true,"hardware_cache_aligned":false,"descriptor_convertible":false},{"name":"kvset16k","used_chunks":1,"huge_pages":1,"used_slabs":1,"used_slabs_size_kb":256,"empty_slabs":0,"allocated_slabs":7,"free_slabs":6,"item_size":16320,"item_alignment":64,"item_aligned_size":16320,"total_items":15,"used_items":6,"allocations":183,"deallocations":177,"huge":true,"packed":true,"hardware_cache_aligned":false,"descriptor_convertible":false},{"name":"kvset8k","used_chunks":2,"huge_pages":2,"used_slabs":2,"used_slabs_size_kb":512,"empty_slabs":0,"allocated_slabs":6,"free_slabs":4,"item_size":8128,"item_alignment":64,"item_aligned_size":8128,"total_items":64,"used_items":4,"allocations":69,"deallocations":65,"huge":true,"packed":true,"hardware_cache_aligned":false,"descriptor_convertible":false},{"name":"kvset4k","used_chunks":3,"huge_pages":3,"used_slabs":3,"used_slabs_size_kb":768,"empty_slabs":0,"allocated_slabs":3,"free_slabs":0,"item_size":4032,"item_alignment":64,"item_aligned_size":4032,"total_items":195,"used_items":41,"allocations":1864,"deallocations":1823,"huge":true,"packed":true,"hardware_cache_aligned":false,"descriptor_convertible":false},{"name":"cn_cursor","used_chunks":0,"huge_pages":0,"used_slabs":0,"used_slabs_size_kb":0,"empty_slabs":0,"allocated_slabs":0,"free_slabs":0,"item_size":3680,"item_alignment":16,"item_aligned_size":3680,"total_items":0,"used_items":0,"allocations":0,"deallocations":0,"huge":true,"packed":true,"hardware_cache_aligned":false,"descriptor_convertible":false},{"name":"bkv_collection","used_chunks":2,"huge_pages":0,"used_slabs":2,"used_slabs_size_kb":512,"empty_slabs":0,"allocated_slabs":11,"free_slabs":9,"item_size":48,"item_alignment":16,"item_aligned_size":48,"total_items":10922,"used_items":6,"allocations":2136,"deallocations":2130,"huge":false,"packed":true,"hardware_cache_aligned":false,"descriptor_convertible":false},{"name":"mpool-rgnmap-1-0","used_chunks":2,"huge_pages":0,"used_slabs":2,"used_slabs_size_kb":512,"empty_slabs":0,"allocated_slabs":2,"free_slabs":0,"item_size":40,"item_alignment":16,"item_aligned_size":48,"total_items":10898,"used_items":349,"allocations":4174,"deallocations":3825,"huge":false,"packed":true,"hardware_cache_aligned":false,"descriptor_convertible":false},{"name":"mpool-rgnmap-1-1","used_chunks":2,"huge_pages":0,"used_slabs":2,"used_slabs_size_kb":512,"empty_slabs":0,"allocated_slabs":2,"free_slabs":0,"item_size":40,"item_alignment":16,"item_aligned_size":48,"total_items":10922,"used_items":343,"allocations":4257,"deallocations":3914,"huge":false,"packed":true,"hardware_cache_aligned":false,"descriptor_convertible":false},{"name":"mpool-rgnmap-1-2","used_chunks":2,"huge_pages":0,"used_slabs":2,"used_slabs_size_kb":512,"empty_slabs":0,"allocated_slabs":3,"free_slabs":1,"item_size":40,"item_alignment":16,"item_aligned_size":48,"total_items":10922,"used_items":345,"allocations":4250,"deallocations":3905,"huge":false,"packed":true,"hardware_cache_aligned":false,"descriptor_convertible":false},{"name":"mpool-rgnmap-1-3","used_chunks":2,"huge_pages":0,"used_slabs":2,"used_slabs_size_kb":512,"empty_slabs":0,"allocated_slabs":2,"free_slabs":0,"item_size":40,"item_alignment":16,"item_aligned_size":48,"total_items":10922,"used_items":333,"allocations":4245,"deallocations":3912,"huge":false,"packed":true,"hardware_cache_aligned":false,"descriptor_convertible":false},{"name":"wal-iowork","used_chunks":4,"huge_pages":0,"used_slabs":16,"used_slabs_size_kb":4096,"empty_slabs":16,"allocated_slabs":46,"free_slabs":30,"item_size":128,"item_alignment":64,"item_aligned_size":128,"total_items":32759,"used_items":0,"allocations":773032,"deallocations":773032,"huge":false,"packed":false,"hardware_cache_aligned":false,"descriptor_convertible":false}]
how do i see this in tabular output?
/opt/hse/bin/hsettp -s $(sudo jq -r '.socket.path' /mnt/kvdbs/kvdb1/kvdb.pid) kmc-vmstat-get --format tab
-s has a default value of /tmp/hse.sock at the moment. Every operation needs it.
$ /opt/hse/bin/hsettp workqueues-get Request failed: cli/lib/rest/client.c:202: Operation canceled (125) (65)
Need better help for this...
/opt/hse/bin/hsettp -s $(sudo jq -r '.socket.path' /mnt/kvdbs/kvdb1/kvdb.pid) kmc-vmstat-get --format tab
--format tab not in the hsettp -h help output...
Nevermind, I see it's an option to the operation...
[64 0 14:10:02] [tpartin@campise] [hse] [tristan957/rest=]
● $ ./build/tools/hsettp workqueues-get
[{"name":"hse_timer","references":3,"minimum_threads":3,"maximum_threads":3,"current_threads":3,"busy":2,"working":0,"delayed":0,"barrier_id":0,"calls":8486,"latency_ns":1082,"wait_message":"jclkslp","state":"S","processor":6,"time":"0:00","thread_id":567152,"thread_name":"(hse_jclock)"},{"name":"hse_timer","references":3,"minimum_threads":3,"maximum_threads":3,"current_threads":3,"busy":2,"working":0,"delayed":0,"barrier_id":0,"calls":13,"latency_ns":5752,"wait_message":"idle","state":"S","processor":8,"time":"0:00","thread_id":567153,"thread_name":"(hse_timer)"},{"name":"hse_timer","references":3,"minimum_threads":3,"maximum_threads":3,"current_threads":3,"busy":2,"working":0,"delayed":0,"barrier_id":0,"calls":13,"latency_ns":4835,"wait_message":"idle","state":"S","processor":5,"time":"0:00","thread_id":567151,"thread_name":"(hse_timer)"},{"name":"hse_kmc_reaper","references":20,"minimum_threads":1,"maximum_threads":1,"current_threads":1,"busy":1,"working":0,"delayed":19,"barrier_id":0,"calls":0,"latency_ns":0,"wait_message":"idle","state":"S","processor":38,"time":"0:00","thread_id":567154,"thread_name":"(hse_kmc_reaper)"}]
[0 0 14:10:54] [tpartin@campise] [hse] [tristan957/rest=]
● $ ./build/tools/hsettp workqueues-get --format tab
NAME REFS MIN_THRD MAX_THRD CURR_THRD BUSY WORKING DELAYED BARR_ID CALLS LAT_NS WAIT_MSG STATE PROC TIME THRD_ID THRD_NAME
hse_timer 3 3 3 3 2 0 0 0 12533 1000 jclkslp S 6 0:00 567152 (hse_jclock)
hse_timer 3 3 3 3 2 0 0 0 19 5362 idle S 8 0:00 567153 (hse_timer)
hse_timer 3 3 3 3 2 0 0 0 19 4390 idle S 5 0:00 567151 (hse_timer)
hse_kmc_reaper 20 1 1 1 1 0 19 0 0 0 idle S 38 0:00 567154 (hse_kmc_reaper)
[0 0 14:10:58] [tpartin@campise] [hse] [tristan957/rest=]
● $ ./build/tools/hsettp kmc-vmstat-get -h
Usage: hsettp [OPTION]... kmc-vmstat-get [OPTION]...
REST Route: /kmc/vmstat
Get virtual memory statistics.
Options:
-f, --format arg
Select the output format.
-h, --help
Print this help output.
-p, --pretty
Request a pretty HTTP response.
Formats:
json
tab
"tab" will print all columns.
"tab:COL1,COL2" will print only COL1 and COL2.
Columns:
NAME
UCHUNKS
HPAGES
USLABS
USLABS_SZ_KB
ESLABS
ASLABS
FSLABS
ITEM_SZ
ITEM_ALIGN
ITEM_ALIGN_SZ
TOTAL_ITEMS
USED_ITEMS
ALLOCS
DEALLOCS
HUGE
PACKED
HW_CACHE_ALIGNED
DESC_CONV