Config file watcher

[Geal]

There's currently a way to configure applications from the static configuration file, but those informations would not be reloaded if the file changes. Most of the configuration options there are static anyway.

There could be a tool that watches for changes in another file, and transforms that into a configuration diff and sends messages to the proxy. The command library already provides a way to make a diff between two configurations and generate the corresponding add and remove messages.

[mbStavola]

I'd like to attempt this.

Quick question though. I see that there is a diff method for ConfigState but not Config and in CommandServer::new there is a TODO for populating a ConfigState from a Config. Does this mean I will also need to implement a Config -> ConfigState function, or instead create a new diff function for Config specifically?

Additionally, instead of a separate tool, wouldn't it be possible to just add a --watch flag to the CLI?

[Geal]

so, about this, the infos in Config that are static will probably not be dynamic any time soon. I'm talking about the routing configuration, ie what's here https://github.com/sozu-proxy/sozu/blob/master/bin/config.toml#L40-L49

I'm thinking of having a separate file where we can put that kind of information, and a tool that would watch changes in there and send modifications from the proxy. So you would only work on a ConfigState.

[Geal]

(adding the link here since the PR is closed) https://github.com/mbStavola/sozu/tree/feature/config-watcherCould you put the config watcher in its own repository? We'll put it under to sozu-proxy organization once it's done. We want to have a list of related config drivers like https://github.com/sozu-proxy/sozu-acme or https://github.com/sozu-proxy/tube-cheese as separate projects, since they'll be easier to contribute to

[mbStavola]

Sure, I can do that.

I think the last technical piece I'm missing is getting and setting the config for the proxy. I know I'm supposed to be using the command lib for this, are there examples I can look at?

[Geal]

you can take a look at this: https://github.com/sozu-proxy/tube-cheese/blob/master/src/api.rs

• create a state object: https://github.com/sozu-proxy/tube-cheese/blob/master/src/api.rs#L28
• whenever you get a config change, create a new state from it: https://github.com/sozu-proxy/tube-cheese/blob/master/src/api.rs#L52
• make a diff between your initial state and the new state, and send the messages to sozu: https://github.com/sozu-proxy/tube-cheese/blob/master/src/api.rs#L56-L59
• the new state replaces the old one: https://github.com/sozu-proxy/tube-cheese/blob/master/src/api.rs#L61

To send the messages to sozu, you can either talk in blocking socket, non blocking with mio, or non blocking with futures and tokio

[mbStavola]

So I found some time today and pulled out what I had in my sozu fork, getting it to send messages with a bit of copy-paste. I think it's in a place where I can actually start making good progress since it kinda works.

Here is the repo if you want to review. I'm going to do some cleanup tomorrow and work on actually making it robust (I also still need to read from sozu for the initial config).

[mbStavola]

Alright, I fixed some issues and cleaned it up a bunch so it's actually readable and reviewable. You can clone it and test whenever you're ready. Any questions, comments, or concerns?

[Geal]

Hi!

This looks like what we need! A few things:

• There are lots of uses of unwrap() and expect(). Since you use error_chain, it should be easy to return an error instead of panicking here and there.
• why do you use your own ConsoleMessage thing? It could be replaced by log and pretty_env_logger
• the current state is updated with the new messages even if sozu returned an error for one of the messages. Even though there's an error displayed in the log, at some point this tool could have an internal state that differs from sozu's.
• you can use the command lib to read to socket path from sozu's configuration file: https://github.com/sozu-proxy/sozu/blob/master/ctl/src/main.rs#L186-L192

[mbStavola]

1. Sure thing, will do!
2. Wasn't aware at the time, just had ansi_term and a dream of colored output. I can replace it with pretty_env_logger no problem, but it seems you need to use an env variable to turn it on? Shouldn't output be on by default for this binary?
3. True! Didn't quite catch that-- but I'm not too sure what the expected behavior would be in this case. Maybe retry until some threshold and then fatally error? Otherwise, we could just keep the last known good config and chug along. Both approaches seem kinda eh to me.
4. I can do that, but is it something you feel strongly about? It doesn't feel right to read the whole file just for one property. Maybe there is something else we could make use of in the future I guess?

[mbStavola]

So it seems like if you close sozu after sozuconfw has initialized from the state dump but before it's next request, it hangs forever on channel.read_message(). I couldn't find any method to set a timeout or an example handling this situation, is there anything you recommend in particular?

[Geal]

I was hoping you could keep using the blocking connection, but it looks like you'll have to switch to the futures-based interface. You should probably try to open a new connection each time you have orders to send to sozu, it makes no sense to keep a long lived connection. Also, maybe dump the state right before sending the order, to check that you still have the same base state?

about pretty_env_logger, IIRC you can pass an option manually to indicate the level when you start it. about reading the config file to get the socket path, it's fine. I'd rather have the config file as "truth source" instead of options all over the place. And reading from the config file will probably be a backup solution on real deployments, since there will be (there already is in sozu, not in other tools) a way to specify the config file path and the socket path at compile time (for distribution packagers).

[mbStavola]

Hey just a quick update:

1. I added pretty_env_logger and updated the README to explain how to turn on logging.
2. Switched over to futures; really fun to learn but a bit tough to work through!
3. I mostly addressed that bug how you describe, but in the reverse way (correct state on error instead of dumping the state before sending orders). I'll probably revisit it to do something more robust.
4. Started on more appropriate error handling.

I'm still working on finishing converting all the unwrap() and expect() calls to error_chain stuff, and after that I'll switch over to using the config file like you wanted. The only reason I haven't finished that yet is because of weird nested things in parser.rs and all the futures stuff in rpc.rs.

I'm mostly interested in feedback related to my futures usage. It works well, but I figure there might be some better approaches I'm missing.

[mbStavola]

Okay, looks like I've mostly covered the concerns. Obviously still room for improvement, but I think this is a good first step for a config file watcher. Here's a link to the repo.

Example usage from the root of sozuconfw:

$ sozu start --config assets/config.toml
$ RUST_LOG=info target/debug/sozuconfw --config assets/config.toml --applications assets/applications.toml --interval 2

[mbStavola]

While investigating the indefinite retry issue, I had found that it was very easy to put the tool out of sync with the proxy since the watcher doesn't take into account other means of modifying sozu. To solve this, I just made it so before every request we pull the latest state. It kinda sucks to do, but until we have a means of broadcasting events I think it's the best we can do.

[Geal]

@mbStavola hey, could you transfer the repository to the sozu-proxy organization? I'm preparing the release, and would like to have this one easily found along other tools

[mbStavola]

Sure thing! I just tried and it seems like I need to be a member of the org to actually do that? I get this: when I attempt to transfer ownership.

I also haven't looked at this in a while, so I'll probably need to test to make sure it still works :^).

[Geal]

ok, I just created https://github.com/sozu-proxy/sozu-confw I'll try to go over the code this week, set up a CI, etc. Thank you so much for your help :)

[Geal]

this could be done inside sozu's main process now, since we have a way to reload the configuration file

[Geal]

doing that in 0.14: spawn a future in the main process and call that method: https://github.com/sozu-proxy/sozu/blob/b38bd170c82c484ca04894761d11c14f2f57501d/bin/src/command/orders.rs#L712