before and after support?

[q66]

it might be handy to be able to specify a constraint that a service has to start before or after (like waits-for but without actually requesting the named service to start, instead only having an effect if the named service is enabled) some other service starts; e.g. like before = foo

this would allow for more flexible handling of ordering when writing service files, without having to touch other service files to get it

[davmac314]

I'd been working off the theory that before/after shouldn't be needed but If there's a good use-case I'll reconsider or come up with an alternative. ("after" wouldn't be too hard to implement as it's a dependency in the same direction as usual, "before" would add some complexity).

Have you go some specific case or example in mind here? It would be good to have examples of where the current design is insufficient.

[q66]

I have a concept of "targets" in my system, which are basically like systemd targets or sysvinit runlevels, specifying different points in the boot process to which enabled services can be attached. Right now, I specify a separate waits-for.d directory for each target (which is simply an internal service) and expect users to enable different services they want into different targets.

This allows me to have semi-flexible ordering which allows separation of stuff that is started early, stuff that is started before getty/login is enabled, and the rest of the stuff, plus a couple auxiliary targets; e.g. I can ensure that stuff like udev or syslog have started before invoking more advanced services, as well as e.g. being able to wait for network services before bringing up some other stuff. The problem is that this requires the user to actually worry about where each service goes. I feel like this should not be the user's responsibility, so my use case is to be able to ship service files that specify they want to invoke after an earlier named target but before a later named target, but while still letting people shove all their enabled services in boot.d.

[davmac314]

Ok, yeah I understand the issue. I'll think on this a little.

[q66]

relatedly to this, i've also been thinking whether we could have some way to have "providers" for services, as targets can still be pretty inflexible for this; e.g. a service may want to say "i want a logger to be available before i start" but there could be multiple providers for that; I guess I could probably use symlinks for it, but that's kinda messy to package in a distribution (one needs some kind of "alternatives" system to be able to switch these around and i've yet to see one that's not terrible), and can be fragile...

[mobin-2008]

Hello. I have an idea for this: Use after and before as follows: Suppose we have several files type = internal with the following names:

network.target
default.target (symlink to tty.target for example)
tty.target (we might call it a console. Understand the idea right now, naming is not a priority right now.)
rescue.target

I gave the end of their name .target for example.

For example, the content of default.target is as follows:

after = network.target
depends-on = udevd
depends-on = dbus
depends-on = elogind
...

I think this dependency management system makes it easier to write scripts for programs that will eventually run.

For example anbox-container-manager instead of this:

depends-on = mount
depends-on = sysusers
depends-on = cgroups
depends-on = fuse
depends-on = net-lo
depends-on = anbox-container-manager-pre

Use this:

after = network.target
depends-on = anbox-container-manager-pre

I think this will be more useful (and maybe faster!). It will also give dinit more flexibility.

[mobin-2008]

Now I have thought about the idea before and after and my opinion is that: I have no idea about after but I think implementing before is a great step forward for the project. but why? : Assume that we have two services a and b. I want b to run before a, so what do I do? I define depends-on = b in a. But this can sometimes be a problem. Read the following position: Consider the position above. There is a problem: b must be executed before a, but a does not need b! In such situations, before is very useful.

[q66]

before is more useful in general, because it allows you to ensure that services will guarantee having attempted to start at a specific point

after doesn't provide any such guarantee (the service specifying it is free to start at any point as long as another point has been reached) but it's still useful sometimes (and is the easier to implement part)

[davmac314]

@mobin-2008 I don't understand your example (and I'm not sure you understand the discussion). Your suggested example change for anbox-container-manager actually removes dependencies. I suspect you're using "after" when you could just be using "waits-for" or even "depends-on". If a service depends on an internal "target" which has its own dependencies, it already inherits those dependencies transitively.

[davmac314]

Let me lay out the problem as I understand it myself:

• dinit dependencies always carry an implicit "after" relationship as part of the dependency (there is no "before" relationship at all)
• the main way of enabling a service is to add it as a dependency of boot (via boot.d for example). Note that dinitctl enable can do this automatically.
• however, some optional services should naturally start before some other service/target. Eg. you want your print service to start before login is possible. Just enabling it in the normal way won't achieve this (and can't). Right now, you have to add a dependency from login-ready (or whatever it is called) to print-service.

@q66 does that probably sum up the problem?

If so I'm leaning towards a solution: have a setting in the service file which specifies which target service (or even services) dinitctl enable will actually apply to. So dinitctl enable print-server would then create the dependency from login-ready (or some other suitable target) rather than from boot. The only downside is that you can't see all optionally-enabled services just by looking at boot any more. This solution is hugely more convenient to implement however than adding a before relationship.

What do you think?

[q66]

the problems with this approach are basically

1. you need to use dinitctl enable, which only works when the service manager is running and its socket is present; therefore, people still have to worry about it in e.g. installation scripts, or when enabling services after they install their system
2. you can still mess up the service relationship if you manually enable services
3. this loses the flexibility of having a before relationship, because with a before, you can pretty much completely influence ordering of services (which i find much nicer than defining a bunch of arbitrary targets, because creating new targets is clunky, you need to make a new waits-for.d directory for them etc.), which is fine-grained and allows people to tweak how they want their services started to their liking

is implementing before really that much of an effort? it should be mostly just looking up the named service's node and internally adding a "dependency" link, i.e. identically to what after would be, just with an additional reverse lookup; if the named service does not exist in the dependency graph, one would simply do nothing for both, and if they do, it is basically adding a waits-for

[q66]

if this was implemented, i was thinking i could in general drop a lot of the target stuff, and instead set up the service relationships in a simpler and more flexible way; some of it would remain (having a login target is handy to have, as it represents a special point in boot, and not something arbitrary) but i could simplify this in general

[davmac314]

is implementing before really that much of an effort? it should be mostly just looking up the named service's node and internally adding a "dependency" link, i.e. identically to what after would be, just with an additional reverse lookup; if the named service does not exist in the dependency graph, one would simply do nothing for both, and if they do, it is basically adding a waits-for

The problems are mainly around:

• handling the case where the named "before" service isn't (yet) loaded, but gets loaded later (still potentially before either has started)
• detecting cycles (eg A is before B, B is before C, D waits-for C and therefore C is implicitly before D, D is before A).

Neither problem is insurmountable, but it's not completely trivial. I'm thinking about it though.

[mobin-2008]

@mobin-2008 I don't understand your example (and I'm not sure you understand the discussion). Your suggested example change for anbox-container-manager actually removes dependencies. I suspect you're using "after" when you could just be using "waits-for" or even "depends-on". If a service depends on an internal "target" which has its own dependencies, it already inherits those dependencies transitively.

Now that I think it would have been better, I would have raised this in another issue. I also thought in the example anbox-container-manager I should format it:

depends-on = network.target
...

In general, my goal was to show that after and before would be useful in this issue, but my idea is still very raw and I need to work on it more. Maybe I will open an issue about it later.

[q66]

The problems are mainly around:

* handling the case where the named "before" service isn't (yet) loaded, but gets loaded later (still potentially before either has started)

can't one load/parse the whole dependency graph before actually starting anything? then it would be basically

1. load every service file that is referenced during a service's startup with either soft or hard dependencies, treat regular deps normally
2. for either before or after, check if the named service is in the graph

• if it is, add a soft link identical to waits-for (for before services, add "this" service to the named service's links; for after, add the named service to this service's links)
• if it isn't, do nothing and ignore

3. once this is all complete, perform cycle checks (neither before nor after would introduce anything that would affect checking cycles) and startup

The desired semanatics is that both before and after only have any sort of effect when they point to something that is to be activated. I.e. when you have a before = foo and foo is not actually requested, the service will still start. Only if foo is also in the graph (through some kind of soft or hard dependency relationship) will the ordering be significant.

[davmac314]

can't one load/parse the whole dependency graph before actually starting anything?

The difficulty is that a service named in a before = might not exist in the whole graph. New services can be added at a later point; so, you have the problem that "foo" has before = bar but "bar" doesn't exist (possibly the service description isn't even there). At some later point, "bar" gets installed and loaded as a service; the dependency between "foo" and "bar" then needs to be put in place.

So, there would need to be a restriction saying that before can only name services for which the description can be found (and it will cause the named service to be loaded, if it's not already). I guess that might be sufficient for the use case you outlined above though? I.e if before is only used for certain targets that are guaranteed to be present, then it would work ok. Cycle checking would still need to be reworked a little because it currently assumes that a dependency on another service which is being loaded at the same time indicates a cycle. Also, the question of which service should fail to load in the case of a cycle is raised - currently it's not an issue since it's always the service that is being loaded right now.

A more complete solution would be, when "foo" is loaded and the before = bar constraint processed, create a dummy "bar" service (not shown by dinitctl list) until the point that the real "bar" is loaded. When (and if) that happens, dinit could treat it as a reload of an existing service rather than loading an entirely new service; the reload functionality already handles cycle checking (but would have to be altered so it didn't discard dependencies that were created via a before constraint from another service).

The desired semanatics is that both before and after only have any sort of effect when they point to something that is to be activated

Yep, that's how I understood it.

[q66]

can't one load/parse the whole dependency graph before actually starting anything?

The difficulty is that a service named in a before = might not exist in the whole graph. New services can be added at a later point; so, you have the problem that "foo" has before = bar but "bar" doesn't exist (possibly the service description isn't even there). At some later point, "bar" gets installed and loaded as a service; the dependency between "foo" and "bar" then needs to be put in place.

So, there would need to be a restriction saying that before can only name services for which the description can be found (and it will cause the named service to be loaded, if it's not already). I guess that might be sufficient for the use case you outlined above though? I.e if before is only used for certain targets that are guaranteed to be present, then it would work ok. Cycle checking would still need to be reworked a little because it currently assumes that a dependency on another service which is being loaded at the same time indicates a cycle. Also, the question of which service should fail to load in the case of a cycle is raised - currently it's not an issue since it's always the service that is being loaded right now.

yeah, for the targets that would probably be enough, since my use case is merely flattening what is already there (i.e. having only one directory with activated services and being able to control their ordering seamlessly; while having support for ordering like this would possibly enable further ideas and much more fine-grained approach to service layout, i have no interest in making the service relationships too complicated - as i see it, the fewer dependency relationships are necessary, the better - and besides, this can always be solved with packaging and properly set up dependencies in there, so that "optional" target service files get installed together with things that reference them) - when you say "to be loaded" i assume parsed and added into the system, but not activated, right?

A more complete solution would be, when "foo" is loaded and the before = bar constraint processed, create a dummy "bar" service (not shown by dinitctl list) until the point that the real "bar" is loaded. When (and if) that happens, dinit could treat it as a reload of an existing service rather than loading an entirely new service; the reload functionality already handles cycle checking (but would have to be altered so it didn't discard dependencies that were created via a before constraint from another service).

this would probably be better overall as far as flexibility goes, but even the constraint should probably work fine enough (at least for my current use case, which would unblock stabilization of the overall service layout, which i need to declare my distribution as ready for end user testing)

The desired semanatics is that both before and after only have any sort of effect when they point to something that is to be activated

Yep, that's how I understood it.

[davmac314]

when you say "to be loaded" i assume parsed and added into the system, but not activated, right?

Yep, exactly

[q66]

sounds good then, that would cover everything at the moment

[davmac314]

I hope to be able to do something about this soon. I have a few other issues (that I've been putting off) that I'm also working on.

[davmac314]

I have begun working on this (currently on the development branch). I'm aiming for before support only at this stage (after is theoretically easier, but before should be more useful, and implementing both before and after at the same time adds even more complexity so I'll avoid it if possible or save it for later).

[q66]

wouldn't one get after pretty much for free once before is implemented without introducing much complexity? i don't strictly need it for things to be functional right now, but I can still see it being useful (mainly for cleanliness of service definitions)

[davmac314]

wouldn't one get after pretty much for free once before is implemented without introducing much complexity?

Well, if the named service does exist, an "after" is equivalent to "waits-for", so yes it's pretty much for free in that case, but it's also unnecessary (you can just use "waits-for"). The interesting case is when the named service might not exist. For a "before" we could reasonably refuse to load a service if it names a service that doesn't exist, but I don't think we can do that for "after". That leaves two options that I can think of:

• silently drop the requirement
• remember the requirement and instate it when (if) the named service does actually get loaded

I guess we could go with the first option: silently drop any "after" requirement for a service that isn't (yet) loaded, but that certainly seems less than ideal as it gives different system behaviour depending on exactly when services are loaded. (For the most part it probably wouldn't matter I guess, but it doesn't seem like a clean solution).

For the second option, there's a similar problem: a cycle can exist if a certain service is loaded, but not exist otherwise. (Consider: "A" has "after = B", and "B" has "depends-on = A". You could load "A" individually, but you could never load "A" and "B" together without breaking the cycle somehow). Proper handling of this is probably to break the cycle (i.e. report a warning and then drop the "after" requirement). With the second option, the behaviour is different depending on load order of services only if there is actually a cycle - so, it's my preferred option at this stage.

I feel like this will be even more complicated if you have both "after" and "before" requirements, but to be honest I'm not completely sure. We'll find out later, I guess.

[q66]

If the named service does exist, after is not equivalent to waits-for, as waits-for will result in the named service being activated.

I'm not sure I understand why you can't refuse to load a service if it names a service that does not exist for after just like for before, mind clearing that up for me?

[davmac314]

If the named service does exist, after is not equivalent to waits-for, as waits-for will result in the named service being activated.

Hmm, right. Ok, I may have been getting myself confused and over-thinking this.

I'm not sure I understand why you can't refuse to load a service if it names a service that does not exist for after just like for before, mind clearing that up for me?

Yeah, given that it's not really a dependency that's not the case. One confusion lead to another.

On the plus side, the whole thing should be easier than I thought it would be.

[davmac314]

before has been implemented on the development branch (f3557a59a98479f1926aaa0c0f2c33b2211f63ef) and seems to be working. Still needs documentation and tests. after support is also still pending.

[q66]

already deployed it, works like a charm

[davmac314]

before and after support (including documentation) is now present on master branch.

[mobin-2008]

Its must be keep open? 0.16 is released!

[davmac314]

Included in 0.16.0.