config.get_enabled_resources

[wu-victor]

config.set_enabled_resources tells Tilt to only run the specified resources.

A user in Slack asks for an analogous config.get_enabled_resources:

is there a corresponding config.get_enabled_resources that'd return an array of resources enabled at the end of a tiltfile evaluation?

It seems like if we go the route of args to define resources, we have to manually encode the resource list. What I'd love to do is get the resource list and be able to subtract services from it easily, and a resolved populated list of resources would be nice for this.

[wu-victor]

@landism would this make sense?

cc @maiamcc

[landism]

The use case makes sense, and having to repeat the resource names from the yaml just to pass them to set_enabled_resources is annoying.

I unfortunately don't see an obvious implementation / UI. Tilt executes the Tiltfile to learn about all the yaml, and then after Tiltfile execution finishes, assembles the yaml into resources. For get_resources to work in the Tiltfile, assembly would need to happen before get_resources is evaluated.

Some options that come to mind:

• make get_resources force assembly during Tiltfile execution (and then do a second assembly at the end of Tiltfile execution and throw an error if anything has changed the results of assembly)
• let the user specify a function that takes a list of resources and returns a list of resources that runs post-Tiltfile execution and post-assembly (though I think we're still a bit uncertain about allowing starlark callbacks to run post-Tiltfile execution)
• provide a general way for the user to write starlark that runs post-assembly, with a restricted list of allowed functions

[nicks]

I'm not sure I understand the problem that they're trying to solve with this.

Is the feature request that they want something like --disable-resource=redis, and be able to disable deployment of all the k8s objects in the 'redis' resource?

would this also be solved with a config.set_disabled_resource function? ya, per Matt's comment, a get_resources function would be hard to make work in Tilt's execution model (see: https://github.com/tilt-dev/company/tree/master/product-principles#infrastructure-as-data), because it creates a circular dependency between the desired state and the desired state.

[bobjackman]

Just to chime in here about a use case where this would be helpful for us:

We have a few different paths that could be calling config.set_enabled_resources() and it's proving difficult to track the list later down the line. Is there some way to retrieve the list of enabled resources (something like config.get_enabled_resources() or similar)?

As for @landism and @nicks concern about circular dependencies, I don't know about the internals of Tilt, but it seems to me like all get_enabled_resources() would have to do is return the last value that was passed to set_enabled_resources(), right?

[landism]

all get_enabled_resources() would have to do is return the last value that was passed to set_enabled_resources(), right?

For your use case, that'd work. My read of the OP was that set_enabled_resources had not been previously called, and the issue is what should get_enabled_resources do in that case? It can't really return all resources, for annoying reasons discussed above. If it errors, then it feels like it'd get people's hopes up just to be very disappointing in any but this special case.

While we're trying to sort this out, you could use something like this as a cheap stopgap (which you might already be doing, given the slack thread?):

last_enabled_resources = None
def set_enabled_resources(resources):
  config.set_enabled_resources(resources)
  last_enabled_resources = resources

def get_enabled_resources():
  if last_enabled_resources == None:
    fail()
  return last_enabled_resources

[amkartashov]

my use case for get_enabled_resources() is to avoid unnecessary calls to bazel to query deps for not enabled resources. Even if I call tilt down it has to query all the targets to find out deps to be passed to custom_build

I can't track enabled resources myself, because of resource dependencies. If I pass resourceA to config.set_enabled_resources and it happen to depend on resourceB, then resourceB is also enabled. So approach from https://github.com/tilt-dev/tilt/issues/3432#issuecomment-794055930 won't work for me.

I understand that get_enabled_resources will be hard to implement because it is required to evaluate Tiltfile first to have deps tree

[bobjackman]

I actually had a thought about this that's been tumbling around in my head. It's only half-baked, but this seems like the perfect thread for it, so figure I'll just put it out there and see what the tilt guys @nicks @landism think about it.

The basic idea is that resources are registered with tilt at eval time, and only executed at run time IF they end up being needed. Basically, Tilt becomes aware of every resource always, but only builds them when they're actually needed. Something like this:

register_resource('foo', foo_maker)
register_resource('bar', bar_maker)

def foo_maker():
  git_resource(blah)
  k8s_resource(blah)
  # other slow stuff

def bar_maker():
  helm_remote(blah);
  # other slow stuff
  k8s_resource('bar', resource_deps=['foo']) # because resources are pre-registered, tilt can still error here if an invalid resource is named

Then:

tilt up -- bar

^^ will call bar_maker(), which depends on foo, so will call foo_maker()

But:

tilt up -- foo

^^ will call foo_maker(), but since bar was never requested, bar_maker() (and its slow tasks) will never be called.

[nicks]

Ya, I really like the idea of making it more ergonomic to skip the helm_remote() call (which could be expensive) if you're not running any resources that the helm_remote call is downloading resources for.

We're working on a big API / engine refactor (we kind of hint at what this system looks like here: https://blog.tilt.dev/2021/04/30/how-many-servers.html#a-small-taste). But there a couple of different ways this could fit into the new API system:

• The kludgey way would be to have a local() call that runs

tilt get session Tiltfile -o jsonpath='{.status.targets[*].name}'

which will give you all the names of the resources that Tilt knows about, and do something hacky to reload the Tiltfile after the first run. I don't think this is a good way to do it, but it should kinda work.

• Making Helm a first-class API type that you register with Tilt, and it will download things asynchronously, and you can enable/disable it if the associated resource is enabled/disabled

• Making Tiltfile execution a first-class API type that you register with Tilt, so you could say, have separate Tiltfiles for 'foo' and 'bar', and have one Tiltfile asynchronously trigger another

The last one is cosmetically similar to what @KOGI suggested - instead of registering Tiltfile functions and executing them asynchronously, we'd register whole Tiltfiles. The hard part, for me, is designing a system that users will be able to debug and reason about. So I'd rather have API objects that you can query with tilt get rather than have complex flow control in the Tiltfile that's much harder to query.

[bobjackman]

You could use some flavor of option 2 that also supports the ability to query with tilt get. As long as all registrations are done first (disallow new registrations inside the builder functions. In other words, registrations are only allowed in the first stage of the tiltfile execution. Once building/instantiating starts, new registrations can no longer be made.) This will allow the entire tree of tiltfiles to be included up front, but only select builders will actually be executed.

[bobjackman]

This issue continues to bite us in the butt. Not only does it take FOREVER to parse the tiltfile every time we make a change, but today we're running into an issue with a bad release of a 3rd party helmchart and because Tilt clones/checksout EVERYTHING at parse time regardless of which resource(s) we asked for, we can't even start up an unrelated portion of our stack because the bad helm chart is causing errors and killing the process. If the definition and execution could be separated as above, then it would still be possible to launch the rest of our stack in a situation like this.

[cellux]

Here is a trick which worked for us:

all_resources = []
if os.path.exists("Tiltfile.all-resources"):
   all_resources = set(decode_json(read_file("Tiltfile.all-resources")))

disabled_resources = []
if os.path.exists("Tiltfile.disabled-resources"):
   disabled_resources = set(decode_json(read_file("Tiltfile.disabled-resources")))

if all_resources and disabled_resources:
   enabled_resources = [r for r in all_resources if r not in disabled_resources]
   config.set_enabled_resources(enabled_resources)

Once Tilt is up and running, you can (re)generate Tiltfile.all-resources with the following command:

tilt dump engine | jq .ManifestDefinitionOrder > Tiltfile.all-resources