Implement lua

[cgwalters]

Today rpm-ostree doesn't implement -p <lua>. We currently today carry overrides for a few packages in Fedora. The reason we don't implement Lua is it directly conflicts with our per-script sandboxing (using bwrap). We theoretically could run Lua in a new process, but doing so basically obviates all of the point of using Lua in the first place. It might as well be shell script.

A major challenge in doing even the new process model is that the posix API that a lot of scripts use is actually implemented inside the guts of librpm. We'd have to implement something like rpm --lua-eval where we pass the script on stdin or so.

For more background, see this thread on rpm-ecosystem around lua and %pretrans.

And this thread: http://lists.rpm.org/pipermail/rpm-ecosystem/2017-May/000477.html

[cgwalters]

Changing this to just Lua, since https://github.com/projectatomic/rpm-ostree/pull/763

[Conan-Kudo]

We theoretically could run Lua in a new process, but doing so basically obviates all of the point of using Lua in the first place. It might as well be shell script.

It does not obviate all the point of using Lua. The only reason this is a problem for you is because you refuse to implement the rpmlua interpreter and you use nearly zero of librpm's facilities to actually handle scriptlets.

While most of what rpm-ostree is doing is good, I vehemently disagree that you should be forcing people to rely on shell when they want to use Lua for scriptlets.

[cgwalters]

The only reason this is a problem for you is because you refuse to implement the rpmlua interpreter

I wouldn't say "refuse"...I'm not opposed to doing it. One thing that holds this type of thing back a bit is rpm-ostree needs to work on RHEL7 today, so it's hard to require new changes to librpm.

and you use nearly zero of librpm's facilities to actually handle scriptlets.

True. The way I see this is rpm-ostree makes some fundamental changes to the rpm stack; there are a lot of improvements, but also risky/invasive.

Perhaps down the line we could try to add the "run scripts via bwrap" logic to librpm at some point, but note it's fairly intertwined with the higher level rpm-ostree model. For example, how we make /var read-only. That's a key aspect of our "offline/transactional" updates; the rpm-ostree stack never touches your user data.

[cgwalters]

One thing we could do fairly easily is support embedding a shell implementation in Lua comments, for projects like glibc that may want to do both, something like this:

%transfiletriggerpostun -p <lua> common -P 2000000 -- /lib /usr/lib /lib64 /usr/lib64
-- nolua begin
-- /sbin/ldconfig
-- nolua end
pid = posix.fork()
if pid == 0 then
  posix.exec("/sbin/ldconfig")
else
  posix.wait(pid)
end
%end

[pmatilai]

We theoretically could run Lua in a new process, but doing so basically obviates all of the point of using Lua in the first place. It might as well be shell script.

This misses the point of -p <lua> by a mile.

I'd actually like to have rpm run -p in a newly forked process to protect the main rpm from side-effects, but there are compatibility etc concerns with that. The original main point of -p <lua> was being able to do stuff when there's nothing at all to exec() in the initial install phase. Plus, people are starting to like and adapt it outside that domain because it's simply more nimble and elegant than throwing shell + dozen helpers at the same problem.

[jlebon]

I'd actually like to have rpm run -p in a newly forked process to protect the main rpm from side-effects, but there are compatibility etc concerns with that.

Would be great if rpm had that. Are there any plans to work through the compat issues there and support this?

[cgwalters]

The original main point of -p was being able to do stuff when there's nothing at all to exec() in the initial install phase.

See: http://lists.rpm.org/pipermail/rpm-ecosystem/2016-August/000396.html

If there's nothing to exec, there's also no reason to have a %pretrans at all - those are always "upgrade hack/workarounds".

Now, it's true today that the Fedora setup RPM ends up re-implementing filesystem in Lua because...well it's just a pile of crazy.

Anyways per that thread I think what we really want is something like %pretrans -p lua --upgrade-workaround which would only run when upgrading from a previous root. Then rpm-ostree could happily ignore it and so could the initial mock root etc.

I'd actually like to have rpm run -p in a newly forked process to protect the main rpm from side-effects, but there are compatibility etc concerns with that.

And agree this would be great! We could probably spare a bit of bandwidth to help with design/testing if a librpm developer was able to start the ball rolling.

[Conan-Kudo]

If there's nothing to exec, there's also no reason to have a %pretrans at all - those are always "upgrade hack/workarounds".

Lua isn't used just for %pretrans. It's used in places where we want to avoid a circular dependency on bash, too. In fact, that's why I wrote the systemd scriptlets in Lua upstream, only for you guys to force the systemd maintainer to rewrite them downstream in Fedora in shell (and thus, reintroduce the circular dependency).

[cgwalters]

It's used in places where we want to avoid a circular dependency on bash, too.

Fair; though...why the heck does bash require systemd?

Anyways, that also came up in the above-linked thread; today rpm-ostree unpacks all packages before running any scripts which solves that problem. And as I said in the thread, there's really no reason for "installing" bash to involve anything more than unpacking the files to get basic execution. IOW we don't need necessarily to unpack all packages before scripts, but it should work to unpack bash and coreutils etc. at least without processing any dependencies.

Is there a specific motivation for the increased activity in this ticket? "We want to use the lua systemd macros"?

[jlebon]

Is there a specific motivation for the increased activity in this ticket? "We want to use the lua systemd macros"?

Likely because I linked to it from https://bugzilla.redhat.com/show_bug.cgi?id=1780827#c30 (see also https://github.com/coreos/fedora-coreos-tracker/issues/459).

[Conan-Kudo]

Please just fix RPM-OSTree to handle Lua scriptlets properly. Being artificially restricted to shell is a terrible situation to be in...

[cgwalters]

Please just fix RPM-OSTree to handle Lua scriptlets properly. Being artificially restricted to shell is a terrible situation to be in...

As is being discussed above, doing so requires new librpm API.

[cgwalters]

I guess one thing we could do is synthesize an RPM containing just the target script to run and install it running something like bwrap ... rpm --dbpath=/tmp/throwawaydb -i /proc/self/fd/N --nodeps --notriggers.

[jlebon]

I'd actually like to have rpm run -p in a newly forked process to protect the main rpm from side-effects, but there are compatibility etc concerns with that.

Would be great if rpm had that. Are there any plans to work through the compat issues there and support this?

OK, I split this out into https://github.com/rpm-software-management/rpm/issues/1273. Let's discuss there how feasible this is before we resort to something a bit more drastic on the rpm-ostree side?

[heyakyra]

Seems related to https://github.com/fedora-silverblue/issue-tracker/issues/210

is there a workaround? or a way to pinpoint the dependency leading to this?

[travier]

~~https://github.com/rpm-software-management/rpm/pull/1867 > This one might enable us to do what we need here.~~

Edit: Apparently it's not enough as we will be missing the execution context.

[martinpitt]

This now breaks Fedora 36 builds due to the filesystem package using lua in posttrans.

Does anyone know some workaround? Like, can we tell rpm-ostree build to simply ignore these scripts? They don't seem to be critical for building the tree.

[lucab]

@martinpitt https://github.com/coreos/rpm-ostree/pull/3909