[wip] [master] Initial purge of community extensions

[dwoz]

What does this PR do?

Initial purge of community extensions for 3008

#65970

Merge requirements satisfied?

[NOTICE] Bug fixes or features added to Salt require tests.

• [ ] Docs
• [x] Changelog - https://docs.saltproject.io/en/master/topics/development/changelog.html
• [x] Tests written/updated

[max-arnold]

Things that seem useful to keep in core (esp. sqlite modules, nacl modules, pkg runner, aptpkg/denconfmod states, rsync state):

engines:
stalekey.py <============== Useful for Cloud environments where minions come and go

modules:
smtp

output:
profile

pillar:
cmd_yaml <===============
cmd_json <==============
http_yaml
http_json
nacl
sqlite3 <===============

queues:
pgjsonb_queue
sqlite_queue <================

renderers:
pydsl

returners:
smtp_return
sqlite3_return <===================

runners:
nacl
pkg <===================
thin <???????? Salt-ssh?

sdb:
cache
rest
sqlite3 <===============

serializers:
keyvalue

states:
aptpkg <===================
debconfmod <==============
ethtool
kernelpkg
rsync <==================
smtp
sqlite3 <==================
xml

Alternatively, what is the process to get them back into core, if there is a significant demand?

[Oloremo]

I'd also like to propose to keep some stuff but before that - what is the current approach of deciding what goes and what stays? What do we consider a core?

[lkubb]

Pretty sure the right place for general proposals is https://github.com/saltstack/great-module-migration. My comment above was intended to point out we shouldn't rip apart a pair of execution/state module.

[dwoz]

Things that seem useful to keep in core (esp. sqlite modules, nacl modules, pkg runner, aptpkg/denconfmod states, rsync state):

@max-arnold We're looking at these. I will follow up with more, hopefully this week.

[whytewolf]

states: aptpkg <===================

so of the list this item should be decommissioned and removed not just turned into an extension. aptpkg has 1 state in it. and that same functionality already exists in pkg. for way more operating systems than just debian based.

[dwoz]

Alternatively, what is the process to get them back into core, if there is a significant demand?

At this point it's very unlikely that anything removed by this PR will get added back in. If there is anything out of this list that is useful then there should be sufficient enough motivation for someone other than the core team to maintain a salt extension. Even if something is deemed to be worth of being supported further by the core team. We'd make an extension rather than adding it back into Salt. The one outlier I can think of would be we find a significant bug caused by something that has been removed.

[KartikShrikantHegde]

Hi @dwoz , what would be the tentative timeline for 3008 release?

[jheiselman]

I noticed that the cache modules are not in any of the lists. Is there a disposition for those yet? We use the redis cache module and noticed that the redis execution and returner modules are being moved out to the community extensions.

[jheiselman]

The etcd state module is listed as being core, but all other etcd-related modules (returner, execution, sdb) are listed as community. The state currently depend on the execution module for some functionality.

[jeffclay]

What is the motivation behind this? It seems like a lot of people are going to upgrade and find lots of missing/broken states. Shouldn't these modules, etc. have been flagged as deprecated prior to being deleted?

[boltronics]

I just went to submit a patch for one of these modules and discovered this change just today. I find this quite scary. There is nothing in the execution module docs that state that this is going to happen, and it impacts me significantly.

Shouldn't these be split out into extension modules, supported or not (even if "archived"), before removing them from the main project? At least that way, people can fork what is needed and upgrade with minimal pain.

As requested months ago, it seems that we should have some transparency around the release timeline for 3008, and we urgently need some deprecation notices on the docs of modules for all currently supported versions.

If the expectation is that the community will pick up support for anything removed, it may take time for that to happen. Such work will likely not even begin if people are not fully across what is happening.

[jleroy]

I totally agree with @boltronics: you can’t remove hundreds of community extensions, some heavily used, without any dialogue with the community and especially without issuing deprecation notices at least 3 major versions before effectively removing them. This is simply madness. Also please explain what’s the motivation behind this decision.

[Oloremo]

Their motivation is quite simple - core team is gutted and they have no capacity to support all these modules. So they cut them off with the idea that they will be available as extensions with community support.

Ansible did something like that in 2.10: https://docs.ansible.com/ansible/latest/collections/community/general/index.html

As for communication and deprecation - they had some communication via Community All-Hands meeting at least and current Salt situation is that they can't really wait for some graceful deprecation period. Salt will do a lot of radical changes to survive.

P.S. I'm paraphrasing things that I heard + add my understanding of the situation. Take it with a grain of...

[dmurphy18]

@Oloremo FYI, the original idea for moving modules to salt-extensions, was generated by discussions in the bar after PyCon 2018 in Cleveland, and the team had slowly started to move modules to salt-extensions, but you are correct, that after the gutting, the speed up to move parts to salt-extensions accelerated, since gutted, but the Core Team was moving that direction.

Back at PyCon in 2018, there were 400+ modules, and we needed to get the number down to allow for a stable core of Salt which could be improved and make thoroughly solid, with appropriate test, code coverage, etc. Also moving parts to salt-extensions would allow for faster resolution of issues for that part, rather than being tied to a Salt release date, for example: saltext.vault has definitely improved Vault support in Salt.

[max-arnold]

A question for everyone:

What additional communication channels (besides the SEP process, Open Hour meetings, blog posts, Slack/Discord community. Github issues) the Salt Core team should have used to inform about the decision to remove these modules from Salt?

[Oloremo]

95% of users don't read any of that, IMO. But I'd believe that at least half would read release notes, so with a graceful deprecation message in the upcoming releases, it would be distributed to a critical mass of users.

Sadly that won't be the case.

[jeffclay]

A question for everyone:

What additional communication channels (besides the SEP process, Open Hour meetings, blog posts, Slack/Discord community. Github issues) the Salt Core team should have used to inform about the decision to remove these modules from Salt?

Following the guidelines laid out here https://docs.saltproject.io/en/master/topics/development/salt_extensions.html#how-do-i-deprecate-a-salt-module-to-a-salt-extension would be nice.

To indicate that a Salt module is being deprecated in favor of a Salt extension, for each Salt module include deprecated tuple in the module. The tuple should include the version of Salt that the module will be removed, the name of the collection of modules that are being deprecated, and the URL where the source for the new extension can be found. The version should be 2 major versions from the next major release. For example, if the next major release of Salt is 3100, the deprecation version should be set to 3102.

[boltronics]

What additional communication channels (besides the SEP process, Open Hour meetings, blog posts, Slack/Discord community. Github issues) the Salt Core team should have used to inform about the decision to remove these modules from Salt?

Thanks for asking. I think that the problem with all of those listed is that they require someone to actively go out of their way to review.

IMO, significant changes such as this are worthy of a post to the salt-announce mailing list. Alternatively, the Salt Blog posts could be pushed to a dedicated mailing list, if they are to be considered important reading.

Unfortunately, the open hour at 10 PST is 3AM AEST, so I've never watched one live. I have watched the odd recording, but not in a while. However, I'd be more likely to check out recordings if there were a mail list that listed discussion points and linked to timestamps that I could jump to.

Aside from the announce mailing list, I mostly just look at https://docs.saltproject.io/en/latest/topics/releases/index.html#upcoming-release and https://docs.saltproject.io/en/latest/ref/index.html unless I run into a problem — which is how I found out about this. So, if you wanted to be really thorough, you could put an "important announcement" text/icon in the documentation header containing a link to where these upcoming changes have been documented.

I seldom use Discord as I generally find it distracting when working.

[edit: fixed a typo]