[Proposal] Application config compilation

[antonkril]

Problem Statement

Magento's modular nature necessitates a complex configuration and metadata retrieval process. However, the inherent complexity of this process is exacerbated by the inconsistent implementation. Magento employs two opposing methods for retrieving developer configuration and metadata: the Just-In-Time approach and the Ahead-Of-Time approach.

The Just-In-Time Approach

In this approach, when the Magento application needs to obtain its configuration (di.xml, events.xml, system.xml, etc.), it reads the relevant configuration files from all modules, merges them, validates the merged configuration, and caches it.

The Ahead-Of-Time Approach

When operating in production mode, Magento assumes that all Dependency Injection (DI) and Interception metadata, as well as generated classes, have already been produced by the Magento compiler, a separate application.

These contradictory approaches lead to confusion and hinder the developer experience by:

• Requiring developers to understand various methods of cache/metadata/generated code cleanup
• Causing discrepancies in application behavior between developer and production modes
• Lacking an easy and clear way to view the merged application configuration, necessitating the use of step debuggers, reading cache storage, or other tools
• Deviating from how non-Magento developers typically work with their frameworks and applications

Proposed Solution

• Remove the just-in-time approach from the Magento application
  • Introduce an alternative configuration component that will directly read the merged configuration from the filesystem (or another storage medium)
  • Eliminate the concept of developer configuration cache from the Magento application
• Rely solely on the ahead-of-time approach
  • Develop a new generation of the compiler
  • Transfer the configuration reading, merging, and validation logic to the new compiler
  • Implement incremental configuration merging capability
  • Incorporate incremental compilation capability (later stage)
  • Add a filesystem watcher to initiate the incremental configuration merging and compilation functions

Contribution checklist (*)

• [ ] Pull request has a meaningful description of its purpose
• [ ] All commits are accompanied by meaningful commit messages
• [ ] All new or changed code is covered with unit/integration tests (if applicable)
• [ ] README.md files for modified modules are updated and included in the pull request if any README.md predefined sections require an update
• [ ] All automated tests passed successfully (all builds are green)