Lazy loading factory classes for config

[ab93]

Currently, the config factory imports are not lazy, and all of classes get imported. This should ideally be a lazy import, where classes are imported on a need basis.

---

Message from the maintainers:

Impacted by this bug? Give it a 👍. We often sort issues this way to know what to prioritize.