plaster
plaster copied to clipboard
Pylons
Add "raw" keyword-only parameter to Loader.get_settings()
Hi,
This makes it possible to copy configurations and avoid multiple interpolations, or interpolation errors.
In a pyramid-cookiecutter-starter patch I pull out configuration from a yaml file and load it into an configparser.Config() configuration, or simply pull configuration from an ini file and load it into alembic's configuration. (No reason to write different code for yaml and ini config file sources).: https://github.com/kpinc/pyramid-cookiecutter-starter/blob/fad9192058e950b17c3e2f0232729e69bdc4984b/%7B%7Bcookiecutter.repo_name%7D%7D/tests/sqlalchemy_conftest.py#L37 https://github.com/kpinc/pyramid-cookiecutter-starter/blob/fad9192058e950b17c3e2f0232729e69bdc4984b/%7B%7Bcookiecutter.repo_name%7D%7D/tests/sqlalchemy_conftest.py#L48
It seems like an important feature, even if pyramid-cookiecutter-starter, enhanced to support yaml config files, might somehow not need to use it.
Further explanation of the use of the "raw" keyword in my pyramid-cookiecutter-starter fork may be warranted.
If a yaml config file is chosen for Pyramid, and sqlalchemy is the db backend, a ini config file must still be generated for alembic. Alembic only takes an ini config file. So, when the regression tests run with a yaml config file the test fixture copies the yaml config programmaticaly and calls the alembic API to give alembic a config to use.
The convenience is that the test code need not ever worry about the format of the supplied config. It just always copies whatever config it gets as input and gives it to alembic.
The test code could be re-engineered to special-case on the config file format in-use, loading alembic's ini config specially and using 2 config files when Pyramid does not take an ini config format. But that seems clunky and opens the possibility that the 2 config files contain different configurations. This would cause confusion.
Best is to be able to cleanly copy configs, without double-interpolation/etc.
I'm trying to follow the things you're saying about this but it's against my original design of plaster to expose these types of details so having trouble wrapping my head around these cases you're describing with the cookiecutter and alembic.
If alembic needs an ini and you move to a different plaster backend that is not ini then it seems to me you just need to define a separate alembic.ini. Is that wrong? Is the proposal to do something else to convert the yaml into an alembic.Config object yourself and not have alembic load it from a file? Seems like that is what you are doing in your conftest.py at least.
It's just kind of a happy little coincidence right now that alembic is compatible with pastedeploy's ini config.
On Wed, 17 Apr 2024 13:02:30 -0700 Michael Merickel @.***> wrote:
I'm trying to follow the things you're saying about this but it's against my original design of plaster to expose these types of details so having trouble wrapping my head around these cases you're describing with the cookiecutter and alembic.
Maybe what I'm doing is "just wrong" and does not fit with your design.
If alembic needs an ini and you move to a different plaster backend that is not yaml then it seems to me you just need to define a separate alembic.ini. Is that wrong?
Not really. There are some disadvantages but nothing that can't be lived with. The existing cookiecutter code reads only a single config file and I tried to continue in that vein without using duplicative config files in different formats. (Most of the duplication is in the logging config. Alembic does not require much itself.)
I am already generating separate alembic.ini files (development, production, etc.) in the cookiecutter so I'd just need to use the testing one in the testing code.
Is the proposal to do something else to convert the yaml into an alembic.Config object yourself and not have alembic load it from a file? Seems like that is what you are doing in your conftest.py at least.
Yes, that is the proposal. That configurations be programmaticly copyable, which means being able to turn interpolation and such off and on. Otherwise, every copy interpolates (etc.), escaping becomes impossible or a matter of knowing the internals of how many copies are made, etc.
However, this may only make real sense when copying between mechanisms that understand similar markup, do similar things with defaults, etc. The cookiecutter fork punts on this and works mostly because the alembic config files are simple. In the general case there's no way of knowing what's going to need post-processing (beyond the initial parse of the underlying data structure) and it is unlikely that the "raw" format is going to be universally interpret-able.
But this does not mean that "raw" is not useful. My original code copied from yaml to ini just fine. The problem was that copying from ini to ini "double interpreted". (So, I leapt forward and started coding. Huzzah! :-)
Programmatic copying does seem to fit in with mixing config files and declarative in-code configuration. But maybe that's only true on the surface.
Given your reticence, I'm starting to think that I should do the direct and simple thing and use two config files. There's some duplicate configuration and a little more code with an explicit special case. Nothing really wonky. And you can keep plaster simple, at least until there's a more compelling use-case for copying configs.
Anyway, I'm happy to take your guidance here. (I'm sure I need it, having worked in something of a vacuum.) Whatever decision you make will let me move forward with the cookiecutter patch.
Thanks for the help.
Regards,
Karl @.***> Free Software: "You don't pay back, you pay forward." -- Robert A. Heinlein
On Wed, 17 Apr 2024 20:13:49 -0500 "Karl O. Pinc" @.***> wrote:
On Wed, 17 Apr 2024 13:02:30 -0700 Michael Merickel @.***> wrote:
I'm trying to follow the things you're saying about this but it's against my original design of plaster to expose these types of details
Maybe what I'm doing is "just wrong" and does not fit with your design.
How does this plan sound? I'll make another fork of the cookecutter starter that does not use "raw" or otherwise require changes to plaster et-al. You can look at the two of them, diff them, etc., and decide whether any changes to plaster are warranted.
The cookiecutter starter fork without "raw" should run and you'll be able to also see all the yaml and ini config files.
Always nice to have something concrete to look at.
If I don't hear anything back from you I'll proceed with this plan and let you know when there's something to look at.
Regards,
Karl @.***> Free Software: "You don't pay back, you pay forward." -- Robert A. Heinlein
Irrespective of pyramid-cookiecutter-starter, there is another consideration. Without a "raw" get_settings() parameter plaster cannot read entire configuration files.
One case is the "stock" pyramid ini config files generated by the cookiecutter.
How relevant this is I can't say. It's unclear how useful it is to be able to read un-interpolated, undefaulted, config file sections. It might sometimes be useful, I guess.
This code (using the stock plaster and plaster_pastedeploy) fails:
import plaster
loader = plaster.get_loader('development.ini')
config = {}
for section in loader.get_sections():
config[section] = loader.get_settings(section)
print(config)
The traceback ends with:
configparser.InterpolationMissingOptionError: Error in file development.ini: Bad value substitution: option 'format' in section 'formatter_generic' contains an interpolation key 'asctime' which is not a valid option name. Raw value: '%(asctime)s %(levelname)-5.5s [%(name)s:%(lineno)s][%(threadName)s] %(message)s'
This code (using the patched plaster and plaster_pastedeploy) works:
import plaster
from configparser import InterpolationMissingOptionError
loader = plaster.get_loader('development.ini')
config = {}
raw_config = {}
for section in loader.get_sections():
try:
config[section] = loader.get_settings(section)
except InterpolationMissingOptionError:
raw_config[section] = loader.get_settings(section, raw=True)
print(config)
print()
print(raw_config)
I don't want to advocate for anything. It is a little odd that a config file reader can't always read entire configs. But it could be that that's a feature.
Regards,
Karl @.***> Free Software: "You don't pay back, you pay forward." -- Robert A. Heinlein
On Wed, 17 Apr 2024 20:13:49 -0500 "Karl O. Pinc" @.***> wrote:
How does this plan sound? I'll make another fork of the cookecutter starter that does not use "raw" or otherwise require changes to plaster et-al. You can look at the two of them, diff them, etc., and decide whether any changes to plaster are warranted.
Done. There's a pyramid-cookiecutter-starter PR #137.
On Wed, 17 Apr 2024 20:13:49 -0500 "Karl O. Pinc" @.***> wrote:
How does this plan sound? I'll make another fork of the cookecutter starter that does not use "raw" or otherwise require changes to plaster et-al.
On reflection, I like the cookiecutter better without the "raw" parameter. It's simpler.