Meson Build System

[anarazel]

Tasks for meson use in postgres:

• [ ] Missing Features
  • [x] #38 libedit support (easy-medium)
  • [x] generate pkg-config files (easy-medium)
  • [ ] fully implement "native language support" (see https://postgr.es/m/bfcd5353-0fb3-a05c-6f62-164d98c5689d%40enterprisedb.com )
  • [x] check vs installcheck (maybe via different test setups?)
  • [x] #50
• [x] Not yet built in meson:
  • [ ] check for other bits and pieces
  • [x] ECPG, except for windows
  • [x] #22
• [x] Not tested yet with meson:
  • [x] ECPG (medium)
  • [ ] search for other "manually invoked" tests (easy)
• [x] OS dependent issues
  • [x] #9 (unknown difficulty)
  • [x] rpath on netbsd broken for at least perl, there's also a possibly related issue on suse
  • [x] AIX symbol generation needs to be fixed
  • [x] windows icon generation (PGAPPICON and PGFILEDESC in makefiles) is implemented only for one or two tools, needs to be generalized
  • [x] simplify exports.txt handling, use it in all the required places (e.g. ecpg)
• [ ] Installation
  • [x] "pkglibdir" logic from src/Makefile.global.in needs to be mirrored to some degree (medium)
  • [x] #30 (partially done)
  • [ ] Header install with install_subdir() is error prone (think tempfiles)
• [x] Cleanups:
  • [x] Centralize operating-system specific logic. There's too many places querying host_machine.system() right now
  • [x] Merge gendef.pl and gendef2.pl back together (medium)
  • [x] src/tools/testwrap should be cleaned up made unnecessary (path unclear as of yet)
  • [x] #36 (medium)
  • [x] #31 (easy-medium, fairly verbose changes)
• [ ] Open questions:
  • [x] figure out how to deal with PGXS / building out-of-tree extension
  • [ ] LLVM bitcode generation implementation isn't even close to generalized enough (hard)
• [ ] Other Todos:
  • [ ] #58 (easy - medium)
  • [x] installation.sgml etc need a lot of adjustment
  • [ ] #33 (easy)
  • [ ] sync up configure tests / meson implementation, add missing ones
• [x] Bugs:
  • [x] #28
  • [x] #32 (easy)
• [ ] Implement missing options:
  • [x] Synchronize option descriptions from configure.ac to meson_options.txt (easy)
  • [x] Check which other options we want to have that configure has but meson_options doesn't
  • [x] #15
  • [x] #16
  • [x] #17
  • [x] #18
  • [ ] #61

Testing Tasks

• [ ] automated testing of various linux distributions, via docker containers, at least (should be optional cirrus tasks):
  • [x] #19
  • [x] #20
  • [x] #21
  • [x] Suse (See #7 )
  • [ ] Debian on linux arm (supported by cirrus), image exists already (easy)
• [x] automated testing via VMs
  • [x] OpenBSD ( https://github.com/anarazel/pg-vm-images/issues/6 )
  • [x] NetBSD ( https://github.com/anarazel/pg-vm-images/issues/7 )
• [x] automated test against all supported meson versions (easy-medium)
• [ ] #81 (easy-medium)
• [ ] Test a few extensions built via PGXS in CI (easy)
• [ ] Automate building a lot of extensions via script (medium)

Improvements to the meson integration that likely aren't critical:

• [ ] build dependency files for xsltproc et al for more accurate incremental builds (medium-hard)

Questions:

• [ ] Do we need to implement "sqlmansectnum" logic for docs?
• [ ] Partial installation support needed? Right now everything will be installed. Likely a problem for things like src/test/modules/dummy_index_am - but those currently need to be installed for tests.

Relevant meson improvements / issues:

• caching for library searches - biggest contributor to configure time now
• mesonbuild/meson#10002 - worked around, but still a trap
• mesonbuild/meson#8511 - not critical, but helpful
• mesonbuild/meson#9493 - worked around, but still a trap
• mesonbuild/meson#9254 - would simplify code
• mesonbuild/meson#9894 - not critical, but would allow easier to analyze test results
• mesonbuild/meson#10032 - should fix so that we eventually can rely on tap subtests (once minimum meson version is high enough)

[samay-sharma]

@anarazel : Can you update the Task list with this issue as well : https://github.com/mesonbuild/meson/issues/9894 . I opened this against meson for the last meson issue.

[anarazel]

@anarazel : Can you update the Task list with this issue as well : mesonbuild/meson#9894 . I opened this against meson for the last meson issue.

Done. You probably now would also have the permissions to do so.

[tristan957]

Could you expand on this?

Header install with install_subdir() is error prone (think tempfiles)

I have submitted https://github.com/mesonbuild/meson/pull/11849 fyi.

[eli-schwartz]

install_subdir() will install an entire directory tree, even if not all the files in that tree are supposed to be installed. This can also include text editor backup files etc.

https://github.com/anarazel/postgres/blob/d1491d36a3cb6ca14ff55b5d1506ba35a1e7f35a/src/include/meson.build#L163-L172

Meson 0.63.0 adds preserve_path: true for install_headers() which allows spelling out every single file and then installing an entire structured tree. This may or may not be a more palatable approach -- either way it's not for today as postgres still targets systems with 0.56.0, though of course any new features would have the same time delay...

[anarazel]

@tristan957 Does @eli-schwartz 's explanation answer your question?

Meson 0.63.0 adds preserve_path: true for install_headers() which allows spelling out every single file and then installing an entire structured tree. This may or may not be a more palatable approach -- either way it's not for today as postgres still targets systems with 0.56.0, though of course any new features would have the same time delay...

It's not really a palatable approach unfortunately. We're talking ~750 files here. Maintaining that by hand is too errorprone.

I suspect that the best way for now is a small helper script that lists the files, separates them by directory using the fs module, and than calls install_headers() for each of those directories :(.

[tristan957]

@anarazel it does. Could you explain what would be a palatable approach? Are you looking for like an exclude_patterns kwarg?

[anarazel]

@anarazel it does. Could you explain what would be a palatable approach? Are you looking for like an exclude_patterns kwarg?

I am thinking of the opposite. I would like to explicitly specify include patterns, with everything not matching not getting installed. IME it's quite rare that one needs to install all but a few patterns, but very common to install just one type of file.

[tristan957]

Are you thinking that glob syntax would be enough?

[tristan957]

Perhaps a patterns kwarg and then doing the equivalent of what github does would be enough.

https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#onpushpull_requestpull_request_targetpathspaths-ignore

Then you get negated globs too

[anarazel]

Are you thinking that glob syntax would be enough?

Depends on the complexity of the globs. I think a full regex would be good - without a regex it's hard to exclude temp files. I'd basically want to say something like include_patterns = ['^[a-zA-Z0-9]+.*.h'], so things like ".#filename.h" etc aren't "included".

[tristan957]

Hmm. A regex module would be interesting. I think what would be annoying is that flavors of regex differ in what they support, so if Meson exposed regex through the re module and Muon exposed it through libpcre2, you would get two different impls.

Another option would be to expose https://docs.python.org/3/library/glob.html.

[tristan957]

Maybe Meson could just define no positive/negative lookaheads/behinds. That would probably be pretty simple at the end of the day since those seem to be the biggest differentiator between regex impls.

[eli-schwartz]

Yeah there's a few reasons why I'm not a fan of regex here. Layering an exclude pattern on top of an include pattern would probably work better here though.

[tristan957]

Eli, so are you saying that we need include_directories and include_files alongside what already exists for exclude_directories and exclude_files? Or are you advocating for patterns? If so, what pattern syntax?

[eli-schwartz]

It's not really a palatable approach unfortunately. We're talking ~750 files here. Maintaining that by hand is too errorprone.

I suspect that the best way for now is a small helper script that lists the files, separates them by directory using the fs module, and than calls install_headers() for each of those directories :(.

One possible alternative here is to have the script generate these lists, and write them to a file in the source tree e.g. headerlist.txt, and then use the fs.read() function to read out that list and dump it into the install_headers() function.

Doesn't need to be maintained by hand, the script can run itself in check mode either at configure time or as a test(), and the generator script doesn't need to understand meson syntax well enough to edit a meson.build file.

[anarazel]

One possible alternative here is to have the script generate these lists, and write them to a file in the source tree e.g. headerlist.txt

I was thinking of using run_command() and using the output to trigger install_headers() - does going via a file have advantages? I'd assume the time overhead of running that script to be neglegible compared to the other costs of running configure.

and then use the fs.read() function to read out that list and dump it into the install_headers() function.

For 0.56 compatibility we'd need to split into multiple install_headers() invocations, one for each directory, right?

I don't love the idea of checking in something like headerslist.txt, that opens up an unnecessary avenue of merge conflicts...

[anarazel]

Hmm. A regex module would be interesting. I think what would be annoying is that flavors of regex differ in what they support, so if Meson exposed regex through the re module and Muon exposed it through libpcre2, you would get two different impls.

Another option would be to expose https://docs.python.org/3/library/glob.html.

I wonder if a good solution could be to have a install_* option to not install hidden files. I'd assume it's very rare that people actually want to do so. And it'd largely remove the need for a full blown regex, I think.

[tristan957]

Hmm. A regex module would be interesting. I think what would be annoying is that flavors of regex differ in what they support, so if Meson exposed regex through the re module and Muon exposed it through libpcre2, you would get two different impls. Another option would be to expose https://docs.python.org/3/library/glob.html.

I wonder if a good solution could be to have a install_* option to not install hidden files. I'd assume it's very rare that people actually want to do so. And it'd largely remove the need for a full blown regex, I think.

I wonder if install_subdir() should exclude meson.build files by default. With the hidden file thing, that would seem to be everything Postgres needs. But maybe there is a workflow where installing meson.build files is appropriate. Perhaps exposing that as an option would be a good idea.

[eli-schwartz]

There's no particular difference between reading it in via a run_command and doing so via fs.read(), except that in the latter case it's more idealistically pure because when you git pull and new files are added, the filelist.txt gets updated at the same time and this triggers a reconfigure. Doing this via run_command is, really, just the meson FAQ entry on support for globs in source lists. You can do it if you want and are willing to accept the unreliability, but meson won't officially support it.

(install_subdir() is the explicit exception to the globbing rule, as it only calculates a directory name at configure time and processes the implicit wildcard glob at install time as part of the single install rule.)

[tristan957]

Where is the Postgres workaround for https://github.com/mesonbuild/meson/issues/9493?

Given that https://github.com/mesonbuild/meson/issues/8511 is now exposed as find_program('meson'), I am wondering if we should add a meson.implementation() method that would tell us what the invoking Meson implementation is. Meson would return 'meson'. Muon would return 'muon'. Meson++ would return 'meson++'.

Otherwise, we need a better way to do https://github.com/postgres/postgres/blob/master/meson.build#L3024-L3027.

[tristan957]

I submitted https://github.com/mesonbuild/meson/pull/12014, which should be a pretty nice cleanup when the minimum version bump occurs.

[anarazel]

@tristan957

Where is the Postgres workaround for https://github.com/mesonbuild/meson/issues/9493?

https://github.com/postgres/postgres/blob/62e9af4c63fbd36fb9af8450fb44bece76d7766f/meson.build#L671-L677

[anarazel]

Given that mesonbuild/meson#8511 is now exposed as find_program('meson'), I am wondering if we should add a meson.implementation() method that would tell us what the invoking Meson implementation is. Meson would return 'meson'. Muon would return 'muon'. Meson++ would return 'meson++'.

Otherwise, we need a better way to do https://github.com/postgres/postgres/blob/master/meson.build#L3024-L3027.

Yea, that'd be nice.

[tristan957]

https://github.com/mesonbuild/meson/pull/12051 is tackling vs_module_defs for executables.

[tristan957]

https://github.com/mesonbuild/meson/pull/12051 is now closed. That leaves us with two more issues to tackle for better Meson support in Postgres. The framework include issue, which I will be working on and better support for install_subdir() somehow...

I guess also https://github.com/anarazel/postgres/issues/61, but that one isn't very urgent.

[anarazel]

Very nice!