required: roadmap for pg versions support

[tsipo]

I just saw this issue concerning pg 14 on Windows (https://github.com/apache/incubator-age/issues/138). The reply there Support for PG 12 and beyond likely won't start until 2022 is very discouraging, and contradicts Age home page which states AGE is currently being developed for the PostgreSQL 11 release and will support PostgreSQL 12 and 13 in 2021 and all the future releases of PostgreSQL. I see now that the readme.md of this project states To focus more on implementing the openCypher specification, the support for PostgreSQL 12 will be added in the Q1 2022. We use in production pg 13 (linux), and going back to 11 is NOT an option (also due to other extensions). Mind you, pg 11 is 3+ y.o. and pg 12 2+ y.o. I tried to build AGE from source with both pg 13 and pg 14 (on linux), it fails to compile due to changes to pg h files (different failures for the two versions). I believe that for many organizations, the decision whether or not even to consider AGE as an option depends on its roadmap for supporting various pg versions. Which is what I am asking you to provide. Not in vague statements, but targets for each version - when do you expect to support 12, 13, 14? Thank you.

[JoshInnis]

Hi tsipo, We are currently finishing up PG 11. We will investigate AGE on PG 12, 13, 14, and 15 and provide a more concrete timeline towards the conclusion of original goal of supporting the openCypher specification in Postgres 11.

[mprimeaux]

@JoshInnis Hi there. I still see here that only Postgres 11 and 12 are supported. Is the documentation correct or are latest versions (e.g. 14 and 15) supported?

Thanks in advance.

[jrgemignani]

@mprimeaux Yes, that documentation is correct. Only PG11 and PG12 are supported releases atm.

PG13 is in alpha (branch alpha/PG13) and will hopefully be released by the end of the month. PG14 is experimental (branch experimental/PG14) and will hopefully be released as an alpha within a few months.

Both of these, however, are available to clone and build. Just remember, they haven't been released, are not necessarily complete, and may have pending PRs for significant changes.

[mprimeaux]

Thanks much and noted.

My focus is running this extension in Postgres within Kubernetes on both linux/amd64 and linux/arm64 and so I'm sure I may have a few more questions once I get my head around the source.

Again, thanks for your help.

[mprimeaux]

I managed to work through what is needed to build a single OCI-compliant multi-stage, multi-architecture container image using nerdctl and qemu that produces a substantially smaller image.

nerdctl images

REPOSITORY                                  TAG       IMAGE ID        CREATED              PLATFORM          SIZE         BLOB SIZE
apache/age                                  latest    f228f7fc52b2    About an hour ago    linux/amd64       914.2 MiB    322.3 MiB
localhost:5000/apache/age                   latest    1a3b9c5f2ae0    11 seconds ago       linux/arm64       366.4 MiB    127.1 MiB
localhost:5000/apache/age                   latest    1a3b9c5f2ae0    11 seconds ago       linux/amd64       0.0 B        131.8 MiB

The first image (above) is the latest AGE image on DockerHub. The second two lines represent the single OCI manifest containing both linux/amd64 and linux/arm64 images.

As you can see, I was able to reduce the size from 914 MiB / 322 MiB to effectively 127.1 MiB (arm64) and 131.8 MiB (amd64). For comparison, this is slightly larger than the official postgres container images, which is expected given the final layer:

REPOSITORY                                  TAG       IMAGE ID        CREATED              PLATFORM          SIZE         BLOB SIZE
postgres                                    14        36c7e29b775b    About an hour ago    linux/amd64       383.8 MiB    131.6 MiB
postgres                                    14        36c7e29b775b    About an hour ago    linux/arm/v5      0.0 B        125.2 MiB
postgres                                    14        36c7e29b775b    About an hour ago    linux/arm/v7      0.0 B        120.1 MiB
postgres                                    14        36c7e29b775b    About an hour ago    linux/arm64/v8    365.7 MiB    126.9 MiB
postgres                                    14        36c7e29b775b    About an hour ago    linux/386         0.0 B        133.1 MiB

I'm still getting my head around the CI process in this repository but more to come. I'm very keen to ensure we build container images (where possible) for all platforms supported by QEMU assuming folks here are also supportive of this direction.

Please let me know.

UPDATE: This is using the experimental/PG14 branch but this process will be the same for PG 11+.

[jrgemignani]

@mprimeaux We are aware of the larger images and I'm not sure who is dealing with those specifically, nor the current status of that issue. I will see if I can find out anything.

I will note that experimental/PG14 is just that. It was submitted by a user, which is really great. However, being that our focus is on 13, we have not really had an opportunity to review it; hence the experimental. Additionally, it is based off 12 which is in the process of being updated to the current 11 version. So, when it comes to full functionality, it is behind 11 atm.

[mprimeaux]

@jrgemignani Thanks for this detail. The reason why I'm highlighting this is that I'm happy to contribute and collaborate on whomever is working on it. To this end, I'm fine to submit a PR once ready for review by any of the committers.

The reason why I'm highlighting this is that with the extreme "uptick" in the usage of ARM64 workloads, I think it's important to first support that CPU architecture before dealing with smaller images but why not address both in the same commit.

I'm in the process of reasoning through the use of this extension and dgraph for our particular set of platforms. Both are not mutually exclusive when considering the set of durable storage technologies our solution domain. Using both for our workloads is worth our evaluation.

[jrgemignani]

@mprimeaux Any help with the project is always appreciated! I will leave this issue open for others who may want to collaborate.

[mprimeaux]

@jrgemignani Thanks much. I don’t mind creating a new ‘issue’ for support of multi-architecture images if that’s more appropriate. Your call but happy to help.

[jrgemignani]

@mprimeaux Let's try to get a few more people in the discussion, if possible, and see what works best. We can always create more issues.

[mprimeaux]

Given the focus is Postgres 13 as per above, I think I’ll first focus on that direction as a priority. Please let me know if that works. Again, thanks for the help.

[mprimeaux]

On a related topic, rather than producing a container image that Postgres + AGE, why wouldn't we just produce a container image with just the AGE-related files (.so, .sql, and .control) and use an init container against whatever Postgres Helm chart that the community uses?

An advantage of this direction would be that for folks running in k8s, they can simply use their existing Helm charts rather than dealing with trying to point to the full Postgres + AGE image, which won't work in cases where the image maintainer has added their own entry point wrapping.

I suppose we can do both. Anyway, just curious if this or creating a Helm chart for AGE has already been discussed.

[mprimeaux]

I took the time this weekend to create an Age CI pipeline and a Helm chart.

The CI pipeline performs the following:

1. Builds Apache AGE container images supporting the linux/amd64 and linux/arm64 CPU architectures using nerdctl and binfmt.
2. Builds a container image of AGE without Postgres (more on this below).
3. Builds a container image of AGE with Postgres.

The Helm chart supports two deployment scenarios:

1. Deploy the Official Postgres and uses an init container to configure AGE.
2. Deploy the Official AGE container image.

The advantage of supporting both scenarios is adoption by (1) folks using existing community Postgres Helm charts that want to use AGE and (2) folks who are just fine using the container image of AGE that includes a Postgres distribution.

I've got a small bit of cleanup to do but am happy to contribute it to this project if there's interest. If there is interest, then my plan is to first make the above implementation available for an initial review. Then I'll fork this repository and submit a PR for review.

Thanks for the consideration.

[jrgemignani]

@mprimeaux I will try to get some others to give their input on this here.

[mprimeaux]

@jrgemignani Hope your week is going well. Any updates that you can share? Again, thanks.

[jrgemignani]

@mprimeaux Still trying, while doing many other things, to get others to look at this. PG13 is close to being supported, as in weeks. PG11 & PG12 are supported and their respective branches have the latest.

[mprimeaux]

@jrgemignani No worries and totally understand. Congrats on the PG13 support. Good to know re: PG 11 and 12. Why don't we revisit once you're through PG 13. In the interim, I'll at least ensure what I have supports 11, 12, and 13 on their respective branches.

I'll check back or just ping me.

[mprimeaux]

@jrgemignani Just checking back to see how things are progressing and if any others care to comment on the above. Thanks, again.

[eyab]

@mprimeaux I'm Just checking in to see if you have any questions.

[eyab]

@tsipo Any updates on this issue?