[Feature Request] Yugabyte Build

[grcooper]

Hello!

I was wondering if it would be possible to get a release of libpg_query that supported Yugabyte specific syntax for AST parsing. There are some differences, the main ones are around PRIMARY KEY usage, but there are some others.

Is this something that we could create for the latest builds of yugabyte?

[lfittl]

Thanks for opening this issue here!

For transparency, I'm currently not in favor of doing this for maintenance reasons, but happy to discuss/consider a bit what it'd take and what the exact requirements would be.

First of all, we'd need to expand the parser syntax to cover YDB specific syntax. I've done a diff of the PG15 parser to the current YugabyteDB master branch (based on PG15), which does not seem terribly bad: https://gist.github.com/lfittl/fe41c6ae8ddf9c11fb2142fd7c456e6c

Then we'd need to decide whether the goal is to accept the syntax and ignore it, or to actually create the right parse nodes.

To help understand this better, it'd be useful to have some examples of syntax that is currently failing for you, so we can determine whether they'd need parse nodes created, or could simply be ignored.

In addition, can you share more on your use case? Is your goal to parse DDL statements (from a schema structure dump maybe?), or are you parsing regular SELECT/DML type statements?

[grcooper]

The use case is to parse DDL statements. We want to do some safety checks on our migrations and being able to do that through parsing the actual SQL is much more sustainable than adding a variety of regex checks.

The main couple that we have run into currently are the hashes on yugabyte indexes ie:

CREATE TABLE foo(
  id INTEGER,
  bar INTEGER,
  uuid VARCHAR(255),
 PRIMARY KEY ((id, bar) HASH, uuid ASC)
) TABLESPACE blah;

If these are the only ones that we need to maintain then just some specific regex to check for these Yugabyte specific cases.

I would be willing to work with the Yugabyte team to build and maintain a fork perhaps, but it would be helpful to know the process or what might need to change in order to build their symbols.

Yugabyte has their own build of postgres: https://github.com/yugabyte/yugabyte-db/tree/master/src/postgres

I was able to locally build against this but I am not sure if it was working properly because the above SQL code was still not being parsed by PgQuery.parse. I suspect I was missing something.

The steps I took were:

1. Pull Yugabyte master
2. Pull lib_pgquery
3. Modify the libpg_query makefile to point at the Yugabyte version of postgres from the folder I shared above
4. Point the ruby gem (what we use) at the local libpg_query build.

It was quite hacky and was wondering if there might be a more reliable way to link them.