Use of libpg_query on a slightly modified postgres.

[merqurio]

Hi Lukas, Thanks for this awesome job, it's really impressive.

I'm trying to compile libpg_query against a slightly modified Postgres (A couple of KEYWORDS and syntax, works as expected in a normal Postgres compilation). I'm trying to understand your process to extract the .c and .h files following the Makefile, but I'm pretty sure I'm missing something.

Can you give a two line description of the process ? I'm aware I might need to add more files to the blacklist and mock a couple of things but I want to be sure I'm following the same process you followed.

Thanks @lfittl !!

[lfittl]

@merqurio The most important part is the extract_source script: https://github.com/lfittl/libpg_query/blob/10-latest/scripts/extract_source.rb

Typically what I would do for a new Postgres version is to run that and then work through any compilation bugs first (followed by changes in node structure, which impact query fingerprints). It does require a bit of iteration, and the script isn't too well documented - so please do reach out as you try :)

I'm also curious what the slightly modified Postgres is that you are working with. Is it a self-built fork, or something that I may be familiar with? (EDB, Greenplum, etc)

[merqurio]

Hi @lfittl ,

I got over all the parser extraction errors and now I'm checking the nodes that are missing or different. I'm working with this fork.

Is there any trick I might need to know about the node structure/JSON generation that comes to your mind ? I've seen that the other scripts might help in this direction.

PS: I have documented the script and the process, so expect a PR once I clean up everything that is not relevant :)

Thanks !

[merqurio]

OK, so far I think I have handled to follow generate_fingerprint_outfuncs.rb and I think that I might have found the source of the error.

The fingerprint functions are accessing the Query structs members, but some of the members in this case are a struct, a situation not handled in the script.

I'll keep you up to date, if you're interested :)

[lfittl]

@merqurio Interesting - yes, please feel free to keep updating this issue, if we can adjust the scripts slightly to accommodate this case, that would be fine I think.

[merqurio]

So far:

• If I don't run generate_fingerprint_outfuncs.rb, generate_fingerprint_tests.rb, generate_json_outfuncs.rb everything compiles and I get almost the expected result when running the tests. But no openCypher output json is generated, although RawStmt is recognized in cypher queries as well as cypher subqueries. I understand that as a win, but I'm not completely sure. For example:
  • SELECT * FROM (MATCH (a) RETURN a) as t; is recognized as:

  [
    {
      "RawStmt": {
        "stmt": {
          "SelectStmt": {
            "targetList": [
              {
                "ResTarget": {
                  "val": {
                    "ColumnRef": {
                      "fields": [
                        {
                          "A_Star": {}
                        }
                      ],
                      "location": 7
                    }
                  },
                  "location": 7
                }
              }
            ],
            "fromClause": [
              {
                "RangeSubselect": {
                  "subquery": { CYPHER QUERY WOULD BE HERE, right now is empty },
                  "alias": {
                    "Alias": {
                      "aliasname": "t"
                    }
                  }
                }
              }
            ],
            "op": 0
          }
        },
        "stmt_len": 39
      }
    }
  ]
  

• Now that I understand the role of fingerprints better, I'm ignoring them and focusing on getting everything compiled after running generate_json_outfuncs.rb.

Does this focus on generate_json_outfuncs.rb make sense to you ?

[merqurio]

Another update!!

I got some JSON output from Cypher queries, I'm missing a lot of things, but at least I have completed the whole cycle starting from a different source code to compile an example using the modified libpg_query.

MATCH (tom:Person)-[:ACTED_IN]->(movie1)<-[:ACTED_IN]-(coActor:Person) 
RETURN coCoActor.name

[
  {
    "RawStmt": {
      "stmt": {
        "CypherStmt": {
          "last": {
            "CypherClause": {
              "detail": {
                "CypherProjection": {
                  "kind": 0
                }
              },
              "prev": {
                "CypherClause": {
                  "detail": {
                    "CypherMatchClause": {}
                  }
                }
              }
            }
          }
        }
      }
    }
  }
]

[lfittl]

@merqurio Nice, well done!

Very happy to see that this actually worked :)

[merqurio]

I think I managed to handle all the diferences to compile correctly json outfuncs and all the outputs tested so far seem to be correct ! Some tests are still not passing, some due to the usage of "?" mark, others due to "$1" others I'm not so sure why, I'm still investigating.

I would really love to give back some of the work I did and also contribute back and help with the maintenance of the library If you're ok with it.

I'm not sure how but maybe we can chat about it ?

Thanks again for this @lfittl !!!!

[lfittl]

@merqurio Good question - do you have a copy of your work available somewhere?

It probably makes sense to keep the generated source tree separate (as that would differ between agensgraph and Postgres), but we could attempt to have one unified Ruby script that supports extracting the parser for both. Happy to work on that and make it a separate project.

[merqurio]

I completely agree that merging the resulting source code makes no sense. I think that a command in the Makefile and the ruby lines would be enough. That way is up to you developer running the makefile what to do. The ruby lines should not interfere at all.

If you like that approach, let me clean up thing, comment and open a PR 👍

[lfittl]

@merqurio I think that sounds good - if you could open a PR that'd be perfect :)