flink icon indicating copy to clipboard operation
flink copied to clipboard

[FLINK-37913][table] Add built-in OBJECT_OF function

Open raminqaf opened this issue 6 months ago • 1 comments

What is the purpose of the change

This pull request implements the OBJECT_OF function as specified in FLIP-520 to enable creation of structured types in SQL and Table API. The function allows users to construct structured objects from key-value pairs without requiring UDFs, making structured type creation SQL-serializable and improving the overall usability of structured types in Flink.

Brief change log

  • Added OBJECT_OF function definition to BuiltInFunctionDefinitions
  • Implemented ObjectOfInputTypeStrategy for input validation (odd argument count, string keys, unique field names)
  • Implemented ObjectOfTypeStrategy for output type inference (creates structured types)
  • Added ObjectOfFunction runtime implementation that creates RowData from key-value pairs
  • Enhanced Expressions.java with objectOf() methods for Table API support
  • Added Python API support with object_of() function in expressions.py
  • Updated SQL functions documentation with OBJECT_OF function details

Verifying this change

This change added tests and can be verified as follows:

  • Added unit tests for ObjectOfInputTypeStrategy to validate argument count, type validation, and field name uniqueness
  • Added integration tests in StructuredFunctionsITCase to test end-to-end functionality
  • Verified SQL usage: SELECT OBJECT_OF('com.example.User', 'name', 'Bob', 'age', 42)
  • Verified Table API usage: objectOf(User.class, "name", "Bob", "age", 42)

Does this pull request potentially affect one of the following parts:

  • Dependencies (does it add or upgrade a dependency): no
  • The public API, i.e., is any changed class annotated with @Public(Evolving): yes (additions to Expressions.java and BuiltInFunctionDefinitions)
  • The serializers: no
  • The runtime per-record code paths (performance sensitive): no
  • Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
  • The S3 file system connector: no

Documentation

  • Does this pull request introduce a new feature? yes
  • If yes, how is the feature documented? docs / JavaDocs (updated SQL functions documentation, comprehensive JavaDocs for all new classes and methods, Python API documentation)

raminqaf avatar Jun 19 '25 16:06 raminqaf

CI report:

  • a49b6ada6d0a5009347d2a8290c66a8d7078def9 Azure: SUCCESS
Bot commands The @flinkbot bot supports the following commands:
  • @flinkbot run azure re-run the last Azure build

flinkbot avatar Jun 19 '25 16:06 flinkbot

I think there's a checkstyle violation making the CI fail. Make sure to run ./mvnw checkstyle:check locally

gustavodemorais avatar Jun 19 '25 18:06 gustavodemorais

@twalthr There is also another case that I have prepared, but so far, it has been stashed on my branch. In the FLIP-520, you mentioned under the table API, this signature:

Expressions.objectOf(DataType, Object... kv);

In SQL, this would be equivalent to:

SELECT OBJECT_OF(OBJECT_OF(...)));

Is this a case we should also support in SQL? Does this mean that we can construct structure types from structured types?

raminqaf avatar Jun 21 '25 18:06 raminqaf