spark

spark copied to clipboard

### What changes were proposed in this pull request? Enhance method verifyNotReadPath to identify subqueries hidden in the filter conditions. ### Why are the changes needed? SparkSQL will throw exception...

xunxunmimi5577

[SPARK-49944][DOCS] Fix broken main.js import and fix image links for streaming documentation

1

### What changes were proposed in this pull request? We use the `rel_path_to_root` Jekyll variable in front of all paths that require it. ### Why are the changes needed? Currently,...

neilramaswamy

[SPARK-50035][SS] Add support for explicit handleExpiredTimer function part of the stateful processor

1

### What changes were proposed in this pull request? Add support for explicit handleExpiredTimer function part of the stateful processor ### Why are the changes needed? Separate function will provide...

anishshri-db

[SPARK-49923][core]Add a configuration item for disabling the setting of javaOptions.

2

What changes were proposed in this pull request? The pr is add AVA_OPTION_PURE_MODE configuration item. When the configuration item is set to true, JavaOptions cannot be configured. Why are the...

cliffchen12

### What changes were proposed in this pull request? - Support deleteIfExists for TransformWithStateInPandas. - Added `close()` support for StatefulProcessor. ### Why are the changes needed? Add parity to TransformWithStateInPandas...

bogao007

### What changes were proposed in this pull request? Currently, we use UnsafeRow to store all state in the StateStores. This change will add StateStore APIs that support byte array...

ericm-db

### What changes were proposed in this pull request? This PR aims to upgrade - `grpcio` and `grpcio-status` to `1.67.0` - `grpc-java` to `1.67.1` Please note that `grpc-java` there is...

dongjoon-hyun

### What changes were proposed in this pull request? `RewriteCollationJoin` should not be an excludable rule any more. ### Why are the changes needed? Enable hash join support for all...

uros-db

### What changes were proposed in this pull request? In this PR collations can now be identified by their fully qualified name, as per the collation project plan. The `Collation`...

stevomitric

[SPARK-49461][SS] Persistent Checkpoint ID to commit logs and read it back

1

### What changes were proposed in this pull request? In this change, we propose to add a new field to the commit log if `STREAMING_STATE_STORE_COMMIT_LOG_VERSION` is 2. The new field...

WweiL