Thomas Newton

Signed-off-by: Thomas Newton ## Description Run doctoc to auto-update the table of contents. This PR was repurposed after what it was originally intended to document was discovered to be a...

# Description When reading the struct column stats (as opposed to json column stats) the null counts where ignored for struct type columns. Completely ignoring is probably less bad than...

`DeltaTable.to_pyarrow_dataset()` fails for tables containing map types

4

# Environment **Delta-rs version**: 0.5.8 **Binding**: Python **Environment**: - **Cloud provider**: Azure - **OS**: Ubuntu 18.04 - **Other**: Python 3.8 *** # Bug **What happened**: When using `DeltaTable.to_pyarrow_dataset()` for a...

[Core feature] Compress workflow state when storing it in etcd

1

### Motivation: Why do you think this is important? Same motivation as https://github.com/flyteorg/flyte/issues/4569 In my mind the bigger we can make our workflows before hitting etcd size limits the better....

[Core feature] Create or document a fast way to query the status of large workflows

10

### Motivation: Why do you think this is important? For large workflows with ~1000s of nodes its difficult to monitor the overall progress. At this scale we need to be...

Clear past errors from workflow state

4

## Tracking issue https://github.com/flyteorg/flyte/issues/4569 ## Why are the changes needed? Reduce un-needed information stored in etcd when using `failure_policy=WorkflowFailurePolicy.FAIL_AFTER_EXECUTABLE_NODES_COMPLETE`. This allows flyte to scale to larger workflows before hitting etcd...

GH-39968: [Python][FS][Azure] Minimal Python bindings for `AzureFileSystem`

35

### Rationale for this change We want to use the new `AzureFileSystem` in `pyarrow`. ### What changes are included in this PR? - Add minimal python bindings for `AzureFileSystem`. This...

For my usecase I generally have a full path including the protocol like `abfss://@.dfs.core.windows.net/`. I've found that different methods of `AzureBlobFileSystem` seem to handle the protocol information in different ways....

Grafana dashboard updates

8

## Tracking issue ## Why are the changes needed? There are a few issues with the dashboards as is. This includes some bugs and some outdated metric names. ## What...

[Feature Request] More performant transaction log parsing for Azure

12

## Feature request ### Overview There is room for performance improvement when parsing delta transaction logs to get the latest state of a delta table stored on Azure blob storage...