Enrico Minack

[SPARK-39931][PYTHON][WIP] Improve applyInPandas performance for very small groups

3

### What changes were proposed in this pull request? Given a batch size to `applyInPandas`, multiple groups are sent to Python UDF at once if they are very small. This...

[SPARK-39876][SQL] Add UNPIVOT to SQL syntax

6

### What changes were proposed in this pull request? This adds UNPIVOT clause to SQL syntax. It follows the same syntax as [BigQuery](https://cloud.google.com/bigquery/docs/reference/standard-sql/query-syntax#unpivot_operator), [T-SQL](https://docs.microsoft.com/en-us/sql/t-sql/queries/from-using-pivot-and-unpivot?view=sql-server-ver15#unpivot-example), [Oracle](https://www.oracletutorial.com/oracle-basics/oracle-unpivot/): ``` FROM ... [ unpivot_clause...

[CI] Upgrade PySpark and Torch

2

Upgrades CI test frameworks PySpark to 3.3.0 and 3.2.2 and torch to 1.12.1. GPU heads test did not test torch head (1.13) but torch 1.12. Added a step to `Dockerfile.test.[cg]pu`...

Allow for elastic tf data service job, overcome tf localhost addresses

2

## Checklist before submitting - [X] Did you read the [contributor guide](https://github.com/horovod/horovod/blob/master/CONTRIBUTING.md)? - [ ] Did you update the docs? - [ ] Did you write any tests to validate...

Elastic torch tests fail

1

The `test_elastic_torch.py` test started to fail on all torch versions across GPU and CPU. In [one build](https://buildkite.com/horovod/horovod/builds/5675), only torchhead GPU failed, on the [next build](https://buildkite.com/horovod/horovod/builds/5689) all torch versions failed. The...

Make Horovod build with OneCCL in GitHub CI

6

Moving the CI to GitHub Workflow, Horovod does not build with OnceCCL anymore. This has to be investigated and resolved here.

Horovod cannot be installed together with ML frameworks with pip>=22.0

5

Since `pip>=22.0`, the following way to install Horovod does not work any more: HOROVOD_WITH_TENSORFLOW=1 pip install tensorflow==2.7.0 horovod[tensorflow] Even though `horovod[tensorflow]` depends on `tensorflow`, pip does not install `tensorflow before...

Ray tests are flaky

4

Ray tests have shown to be flaky, especially with GPU (Buildkite CI). There are two places that cause these flakiness: 1. Some tests fetch `ray.available_resources()` at the beginning of the...

RayExecutor does not work with Horovod Settings

1

The documentation of RayExecutor says `You can use a standard Horovod Settings object` for the `settings` parameter, but this fails with: ``` Traceback (most recent call last): File "bug.py", line...

Is the PyGithub project dead? How can the community help?

16

Over the last months, I have seen many very good pull requests from the community that fix blocking bugs or add new features, as well as many questions around issues...