pyspark-ai
pyspark-ai copied to clipboard
databrickslabs
English SDK for Apache Spark
Close #142 Add all notebooks as pages to the website. That's how it looks like:  1. I moved notebooks from `examples` to `docs/examples` 2. I added `poetry` group `mkdocs`...
Some project dependencies should be flagged as `dev-dependencies`. We want the fewest dependencies possible when this project is pip installed. A quinn user [recently informed me](https://github.com/MrPowers/quinn/issues/124) that dev-dependencies are being...
Currently the code is tested only against regular `spark.sql.SparkSession`. I can extend tests to `spark.sql.connect.SparkSession` too; there is some difference in returned types and possible that they won't pass some...
I can create templates for different kinds of issues: 1. Feature-request or proposal 2. Databricks-related bug (in such a case we need to ask user about DBR version too) 3....
Currently, there are no API documents. We should start creating one
The Spark job failed due to a Python worker crashing unexpectedly. The root cause is likely a java.io.EOFException, indicating an unexpected end of file or communication issue.
I'm thinking about how to use this in my business to bring people into databricks instead of using low code platforms that are more difficult to support. One limitation I...
Improves #138 - Removes .flake8 - Adds pyproject toml ruff with bandit security scan settings. - Other rules available. https://docs.astral.sh/ruff/rules/ - Update the lint workflow and makefile Requires some GitHub...
It looks like it is possible to add Jupyter notebooks as part of documentation with the following extension: https://pypi.org/project/mkdocs-jupyter/
Create a new class for all the utility method related to Spark, for example, `_get_dataframe_results`, `_get_df_schema`, etc. The refactoring can simplify the code in `pyspark_ai.py`.