Canner
Support Spark SQL data source
Is your feature request related to a problem? Please describe.
Provides users with the ability to connect and set up Spark SQL as a data source, enabling integration with distributed data processing frameworks directly within Wren AI's connection setup.
Describe the solution you'd like
- Add a
Spark SQLconnector in ibis-server - Implement
Spark SQLconnection handling between ibis-server and Wren UI backend. - Create a UI form for users to set up the
Spark SQLdata source (e.g., server address, port, authentication).
For more details, see Creating a New Data Source Connector in our CONTRIBUTING.md.
Additional context Add any other context or screenshots about the feature request here.
The first step of this issue is implementing the SparkSQL connector on Wren engine side. Wren engine is powered by ibis-project to connect to different data sources.
Refer to ibis pyspark backend, you can follow the guide of adding DataSource to:
- define the connection info
- add the connector
- add test cases
By the way, It would be appreciated if you could also survey how to connect to Databricks using ibis-project. 🙇
@chilijung Since there hasnt been any updates for this issue, I hope you dont mind if i can have a look at it.
@ichuniq Are you working on this? I noticed you submitted a PR for a typo. I think you may be looking into this issue. Please help confirm. If yes, could you also share the correct progress? Thanks.
@Prototype4988 before you start, let me double check and confirm with @ichuniq thanks.
Hi @chilijung, I’ve looked into the task but haven’t started implementing it yet due to limited capacity. However, I’ll have some free time between now and the end of October.
Thank you for the update, @Prototype4988. I will find other issues for you to solve, and I've cc'd you on other ones.
@ichuniq , I look forward to your contribution, and please keep us updated!