langchain icon indicating copy to clipboard operation
langchain copied to clipboard

Published 20 hours ago • verified publisher icon langchain-ai

Add Spark SQL support

Open gengliangwang opened this issue 2 years ago • 2 comments

Add Spark SQL support

  • Add Spark SQL support, based on SQL Database agent. It can connect to Spark via building a local/remote SparkSession.
  • Include a notebook example

I tried some complicated queries (window function, table joins), and the tool works well. Compared to the Spark Dataframe agent, this tool is able to generate queries across multiple tables.

gengliangwang avatar May 09 '23 05:05 gengliangwang

cc @hwchase17 @vowelparrot @skcoirz

gengliangwang avatar May 09 '23 05:05 gengliangwang

Hi @dev2049, I have addressed your comments. Could you take another look?

gengliangwang avatar May 11 '23 02:05 gengliangwang

Created a new PR which doesn't inherit from SQLDatabase: https://github.com/hwchase17/langchain/pull/4602 I am closing this one for now.

gengliangwang avatar May 12 '23 22:05 gengliangwang