pyspark-ai
pyspark-ai copied to clipboard
databrickslabs
English SDK for Apache Spark
Any other model in addition to GPT-4 that I can use with pyspark-ai?
--------------------------------------------------------------------------- ModuleNotFoundError Traceback (most recent call last) Cell In[14], line 2 1 # Activate partial functions for Spark DataFrame ----> 2 spark_ai.activate() File ~/anaconda3/envs/python3/lib/python3.10/site-packages/pyspark_ai/pyspark_ai.py:428, in SparkAI.activate(self) 426 DataFrame.ai = AIUtils(self)...
Hi, could you add a parameter "tablename" for the function create_df()? In same cases a table with an automaticlly detected name cannot be created, but the SQL works with different...
``` from pyspark.sql import SparkSession import pyspark_ai spark = SparkSession.builder.appName('dummy').getOrCreate() data = [("Java", "20000"), ("Python", "100000"), ("Scala", "3000")] columns = ["language","user_cnt"] rdd = spark.sparkContext.parallelize(data) from pyspark.sql.types import StructType, StructField, StringType...
Right now, every DF transformation creates a new temp view and the transformation is applied as a SQL query on top of the temp view. Unfortunately, this creates a lot...
Steps to reproduce: Start a python notebook. !pip install pyspark-ai !export OPENAI_API_KEY='sk-...' from pyspark_ai import * TypeError: dataclass_transform() got an unexpected keyword argument 'field_specifiers'
facing TypeError: dataclass_transform() got an unexpected keyword argument 'field_specifiers' code used : from pyspark_ai import SparkAI spark_ai=SparkAI(verbose=True) spark_ai.activate() Also, Please clarify whether it is pyspark-ai or pyspark_ai to be imported....
Right now, I don't see a way to load data from a saved source (i.e., a table saved in Databricks or snowflake). It would be helpful if this can be...
It is convenient for a developer to ship code in English and the cache data together to optimize speed, save cost, and more importantly reproduce the result. However, we need...
We have been playing around with a tool for a bit, super cool! But is there a way to get the SQL query to help us debug? We noticed that...
