Nilo

add spark-scala-quickstart

4

Dataproc - Spark Scala Quickstart is an effort to assist in the creation of Spark jobs written in Scala to run on Dataproc. It provides different pre-implemented Spark jobs and...

Sorry, created the pull request before the issue. I can recreate it if needed. My proposed change is to add a spark-scala-quickstart session. It is described in the pull request:...

Allow querying the bigframes dataframe with custom BigQuery SQL like pandasql

4

### **Feature Request** Supposed I used bigframes and processed my data like this: `res_df = df.assign(pred=input_remote_function.apply(generate_text))` And got this output dataframe **res_df**: uri | pred | -------------------------------------------------- | -----------------------------------------------| gs://dataproc-metastore-public-binaries/ads_ba......

Create an integration test to run the notebook programmatically periodically from Jenkins

[Notebook] Create first notebook using Spark Sessions in Vertex AI

2

Create the first implementation of a notebook focusing on the AI/ML use case, leveraging Dataproc Spark Sessions in Vertex AI. It should be a template to facilitate the development of...

Add integration test for the [mysql2spanner](https://github.com/GoogleCloudPlatform/dataproc-templates/tree/main/notebooks/mysql2spanner) notebook. The integration test is run by leveraging the [MySqlToSpanner_parameterize_script](https://github.com/GoogleCloudPlatform/dataproc-templates/blob/main/notebooks/mysql2spanner/MySqlToSpanner_parameterize_script.py), which allows running the notebook as a python script with the parameters specified by...