data-engineer-handbook
data-engineer-handbook copied to clipboard
Fix: Incorrect table name in `do_team_vertex_transformation`'s `write` operation
In the team_vertex_job.py file, the main function incorrectly attempts to write the output DataFrame into a table named "players_scd", which is likely meant for the players SCD job. This will cause an error if the table does not exist or write data to the wrong table.
Fix:
--- a/bootcamp/materials/3-spark-fundamentals/src/jobs/team_vertex_job.py
+++ b/bootcamp/materials/3-spark-fundamentals/src/jobs/team_vertex_job.py
@@ -42,4 +42,4 @@
.appName("players_scd")
.getOrCreate()
output_df = do_team_vertex_transformation(spark, spark.table("players"))
- output_df.write.mode("overwrite").insertInto("players_scd")
- output_df.write.mode("overwrite").insertInto("team_vertex")