data-engineer-handbook icon indicating copy to clipboard operation
data-engineer-handbook copied to clipboard

Fix: Incorrect table name in `do_team_vertex_transformation`'s `write` operation

Open PrinceSajjadHussain opened this issue 5 months ago • 0 comments

In the team_vertex_job.py file, the main function incorrectly attempts to write the output DataFrame into a table named "players_scd", which is likely meant for the players SCD job. This will cause an error if the table does not exist or write data to the wrong table.

Fix: --- a/bootcamp/materials/3-spark-fundamentals/src/jobs/team_vertex_job.py +++ b/bootcamp/materials/3-spark-fundamentals/src/jobs/team_vertex_job.py @@ -42,4 +42,4 @@ .appName("players_scd")
.getOrCreate() output_df = do_team_vertex_transformation(spark, spark.table("players"))

  • output_df.write.mode("overwrite").insertInto("players_scd")
  • output_df.write.mode("overwrite").insertInto("team_vertex")

PrinceSajjadHussain avatar Jun 20 '25 15:06 PrinceSajjadHussain