Clarity regarding dotnet Spark Job definition dependencies for Synapse

[cutecycle]

There are a few things that were unintuitive about the documentation when we initially set up our .NET Spark project.

• We built using 2.1.0, and spent time trying to figure out why the DotNetRunner was timing out.
• We found out we had to use 1.0.0 with our spark pool, and got a SerDe error when attempting to debug locally.

[PRMerger15]

@cutecycle : Thanks for your contribution! The author(s) have been notified to review your proposed change.

[PRMerger20]

@cutecycle : Thanks for your contribution! The author(s) have been notified to review your proposed change.

[PRMerger19]

@cutecycle : Thanks for your contribution! The author(s) have been notified to review your proposed change.

[PRMerger19]

@cutecycle : Thanks for your contribution! The author(s) have been notified to review your proposed change.

[ShannonLeavitt]

@juluczni - If you approve these changes, lines 27-40 must be indented three spaces before this PR can be merged.

#label:"aq-pr-triaged"

[cutecycle]

after going through some local development environment setup, there's a few things worthy of rewriting:

• When setting up a spark pool, the Studio interface shows you what version of .NET Spark you should use. Of course, we started with scala notebooks and had the workspace deploying in ARM for months before prototyping the .NET Spark workflow, so we hadn't seen this one. I think it'd be ideal to call that out in a small compatibility table + screenshot at the top
• I think the dotnet spark example should be project source code, rather than a pre-built executable, though that's out of scope of this PR.

[PRMerger15]

@cutecycle : Thanks for your contribution! The author(s) have been notified to review your proposed change.

[prmerger-automator[bot]]

@cutecycle : Thanks for your contribution! The author(s) have been notified to review your proposed change.

[JasonWHowell]

#assign:SnehaGunda for review

[prmerger-automator[bot]]

The following users don't have valid GitHub IDs, or are not collaborators on this repo: snehagunda for review

[prmerger-automator[bot]]

@cutecycle : Thanks for your contribution! The author(s) have been notified to review your proposed change.

[JasonWHowell]

@SnehaGunda can you check this one? I don't know why but it won't let me assign it to you.

[prmerger-automator[bot]]

@cutecycle : Thanks for your contribution! The author(s) have been notified to review your proposed change.

[SnehaGunda]

@SnehaGunda can you check this one? I don't know why but it won't let me assign it to you.

Yes, I think you missed @ in the assignment.

[SnehaGunda]

@cutecycle thanks for the contribution. @mlee3gsd validated your fixes. Your PR will be merged soon.

[SnehaGunda]

#sign-off

[prmerger-automator[bot]]

Invalid command: '#sign-off'. Only the assigned author of one or more file in this PR can sign off. @juluczni

[SnehaGunda]

@ShannonLeavitt can you please fix the indentation and merge the PR.

[prmerger-automator[bot]]

@ShannonLeavitt : Thanks for your contribution! The author(s) have been notified to review your proposed change.

[prmerger-automator[bot]]

@ShannonLeavitt : Thanks for your contribution! The author(s) have been notified to review your proposed change.

[ShannonLeavitt]

@SnehaGunda - I've fixed the indentation and made other minor edits as you requested. The indentation didn't look right in the GitHub view, so I staged the content in the private repo to confirm and also to take advantage of automated checks.

Can you please address the following issue in a new PR in the private repo?

articles/synapse-analytics/spark/media/apache-spark-job-definitions/net-spark-workspace-compatibility.png

• This new image needs to be moved to a folder dedicated to spark-dotnet.md; it shouldn't be in the apache-spark-job-definitions media folder. Please open a new PR to move the image.

I'll merge now.