Remove casts for toDartObject

[srujzs]

ExternalDartReference is now generic, so we can just type it and toDartObject will do the cast for us if needed. Also removes now-stale ignores.

[flutter-dashboard[bot]]

It looks like this pull request may not have tests. Please make sure to add tests before merging. If you need an exemption to this rule, contact "@test-exemption-reviewer" in the #hackers channel in Chat (don't just cc them here, they won't see it! Use Discord!).

If you are not sure if you need tests, consider this rule of thumb: the purpose of a test is to make sure someone doesn't accidentally revert the fix. Ask yourself, is there anything in your PR that you feel it is important we not accidentally revert back to how it was before your fix?

Reviewers: Read the Tree Hygiene page and make sure this patch meets those guidelines before LGTMing.

[srujzs]

Friendly ping @eyebrowsoffire :)

[srujzs]

Thanks!