Tdml gen app

[arosien]

trafficstars

[NolanMatt]

@arosien I updated it to only apply to the daffodil-tdml-lib portion of the build, but still getting errors, my new thought is that on the xjcCommandLine it's referencing org.apache.daffodil.tdml which seems to me like a bit of a circular reference that might not resolve since this is the project we're building it in. I'm not sure though, just seems a bit off to me. Any ideas?

[tuxji]

I saw that this PR needed a maintainer's approval to run CI on it, so I looked over the changes and let CI run but unfortunately the changes don't seem to work on GitHub runner machines. Some runs printed their error messages in foreign languages (I wonder how that happened?), but one run's English errors seem to indicate that DFDL additions to XML Schema confuse xjc. Did you run xjc successfully on your local machine?

[stevedlawrence]

Some runs printed their error messages in foreign languages (I wonder how that happened?)

For a while we had a few tests that expected error messages that were generated and internationalized by the OS (e.g. "could not open file"). But the tests expected english error messages so would fail on systems that defaulted to another language. These tests were fixed to not look for these specific error messages, but to avoid new tests that might do something similar, a couple of github action jobs were updated to run with LANG set to de_DE and ja_JP.

https://github.com/apache/daffodil/blob/main/.github/workflows/main.yml#L61-L68

[arosien]

FYI the SbtXjcPlugin unfortunately installs itself globally, so it applies to all projects that have xsd schemas in them, hence the need for the disablePlugins(SbtXjcPlugin) expressions sprinkled around everywhere except the tdml-lib project.

[michael-hoke]

@mbeckerle @stevedlawrence

I refactored the local parts of the TDML schema into tdml-core.xsd. Other than what I showed earlier in the patch file, the only other notable change was to replace the choice in testSuite with a substitutionGroup.

Everything appears to have been properly separated - the generated JAXB classes don't allow the internal schema/configs in testSuite, but the compile/test process is still working for me locally.

I will squash once we decide whether this is a direction we want to continue in.

[michael-hoke]

One more (hopefully, the last) push. It's been squashed and should pass the formatting check as well.

This also is tracked in JIRA - DAFFODIL-2830.

[arosien]

Yay, all tests pass @NolanMatt. Now on to linting, etc.

[stevedlawrence]

With the changes to scalaxb, is this ready for re-review?

[arosien]

With the changes to scalaxb, is this ready for re-review?

Yes! See some infoset translation and here is the top-level test suite:

case class TestSuite(var testSuiteChoices: Seq[scalaxb.DataRecord[Any]] = Nil,
  var attributes: Map[String, scalaxb.DataRecord[Any]] = Map.empty) {
  def suiteName = attributes.get("@suiteName") map { _.as[String]}
  def suiteName_=(_value: Option[String])(implicit evidence: scalaxb.CanWriteXML[Option[String]]) = attributes += "@suiteName" -> scalaxb.DataRecord(_value)
  def ID = attributes.get("@ID") map { _.as[String]}
  def ID_=(_value: Option[String])(implicit evidence: scalaxb.CanWriteXML[Option[String]]) = attributes += "@ID" -> scalaxb.DataRecord(_value)
  def description = attributes.get("@description") map { _.as[String]}
  def description_=(_value: Option[String])(implicit evidence: scalaxb.CanWriteXML[Option[String]]) = attributes += "@description" -> scalaxb.DataRecord(_value)
  def defaultRoundTrip = attributes.get("@defaultRoundTrip") map { _.as[RoundTripType]}
  def defaultRoundTrip_=(_value: Option[RoundTripType])(implicit evidence: scalaxb.CanWriteXML[Option[RoundTripType]]) = attributes += "@defaultRoundTrip" -> scalaxb.DataRecord(_value)
  def defaultValidation = attributes.get("@defaultValidation") map { _.as[ValidationType]}
  def defaultValidation_=(_value: Option[ValidationType])(implicit evidence: scalaxb.CanWriteXML[Option[ValidationType]]) = attributes += "@defaultValidation" -> scalaxb.DataRecord(_value)
  def defaultConfig = attributes.get("@defaultConfig") map { _.as[String]}
  def defaultConfig_=(_value: Option[String])(implicit evidence: scalaxb.CanWriteXML[Option[String]]) = attributes += "@defaultConfig" -> scalaxb.DataRecord(_value)
  def defaultImplementations = attributes.get("@defaultImplementations") map { _.as[Seq[ImplementationItem]]}
  def defaultImplementations_=(_value: Option[Seq[ImplementationItem]])(implicit evidence: scalaxb.CanWriteXML[Option[Seq[ImplementationItem]]]) = attributes += "@defaultImplementations" -> scalaxb.DataRecord(_value)
}

Users of these data structures will need to build up, for example, ParserTestCaseType values, serialize them via scalaxb.toXML, and push them into the testSuiteChoices member.

[stevedlawrence]

Can you please rebase to the latest main branch and resolve the conflicts?

[arosien]

I think this could just use a bit more extensive test, and maybe some license questions?

Also, it looks like you squashed into a single commit, but it still needs to be rebased to fix conflicts.

Also, can you update the commit message to include a bug reference, right now it has a broken link. I think DAFFODIL-2830 is the right bug?

I don't know why it says build.sbt has conflicts, locally I don't see them. I'll poke around more.

Will update the bug reference.

[shanedell]

@stevedlawrence I know there was a discuss out for releasing Daffodil 3.7.0. Would this possibly be a PR we could get in before the actual release or where do you stand on what still needs done for this to meet requirements?

[stevedlawrence]

@stevedlawrence I know there was a discuss out for releasing Daffodil 3.7.0. Would this possibly be a PR we could get in before the actual release or where do you stand on what still needs done for this to meet requirements?

Based on @arosien's comments, it sounds like there are still some open issues to resolve with this PR? There's also merge conflicts and a potential licensing issue to figure out. So it might be best to push to the next release. Were thinking we might try to get 3.8.0 out quicker than usual with the new layer API. We could try to get this in that release so you wouldn't have to wait too long.

[mbeckerle]

Since I'm re-reading this ticket.... I had a thought. Rather than making the VSCode read/write TDML, which we can see using the scalaxb approach, is hard to make compatible with human authored stuff like CDATA bracketing, comments, etc,, why don't we define this to be different from HA (Human Authored) TDML, can explicitly call it MG (Machine Generated) TDML. This is really just a shift in perspective, but we're giving ourselves the freedom to simply be incompatible with Human-Authored TDML, and we can add features like an explicit 'tdml:comment.' element to overcome the XML-Comments issue.

[michael-hoke]

Since I'm re-reading this ticket.... I had a thought. Rather than making the VSCode read/write TDML, which we can see using the scalaxb approach, is hard to make compatible with human authored stuff like CDATA bracketing, comments, etc,, why don't we define this to be different from HA (Human Authored) TDML, can explicitly call it MG (Machine Generated) TDML. This is really just a shift in perspective, but we're giving ourselves the freedom to simply be incompatible with Human-Authored TDML, and we can add features like an explicit 'tdml:comment.' element to overcome the XML-Comments issue.

This is more or less how it is now. Our use cases currently assume MG TDML files, and we've been testing with MG files. I think we've been trying to get basic parsing working with HA files because, in theory, we should be able to load both in. We should be able append/execute any TDML as well, but, to my knowledge, we haven't tested with that yet.

[arosien]

Closing for multiple reasons:

• TDML XML object mapping support isn't used, or can be avoided, in downstream Scala projects like daffodil-vscode. See above for more context.
• The original JAXB solution was deemed to have too particular build configuration given differences between JVM versions, so scalaxb was chosen instead. But since JVM 21 was introduced it also appears to have JVM version dependent build configuration, which is slightly annoying. It's not a blocker, however.
• Given the oddities of XML object binding, the need for an alternative was proposed above, say, JSON, which may fit more downstream languages like Scala and Typescript.