drill icon indicating copy to clipboard operation
drill copied to clipboard

Published 20 hours ago • verified publisher icon apache

Feature for Daffodil DFDL Data querying

Open mbeckerle opened this issue 1 year ago • 2 comments

Use DFDL language to describe data, and then enable Drill to query that data immediately by way of Apache Daffodil's DFDL implementation.

(Creating this so I have a ticket number to cite in commits/PRs)

mbeckerle avatar Oct 11 '23 12:10 mbeckerle

Very excited to see this! DFDL + Drill is really a great combo.

cgivre avatar Oct 11 '23 13:10 cgivre

PR updated: https://github.com/apache/drill/pull/2836

  • Metadata bridge working,
  • data bridge working.
  • simple types still tbd are: unsigned integers, decimal, date/time/dateTime

Not done: Distribution of schemas (or compiled schemas) across drill's computation fabric.

Testing of nillability, and real queries against realistic DFDL schemas are needed.

mbeckerle avatar Dec 22 '23 01:12 mbeckerle