dagster icon indicating copy to clipboard operation
dagster copied to clipboard

Published 20 hours ago • verified publisher icon dagster-io

[external assets] source assets -> external assets

Open smackesey opened this issue 1 year ago • 3 comments

Summary & Motivation

Internally convert source assets to external assets. This happens at the level of RepositoryData-- we do the conversion here instead of in Definitions so that conversion works for both @repository and Definitions.

"Source assets" no longer exist as a concept at the level of ExternalAssetNode, but still exist over the GQL API to keep the representation in Dagster UI constant. A new isExternal field is also added to GQL asset nodes.

A caveat is that auto_observe_interval_minutes can not be perfectly translated to an auto-materialize policy, because there is no way to capture "every N minutes" exactly as a cron expression (and AutoMaterializeRule.materialize_on_cron is the closest option available). I used a mapping that gets us pretty close though where we round to the nearest minute/hour/day for the number of minutes.

How I Tested These Changes

smackesey avatar Nov 22 '23 15:11 smackesey

[!WARNING] This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite. Learn more

  • #18217 Graphite 👈
  • #19702 Graphite: 1 other dependent PR (#19741 Graphite)
  • #19685 Graphite
  • #19684 Graphite
  • #19774 Graphite
  • master

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @smackesey and the rest of your teammates on Graphite Graphite

smackesey avatar Nov 22 '23 15:11 smackesey

Deploy preview for dagit-core-storybook ready!

✅ Preview https://dagit-core-storybook-9n1yxcik2-elementl.vercel.app https://sean-convert-source-assets.core-storybook.dagster-docs.io

Built with commit 5ebeac7dbb960abba62bb484e50ae0588f846077. This pull request is being automatically deployed with vercel-action

github-actions[bot] avatar Nov 22 '23 15:11 github-actions[bot]

Deploy preview for dagster-docs ready!

Preview available at https://dagster-docs-ao3tft7y7-elementl.vercel.app https://sean-convert-source-assets.dagster.dagster-docs.io

Direct link to changed pages:

  • https://dagster-docs-ao3tft7y7-elementl.vercel.app https://sean-convert-source-assets.dagster.dagster-docs.io/concepts/partitions-schedules-sensors/sensors
  • https://dagster-docs-ao3tft7y7-elementl.vercel.app https://sean-convert-source-assets.dagster.dagster-docs.io/dagster-cloud/account/authentication
  • https://dagster-docs-ao3tft7y7-elementl.vercel.app https://sean-convert-source-assets.dagster.dagster-docs.io/dagster-cloud/account/authentication/okta/saml-sso
  • https://dagster-docs-ao3tft7y7-elementl.vercel.app https://sean-convert-source-assets.dagster.dagster-docs.io/dagster-cloud/account/authentication/okta/scim-provisioning
  • https://dagster-docs-ao3tft7y7-elementl.vercel.app https://sean-convert-source-assets.dagster.dagster-docs.io/dagster-cloud/account/authentication/setting-up-azure-ad-saml-sso
  • https://dagster-docs-ao3tft7y7-elementl.vercel.app https://sean-convert-source-assets.dagster.dagster-docs.io/dagster-cloud/account/authentication/setting-up-google-workspace-saml-sso
  • https://dagster-docs-ao3tft7y7-elementl.vercel.app https://sean-convert-source-assets.dagster.dagster-docs.io/dagster-cloud/account/authentication/setting-up-onelogin-saml-sso
  • https://dagster-docs-ao3tft7y7-elementl.vercel.app https://sean-convert-source-assets.dagster.dagster-docs.io/dagster-cloud/account/authentication/setting-up-pingone-saml-sso
  • https://dagster-docs-ao3tft7y7-elementl.vercel.app https://sean-convert-source-assets.dagster.dagster-docs.io/dagster-cloud/account/authentication/utilizing-scim-provisioning
  • https://dagster-docs-ao3tft7y7-elementl.vercel.app https://sean-convert-source-assets.dagster.dagster-docs.io/dagster-cloud/account/managing-users
  • https://dagster-docs-ao3tft7y7-elementl.vercel.app https://sean-convert-source-assets.dagster.dagster-docs.io/dagster-cloud/account/managing-users/managing-teams
  • https://dagster-docs-ao3tft7y7-elementl.vercel.app https://sean-convert-source-assets.dagster.dagster-docs.io/dagster-cloud/account/managing-users/managing-user-roles-permissions
  • https://dagster-docs-ao3tft7y7-elementl.vercel.app https://sean-convert-source-assets.dagster.dagster-docs.io/dagster-cloud/deployment/agents/running-multiple-agents
  • https://dagster-docs-ao3tft7y7-elementl.vercel.app https://sean-convert-source-assets.dagster.dagster-docs.io/dagster-cloud/deployment/serverless
  • https://dagster-docs-ao3tft7y7-elementl.vercel.app https://sean-convert-source-assets.dagster.dagster-docs.io/dagster-cloud/managing-deployments/branch-deployments
  • https://dagster-docs-ao3tft7y7-elementl.vercel.app https://sean-convert-source-assets.dagster.dagster-docs.io/dagster-cloud/managing-deployments/managing-deployments
  • https://dagster-docs-ao3tft7y7-elementl.vercel.app https://sean-convert-source-assets.dagster.dagster-docs.io/integrations/embedded-elt

github-actions[bot] avatar Feb 07 '24 12:02 github-actions[bot]

Superseded by another stack.

smackesey avatar Mar 20 '24 16:03 smackesey