evidence icon indicating copy to clipboard operation
evidence copied to clipboard

Python based pipelines support

Open ItsMeBrianD opened this issue 1 year ago • 5 comments

Description

This PR adds support for python pipelines that can modify source data and overwrite or create new tables. Docs, changesets, and the supporting @evidence-dev/pipelines package to follow

Usage:

Update connection.yaml to specify one or more pipelines:

pipelines:
  - name: Converts all text in comments to uppercase
    # Use the comments table
    source: comments
    # Overwrite the comments table
    destination: comments
    steps:
      # Specify which python files will be run with this data
      - uppercaseStrings
  
  - name: Converts all columns to text, then uppercases them
    source: comments
    # Create a new table instead of overwriting comments
    destination: all_upper_comments
    steps:
      - convertColumnsToStrings
      - uppercaseStrings

Add some pipelines to ./pipelines, in these files data is a pandas dataframe that contains the source table for mutation. After the file is run, the updated content of data is written to the destination (defaults to overwriting the input table) source table, updating the schema as needed.

uppsercaseStrings.py

for col in data.select_dtypes(include=[object]):
    data[col] = data[col].str.upper()

Checklist

  • [ ] ~~For UI or styling changes, I have added a screenshot or gif showing before & after~~
  • [ ] I have added a changeset
  • [ ] I have added to the docs where applicable
  • [ ] ~~I have added to the VS Code extension where applicable~~

ItsMeBrianD avatar Aug 28 '24 14:08 ItsMeBrianD

⚠️ No Changeset found

Latest commit: b8ad97285302addc40c5c8941e6eb65c2c4703c1

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

changeset-bot[bot] avatar Aug 28 '24 14:08 changeset-bot[bot]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name Status Preview Comments Updated (UTC)
docs ❌ Failed (Inspect) Aug 28, 2024 2:56pm
next-docs ❌ Failed (Inspect) Aug 28, 2024 2:56pm

vercel[bot] avatar Aug 28 '24 14:08 vercel[bot]

Deploy Preview for evidence-development-workspace failed. Why did it fail? →

Name Link
Latest commit b8ad97285302addc40c5c8941e6eb65c2c4703c1
Latest deploy log https://app.netlify.com/sites/evidence-development-workspace/deploys/66cf3a8f951d9a000923235c

netlify[bot] avatar Aug 28 '24 14:08 netlify[bot]

Deploy Preview for evidence-test-env failed. Why did it fail? →

Name Link
Latest commit b8ad97285302addc40c5c8941e6eb65c2c4703c1
Latest deploy log https://app.netlify.com/sites/evidence-test-env/deploys/66cf3a8fbf5b4d00088d9c67

netlify[bot] avatar Aug 28 '24 14:08 netlify[bot]

Deploy Preview for next-docs-evidence failed. Why did it fail? →

Name Link
Latest commit b8ad97285302addc40c5c8941e6eb65c2c4703c1
Latest deploy log https://app.netlify.com/sites/next-docs-evidence/deploys/66cf3a8f9a091d0008160803

netlify[bot] avatar Aug 28 '24 14:08 netlify[bot]