evidence
evidence copied to clipboard
Python based pipelines support
Description
This PR adds support for python pipelines that can modify source data and overwrite or create new tables. Docs, changesets, and the supporting @evidence-dev/pipelines package to follow
Usage:
Update connection.yaml to specify one or more pipelines:
pipelines:
- name: Converts all text in comments to uppercase
# Use the comments table
source: comments
# Overwrite the comments table
destination: comments
steps:
# Specify which python files will be run with this data
- uppercaseStrings
- name: Converts all columns to text, then uppercases them
source: comments
# Create a new table instead of overwriting comments
destination: all_upper_comments
steps:
- convertColumnsToStrings
- uppercaseStrings
Add some pipelines to ./pipelines, in these files data is a pandas dataframe that contains the source table for mutation. After the file is run, the updated content of data is written to the destination (defaults to overwriting the input table) source table, updating the schema as needed.
uppsercaseStrings.py
for col in data.select_dtypes(include=[object]):
data[col] = data[col].str.upper()
Checklist
- [ ] ~~For UI or styling changes, I have added a screenshot or gif showing before & after~~
- [ ] I have added a changeset
- [ ] I have added to the docs where applicable
- [ ] ~~I have added to the VS Code extension where applicable~~
⚠️ No Changeset found
Latest commit: b8ad97285302addc40c5c8941e6eb65c2c4703c1
Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.
This PR includes no changesets
When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types
Click here to learn what changesets are, and how to add one.
Click here if you're a maintainer who wants to add a changeset to this PR
The latest updates on your projects. Learn more about Vercel for Git ↗︎
| Name | Status | Preview | Comments | Updated (UTC) |
|---|---|---|---|---|
| docs | ❌ Failed (Inspect) | Aug 28, 2024 2:56pm | ||
| next-docs | ❌ Failed (Inspect) | Aug 28, 2024 2:56pm |
Deploy Preview for evidence-development-workspace failed. Why did it fail? →
| Name | Link |
|---|---|
| Latest commit | b8ad97285302addc40c5c8941e6eb65c2c4703c1 |
| Latest deploy log | https://app.netlify.com/sites/evidence-development-workspace/deploys/66cf3a8f951d9a000923235c |
Deploy Preview for evidence-test-env failed. Why did it fail? →
| Name | Link |
|---|---|
| Latest commit | b8ad97285302addc40c5c8941e6eb65c2c4703c1 |
| Latest deploy log | https://app.netlify.com/sites/evidence-test-env/deploys/66cf3a8fbf5b4d00088d9c67 |
Deploy Preview for next-docs-evidence failed. Why did it fail? →
| Name | Link |
|---|---|
| Latest commit | b8ad97285302addc40c5c8941e6eb65c2c4703c1 |
| Latest deploy log | https://app.netlify.com/sites/next-docs-evidence/deploys/66cf3a8f9a091d0008160803 |