pipedream
pipedream copied to clipboard
PipedreamHQ
Loops in workflows
I'd like to run a set of steps or actions within a loop construct. For example, I might receive an array of 100 events in my HTTP payload, and I need to run an action on each. I can do this with code, but I have no way to loop through the elements of this array using Pipedream's actions. I'd like a "loop" action that can handle this use case.
As a temporary workaround, can you please add the ability to duplicate an existing step in a workflow? So, if I want to do something ten times, I can create the action, duplicate it nine times, and then just modify the parameters in the duplicate actions.
This functionality is a must! If a step receives an array, it should simply run the same step in the sequence for X times.
What is the status on this? It's a killer for when moving from Make.
@alexanderahlsen This is on the backlog for 2022! We're working on a GitHub integration now, and plan to work on control flow (loops, conditionals, etc.) soon after!
Here's a temporary workaround by utilizing two different workflows.
The first workflow is the fetching workflow. It grabs this array of records from Airtable, Google Sheets, Quickbooks, or some database or API.
The second workflow is the processing workflow. It's responsible for processing a single record at a time, so that way you can use all of the pre-built actions Pipedream has to offer.
How to create a processing workflow
First, create a new workflow with an HTTP trigger:
In this case, the HTTP endpoint for our processing workflow is https://eo2b9ksk3fy3nrb.m.pipedream.net.
How to connect the fetching workflow to the processing workflow
In your fetching workflow after the fetching step, whether that be a trigger or an action add a new Node.js code step and paste in this snippet:
import { axios } from '@pipedream/platform';
// To use previous step data, pass the `steps` object to the run() function
export default defineComponent({
props: {
records: {
type: "any",
label: "Records to loop",
description: "The array of records to send to processing workflow",
},
workflow_url: {
type: "string",
label: "Processing workflow URL",
description: "The HTTP endpoint to connect to that's processing single individual records from this workflow"
}
},
async run({ steps, $ }) {
for await(const record of this.records) {
await axios($, {
url: this.workflow_url,
method: 'POST',
data: record
})
}
},
})
Then populate the Records prop with the array of records you need processed, and paste in the URL endpoint to your processing workflow. In our example above, the URL would be https://eo2b9ksk3fy3nrb.m.pipedream.net.
After clicking Test in your fetching workflow, you should see individual HTTP requests hitting your processing workflow with each record individual in the body of the request.
Now you can process each record individually, and leverage the hundreds of pre-built actions in your processing workflow.
Hey ctrlaltdylan, you made my day! This is an excellent solution to the missing iterator feature in pipedream! Thank you! Johannes
I ran into an issue where my API came back as an object instead of an array. This modified code should handle single values relatively gracefully 🤞 @ctrlaltdylan
import { axios } from '@pipedream/platform';
// To use previous step data, pass the `steps` object to the run() function
export default defineComponent({
props: {
records: {
type: "any",
label: "Records to loop",
description: "The array of records to send to processing workflow",
},
workflow_url: {
type: "string",
label: "Processing workflow URL",
description: "The HTTP endpoint to connect to that's processing single individual records from this workflow"
}
},
async run({ steps, $ }) {
// Break this if a single value (Array | Object)
if (Array.isArray(this.records)) {
for await(const record of this.records) {
await axios($, {
url: this.workflow_url,
method: 'POST',
data: record
})
}
} else {
await axios($, {
url: this.workflow_url,
method: 'POST',
data: this.records
})
}
},
})
Not the cleanest grammatically but y'all get the point right?
Hi @djpecot , yup I follow!
Though I'm curious how your data is being passed to the records prop is sometimes an array and sometimes a single object.
What's the source of your data if you don't mind me asking?
@ctrlaltdylan It's a webhook that's hitting me with a variable amount of objects (Toggl). It was hitting me with either an array or a single object as I recall, so had to account for it.
This functionality is a must! If a step receives an array, it should simply run the same step in the sequence for X times.
I'm looking for this feature as well 😄
This functionality is a must! If a step receives an array, it should simply run the same step in the sequence for X times.
Any update for this feature? Two years and I hard to see the future of this simple implementation.
Just created an account on this website an hour ago, and already deleting my account due to the lack of this feature. What a shame.
Just created an account on this website an hour ago, and already deleting my account due to the lack of this feature. What a shame.
Exactly my case as well 😅. Pipedream seems to be the best solution, otherwise! ❤️
@alexanderahlsen This is on the backlog for 2022! We're working on a GitHub integration now, and plan to work on control flow (loops, conditionals, etc.) soon after!
like people care about GitHub integration?
I think this has to be the deal breaker for most users once they realize this limitation. Perhaps its because making an extra call to a different workflow allows them to bill you separately for each iteration? Correct me if I am wrong , but each iteration would be an additional invocation? It's a shame, because I was really enjoying the building process only to find out I can't afford to use it.
Yes I can confirm that calling a different workflow counts as a separate invocation. I have to process orders with 20 items, so instead of having one invocation i have 21. Still I think we need this feature and need the ability to handle this within one invocation. It gets confusing and hard to monitor with a lot of single workflows.
eh, back to Zapier I guess :( I was really looking towards switching away from them.
I can code loop myself but if have to bother with that I may as well code the rest and self host it instead of pushing data via external services...
Just a quick note for other people stumbling accross this issue:
following @ctrlaltdylan's step-by-step explanation above, I was able to solve it for now. Though, this is somewhat tedious ;-)
{{Steps.Not.A.Coder ;-)}} The workaround offered with invoking another workflow by pushing it via webhook is elegant since the 2nd workflow runs independently for each item and in parallel - so I am guessing there's a huge time saver if you need to call another API for each item. However, I do not see a way to stitch them all together at the end in the sequence of the initial array - any ideas, please? @ctrlaltdylan
My used case: Take long-form text, split into pieces, do something with each piece, and then put them back together in the order of the original long-from text.
Just wanted to add my +1!
Use case: I want to add each person who is registered for an event in my appointment scheduler to my CRM.
A no-code way to do simple iteration over array elements would be really nice! There are lots of cases where you can end up with an array of data but the API you're working with won't accept array/bulk input so you need to send a request for each one.
I guess there is nothing technical preventing them to implement the loop system. What could limit them to do it is the business model: Currently, user/organization pay per event. The problem is that if a step retrieves 100 elements to process in a loop, it is still counted as one event. Since pipedream is probably using a pay as you go server (like AWS), they need to think about a new business model before going further. This is probably why this issue is in standby since 4 years...
Hi y'all, just want to let you know we read all this feedback and very much appreciate the use cases! This is on our roadmap for this year, after we ship our Git integration, folders, and other big features we're working on right now. So we plan to address this soon.
Hi y'all, just want to let you know we read all this feedback and very much appreciate the use cases! This is on our roadmap for this year, after we ship our Git integration, folders, and other big features we're working on right now. So we plan to address this soon.
Hello, great news 🎉 Will the tiers change ?
We haven't decided exactly how this will impact the invocations / credits model, but we'll comment here / share in future announcements on all the details.
There was at one stage where I was looking to get the payload data from Woocommerce. I was more interested in getting the licenses keys to uploaded to Dropbox folder in a form of txt file. My use case may not be as complicated. The payload created 40 lines of code of which will never be the real scenario for my situation. I did not see 40 invocations when I was looping through an an array, only one.
My payload looks like this
I am sure this is only applicable to me.
Seems like this is still a pipedream eh.
+1 its unfair if each looped item counts as a new invocation and gets billed. I love pipedream so far, but If this happens, I'm moving to autocode or something else.
