How to achieve a while loop and reduce the certain state for each iteration?

[brlala]

What is a good approach for a while loop implementation using serverless workflow specs? Giving a simplified example, we have a list of book title in the state data and list of libraries database. We want to search through the list of books in each libraries until all books are found(early termination) or all libraries are exhausted.

StateData
{
   "otherImportantData": {}, 
   ...
   "books": ["Harry Potter", "Bad Blood"],
   "libraries": [{"name": "New York Library", "url": "www.libraryny.com/search"}, {"name": "Japan Library", "url": "www.libraryjp.com.jp/search"}]
   }

The current pseudocode I have is as follow but requires the abilitity to replace stateData.

1. call SearchLibrary workflow with inputs from StateData.
2. Search library will take the first library from the []libraries and search all available []books. If found, it will output it to []results and remove it from the books list.
3. The workflow will then output the reduced list of books and libraries(oldState minus the books or library that they have searched) as an input to a switch state workflow.
4. The switch workflow will then check if the length of books or libraries is 0, if yes, terminate, if no, go back to step 1 with the output as the new input.

Step 2 above can't be achieve because by default it will merge into the state data of array. Is there anyway we can force replace or have another flow design to achieve the intention above?

[brlala]

A workaround that we think of is having a statedata that contains a wrapper to a list of objects, not sure if this is intended

{
   "otherImportantData": {}, 
   ...
   "books": {"bookList": ["Harry Potter", "Bad Blood"]},
   "libraries": { "libraryList": [{"name": "New York Library", "url": "www.libraryny.com/search"}, {"name": "Japan Library", "url": "www.libraryjp.com.jp/search"}]}
   }

this way it will do a replace instead of merging

[ricardozanini]

Hi! How do you plan to implement the searchLibrary function? You will pass all the books in the array, and it will return the books they have?

What I would do:

1. For each library
2. callSearchLibrary, pass books
3. Add the books found in a new structure { booksSearchResult: [ "library": "Japan", "books": [ "BookA"] ]}
4. Apply jq operation to remove from books the books returned by thecallSearchLibrary function
5. If books.length == 0, end the workflow, otherwise return to step 1

[tsurdilo]

Whats the expected result that you workflow needs to provide?

One idea could be to apply transformation on your two arrays for example (jq):

[
  .libraries as $l
| .books[]
| {
    book: .,
    libraries: $l
  }
]     

that would give you:

[
  {
    "book": "Harry Potter",
    "libraries": [
      {
        "name": "New York Library",
        "url": "www.libraryny.com/search"
      },
      {
        "name": "Japan Library",
        "url": "www.libraryjp.com.jp/search"
      }
    ]
  },
  {
    "book": "Bad Blood",
    "libraries": [
      {
        "name": "New York Library",
        "url": "www.libraryny.com/search"
      },
      {
        "name": "Japan Library",
        "url": "www.libraryjp.com.jp/search"
      }
    ]
  }
]

and then use foreach state with your function results go into outputCollection.

If you define what your expected result workflow json structure should be that would help.

[brlala]

Hi! How do you plan to implement the searchLibrary function? You will pass all the books in the array, and it will return the books they have?

What I would do:

1. For each library
2. callSearchLibrary, pass books
3. Add the books found in a new structure { booksSearchResult: [ "library": "Japan", "books": [ "BookA"] ]}
4. Apply jq operation to remove from books the books returned by the callSearchLibrary function
5. If books.length == 0, end the workflow, otherwise return to step 1

Thanks this is exactly what I'm looking for, if my understanding of the specs is correct, we can remove the books from the return by the callSearchLibrary function, but it will eventually be merged into the state hence concatenating instead of replacing it. How do we do a jq operation to remove books instead of it merging?

[brlala]

Whats the expected result that you workflow needs to provide?

One idea could be to apply transformation on your two arrays for example (jq):

[
  .libraries as $l
| .books[]
| {
    book: .,
    libraries: $l
  }
]     

that would give you:

[
  {
    "book": "Harry Potter",
    "libraries": [
      {
        "name": "New York Library",
        "url": "www.libraryny.com/search"
      },
      {
        "name": "Japan Library",
        "url": "www.libraryjp.com.jp/search"
      }
    ]
  },
  {
    "book": "Bad Blood",
    "libraries": [
      {
        "name": "New York Library",
        "url": "www.libraryny.com/search"
      },
      {
        "name": "Japan Library",
        "url": "www.libraryjp.com.jp/search"
      }
    ]
  }
]

and then use foreach state with your function results go into outputCollection.

If you define what your expected result workflow json structure should be that would help.

The flow is that we will go through each library and search for the books, with the possible few scenarios, assuming there's two item in books and libraries:

1. All 2 books are found in the first library, reaches the switch statement(check if length of books or libraries are 0) and terminates the flow
2. 1 book is found in the first library, the flow does some cleanup logic, reaches the switch statement(check if length of books or libraries are 0) and found that the length of books and libraries are still more than one, hence it picks the 2nd library and search again.
3. 0-1 books are found, but the list of libraries to search are exhausted, reaches the switch statement(check if length of books or libraries are 0) and terminates the flow.

The statedata input is as follow

# iteration 0th - first entering the flow
{
   "otherImportantData": {}, 
   ...
   "books": ["Harry Potter", "Bad Blood"],
   "libraries": [{"name": "New York Library", "url": "www.libraryny.com/search"}, {"name": "Japan Library", "url": "www.libraryjp.com.jp/search"}]
 }
   
# iteration 1 - found "Bad Blood" item in the first library, add metadata of found item, run the `cleanup` flow, and remove the found item from `books` and the library from `libraries`
{
   "otherImportantData": {}, 
   ...
   "books": ["Harry Potter"],
   "libraries": [{"name": "Japan Library", "url": "www.libraryjp.com.jp/search"}]
   "metadata": [{"bookTitle": "Bad Blood", "Author": "John Carreyrou"}]
}

# iteration 2 - found "Harry Potter" item in the 2nd library, add metadata of found item, run the `cleanup` flow, and remove the found item from `books` and the library from `libraries`
{
   "otherImportantData": {}, 
   ...
   "books": [],
   "libraries": []
   "metadata": [{"bookTitle": "Bad Blood", "Author": "John Carreyrou"}, {"bookTitle": "Harry Potter", "Author": "J.K. Rowling"}]
}

It is necessary to search the item based on the order of the libraries from index[0] to the last and it's possible to find all books in the first library, hence it is not possible to go for the transformation approach you suggested. Would there be other approach that you'd recommend?

[brlala]

bump, will need help on this, trying to achieve deleting state with serverless workflow

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.