[Feature Request]: Clear Workflow Instances After Execution to Prevent Cache Issues

[droideronline]

Do you need to file an issue?

• [x] I have searched the existing issues and this feature is not already filed.
• [x] My model is hosted on OpenAI or Azure. If not, please look at the "model providers" issue and don't file a new one here.
• [x] I believe this is a legitimate feature request, not just a question. If this is a question, please use the Discussions area.

Is your feature request related to a problem? Please describe.

Currently, based on the workflow type, if the workflow is "extract graph," we instantiate an fnllm instance with the name "extract graph." Similarly, for other workflows, we create instances with the workflow name. However, these instances are not cleared after execution is completed. This leads to issues when creating a new graphrag project and deleting the old graphrag cache. The system uses the old instance and tries to find the old cache, which has been deleted, causing the entire pipeline to fail.

This error occurs in Docker while running the build index inside a Celery worker.

Describe the solution you'd like

1. Instance Naming: Modify the instance naming convention to include a unique project ID. This ensures that each instance is uniquely identifiable and does not conflict with previous instances.
2. Instance Cleanup: Implement a mechanism to clear or delete instances after their execution is completed. This will prevent old instances from lingering and causing cache issues.
3. Error Handling: Enhance error handling to detect and manage cases where an instance tries to access a deleted cache, providing a clear error message and recovery steps.

Additional context

No response