Fixed issue #775, #824, #894 and refactored Ensemble optimizer

[drawal1]

EnsembledProgram can now be loaded/saved ~~Inference of programs (forward function) is now concurrent~~ Added checks for size = 0 or size > number of programs

[okhat]

This is so cool.... I need to check the parallelization, it's a bit tricky in DSPy due to dspy.settings.

In dspy.evaluate.Evaluate we do things in a bit of a special way, which has to be applied when there's threading

[drawal1]

@okhat @arnavsinghvi11 - This PR now also includes a fix for issue #824. The fix is in teleprompt/random_search.py

[drawal1]

@okhat - any progress on this review? I am continuing to work on intelligent ensemble routing and don't want to muddy this PR with commits related to that work

btw, I have been testing ensemble concurrency and so far it works like a charm

also, I can reverse the commit for issue #824 and submit a separate PR for that, if it would expedite this PR review

[drawal1]

This is so cool.... I need to check the parallelization, it's a bit tricky in DSPy due to dspy.settings.

In dspy.evaluate.Evaluate we do things in a bit of a special way, which has to be applied when there's threading

I think I understand. I see this code in evaluate.py:

        def wrapped_program(example_idx, example):
            # NOTE: TODO: Won't work if threads create threads!
            thread_stacks = dspy.settings.stack_by_thread

So I need to test and verify the parallelization NOT in inference but as part of another optimization pipeline

[drawal1]

Fixed #894 in dsp/modules/aws_models.py, in class AWSMeta(AWSModel) by moving below functionality from init function:

        max_tokens = query_args.pop("max_tokens", None)
        if max_tokens:
            query_args["max_gen_len"] = max_tokens

to the create_body function

and copying kwargs to base_args by value

[drawal1]

@okhat - I have reverted the parallelization in EnsembledProgram.forward() based on your review and feedback.

The remaining fixes/improvements are targeted and should not have side-effects

[arnavsinghvi11]

Thanks @drawal1 for the PR! left a few comments. The changes to Ensemble look correct to me, but I think the ones for RandomSearch are not needed? or can be handled separately without impacting existing behavior. lmk what you think.

There's also a merge conflict to resolve and feel free to remove extraneous comments from the changes that do not help with following the code. Thanks!

tagging @okhat for reference on Ensemble/RandomSearch again (but I believe the parallelization changes are removed so should be good there).

[drawal1]

@arnavsinghvi11 - I have reverted ALL the complicated changes. Hopefully now this PR is acceptable

[arnavsinghvi11]

Hi @drawal1 , thanks for the changes. can you also revert the changes to BootstrapWithRandomSearch since those changes are not relevant to patching #858 .

[drawal1]

@arnavsinghvi11 - BootstrapWithRandomSearch changes are now reverted

[arnavsinghvi11]

Thanks @drawal1 !