[Bug]: Specifying a seed dramatically slows down performance

[Ben-Epstein]

Contact Details

[email protected]

What happened?

You can't directly set a seed in the python SDK (see https://github.com/googleapis/python-aiplatform/pull/3186), but looking through the code im doing this, which works fine.

But you can see the performance slows dramatically.

import vertexai
from vertexai.language_models import TextGenerationModel

model = TextGenerationModel.from_pretrained("text-bison@002")
prompt = "tell me a story and make it very creative"

def predict(prompt, seed=None):
    kwargs = {"maxDecodeSteps": 1024, "temperature": 0.0, "topP": 1, "topK": 40}
    if seed is not None:
        kwargs["seed"] = seed
    instance = {"content": prompt}
    raw_pred = model._endpoint.predict(instances=[instance], parameters=kwargs)
    response = vertexai.language_models._language_models._parse_text_generation_model_multi_candidate_response(raw_pred)
    return response.text.strip()

%timeit predict(prompt, seed=None);  # 3.96 s ± 531 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
%timeit predict(prompt, seed=42);    #  7.42 s ± 797 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

Also, setting a seed isn't guaranteeing determinism. I ran this 500 times, both with and without a seed

• without a seed: 52 unique responses
• with a seed: 7 unique responses

Relevant log output

No response

Code of Conduct

• [X] I agree to follow this project's Code of Conduct