S3 download issues

[joaopcm1996]

trafficstars

System Info

I've run into 2 unexpected issues/inconsistencies when downloading adapters from S3.

Issue 1: With PREDIBASE_ADAPTERS_BUCKET=sagemaker-us-east-1-000000000000 Several prefixes with naming lorax/mistral-adapters/{id}, with id being an integer from 1 to 50. Exact same adapter files replicated into each prefix.

Lorax Client call:

client.generate(prompt, max_new_tokens=64,adapter_id='lorax/mistral-adapters/{id}', adapter_source='s3')

For most id's, adapter is download as expected. For some, such as id's 1 through 5 from my testing (e.g. adapter_id='lorax/mistral-adapters/1') I get the following error:

Traceback (most recent call last):
  File "/home/ubuntu/user/invoke.py", line 8, in <module>
    print(client.generate(prompt, max_new_tokens=64,adapter_id=adapter_id, adapter_source='s3').generated_text)
  File "/opt/conda/envs/pytorch/lib/python3.10/site-packages/lorax/client.py", line 157, in generate
    raise parse_error(resp.status_code, payload)
lorax.errors.GenerationError: Request failed during generation: Server error: An error occurred (404) when calling the HeadObject operation: Not Found

This does not happen for adapters 6 through 50.

Issue 2: With PREDIBASE_ADAPTERS_BUCKET=lorax-adapters Several prefixes with naming adapter_{id}, with id being an integer from 1 to 50. Exact same adapter files replicated into each prefix.

Lorax Client call:

client.generate(prompt, max_new_tokens=64,adapter_id=f'adapter_{id}', adapter_source='s3')

For most id's, adapter is donwloaded as expected. For some others (e.g. adapter_id='adapter_2') I get the following error:

Traceback (most recent call last):
  File "/home/ubuntu/user/invoke.py", line 8, in <module>
    print(client.generate(prompt, max_new_tokens=64,adapter_id=adapter_id, adapter_source='s3').generated_text)
  File "/opt/conda/envs/pytorch/lib/python3.10/site-packages/lorax/client.py", line 157, in generate
    raise parse_error(resp.status_code, payload)
lorax.errors.GenerationError: Request failed during generation: Server error: Multiple models found for model_id adapter_2

Even though files at S3 prefix are:

aws s3 ls s3://lorax-adapters/adapter_2/
2024-01-03 21:28:44       1519 .gitattributes
2024-01-03 21:28:44       1307 README.md
2024-01-03 21:28:44        501 adapter_config.json
2024-01-03 21:28:44   13677261 adapter_model.bin
2024-01-03 21:28:44       4091 training_args.bin

Information

• [X] Docker
• [ ] The CLI directly

Tasks

• [X] An officially supported command
• [ ] My own modifications

Reproduction

See repro notebook here

Expected behavior

All adapter prefixes in the same bucket are loaded without error

[tgaddair]

Thanks for reporting this @joaopcm1996, that behavior is surprising. It should work now without needing to set the PREDIBASE_ADAPTERS_BUCKET param. I'll see if I can repro the issue.

[joaopcm1996]

Oh I misinterpreted your suggestion in #155 then. I just experimented not passing in PREDIBASE_ADAPTERS_BUCKET at launch and setting adapter_id as the full S3 path; still errored and didn't work for any prefix, looking for the adapter in the local cache folder instead of donwloading it. I made sure to pull the latest image (acct id is anonymized here).

from lorax import Client

client = Client("http://127.0.0.1:8080")
prompt = "[INST] Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May? [/INST]"
adapter_id = "s3://sagemaker-us-east-1-000000000000/lorax/mistral-adapters/1"

print(client.generate(prompt, max_new_tokens=64,adapter_id=adapter_id, adapter_source='s3').generated_text)  

Error: lorax.errors.GenerationError: Request failed during generation: Server error: Can't find 'adapter_config.json' at '/data/models--s3:----sagemaker-us-east-1-000000000000--lorax--mistral-adapters--11--/snapshots'