datarobot-user-models icon indicating copy to clipboard operation
datarobot-user-models copied to clipboard

Minor improvements to GPU examples

Open elatt opened this issue 4 months ago • 2 comments

This repository is public. Do not put here any private DataRobot or customer's data: code, datasets, model artifacts, .etc.

Summary

  • Remove concurrency param from example
  • Set vLLM model via engine config

Rationale

This makes some minor improvements to our examples. First it is better to set the vLLM model param via the config file because then it is versioned and can't be overriden at runtime with no auditing. Also don't showcase the concurrency flag yet as we haven't tested it extensively.

elatt avatar Oct 12 '24 00:10 elatt