lmql
lmql copied to clipboard
Additional backends aren't supported with serve-model
I implemented a PeftLLM backend (which I pasted into #152 ) but I cannot load it since there's no way to insert it into the registry via serve-model.
I think this could be fixed with an additional parameter such as --import_extra <some_package>
that allowed it to be evaluated just like lmql.models.lmtp.backends
is via its __init__.py
script, so that additional custom backends are supported.
I think this feature would be very beneficial so that the lmql implementation doesn't have to carry the burden of maintaining a bazillion backends with this weeks new hot stuff (that could be in a lmql-extras
package, for instance).
I like the suggestion. Did you see https://github.com/eth-sri/lmql/blob/main/src/lmql/models/lmtp/lmtp_programmatic_serve_example.py which allows you to run serve-model
from a custom launch script, which also lets you import custom modules beforehand.