Can it be launched on usual VPS? For example, 6 CPU 16 RAM (usual chips)

[CombainerA19]

Sorry for maybe a stupid question, fortunately, I find your product and want to integrate it into social media accounts; I don't release how to use it from the box (via Docker). As it said in the instruction I need a strong PC with GPU chips (which are pretty expensive for me) and I wonder if there is a way to utilize " input text prompt/variables - get a response in console / API"? Can you please, @artnitolog, comment on this?

For reference, a way how https://porfirevich.ru/ works.

[blokhin]

As you can see, you have to use very advanced GPU-hardware in order to run the inference.

On the other hand, it should be possible to run it using also much limited GPU memory via huggingface-accelerate. Generally, you have to use the hook for the pytorch forward method. This hook should load the weights into GPU memory iteratively and save the intermediate representation to disk until a working inference is obtained. There is a discussion here pointing out it is in principle possible.

[CombainerA19]

As you can see, you have to use very advanced GPU-hardware in order to run the inference.

On the other hand, it should be possible to run it using also much limited GPU memory via huggingface-accelerate. Generally, you have to use the hook for the pytorch forward method. This hook should load the weights into GPU memory iteratively and save the intermediate representation to disk until a working inference is obtained. There is a discussion here pointing out it is in principle possible.

Thanks, Evgeny for the help and links, I will listen to discussions and check product updates. AI is cool 🤘🏻