runpod-python

runpod-python copied to clipboard

Is your feature request related to a problem? Please describe. I would like to use a regular Pod to handle the average traffic, and then scale on serverless if the Pod gets overloaded. By doing so I would be able to handle traffic spikes with no effort and being able to use the Saving Plans at the same time as well.

Describe the solution you'd like As there is the possibility to use the same Serverless template with regular Pods (ref. 1, ref. 2), I would like to use the same package to handle requests either to Serverless endpoints or on-demand Pods.

An ideal usage example follows:

            runpod.api_key = self.runpod_api_key
            runpod.endpoint_url_base = "https://some-random-pod-host-8000.proxy.runpod.net"
            endpoint = runpod.Endpoint(None)

            # then use the API as usual
            run_request = endpoint.run(input_payload)
            status = run_request.status()

Or something like

            runpod.api_key = self.runpod_api_key
            endpoint = runpod.CustomEndpoint("https://some-random-pod-host-8000.proxy.runpod.net")

            # then use the API as usual
            run_request = endpoint.run(input_payload)
            status = run_request.status()

Describe alternatives you've considered

• Execute the request using requests as specified in the docs; this solution is working of course, but it makes me write more code with different handling methods according to what type of executor I am using (serverless/pod)

Additional context N/A