deepsparse
deepsparse copied to clipboard
neuralmagic
Sparsity-aware deep learning inference runtime for CPUs
Since we've made a quantized CLIP (https://huggingface.co/neuralmagic/CLIP-ViT-B-32-256x256-DataComp-s34B-b86K-quant-ds), let's use it to test our pipeline!
from deepsparse import Pipeline sa_pipeline = Pipeline.create( task="sentiment-analysis", model_path="/content/bert-sentiment-onnx-fp16-opset" ) inference = sa_pipeline("Aku suka itu") print(inference) [/usr/local/lib/python3.10/dist-packages/deepsparse/engine.py](https://localhost:8080/#) in run(self, inp, val_inp) 530 self._validate_inputs(inp) 531 --> 532 return self._eng_net.execute_list_out(inp) 533 534...
**Describe the bug** I'm trying to pull the docker image but hitting an access denied error **Expected behavior** i would expect `docker pull ghcr.io/neuralmagic/deepsparse:1.5.1` to work **Environment** I'm on a...
It seems that fundamentally at the `Pipeline` level, there is an assumption that `ops` is a list, not a dictionary. To reproduce: ```python from deepsparse.v2.text_generation import TextGenerationPipelineNoCache prompt = ["Some...
Despite the code saying that you can specify "token_embeddings" for the DeepSparseSentenceTransformer.encode method as the value for the output_value argument, it is not actually set up properly to do this....
Just a FYI of a new discovery that should be super relevant for you guys: https://twitter.com/aaquib_syed1/status/1714386165237776653
This PR updates the validation of CLI command tests. The checks for things like "error" or "failure" in the command output are problematic because they can simply be part of...
Question - should this script include some sort of metric generation for zeroshot classification on some dataset? Right now, the script just takes in local samples ## Summary - Validation...
Example: ``` numactl -C0-15 python deepsparse/src/deepsparse/transformers/eval_downstream.py \ \ --num-cores 16 \ --dataset openai_humaneval \ --humaneval-method pass_at_k \ --engine deepsparse \ --start 0 \ --max-samples 2 ``` - This will create...
Metadata
Owner
Metadata
Sparsity-aware deep learning inference runtime for CPUs