code-rag-bench

code-rag-bench copied to clipboard

Issue in Reproduce Retrieval Performance

1

Hi there, Thank you for the excellent work and for publishing the code base. I am attempting to reproduce the retrieval performance of BGE-base as shown in Table 3 but...

yeliu918

Hello, We have come across your work on Code Retrieval. Can you benchmark our code embedding models, [CodeSage](https://huggingface.co/codesage)? Thank you.

wasiahmad

I want to run benchmark with my custom search pipeline (for example add bm25 and embedding vector search with LLM reranker). Are there any simple ways to insert it into...

NikitaShalagin

Your project doesn't appear to specify an open source license, is it possible to add one? Thanks.

wuzhenqing

Reproduction issue

1

Hi. I really appreciate your helpful contribution in this field. I have always wondered why Code RAG does not have a well-composed benchmark, and you are the one who resolved...

ArtemisDicoTiar

During the process of using your repository, I encountered some issues that I would like to bring to your attention. While attempting to build and evaluate the code, I ran...

zhenlongDai

About evaluating repoval-function tasks in Retrieval-Augmented Code Generation

2

After downloading the repositories for RepoEval (e.g., under retrieval/output/repoeval/repositories/function_level/), obtaining retrieval results (e.g., /path/to/retrieval/results/retriever-name.jsonl), **and obtaining the code generation outputs in previous steps (e.g., /path/to/generation/outputs/model-name.json),** we can run execution-based evaluation....

trartcode

Bug: Required Dataset for Livecodebench Retrieval project-x/programming is Unavailable

8

### **Description** Hello, I am trying to run the retrieval script to generate the index and `qrels` file for the `live_code_bench` task, following the project's documentation. The process fails because...

begineri

Potential mismatch between beir and vllm

2

It seems install older vllm version lacks "PoolerConfig" class inside vllm/config.py package. Beir tries to use it and ends up crashing. I tried running the following line: python3 -u eval_beir_sbert_canonical.py...

KrossKinetic

Hi, thank you very much for your excellent work and contributions in this domain! According to the CODERAG-BENCH paper, a reproducible execution environment is provided for RepoEval. The file `code-rag-bench/tree/main/generation/eval/tasks/custom_metrics/repoeval_task_id2tests.json`...

ZimaBlue307