LLM-as-a-Rel: Benchmarking Automatic Relevance Judgments

Hossein A. Rahmani¹, Emine Yilmaz¹, Nick Craswell², Bhaskar Mitra², Paul Thomas², Charles L. A. Clarke³, Mohammad Aliannejadi⁴, Clemencia Siro⁴, Guglielmo Faggioli⁵

¹University College London, ²Microsoft, ³University of Waterloo, ⁴University of Amsterdam, ⁵University of Padua

LLMJudge Overview Challenge Coda and Data Data Report Leaderboard

LLMJudge Benchamrk TBA

Summary

To be added.

LLMJudge Challenge Dataset

LLMJudge Benchamrk

Results and Analysis

Main Results

Analysis

(A) Ablations

BibTeX

@article{rahmani2024llmhudgebench,
      author    = {Rahmani, Hossein A. and Yilmaz, Emine and Craswell, Nick and Mitra, Bhaskar and Thomas, Paul and Clarke, Charles L. A. and Aliannejadi, Mohammad and Siro, Clemencia and Faggioli, Guglielmo},
      title     = {LLMJudge: Automatic Relevance Judgments for Search and Retrieval Systems},
      year      = {2024},
     journal    = {#},
     url        = {#}
    }