Discussions

• 1 You must be logged in to vote

  💬

  MMTEB Benchmarks considerations

  KennethEnevoldsen started Sep 29, 2024 in General

  

  

  0
• 2 You must be logged in to vote

  💬

  Authors Missing openreview IDs

  KennethEnevoldsen started Sep 26, 2024 in General

  

  

  

  

  10
• 1 You must be logged in to vote

  🙏

  Support for reranking tasks with multi-level relevance scores?

  omihub777 asked Sep 26, 2024 in Q&A · Unanswered

  

  

  

  

  3
• 2 You must be logged in to vote

  💬

  MMTEB Author check

  KennethEnevoldsen started Sep 26, 2024 in General

  

  

  

  

  39
• 1 You must be logged in to vote

  🙏

  Long evaluation time using custom distance metric and token_embeddings for queries on retrieval task

  gnatesan asked Sep 12, 2024 in Q&A · Unanswered

  

  

  5
• 1 You must be logged in to vote

  🙏

  Cross-encoder and MTEB reranking tasks

  qiisziilbash asked Sep 10, 2024 in Q&A · Answered

  

  

  

  

  8
• 5 You must be logged in to vote

  💬

  Running models

  imenelydiaker started May 14, 2024 in General

  

  

  

  

  33
• 1 You must be logged in to vote

  💡

  Tracking CO2 Emissions

  x-tabdeveloping started May 14, 2024 in Ideas

  

  

  

  

  23
• 2 You must be logged in to vote

  💬

  Paper writing

  Muennighoff started Apr 29, 2024 in General

  

  

  

  

  60
• 1 You must be logged in to vote

  🙏

  Test run for a few good samples

  su-park asked Jul 18, 2024 in Q&A · Unanswered

  

  

  2
• 1 You must be logged in to vote

  💡

  Add a documentation site

  enhancement New feature or request

  KennethEnevoldsen started Jul 10, 2024 in Ideas

  

  

  0
• 1 You must be logged in to vote

  💬

  Paper writing - Description of abstention metrics

  hgissbkh started Jun 10, 2024 in General

  

  

  4
• 1 You must be logged in to vote

  💬

  Sampling MTEB

  NehaB18 started May 8, 2024 in General

  

  

  2
• 1 You must be logged in to vote

  💬

  Experiments for the paper

  imenelydiaker started May 27, 2024 in General

  

  

  

  

  3
• 4 You must be logged in to vote

  💬

  Score Aggregation for Multilingual benchmark (and in general)

  KennethEnevoldsen started May 17, 2024 in General

  

  

  

  

  6
• 1 You must be logged in to vote

  💬

  Stratification of multilabel data

  dokato started May 14, 2024 in General

  

  

  7
• 1 You must be logged in to vote

  💬

  Ideas on Conversational Retrieval Task Category

  vaibhavad started May 10, 2024 in General

  

  

  

  

  7
• 2 You must be logged in to vote

  💬

  Table to count number of tasks per language

  isaac-chung started May 13, 2024 in General

  

  

  

  

  15
• 3 You must be logged in to vote

  💬

  Leaderboard changes for massive multilinguality

  Muennighoff started May 12, 2024 in General

  

  

  

  

  6
• 2 You must be logged in to vote

  💬

  Retrieval speed up

  isaac-chung started May 6, 2024 in General

  

  

  

  

  12
• 1 You must be logged in to vote

  💡

  Linguistic Probing and New Task

  asparius started Apr 26, 2024 in Ideas

  

  

  

  

  3
• 1 You must be logged in to vote

  🙏

  Retrieval evaluator with 0 scores

  sroy-forr asked Apr 28, 2024 in Q&A · Answered

  

  

  3
• 2 You must be logged in to vote

  💬

  Datasets of language identification

  imenelydiaker started Apr 29, 2024 in General

  

  

  

  

  10
• 1 You must be logged in to vote

  🙏

  Required dataformat for corpus.jsonl, queries.jsonl, test.tsv

  paulesause asked Apr 26, 2024 in Q&A · Answered

  

  

  1
• 1 You must be logged in to vote

  💬

  Should we include Cross-Lingual tasks in MMTEB?

  orionw started Apr 13, 2024 in General

  

  

  

  

  13