Das IBM watsonx Leaderboard

Ilia Ris

• Best experiment: Dense Retrieval; Router; LLM reranking; Self-Consistency; llama-3.3 70b
• Signature: 25fabf
• Summary: Dense retrieval combined with LLM reranking, Self-Consistency and Schema Repair.
• Article: How I Won the Enterprise RAG Challenge
• Source code: Github

Models used:

• llama-3.3 70b

Architecture

Ilia Ris solved the problem by making it easy to run numerous experiments before the competition has even started. He created an evaluation pipeline that let him quickly evaluate different architectural solutions. The best solution was also among the fastest ones.

This solution used IBM WatsonX AI API. It had the following configuration:

• PDF Analysis: Documents are processed using a highly modified Docling Library from IBM. Modifications were needed to preserve page references.
• Router Pattern: First step in question answering flow picks the most suitable agent.
• Dense Retrieval: The system searches for relevant information based on semantic similarity (FAISS library and OpenAI vector embeddings).
• Parent Document Retrieval: Instead of retrieving only the chunk, full page is loaded to preserve relevant context.
• LLM Reranking: Retrieved information is re-evaluated and reordered by the LLM.
• Reasoning Patterns: Improve LLM accuracy within a single prompt by controlling its thinking process with Custom Chain-of-Thought and Structured Outputs.
• Final Answer generation: The optimized result was generated using llama-3.3 70b via WatsonX AI API. Since Structured Outputs were not supported by this API, an additional Schema Repair step was needed.
• Self-Consistency with Majority Vote: Multiple answer variations are generated, compared, and the most consistent one is selected.

R&D Experiments

Total experiments submitted: 11

Other approaches:

• Dense Retrieval; LLM Reranking; Router; SO CoT; o3-mini
• Dense Retrieval; Router; SO CoT; llama3.3-70b
• Dense Retrieval; Tables serialization; Router; LLM reranking; o3-mini
• Dense Retrieval; llama-3.3 70b
• Dense Retrieval; llama-3.1 8b
• Full Context; gemini-2.0 thinking
• Dense Retrieval; Router; LLM reranking; Self-Consistency; o3-mini
• Dense Retrieval; Router; LLM reranking; Self-Consistency; llama-3.3 70b

What didn't work?

• Using llama-3.1 8b for reranking
• Incorporating Full Context with gemini-2.0 thinking

Future experiments:

• Evaluating various local embedding models for fully offline solutions

Experiment journal:

• 16 min → R: 83.9, G: 72.8, Score: 114.8 ▲ - Dense Retrieval; LLM Reranking; Router; SO CoT; o3-mini
• 23 min → R: 81.4, G: 74.7, Score: 115.4 ▲ - Dense Retrieval; llama-3.3 70b
• 49 min → R: 83.8, G: 81.8, Score: 123.7 ▲ - Dense Retrieval; Router; LLM reranking; o3-mini
• 50 min → R: 81.1, G: 68.7, Score: 109.3 - Dense Retrieval; llama-3.1 8b
• 51 min → R: 75.5, G: 75.0, Score: 112.8 - Full Context; gemini-2.0 thinking
• 66 min → R: 83.0, G: 78.8, Score: 120.3 - Dense Retrieval; Tables serialization; Router; LLM reranking; o3-mini
• 22 hours → R: 83.5, G: 81.8, Score: 123.6 - Dense Retrieval; Router; LLM reranking; o3-mini
• 22 hours → R: 80.8, G: 75.7, Score: 116.1 - Dense Retrieval; llama-3.3 70b
• 33 hours → R: 83.4, G: 79.8, Score: 121.6 - Dense Retrieval; Router; LLM reranking; Self-Consistency; o3-mini
• 33 hours → R: 81.3, G: 79.7, Score: 120.3 - Dense Retrieval; Router; LLM reranking; Self-Consistency; llama-3.3 70b 👈

A.Rasskazov/V.Kalesnikau

• Experiment: ibm/granite-embedding-107m-multilingual + deepseek/deepseek-r1-distill-llama-70b
• Signature: 28a3dc
• Summary: A multi-agent system leveraging LLMs for question answering using similarity-based retrieval.

Models used:

• ibm/granite-embedding-107m-multilingual
• deepseek/deepseek-r1-distill-llama-70b

Architecture

The system preprocesses questions to extract key metrics and synonyms, retrieves relevant PDF pages via a multi-embedding vector database, and then uses an LLM to verify and extract the answer with its corresponding page reference.

R&D Experiments

Total experiments submitted: 2

Other approaches:

• pjatk_team_002: A system that preprocesses questions, retrieves relevant PDF pages using a vector database, and extracts answers with page references using LLMs.

What didn't work?

• Alternative embedding models for retrieval.
• Different strategies for key metric extraction.

Experiment journal:

• 30 hours → R: 84.0, G: 67.2, Score: 109.3 ▲ - multi_agent_ibm_openai
• 7 days → R: 82.5, G: 64.0, Score: 105.2 - pjatk_team_002 👈

nightwalkers

• Best experiment: nightwalkers-baseline
• Signature: 356ef4
• Summary: Utilized a vector database for efficient document retrieval and LLM for response generation.

Models used:

• deepseek-r1-distill-llama-70b

Architecture

The team implemented vector database search using embeddings from all-MiniLM-L6-v2 and ibm/granite-embedding-107m-multilingual models. This facilitated the retrieval of the most relevant page and document based on the query. The retrieved information was then processed by the deepseek-r1-distill-llama-70b LLM to generate relevant answers.

Felix-TAT

• Best experiment: IBM-4o-based Multiagent RAG
• Signature: 2ff9d6
• Summary: Multiagent, mixed-model approach with delegation and execution agents.

Models used:

• ibm/granite-20b-code-instruct
• gpt-4o-2024-08-06

Architecture

Multiagent approach using qdrant vector database. A delegation manager (openAI) splits the question into the touched companies and delegates company specific queries to expert agents (IBM), trying to retrieve parts of the answer from the chunks in the vector qdrant database. These responses are then combined and put together by an execution agent (openAI), coming up with the final answer. PDFs were parsed using docling, chunked in a custom way to keep tables as a whole including context.

R&D Experiments

Total experiments submitted: 4

Other approaches:

• Gemini Naive
• IBM-4o-based Multiagent RAG
• OpenAI Multiagent RAG

What didn't work?

• Using a single model without multiagent delegation
• Relying solely on vector database retrieval without full PDF context

Experiment journal:

• 6 days → R: 79.0, G: 60.3, Score: 99.8 ▲ - Gemini Naive
• 7 days → R: 81.7, G: 47.3, Score: 88.2 - IBM-4o-based Multiagent RAG 👈
• 7 days → R: 82.2, G: 66.0, Score: 107.1 ▲ - OpenAI Multiagent RAG
• 7 days → R: 80.2, G: 69.3, Score: 109.4 ▲ - Gemini-4o Multiagent RAG

AValiev

• Best experiment: IBM-mixtral-agentic-rag
• Signature: 9881dd
• Summary: Agentic RAG with type validation, Pydantic typing, Qdrant vector store querying. PDFs were extracted with PyPDF + Docling

Models used:

• mistralai/mixtral-8x7b-instruct-v01

Architecture

This RAG solution was based on an Agentic Retrieval-Augmented Generation (RAG) architecture.

It utilized type validation and Pydantic typing for robust data handling, and Qdrant vector store querying for efficient information retrieval. PDF documents were processed using PyPDF and Docling for accurate text extraction.

R&D Experiments

Total experiments submitted: 5

Other approaches:

• openai-agentic-rag
• IBM-mixtral-agentic-rag
• granite-3-8b-instruct_rag_agentic
• deepseek/deepseek-r1-distill-llama-70b_sophisticated_chunking_rag_agentic

What didn't work?

• Alternative LLM models such as OpenAI-gpt-4o-mini and mistralai/mixtral-8x7b-instruct-v01 were explored but did not achieve the same performance as the winning model.

Experiment journal:

• 54 min → R: 43.5, G: 60.0, Score: 81.8 ▲ - openai-agentic-rag
• 3 hours → R: 43.5, G: 33.0, Score: 54.8 - IBM-mixtral-agentic-rag 👈
• 4 hours → R: 43.5, G: 60.0, Score: 81.8 - IBM-deepseek-agentic-rag
• 4 hours → R: 43.5, G: 48.5, Score: 70.2 - granite-3-8b-instruct_rag_agentic
• 34 hours → R: 35.8, G: 53.0, Score: 70.9 - deepseek/deepseek-r1-distill-llama-70b_sophisticated_chunking_rag_agentic

ragtastic

• Best experiment: ragtastic
• Signature: 43d4fd
• Summary: The architecture leverages the Mistral-large model for its implementation.

Models used:

• mistral-large

Architecture

The solution used Mistral-large model to achieve its objectives. The architecture is designed to optimize performance and accuracy, ensuring robust results.