Improve Deep Research LLM Input Handling to Reduce Irrelevant Responses

[vishiivivek]

Description:
Currently, the Deep Research tool sends all retrieved webpage content in a single request to the LLM, often exceeding 10+ pages. This causes the model to assume that all the provided content is relevant, leading to incorrect or misleading responses.

Problems Identified:

• The LLM assumes the answer lies within the provided content, even when it's irrelevant.
• Large context size (up to 25,000 tokens) can dilute relevant information and it is proven that LLM performs very poorly.
• Mixing multiple sources in one go makes it difficult for the model to focus on important details.

Problematic function: process_serp_result

Suggested Improvements:

1. Chunked Processing:

• Instead of sending all pages at once, process them one by one and filter useful insights.
• Allow step-by-step integration of information, improving response accuracy.

2. Explicit Instructions to the LLM:

• Add instructions making it clear that some content may be irrelevant.
• Use structured prompts that tell the model to prioritize key insights.

3. Relevance Filtering Before Sending to LLM:

• Implement a preprocessing step to discard low-relevance text.
• Consider ranking content based on keyword relevance before including it in the prompt. (Similar to RAG)

Expected Benefits:

• Eliminates noise by discarding ~90% of unnecessary pages
• Reduces token usage, optimizing LLM performance
• Improves response accuracy by sending only highly relevant context

Note: This is a highly experimental feature and takes time to arrive at final solution. Also have to come up with an evaluation technique