[Question]: OpenAI Agent Responding Beyond Available Tools

[Akash0463]

Question Validation

• I have searched both the documentation and discord for an answer.

Question

Hi,

I’m using the OpenAI agent with the GPT-4o-mini model in LlamaIndex, and I've noticed that the agent seems to be answering from its own knowledge base rather than being restricted to the data available through the tools provided. This results in the assistant responding with information not present in the tool’s output.

Question:
Is there a way to restrict the OpenAI agent to only respond with data retrieved from the available tools, rather than allowing it to answer based on its own general knowledge?

Any help or suggestions on how to enforce this restriction would be greatly appreciated!

[dosubot]

To restrict the OpenAI agent using the GPT-4o-mini model in LlamaIndex to only respond with data retrieved from the available tools, you can implement the following solutions:

1. Reduce the Temperature: Set the temperature to 0.0 to ensure the agent follows specifications more strictly. This approach has been confirmed to help in similar scenarios.

2. Detailed Tool Descriptions: Provide detailed descriptions for each tool, specifying when to use the tool, the expected input and output formats, and examples.

3. Modify System Message: Adjust the agent's system message to prime it to use tools when specific criteria are met.

4. Explicit Instructions: Configure the system to explicitly state that if the query is out of scope or unavailable for semantic search using only data in the document, the response should indicate that this information is not found in the context ^[1].

These steps are designed to ensure that the agent uses the available tools rather than relying on its own general knowledge.

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other}

[logan-markewich]

Set a system prompt with instructions for how you want the agent to act