Skip to content

Added refreshed docs for terminal arguments and python streaming responses #864

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Jan 3, 2024
Merged

Added refreshed docs for terminal arguments and python streaming responses #864

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
137 changes: 83 additions & 54 deletions docs/usage/python/streaming-response.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,70 +8,95 @@ You can stream messages, code, and code outputs out of Open Interpreter by setti
for chunk in interpreter.chat("What's 34/24?", stream=True, display=False):
print(chunk)
```

```
{'language': 'python'}
{'code': '34'}
{'code': ' /'}
{'code': ' '}
{'code': '24'}
{'executing': {'code': '34 / 24', 'language': 'python'}}
{'active_line': 1}
{'output': '1.4166666666666667\n'}
{'active_line': None}
{'end_of_execution': True}
{'message': 'The'}
{'message': ' result'}
{'message': ' of'}
{'message': ' the'}
{'message': ' division'}
{'message': ' '}
{'message': '34'}
{'message': '/'}
{'message': '24'}
{'message': ' is'}
{'message': ' approximately'}
{'message': ' '}
{'message': '1'}
{'message': '.'}
{'message': '42'}
{'message': '.'}
{"role": "assistant", "type": "code", "format": "python", "start": True}
{"role": "assistant", "type": "code", "format": "python", "content": "34"}
{"role": "assistant", "type": "code", "format": "python", "content": " /"}
{"role": "assistant", "type": "code", "format": "python", "content": " "}
{"role": "assistant", "type": "code", "format": "python", "content": "24"}
{"role": "assistant", "type": "code", "format": "python", "end": True}

{"role": "computer", "type": "confirmation", "format": "execution", "content": {"type": "code", "language": "python", "code": "34 / 24"}}

{"role": "computer", "type": "console", "start": True}
{"role": "computer", "type": "console", "format": "active_line", "content": "1"}
{"role": "computer", "type": "console", "format": "output", "content": "1.4166666666666667\n"}
{"role": "computer", "type": "console", "end": True}

{"role": "assistant", "type": "message", "start": True}
{"role": "assistant", "type": "message", "content": "The"}
{"role": "assistant", "type": "message", "content": " result"}
{"role": "assistant", "type": "message", "content": " of"}
{"role": "assistant", "type": "message", "content": " the"}
{"role": "assistant", "type": "message", "content": " division"}
{"role": "assistant", "type": "message", "content": " "}
{"role": "assistant", "type": "message", "content": "34"}
{"role": "assistant", "type": "message", "content": "/"}
{"role": "assistant", "type": "message", "content": "24"}
{"role": "assistant", "type": "message", "content": " is"}
{"role": "assistant", "type": "message", "content": " approximately"}
{"role": "assistant", "type": "message", "content": " "}
{"role": "assistant", "type": "message", "content": "1"}
{"role": "assistant", "type": "message", "content": "."}
{"role": "assistant", "type": "message", "content": "42"}
{"role": "assistant", "type": "message", "content": "."}
{"role": "assistant", "type": "message", "end": True}
```

**Note:** Setting `display=True` won't change the behavior of the streaming response, it will just render a display in your terminal.

# Anatomy

Each chunk of the streamed response is a dictionary, that has a "role" key that can be either "assistant" or "computer". The "type" key describes what the chunk is. The "content" key contains the actual content of the chunk.

Every 'message' is made up of chunks, and begins with a "start" chunk, and ends with an "end" chunk. This helps you parse the streamed response into messages.

Let's break down each part of the streamed response.

## Code

In this example, the LLM decided to start writing code first. It could have decided to write a message first, or to only write code, or to only write a message.

Before writing any code, the LLM will always set the `language` for the code it's about to write. In this case it decided to write `python`.
Every streamed chunk of type "code" has a format key that specifies the language. In this case it decided to write `python`.

This can be any language defined in [our languages directory.](https://github.com/KillianLucas/open-interpreter/tree/main/interpreter/computer/languages)

```
{'language': 'python'}

{"role": "assistant", "type": "code", "format": "python", "start": True}

```

Then, the LLM decided to write some code. The code is sent token-by-token:

```
{'code': '34'}
{'code': ' /'}
{'code': ' '}
{'code': '24'}

{"role": "assistant", "type": "code", "format": "python", "content": "34"}
{"role": "assistant", "type": "code", "format": "python", "content": " /"}
{"role": "assistant", "type": "code", "format": "python", "content": " "}
{"role": "assistant", "type": "code", "format": "python", "content": "24"}

```

When the LLM finishes writing code, it will send an "end" chunk:

```

{"role": "assistant", "type": "code", "format": "python", "end": True}

```

## Code Output

After the LLM finishes writing a code block, Open Interpreter will attempt to run it.

**Before** it runs it, the following message is sent:
**Before** it runs it, the following chunk is sent:

```
{'executing': {'code': '34 / 24', 'language': 'python'}}

{"role": "computer", "type": "confirmation", "format": "execution", "content": {"type": "code", "language": "python", "code": "34 / 24"}}

```

If you check for this object, you can break (or get confirmation) **before** executing the code.
Expand All @@ -88,42 +113,46 @@ for chunk in interpreter.chat("What's 34/24?", stream=True):
**While** the code is being executed, you'll recieve the line of code that's being run:

```
{'active_line': 1}
{"role": "computer", "type": "console", "format": "active_line", "content": "1"}
```

We use this to highlight the active line of code on our UI, which keeps the user aware of what Open Interpreter is doing.

You'll then recieve its output, if it produces any:

```
{'output': '1.4166666666666667\n'}
{"role": "computer", "type": "console", "format": "output", "content": "1.4166666666666667\n"}
```

When the code is **finished** executing, this flag will be sent:

```
{'end_of_execution': True}
{"role": "computer", "type": "console", "end": True}
```

## Message

Finally, the LLM decided to write a message. This is streamed token-by-token as well:

```
{'message': 'The'}
{'message': ' result'}
{'message': ' of'}
{'message': ' the'}
{'message': ' division'}
{'message': ' '}
{'message': '34'}
{'message': '/'}
{'message': '24'}
{'message': ' is'}
{'message': ' approximately'}
{'message': ' '}
{'message': '1'}
{'message': '.'}
{'message': '42'}
{'message': '.'}
```
{"role": "assistant", "type": "message", "start": True}
{"role": "assistant", "type": "message", "content": "The"}
{"role": "assistant", "type": "message", "content": " result"}
{"role": "assistant", "type": "message", "content": " of"}
{"role": "assistant", "type": "message", "content": " the"}
{"role": "assistant", "type": "message", "content": " division"}
{"role": "assistant", "type": "message", "content": " "}
{"role": "assistant", "type": "message", "content": "34"}
{"role": "assistant", "type": "message", "content": "/"}
{"role": "assistant", "type": "message", "content": "24"}
{"role": "assistant", "type": "message", "content": " is"}
{"role": "assistant", "type": "message", "content": " approximately"}
{"role": "assistant", "type": "message", "content": " "}
{"role": "assistant", "type": "message", "content": "1"}
{"role": "assistant", "type": "message", "content": "."}
{"role": "assistant", "type": "message", "content": "42"}
{"role": "assistant", "type": "message", "content": "."}
{"role": "assistant", "type": "message", "end": True}
```

For an example in JavaScript on how you might process these streamed chunks, see the [migration guide](https://github.com/KillianLucas/open-interpreter/blob/main/docs/NCU_MIGRATION_GUIDE.md)
Loading