Skip to content

Commit 38f7dea

Browse files
committed
Update README and docs
1 parent 267d364 commit 38f7dea

File tree

2 files changed

+54
-4
lines changed

2 files changed

+54
-4
lines changed

README.md

Lines changed: 23 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -15,15 +15,15 @@ This package provides:
1515
- OpenAI-like API
1616
- LangChain compatibility
1717

18-
# Installation
18+
## Installation
1919

2020
Install from PyPI:
2121

2222
```bash
2323
pip install llama-cpp-python
2424
```
2525

26-
# Usage
26+
## High-level API
2727

2828
```python
2929
>>> from llama_cpp import Llama
@@ -51,6 +51,27 @@ pip install llama-cpp-python
5151
}
5252
```
5353

54+
## Web Server
55+
56+
`llama-cpp-python` offers a web server which aims to act as a drop-in replacement for the OpenAI API.
57+
This allows you to use llama.cpp compatible models with any OpenAI compatible client (language libraries, services, etc).
58+
59+
To install the server package and get started:
60+
61+
```bash
62+
pip install llama-cpp-python[server]
63+
export MODEL=./models/7B
64+
python3 -m llama_cpp.server
65+
```
66+
67+
Navigate to [http://localhost:8000/docs](http://localhost:8000/docs) to see the OpenAPI documentation.
68+
69+
## Low-level API
70+
71+
The low-level API is a direct `ctypes` binding to the C API provided by `llama.cpp`.
72+
The entire API can be found in [llama_cpp/llama_cpp.py](https://github.com/abetlen/llama-cpp-python/blob/master/llama_cpp/llama_cpp.py) and should mirror [llama.h](https://github.com/ggerganov/llama.cpp/blob/master/llama.h).
73+
74+
5475
# Documentation
5576

5677
Documentation is available at [https://abetlen.github.io/llama-cpp-python](https://abetlen.github.io/llama-cpp-python).

docs/index.md

Lines changed: 31 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,9 @@
1-
# 🦙 Python Bindings for `llama.cpp`
1+
# Getting Started
22

3+
## 🦙 Python Bindings for `llama.cpp`
4+
5+
[![Documentation](https://img.shields.io/badge/docs-passing-green.svg)](https://abetlen.github.io/llama-cpp-python)
6+
[![Tests](https://github.com/abetlen/llama-cpp-python/actions/workflows/test.yaml/badge.svg?branch=main)](https://github.com/abetlen/llama-cpp-python/actions/workflows/test.yaml)
37
[![PyPI](https://img.shields.io/pypi/v/llama-cpp-python)](https://pypi.org/project/llama-cpp-python/)
48
[![PyPI - Python Version](https://img.shields.io/pypi/pyversions/llama-cpp-python)](https://pypi.org/project/llama-cpp-python/)
59
[![PyPI - License](https://img.shields.io/pypi/l/llama-cpp-python)](https://pypi.org/project/llama-cpp-python/)
@@ -21,7 +25,7 @@ Install from PyPI:
2125
pip install llama-cpp-python
2226
```
2327

24-
## Usage
28+
## High-level API
2529

2630
```python
2731
>>> from llama_cpp import Llama
@@ -49,8 +53,33 @@ pip install llama-cpp-python
4953
}
5054
```
5155

56+
## Web Server
57+
58+
`llama-cpp-python` offers a web server which aims to act as a drop-in replacement for the OpenAI API.
59+
This allows you to use llama.cpp compatible models with any OpenAI compatible client (language libraries, services, etc).
60+
61+
To install the server package and get started:
62+
63+
```bash
64+
pip install llama-cpp-python[server]
65+
export MODEL=./models/7B
66+
python3 -m llama_cpp.server
67+
```
68+
69+
Navigate to [http://localhost:8000/docs](http://localhost:8000/docs) to see the OpenAPI documentation.
70+
71+
## Low-level API
72+
73+
The low-level API is a direct `ctypes` binding to the C API provided by `llama.cpp`.
74+
The entire API can be found in [llama_cpp/llama_cpp.py](https://github.com/abetlen/llama-cpp-python/blob/master/llama_cpp/llama_cpp.py) and should mirror [llama.h](https://github.com/ggerganov/llama.cpp/blob/master/llama.h).
75+
76+
5277
## Development
5378

79+
This package is under active development and I welcome any contributions.
80+
81+
To get started, clone the repository and install the package in development mode:
82+
5483
```bash
5584
git clone [email protected]:abetlen/llama-cpp-python.git
5685
git submodule update --init --recursive

0 commit comments

Comments
 (0)