Skip to content

Pull requests: openai/tiktoken

New pull request New
28 Open 82 Closed
28 Open 82 Closed
Author
Filter by author
Loading
Label
Filter by label
Loading
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Loading
Milestones
Filter by milestone
Loading
Reviews
Filter by reviews
No reviews Review required Approved review Changes requested
Assignee
Filter by who’s assigned
Assigned to nobody Loading
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Pull requests list

Add back in musllinux_aarch64 fix #350
#402 opened May 19, 2025 by rhuddleston Loading…
1
Add support for free-threaded CPython 3.13t
#401 opened May 13, 2025 by rostan-t Loading…
Update load.py
#399 opened Apr 23, 2025 by Shivam-Singh-Dev Loading…
Add o4 model prefix to MODEL_PREFIX_TO_ENCODING and MODEL_TO_ENCODING
#397 opened Apr 17, 2025 by ColemanDunn Loading…
Add gpt-4.1 to model
#396 opened Apr 14, 2025 by AlisaLC Loading…
7
Make decoder and sorted_token_bytes re-use existing memory - Unit Tests
#384 opened Feb 19, 2025 by CodeBeaverAI Loading…
In-code documentation update
#383 opened Feb 17, 2025 by louisbrulenaudet Loading…
1
Option to use TIKTOKEN_BPE_HOST environment variable for configurable BPE host URL
#357 opened Nov 4, 2024 by AitBaali-Hamza-bcg Loading…
2
make decoder and sorted_token_bytes re-use existing memory
#352 opened Oct 17, 2024 by tmm1 Loading…
use HashSet from rustc-hash too
#351 opened Oct 17, 2024 by tmm1 Loading…
Uses Regex instead of fancy-regex - 6x speedup
#331 opened Aug 5, 2024 by Majdoddin Loading…
3
Cache for Encoding - Runtime Boosted by 12%
#319 opened Jul 10, 2024 by Majdoddin Loading…
Add Terminal-Based Visualization Tool for Tokenized Data Points in Tiktoken Tokenizer
#314 opened Jun 18, 2024 by LVivona Loading…
1
Remove dependency on requests
#271 opened Mar 26, 2024 by tristan-jl Loading…
1
Optimize byte pair merge for really big tokens (40x faster for a 2500 token word)
#239 opened Jan 15, 2024 by l0rinc Loading…
26
Support enterprise network support for self-hosted encodings (UPDATED v0.6.0)
#238 opened Jan 12, 2024 by blaney83 Loading…
1
0) Add the jtokkit test suite examples to validate the cl100k_base, p50k_base & r50k_base encodings
#237 opened Jan 6, 2024 by l0rinc Loading…
1
feat: Add option TIKTOKEN_FORCE_CACHE
#170 opened Jul 28, 2023 by Kontrolix Loading…
4
✨ feat: Split code rust, for run as python lib and rust lib
#167 opened Jul 18, 2023 by Miuler Loading…
1
5
fix pycache folder being added instead of encoding info files
#152 opened Jun 23, 2023 by dheerajiiitv Loading…
1
9
Add support for fine-tuned models in encoding_for_model
#135 opened May 23, 2023 by thespino Loading…
2
Option to use env variable for BPE_FILE location
#128 opened May 15, 2023 by csnelsonchu Loading…
update license metadata to use SPDX identifier as per recommendation
#125 opened May 12, 2023 by imrehg Loading…
Allow older versions of dependencies fancy-regex and bstr.
#108 opened Apr 16, 2023 by petterreinholdtsen Loading…
1
Add ruby bindings
#83 opened Mar 28, 2023 by arjun810 Loading…
ProTip! no:milestone will show everything without a milestone.