ImportError: cannot import name '_tiktoken' from partially initialized module 'tiktoken'

[onlychenchen]

I pulled the code and tried to run the use case:
"""
import tiktoken
enc = tiktoken.get_encoding("cl100k_base")
assert enc.decode(enc.encode("hello world")) == "hello world"
enc = tiktoken.encoding_for_model("gpt-4")
"""
but error:
File "D:\tiktoken-main\tiktoken\core.py", line 9, in
from tiktoken import _tiktoken
ImportError: cannot import name '_tiktoken' from partially initialized module 'tiktoken'(most likely due to a circular import)
how to fix this error? i need help, thanks!

[hauntsaninja]

Do you have some other file name tiktoken.py, e.g. like #106? There isn't enough information to reproduce this error.

[akshat-g]

@hauntsaninja - Actually, I am facing the same issue. I am using marvin - https://pypi.org/project/marvin/
It uses tiktoken as a dependency. So it installes tiktoken as well.
I am trying to use marvin on AWS lambda. When I invoke my lambda, error occurs:
cannot import name '_tiktoken' from partially initialized module 'tiktoken' (most likely due to a circular import) (/opt/python/lib/python3.9/site-packages/tiktoken/__init__.py)

It's here in tiktoken/core.py

from __future__ import annotations

import functools
from concurrent.futures import ThreadPoolExecutor
from typing import AbstractSet, Collection, Literal, NoReturn, Optional, Union

import regex

from tiktoken import _tiktoken

I have no file names as tiktoken.py so probably that is not causing the issue. Also I am using openai library as well, but not sure if that is causing the issue.