[버그]: HTTPSConnectionPool이 제대로 닫히지 않는 것 같습니다.

[tasoo-oos]

빠른 문제 해결을 위해 다음을 확인했나요?

• PyKis Issues에서 검색했지만 유사한 버그를 찾지 못했습니다.

버그 설명

for i, ticker in enumerate(tickers):
    asd = kis.stock(symbol=ticker, market="KRX")
    quote = asd.quote()
    print(f"{i}: {quote.name}")
    time.sleep(1)

이런 예시 프로그램을 다음 상황에서 실행합니다.

(fetch_krx_data) tasoo1118@tasoo-desktop:~/projects/python_projects/fetch_krx_data$ ulimit -n 128
(fetch_krx_data) tasoo1118@tasoo-desktop:~/projects/python_projects/fetch_krx_data$ python err_dup.py
0: AJ네트웍스
1: AK홀딩스

...

121: LX홀딩스1우
122: MH에탄올

그 뒤 이런 에러가 나옵니다.

Traceback (most recent call last):
  File "/home/tasoo1118/.miniconda3/envs/fetch_krx_data/lib/python3.11/site-packages/urllib3/connection.py", line 199, in _new_conn
  File "/home/tasoo1118/.miniconda3/envs/fetch_krx_data/lib/python3.11/site-packages/urllib3/util/connection.py", line 60, in create_connection
  File "/home/tasoo1118/.miniconda3/envs/fetch_krx_data/lib/python3.11/socket.py", line 974, in getaddrinfo
OSError: [Errno 24] Too many open files

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/tasoo1118/.miniconda3/envs/fetch_krx_data/lib/python3.11/site-packages/urllib3/connectionpool.py", line 789, in urlopen
  File "/home/tasoo1118/.miniconda3/envs/fetch_krx_data/lib/python3.11/site-packages/urllib3/connectionpool.py", line 490, in _make_request
  File "/home/tasoo1118/.miniconda3/envs/fetch_krx_data/lib/python3.11/site-packages/urllib3/connectionpool.py", line 466, in _make_request
  File "/home/tasoo1118/.miniconda3/envs/fetch_krx_data/lib/python3.11/site-packages/urllib3/connectionpool.py", line 1095, in _validate_conn
  File "/home/tasoo1118/.miniconda3/envs/fetch_krx_data/lib/python3.11/site-packages/urllib3/connection.py", line 693, in connect
  File "/home/tasoo1118/.miniconda3/envs/fetch_krx_data/lib/python3.11/site-packages/urllib3/connection.py", line 214, in _new_conn
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPSConnection object at 0x7905fe5dc250>: Failed to establish a new connection: [Errno 24] Too many open files

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/tasoo1118/.miniconda3/envs/fetch_krx_data/lib/python3.11/site-packages/requests/adapters.py", line 667, in send
  File "/home/tasoo1118/.miniconda3/envs/fetch_krx_data/lib/python3.11/site-packages/urllib3/connectionpool.py", line 843, in urlopen
  File "/home/tasoo1118/.miniconda3/envs/fetch_krx_data/lib/python3.11/site-packages/urllib3/util/retry.py", line 519, in increment
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='openapi.koreainvestment.com', port=9443): Max retries exceeded with url: /uapi/domestic-stock/v1/quotations/inquire-price?FID_COND_MRKT_DIV_CODE=J&FID_INPUT_ISCD=035420 (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7905fe5dc250>: Failed to establish a new connection: [Errno 24] Too many open files'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/tasoo1118/projects/python_projects/fetch_krx_data/err_dup.py", line 25, in <module>
  File "/home/tasoo1118/.miniconda3/envs/fetch_krx_data/lib/python3.11/site-packages/pykis/api/stock/quote.py", line 757, in product_quote
  File "/home/tasoo1118/.miniconda3/envs/fetch_krx_data/lib/python3.11/site-packages/pykis/api/stock/quote.py", line 729, in quote
  File "/home/tasoo1118/.miniconda3/envs/fetch_krx_data/lib/python3.11/site-packages/pykis/api/stock/quote.py", line 644, in domestic_quote
  File "/home/tasoo1118/.miniconda3/envs/fetch_krx_data/lib/python3.11/site-packages/pykis/kis.py", line 618, in fetch
  File "/home/tasoo1118/.miniconda3/envs/fetch_krx_data/lib/python3.11/site-packages/pykis/kis.py", line 553, in request
  File "/home/tasoo1118/.miniconda3/envs/fetch_krx_data/lib/python3.11/site-packages/requests/api.py", line 59, in request
  File "/home/tasoo1118/.miniconda3/envs/fetch_krx_data/lib/python3.11/site-packages/requests/sessions.py", line 589, in request
  File "/home/tasoo1118/.miniconda3/envs/fetch_krx_data/lib/python3.11/site-packages/requests/sessions.py", line 703, in send
  File "/home/tasoo1118/.miniconda3/envs/fetch_krx_data/lib/python3.11/site-packages/requests/adapters.py", line 700, in send
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='openapi.koreainvestment.com', port=9443): Max retries exceeded with url: /uapi/domestic-stock/v1/quotations/inquire-price?FID_COND_MRKT_DIV_CODE=J&FID_INPUT_ISCD=035420 (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7905fe5dc250>: Failed to establish a new connection: [Errno 24] Too many open files'))

ulimit -n 1024 의 상태에서 종목별로 quote 및 당일 1분봉 chart를 저장하는 프로그램을 짰는데, 그 프로그램에서는 1000종목째쯤에 이 오류가 뜨더군요. 아마 종목별로 커넥션을 관리해서 생기는 문제가 아닐까 싶은데, 정확히는 모르겠습니다. gc.collect()를 주기적으로 돌려주는 방법도 시도했으나, 증상은 그대로였습니다.

GPT에게 pykis/kis.py 의 request() 함수 코드를 주며 물어보니 이렇게 대답하더라고요.

This code uses requests.request directly, which does not benefit from connection pooling unless it’s within a requests.Session. Creating a new connection for each API call without a session could lead to multiple open connections that do not close properly.
Solution: Use a requests.Session() for connection pooling, which allows persistent connections to the API server.

종속성 버전 문제 진단

(fetch_krx_data) tasoo1118@tasoo-desktop:~/projects/python_projects/fetch_krx_data$ python
Python 3.11.10 (main, Oct  3 2024, 07:29:13) [GCC 11.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from pykis.utils.diagnosis import check; check()
Version: PyKis/2.1.2
Python: CPython 3.11.10
System: Linux #1 SMP PREEMPT_DYNAMIC Tue Nov 12 18:57:56 KST 2024 [x86_64]

Installed Packages:
========== requests  ===========
Required: 2.32.3>=
Installed: Not Found
====== websocket-client  =======
Required: 1.8.0>=
Installed: Not Found
======== cryptography  =========
Required: 43.0.0>=
Installed: Not Found
========== colorlog  ===========
Required: 6.8.2>=
Installed: Not Found
================================

(fetch_krx_data) tasoo1118@tasoo-desktop:~/projects/python_projects/fetch_krx_data$ pip freeze | grep requests
requests==2.32.3
(fetch_krx_data) tasoo1118@tasoo-desktop:~/projects/python_projects/fetch_krx_data$ pip freeze | grep websocket
websocket-client==1.8.0
(fetch_krx_data) tasoo1118@tasoo-desktop:~/projects/python_projects/fetch_krx_data$ pip freeze | grep cryptography
cryptography==43.0.3
(fetch_krx_data) tasoo1118@tasoo-desktop:~/projects/python_projects/fetch_krx_data$ pip freeze | grep colorlog
colorlog==6.9.0
(fetch_krx_data) tasoo1118@tasoo-desktop:~/projects/python_projects/fetch_krx_data$ which python
/home/tasoo1118/.miniconda3/envs/fetch_krx_data/bin/python
(fetch_krx_data) tasoo1118@tasoo-desktop:~/projects/python_projects/fetch_krx_data$ conda --version
conda 24.9.1

재현 방법

KRX_HTS_ID=
KRX_ACCOUNT_NUMBER=
KRX_APP_KEY=
KRX_APP_SECRET=

import time

from pykis import PyKis
from dotenv import dotenv_values
from pykrx import stock

config = dotenv_values(".env")  # 개인정보를 저장한 .env 파일을 읽어옵니다.

kis = PyKis(
    id=config["KRX_HTS_ID"],
    account=config["KRX_ACCOUNT_NUMBER"],
    appkey=config["KRX_APP_KEY"],
    secretkey=config["KRX_APP_SECRET"],
    keep_token=True
)

tickers = (
        stock.get_market_ticker_list(market="KOSPI") +
        stock.get_market_ticker_list(market="KOSDAQ") +
        stock.get_etf_ticker_list()
)  # 코스피, 코스닥, ETF 종목을 모두 가져옵니다. (총 3,000여개)

for i, ticker in enumerate(tickers):
    asd = kis.stock(symbol=ticker, market="KRX")
    quote = asd.quote()
    print(f"{i}: {quote.name}")
    time.sleep(1)

추가 정보

No response

PR를 통해 라이브러리에 기여하고 싶으신가요?

• 네, PR을 제출하여 도움을 주고 싶습니다!

[Soju06]

pykis의 모든 api를 호출하는 PyKis.fetch 함수에서 응답 결과를 requests.Response와 함께 Kis�Dynamic 객체에 저장하게 되는데, 이 때문에 커넥션이 종료되지 않아 발생한 문제인 것 같습니다.
requests.request대신 requests.Session.request를 사용하도록 변경하고, dry response를 받을 수 있게 self.session.request(..., stream=False) 및 response.close()가 필요해보이네요