Add Llama.cpp as inference backend for Answer section #114
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
*Use interface to exchange backends *Add llama cpp backend *Move ipex llm to class Re-enable RAG for both backends (#32) llama.cpp and ipex run alternate (#32) Adjust to llama.cpp branch to upstream changes. Improve UX. Adds separate Llama cpp service Update parameters for arc usage Update gitignore Uses own llama.cpp backend service for llama.cpp support (#32) Move llama.cpp out of ipex service (#32) Makes llama.cpp accessible (#32)
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| def __calculate_md5(self, file_path: str) -> str: | ||
| import hashlib | ||
|
|
||
| hasher = hashlib.md5() |
Check warning
Code scanning / Bandit
Use of insecure MD2, MD4, MD5, or SHA1 hash function. Warning
| parser = argparse.ArgumentParser(description="AI Playground Web service") | ||
| parser.add_argument("--port", type=int, default=59997, help="Service listen port") | ||
| args = parser.parse_args() | ||
| app.run(host="127.0.0.1", port=args.port, debug=True, use_reloader=False) |
Check failure
Code scanning / Bandit
A Flask app appears to be run with debug=True, which exposes the Werkzeug debugger and allows the execution of arbitrary code. Error
Nuullll
reviewed
Dec 16, 2024
| import gc | ||
| import json | ||
| import os | ||
| import re |
There was a problem hiding this comment.
Suggested change
| import re |
| parser = argparse.ArgumentParser(description="AI Playground Web service") | ||
| parser.add_argument("--port", type=int, default=59997, help="Service listen port") | ||
| args = parser.parse_args() | ||
| app.run(host="127.0.0.1", port=args.port, debug=True, use_reloader=False) |
There was a problem hiding this comment.
Suggested change
| app.run(host="127.0.0.1", port=args.port, debug=True, use_reloader=False) | |
| app.run(host="127.0.0.1", port=args.port, use_reloader=False) |
Nuullll
reviewed
Dec 16, 2024
| } | ||
|
|
||
| device = "xpu" | ||
| env_type = "arc" |
There was a problem hiding this comment.
this is no longer needed.
Suggested change
| env_type = "arc" |
EffortlessSteven
pushed a commit
to zimm0140/AI-Playground
that referenced
this pull request
Mar 27, 2025
**Description:** This adds a new python backend which uses python-llama-cpp as an inference backend for the Answer section. This allows users to run text generation using single file GGUF models. **Changes Made:** * add llama.cpp backend * add installation management for llama.cpp * adjust build scripts * adjust add model dialog **Testing Done:** Tested locally on BMG. **Checklist:** - [x] I have tested the changes locally. - [x] I have self-reviewed the code changes.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description:
This adds a new python backend which uses python-llama-cpp as an inference backend for the Answer section. This allows users to run text generation using single file GGUF models.
Changes Made:
Testing Done:
Tested locally on BMG.
Checklist: