Collapse file tree

Files

 master

Search this repository

/

src

/

Copy path

Directory actions

More options

More options

Directory actions

More options

More options

Latest commit

rkazants

[GGUF] Create tokenizers factory for GGUF support in OpenVINO GenAI (#…

Open commit detailsfailure

May 26, 2025

ad1d555 · May 26, 2025

History

History

Open commit details

/

src

/

Top

Folders and files

Name		Name	Last commit message	Last commit date
parent directory ..
CMakeLists.txt		CMakeLists.txt	Alternative fix for JIT in PRCE2 (#470)	Apr 23, 2025
bpe_tokenizer.cpp		bpe_tokenizer.cpp	2025 Copyright year update (#370)	Jan 15, 2025
bpe_tokenizer.hpp		bpe_tokenizer.hpp	Fix Coverity Sentencepiece (#474)	May 7, 2025
byte_fallback.cpp		byte_fallback.cpp	2025 Copyright year update (#370)	Jan 15, 2025
byte_fallback.hpp		byte_fallback.hpp	2025 Copyright year update (#370)	Jan 15, 2025
bytes_to_chars.cpp		bytes_to_chars.cpp	Use const pointers for const tensors (#450)	Mar 24, 2025
bytes_to_chars.hpp		bytes_to_chars.hpp	2025 Copyright year update (#370)	Jan 15, 2025
case_fold.cpp		case_fold.cpp	Del cerr (#402)	Feb 5, 2025
case_fold.hpp		case_fold.hpp	Revert "Revert "Update Normalization"" (#401)	Feb 5, 2025
chars_to_bytes.cpp		chars_to_bytes.cpp	2025 Copyright year update (#370)	Jan 15, 2025
chars_to_bytes.hpp		chars_to_bytes.hpp	2025 Copyright year update (#370)	Jan 15, 2025
charsmap_normalization.cpp		charsmap_normalization.cpp	Fixed compilation with external protobuf / abseil (#414)	Feb 10, 2025
charsmap_normalization.hpp		charsmap_normalization.hpp	Revert "Revert "Update Normalization"" (#401)	Feb 5, 2025
combine_segments.cpp		combine_segments.cpp	Pass max_length to convert_model, add layer tests to RaggedToDense,…	Feb 7, 2025
combine_segments.hpp		combine_segments.hpp	Pass max_length to convert_model, add layer tests to RaggedToDense,…	Feb 7, 2025
equal_str.cpp		equal_str.cpp	2025 Copyright year update (#370)	Jan 15, 2025
equal_str.hpp		equal_str.hpp	2025 Copyright year update (#370)	Jan 15, 2025
fuze.cpp		fuze.cpp	2025 Copyright year update (#370)	Jan 15, 2025
fuze.hpp		fuze.hpp	2025 Copyright year update (#370)	Jan 15, 2025
normalize_unicode.cpp		normalize_unicode.cpp	Fixed compilation with external protobuf / abseil (#414)	Feb 10, 2025
normalize_unicode.hpp		normalize_unicode.hpp	Revert "Revert "Update Normalization"" (#401)	Feb 5, 2025
ov_extension.cpp		ov_extension.cpp	Improve pair input (#455)	Apr 18, 2025
ragged_tensor_pack.cpp		ragged_tensor_pack.cpp	2025 Copyright year update (#370)	Jan 15, 2025
ragged_tensor_pack.hpp		ragged_tensor_pack.hpp	2025 Copyright year update (#370)	Jan 15, 2025
ragged_to_dense.cpp		ragged_to_dense.cpp	Pass max_length to convert_model, add layer tests to RaggedToDense,…	Feb 7, 2025
ragged_to_dense.hpp		ragged_to_dense.hpp	Pass max_length to convert_model, add layer tests to RaggedToDense,…	Feb 7, 2025
ragged_to_ragged.cpp		ragged_to_ragged.cpp	Fix Coverity Sentencepiece (#474)	May 7, 2025
ragged_to_ragged.hpp		ragged_to_ragged.hpp	2025 Copyright year update (#370)	Jan 15, 2025
ragged_to_sparse.cpp		ragged_to_sparse.cpp	2025 Copyright year update (#370)	Jan 15, 2025
ragged_to_sparse.hpp		ragged_to_sparse.hpp	2025 Copyright year update (#370)	Jan 15, 2025
regex_normalization.cpp		regex_normalization.cpp	Fix Coverity Sentencepiece (#474)	May 7, 2025
regex_normalization.hpp		regex_normalization.hpp	2025 Copyright year update (#370)	Jan 15, 2025
regex_split.cpp		regex_split.cpp	Fix Coverity Sentencepiece (#474)	May 7, 2025
regex_split.hpp		regex_split.hpp	2025 Copyright year update (#370)	Jan 15, 2025
sentence_piece.cpp		sentence_piece.cpp	Support Sentencepiece Chars Tokenizer (#476)	May 9, 2025
sentence_piece.hpp		sentence_piece.hpp	2025 Copyright year update (#370)	Jan 15, 2025
special_tokens_split.cpp		special_tokens_split.cpp	Fix Coverity Sentencepiece (#474)	May 7, 2025
special_tokens_split.hpp		special_tokens_split.hpp	2025 Copyright year update (#370)	Jan 15, 2025
string_tensor_pack.hpp		string_tensor_pack.hpp	Turn on StringPack/Unpack on master from opset15: 3rd attempt (#386)	Feb 4, 2025
string_tensor_unpack.cpp		string_tensor_unpack.cpp	2025 Copyright year update (#370)	Jan 15, 2025
string_tensor_unpack.hpp		string_tensor_unpack.hpp	Turn on StringPack/Unpack on master from opset15: 3rd attempt (#386)	Feb 4, 2025
string_to_hash_bucket.cpp		string_to_hash_bucket.cpp	2025 Copyright year update (#370)	Jan 15, 2025
string_to_hash_bucket.hpp		string_to_hash_bucket.hpp	2025 Copyright year update (#370)	Jan 15, 2025
tensorflow_translators.cpp		tensorflow_translators.cpp	Fix Coverity Sentencepiece (#474)	May 7, 2025
tensorflow_translators.hpp		tensorflow_translators.hpp	Revert "Revert "Update Normalization"" (#401)	Feb 5, 2025
tokenizer.hpp		tokenizer.hpp	Improve pair input (#455)	Apr 18, 2025
tokenizers_factory.cpp		tokenizers_factory.cpp	[GGUF] Create tokenizers factory for GGUF support in OpenVINO GenAI (#…	May 26, 2025
tokenizers_factory.hpp		tokenizers_factory.hpp	[GGUF] Create tokenizers factory for GGUF support in OpenVINO GenAI (#…	May 26, 2025
trie_tokenizer.cpp		trie_tokenizer.cpp	2025 Copyright year update (#370)	Jan 15, 2025
trie_tokenizer.hpp		trie_tokenizer.hpp	2025 Copyright year update (#370)	Jan 15, 2025
truncate.cpp		truncate.cpp	Fix Coverity Sentencepiece (#474)	May 7, 2025
truncate.hpp		truncate.hpp	Improve pair input (#455)	Apr 18, 2025
unigram_tokenizer.cpp		unigram_tokenizer.cpp	Add JiT Compilation to PCRE2 (#449)	Mar 24, 2025
unigram_tokenizer.hpp		unigram_tokenizer.hpp	Add JiT Compilation to PCRE2 (#449)	Mar 24, 2025
utf8_validate.cpp		utf8_validate.cpp	Use const pointers for const tensors (#450)	Mar 24, 2025
utf8_validate.hpp		utf8_validate.hpp	2025 Copyright year update (#370)	Jan 15, 2025
utils.cpp		utils.cpp	Fix Coverity Sentencepiece (#474)	May 7, 2025
utils.hpp		utils.hpp	Fix Coverity Sentencepiece (#474)	May 7, 2025
vocab_decoder.cpp		vocab_decoder.cpp	2025 Copyright year update (#370)	Jan 15, 2025
vocab_decoder.hpp		vocab_decoder.hpp	2025 Copyright year update (#370)	Jan 15, 2025
vocab_encoder.cpp		vocab_encoder.cpp	Revert "Revert "Update Normalization"" (#401)	Feb 5, 2025
vocab_encoder.hpp		vocab_encoder.hpp	Revert "Revert "Update Normalization"" (#401)	Feb 5, 2025
wordpiece_tokenizer.cpp		wordpiece_tokenizer.cpp	2025 Copyright year update (#370)	Jan 15, 2025
wordpiece_tokenizer.hpp		wordpiece_tokenizer.hpp	2025 Copyright year update (#370)	Jan 15, 2025
View all files