Skip to content

Commit 0d936bf

Browse files
authored
Update test-tokenizer-1-bpe.cpp
Cosmetics.
1 parent 1244b00 commit 0d936bf

File tree

1 file changed

+2
-1
lines changed

1 file changed

+2
-1
lines changed

tests/test-tokenizer-1-bpe.cpp

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -91,7 +91,7 @@ int main(int argc, char **argv) {
9191
}
9292
}
9393
}
94-
// Restrict to unicode planes
94+
// Restrict to assigned unicode planes
9595
// for (uint32_t cp = 0x10000; cp < 0x0010ffff; ++cp) {
9696
for (uint32_t cp = 0x10000; cp < 0x00040000; ++cp) {
9797
std::string str = codepoint_to_utf8(cp);
@@ -113,6 +113,7 @@ int main(int argc, char **argv) {
113113
return 4;
114114
}
115115
}
116+
116117
llama_free_model(model);
117118
llama_free(ctx);
118119

0 commit comments

Comments
 (0)