Skip to content

Define non-positive top_k; top_k range check #779

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Apr 5, 2023

Conversation

ivanstepanovftw
Copy link
Collaborator

  1. With --top_k 0 expected that all logits will be sampled (by top_p)
  2. Range check added to prevent from resizing logits more than vocab have. Resizing to more than vocab have causes token 0 to be appended to logits to be sampled

@ggerganov ggerganov merged commit 5a8c4f6 into ggml-org:master Apr 5, 2023
@ivanstepanovftw ivanstepanovftw deleted the non-positive-topk branch April 5, 2023 16:20
@Piezoid
Copy link
Contributor

Piezoid commented Apr 6, 2023

@ivanstepanovftw @ggerganov
Correct me if I'm mistaken, but I think the top_p filter needs the logits to be sorted. With this PR, disabling the top_k filter bypasses the sorting.

Instead, the whole array should be sorted when there is no top_k filtering:

sample_top_k(logits_id, top_k > 0 ? std::min(top_k, n_logits) : n_logits);

@ivanstepanovftw
Copy link
Collaborator Author

You are right!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants