[Core] Remove tokenizer group in vLLM

[zhuohan123]

Purpose

Tokenizer group is an abstraction introduced in early day vLLM to support the case where different LoRA adapters use different tokenizers. Looking back, LoRA is a niche feature among all vLLM users, and different tokenizers for different LoRAs is a "niche of the niche" feature. However, This niche of the niche feature spreads all around in vLLM code base, which becomes technical debt.

In the long term, I believe it's a good idea to eliminate the use of tokenizer in vLLM core and make most part of the core only works on token-IDs. This reduces our coupling with huggingface tokenizer and will make developing vLLM core easier.

Also see #23474 #23540

Test Plan

Make sure all the existing tests pass.

Test Result

---

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
• (Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @zhuohan123.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @zhuohan123.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork