Migrate users of llm tokenizer to use pytorch-labs/tokenizers #9114

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Merged

facebook-github-bot merged 1 commit into main from export-D70932091

Mar 17, 2025

Contributor

larryliu0820 commented Mar 10, 2025

Summary: Finally migrate llm tokenizer usages to pytorch-labs/tokenizers.

Differential Revision: D70932091

larryliu0820 requested review from jackzhxng, iseeyuan and lucylq as code owners

March 10, 2025 23:56

pytorch-bot bot commented Mar 10, 2025 •

edited

Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/9114

📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 New Failure, 3 Cancelled Jobs, 2 Unrelated Failures

As of commit 9ce3d49 with merge base f789df2 ():

NEW FAILURE - The following job has failed:

pull / unittest-buck / linux / linux-job (gh)
RuntimeError: Command docker exec -t 2580992ba6ed3bc4ddaaf62ede95716683dd3015fe66817b0e21b711003deb2e /exec failed with exit code 3

CANCELLED JOBS - The following jobs were cancelled. Please retry:

trunk / test-models-macos (llama3_2_vision_encoder, portable) / macos-job (gh)
##[error]The operation was canceled.
trunk / test-models-macos (vit, xnnpack-quantization-delegation) / macos-job (gh)
##[error]The operation was canceled.
trunk / test-selective-build-macos (cmake) / macos-job (gh)
##[error]The operation was canceled.

BROKEN TRUNK - The following jobs failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

pull / unittest-buck / macos / macos-job (gh) (trunk failure)
RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 3
trunk / test-arm-backend-delegation / linux-job (gh) (trunk failure)

This comment was automatically generated by Dr. CI and updates every 15 minutes.

facebook-github-bot added the CLA Signed label

Contributor

facebook-github-bot commented Mar 10, 2025

This pull request was exported from Phabricator. Differential Revision: D70932091

facebook-github-bot added the fb-exported label

Contributor

facebook-github-bot commented Mar 10, 2025

This pull request was exported from Phabricator. Differential Revision: D70932091

larryliu0820 force-pushed the export-D70932091 branch from 788d5e9 to 70675c3 Compare

March 10, 2025 23:59

larryliu0820 added the topic: not user facing label

facebook-github-bot pushed a commit that referenced this pull request


          Migrate users of llm tokenizer to use pytorch-labs/tokenizers (#9114)

add8c5a

Summary:

Finally migrate llm tokenizer usages to pytorch-labs/tokenizers.

Differential Revision: D70932091

facebook-github-bot force-pushed the export-D70932091 branch from 70675c3 to add8c5a Compare

March 11, 2025 01:30

Contributor

facebook-github-bot commented Mar 11, 2025

This pull request was exported from Phabricator. Differential Revision: D70932091

facebook-github-bot pushed a commit that referenced this pull request


          Migrate users of llm tokenizer to use pytorch-labs/tokenizers (#9114)

8262bad

Summary:

Finally migrate llm tokenizer usages to pytorch-labs/tokenizers.

Differential Revision: D70932091

facebook-github-bot force-pushed the export-D70932091 branch from add8c5a to 8262bad Compare

March 11, 2025 04:05

facebook-github-bot requested a review from kirklandsign as a code owner

March 11, 2025 04:05

Contributor

facebook-github-bot commented Mar 11, 2025

This pull request was exported from Phabricator. Differential Revision: D70932091

iseeyuan approved these changes

View reviewed changes

Contributor

facebook-github-bot commented Mar 16, 2025

This pull request was exported from Phabricator. Differential Revision: D70932091

larryliu0820 added a commit that referenced this pull request


          Migrate users of llm tokenizer to use pytorch-labs/tokenizers (#9114)

dd7985e

Summary:
Pull Request resolved: #9114

Finally migrate llm tokenizer usages to pytorch-labs/tokenizers.

Reviewed By: iseeyuan

Differential Revision: D70932091

larryliu0820 force-pushed the export-D70932091 branch from 8262bad to dd7985e Compare

March 16, 2025 07:20

larryliu0820 requested review from swolchok and cccclai as code owners

March 16, 2025 07:20

Contributor

facebook-github-bot commented Mar 16, 2025

This pull request was exported from Phabricator. Differential Revision: D70932091

larryliu0820 added a commit that referenced this pull request


          Migrate users of llm tokenizer to use pytorch-labs/tokenizers (#9114)

c9428eb

Summary:
Pull Request resolved: #9114

Finally migrate llm tokenizer usages to pytorch-labs/tokenizers.

Reviewed By: iseeyuan

Differential Revision: D70932091

larryliu0820 force-pushed the export-D70932091 branch from dd7985e to c9428eb Compare

March 16, 2025 07:55

Contributor

facebook-github-bot commented Mar 16, 2025

This pull request was exported from Phabricator. Differential Revision: D70932091

larryliu0820 added a commit that referenced this pull request


          Migrate users of llm tokenizer to use pytorch-labs/tokenizers (#9114)

a6c67f5

Summary:
Pull Request resolved: #9114

Finally migrate llm tokenizer usages to pytorch-labs/tokenizers.

Reviewed By: iseeyuan

Differential Revision: D70932091

larryliu0820 force-pushed the export-D70932091 branch from c9428eb to a6c67f5 Compare

March 16, 2025 08:02

Contributor

facebook-github-bot commented Mar 16, 2025

This pull request was exported from Phabricator. Differential Revision: D70932091

Contributor

facebook-github-bot commented Mar 16, 2025

This pull request was exported from Phabricator. Differential Revision: D70932091

larryliu0820 added a commit that referenced this pull request


          Migrate users of llm tokenizer to use pytorch-labs/tokenizers (#9114)

0d875b5

Summary:
Pull Request resolved: #9114

Finally migrate llm tokenizer usages to pytorch-labs/tokenizers.

Reviewed By: iseeyuan

Differential Revision: D70932091

larryliu0820 force-pushed the export-D70932091 branch from 6805b10 to 0d875b5 Compare

March 16, 2025 18:40

Contributor

facebook-github-bot commented Mar 16, 2025

This pull request was exported from Phabricator. Differential Revision: D70932091

larryliu0820 added a commit that referenced this pull request


          Migrate users of llm tokenizer to use pytorch-labs/tokenizers (#9114)

605383f

Summary:
Pull Request resolved: #9114

Finally migrate llm tokenizer usages to pytorch-labs/tokenizers.

Reviewed By: iseeyuan

Differential Revision: D70932091

larryliu0820 force-pushed the export-D70932091 branch from 0d875b5 to 605383f Compare

March 16, 2025 18:48

larryliu0820 added a commit that referenced this pull request


          Migrate users of llm tokenizer to use pytorch-labs/tokenizers (#9114)

138f43f

Summary:
Pull Request resolved: #9114

Finally migrate llm tokenizer usages to pytorch-labs/tokenizers.

Reviewed By: iseeyuan

Differential Revision: D70932091

larryliu0820 force-pushed the export-D70932091 branch from 605383f to 138f43f Compare

March 16, 2025 20:14

Contributor

facebook-github-bot commented Mar 16, 2025

This pull request was exported from Phabricator. Differential Revision: D70932091

1 similar comment

Contributor

facebook-github-bot commented Mar 16, 2025

This pull request was exported from Phabricator. Differential Revision: D70932091

larryliu0820 added a commit that referenced this pull request


          Migrate users of llm tokenizer to use pytorch-labs/tokenizers (#9114)

f0e2850

Summary:
Pull Request resolved: #9114

Finally migrate llm tokenizer usages to pytorch-labs/tokenizers.

Reviewed By: iseeyuan

Differential Revision: D70932091

larryliu0820 force-pushed the export-D70932091 branch from 138f43f to f0e2850 Compare

March 16, 2025 20:18

Contributor

facebook-github-bot commented Mar 16, 2025

This pull request was exported from Phabricator. Differential Revision: D70932091

larryliu0820 added a commit that referenced this pull request


          Migrate users of llm tokenizer to use pytorch-labs/tokenizers (#9114)

61230b6

Summary:
Pull Request resolved: #9114

Finally migrate llm tokenizer usages to pytorch-labs/tokenizers.

Reviewed By: iseeyuan

Differential Revision: D70932091

larryliu0820 force-pushed the export-D70932091 branch from f0e2850 to 61230b6 Compare

March 16, 2025 20:29

Contributor

facebook-github-bot commented Mar 16, 2025

This pull request was exported from Phabricator. Differential Revision: D70932091

larryliu0820 added a commit that referenced this pull request


          Migrate users of llm tokenizer to use pytorch-labs/tokenizers (#9114)

71ea375

Summary:
Pull Request resolved: #9114

Finally migrate llm tokenizer usages to pytorch-labs/tokenizers.

Reviewed By: iseeyuan

Differential Revision: D70932091

larryliu0820 force-pushed the export-D70932091 branch from 61230b6 to 71ea375 Compare

March 16, 2025 22:02

Contributor

facebook-github-bot commented Mar 17, 2025

This pull request was exported from Phabricator. Differential Revision: D70932091

larryliu0820 added a commit that referenced this pull request


          Migrate users of llm tokenizer to use pytorch-labs/tokenizers (#9114)

d8f0e32

Summary:
Pull Request resolved: #9114

Finally migrate llm tokenizer usages to pytorch-labs/tokenizers.

Reviewed By: iseeyuan

Differential Revision: D70932091

larryliu0820 force-pushed the export-D70932091 branch from 71ea375 to d8f0e32 Compare

March 17, 2025 01:32


          Migrate users of llm tokenizer to use pytorch-labs/tokenizers (#9114)

9ce3d49

Summary:
Pull Request resolved: #9114

Finally migrate llm tokenizer usages to pytorch-labs/tokenizers.

Reviewed By: iseeyuan

Differential Revision: D70932091

Contributor

facebook-github-bot commented Mar 17, 2025

This pull request was exported from Phabricator. Differential Revision: D70932091

larryliu0820 force-pushed the export-D70932091 branch from d8f0e32 to 9ce3d49 Compare

March 17, 2025 01:40

facebook-github-bot merged commit 206a4df into main

157 of 165 checks passed

facebook-github-bot deleted the export-D70932091 branch

March 17, 2025 04:50

larryliu0820 mentioned this pull request

Fix benchmark job and apple demo app job #9329

Merged

larryliu0820 added a commit that referenced this pull request


          Fix benchmark job and apple demo app job (#9329)

8c32da7

### Summary
Fixes CI failures caused by #9114 

We need to tell xcode project about the new tokenizers repo and include
path.

### Test plan
These jobs need to pass:

```
Apple / build-benchmark-app / macos-job 
Apple / build-demo-ios / macos-job
```

DannyYuyang-quic pushed a commit to CodeLinaro/executorch that referenced this pull request


          Migrate users of llm tokenizer to use pytorch-labs/tokenizers

430af43

Differential Revision: D70932091

Pull Request resolved: pytorch#9114

DannyYuyang-quic pushed a commit to CodeLinaro/executorch that referenced this pull request


          Fix benchmark job and apple demo app job (pytorch#9329)

38906b2

### Summary
Fixes CI failures caused by pytorch#9114 

We need to tell xcode project about the new tokenizers repo and include
path.

### Test plan
These jobs need to pass:

```
Apple / build-benchmark-app / macos-job 
Apple / build-demo-ios / macos-job
```

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

iseeyuan iseeyuan approved these changes

jackzhxng Awaiting requested review from jackzhxng jackzhxng is a code owner

lucylq Awaiting requested review from lucylq lucylq is a code owner

kirklandsign Awaiting requested review from kirklandsign kirklandsign is a code owner

swolchok Awaiting requested review from swolchok swolchok is a code owner

cccclai Awaiting requested review from cccclai cccclai is a code owner

Labels

CLA Signed fb-exported topic: not user facing