support for answerdotai/ModernBERT-base

[mathcass]

Model description

I tried running this on AWS SageMaker with the config,

    config = {
        "HF_MODEL_ID": "answerdotai/ModernBERT-base",
        "POOLING": "mean",
    }

but it failed with the following error message from the AWS console,

Error: Could not create backend
Caused by:
    Could not start backend: Model is not supported: unknown variant `modernbert`, expected one of `bert`, `xlm-roberta`, `camembert`, `roberta`, `distilbert`, `nomic_bert` at line 32 column 28

Open source status

• The model implementation is available
• The model weights are available

Provide useful links for the implementation

Released today, https://huggingface.co/blog/modernbert

[mhillebrand]

I tried launching TEI-gRPC with Docker, using a fine-tuned ModernBERT model, and I got this error:

2024-12-20T20:05:51.464907Z  INFO text_embeddings_router: router/src/main.rs:175: Args { model_id: "/mod**/**nal", revision: None, tokenization_workers: None, dtype: Some(Float16), pooling: None, max_concurrent_requests: 512, max_batch_tokens: 16384, max_batch_requests: None, max_client_batch_size: 4000, auto_truncate: false, default_prompt_name: None, default_prompt: None, hf_api_token: None, hostname: "81f1e6a2b3e2", port: 80, uds_path: "/tmp/text-embeddings-inference-server", huggingface_hub_cache: Some("/data"), payload_limit: 2000000, api_key: None, json_output: false, otlp_endpoint: None, otlp_service_name: "text-embeddings-inference.server", cors_allow_origin: None }
thread 'main' panicked at /usr/src/router/src/lib.rs:134:62:
tokenizer.json not found. text-embeddings-inference only supports fast tokenizers: Error("data did not match any variant of untagged enum ModelWrapper", line: 251498, column: 1)
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

And when I tried setting dtype to bfloat16:

error: invalid value 'bfloat16' for '--dtype <DTYPE>'
  [possible values: float16, float32]

tranformers 4.47.1
tokenizers 0.21.0
TEI 1.6.0

Upgrading to the latest main branch of transformers had the same problem.

[kozistr]

@mhillebrand @mathcass hi. I implemented the ModernBert model, and you can use it by building from the source based on #459! please feel free to leave a comment if you have an issue :)

[mhillebrand]

@mhillebrand @mathcass hi. I implemented the ModernBert model, and you can use it by building from the source based on #459! please feel free to leave a comment if you have an issue :)

Thank you, @kozistr

I've never built TEI from source. Any pointers on getting your branch to work with gRPC?

[kozistr]

@mhillebrand @mathcass hi. I implemented the ModernBert model, and you can use it by building from the source based on #459! please feel free to leave a comment if you have an issue :)

Thank you, @kozistr

I've never built TEI from source. Any pointers on getting your branch to work with gRPC?

you could try with Docker.

1. clone kozistr/text-embeddings-inference repo, feature/modernbert branch
2. build Docker image (e.g. docker build . -f Dockerfile-cuda --build-arg CUDA_COMPUTE_CAP=$runtime_compute_cap) (docs: https://github.com/huggingface/text-embeddings-inference?tab=readme-ov-file#docker-build)

[touhi99]

+1

[mhillebrand]

@kozistr I built a docker image from your latest feature/modernbert repo, but I'm seeing some errors.

Bash commands:

git clone git@github.com:kozistr/text-embeddings-inference.git .
git checkout -b feature/modernbert
docker build . -f Dockerfile-cuda --build-arg CUDA_COMPUTE_CAP=86 -t tei-modernbert:latest

markdown_classifier.sh:

docker run \
    --runtime=nvidia \
    --gpus device=0 \
    -p 8120:80 \
    -v /opt/markdown/model:/model \
    --rm tei-modernbert:latest \
    --model-id /model/final \
    --dtype float16

Error:

$ /opt/boot/markdown_classifier.sh 
2025-02-28T19:04:43.751161Z  INFO text_embeddings_router: router/src/main.rs:175: Args { model_id: "/mod**/**nal", revision: None, tokenization_workers: None, dtype: Some(Float16), pooling: None, max_concurrent_requests: 512, max_batch_tokens: 16384, max_batch_requests: None, max_client_batch_size: 20, auto_truncate: true, default_prompt_name: None, default_prompt: None, hf_api_token: None, hostname: "6269c46b1d6b", port: 80, uds_path: "/tmp/text-embeddings-inference-server", huggingface_hub_cache: Some("/data"), payload_limit: 2000000, api_key: None, json_output: false, otlp_endpoint: None, otlp_service_name: "text-embeddings-inference.server", cors_allow_origin: None }

thread 'main' panicked at /usr/src/router/src/lib.rs:134:62:
tokenizer.json not found. text-embeddings-inference only supports fast tokenizers: Error("data did not match any variant of untagged enum ModelWrapper", line: 251510, column: 1)
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

I initially saw an error about missing id2label and label2id in config.json, so Cursor + Claude Sonnet 3.7 added this:

  "id2label": {
    "0": "LABEL_0",
    "1": "LABEL_1"
  },
  "label2id": {
    "LABEL_0": 0,
    "LABEL_1": 1
  }

[kozistr]

@kozistr I built a docker image from your latest feature/modernbert repo, but I'm seeing some errors.

Bash commands:

git clone git@github.com:kozistr/text-embeddings-inference.git .
git checkout -b feature/modernbert
docker build . -f Dockerfile-cuda --build-arg CUDA_COMPUTE_CAP=86 -t tei-modernbert:latest

markdown_classifier.sh:

docker run
--runtime=nvidia
--gpus device=0
-p 8120:80
-v /opt/markdown/model:/model
--rm tei-modernbert:latest
--model-id /model/final
--dtype float16
Error:

$ /opt/boot/markdown_classifier.sh
2025-02-28T19:04:43.751161Z INFO text_embeddings_router: router/src/main.rs:175: Args { model_id: "/mod**/**nal", revision: None, tokenization_workers: None, dtype: Some(Float16), pooling: None, max_concurrent_requests: 512, max_batch_tokens: 16384, max_batch_requests: None, max_client_batch_size: 20, auto_truncate: true, default_prompt_name: None, default_prompt: None, hf_api_token: None, hostname: "6269c46b1d6b", port: 80, uds_path: "/tmp/text-embeddings-inference-server", huggingface_hub_cache: Some("/data"), payload_limit: 2000000, api_key: None, json_output: false, otlp_endpoint: None, otlp_service_name: "text-embeddings-inference.server", cors_allow_origin: None }

thread 'main' panicked at /usr/src/router/src/lib.rs:134:62:
tokenizer.json not found. text-embeddings-inference only supports fast tokenizers: Error("data did not match any variant of untagged enum ModelWrapper", line: 251510, column: 1)
note: run with RUST_BACKTRACE=1 environment variable to display a backtrace
I initially saw an error about missing id2label and label2id in config.json, so Cursor + Claude Sonnet 3.7 added this:

  "id2label": {
    "0": "LABEL_0",
    "1": "LABEL_1"
  },
  "label2id": {
    "LABEL_0": 0,
    "LABEL_1": 1
  }

it seems like there's no tokenizer under the path (error message: tokenizer.json not found. text-embeddings-inference only supports fast tokenizers:). maybe, placing tokenizer-related files (e.g. tokenizer.json, tokenizer_config.json, special_token_maps.json) along with the model files would resolve that issue. and, currently fp16 (w/ fa) is not supported, so please use it with fp32.

[mhillebrand]

Heh, I definitely checked the existence of tokenizer.json. It's there. BTW, I'm getting the same error with --dtype float32.

[mhillebrand]

I also tried to convert ModernBERT's tokenizer to a fast tokenizer, but I failed. I added -e RUST_BACKTRACE=full, but it wasn't much help

Args { model_id: "/mod**/**nal", revision: None, tokenization_workers: None, dtype: Some(Float32), pooling: None, max_concurrent_requests: 512, max_batch_tokens: 16384, max_batch_requests: None, max_client_batch_size: 32, auto_truncate: false, default_prompt_name: None, default_prompt: None, hf_api_token: None, hostname: "49f60a98aedc", port: 80, uds_path: "/tmp/text-embeddings-inference-server", huggingface_hub_cache: Some("/data"), payload_limit: 2000000, api_key: None, json_output: false, otlp_endpoint: None, otlp_service_name: "text-embeddings-inference.server", cors_allow_origin: None }

thread 'main' panicked at /usr/src/router/src/lib.rs:134:62:
tokenizer.json not found. text-embeddings-inference only supports fast tokenizers: Error("data did not match any variant of untagged enum ModelWrapper", line: 251510, column: 1)
stack backtrace:
   0:     0x58e788d860e9 - <unknown>
   1:     0x58e7889561c3 - <unknown>
   2:     0x58e788d859d2 - <unknown>
   3:     0x58e788d85f43 - <unknown>
   4:     0x58e788d8580b - <unknown>
   5:     0x58e788dc2a68 - <unknown>
   6:     0x58e788dc29c9 - <unknown>
   7:     0x58e788dc2e6c - <unknown>
   8:     0x58e78825616f - <unknown>
   9:     0x58e7882564e5 - <unknown>
  10:     0x58e78865117b - <unknown>
  11:     0x58e78879e627 - <unknown>
  12:     0x58e78879abee - <unknown>
  13:     0x58e7886c6723 - <unknown>
  14:     0x58e78879edc4 - <unknown>
  15:     0x7986f6cecd90 - <unknown>
  16:     0x7986f6cece40 - __libc_start_main
  17:     0x58e7885e1855 - <unknown>
  18:                0x0 - <unknown>

[kozistr]

I also tried to convert ModernBERT's tokenizer to a fast tokenizer, but I failed.

could you please double check the directory inside the docker to check the files are correctly located? looks like you mounted your local path (/opt/markdown/model) to /model and loaded the model from /model/final.

[mhillebrand]

final is a symbolic link to the directory in the screenshot above.

(infer) /opt/markdown/model$ l
total 8.0K
drwxrwxr-x 7 matt matt 4.0K Feb 28 14:23 ModernBERT-base
lrwxrwxrwx 1 matt matt   31 Feb 28 09:55 final -> ModernBERT-base/checkpoint-2100
drwxrwxr-x 3 matt matt 4.0K Feb 28 09:55 qwen
(infer) /opt/markdown/model$ ll final/*
-rw-rw-r-- 1 matt matt 1.5K Feb 28 22:23 final/config.json
-rw-rw-r-- 1 matt matt 286M Feb 28 09:32 final/model.safetensors
-rw-rw-r-- 1 matt matt 571M Feb 28 09:32 final/optimizer.pt
-rw-rw-r-- 1 matt matt  14K Feb 28 09:32 final/rng_state.pth
-rw-rw-r-- 1 matt matt 1.1K Feb 28 09:32 final/scheduler.pt
-rw-rw-r-- 1 matt matt  694 Feb 28 09:32 final/special_tokens_map.json
-rw-rw-r-- 1 matt matt 3.5M Feb 28 14:22 final/tokenizer.json
-rw-rw-r-- 1 matt matt  21K Feb 28 14:22 final/tokenizer_config.json
-rw-rw-r-- 1 matt matt 3.5K Feb 28 09:32 final/trainer_state.json
-rw-rw-r-- 1 matt matt 5.3K Feb 28 09:32 final/training_args.bin

[kozistr]

final is a symbolic link to the directory in the screenshot above.

(infer) /opt/markdown/model$ l
total 8.0K
drwxrwxr-x 7 matt matt 4.0K Feb 28 14:23 ModernBERT-base
lrwxrwxrwx 1 matt matt 31 Feb 28 09:55 final -> ModernBERT-base/checkpoint-2100
drwxrwxr-x 3 matt matt 4.0K Feb 28 09:55 qwen
(infer) /opt/markdown/model$ ll final/*
-rw-rw-r-- 1 matt matt 1.5K Feb 28 22:23 final/config.json
-rw-rw-r-- 1 matt matt 286M Feb 28 09:32 final/model.safetensors
-rw-rw-r-- 1 matt matt 571M Feb 28 09:32 final/optimizer.pt
-rw-rw-r-- 1 matt matt 14K Feb 28 09:32 final/rng_state.pth
-rw-rw-r-- 1 matt matt 1.1K Feb 28 09:32 final/scheduler.pt
-rw-rw-r-- 1 matt matt 694 Feb 28 09:32 final/special_tokens_map.json
-rw-rw-r-- 1 matt matt 3.5M Feb 28 14:22 final/tokenizer.json
-rw-rw-r-- 1 matt matt 21K Feb 28 14:22 final/tokenizer_config.json
-rw-rw-r-- 1 matt matt 3.5K Feb 28 09:32 final/trainer_state.json
-rw-rw-r-- 1 matt matt 5.3K Feb 28 09:32 final/training_args.bin

well, as far as I know, when mounting a local path to a Docker container, symbolic links within the mounted directory are not automatically resolved or followed. So, it'd be great to check the mounted directory inside the container.

[mhillebrand]

Symbolic links have always worked for me when using Docker. I just did away with the symbolic link, and I got the same bogus error.

docker run \
    --gpus device=0 \
    -p 8120:80 \
    -v /opt/markdown/model:/model \
    -e RUST_BACKTRACE=full \
    --rm tei-modernbert:latest \
    --model-id /model/ModernBERT-base/checkpoint-2100

2025-03-01T06:49:31.005543Z  INFO text_embeddings_router: router/src/main.rs:175: Args { model_id: "/mod**/**********-****/**********-*100", revision: None, tokenization_workers: None, dtype: None, pooling: None, max_concurrent_requests: 512, max_batch_tokens: 16384, max_batch_requests: None, max_client_batch_size: 32, auto_truncate: false, default_prompt_name: None, default_prompt: None, hf_api_token: None, hostname: "49770c8319fe", port: 80, uds_path: "/tmp/text-embeddings-inference-server", huggingface_hub_cache: Some("/data"), payload_limit: 2000000, api_key: None, json_output: false, otlp_endpoint: None, otlp_service_name: "text-embeddings-inference.server", cors_allow_origin: None }

thread 'main' panicked at /usr/src/router/src/lib.rs:134:62:
tokenizer.json not found. text-embeddings-inference only supports fast tokenizers: Error("data did not match any variant of untagged enum ModelWrapper", line: 251510, column: 1)
stack backtrace:
   0:     0x6491206bd0e9 - <unknown>
   1:     0x64912028d1c3 - <unknown>
   2:     0x6491206bc9d2 - <unknown>
   3:     0x6491206bcf43 - <unknown>
   4:     0x6491206bc80b - <unknown>
   5:     0x6491206f9a68 - <unknown>
   6:     0x6491206f99c9 - <unknown>
   7:     0x6491206f9e6c - <unknown>
   8:     0x64911fb8d16f - <unknown>
   9:     0x64911fb8d4e5 - <unknown>
  10:     0x64911ff8817b - <unknown>
  11:     0x6491200d5627 - <unknown>
  12:     0x6491200d1bee - <unknown>
  13:     0x64911fffd723 - <unknown>
  14:     0x6491200d5dc4 - <unknown>
  15:     0x7243705bdd90 - <unknown>
  16:     0x7243705bde40 - __libc_start_main
  17:     0x64911ff18855 - <unknown>
  18:                0x0 - <unknown>

[kozistr]

Symbolic links have always worked for me when using Docker. I just did away with the symbolic link, and I got the same bogus error.

docker run
--gpus device=0
-p 8120:80
-v /opt/markdown/model:/model
-e RUST_BACKTRACE=full
--rm tei-modernbert:latest
--model-id /model/ModernBERT-base/checkpoint-2100
2025-03-01T06:49:31.005543Z INFO text_embeddings_router: router/src/main.rs:175: Args { model_id: "/mod**/-/****-*100", revision: None, tokenization_workers: None, dtype: None, pooling: None, max_concurrent_requests: 512, max_batch_tokens: 16384, max_batch_requests: None, max_client_batch_size: 32, auto_truncate: false, default_prompt_name: None, default_prompt: None, hf_api_token: None, hostname: "49770c8319fe", port: 80, uds_path: "/tmp/text-embeddings-inference-server", huggingface_hub_cache: Some("/data"), payload_limit: 2000000, api_key: None, json_output: false, otlp_endpoint: None, otlp_service_name: "text-embeddings-inference.server", cors_allow_origin: None }

thread 'main' panicked at /usr/src/router/src/lib.rs:134:62:
tokenizer.json not found. text-embeddings-inference only supports fast tokenizers: Error("data did not match any variant of untagged enum ModelWrapper", line: 251510, column: 1)
stack backtrace:
0: 0x6491206bd0e9 -
1: 0x64912028d1c3 -
2: 0x6491206bc9d2 -
3: 0x6491206bcf43 -
4: 0x6491206bc80b -
5: 0x6491206f9a68 -
6: 0x6491206f99c9 -
7: 0x6491206f9e6c -
8: 0x64911fb8d16f -
9: 0x64911fb8d4e5 -
10: 0x64911ff8817b -
11: 0x6491200d5627 -
12: 0x6491200d1bee -
13: 0x64911fffd723 -
14: 0x6491200d5dc4 -
15: 0x7243705bdd90 -
16: 0x7243705bde40 - __libc_start_main
17: 0x64911ff18855 -
18: 0x0 -

that's weird. I've just tested the ModernBERT-base from here and it worked.

2025-03-01T07:02:55.572364Z  INFO text_embeddings_router: router/src/main.rs:175: Args { model_id: "./Mod*******-*ase", revision: None, tokenization_workers: None, dtype: Some(Float32), pooling: None, max_concurrent_requests: 512, max_batch_tokens: 16384, max_batch_requests: None, max_client_batch_size: 32, auto_truncate: true, default_prompt_name: None, default_prompt: None, hf_api_token: None, hostname: "0.0.0.0", port: 8888, uds_path: "/tmp/text-embeddings-inference-server", huggingface_hub_cache: None, payload_limit: 2000000, api_key: None, json_output: false, otlp_endpoint: None, otlp_service_name: "text-embeddings-inference.server", cors_allow_origin: None }
2025-03-01T07:02:55.573023Z  WARN text_embeddings_router: router/src/lib.rs:392: The `--pooling` arg is not set and we could not find a pooling configuration (`1_Pooling/config.json`) for this model but the model is a BERT variant. Defaulting to `CLS` pooling.
2025-03-01T07:02:55.639180Z  WARN text_embeddings_router: router/src/lib.rs:184: Could not find a Sentence Transformers config
2025-03-01T07:02:55.639210Z  INFO text_embeddings_router: router/src/lib.rs:188: Maximum number of tokens per request: 8192
2025-03-01T07:02:55.639377Z  INFO text_embeddings_core::tokenization: core/src/tokenization.rs:28: Starting 8 tokenization workers
2025-03-01T07:02:55.714197Z  INFO text_embeddings_router: router/src/lib.rs:230: Starting model backend
2025-03-01T07:02:55.715244Z  INFO text_embeddings_backend_candle: backends/candle/src/lib.rs:239: Starting ModernBert model on Cpu
2025-03-01T07:02:56.509923Z  WARN text_embeddings_router: router/src/lib.rs:258: Backend does not support a batch size > 4
2025-03-01T07:02:56.509950Z  WARN text_embeddings_router: router/src/lib.rs:259: forcing `max_batch_requests=4`
2025-03-01T07:02:56.510797Z  INFO text_embeddings_router::http::server: router/src/http/server.rs:1812: Starting HTTP server: 0.0.0.0:8888
2025-03-01T07:02:56.510826Z  INFO text_embeddings_router::http::server: router/src/http/server.rs:1813: Ready