Added LayoutLMv3 #2178

carrycooldude · 2025-03-30T08:19:16Z

Description

This PR fixes the LayoutLMv3 checkpoint conversion script to properly handle different spatial embedding dimensions between the base and large models. The base model uses 128 dimensions for all spatial embeddings, while the large model uses 171 dimensions for x/y coordinates and 170 dimensions for height/width.

Changes Made

Added dynamic detection of spatial embedding dimensions from the Hugging Face model
Implemented padding for smaller embeddings to match the maximum dimension
Updated projection matrices to use consistent dimensions
Added detailed debug output for spatial embedding shapes

Technical Details

The conversion script now:

Detects individual dimensions for x, y, h, w embeddings
Uses the maximum dimension (171 for large model) for all embeddings
Pads smaller embeddings (170) with zeros to match the larger dimension
Creates projection matrices with consistent dimensions

Testing

Successfully converted both base and large models
Verified output shapes match expected dimensions
Confirmed no dimension mismatch errors during conversion

Output Example

divyashreepathihalli · 2025-04-04T21:31:30Z

@carrycooldude That you for the PR - the code structure does not match KerasHub style.
please go through the guide here - https://github.com/keras-team/keras-hub/blob/master/CONTRIBUTING_MODELS.md
Take a look at other model folders.
What would the task model look like?
the preset file contents should be just metadata and kaggle hub path
Can you provide a model code usage example?

sachinprasadhs · 2025-04-25T23:15:21Z

keras_hub/src/models/layoutlmv3/layoutlmv3_backbone_test.py

@@ -0,0 +1,152 @@
+"""Tests for LayoutLMv3 backbone."""


Remove these docstring at the start of the file.

sachinprasadhs · 2025-04-25T23:23:12Z

Adding General code structuring comments.

Add all the files under the model directory only, we don't recommend using sub directories.
We don't encourage using Tensorflow specific operation, like tf. , we make the mode design to support backend agnostic.
The code does not follow the general code format we follow in Keras Hub, I suggest you to refer other model implementations in detail.
Arguments needs to be descriptive, with type of data it accepts and what is the default arguments etc.

Refer any existing model implementations here https://github.com/keras-team/keras-hub/tree/master/keras_hub/src/models

The test cases also should follow the template we are following in the models.

…kend-agnostic

sachinprasadhs

I have added few comments, most of it are general practice which we follow. Incorporate those general suggested changes across all the files.
And remove the files and directory which are not required like env directory.

sachinprasadhs · 2025-04-29T22:04:29Z

examples/layoutlmv3_document_classification.ipynb

@@ -0,0 +1 @@
+


Remove this directory and file

This still needs to be removed

sachinprasadhs · 2025-04-29T22:08:15Z

keras_hub/src/models/__init__.py

@@ -0,0 +1,4 @@
+"""LayoutLMv3 document classifier."""


This file needs to be empty, all the import is handled in keras_hub/api directory and will be automatically generated whenever you run git commit -m "<message>"
Make sure you run pre-commit install for the first time.

sachinprasadhs · 2025-04-29T22:09:53Z

keras_hub/src/models/layoutlmv3/__init__.py

@@ -0,0 +1,15 @@
+from keras_hub.src.models.layoutlmv3.layoutlmv3_backbone import LayoutLMv3Backbone


This file is mainly to register presets, follow other models to understand the format we follow.

sachinprasadhs · 2025-04-29T22:12:40Z

keras_hub/src/models/layoutlmv3/layoutlmv3_backbone.py

+
+    def __init__(
+        self,
+        vocab_size: int = 30522,


Remove type annotation from everywhere, we don't follow type annotation in Keras Hub

Still type annotation needs to be removed

sachinprasadhs · 2025-04-29T22:18:30Z

keras_hub/src/models/layoutlmv3/layoutlmv3_backbone.py

+References:
+- [LayoutLMv3 Paper](https://arxiv.org/abs/2204.08387)
+- [LayoutLMv3 GitHub](https://github.com/microsoft/unilm/tree/master/layoutlmv3)
+"""


This entire doctring needs to be inside the Backbone class

sachinprasadhs · 2025-04-29T22:19:12Z

keras_hub/src/models/layoutlmv3/layoutlmv3_backbone.py

+"""
+
+import os
+from typing import Dict, List, Optional, Tuple, Union


Remove this once type annotation is removed

sachinprasadhs · 2025-04-29T22:20:53Z

keras_hub/src/models/layoutlmv3/layoutlmv3_backbone.py

+
+from .layoutlmv3_tokenizer import LayoutLMv3Tokenizer
+from .layoutlmv3_presets import backbone_presets
+from .layoutlmv3_transformer import LayoutLMv3TransformerLayer


change from relative imports to absolute imports everywhere.

sachinprasadhs · 2025-04-29T22:26:20Z

keras_hub/src/models/layoutlmv3/layoutlmv3_backbone.py

+    maintaining spatial relationships in documents.
+
+    Args:
+        vocab_size: int, defaults to 30522. Size of the vocabulary.


Format for Args we follow is:
vocab_size: int. Size of the vocabulary. Defaults to 30522

This format should be followed for all and make sure it conveys the proper and complete required information.

sachinprasadhs · 2025-04-29T22:28:35Z

keras_hub/src/models/layoutlmv3/layoutlmv3_backbone.py

+    ```
+    """
+
+    presets = backbone_presets


No need of this here.

You can keep the example, but we don't need presets = backbone_presets

sachinprasadhs · 2025-04-29T22:32:18Z

keras_hub/src/models/layoutlmv3/layoutlmv3_backbone.py

+        self.use_rel_pos = use_rel_pos
+        self.rel_pos_bins = rel_pos_bins
+        self.max_rel_pos = max_rel_pos
+        self.spatial_embedding_dim = spatial_embedding_dim


This should come at last.
You can follow below order:

# === Layers === # === Functional Model === # === Config ===

carrycooldude · 2025-05-07T16:01:46Z

@sachinprasadhs any updates on this one?

sachinprasadhs · 2025-05-07T20:25:05Z

Still the review comments are not addressed, could you please fix those before I can suggest any more changes

carrycooldude · 2025-05-08T13:39:02Z

Still the review comments are not addressed, could you please fix those before I can suggest any more changes

I guess I fixed it , can you tell me which are those?

sachinprasadhs

Pointed the comments where previous reviews were not addressed.

Also, remove layoutmv3_env directory

sachinprasadhs · 2025-05-08T20:36:25Z

keras_hub/src/models/layoutlmv3/layoutlmv3_backbone.py

+    ```
+    """
+
+    presets = backbone_presets


You can keep the example, but we don't need presets = backbone_presets

sachinprasadhs · 2025-05-08T20:36:48Z

keras_hub/src/models/layoutlmv3/layoutlmv3_backbone.py

+
+    def __init__(
+        self,
+        vocab_size: int = 30522,


Still type annotation needs to be removed

sachinprasadhs · 2025-05-08T20:37:15Z

keras_hub/src/models/__init__.py

@@ -0,0 +1,4 @@
+"""LayoutLMv3 document classifier."""


sachinprasadhs · 2025-05-08T20:37:36Z

keras_hub/src/models/layoutlmv3/__init__.py

@@ -0,0 +1,15 @@
+from keras_hub.src.models.layoutlmv3.layoutlmv3_backbone import LayoutLMv3Backbone


sachinprasadhs · 2025-05-08T20:38:31Z

keras_hub/src/models/layoutlmv3/layoutlmv3_backbone_test.py

+# Copyright 2024 The Keras Hub Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ==============================================================================
+


remove this

sachinprasadhs · 2025-05-09T17:08:21Z

keras_hub/src/models/layoutlmv3/layoutlmv3_tokenizer.py

+"""LayoutLMv3 tokenizer implementation.
+
+This tokenizer inherits from WordPieceTokenizer and adds LayoutLMv3-specific
+functionality for document understanding tasks.
+
+Example:
+```python
+# Initialize the tokenizer
+tokenizer = LayoutLMv3Tokenizer.from_preset("layoutlmv3_base")
+
+# Tokenize text
+tokens = tokenizer("Hello world!")
+```
+"""
+


Remove this, move the example inside LayoutLMv3Tokenizer if necessary.

sachinprasadhs · 2025-05-09T17:08:42Z

keras_hub/src/models/layoutlmv3/layoutlmv3_tokenizer_test.py

+"""Tests for LayoutLMv3 tokenizer."""
+


Remove this

sachinprasadhs · 2025-05-09T17:09:01Z

keras_hub/src/models/layoutlmv3/layoutlmv3_tokenizer_test.py

+from ..layoutlmv3.layoutlmv3_tokenizer import LayoutLMv3Tokenizer
+


No relative imports

sachinprasadhs · 2025-05-09T17:09:29Z

keras_hub/src/models/layoutlmv3/layoutlmv3_transformer.py

+"""LayoutLMv3 transformer layer implementation.
+
+This module implements the transformer layer used in the LayoutLMv3 model.
+"""
+


Remove this

sachinprasadhs · 2025-05-09T17:09:49Z

keras_hub/src/models/layoutlmv3/layoutlmv3_transformer.py

+from typing import Dict, Optional
+


No need of this

added the files

ae79d15

carrycooldude mentioned this pull request Mar 31, 2025

Add LayoutLMv3 Model #2176

Open

Restructure LayoutLMv3 implementation to match KerasHub style

737f03a

sachinprasadhs reviewed Apr 25, 2025

View reviewed changes

sachinprasadhs added the stat:awaiting response from contributor label Apr 25, 2025

carrycooldude added 2 commits April 27, 2025 12:59

Refactor: Move LayoutLMv3 files to models directory and make code bac…

455a140

…kend-agnostic

refactor: Move LayoutLMv3 files to dedicated directory

d92c8c4

sachinprasadhs requested changes Apr 29, 2025

View reviewed changes

carrycooldude added 2 commits April 30, 2025 13:07

fix: Update LayoutLMv3 init files to follow correct format

0948f95

fix: Update LayoutLMv3 backbone to follow project standards

3c02f78

sachinprasadhs requested changes May 9, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Added LayoutLMv3 #2178

Added LayoutLMv3 #2178

carrycooldude commented Mar 30, 2025

divyashreepathihalli commented Apr 4, 2025

sachinprasadhs Apr 25, 2025

sachinprasadhs commented Apr 25, 2025

sachinprasadhs left a comment

sachinprasadhs Apr 29, 2025

sachinprasadhs May 8, 2025

sachinprasadhs Apr 29, 2025

sachinprasadhs May 8, 2025

sachinprasadhs Apr 29, 2025

sachinprasadhs May 8, 2025

sachinprasadhs Apr 29, 2025

sachinprasadhs May 8, 2025

sachinprasadhs Apr 29, 2025

sachinprasadhs Apr 29, 2025

sachinprasadhs Apr 29, 2025

sachinprasadhs Apr 29, 2025

sachinprasadhs Apr 29, 2025

sachinprasadhs May 8, 2025

sachinprasadhs Apr 29, 2025

carrycooldude commented May 7, 2025

sachinprasadhs commented May 7, 2025

carrycooldude commented May 8, 2025

sachinprasadhs left a comment

sachinprasadhs May 8, 2025

sachinprasadhs May 8, 2025

sachinprasadhs May 8, 2025

sachinprasadhs May 8, 2025

sachinprasadhs May 8, 2025

sachinprasadhs May 9, 2025

sachinprasadhs May 9, 2025

sachinprasadhs May 9, 2025

sachinprasadhs May 9, 2025

sachinprasadhs May 9, 2025

		@@ -0,0 +1,15 @@
		from keras_hub.src.models.layoutlmv3.layoutlmv3_backbone import LayoutLMv3Backbone

		from ..layoutlmv3.layoutlmv3_tokenizer import LayoutLMv3Tokenizer

Added LayoutLMv3 #2178

Are you sure you want to change the base?

Added LayoutLMv3 #2178

Conversation

carrycooldude commented Mar 30, 2025

Description

Changes Made

Technical Details

Testing

Output Example

divyashreepathihalli commented Apr 4, 2025

Choose a reason for hiding this comment

sachinprasadhs commented Apr 25, 2025

sachinprasadhs left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

carrycooldude commented May 7, 2025

sachinprasadhs commented May 7, 2025

carrycooldude commented May 8, 2025

sachinprasadhs left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment