Add CodeGen model #17443

rooa · 2022-05-26T17:29:17Z

What does this PR do?

Adds CodeGen PyTorch model.

Before submitting

Did you read the contributor guideline,
Pull Request section?
This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
Was this discussed/approved via a Github issue or the forum? ==> Discussed with @lvwerra and @patil-suraj.
Did you make sure to update the documentation with your changes? Here are the
documentation guidelines, and
here are tips on formatting docstrings.
Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@lvwerra @patil-suraj @loubnabnl

HuggingFaceDocBuilderDev · 2022-05-26T17:39:37Z

The documentation is not available anymore as the PR was closed or merged.

patil-suraj

Thanks a lot for adding this model @rooa ! The code is looking good already. Left some comments. Specifically:

We should add as much Copied from ... statements as possible
We should remove the manual parallelization logic, more details in the comment below.

Let me know if you have any other questions. I will look into tests after these changes :)

docs/source/en/index.mdx

docs/source/en/model_doc/codegen.mdx

src/transformers/models/codegen/__init__.py

src/transformers/models/codegen/configuration_codegen.py

src/transformers/models/codegen/modeling_codegen.py

patil-suraj

Thanks a lot for addressing the comments, the PR looks almost ready. Just left a comment about tie_word_embeddings.

docs/source/en/model_doc/codegen.mdx

src/transformers/models/codegen/configuration_codegen.py

sgugger

Thanks for adding this new model!
Good for me with @patil-suraj comments.

README.md

src/transformers/models/codegen/modeling_codegen.py

patrickvonplaten

Looks good to me in general. Would be nice if we could give self.bias a better name - think it'd make reading the code much easier

src/transformers/models/codegen/modeling_codegen.py

patrickvonplaten · 2022-06-02T16:46:31Z

src/transformers/models/codegen/modeling_codegen.py

+    ]:
+
+        qkv = self.qkv_proj(hidden_states)
+        # TODO(enijkamp): factor out number of logical TPU-v4 cores or make forward pass agnostic


(out of curiosity) what does the comment mean here?

@patil-suraj why resolve here?

tests/models/codegen/test_modeling_codegen.py

patil-suraj

Thanks a lot @rooa ! This looks good for merge now, once @patrickvonplaten's comment is addressed.

src/transformers/models/codegen/modeling_codegen.py

patrickvonplaten

Some things to clean up before merging:

Some tests are failing
We should add a truncate_before_pattern function arg that takes a list of patterns before which we truncate. I think it's important to stay flexible here

src/transformers/models/codegen/tokenization_codegen_fast.py

patrickvonplaten · 2022-06-07T23:00:08Z

src/transformers/models/codegen/tokenization_codegen.py

+    return pairs
+
+
+class CodeGenTokenizer(PreTrainedTokenizer):


Think we should add some more # Copied from ... statements here

@patil-suraj why resolve here if there hasn't been an answer or change?

My bad. Here we add an extra method truncate in the tokenizer , so didn't add the # Copied from ... statement.

src/transformers/models/codegen/tokenization_codegen.py

tests/models/codegen/test_tokenization_codegen.py

src/transformers/models/codegen/tokenization_codegen.py

sam-h-bean · 2022-06-22T18:08:48Z

Hey @patil-suraj @rooa you should go fetch upstream on your fork. There were some test fixes that I think you are missing which is causing the red exes that no one likes to see. I actually would love to use this but I can't because this PR is not merged yet!

…to add_codegen

src/transformers/models/codegen/modeling_codegen.py

src/transformers/models/codegen/tokenization_codegen.py

tests/models/codegen/test_modeling_codegen.py

tests/models/codegen/test_tokenization_codegen.py

patrickvonplaten

Good to merge for me!

Co-authored-by: Patrick von Platen <[email protected]>

…to add_codegen

patil-suraj · 2022-06-24T15:10:30Z

Merging now! Thanks a lot @rooa for working on this and being patient with the review and tests.

patil-suraj reviewed May 30, 2022

View reviewed changes

patil-suraj requested review from patrickvonplaten and sgugger June 1, 2022 12:49

patil-suraj reviewed Jun 1, 2022

View reviewed changes

docs/source/en/model_doc/codegen.mdx Outdated Show resolved Hide resolved

docs/source/en/model_doc/codegen.mdx Outdated Show resolved Hide resolved

src/transformers/models/codegen/configuration_codegen.py Show resolved Hide resolved

sgugger approved these changes Jun 1, 2022

View reviewed changes

README.md Outdated Show resolved Hide resolved

README.md Outdated Show resolved Hide resolved

src/transformers/models/codegen/modeling_codegen.py Show resolved Hide resolved

patrickvonplaten reviewed Jun 2, 2022

View reviewed changes

src/transformers/models/codegen/modeling_codegen.py Outdated Show resolved Hide resolved

patrickvonplaten reviewed Jun 2, 2022

View reviewed changes

src/transformers/models/codegen/modeling_codegen.py Outdated Show resolved Hide resolved

patrickvonplaten approved these changes Jun 2, 2022

View reviewed changes

patil-suraj approved these changes Jun 3, 2022

View reviewed changes

src/transformers/models/codegen/modeling_codegen.py Outdated Show resolved Hide resolved

patil-suraj requested a review from patrickvonplaten June 7, 2022 16:26

patrickvonplaten reviewed Jun 7, 2022

View reviewed changes

src/transformers/models/codegen/tokenization_codegen.py Outdated Show resolved Hide resolved

gongel reviewed Jun 21, 2022

View reviewed changes

src/transformers/models/codegen/tokenization_codegen.py Outdated Show resolved Hide resolved

rooa and others added 16 commits June 22, 2022 13:28

Add CodeGen model

0300cc1

Add missing key and switch order of super()

3bd802e

Fix torch.ones init with uint8 instead of bool

24b8389

Address comments: copy statements and doc

96217c9

update tests

37a174f

remove old model parallel

7935424

fix batch gen tests

ee340b3

fix batch gen test

a7ee6bf

update test_gpt2_sample_max_time

3f53403

fix codgen test and revert gpt2 test change

79b988c

Fix incorrect tie_word_embedding value, typo, URL

0d7eda5

Fix model order in README and styling

4f0022d

Reorder model list alphabetically

37eeb99

Set tie_word_embedding to False by default

9c08c1b

Apply suggestions from code review

a0b4fda

Better attn mask name & remove attn masked_bias

e783f75

patil-suraj added 8 commits June 22, 2022 13:33

quality

d881c05

doc tokenizer

0eb2112

fix-copies

b09ff84

add CodeGenTokenizer in converter

12e4509

make truncation optional

d0a7883

add test for truncation

5689a00

add copyright

7c1758c

fix-copies

d2bd258

rooa and others added 3 commits June 22, 2022 11:11

Merge branch 'huggingface:main' into add_codegen

5ffe8c9

fix fast tokenizer decode

0dad72b

Merge branch 'add_codegen' of https://github.com/rooa/transformers in…

a112244

…to add_codegen

patrickvonplaten reviewed Jun 23, 2022

View reviewed changes

src/transformers/models/codegen/modeling_codegen.py Show resolved Hide resolved

patrickvonplaten reviewed Jun 23, 2022

View reviewed changes

src/transformers/models/codegen/tokenization_codegen.py Outdated Show resolved Hide resolved

patrickvonplaten reviewed Jun 23, 2022

View reviewed changes

tests/models/codegen/test_modeling_codegen.py Show resolved Hide resolved

patrickvonplaten reviewed Jun 23, 2022

View reviewed changes

tests/models/codegen/test_modeling_codegen.py Show resolved Hide resolved

patrickvonplaten reviewed Jun 23, 2022

View reviewed changes

tests/models/codegen/test_tokenization_codegen.py Show resolved Hide resolved

patrickvonplaten approved these changes Jun 23, 2022

View reviewed changes

rooa mentioned this pull request Jun 23, 2022

add web demo/model to Huggingface salesforce/CodeGen#2

Closed

rooa and others added 5 commits June 23, 2022 16:38

Merge branch 'huggingface:main' into add_codegen

49c6868

Update src/transformers/models/codegen/tokenization_codegen.py

a24f576

Co-authored-by: Patrick von Platen <[email protected]>

Merge branch 'main' of https://github.com/huggingface/transformers in…

5839839

…to add_codegen

increase vocab_size in tests

8930e49

Merge branch 'main' of https://github.com/huggingface/transformers in…

f29ef0a

…to add_codegen

patil-suraj merged commit d6b6fb9 into huggingface:main Jun 24, 2022

SaulLu mentioned this pull request Jun 29, 2022

codegen-16B-mono (Salesforce) fails to load tokenizer and model #17954

Closed

4 tasks

Sai-Suraj-27 mentioned this pull request Aug 16, 2024

fix: Fixed CodeGenTokenizationTest::test_truncation failing test #32850

Merged

5 tasks

Add CodeGen model #17443

Add CodeGen model #17443

Uh oh!

Conversation

rooa commented May 26, 2022 • edited by loubnabnl Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Before submitting

Who can review?

Uh oh!

HuggingFaceDocBuilderDev commented May 26, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

patil-suraj left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

patil-suraj left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sgugger left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

patrickvonplaten left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

patrickvonplaten Jun 2, 2022

Choose a reason for hiding this comment

Uh oh!

patrickvonplaten Jun 23, 2022

Choose a reason for hiding this comment

Uh oh!

Uh oh!

patil-suraj left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

patrickvonplaten left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

patrickvonplaten Jun 7, 2022

Choose a reason for hiding this comment

Uh oh!

patrickvonplaten Jun 23, 2022

Choose a reason for hiding this comment

Uh oh!

patil-suraj Jun 24, 2022

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sam-h-bean commented Jun 22, 2022

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rooa commented May 26, 2022 •

edited by loubnabnl

Loading

HuggingFaceDocBuilderDev commented May 26, 2022 •

edited

Loading

sgugger left a comment •

edited

Loading