[test] Use LLM API for Nemotron-H correctness test

[tomeras91]

Nemotron-H correctness tests doesn't just check that the completions are as expected, but rather also validates that logprobs aren't too far from reference logprobs. Initially, the test was implemented using a manual generation for-loop, since the LLM API didn't support returning context and generation logprobs.

After PRs #4538 and #4819 added context and generation logits support for the LLM API, it can now be used to compare logprobs as well, removing the need for a manual generation loop. It is also better to use the LLM API in the unittest since that's the standard way to use a model in TRTLLM.

Therefore, this PR merges the old Nemotron-H correctness and LLM API unittests, such that now the LLM API is used in the Nemotron-H correctness test.

Also, the reference generation logprobs were updated, since turns out there was a bug in the position_ids in the old manual generation loop. This further emphasizes why, when possible, standard APIs should be used instead of custom code.

[Copilot]

Copilot reviewed 1 out of 1 changed files in this pull request and generated no comments.

[tomeras91]

FYI @vegaluisjose @suyoggupta

[tomeras91]

/bot run

[tensorrt-cicd]

PR_Github #8337 [ run ] triggered by Bot

[vegaluisjose]

LGTM

[omera-nv]

LGTM

[tensorrt-cicd]

PR_Github #8337 [ run ] completed with state SUCCESS
/LLM/main/L0_MergeRequest_PR pipeline #6039 completed with status: 'FAILURE'

[tomeras91]

/bot run

[tensorrt-cicd]

PR_Github #8358 [ run ] triggered by Bot

[tensorrt-cicd]

PR_Github #8358 [ run ] completed with state FAILURE
/LLM/main/L0_MergeRequest_PR pipeline #6052 completed with status: 'FAILURE'

[tomeras91]

/bot run

[tensorrt-cicd]

PR_Github #8419 [ run ] triggered by Bot

[tomeras91]

/bot run

[tensorrt-cicd]

PR_Github #8440 [ run ] triggered by Bot

[tensorrt-cicd]

PR_Github #8419 [ run ] completed with state ABORTED

[tensorrt-cicd]

PR_Github #8440 [ run ] completed with state FAILURE
/LLM/main/L0_MergeRequest_PR pipeline #6119 completed with status: 'FAILURE'

[tomeras91]

/bot run

[tensorrt-cicd]

PR_Github #8493 [ run ] triggered by Bot

[amitz-nv]

Use of get context & generation logits LGTM.
Note that logprobs support was merged this morning: #4836 - using it can probably save some more code from this test

[tensorrt-cicd]

PR_Github #8493 [ run ] completed with state FAILURE
/LLM/main/L0_MergeRequest_PR pipeline #6155 completed with status: 'FAILURE'

[tomeras91]

/bot run

[tensorrt-cicd]

PR_Github #8530 [ run ] triggered by Bot

[tensorrt-cicd]

PR_Github #8530 [ run ] completed with state FAILURE

[tomeras91]

/bot run

[tensorrt-cicd]

PR_Github #8535 [ run ] triggered by Bot

[tensorrt-cicd]

PR_Github #8535 [ run ] completed with state SUCCESS
/LLM/main/L0_MergeRequest_PR pipeline #6188 completed with status: 'SUCCESS'