Skip to content

Fix EndTime nil pointer when persisting metrics #2259

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 7 commits into from
Jun 18, 2021

Conversation

vishalbollu
Copy link
Contributor

Previously we used to get a snapshot of the batch metrics based on the job EndTime. The EndTime field would be nil if the job didn't complete due to an unhandled worker error. In such cases, controller would encounter a nil pointer error.

The controller has been updated to get a snapshot of the metrics in prometheus based on the current time rather than the EndTime so that even in situations where EndTime is nil, it tries to get the most accurate metrics.


checklist:

  • run make test and make lint
  • test manually (i.e. build/push all images, restart operator, and re-deploy APIs)

@vishalbollu vishalbollu requested a review from miguelvr June 17, 2021 21:38
@vishalbollu vishalbollu changed the title Fix nil pointer in Fix EndTime nil pointer when persisting metrics Jun 17, 2021
Copy link
Collaborator

@miguelvr miguelvr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM after the suggested change

@vishalbollu vishalbollu merged commit 61c4397 into master Jun 18, 2021
@vishalbollu vishalbollu deleted the set-endtime-incomplete-jobs branch June 18, 2021 17:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants