Skip to content

Workflow timeout a better scenario #208

Closed
@DavidGOrtega

Description

@DavidGOrtega

Github Actions max workflow timeout is 72 hours. This is a very limited time for training a model.
Depending on how the vendor's runners handle this a nice way to handle this should be restarting the workflow to be able to get the green light.

However, two possible scenarios comes to mind (if not more)

  1. The runner is able to finish the job (training)
  2. The runner stops since the workflow fails

In both cases the solution would be a mechanism to restart the workflow having a cache to save the intermediate models/state.

This is related to #174 and #161

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions