Save the .nn file every checkpoint

[devedse]

Hi all,

Would it be possible to automatically save a .nn file every time a checkpoint is made. That way I can have one PC training until infinity and basically develop the game further using the latest .nn model.

[anupam-142857]

@devedse this sounds like a useful feature. I have added it as a feature request into our backlog.

[devedse]

@anupambhatnagar , what I'm running into currently is that the training process crashes sometimes. Sometimes due to something I can't really pinpoint and also a few times due to an out of memory exception.

Is it possible to manually generate a .nn file from the last snapshot without starting to train? (I noticed some issues when I continued training, but lets leave that for another day)

[anupam-142857]

@ervteng do you have any insights into generating a .nn file from the last saved checkpoint? Due to the training process crashing @devedse would like to extract the model from the saved checkpoint.

[ervteng]

Yes, that can be done pretty easily by running mlagents-learn and adding the --load parameter, with the same run-id, and quitting immediately using Ctrl+C when the agents start training. This should load the latest saved checkpoint and then export to barracuda.

[stale]

This issue has been automatically marked as stale because it has not had activity in the last 14 days. It will be closed in the next 14 days if no further activity occurs. Thank you for your contributions.

[devedse]

Would it be possible to add the logic to save it every checkpoint?

[Tobbse]

I'm wondering if there might be a way to export a .nn file from checkpoints that are not the latest checkpoint?
This would be very useful to me as I would sometimes after a long training session like to use policies at specific points of time during the training.
For example, I have a project that involves some very unstable training, which is likely my own fault. But nevertheless I would like to use the policy given at the checkpoint with the highest reward which is not nessecarily the latest one.

[devedse]

Well, if the tooling would generate a .nn file during creation of a snapshot then your issue would also be tackled :)

[Tobbse]

True, but then we'd end up with a whole bunch of .nn files :)

[devedse]

Well, now we have a whole bunch of snapshot files so I couldn't really see one file making the difference. Besides that a CLI option could be added to enable/disable this feature.