Provide load_seed_checkpoint_only option #1800
Conversation
Seed checkpoint loading requires checkpoint.enable to be True. However, when using seed checkpoints, users typically don't want to save subsequent checkpoints since seed checkpoints serve verification purposes only.
meta-cla
bot
added
the
CLA Signed
There was a problem hiding this comment.
Had one comment. Would be helpful to also update https://github.com/pytorch/torchtitan/blob/main/docs/debugging.md#seed-checkpoint-based-reproducibility accordingly.
torchtitan/config/job_config.py
Outdated
| Could be implemented as a separate script, but this way shares more code. | ||
| """ | ||
|
|
||
| load_seed_checkpoint_only: bool = False |
There was a problem hiding this comment.
It seems this logic could be used / useful not just for seed checkpoint only, but any checkpoint in general? e.g. when we debug uneven FSDP vocab dim sharding
Do you think we can rename it to load_only (and update the comment)?
There was a problem hiding this comment.
I used initial_load_path and initial_load_in_hf parameter a lot, sometimes I only want to load from HF checkpoints and don't want to save checkpoints. I think initial_load_path can be used together with the new parameter together for my use case as well.
I agree we can name is more general.
Seed checkpoint loading requires checkpoint.enable to be True. However, when using seed checkpoints, users typically don't want to save subsequent checkpoints since seed checkpoints serve verification purposes only.