Question about integration with DeepSpeed-Ulysses

[zigzagcai]

Hi developers,

Thanks for such a great project that can demonstrate the power of newly released features in torch.

When I want to run llama2 model with 128k long sequence, how can we enable it? I have some experience with DeepSpeed-Ulysses, so the question becomes does torchtitan support sequence parallelism in DeepSpeed-Ulysses?

Thanks!

[gnadathur]

@zigzagcai
titan support sequence and context parallelism for ultra long sequence length. cc: @XilunWu , could you share the details on the config.

Oh -- misread this comment, titan supports native FSDP, no deepspeed integration

[XilunWu]

Sequence parallel should be enabled by default if TP is used.

To enable CP, please change the context_parallel_degree (see PR #592 description for details/examples)