enhance partition utilities

[wyli]

enhances the partition utilities, part of #816

Status

Ready

Types of changes

• Non-breaking change (fix or new feature that would not break existing functionality).
• Quick tests passed locally by running ./runtests.sh --quick.
• In-line docstrings updated.
• Documentation updated, tested make html command in the docs/ folder.

[wyli]

addressed some corner cases, not sure whether we should use np.random.RandomState instead of seed for shuffle

[Nic-Ma]

addressed some corner cases, not sure whether we should use np.random.RandomState instead of seed for shuffle

Hi @wyli ,

Current shuffle logic can work fine cross multiple processes, I am not sure whether np.random.RandomState also work well?
We are using this API to partition dataset for CacheDataset and SmartCacheDataset in distributed training of Clara.

Thanks.

[wyli]

addressed some corner cases, not sure whether we should use np.random.RandomState instead of seed for shuffle

Hi @wyli ,

Current shuffle logic can work fine cross multiple processes, I am not sure whether np.random.RandomState also work well?

Thanks.

ok, I was not aware of that, but since each process has a different partition, I guess it's fine to share the randomstate instance? would be great to create a few test cases so that I can understand the context. anyway we shouldn't change the global seed here.

[Nic-Ma]

Hi @wyli ,

The multiple process usage is something like this: https://github.com/Project-MONAI/tutorials/blob/master/acceleration/distributed_training/unet_training_smartcache.py#L176
I am not sure whether all the processes can get the exact same shuffle result with np.random.RandomState?
Could you please help double confirm it? If yes, others look good to me, we can merge your PR immediately.
Then I can create a PR to update the examples of distributed training.

Thanks.

[wyli]

Hi @wyli ,

The multiple process usage is something like this: https://github.com/Project-MONAI/tutorials/blob/master/acceleration/distributed_training/unet_training_smartcache.py#L176
I am not sure whether all the processes can get the exact same shuffle result with np.random.RandomState?
Could you please help double confirm it? If yes, others look good to me, we can merge your PR immediately.
Then I can create a PR to update the examples of distributed training.

Thanks.

i don’t see any issue