Skip to content

WIP: Add smsc endpoints to HAN #12272

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 1 commit into from
Closed

Conversation

lrbison
Copy link
Contributor

@lrbison lrbison commented Jan 24, 2024

PR for discussion on how to add smsc endpoints to HAN for local memory exchange.

Motivation: I have a working POC using XPMEM and HAN to increase alltoall exchange speed, but I would prefer to cache the creation of the XPMEM endpoints rather than creating them new each time in the collective.

Questions:

  • Where to store the endpoints, and how to properly initialize and deconstruct them.
  • What is the difference between mca_coll_han_comm_create_new and mca_coll_han_comm_create?

@lrbison
Copy link
Contributor Author

lrbison commented Jan 29, 2024

Motivating this change is a branch I'm working on to add alltoall (and soon alltoallv) algorithm to HAN.

My testing shows that:

  • SMSC End point creation is to slow to be done on-the-fly, so we need a cache somewhere.
  • XPMEM SMSC module has the lowest latency, beating out CMA and KNEM, even for copy_to/from, but especially using mapping.

Here are the results I have so far:

image

@lrbison lrbison closed this Mar 1, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant