Skip to content

btl: introduce flag MCA_BTL_FLAGS_RDMA_REMOTE_COMPLETION #9694

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Nov 24, 2021
Merged

btl: introduce flag MCA_BTL_FLAGS_RDMA_REMOTE_COMPLETION #9694

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 23 additions & 0 deletions opal/mca/btl/base/btl_base_am_rdma.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1138,5 +1138,28 @@ int mca_btl_base_am_rdma_init(mca_btl_base_module_t *btl)
OBJ_CONSTRUCT(&default_module, mca_btl_base_am_rdma_module_t);
}

/* This section check whether we can claim support of remote completion.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think you need this first line of the comment. But I do love the detailed comment!

*
* In terms of remote completion, we are mainly interested in put and atomic ops,
* because get, atomics fops and atomic cswap support remote completion by their nature.
*
* For active message put (AM put), the target side will send a response, and the initiator
* side will wait for the response to complete the put operation. Thus if AM put is based on send,
* it support remote completion. (If AM put is based on get, it does not support remote
* completion because the target side does not wait for get's completion to send response).
*
* active message RDMA/atomics does not implement atomic ops. User was suppose to
* use atomic fops (unless the btl support atomic ops natively).
*
* In all, the conditions for AM rdma to claim support of remote completion are:
* 1. AM put is enabled (which means the btl does not support put)
* 2. AM put does not use get (so it must use send)
* 3. btl does not have native atomics ops support.
*/
if ((btl->btl_flags & MCA_BTL_FLAGS_PUT_AM) && !mca_btl_base_rdma_use_rdma_get(btl) &&
!(btl->btl_flags & MCA_BTL_FLAGS_ATOMIC_OPS)) {
btl->btl_flags |= MCA_BTL_FLAGS_RDMA_REMOTE_COMPLETION;
}

return OPAL_SUCCESS;
}
10 changes: 10 additions & 0 deletions opal/mca/btl/btl.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -263,6 +263,16 @@ typedef uint8_t mca_btl_base_tag_t;
/* The BTL has active-message based atomics */
#define MCA_BTL_FLAGS_ATOMIC_AM_FOP 0x400000

/** Ths BTL's RDMA/atomics operation supports remote completion.
* When the BTL reported the completion of a RDMA/atomic operation
* on the initator side, the operation also finished on the target side.
*
* Note, this flag is for put and atomic write operations. Operations
* like get, atomic fetch and atomic swap support remote
* completion by nature.
*/
#define MCA_BTL_FLAGS_RDMA_REMOTE_COMPLETION 0x800000

/* Default exclusivity levels */
#define MCA_BTL_EXCLUSIVITY_HIGH (64 * 1024) /* internal loopback */
#define MCA_BTL_EXCLUSIVITY_DEFAULT 1024 /* GM/IB/etc. */
Expand Down
4 changes: 3 additions & 1 deletion opal/mca/btl/ofi/btl_ofi_module.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -390,8 +390,10 @@ mca_btl_ofi_module_t *mca_btl_ofi_module_alloc(int mode)
module->super.btl_register_mem = mca_btl_ofi_register_mem;
module->super.btl_deregister_mem = mca_btl_ofi_deregister_mem;

/* btl/ofi support remote completion because it required FI_DELIVERY_COMPLETE capability
*/
module->super.btl_flags |= MCA_BTL_FLAGS_ATOMIC_FOPS | MCA_BTL_FLAGS_ATOMIC_OPS
| MCA_BTL_FLAGS_RDMA;
| MCA_BTL_FLAGS_RDMA | MCA_BTL_FLAGS_RDMA_REMOTE_COMPLETION;

module->super.btl_atomic_flags = MCA_BTL_ATOMIC_SUPPORTS_ADD | MCA_BTL_ATOMIC_SUPPORTS_SWAP
| MCA_BTL_ATOMIC_SUPPORTS_CSWAP
Expand Down
2 changes: 2 additions & 0 deletions opal/mca/btl/self/btl_self_component.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -107,6 +107,8 @@ static int mca_btl_self_component_register(void)
mca_btl_self.btl_rdma_pipeline_frag_size = INT_MAX;
mca_btl_self.btl_min_rdma_pipeline_size = 0;
mca_btl_self.btl_flags = MCA_BTL_FLAGS_RDMA | MCA_BTL_FLAGS_SEND_INPLACE | MCA_BTL_FLAGS_SEND;
/* for self, remote completion is local completion */
mca_btl_self.btl_flags |= MCA_BTL_FLAGS_RDMA_REMOTE_COMPLETION;
mca_btl_self.btl_bandwidth = 100;
mca_btl_self.btl_latency = 0;
mca_btl_base_param_register(&mca_btl_self_component.super.btl_version, &mca_btl_self);
Expand Down
1 change: 1 addition & 0 deletions opal/mca/btl/ugni/btl_ugni_component.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -469,6 +469,7 @@ static int btl_ugni_component_register(void)
mca_btl_ugni_module.super.btl_flags = MCA_BTL_FLAGS_SEND | MCA_BTL_FLAGS_RDMA
| MCA_BTL_FLAGS_SEND_INPLACE | MCA_BTL_FLAGS_ATOMIC_OPS
| MCA_BTL_FLAGS_ATOMIC_FOPS;
mca_btl_ugni_module.super.btl_flags |= MCA_BTL_FLAGS_RDMA_REMOTE_COMPLETION;
mca_btl_ugni_module.super.btl_atomic_flags = MCA_BTL_ATOMIC_SUPPORTS_ADD
| MCA_BTL_ATOMIC_SUPPORTS_AND
| MCA_BTL_ATOMIC_SUPPORTS_OR
Expand Down