[DRAFT][flang] Multi-Image Fortran (MIF) Dialect

[JDPailleux]

Hello,

This PR is a discussion around a dialect to define “coarrray” operations in Flang. This is a draft and a few operations have been proposed to start with and see if they might be suitable.

[bonachea]

@JDPailleux I'm not sure why this was moved from PR #3 without explanation, but the only difference seems to be the branch names. I've copied over my initial high-level comments from #3.

I have plenty of other feedback on the actual content of the dialect, but before diving into that I was hoping for some clarity on the intended scope and name of the dialect.

Is this dialect intended to capture all of Fortran's multi-image functionality (i.e. everything that gets lowered into a PRIF call)? Or is it really intended to just capture coarrays (in which case large sections of the current draft seem out-of-scope)?

[bonachea]

Do Collective Subroutines really belong in this dialect?

These are often conflated during informal conversations, but Collective Subroutines have absolutely nothing to do with coarrays. They are a completely orthogonal multi-image feature. In fact the argument to collective subroutines is even prohibited from being a coindexed object.

I'm not sure these belong in ANY dialect - for most static purposes, collective subroutines are just a regular intrinsic subroutine. IMHO they don't belong lumped into a coarray dialect any more than say, the SQRT intrinsic.

[bonachea]

The STOP and FAIL IMAGE statements and intrinsics NUM_IMAGES, STOPPED_IMAGES, FAILED_IMAGES and IMAGE_STATUS also have nothing to do with coarrays. Neither does SYNC ALL, nor (arguably) all the TEAM support routines.

Perhaps this dialect should be renamed to the multi-image dialect? That name could encompass all the features related to multi-image parallel execution, of which coarrays are an important subset.

[bonachea]

Resolved in 3fef235

[JDPailleux]

Hi @bonachea , following your feedback and what was mentioned during the committee meeting, I renamed the dialect to Multi-Image (MIF) dialect because it make more sense.

[bonachea]

Thanks for the great start on this dialect!

I made many comments/requests. These include may simple suggestions that can probably be applied immediately (I highly recommend using the "commit suggestion" button in the GitHub UI for this purpose, it saves alot of typing and is less error-prone than manual application). There are also a few comments that may require more sweeping changes and possibly further discussion.

I will also likely have more feedback once we start to address some of these issues I've raised.

[bonachea]

Several operations seem to be notably missing from this dialect:

1. non-coarray ALLOCATE
2. MOVE_ALLOC with coarray arguments

I assume these pre-existing serial features already exists as operations elsewhere in HLFIR/MLIR, but they will need to be lowered to PRIF calls when compiling in multi-image mode.

How do we propose to handle such operations?

[bonachea]

Resolved in 3fef235

[JDPailleux]

Hi @bonachea, sorry for the delay.
Thank you for all feedback and sorry for the many careless mistakes and confusions. I've updated the code with some of your comments.

[bonachea]

Hi @JDPailleux - Thanks for all the improvements!

I've made a few additional comments on open threads and added a few more suggestions.

I again highly recommend using the "commit suggestion" button in the GitHub UI for suggestions you agree with: this wonderful button saves alot of typing and is less error-prone than manual application. There is also an "Add suggestion to batch" button that can be used to batch suggestions into a single commit.

Also a few comments from the first review remain unresolved.

[bonachea]

As previously mentioned, ATOMIC_LOGICAL_KIND is not an integer kind. Strictly speaking the two atomic types in Fortran are:

• integer(ATOMIC_INT_KIND) and
• logical(ATOMIC_LOGICAL_KIND)

It may be that flang internally conflates integer and logical (not sure?), but I think at least in descriptions we should try to express the types correctly.

In the suggestion below I'm unsure whether AnyIntegerType will encompass logical(ATOMIC_LOGICAL_KIND), so this might have to be something more generic.

Suggested change

	- `atom`: Shall be a coarray of type integer with kind
	ATOMIC_INT_KIND or ATOMIC_LOGICAL_KIND from module ISO_FORTRAN_ENV
	- `old`, `new` and `compared`: Shall be a scalar integer with kind ATOMIC_LOGICAL_KIND.
	They need to have the same type and kind as `atom`.
	}];
	let arguments = (ins AnyIntegerType:$image_num,
	AnyIntegerType:$atom,
	Arg<AnyIntegerType, "", [MemWrite]>:$old,
	Arg<AnyIntegerType, "", [MemRead]>:$compare,
	Arg<AnyIntegerType, "", [MemRead]>:$new_val,
	Arg<Optional<AnyRefOrBoxType>, "", [MemWrite]>:$errmsg);
	- `atom`: Shall be a coarray of type integer with kind
	ATOMIC_INT_KIND or of type logical with kind ATOMIC_LOGICAL_KIND
	from module ISO_FORTRAN_ENV
	- `old`, `new` and `compared`: Shall be a scalars with the same type and kind as `atom`.
	}];
	let arguments = (ins AnyIntegerType:$image_num,
	AnyIntegerType:$atom,
	Arg<AnyIntegerType, "", [MemWrite]>:$old,
	Arg<AnyIntegerType, "", [MemRead]>:$compare,
	Arg<AnyIntegerType, "", [MemRead]>:$new_val,
	Arg<Optional<AnyRefOrBoxType>, "", [MemWrite]>:$errmsg);

The same comment also applies to mif_AtomicDefineOp and mif_AtomicRefOp, the two other operations that accept both integer or logical arguments.

[bonachea]

co_broadcast is a data movement collective, not a computational collective. It does not perform a "sum" (or any other ALU computation).

Suggested change

	The co_broadcast operation performs the computation of the sum
	across images.
	The co_broadcast operation broadcasts a value across images.

[bonachea]

Suggested change

	The co_sum operation performs the reduction across images.
	The co_reduce operation performs the reduction across images.

[bonachea]

I'm not sure I fully understand your answer.

My main concern is to ensure we faithfully represent the presence of absence of the optional STAT= argument from the user's original code in the dialect, so that when it comes time to lower this operation to a PRIF call you can match the user's choice to provide STAT or not in the corresponding PRIF call.

I should note this detail only matters when an error occurs, and maybe we initially don't care about error behavior. But if we expect to some day provide spec-compliant error behavior, then we probably need to preserve that information somehow in the IR.

[bonachea]

The F23 standard does not actually require the image_set to be non-empty:

but of course you can pass an empty rank-one array of integers, e.g.:

PROGRAM SYNCIMGS
  integer, dimension(1:0) :: empty_array

  SYNC IMAGES(*)
  SYNC IMAGES(THIS_IMAGE())
  SYNC IMAGES([INTEGER ::])
  SYNC IMAGES(empty_array)
  print *, THIS_IMAGE(), size(empty_array)
END

I've just verified this program compiles and works with Cray Fortran. Both flang/trunk and the SiPearl fork accept this program without compile-time errors (and in general one cannot statically prove an array variable is non-empty).

However an empty input array (SYNC IMAGES([INTEGER ::])) means "synchronize with no images" whereas SYNC IMAGES(*) means the opposite "synchronize with all the images". For this reason PRIF makes the image_set optional and uses the absence of the argument to mean *, a state which is syntactically and semantically distinct from a provided empty array.

[bonachea]

OK so in that case for completeness I think we need to introduce a dialect operation to perform that image translation.

The semantics are identical to prif_initial_team_index, but of course we don't have to call it that.

[bonachea]

I definitely understand we are trying to match Fortran and not PRIF here, but in most cases the design of PRIF was determined by what's possible to express in Fortran.

My specific concern here is that the user code might not always match the simple case:

LOCK(coarray[image_num])

This is definitely a common/important case, but in general I believe Fortran also allows more complicated statements like:

LOCK(some_var%some_component(2)%coarray[image_num]%some_component(1,4,7)%lock_var(2))

In other words, the argument to LOCK can be any valid coindexed expression that leads to a LOCK_TYPE variable. The same is true for UNLOCK, also EVENT POST (with EVENT_TYPE) and ATOMIC_* (with appropriate atomic type).

You can even see an example of this in F23 sec C.7.9:

 EVENT POST (worker[1]%free(THIS_IMAGE()))

Does fir_BoxType preserve sufficient information about the full coindexed expression used to reach the appropriate variable? If not then we may eventually have to address the general case more explicitly in the dialect.

[bonachea]

OK I didn't realize there was an LLVM precedent for conflating these two separate statements.

I just wanted to note that while STOP and ERROR STOP have very similar runtime behavior in single-image execution, they have very different runtime behavior in multi-image execution. So we may eventually want this difference to be more visible in the dialect (to control analysis of surrounding code), but I guess it's fine this way for now provided we document it.