Add integer-based access to MPI_Wtime

[mahermanns]

Problem

MPI provides standardized access to a time source through MPI_Wtime(), however, the returned timestamp is a floating-point number based on seconds since some time in the past. If that time in the past is significantly far in the past, the floating point value loses resolution. Furthermore, most common time sources are integer-based, thus the time information needs to be converted to a floating-point value with additional effort.

Proposal

Provide two additional calls returning integer values for ticks since some time in the past and ticks per second. The time source should be the same as that for MPI_Wtime.

Changes to the Text

See the corresponding pull request.

Impact on Implementations

Implementations need to support the additional two function calls.

Impact on Users

Users can access integer-based timing information, with potentially lower overhead, while still benefiting from the convenient floating-point interface in less time/overhead-critical parts of the code (e.g., printf, write).

References

Tools Ticket: mpiwg-tools/tools-issues#8

[jeffhammond]

I have many objections:

• I do not see how 64-bit floating-point numbers of seconds can lose precision in the scenario you describe. Can you give an example where this happens? If the difference of two very large numbers is a problem, then the implementation is permitted (and should!) start the counter at zero when the job starts.

• Both seconds per tick and ticks per second are meaningless quantities in the context of variable-frequency processors. The duration over which these might be meaningful is less than a millisecond.

• We already have MPI_Wtick. I see no reason to add a new function that returns the same information, albeit inverted and cast to an integer. And, as noted in the previous bullet, MPI_Wtick does not return a well-defined quantity anyways.

• Good time interfaces like POSIX clock_gettime return an integer number of seconds and nanoseconds, rather than "ticks", because nanoseconds are a meaningful term.

• If programmers want to count "ticks" on a machine where this is meaningful (e.g. Blue Gene), they should use the machine-specific implementation. MPI should not attempt to standardize a feature that is not portable.

[bosilca]

Interesting idea. I wondered what accuracy the user can get via the double timer depending on the OS timer accuracy. For us timers (1e-6) it will take about 544 years before losing one us. Going for ns (1e-9) the result is more drastic, it will take 194 days before losing one ns. Thus, counting the time from the MPI job start as proposed by @jeffhammond, will give us 194 days before losing our first nano-second, and from there the accuracy will sharply decline.

[mahermanns]

@jeffhammond I agree that counting the time from the MPI job start does take care of the accuracy issue. Regarding the other objections, I did not necessarily mean clock ticks of a varying frequency processor, but rather the abstract tick of a "clock on the wall", much like MPI_Wtick does not reflect the processor speed at the moment either.

The proposal is not trying to provide a new time source, but rather a complementary interface to the existing time source. As you mention, interfaces such as clock_gettime return time as an integer-based entity, and some performance tools use this timer already as is. The background of this proposal is a larger ticket that I currently only have an tools WG ticket for (mpiwg-tools/tools-issues#11), where the MPI implementation will return time-stamped information to the user/tool. When the tool uses an integer-based time format, it could use this call, if it uses a double-based time format it could use MPI_Wtime as before. If the MPI implementation uses clock_gettime as its source, the tool would avoid the conversion from integer to double and back to integer.

[jeffhammond]

@bosilca Well, I suppose if MPI fault-tolerance works as designed, people might actually attempt to run jobs for 6 months 😮

@mahermanns Indeed, @jdinan corrected my misunderstanding of "tick".

Since clock_gettime is sufficient, why can't tools just use that? In the past, MPI has added wrappers for a wide range of POSIX and ISO language features because we don't assume these are present, but it's lead to quite a bit of ugliness. In many cases, there is a good justification "because (legacy) Fortran", but I recall the MPI_T interface doesn't support Fortran and - for good reason - targets C/C++ code. Are we really concerned that MPI tool developers can't use clock_gettime that we need to provide another trivial wrapper for a POSIX call?

[mahermanns]

@jeffhammond If you take a look at mpiwg-tools/tools-issues#11 and the corresponding branch https://github.com/mpiwg-tools/mpi-standard/tree/issue_11_mpi_t_events, the plan here is that instead of a tool querying clock_gettime or whatever time source they use internally within a callback function, the tool can use a special call to query the time an event happened. The MPI runtime can then either provide the current time (as the tool would have done with its own timer) or provide a time of the past, if the event was buffered internally---all transparent to the tool querying the event's timestamp.

The desire here was to use an integer-based timestamp (i.e., same type as current timing routines in the backend), as that might enable a lower overhead, when we don't need to convert the time into double (and potentially back into an integer, if the tool's timestamps are integer-based). To put the event time source into relation of the tool's other time sources, the tool would need to query reference timestamps at some point (e.g., beginning and end of the measurement).

[jdinan]

To correct an earlier comment by @jeffhammond --

Both seconds per tick and ticks per second are meaningless quantities in the context of variable-frequency processors. The duration over which these might be meaningful is less than a millisecond.

In current Intel processors, the timestamp counter (accessed via the rdtsc instruction) ticks at the nominal processor frequency, regardless of power saving or frequency boosting measures. Other common time sources, e.g. RTC and HPET, are also independent.

Regarding the suggestion that Linux clock_gettime is sufficient, you might be disappointed if you look at how it's implemented: http://linuxmogeb.blogspot.com/2013/10/how-does-clockgettime-work.html (for the spoiler, skip to the conclusion at the end of the article).

[jeffhammond]

Even though I work for Intel, I do not support making decisions about the MPI standard based upon the fact that Intel got this right starting in 2008.

And now that I've seen exactly how much of a pain it is to implement MPI_Wti** properly in Open-MPI, I'm even more reluctant to want to add new features to the standard of a similar nature.

[jdinan]

@mahermanns Is "The number of ticks per second must be constant over the execution of the program" intended to specify that the number of ticks elapsed is monotonic increasing? If so, I'm having trouble convincing myself that this is sufficient. When you make adjustments to the clock to maintain MPI_WTIME_IS_GLOBAL I think you are adjusting the number of seconds that have elapsed, not the number of ticks per second.

A second question -- why tie this new routine to the resolution of MPI_Wtick? If I implement this routine using a different time source (e.g. the processor timestamp counter), I may need to reduce the resolution to match the time source being used by MPI_Wtime.

[mahermanns]

@jdinan The phrase is to ensure that the call always reports the same ticks per second during the run, i.e., a tool can query that at the beginning of a run and does not have to query it again.

Explicit mention of monotonic time is not part of this proposal, as I did not want to overload it (separation of concerns) ... of course, bad things happen to a number of tools when the time is not monotonically increasing and as a tools developer I would like a way to ensure this. I think we talked about this in Aachen and the idea was that the synchronization is only allowed to re-set the clocks to a future time.

[jdinan]

Could you clarify the problem solved by the proposed API? As @bosilca mentioned earlier, a good implementation of MPI_Wtime will take 194 days to lose a nanosecond. Is the problem that MPI_Wtick is not constant? If so, would this new routine indirectly change the semantics of MPI_Wtick?