config/opal_setup_java.m4: Improve JDK Tool Path Resolution on OS X/macOS #5015

RandomDSdevel · 2018-04-04T23:33:14Z

This is just a first pass at this, as I expect my changes to be thoroughly reviewed. I will, of course, address feedback as it comes in.

ompiteam-bot · 2018-04-04T23:33:15Z

Can one of the admins verify this patch?

kawashima-fj · 2018-04-05T06:16:30Z

ok to test

The issue #5000 identified two problems which are related but independent (JDK 10 and macOS).
The PR #5001 addresses the JDK 10 problem and this PR addresses the macOS problem.
I'm not familiar with macOS. Can someone OMPI developer confirm this patch?

ggouaillardet · 2018-04-06T03:54:58Z

@RandomDSdevel can you please describe what this PR is fixing ?

#5001 has been merged into master, and I will PR shortly vs the release branches.
Since javac -h ... is working on JDK 10, we now do not even care if javah is available/unavailable/working/busted, so I am wondering whether this PR is still needed.

RandomDSdevel · 2018-04-06T17:04:09Z

Whoops, accidentally forgot to make sure AC_MSG_CHECKING([OSX locations]) from config/opal_setup_java.m4's OS X/macOS configuration code path was still in there after I was done messing with that. Hold on, fixing that'll only take a sec…

RandomDSdevel · 2018-04-06T17:38:25Z

Sure thing, @ggouaillardet; as I understand things, Java headers and executables can be found in a few different ways:

Under /usr/, where:
- counterparts to headers don't seem to exist, at least as far as I can see at the moment
- executables live under /usr/bin/ as symbolic links to shim executables hosted in /System/Library/Frameworks/JavaVM.framework/Versions/Current/Commands/
Under /System/Library/Frameworks/JavaVM.framework/Versions/Current/, where:
- headers live in /System/Library/Frameworks/JavaVM.framework/Versions/Current/Headers/
- Apple-distributed executables exist under /System/Library/Frameworks/JavaVM.framework/Versions/Current/Commands/ as shims that redirect to the real ones hosted in /Library/Java/JavaVirtualMachines/jdk-$JDK_VERSION.jdk/Contents/Home/bin/ for valid $JDK_VERSIONs. (I have reason to suspect that these are relics left over from when Apple used to ship its own JDK variant.)
Under /Library/Java/JavaVirtualMachines/jdk-$JDK_VERSION.jdk/Contents/Home/ (this is typically the result of calling /usr/libexec/java_home, a command that is also just the beginning of a wild goose chase of indirections) for valid $JDK_VERSIONs, where:
- headers live in /Library/Java/JavaVirtualMachines/jdk-$JDK_VERSION.jdk/Contents/Home/include/
- Oracle-distributed executables live in /Library/Java/JavaVirtualMachines/jdk-$JDK_VERSION.jdk/Contents/Home/bin/.

From what I understand of config/opal_setup_java.m4 and the points listed above, Open MPI's OS X/macOS configuration code was previously:

Only structured to handle looking for Java headers in either of locations 2 or 3 listed above, favoring the latter.
Hard-coded to pick up Java executables from out of location 1.

With this PR, the same make code should behave less inappropriately and now:

Avoid reaching into location 1 listed above entirely for everything except /usr/libexec/java_home.
Handle looking for Java headers and executables in either of locations 2 or 3 listed above, favoring the latter.
Handle the …/Headers//…/include/ and …/Commands//…/bin/ distinction correctly by search location.

As such, I believe Open MPI's configuration system should behave along the lines explained here even though #5001 has already been merged, as this would actually be more correct. Please let me know, however, if I've missed anything either in this analysis or, again, in this PR.

RandomDSdevel · 2018-04-16T23:23:31Z

@ggouaillardet and/or @bwbarrett: Does anything else remain to be done on this PR on my side?

RandomDSdevel · 2018-04-23T01:53:32Z

I've been keeping things rebased against the latest commits to master locally, if that helps any.

bwbarrett · 2018-04-23T15:52:28Z

@kawashima-fj and @ggouaillardet, can one of you comment on @RandomDSdevel's questions and if it looks right to you, merge it into master?

kawashima-fj · 2018-04-24T01:23:10Z

@RandomDSdevel Could you remove trailing spaces at the end of lines and update the PR (git commit --amend/git push --force)?

Though I cannot understand the need of this change fully because of my macOS knowledge shortage, the intent of this change and the change itself seem reasonable. I'll merge it once the PR is updated.

kawashima-fj · 2018-04-24T01:34:19Z

@RandomDSdevel, Without this change, who suffers the problem? All OS X/macOS users who want Java binding? Or users with a particular JDK installation? I want to know impact of this problem.

ggouaillardet · 2018-04-24T02:30:06Z

config/opal_setup_java.m4

                     [AC_MSG_RESULT([found ($opal_java_dir)])
                      opal_java_found=1
-                      with_jdk_headers=$opal_java_dir
-                      with_jdk_bindir=/usr/bin],
+                      if test "$opal_java_dir" = /System/Library/Frameworks/JavaVM.framework/Versions/Current; then


instead of testing the value of $opal_java_dir, could we check the existence of $opal_java_dir/{include,bin} ?

@ggouaillardet:

Not unless you want to drop support for Apple's Java shims. (I'm not sure what their status is, but Apple stopped shipping its own derivative Java distribution after Java…6, I believe, so odds are they've actually been deprecated since some time after that…?) I noted in my toolpath analysis above that the header and executable directories under $opal_java_dir differ in naming scheme depending on what that variable's contents are, did I not? In any case, I felt it safer to leave messing with that to the future.

@RandomDSdevel First of all, many thanks for your patience. This is a complex topic; forgive us for being so pedantic trying to understand it.

I would actually love to see much of config/opal_setup_java.m4: Improve JDK Tool Path Resolution on OS X/macOS #5015 (comment) turned into a comment in the .m4 file. This is complex stuff; putting all that hard-won knowledge down as a comment in the code could save someone from making a mistake in the future while "fixing" something (but not having the background knowledge of why the code currently is the way it is).

I think what @ggouaillardet was suggesting was not incongruent with your analysis. I think he was suggesting something like this (typed directly here in github / not tested -- YMMV):

if test -d "$opal_java_dir/Headers" && test -d "$opal_java_dir/Commands"; then with_jdk_headers="$opal_java_dir/Headers" with_jdk_bindir="$opal_java_dir/Commands" elif test -d "$opal_java_dir/include" && test -d "$opal_java_dir/bin"; then with_jdk_headers="$opal_java_dir/include" with_jdk_bindir="$opal_java_dir/bin" else AC_MSG_WARN([No recognized directory structure found under $opal_java_dir]) AC_MSG_ERROR([Cannot continue]) fi

This simply prevents us from having the hard-coded ".../Current" directory name in 2 places. It also is marginally more defensive in that it also checks for the presence of include and bin if the Headers and Commands checks fail.

Is there a reason that would drop support for Apple's Java shims? It looks like the same logic to me -- is there a subtle implication that I'm missing?

@jsquyres:

@RandomDSdevel First of all, many thanks for your patience. This is a complex topic; forgive us for being so pedantic trying to understand it.

⋮

     You're welcome, and no problem.

⋮

I would actually love to see much of #5015 (comment) turned into a comment in the .m4 file. This is complex stuff; putting all that hard-won knowledge down as a comment in the code could save someone from making a mistake in the future while "fixing" something (but not having the background knowledge of why the code currently is the way it is).

⋮

     Agreed, but…that would be rather a lot of text, don't you think? I could just put URIs pointing to the relevant discussion into such a comment instead; that would be a lot shorter.

⋮

I think what @ggouaillardet was suggesting was not incongruent with your analysis. I think he was suggesting something like this (typed directly here in github / not tested -- YMMV):

if test -d "$opal_java_dir/Headers" && test -d "$opal_java_dir/Commands"; then with_jdk_headers="$opal_java_dir/Headers" with_jdk_bindir="$opal_java_dir/Commands" elif test -d "$opal_java_dir/include" && test -d "$opal_java_dir/bin"; then with_jdk_headers="$opal_java_dir/include" with_jdk_bindir="$opal_java_dir/bin" else AC_MSG_WARN([No recognized directory structure found under $opal_java_dir]) AC_MSG_ERROR([Cannot continue]) fi

This simply prevents us from having the hard-coded ".../Current" directory name in 2 places. It also is marginally more defensive in that it also checks for the presence of include and bin if the Headers and Commands checks fail.

⋮

     Ah, OK; that does seem a bit nicer, so I'll put that in place now.

⋮

Is there a reason that would drop support for Apple's Java shims? It looks like the same logic to me -- is there a subtle implication that I'm missing?

     Sorry, I should have repeated some additional context from downstream in Homebrew/homebrew-core#26009 (comment) here; here it is:

… (Per an old message which I found on one of the OpenJDK mailing lists via this Ask Different answer when I was looking around to see what information I could jump off from to troubleshoot [a possible lead on this issue]:

⋮

Do NOT remove any content in the JavaVM.framework. Those items are required by Java 7, 8, 9+ as well as Java SE 6. No modern version of OS X has a Java JDK inside there anyway.

⋮

⋮

However, if you run:

…$ sudo -i ⋮ # Enter password… …# find / -type f -perm +111 -exec file -p {} \; -exec head -c 1 /dev/zero \; | xargs -0 -n 1 sh -c 'echo "$@" | head -n 1' -- | grep -v x86_64 | grep -E Mach-O.+i386 ⋮ # Wait for that to finish… …# logout

(at least on my system, which has OS X v10.11.6 'El Capitan,' build 15G20015, installed,) then you'll notice that the resulting long list includes the following output:

⋮ /System/Library/Frameworks/JavaVM.framework/Versions/A/Resources/.compatibility/libIASupport.jnilib: Mach-O dynamically linked shared library i386 ⋮

There might also be some 32-bit components elsewhere on Macs upon which /System/Library/Frameworks/JavaMV.framework might additionally depend, but I haven't looked into what those might be. (I also haven't asked anywhere to see if this might have changed in versions of OS X/macOS newer than the one I have installed on my system.)
     What implications this factoid might have, if any, for the future of Apple's Java shims, I don't know, but all 32-bit OS X/macOS binaries currently have a limited lifespan, as noted in this notice on Apple's Developer portal:

64-bit Requirement for Mac Apps

June 28, 2017

At WWDC 2017, we announced new apps submitted to the Mac App Store must support 64-bit starting January 2018, and Mac app updates and existing apps must support 64-bit starting June 2018. If you distribute your apps outside the Mac App Store, we highly recommend distributing 64-bit binaries to make sure your users can continue to run your apps on future versions of macOS. macOS High Sierra will be the last macOS release to support 32-bit apps without compromise.

Whether Apple will use this aa an excuse to remove their vestigial Java shims entirely or choose to update that lone 32-bit component inside it (and any the entire shim framework might depend on) to be 64-bit executables, I again don't know. Relying solely on users only having instances of Oracle's official JDK installed on their system might, however, turn out to be a safer idea in the long run, but I'm deeming that outside the scope of this PR for now.

@jsquyres:

I've addressed most of your review comments with a new revision of this PR's sole commit pushed to its staging branch on my Open MPI repository fork. I left out adding an additional comment as you asked for now, though, pending any guidance you might have on what you might want it to look like (just something short pointing through one or more link vs. a copied-and-pasted variant of my giant wall of text from #5015 (comment).) (I should also note that said wall of text was adapted from an e-mail that I posted to [email protected] and mentioned in my commit message.)

RandomDSdevel · 2018-04-25T01:43:44Z

@kawashima-fj:

@RandomDSdevel Could you remove trailing spaces at the end of lines and update the PR (git commit --amend/git push --force)?

⋮

Does.

Edit: Did a final rebase but nearly forgot to remove trailing space as requested (at least I think I got them all…)

RandomDSdevel · 2018-04-25T01:59:04Z

@kawashima-fj:

@RandomDSdevel…

⋮

Though I cannot understand the need of this change fully because of my macOS knowledge shortage, the intent of this change and the change itself seem reasonable. I'll merge it once the PR is updated.

@RandomDSdevel, Without this change, who suffers the problem? All OS X/macOS users who want Java binding? Or users with a particular JDK installation? I want to know impact of this problem.

Without this change, the Open MPI build system mixes up Apple's Java shims with users' official Oracle Java installations as I explained above. With this change, #5000 would have been caught during build configuration instead of later on in the build process much closer to its end when an attempt to invoke javah was made and failed due to the fact that said tool is no longer present in Java 10. (Again, #5001 renders this PR more a correctness issue with respect to path handling more than anything else, but I still feel like it's somewhat worthwhile to pursue for that reason alone on its own.)

RandomDSdevel · 2018-04-25T20:19:24Z

@jsquyres: #5015 (comment)

kawashima-fj · 2018-04-26T10:18:57Z

config/opal_setup_java.m4

               AS_IF([test -d $opal_java_dir],
                     [AC_MSG_RESULT([found ($opal_java_dir)])
                      opal_java_found=1
-                      with_jdk_headers=$opal_java_dir
-                      with_jdk_bindir=/usr/bin],
+                      if test "$opal_java_dir/Headers" && test -d "$opal_java_dir/Commands"; then


Did you forget -d before "$opal_java_dir/Headers"?

Whoops, clumsy me; it appears I did. Fix incoming…

jsquyres · 2018-04-26T19:46:27Z

That looks good to me (i.e., the -d fix).

As for the comment, I honestly don't mind long comments. I leave it to your discretion: a long comment, or a pointer to #5015 (comment).

Either way, this overall issue should be cited in the comment (i.e., #5015), since the overall discussion may be useful in the future. But that one specific comment (or something derived from it) is the crux of the design philosophy, and should specifically be called out, either by reference or by value.

Thanks!

…acOS. Also avoid picking up Apple's Java shims via the sym. links to them in `/usr/bin` on systems where any one of them could possibly exhibit behavior that is erratic and, to some extent, likely to be incorrect nowadays (cf.: - https://www.mail-archive.com/[email protected]/msg20551.html - #5015 (comment) - the last part of #5015 (comment) - #5015 (comment) for more detailed context.) Works alongside #5001 to close #5000. Signed-off-by: Bryce Glover <[email protected]>

RandomDSdevel · 2018-04-26T21:18:06Z

How's that for a first pass at adding documentation about this PR's associated changes to the makefile source, @jsquyres? I'm sure the wording could see some improvement, but I think I covered everything sufficiently with it.

RandomDSdevel · 2018-04-26T23:46:45Z

Looks like there might have been a CI infrastructure hiccup…? Can somebody restart the build, please?

bwbarrett · 2018-04-26T23:52:12Z

bot:ompi:retest

Looks like the same bad merge issue

kawashima-fj · 2018-04-27T01:14:36Z

@RandomDSdevel Thanks for your update and explanation!

I understand that users who are affected by this problem are:

Older versions of Mac OS X (Apple JDK is installed), and
Oracle JDK is manually(?) installed, and
Open MPI Java binding is enabled

RandomDSdevel · 2018-04-27T01:47:08Z

Thanks, @bwbarrett!

RandomDSdevel · 2018-04-27T01:50:44Z

@kawashima-fj:

@RandomDSdevel Thanks for your update and explanation!

I understand that users who are affected by this problem are:

Older versions of Mac OS X (Apple JDK is installed), and

Oracle JDK is manually(?) installed, and

Open MPI Java binding is enabled

Not exactly: those shims stay in place even if you don't have an Apple JDK installed, and they're still around on newer systems. Otherwise that's correct aside from how it interacted with the old path-detection logic, which is now fixed, and how things were before #5001.

kawashima-fj · 2018-04-27T02:44:16Z

@RandomDSdevel, sorry, I misunderstood. Thanks for your clarification.

RandomDSdevel · 2018-04-30T20:23:07Z

@kawashima-fj, @ggouaillardet, @bwbarrett, and/or @jsquyres:

Can we get this backported from master to the v3.0.x branch, please? I'm preparing a PR to apply both #5001, as backported by #5029, and this PR downstream in Homebrew in lieu of v3.0.2 being released soon, but don't see a similar backport PR/commit for this one.

…acOS. Also avoid picking up Apple's Java shims via the sym. links to them in `/usr/bin` on systems where any one of them could possibly exhibit behavior that is erratic and, to some extent, likely to be incorrect nowadays (cf.: - https://www.mail-archive.com/[email protected]/msg20551.html - open-mpi#5015 (comment) - the last part of open-mpi#5015 (comment) - open-mpi#5015 (comment) for more detailed context.) Works alongside open-mpi#5001 to close open-mpi#5000. Signed-off-by: Bryce Glover <[email protected]> (cherry picked from commit open-mpi/ompi@8c32cd8)

ggouaillardet · 2018-05-01T00:52:49Z

I made #5118 and #5119 for the v3 branches

ggouaillardet · 2018-05-01T00:53:28Z

@RandomDSdevel note v3.1.0 might be released before 3.0.2

RandomDSdevel · 2018-05-01T01:52:18Z

@ggouaillardet:

Funny how that works sometimes, eh?

RandomDSdevel · 2018-05-03T21:32:37Z

@ggouaillardet:

I'm still looking to downstream #5119 to Homebrew as a patch to its distribution of v3.0.1 until v3.0.2 has finished baking, but said PR still hasn't received the review you requested from @kawashima-fj.

kawashima-fj · 2018-05-07T02:01:53Z

Sorry, I'm late. There was 9-day holiday in Japan. I reviewed and approved v3.0/v3.1 PRs.

Merging this commit to v2.x branch fails because it requires the commit ff5af73 beforehand. But it does not have signed-off. Can I cherry-pick it without signed-off?

ggouaillardet · 2018-05-07T02:26:52Z

@kawashima-fj 8c32cd8 is the commit that should be backported (and it was signed off)

kawashima-fj · 2018-05-07T03:50:07Z

@ggouaillardet 8c32cd8 is based on ff5af73 (AS_IF([test -x /usr/libexec/java_home], ...) is added).

git cherry-pick 8c32cd8 in v2.x branch fails because of a merge conflict. I have five options.

Cherry-pick ff5af73 beforehand -- Technically this is the best option but I'm not sure cherry-picking a commit without signed-off is allowed in the OMPI community.
Manually resolve the conflict without AS_IF([test -x /usr/libexec/java_home], ...) -- I can do it but the fix is incomplete.
Manually resolve the conflict with AS_IF([test -x /usr/libexec/java_home], ...) -- How about copyright issue?.
Ask original author of ff5af73 to add signed-off
Don't cherry-pick -- We need to tell OS X/macOS users to switch to v3.0/v3.1 if they want Open MPI Java binding + Oracle JDK.

ggouaillardet · 2018-05-07T04:14:01Z

@kawashima-fj oh i see, this is an old commit that was never merged into the master.
The commit landed into master way before the sign-off policy was introduced, so it can be cherry-picked as-is (we did that before, but though legal, the sign-off checker might reject such a commit today)

the last option looks good enough to me (e.g. do nothing but ask users to upgrade)

My second option would be to cherry-pick both commits, squash them and keep the last commit message, and append a credit to the original author in the commit message.

kawashima-fj · 2018-05-07T04:41:55Z

Ok. If no one wants OMPI v2.1 + OS X/macOS + Oracle JDK + Open MPI Java binding, I'll update the README in v2.1 branch to upgrade to v3.0.2 or v3.1.1 after corresponding PRs are merged and they are released.

RandomDSdevel · 2018-05-07T19:44:30Z

@kawashima-fj:

Sorry, I'm late. There was 9-day holiday in Japan. …

Golden Week, eh? (I looked it up.) Well, that's as good a reason as any! I hope you enjoyed it!

jsquyres · 2018-05-07T19:57:39Z

FWIW, I would not advise cherry picking 2 commits and squashing them -- that just makes them harder to track.

Instead, I would cherry pick the 2 commits in the correct order (to avoid the conflicts), and then manually add a signed-off-by line on the commit that is missing it (e.g., via an interactive rebase or whatever your favorite commit-message-edit method you prefer).

RandomDSdevel · 2018-05-09T20:30:52Z

Per #5001 (comment) addressing #5001 (comment), see also #5160.

config/opal_setup_java.m4: Fix #5015.

@bosilca

That PR accidentally changed Open MPI's build configuration infrastruc- ture's Java toolchain detection logic so that it would, as reported by @bosilca in open-mpi#5001 (comment) and tracked down by me in open-mpi#5001 (comment), abort your entire in-progress Open MPI build when it failed to find an OS X/macOS JDK instead of simply falling back to checking for a JDK in locations where it would be found on other platforms. _Oops…!_ Signed-off-by: Bryce Glover <[email protected]> (cherry picked from commit 4a05c7e)

@bosilca

That PR accidentally changed Open MPI's build configuration infrastruc- ture's Java toolchain detection logic so that it would, as reported by @bosilca in open-mpi#5001 (comment) and tracked down by me in open-mpi#5001 (comment), abort your entire in-progress Open MPI build when it failed to find an OS X/macOS JDK instead of simply falling back to checking for a JDK in locations where it would be found on other platforms. _Oops…!_ Signed-off-by: Bryce Glover <[email protected]> (cherry picked from commit open-mpi/ompi@4a05c7e)

@bosilca

That PR accidentally changed Open MPI's build configuration infrastruc- ture's Java toolchain detection logic so that it would, as reported by @bosilca in open-mpi#5001 (comment) and tracked down by me in open-mpi#5001 (comment), abort your entire in-progress Open MPI build when it failed to find an OS X/macOS JDK instead of simply falling back to checking for a JDK in locations where it would be found on other platforms. _Oops…!_ Signed-off-by: Bryce Glover <[email protected]> (cherry picked from commit 4a05c7e)

…acOS. Also avoid picking up Apple's Java shims via the sym. links to them in `/usr/bin` on systems where any one of them could possibly exhibit behavior that is erratic and, to some extent, likely to be incorrect nowadays (cf.: - https://www.mail-archive.com/[email protected]/msg20551.html - #5015 (comment) - the last part of #5015 (comment) - #5015 (comment) for more detailed context.) Works alongside #5001 to close #5000. Signed-off-by: Bryce Glover <[email protected]> (cherry picked from commit 8c32cd8) Signed-off-by: KAWASHIMA Takahiro <[email protected]>

RandomDSdevel mentioned this pull request Apr 4, 2018

[Formula/open-mpi] Open MPI Fails to Build and Install Under OS X v10.11.6 'El Capitan' Homebrew/homebrew-core#26009

Closed

6 tasks

kawashima-fj mentioned this pull request Apr 6, 2018

configury: use javac vs javah whenever possible. #5001

Merged

RandomDSdevel mentioned this pull request Apr 6, 2018

Open MPI Fails to Build Under OS X v10.11.6 'El Capitan' Due to the Absence of javah in Java 10 #5000

Closed

bwbarrett added the Target: main label Apr 10, 2018

kawashima-fj self-assigned this Apr 24, 2018

kawashima-fj added the java bindings label Apr 24, 2018

ggouaillardet reviewed Apr 24, 2018

View reviewed changes

kawashima-fj reviewed Apr 26, 2018

View reviewed changes

kawashima-fj merged commit dc0f984 into open-mpi:master Apr 27, 2018

RandomDSdevel deleted the fix-configure-OS-X-and-macOS-JDK-path-resolution branch April 27, 2018 20:05

This was referenced May 1, 2018

v3.1: config/opal_setup_java.m4: Improve JDK tool path resolution on OS X/… #5118

Merged

v3.0: config/opal_setup_java.m4: Improve JDK tool path resolution on OS X/… #5119

Merged

kawashima-fj mentioned this pull request May 8, 2018

v2.x: Improve JDK tool path resolution on OS X/macOS #5153

Merged

jsquyres added a commit that referenced this pull request May 9, 2018

Merge pull request #5160 from RandomDSdevel/fix-5015

93930a2

config/opal_setup_java.m4: Fix #5015.

config/opal_setup_java.m4: Improve JDK Tool Path Resolution on OS X/macOS #5015

config/opal_setup_java.m4: Improve JDK Tool Path Resolution on OS X/macOS #5015

Conversation

RandomDSdevel commented Apr 4, 2018

ompiteam-bot commented Apr 4, 2018

kawashima-fj commented Apr 5, 2018

ggouaillardet commented Apr 6, 2018

RandomDSdevel commented Apr 6, 2018

RandomDSdevel commented Apr 6, 2018 • edited Loading

RandomDSdevel commented Apr 16, 2018

RandomDSdevel commented Apr 23, 2018

bwbarrett commented Apr 23, 2018

kawashima-fj commented Apr 24, 2018

kawashima-fj commented Apr 24, 2018

ggouaillardet Apr 24, 2018

Choose a reason for hiding this comment

RandomDSdevel Apr 25, 2018

Choose a reason for hiding this comment

jsquyres Apr 25, 2018

Choose a reason for hiding this comment

RandomDSdevel Apr 25, 2018

Choose a reason for hiding this comment

64-bit Requirement for Mac Apps

RandomDSdevel Apr 25, 2018

Choose a reason for hiding this comment

RandomDSdevel commented Apr 25, 2018 • edited Loading

RandomDSdevel commented Apr 25, 2018 • edited Loading

RandomDSdevel commented Apr 25, 2018

kawashima-fj Apr 26, 2018

Choose a reason for hiding this comment

RandomDSdevel Apr 26, 2018

Choose a reason for hiding this comment

jsquyres commented Apr 26, 2018

RandomDSdevel commented Apr 26, 2018

RandomDSdevel commented Apr 26, 2018 • edited Loading

bwbarrett commented Apr 26, 2018

kawashima-fj commented Apr 27, 2018

RandomDSdevel commented Apr 27, 2018

RandomDSdevel commented Apr 27, 2018

kawashima-fj commented Apr 27, 2018

RandomDSdevel commented Apr 30, 2018 • edited Loading

ggouaillardet commented May 1, 2018

ggouaillardet commented May 1, 2018

RandomDSdevel commented May 1, 2018

RandomDSdevel commented May 3, 2018 • edited Loading

kawashima-fj commented May 7, 2018

ggouaillardet commented May 7, 2018

kawashima-fj commented May 7, 2018

ggouaillardet commented May 7, 2018

kawashima-fj commented May 7, 2018

RandomDSdevel commented May 7, 2018

jsquyres commented May 7, 2018

RandomDSdevel commented May 9, 2018

RandomDSdevel commented Apr 6, 2018 •

edited

Loading

RandomDSdevel commented Apr 25, 2018 •

edited

Loading

RandomDSdevel commented Apr 25, 2018 •

edited

Loading

RandomDSdevel commented Apr 26, 2018 •

edited

Loading

RandomDSdevel commented Apr 30, 2018 •

edited

Loading

RandomDSdevel commented May 3, 2018 •

edited

Loading