Add tests (here or in Chakra-Samples) to ensure ChakraCore Samples don't break.

[dilijev]

Previously it has been noted by some members of the community that ChakraCore Samples in the Chakra-Samples project have not worked with the current ChakraCore master. These could be temporary regressions but they are not tracked by any automation or with any visible build badges, so it is not possible for the community to know whether they are expected to work.

It seems to me that we should have some kind of automation to ensure that Chakra-Samples continues to work. I think the most sensible place to add this (for community visibility) would be to the Chakra-Samples project itself. We could add Jenkins automation to clone ChakraCore master, build it, and then build the samples against it. That's the end-to-end scenario we plan to support. This also has the benefit of adding build scripts that show the full set of steps needed for the samples to work, and so would be valuable for documentation.

This unfortunately runs the risk of falling under the radar of the ChakraCore project maintainers, so we should, at a minimum, link from the Build Status table to the same daily build badges that indicate the health of the ChakraCore samples.

As a nice-to-have: having branches in Chakra-Samples and corresponding automation for master, the release milestone in progress (currently 1.4) and any supported releases (currently 1.2 and 1.3), will help users of ChakraCore find the Samples that work with the version of ChakraCore they are trying to build. Short of that, clearly indicating in a README under ChakraCore Samples which branch(es) of ChakraCore the samples are compatible with would be helpful as well.

See conversation here: #2001 (comment)

Related to: #2136

/cc @obastemur @liminzhu

[obastemur]

Previously it has been noted by some members of the community that ChakraCore Samples in the Chakra-Samples project have not worked with the current ChakraCore master.

1 - We weren't testing native tests before. Added them after the first wave of sample break reports.

2 - We had updated most JSRT.. Between merge here and updating the samples there, small delay and expected break.

Repeat: We have native tests on this very repository now. Those tests use the same code and compile script as we have on ChakraSamples repository. What else do we want here?

[dilijev]

@obastemur - I guess I didn't realize that they were exactly the same code. Is the plan to always keep them in sync? Is all the sample code represented in our unit tests?

From a community perspective, a badge showing the build is good and the automation scripts seems like they would still be nice.

But anyway it sounds like we're pretty much covered. I didn't understand the full scope of the native tests.

[Oceanswave]

If you're needing an automated way of generating the P/Invoke layer for the C# hosting samples to get full coverage, I'm using CppSharp to do so over here with a sample usage of the generated pinvoke layer here

[obastemur]

@Oceanswave looks nice, Thanks!

[agarwal-sandeep]

@dilijev @liminzhu what's the plan for this?

[dilijev]

An alternative to the above proposal for CI under Chakra-Samples is to migrate the samples directly into the ChakraCore repo under a samples/ directory to keep them close and ensure they build along with every change we make, so we know right away. As @kfarnung pointed out, this would also help users with being able to clone, build samples, and play around with ChakraCore right away (from a hosting perspective, instead of just using ch or node-ChakraCore which is a language-level playground).