Design and create a test suite

[abhijithnraj]

Hi, I have been following this project for some time since its release. Great Work.

But support for several git features coming in, is there any plans or ideas on testing the features. I would like to work on this.

For Manim based tests, we can have a look at the same way manimCE tests their own code. https://docs.manim.community/en/stable/contributing/testing.html and https://github.com/ManimCommunity/manim/tree/main/tests

Hi, @initialcommit-io , is there any test design you have in mind.?

Some of my ideas:

1. Some basic Robustness tests to ensure no failures, like object creation, function calls, running from out of repo, support for the available python versions etc.
2. Some Functional Test cases by creating a story, a basic git repo: run command compare output, first commit: run command compare output. But with changes happening to the output representation itself, need to inspect the Manim MObject that is being rendered and assert on its properties.
3. Checking the command line flags are properly being dispatched, ensuring no regressions on new modifications etc.

Once done we can add an actions pipeline to ensure continuous integration.

I am working on this, let me know if there are any existing plans on this.

Once again, Thank You for this wonderful project.

[abhijitnathwani]

I was about to create an issue to track this @abhijithnraj. This a good point to start integrating tests NOW. @initialcommit-io even I was skeptical while adding support for the stash subcommand that I may break something in some other commands while overriding the GitSimBaseCommand methods. I think we need to start designing the test plan aggressively now. The problems you caught with Typer integration could be caught easily with some functional test plans.

For Manim based tests, we can have a look at the same way manimCE tests their own code. https://docs.manim.community/en/stable/contributing/testing.html and https://github.com/ManimCommunity/manim/tree/main/tests

I didn't know how to test the images that will be generated but this looks promising.

Once done we can add an actions pipeline to ensure continuous integration.

Exactly. This will ensure great test coverage and using tools like pytest we shall be in a good form to support this for the long term with new features and guaranteeing support for the already existing features.

Python version is already an issue with the new Typer implementation which I have discussed here.

[initialcommit-io]

@abhijithnraj Thank you for creating this issue and for your proposed contributions! Yes, it's clear we need to start on this now, as @abhijitnathwani brought up some breaking changes across Python versions that we had not been testing for.

For context - I didn't have any tests going into the project because I wanted to gauge user interest before spending too much time on it, but now that we know the project has a solid user base, we need to prioritize a robust test suite for the reasons you both mentioned.

@abhijithnraj I don't have any specific test design in mind, but I agree with all of your points, and good find on the Manim test option. Maybe a good place to start is if you want to put together a PR with a simplified version of an initial test suite, that covers a single aspect of the codebase - probably git-sim log subcommand since that is the simplest - from all perspectives (1), (2), and (3) that you mentioned.

Then we can all discuss the PR and see if it will work for the whole codebase/program.

[initialcommit-io]

@abhijitnathwani @abhijithnraj FYI I meant to mention - I will be working on a small tool that can generate dummy Git repos with specific numbers of commits / branch structures / etc. I am thinking we can use this in functional tests to create the structure we need for testing certain commands, like merge/rebase, etc.

[abhijithnraj]

@initialcommit-io Yes, that can help with generating test data.

[initialcommit-io]

@abhijithnraj Do you have a timeline for an initial version of the test suite that provides coverage for the git log feature?

[initialcommit-io]

@abhijithnraj @abhijitnathwani @paketb0te Here is the first version of a tool I created to generate dummy Git repos:

https://github.com/initialcommit-com/git-dummy

I'm thinking we can use this in our functional tests.

[abhijithnraj]

@initialcommit-io I am studying the graphical test cases for manim. I think I can raise the first PR at max 2 days. We will add more testcases once we finalize the framework in the PR. Meanwhile I will checkout git-dummy. currently for testing, I wrote a simple code to generate some commits.

[initialcommit-io]

@abhijithnraj Sounds great. FYI I added a --branches=N option to git-dummy, so now it can easily generate dummy Git repos with an arbitrary number of branches. This will be good for simulating commands that show multiple branches, like merge, rebase, etc.

[initialcommit-io]

@abhijithnraj @abhijitnathwani @paketb0te Hi again - last update for a bit on git-dummy, I added flags --diverge-at to set the commit number at which the branches will diverge from main.

I also added --merge=x,y,...,n so the user can select which branches to merge back into main if desired.

Its not much, but I think at this point git-dummy has enough so that we can use it for basic functional test coverage, to set up automated scenarios/repos we need for the tests.

I am also planning to add hooks for git-dummy into Git-Sim, with something like a --generate flag. This will allow users to generate a test repo automatically before running a Git-Sim command.

[initialcommit-io]

@abhijithnraj Hi again! Just wanted to check in and see if this is still something you were working on?

[abhijithnraj]

Hi, Still working on it.

[initialcommit-io]

Hi @abhijithnraj @abhijitnathwani @paketb0te ! Hope you all are doing great =)

Just a note I released a significant refactor in git-sim v0.2.6 which overhauls the way it parses the commit history and adds several important new options/flags. Git-Sim now properly traces parent/child relationships, which means it can display more accurate and complex graph structures, with multiple branches and the relationships between them.

See the release notes for more details: https://github.com/initialcommit-com/git-sim/releases/tag/v0.2.6

I figured I'd mention it here since it might affect the test suite, depending on how it was being written.

Let me know if you get the chance to try it out. It's likely there are some bugs since we still don't have the test suite set up yet, but I really wanted to get this released since I feel it's an important milestone for the project, will make the tool much more useful for users, and will make future development smoother.