Skip to content

Introduced a basic logger to the integration tests framework #2293

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged

Conversation

pracucci
Copy link
Contributor

What this PR does:
We have some flaky integration tests in query_frontend_test.go. I can't understand the root cause but I've the feeling it's a timing issue. In this PR I'm adding a very simple logger to prefix each log with the time, so that we can have a better idea where the time is being spent when running in CI.

An example of the tests output is:

=== RUN   TestQueryFrontendWithChunksStorageViaConfigFile
10:00:47 Starting consul
10:00:48 consul: ==> Starting Consul agent...
10:00:48 consul: ==> Consul agent running!
10:00:48 consul: Version: 'v0.9.4'
10:00:48 consul: Node ID: 'b67d8cc8-029c-d764-dc65-a7205203b3cd'
10:00:48 consul: Node name: 'consul'

Which issue(s) this PR fixes:
N/A

Checklist

  • Tests updated
  • Documentation added
  • CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

@pracucci pracucci force-pushed the add-logger-to-integration-tests branch from b86b16b to 9832720 Compare March 19, 2020 09:06
Signed-off-by: Marco Pracucci <[email protected]>
@pracucci
Copy link
Contributor Author

And thanks to the timing displayed in logs I've also found and fixed the root cause, which is trivial. We currently have a 300s timeout for the integration tests, but due to the new tests we're adding, the whole suite sometimes takes more than 5 minutes to run in CI and thus it breaks due to the max execution time reached.

I've increased the timeout to 10 minutes (commit). In the future we may experiment with parallel integration tests execution.

@pracucci pracucci merged commit 305b16c into cortexproject:master Mar 19, 2020
@pracucci pracucci deleted the add-logger-to-integration-tests branch March 19, 2020 09:46
pracucci added a commit to grafana/cortex that referenced this pull request Mar 19, 2020
commit 8e9c601
Author: Marco Pracucci <[email protected]>
Date:   Thu Mar 19 10:44:34 2020 +0100

    Avoid a superfluous querier integration test

    Signed-off-by: Marco Pracucci <[email protected]>

commit 9691dc6
Author: Marco Pracucci <[email protected]>
Date:   Thu Mar 19 09:26:30 2020 +0100

    Separated metrics helpers unit tests using go blocks

    Signed-off-by: Marco Pracucci <[email protected]>

commit d32afe8
Author: Marco Pracucci <[email protected]>
Date:   Thu Mar 19 09:23:34 2020 +0100

    Reworded util.StringsContain() comment and params

    Signed-off-by: Marco Pracucci <[email protected]>

commit a5000a8
Author: Marco Pracucci <[email protected]>
Date:   Thu Mar 19 09:21:54 2020 +0100

    Update docs/operations/dns-service-discovery.md

    Signed-off-by: Marco Pracucci <[email protected]>

    Co-Authored-By: Peter Štibraný <[email protected]>

commit 1babd96
Author: Marco Pracucci <[email protected]>
Date:   Thu Mar 19 09:21:37 2020 +0100

    Update integration/e2e/service.go

    Signed-off-by: Marco Pracucci <[email protected]>

    Co-Authored-By: Peter Štibraný <[email protected]>

commit c1ed809
Author: Marco Pracucci <[email protected]>
Date:   Thu Mar 19 09:01:57 2020 +0100

    Added a timeout to the 'docker inspect' command to avoid the command being stuck

    Signed-off-by: Marco Pracucci <[email protected]>

commit 271fb8d
Author: Marco Pracucci <[email protected]>
Date:   Thu Mar 19 08:52:09 2020 +0100

    Updated doc

    Signed-off-by: Marco Pracucci <[email protected]>

commit a922d32
Author: Marco Pracucci <[email protected]>
Date:   Thu Mar 19 08:43:20 2020 +0100

    Fixed unit tests

    Signed-off-by: Marco Pracucci <[email protected]>

commit 50398ab
Author: Marco Pracucci <[email protected]>
Date:   Thu Mar 19 08:40:40 2020 +0100

    Fixed doc

    Signed-off-by: Marco Pracucci <[email protected]>

commit f077389
Author: Marco Pracucci <[email protected]>
Date:   Wed Mar 18 18:57:52 2020 +0100

    Documented blocks storage index cache backends

    Signed-off-by: Marco Pracucci <[email protected]>

commit 330e449
Author: Marco Pracucci <[email protected]>
Date:   Wed Mar 18 18:14:12 2020 +0100

    Updated CHANGELOG

    Signed-off-by: Marco Pracucci <[email protected]>

commit 0233519
Author: Marco Pracucci <[email protected]>
Date:   Thu Mar 12 17:40:21 2020 +0100

    Added memcached support to the blocks storage index cache

    Signed-off-by: Marco Pracucci <[email protected]>

commit 305b16c
Author: Marco Pracucci <[email protected]>
Date:   Thu Mar 19 10:46:43 2020 +0100

    Introduced a basic logger to the integration tests framework (cortexproject#2293)

    * Introduced a basic logger to the integration tests framework

    Signed-off-by: Marco Pracucci <[email protected]>

    * Fixed linter

    Signed-off-by: Marco Pracucci <[email protected]>

    * Increased integration tests timeout in CI

    Signed-off-by: Marco Pracucci <[email protected]>

commit 51f7047
Author: Jay Batra <[email protected]>
Date:   Thu Mar 19 13:19:39 2020 +0530

    Removes all the occurences of omitempty (cortexproject#2209)

    * Removes all the occurences of omitempty(cortexproject#2209)

    This PR removes all the occurences of omitempty
    from structs.

    Signed-off by Jay Batra <[email protected]>

    Signed-off-by: Jay Batra <[email protected]>

    * Update CHANGELOG.md

    Co-authored-by: Marco Pracucci <[email protected]>

commit bf59f9f
Author: Sandeep Sukhani <[email protected]>
Date:   Wed Mar 18 17:00:11 2020 +0530

    support for cassandra in integration tests (cortexproject#2275)

    * support for cassandra in integration tests

    Signed-off-by: Sandeep Sukhani <[email protected]>

    * changes suggested from PR review

    Signed-off-by: Sandeep Sukhani <[email protected]>

    * using ConcreteService.Endpoint for getting http endpoint for readiness probe

    Signed-off-by: Sandeep Sukhani <[email protected]>

commit 87fa252
Author: Rajat Vig <[email protected]>
Date:   Wed Mar 18 08:49:45 2020 +0000

    Fix the import for test for bucket_client (cortexproject#2285)

    Signed-off-by: Rajat Vig <[email protected]>
simonswine pushed a commit to grafana/e2e that referenced this pull request Jan 13, 2022
…roject/cortex#2293)

* Introduced a basic logger to the integration tests framework

Signed-off-by: Marco Pracucci <[email protected]>

* Fixed linter

Signed-off-by: Marco Pracucci <[email protected]>

* Increased integration tests timeout in CI

Signed-off-by: Marco Pracucci <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants