add support for bigtable as index store for integration tests #2249

sandeepsukhani · 2020-03-11T10:42:27Z

What this PR does:
Add support for Bigtable in integration tests using https://github.com/Shopify/bigtable-emulator
It also adds a simple test for all the supported index stores in integration tests while keeping dynamo-db as default for rest of the tests.

pstibrany

LGTM. I think number of parameters to various New<Service> functions is so high now that we should refactor it to use functional options pattern (in new PR)

sandeepsukhani · 2020-03-11T11:54:45Z

LGTM. I think number of parameters to various New<Service> functions is so high now that we should refactor it to use functional options pattern (in new PR)

Yeah, right. I will add it to my todo list.
Thanks for the review!

pracucci · 2020-03-11T12:16:14Z

integration/configs.go

@@ -99,6 +105,9 @@ storage:
  aws:
    dynamodbconfig:
      dynamodb: {{.DynamoDBURL}}
+  bigtable:


By contract, ChunksStorageConfig should be specular to ChunksStorageFlags, so if you do a change to ChunksStorageConfig you should do the same change to ChunksStorageFlags and viceversa.

However, these configs where designed to be ready to use and should not have mixed config (either the storage is AWS or BigTable). I would suggest to:

Rename ChunksStorageConfig to ChunksStorageDynamoDBConfig

Rename ChunksStorageFlags to ChunksStorageDynamoDBFlags

Create ChunksStorageBigtableConfig and ChunksStorageBigtableFlags with the config you want

It seems I can get rid of this change completely because it is not used in the new test that I added.

pracucci · 2020-03-11T12:25:05Z

integration/e2ecortex/services.go

 	)
 }

-func NewQuerier(name string, consulAddress string, flags map[string]string, image string) *CortexService {
-	return NewQuerierWithConfigFile(name, consulAddress, "", flags, image)
+func NewQuerier(name string, consulAddress string, flags map[string]string, image string, envVars map[string]string) *CortexService {


I'm not sure it's worth adding envVars to all such functions, considering it's not a commong use case and you could just call service.SetEnvVars() in the single integration test where you need it. What's your take?

I had started with that approach first but then it made me feel someone could miss setting that env var, while setting all the params in the function is explicitly done so less of a chance to miss it. But I think we would mostly continue using dynamo for all the other test so I can use that function for now. We can refactor it again later if we see more needs for setting environment variables.
Thanks!

then it made me feel someone could miss setting that env var

It would miss anyway if that person is not aware of the required env var. If you want to guaraantee it you should take bigtableAddress in input, like we do for consulAddress, but to me looks a bit over-engineered adding bigtableAddress to each of such functions given it's only required by 1 single scenario (contrary to consul).

pracucci · 2020-03-11T12:31:38Z

integration/all_stores_test.go

+
+	// here we are starting and stopping table manager for each index store
+	// this is a workaround to make table manager create tables for each config since it considers only latest schema config while creating tables
+	for i := range storeConfigs {


This looks a bit hacky. Why don't we run the table manager only once, but with a schema config containing both storages (two entries in the same schema config)? Doing it with buildSchemaConfigWith() may be engineered (would require extra refactoring I'm not sure it's worth). You could just rollback buildSchemaConfigWith() and have a static schema config defined in this file.

This is done because TableManager only uses latest config to create tables i.e it does not consider older configs while creating tables. There is a PR open to fix this behaviour in Cortex. See #1446

We have not seen someone complain about it because we add only one schema at a time to cortex as and when needed but here I am setting multiple schema configs at a time.
Does it make sense?

Yes. I wasn't aware of the issue. Could you add a // TODO comment referencing the PR and saying that we can merge two two schema configs into 1 and just run the table-manager once once that PR is merged, please?

pracucci · 2020-03-11T13:54:44Z

integration/all_stores_test.go

+		require.NoError(t, err)
+		require.Equal(t, 200, res.StatusCode)
+
+		// Query the series both from the querier and query-frontend (to hit the read path).


Suggested change

// Query the series both from the querier and query-frontend (to hit the read path).

// Query back the series.

pracucci · 2020-03-11T13:55:36Z

integration/all_stores_test.go

+		require.Equal(t, 200, res.StatusCode)
+
+		// Query the series both from the querier and query-frontend (to hit the read path).
+		c, err = e2ecortex.NewClient("", querier.HTTPEndpoint(), "", "user-1")


This test should hit the storage, but how is it guaranteed that the series have been flushed to the storage and offloaded from the ingesters memory at this point?

I've the feeling we're not hitting the backend store neither on the write and read path, because the chunks are just kept in the ingesters memory.

it also adds a simple test for all the supported index stores in integration tests Signed-off-by: Sandeep Sukhani <[email protected]>

Signed-off-by: Marco Pracucci <[email protected]>

Signed-off-by: Sandeep Sukhani <[email protected]>

Signed-off-by: Marco Pracucci <[email protected]>

pracucci

Thanks @sandeepsukhani! LGTM. As agreed offline, I pushed a couple of small changes.

…project/cortex#2249) * add support for bigtable as index store for integration tests it also adds a simple test for all the supported index stores in integration tests Signed-off-by: Sandeep Sukhani <[email protected]> * Added comment to Bigtable image Signed-off-by: Marco Pracucci <[email protected]> * changes suggested from PR review Signed-off-by: Sandeep Sukhani <[email protected]> * changes suggested from PR review Signed-off-by: Sandeep Sukhani <[email protected]> * Simplified client Signed-off-by: Marco Pracucci <[email protected]> * Renamed test file Signed-off-by: Marco Pracucci <[email protected]> Co-authored-by: Marco Pracucci <[email protected]>

pull-request-size bot added the size/L label Mar 11, 2020

sandeepsukhani force-pushed the bigtable-integration-test branch from 16b5444 to c9b35bb Compare March 11, 2020 10:56

pstibrany approved these changes Mar 11, 2020

View reviewed changes

pracucci reviewed Mar 11, 2020

View reviewed changes

sandeepsukhani and others added 5 commits March 12, 2020 12:28

add support for bigtable as index store for integration tests

0789a9a

it also adds a simple test for all the supported index stores in integration tests Signed-off-by: Sandeep Sukhani <[email protected]>

Added comment to Bigtable image

8bcf0f9

Signed-off-by: Marco Pracucci <[email protected]>

changes suggested from PR review

e01c45b

Signed-off-by: Sandeep Sukhani <[email protected]>

changes suggested from PR review

07b7f7d

Signed-off-by: Sandeep Sukhani <[email protected]>

Simplified client

b2ad1d9

Signed-off-by: Marco Pracucci <[email protected]>

pracucci force-pushed the bigtable-integration-test branch from 9288ae3 to b2ad1d9 Compare March 12, 2020 11:33

Renamed test file

385b1b5

Signed-off-by: Marco Pracucci <[email protected]>

pracucci approved these changes Mar 12, 2020

View reviewed changes

pracucci merged commit 8b04fac into cortexproject:master Mar 12, 2020

pracucci mentioned this pull request Mar 30, 2020

Add integration tests for cassandra, bigtable and etcd #2137

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

add support for bigtable as index store for integration tests #2249

add support for bigtable as index store for integration tests #2249

Uh oh!

sandeepsukhani commented Mar 11, 2020

Uh oh!

pstibrany left a comment

Uh oh!

sandeepsukhani commented Mar 11, 2020

Uh oh!

pracucci Mar 11, 2020

Uh oh!

sandeepsukhani Mar 11, 2020

Uh oh!

pracucci Mar 11, 2020

Uh oh!

sandeepsukhani Mar 11, 2020 •

edited

Loading

Uh oh!

pracucci Mar 11, 2020

Uh oh!

pracucci Mar 11, 2020

Uh oh!

sandeepsukhani Mar 11, 2020

Uh oh!

pracucci Mar 11, 2020

Uh oh!

pracucci Mar 11, 2020

Uh oh!

pracucci Mar 11, 2020

Uh oh!

pracucci left a comment

Uh oh!

Uh oh!

	// Query the series both from the querier and query-frontend (to hit the read path).
	// Query back the series.

add support for bigtable as index store for integration tests #2249

add support for bigtable as index store for integration tests #2249

Uh oh!

Conversation

sandeepsukhani commented Mar 11, 2020

Uh oh!

pstibrany left a comment

Choose a reason for hiding this comment

Uh oh!

sandeepsukhani commented Mar 11, 2020

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sandeepsukhani Mar 11, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pracucci left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

sandeepsukhani Mar 11, 2020 •

edited

Loading