Flush mode for ingester #1747

codesome · 2019-10-21T18:43:48Z

This is as per this design. Ingester would be run in flush mode in case someone forgets to hit the shutdown endpoint as described in #1746 before scaling down the ingesters with WAL.

Still in draft mode, as I have to check about the ingester quitting gracefully and not just stopping the operations.

bboreham

I guess it's early to review but I took a look and made a couple of comments.

pkg/ingester/ingester.go

pkg/cortex/cortex.go

jtlisi

Would it be possible to refactor this code so the shutdown is initiated by the ingester module?

That way the main.go should not need any changes, and the IsIngesterJob function could be removed?

jtlisi · 2019-11-21T16:53:30Z

pkg/ingester/ingester.go

+	if cfg.FlushMode {
+		// Flush and don't update the ring.
+		i.Flush()
+		return i, nil


Could we initiate the shutdown here?

I have been staring at this for a while and I might need some pointers on how cortex can be shut down within ingester. I can only see signals as an option.

Hrm, let me see. Can we pass t.Stop to the ingester when initiating and then just call that?

Will try that! Should work

Causes an initialization loop :/

Also, I think it's not safe to stop all modules while we are in the process of initing all the modules (additionally, the server is started after all modules are inited).

I tried another modification of my earlier attempt - the modules propagate the info of shutting down via a bool instead of cortex explicitly knowing about the ingester job.

jtlisi · 2019-11-21T16:53:51Z

cmd/cortex/main.go


 	runtime.KeepAlive(ballast)
-	err = t.Stop()
+
+	if t.IsIngesterJob() {


Is it possible to refactor this logic into the ingester module?

codesome · 2019-11-21T23:59:16Z

Would it be possible to refactor this code so the shutdown is initiated by the ingester module?

One blocker that I faced was that the server was started by the cortex package, and the Run() function is blocked on that. I could not find a way to stop the server from the ingester, would it be sane to pass the server object into the ingester which can stop it there?

codesome · 2019-11-22T00:05:26Z

I have a clarification to make here: The flushing happens even before the server starts. But would the metrics be served via the default registry even before the server starts? If not, then the code needs to be refactored so that the server starts before flushing.

jtlisi · 2019-11-22T19:23:44Z

I have a clarification to make here: The flushing happens even before the server starts. But would the metrics be served via the default registry even before the server starts? If not, then the code needs to be refactored so that the server starts before flushing.

Yea we are going to need the server to be running if we want metrics from this job.

pkg/cortex/cortex.go

gouthamve · 2019-12-03T21:34:46Z

pkg/ingester/ingester.go

+	if cfg.FlushMode {
+		// Flush and don't update the ring.
+		i.Flush()
+		return i, nil


Hrm, let me see. Can we pass t.Stop to the ingester when initiating and then just call that?

pkg/ingester/ingester.go

codesome · 2019-12-05T07:15:57Z

I am curious how can we scrape the metrics here even if we have the server running. Reason: The flushing happens in cortex.New when the ingester is being created. The server is stopped right after it was started, leaving little chance for a scrape to gather any metrics.

codesome · 2019-12-05T11:50:25Z

I tested this with #1103 and seems to be working fine.

@jtlisi

This is my attempt to simplify things compared to cortexproject#1747 /cc @jtlisi @codesome Signed-off-by: Goutham Veeramachaneni <[email protected]>

jtlisi · 2019-12-09T21:28:27Z

@codesome Ideally we use something like a push gateway for something like this. However, that may take a significant bit of refactoring. Otherwise we could add a wait period before shutting down.

Signed-off-by: Ganesh Vernekar <[email protected]>

codesome · 2019-12-10T09:49:19Z

I have now rebased this PR on top of #1885 with a small fix

Signed-off-by: Ganesh Vernekar <[email protected]>

codesome · 2019-12-12T09:34:37Z

Tested it. Worked fine.

bboreham · 2019-12-12T11:38:09Z

Sorry to come up with another suggestion after so many other discussions, but it seemed cleaner to me: how about doing it from a different -target ? That way Cortex.Run() doesn't need to change.

@jtlisi

This is my attempt to simplify things compared to cortexproject#1747 /cc @jtlisi @codesome Signed-off-by: Goutham Veeramachaneni <[email protected]>

gouthamve

Not the most ideal but should do the job, @codesome could you test it again?

Didn't see the conflicts

codesome · 2020-02-04T11:52:19Z

Superseded by #2075

codesome force-pushed the flush-only-ingester branch from 1fc5f35 to a5af8ea Compare October 22, 2019 12:57

codesome marked this pull request as ready for review October 22, 2019 13:14

codesome force-pushed the flush-only-ingester branch from a5af8ea to d1c3880 Compare October 22, 2019 17:08

bboreham reviewed Oct 24, 2019

View reviewed changes

pkg/ingester/ingester.go Outdated Show resolved Hide resolved

pkg/cortex/cortex.go Outdated Show resolved Hide resolved

codesome force-pushed the flush-only-ingester branch 3 times, most recently from cb5956b to fc24cce Compare November 18, 2019 18:31

jtlisi self-assigned this Nov 18, 2019

jtlisi suggested changes Nov 21, 2019

View reviewed changes

gouthamve reviewed Dec 3, 2019

View reviewed changes

codesome force-pushed the flush-only-ingester branch from feac537 to 7c362db Compare December 4, 2019 12:06

gouthamve mentioned this pull request Dec 5, 2019

Add flush job mode to ingester. #1885

Closed

3 tasks

Add flush job mode to ingester.

f7c407d

This is my attempt to simplify things compared to cortexproject#1747 /cc @jtlisi @codesome Signed-off-by: Goutham Veeramachaneni <[email protected]>

Handle multiple stop calls gracefully

212ebd3

Signed-off-by: Ganesh Vernekar <[email protected]>

codesome force-pushed the flush-only-ingester branch from d5efd1d to 212ebd3 Compare December 10, 2019 09:48

Merge remote-tracking branch 'upstream/master' into flush-only-ingester

9e46323

Signed-off-by: Ganesh Vernekar <[email protected]>

gouthamve added a commit to gouthamve/cortex that referenced this pull request Jan 3, 2020

Add flush job mode to ingester.

cf877d6

This is my attempt to simplify things compared to cortexproject#1747 /cc @jtlisi @codesome Signed-off-by: Goutham Veeramachaneni <[email protected]>

gouthamve previously approved these changes Jan 3, 2020

View reviewed changes

codesome mentioned this pull request Feb 4, 2020

Flusher target to flush WAL #2075

Merged

3 tasks

codesome closed this Feb 4, 2020

Flush mode for ingester #1747

Flush mode for ingester #1747

Uh oh!

Conversation

codesome commented Oct 21, 2019

Uh oh!

bboreham left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

jtlisi left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

codesome commented Nov 21, 2019

Uh oh!

codesome commented Nov 22, 2019

Uh oh!

jtlisi commented Nov 22, 2019

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

codesome commented Dec 5, 2019

Uh oh!

codesome commented Dec 5, 2019

Uh oh!

jtlisi commented Dec 9, 2019

Uh oh!

codesome commented Dec 10, 2019

Uh oh!

codesome commented Dec 12, 2019

Uh oh!

bboreham commented Dec 12, 2019

Uh oh!

gouthamve left a comment

Choose a reason for hiding this comment

Uh oh!

codesome commented Feb 4, 2020

Uh oh!

Uh oh!