BasicBuffer breaks the chain if queue exceeds limit

[dterror-zz]

https://github.com/fluent/fluentd/blob/master/lib/fluent/buffer.rb#L182

Raising an exception at this point will break the chain, so in the context of a CopyOutput being used, if one plugin type runs into a problem, it causes all to fail.

One reasonable use case would be: I have the copy plugin sending my data to S3 and ElasticSearch, if the ElasticSearch cluster goes down, I'm ok to lose that data provided that I know it will still be sent to S3. This type of configuration - based on copy - helps (that is, would help) reduce SPFs in logging.

Another point is, why would BasicBuffer ever raise an exception, losing data is a big thing and I understand it should 'fail loudly', but not at the cost of other outputs.

This code is really easy to change (error logging instead of raising), but if you feel like another solution is needed, I'm willing to submit a pull-request.

[repeatedly]

Maybe, adding buffer_ignore_exceeded_chunk or similar option is better.
Raising exeception is needed because some plugins return an error to the client for fail over.

[dterror-zz]

@repeatedly do you think that should be available only to BasicBuffer or be a property of CopyOutput? My reasoning is that the contract for failing is only different for out_copy.

[repeatedly]

@dterror CopyOutput itself can't handle above case because CopyOutput is simple output and doesn't touch a store's plugin.
How about following configuration?

<match foo.bar.*>
  type copy

  <store>
    type s3
    #...
  </store>
  <store>
    type elasticsearch
    buffer_ignore_exceeded_chunk true # output error to log, not raise an exception.
    #...
  </store>
</match>

[dterror-zz]

I was thinking of something like (in out_copy) (needs to be made more sophisticated):

    def emit(tag, es, chain)
      begin
         chain.next
       rescue Exception => e
          raise e unless @ignore_individual_errors
          chain.next
       end
    end

But I'm ok with your idea, too. Let me know what you think and I'll implement it.

[repeatedly]

I see. Maybe implementing I and your ideas is good for covering more cases.

We have a plan to implement error stream which sends error record to another plugin.
But it needs more time so implementing ignore_individual_errors parameter is good for now.

Could you send ignore_individual_errors pull request?

[dterror-zz]

sure, will work on it

[repeatedly]

Related issues are closed.
So this pull request is also closed.