Change web server from Thin to Puma (or other alternative)

[sethboyles]

Why?

Thin has not been under active support for many years and has not seen a release since 2016.

Additionally, Thin (and EventMachine, which it is based off of) does not seem to always handle concurrent requests correctly. In rare cases, a single API call can block Thin from responding to other requests.

Potential alternatives:

Puma -- the default web server included with Rails 5
Unicorn
Rainbows!
Others?

What do we need to do before we can switch off of Thin?

Performance

We currently don't have a baseline of CCNG performance on Thin, and thus no way to measure the effect this may have on Cloud Controller's performance. We probably would like to have a suite of performance measurements to ensure that at the very least we don't end up degrading performance in any way.

Based on the BOSH team's measurements from switching from Thin to Puma, it's likely that we will only see improvements to response times and throughput. But we should be able to back this up.

Functionality

We use EventMachine's periodic timer functionality to emit metrics:

cloud_controller_ng/lib/cloud_controller/metrics/periodic_updater.rb

Lines 18 to 20 in 4a23a41

	EM.add_periodic_timer(600) { catch_error { record_user_count } }
	EM.add_periodic_timer(30) { catch_error { update_job_queue_length } }
	EM.add_periodic_timer(30) { catch_error { update_thread_info } }

We would need to figure out how we could replace this functionality.

We also have EventMachine specific code in async_requests.rb and runner.rb

Dependencies

is the dependency of

• #221
• #218
• #217

Notes

It may be helpful to review the BOSH team's previous successful effort when changing the BOSH Director's web server from Thin to Puma:

https://www.pivotaltracker.com/n/projects/1456570/stories/150958544
cloudfoundry/bosh#1800

The CAPI team also previously investigated making the switch in 2017, but it seems that track of work was dropped. I can't find a recorded reason why it wasn't pursued--@Gerg do you have any recollection around this?

@andy-paine @FloThinksPi @stephanme

[Gerg]

I don't remember a particular reason why we didn't do it. I think, like you identified, it isn't a trivial swap, and we had other priorities at the time.

[sethboyles]

I just discovered another alternative, created by a Ruby core contributor (who is also apparently doing minor updates to Thin): https://github.com/socketry/falcon

It is based off of https://github.com/socketry/async. I haven't dug too deep, but it seems that it is taking advantage of Ruby Fibers to implement the Reactor pattern. Since Thin is also based off of the Reactor pattern, I wonder if this would be less of behavior change for CCNG than to switch to Puma.

That said, I'm not clear on the stability or current viability of Falcon. There is not a 1.0 release and there is a PR to move forward with a Ruby 3.0 only version of the underlying library: socketry/async#111

[sethboyles]

Looks like Thin got a potential security fix: https://github.com/macournoyer/thin/releases/tag/v1.8.1

[mkocher]

This would be awesome, I'd love to see us do it.

Converting CC to the new way of doing component metrics using prom-scraper could eliminate that use of event machine and make metrics just another http endpoint.

[philippthun]

I'm going to close this issue as Puma can already be used (still marked as experimental) and I'm confident that we will be able to make it the default webserver at some point in time.