RFC: add debug functionality to test with mock server #69

tomerd · 2020-05-05T03:38:51Z

motivation: allow end to end testing locally

changes:

add a Lambda+LocalServer which exposes Lambda.withLocalServer available only in DEBUG mode
local server can recieve POST requests with payloads on a configurable endpoint and and send them to the Lambda
add a "noContent" mode to Lambda runtime to allow polling

tomerd · 2020-05-05T03:41:55Z

Sources/AWSLambdaRuntime/Lambda+LocalServer.swift

+    public static func withLocalServer(invocationEndpoint: String? = nil, _ body: @escaping () -> Void) throws {
+        let server = LocalLambda.Server(invocationEndpoint: invocationEndpoint)
+        try server.start().wait()
+        defer { try! server.stop() } // FIXME:


tomerd · 2020-05-05T03:42:21Z

Sources/AWSLambdaRuntime/Lambda+LocalServer.swift

+        func processRequest(context: ChannelHandlerContext, request: (head: HTTPRequestHead, body: ByteBuffer?)) {
+            if request.head.uri.hasSuffix(self.invocationEndpoint) {
+                if let work = request.body {
+                    let requestId = "\(DispatchTime.now().uptimeNanoseconds)" // FIXME:


👀 need to think of something better?

An aws request id is a UUID. So I guess we should use a uuid for this?

we could, that will pull in Foundation

Sources/AWSLambdaRuntime/Lambda+LocalServer.swift

tomerd · 2020-05-05T03:43:31Z

Sources/AWSLambdaRuntime/Lambda+LocalServer.swift

+// MARK: - Local Mock Server
+
+private enum LocalLambda {
+    struct Server {


maybe we should call this MockServer to make it even clearer?

Sources/AWSLambdaRuntime/Lambda+LocalServer.swift

tomerd · 2020-05-05T05:06:28Z

@swift-server-bot test this please

tomerd · 2020-05-05T05:07:02Z

example how this can be used
https://github.com/swift-server/swift-aws-lambda-deubgging-example

cc @fabianfett

tomerd · 2020-05-05T05:08:37Z

Sources/AWSLambdaRuntime/LambdaRunner.swift

+                }
+                throw error
+            }
+            .always { result in
                // we are done!


@fabianfett these are the changes I like the least. in real work lambda, this will never happen since the lambda will be frozen until work is available, but in this case we continue to poll if "noContent" is received. happy to hear better ideas

Within the Lambda.run we already listen for a stop signal:

internal static func runAsync(eventLoopGroup: EventLoopGroup, configuration: Configuration, factory: @escaping HandlerFactory) -> EventLoopFuture<Int> { Backtrace.install() var logger = Logger(label: "Lambda") logger.logLevel = configuration.general.logLevel let lifecycle = Lifecycle(eventLoop: eventLoopGroup.next(), logger: logger, configuration: configuration, factory: factory) let signalSource = trap(signal: configuration.lifecycle.stopSignal) { signal in logger.info("intercepted signal: \(signal)") lifecycle.shutdown() } return lifecycle.start().flatMap { return lifecycle.shutdownFuture.always { _ in signalSource.cancel() } } }

The shutdown method triggers a state change, which is only evaluated before a new cycle. That means if we are waiting for a new task cancel, shutdown() should cancel the next request.

tomerd · 2020-05-05T06:21:47Z

5.3 failure is due to a swift-nio issue on 5.3, should be fixed soon and safe to ignore for now.

fabianfett

Great start, there are a couple of bigger issues that we will need to tackle, but overall this is a move into the right direction!

fabianfett · 2020-05-05T15:32:04Z

Sources/AWSLambdaRuntime/Lambda+LocalServer.swift

+            var logger = Logger(label: "LocalLambdaServer")
+            logger.logLevel = configuration.general.logLevel
+            self.logger = logger
+            self.group = MultiThreadedEventLoopGroup(numberOfThreads: System.coreCount)


Do we really want to support multiple eventLoops here? I think @weissi's main event grouplool could also be sufficient here. Could save us locks (which would make the code simpler) and therefore easier to debug. And performance shouldn't be a problem anyways.

We should not take over the main thread, since we are running in the same process as the lambda executor/runner (which will take over the main thread itself). But we should use only one thread though ;)

I think one single threaded ELG (with the new takeover stuff) should be enough for lambda. Server & client can also share.

fabianfett · 2020-05-05T15:35:42Z

Sources/AWSLambdaRuntime/Lambda+LocalServer.swift

+        func processRequest(context: ChannelHandlerContext, request: (head: HTTPRequestHead, body: ByteBuffer?)) {
+            if request.head.uri.hasSuffix(self.invocationEndpoint) {
+                if let work = request.body {
+                    let requestId = "\(DispatchTime.now().uptimeNanoseconds)" // FIXME:


An aws request id is a UUID. So I guess we should use a uuid for this?

Sources/AWSLambdaRuntime/Lambda+LocalServer.swift

fabianfett · 2020-05-05T15:46:07Z

Sources/AWSLambdaRuntime/Lambda+LocalServer.swift

+                        (AmazonHeaders.requestID, pending.key),
+                        (AmazonHeaders.invokedFunctionARN, "arn:aws:lambda:us-east-1:\(Int16.random(in: Int16.min ... Int16.max)):function:custom-runtime"),
+                        (AmazonHeaders.traceID, "Root=\(Int16.random(in: Int16.min ... Int16.max));Parent=\(Int16.random(in: Int16.min ... Int16.max));Sampled=1"),
+                        (AmazonHeaders.deadline, "\(DispatchWallTime.distantFuture.millisSinceEpoch)"),


We should add something here. Maybe even make this configurable?

We should probably create even have a real timeout schedule here as well.

Sources/AWSLambdaRuntime/Lambda+LocalServer.swift

fabianfett · 2020-05-05T16:22:21Z

Sources/AWSLambdaRuntime/LambdaRunner.swift

+                }
+                throw error
+            }
+            .always { result in
                // we are done!


Within the Lambda.run we already listen for a stop signal:

internal static func runAsync(eventLoopGroup: EventLoopGroup, configuration: Configuration, factory: @escaping HandlerFactory) -> EventLoopFuture<Int> { Backtrace.install() var logger = Logger(label: "Lambda") logger.logLevel = configuration.general.logLevel let lifecycle = Lifecycle(eventLoop: eventLoopGroup.next(), logger: logger, configuration: configuration, factory: factory) let signalSource = trap(signal: configuration.lifecycle.stopSignal) { signal in logger.info("intercepted signal: \(signal)") lifecycle.shutdown() } return lifecycle.start().flatMap { return lifecycle.shutdownFuture.always { _ in signalSource.cancel() } } }

The shutdown method triggers a state change, which is only evaluated before a new cycle. That means if we are waiting for a new task cancel, shutdown() should cancel the next request.

tomerd · 2020-05-06T02:48:06Z

@fabianfett i'v updated this based on #70 + fixup on top. I think it looks much better, thank you. the request/response bit is still order sensitive (since not using a dictionary any longer) but maybe good enough for what this is designed for?

followups:

do we feel this is a good direction to go?
anything open in the code we need to address?
do we need a withLocalAPIGatewatServer variant that will wrap the invocation request in an APIGateway.Request to help build such popular Lambda functions?

wdyt?

motivation: allow end to end testing locally changes: * add a Lambda+LocalServer which exposes Lambda.withLocalServer available only in DEBUG mode * local server can recieve POST requests with payloads on a configurable endpoint and and send them to the Lambda * add a "noContent" mode to Lambda runtime to allow polling

* Don’t exit immediately * Removed locks. Just running in one EL

tomerd · 2020-05-15T18:57:56Z

replaced by #73

tomerd requested a review from fabianfett May 5, 2020 03:38

tomerd commented May 5, 2020

View reviewed changes

Sources/AWSLambdaRuntime/Lambda+LocalServer.swift Outdated Show resolved Hide resolved

tomerd commented May 5, 2020

View reviewed changes

Sources/AWSLambdaRuntime/Lambda+LocalServer.swift Outdated Show resolved Hide resolved

tomerd commented May 5, 2020

View reviewed changes

Sources/AWSLambdaRuntime/Lambda+LocalServer.swift Show resolved Hide resolved

tomerd force-pushed the feature/local-server branch 2 times, most recently from 265d08e to 11bc8d1 Compare May 5, 2020 04:14

tomerd commented May 5, 2020

View reviewed changes

tomerd force-pushed the feature/local-server branch 2 times, most recently from b5a8f74 to 9863db8 Compare May 5, 2020 05:13

fabianfett reviewed May 5, 2020

View reviewed changes

tomerd force-pushed the feature/local-server branch 4 times, most recently from 4857d6f to c24bce7 Compare May 6, 2020 02:44

tomerd and others added 3 commits May 7, 2020 13:26

Feature/ff local server (#70)

26f6a4b

* Don’t exit immediately * Removed locks. Just running in one EL

fixup

2cf4e1b

fabianfett force-pushed the feature/local-server branch from bf4249c to 2cf4e1b Compare May 7, 2020 11:27

tomerd closed this May 15, 2020

fabianfett deleted the feature/local-server branch May 19, 2020 15:18

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

RFC: add debug functionality to test with mock server #69

RFC: add debug functionality to test with mock server #69

tomerd commented May 5, 2020

tomerd May 5, 2020

tomerd May 5, 2020

fabianfett May 5, 2020

tomerd May 5, 2020

tomerd May 5, 2020 •

edited

Loading

tomerd commented May 5, 2020

tomerd commented May 5, 2020

tomerd May 5, 2020

fabianfett May 5, 2020

tomerd commented May 5, 2020

fabianfett left a comment

fabianfett May 5, 2020

fabianfett May 5, 2020

weissi May 5, 2020

fabianfett May 5, 2020

fabianfett May 5, 2020

fabianfett May 5, 2020

fabianfett May 5, 2020

tomerd commented May 6, 2020 •

edited

Loading

tomerd commented May 15, 2020

RFC: add debug functionality to test with mock server #69

RFC: add debug functionality to test with mock server #69

Conversation

tomerd commented May 5, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tomerd May 5, 2020 • edited Loading

Choose a reason for hiding this comment

tomerd commented May 5, 2020

tomerd commented May 5, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tomerd commented May 5, 2020

fabianfett left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tomerd commented May 6, 2020 • edited Loading

tomerd commented May 15, 2020

tomerd May 5, 2020 •

edited

Loading

tomerd commented May 6, 2020 •

edited

Loading