You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Hey @tcmbilozub
I ran into this same error message, but it appears to be a different issue.
For me, it turned out to be an incorrect path to the python executable that was passed to the runtime interface emulator. Posting here to hopefully help others.
Starting mock Lambda runtime:
16 Aug 2023 16:42:49,811 [INFO] (rapid) exec '/usr/bin/python' (cwd=/, handler=awslambdaric)
16 Aug 2023 16:42:51,574 [INFO] (rapid) extensionsDisabledByLayer(/opt/disable-extensions-jwigqn8j) -> stat /opt/disable-extensions-jwigqn8j: no such file or directory
16 Aug 2023 16:42:51,574 [INFO] (rapid) Configuring and starting Operator Domain
16 Aug 2023 16:42:51,574 [INFO] (rapid) Starting runtime domain
16 Aug 2023 16:42:51,574 [WARNING] (rapid) Cannot list external agents error=open /opt/extensions: no such file or directory
16 Aug 2023 16:42:51,574 [INFO] (rapid) Starting runtime without AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN , Expected?: false
START RequestId: e4769e06-19d6-45c7-969f-a310d8769df9 Version: $LATEST
16 Aug 2023 16:42:51,574 [WARNING] (rapid) First fatal error stored in appctx: Runtime.InvalidEntrypoint
******************************************************************************************************************************************
16 Aug 2023 16:42:51,574 [ERROR] (rapid) Init failed InvokeID= error=fork/exec /usr/bin/python: no such file or directory
******************************************************************************************************************************************
16 Aug 2023 16:42:51,574 [INFO] (rapid) Starting runtime domain
16 Aug 2023 16:42:51,574 [WARNING] (rapid) Cannot list external agents error=open /opt/extensions: no such file or directory
16 Aug 2023 16:42:51,575 [INFO] (rapid) Starting runtime without AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN , Expected?: false
16 Aug 2023 16:42:51,575 [WARNING] (rapid) Omitting fatal error Runtime.InvalidEntrypoint: Runtime.InvalidEntrypoint already stored
START RequestId: 9098b9f6-2ec1-4574-976c-c1f708284870 Version: $LATEST
16 Aug 2023 16:42:55,910 [INFO] (rapid) ReserveFailed: AlreadyReserved
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x30 pc=0x699b8f]
goroutine 51 [running]:
go.amzn.com/lambda/rapidcore.(*Server).Invoke.func2()
/LambdaRuntimeLocal/lambda/rapidcore/server.go:653 +0xef
created by go.amzn.com/lambda/rapidcore.(*Server).Invoke
/LambdaRuntimeLocal/lambda/rapidcore/server.go:636 +0x23d
It checks if Reserve() fails and logs message but then continues on as if nothing was wrong while reserveResp is nil causing this segfault immediatly when trying to access reserveResp.Token
I looked at this code, and I'm not sure as to what a good fix might be. One possibility would be to pull the s.Reserve() call out of the go func into the main Invoke() body, which would make the s.Reserve() call synchronous, allowing us to return from Invoke() with an error. This appears to be roughly what the previous version of Invoke() did:
Temporarily solved this problem by downgrading to a lower version.
RUN curl -Lo /usr/local/bin/aws-lambda-rie https://github.com/aws/aws-lambda-runtime-interface-emulator/releases/download/v1.10/aws-lambda-rie \ && chmod +x /usr/local/bin/aws-lambda-rie
Thank you, @tcmbilozub , for this workaround. With the earlier version you identified at least the first request made can complete (tested with python 3.11).
Given its effect on concurrency and the popularity of lambda, I'm surprised this regression hasn't garnered more attention by now. Someone once said "speed matters in business." AWS lambda team, please consider prioritizing this for your customers.
As alluded to above, downgrading isn't really a solution, since it'll still error '[INFO] (rapid) ReserveFailed: AlreadyReserved', it just won't panic on it.
The issue seems to be that Reserve tries to update the single context, on the single Server, and so it just inherently can't serve multiple concurrent invocations:
Activity
nparker2020 commentedon Aug 16, 2023
Hey @tcmbilozub
I ran into this same error message, but it appears to be a different issue.
For me, it turned out to be an incorrect path to the python executable that was passed to the runtime interface emulator. Posting here to hopefully help others.
tcmbilozub commentedon Sep 5, 2023
Temporarily solved this problem by downgrading to a lower version.
RUN curl -Lo /usr/local/bin/aws-lambda-rie https://github.com/aws/aws-lambda-runtime-interface-emulator/releases/download/v1.10/aws-lambda-rie \ && chmod +x /usr/local/bin/aws-lambda-rie
benoit-laplante commentedon Sep 18, 2023
Having same issue with simultaneous requests under python 3.8
achernyakov-rvbd commentedon Dec 23, 2023
I too am hitting this when making simultaneous requests. Is there any workaround other than downgrading to an old version?
fs-aikito commentedon Feb 29, 2024
The offending code is here
https://github.com/aws/aws-lambda-runtime-interface-emulator/blob/develop/lambda/rapidcore/server.go#L661-L666
It checks if
Reserve()
fails and logs message but then continues on as if nothing was wrong whilereserveResp
isnil
causing this segfault immediatly when trying to accessreserveResp.Token
jmehnle commentedon May 16, 2024
The permanent link for this code section is:
aws-lambda-runtime-interface-emulator/lambda/rapidcore/server.go
Lines 661 to 666 in d37e08c
I looked at this code, and I'm not sure as to what a good fix might be. One possibility would be to pull the
s.Reserve()
call out of thego func
into the mainInvoke()
body, which would make thes.Reserve()
call synchronous, allowing us to return fromInvoke()
with an error. This appears to be roughly what the previous version ofInvoke()
did:aws-lambda-runtime-interface-emulator/lambda/rapidcore/server.go
Lines 577 to 594 in a08886c
… but it seems undoing that would contravene the intent behind moving the
s.Reserve()
call into an asynchronous goroutine.AWS, can someone please comment on what your plans are for addressing this regression?
AlreadyReserved
error #131jperezr21 commentedon Aug 13, 2024
Any updates on this? Makes testing locally impossible for some use cases.
myedibleenso commentedon Aug 17, 2024
Thank you, @tcmbilozub , for this workaround. With the earlier version you identified at least the first request made can complete (tested with python 3.11).
Given its effect on concurrency and the popularity of lambda, I'm surprised this regression hasn't garnered more attention by now. Someone once said "speed matters in business." AWS lambda team, please consider prioritizing this for your customers.
OJFord commentedon Sep 19, 2024
As alluded to above, downgrading isn't really a solution, since it'll still error '[INFO] (rapid) ReserveFailed: AlreadyReserved', it just won't panic on it.
The issue seems to be that
Reserve
tries to update the single context, on the singleServer
, and so it just inherently can't serve multiple concurrent invocations:aws-lambda-runtime-interface-emulator/lambda/rapidcore/server.go
Lines 159 to 199 in 71388dd
but I'm not that familiar with Go, nevermind this project, and I don't know the solution.
Are you able to comment @valerena?
Fix already reserved panic on concurrent invokes
OJFord commentedon Sep 19, 2024
I believe I have a fix/workaround (?) in #133.
13 remaining items