Latest version is causing codedeploy to fail on applicationstop everytime

[jasong1987]

Upgraded the host agent to latest version and its now failing deployment on applicationstop on the replacement instance everytime. this did not do this in v1.0.0

[brndnblck]

It's highly likely that this is a failure in a deployment script and not related to the agent itself.

What do the agent logs tell you?

Please do the following and provide log data for a failed deployment:

1. Enable verbose logging on your agent.
   https://docs.aws.amazon.com/codedeploy/latest/userguide/reference-agent-configuration.html

2. Provide agent logs which capture the unpack step of the deployment process.
   https://docs.aws.amazon.com/codedeploy/latest/userguide/deployments-view-logs.html

[stekern]

I seem to be getting the same behavior after updating from 1.2.1 to 1.3.0 -- with no updates to the deployment script. Deployments do not seem to work, and the logs are filled with errors.

On another note: I can't see that version 1.3.0 of the CodeDeploy agent has been released on GitHub, but the SSM document AWS-ConfigurePackage was able to install it when targeting the latest version.

EDIT: I downgraded the agent from 1.3.0 to 1.2.1, and now the deployments work as expected. Seems like there are issues with the newest version.

Here is a log snippet from 1.3.0:

2020-11-25 15:04:03 INFO  [codedeploy-agent(71353)]: master 71353: Spawned child 1/1
2020-11-25 15:04:04 INFO  [codedeploy-agent(71353)]: Started master 71353 with 1 children
2020-11-25 15:04:06 INFO  [codedeploy-agent(71516)]: Could not acquire lock on /opt/codedeploy-agent/state/.pid/codedeploy-agent.pid.lock - aborting start!
2020-11-25 15:06:04 INFO  [codedeploy-agent(71357)]: IMDSv2 http request failed, falling back to IMDSv1.
2020-11-25 15:08:04 ERROR [codedeploy-agent(71357)]: booting child: error during start or run: Net::OpenTimeout - execution expired - /usr/share/ruby/net/http.rb:878:in `initialize'
/usr/share/ruby/net/http.rb:878:in `open'
/usr/share/ruby/net/http.rb:878:in `block in connect'
/usr/share/ruby/net/http.rb:877:in `connect'
/usr/share/ruby/net/http.rb:862:in `do_start'
/usr/share/ruby/net/http.rb:851:in `start'
/usr/share/ruby/net/http.rb:582:in `start'
/opt/codedeploy-agent/lib/instance_metadata.rb:57:in `http_request'
/opt/codedeploy-agent/lib/instance_metadata.rb:77:in `get_request'
/opt/codedeploy-agent/lib/instance_metadata.rb:86:in `rescue in doc'
/opt/codedeploy-agent/lib/instance_metadata.rb:81:in `doc'
/opt/codedeploy-agent/lib/instance_metadata.rb:30:in `region'
/opt/codedeploy-agent/vendor/gems/codedeploy-commands-1.0.0/lib/aws/plugins/deploy_control_endpoint.rb:8:in `block in <class:DeployControlEndpoint>'
/opt/codedeploy-agent/vendor/gems/aws-sdk-core-3.109.0/lib/seahorse/client/configuration.rb:72:in `call'
/opt/codedeploy-agent/vendor/gems/aws-sdk-core-3.109.0/lib/seahorse/client/configuration.rb:72:in `call'
/opt/codedeploy-agent/vendor/gems/aws-sdk-core-3.109.0/lib/seahorse/client/configuration.rb:215:in `block in resolve_defaults'
/opt/codedeploy-agent/vendor/gems/aws-sdk-core-3.109.0/lib/seahorse/client/configuration.rb:59:in `each'
/opt/codedeploy-agent/vendor/gems/aws-sdk-core-3.109.0/lib/seahorse/client/configuration.rb:59:in `each'
/opt/codedeploy-agent/vendor/gems/aws-sdk-core-3.109.0/lib/seahorse/client/configuration.rb:214:in `resolve_defaults'
/opt/codedeploy-agent/vendor/gems/aws-sdk-core-3.109.0/lib/seahorse/client/configuration.rb:207:in `value_at'
/opt/codedeploy-agent/vendor/gems/aws-sdk-core-3.109.0/lib/seahorse/client/configuration.rb:191:in `block in resolve'
/usr/share/ruby/set.rb:232:in `each_key'
/usr/share/ruby/set.rb:232:in `each'
/opt/codedeploy-agent/vendor/gems/aws-sdk-core-3.109.0/lib/seahorse/client/configuration.rb:191:in `resolve'
/opt/codedeploy-agent/vendor/gems/aws-sdk-core-3.109.0/lib/seahorse/client/configuration.rb:179:in `apply_defaults'
/opt/codedeploy-agent/vendor/gems/aws-sdk-core-3.109.0/lib/seahorse/client/configuration.rb:152:in `build!'
/opt/codedeploy-agent/vendor/gems/aws-sdk-core-3.109.0/lib/seahorse/client/base.rb:65:in `build_config'
/opt/codedeploy-agent/vendor/gems/aws-sdk-core-3.109.0/lib/seahorse/client/base.rb:22:in `initialize'
(eval):777:in `initialize'
/opt/codedeploy-agent/vendor/gems/aws-sdk-core-3.109.0/lib/seahorse/client/base.rb:102:in `new'
/opt/codedeploy-agent/lib/instance_agent/plugins/codedeploy/codedeploy_control.rb:41:in `get_client'
/opt/codedeploy-agent/lib/instance_agent/plugins/codedeploy/command_poller.rb:46:in `initialize'
/opt/codedeploy-agent/lib/instance_agent/agent/base.rb:10:in `new'
/opt/codedeploy-agent/lib/instance_agent/agent/base.rb:10:in `runner'
/opt/codedeploy-agent/lib/instance_agent/runner/child.rb:32:in `block in prepare_run'
/opt/codedeploy-agent/lib/instance_agent/runner/child.rb:78:in `with_error_handling'
/opt/codedeploy-agent/lib/instance_agent/runner/child.rb:31:in `prepare_run'
/opt/codedeploy-agent/vendor/gems/process_manager-0.0.13/lib/process_manager/child.rb:64:in `block in prepare_run_with_error_handling'
/opt/codedeploy-agent/lib/instance_agent/runner/child.rb:78:in `with_error_handling'
/opt/codedeploy-agent/vendor/gems/process_manager-0.0.13/lib/process_manager/child.rb:63:in `prepare_run_with_error_handling'
/opt/codedeploy-agent/vendor/gems/process_manager-0.0.13/lib/process_manager/child.rb:20:in `start'
/opt/codedeploy-agent/vendor/gems/process_manager-0.0.13/lib/process_manager/master.rb:206:in `block in spawn_child'
/opt/codedeploy-agent/vendor/gems/process_manager-0.0.13/lib/process_manager/master.rb:204:in `fork'
/opt/codedeploy-agent/vendor/gems/process_manager-0.0.13/lib/process_manager/master.rb:204:in `spawn_child'
/opt/codedeploy-agent/vendor/gems/process_manager-0.0.13/lib/process_manager/master.rb:196:in `block in spawn_children'
/opt/codedeploy-agent/vendor/gems/process_manager-0.0.13/lib/process_manager/master.rb:195:in `times'
/opt/codedeploy-agent/vendor/gems/process_manager-0.0.13/lib/process_manager/master.rb:195:in `spawn_children'
/opt/codedeploy-agent/vendor/gems/process_manager-0.0.13/lib/process_manager/master.rb:134:in `start'
/opt/codedeploy-agent/vendor/gems/process_manager-0.0.13/lib/process_manager/master.rb:37:in `block in start'
/opt/codedeploy-agent/vendor/gems/process_manager-0.0.13/lib/process_manager/master.rb:36:in `fork'
/opt/codedeploy-agent/vendor/gems/process_manager-0.0.13/lib/process_manager/master.rb:36:in `start'
/opt/codedeploy-agent/bin/../lib/codedeploy-agent.rb:43:in `block (2 levels) in <main>'

[philstrong]

Confirmed we released a bug in 1.3.0 for on premise deployments unrelated to this issue. 1.3.1 has a tested fix, and we're rolling it out soon.

[philstrong]

1.3.1 should resolve this issue.