Repository navigation
Improve error message when ECS credentials endpoint returns HTTP 429 (rate limited) #3350
Description
Activity
Hey, thanks for opening an issue and I apologize for the misleading error. I'll look into improving the error message.
Just for my understanding, were you suppressing stderr outputs? When reproducing locally I see
Error retrieving ECS Credentials: Aws::ECSCredentials::Non200Response Error Class: Aws::Errors::MissingCredentialsError Message: unable to sign request without credentials set FULL STACK TRACE: 0: /Users/wangrch/.gem/ruby/3.4.1/gems/aws-sdk-core-3.241.4/lib/aws-sdk-core/plugins/sign.rb:145:in 'Aws::Plugins::Sign::SignatureV4#sign' 1: /Users/wangrch/.gem/ruby/3.4.1/gems/aws-sdk-core-3.241.4/lib/aws-sdk-core/plugins/sign.rb:48:in 'Aws::Plugins::Sign::Handler#call' 2: /Users/wangrch/.gem/ruby/3.4.1/gems/aws-sdk-core-3.241.4/lib/aws-sdk-core/plugins/transfer_encoding.rb:27:in 'Aws::Plugins::TransferEncoding::Handler#call' 3: /Users/wangrch/.gem/ruby/3.4.1/gems/aws-sdk-core-3.241.4/lib/aws-sdk-core/plugins/helpful_socket_errors.rb:12:in 'Aws::Plugins::HelpfulSocketErrors::Handler#call' 4: /Users/wangrch/.gem/ruby/3.4.1/gems/aws-sdk-core-3.241.4/lib/aws-sdk-core/plugins/retry_errors.rb:365:in 'block in Aws::Plugins::RetryErrors::LegacyHandler#call' 5: /Users/wangrch/.gem/ruby/3.4.1/gems/aws-sdk-core-3.241.4/lib/aws-sdk-core/plugins/user_agent.rb:92:in 'Aws::Plugins::UserAgent.metric' 6: /Users/wangrch/.gem/ruby/3.4.1/gems/aws-sdk-core-3.241.4/lib/aws-sdk-core/plugins/retry_errors.rb:385:in 'Aws::Plugins::RetryErrors::LegacyHandler#with_metric' 7: /Users/wangrch/.gem/ruby/3.4.1/gems/aws-sdk-core-3.241.4/lib/aws-sdk-core/plugins/retry_errors.rb:365:in 'Aws::Plugins::RetryErrors::LegacyHandler#call' 8: /Users/wangrch/.gem/ruby/3.4.1/gems/aws-sdk-core-3.241.4/lib/aws-sdk-core/plugins/http_checksum.rb:20:in 'Aws::Plugins::HttpChecksum::Handler#call'which still isn't the best message but it does give me more information about the underlying issue.
Hello @richardwang1124 and thank you for having a look. We were not suppressing the stderr outputs and we did get the stack which led us into looking at the ECS_TASK_METADATA_RPS_LIMIT but after spending sometime thinking the issue was credential not being available initially.
Sorry, my previous comment was a little vague; I meant to ask if you were able to see the
Error retrieving ECS Credentials: Aws::ECSCredentials::Non200Responsewarning since I didn't see it in your original post.I've opened a PR to help clarify the warning message and it now looks like
Error retrieving ECS Credentials: HTTP 429: You have reached maximum request limit. Error: Aws::Errors::MissingCredentialsError - unable to sign request without credentials setI don't think I'll be able to change the actual error message surfaced (
unable to sign request without credentials set) but the team is exploring ways to create a log of credentials providers that weren't able to be resolved and why and surfacing that with the error. In the meantime let me know if the updated warning message helps!Hello @richardwang1124 Yes, the message is clear and provides the guidance needed regarding reaching the limit. thanks for your help!
This issue is now closed. Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one.
Describe the feature
When the ECS task metadata credentials endpoint returns HTTP 429 (rate limited), the Ruby SDK raises Aws::Errors::MissingCredentialsError. in my case the complete error was:
This is misleading because credentials are not missing—they are being rate-limited by the ECS agent. misleading my troubleshooting efforts
The Python SDK (boto3/botocore) provides a much clearer error message that includes the HTTP status code and the actual response from the endpoint.
Python SDK (for comparison):
The ECS agent logs do not help either as there were no rate limiting errors in the agent logs. I had to run a tcp dump command below to troubleshoot the issue:
Use Case
When running applications on Amazon ECS with IAM task roles, the ECS agent enforces rate limiting on the credentials endpoint (http://169.254.170.2/v2/credentials/) via the ECS_TASK_METADATA_RPS_LIMIT configuration (default: 40 steady, 60 burst requests per second).
When this rate limit is exceeded, the ECS agent returns HTTP 429 with the body: You have reached maximum request limit.
The current Ruby SDK error message leads developers to debug credential provider chain configuration burning hours investigating the wrong problem When the actual issue is simply rate limiting, which can be resolved by Caching/reusing SDK clients instead of creating new ones per request or adjusting ECS_TASK_METADATA_RPS_LIMIT on the ECS agent.
References
[1] https://github.com/aws/amazon-ecs-agent/blob/master/README.md#:~:text=ECS_TASK_METADATA_RPS_LIMIT
Proposed Solution
Include HTTP status code and response body in the error message
Other Information
No response
Acknowledgements
SDK version used
3.241.4
Environment details (OS name and version, etc.)
amazonlinux 2023