Repository navigation
Invalid method call in credential_provider_chain to Aws::Plugins::UserAgent:Class #3329
Description
Activity
Hey, thanks for opening an issue. We'll be taking a look at this.
Reacted by Frederik SchubertStill investigating, it's strange that you're receiving this error since the UserAgent plugin is part of aws-sdk-core and the
metricmethod has existed since many versions before3.233.0. Are you able to try using another type of credentials to see if you still get the same error? Not sure how feasible this is, but if you're able to provide some code to reproduce this error that would be helpful too.I would not expect the issue to appear on any other authentication method as this code is located explicitly inside the
assume_role_web_identity_credentialsmethod. Authentication works this way and logstash can operate normally so this is not a high-impact issue. I would assume simply metrics collection does not work in this case. I have noticed other authentication methods are using thewith_metricsmethod instead of callingAws::Plugins::UserAgent.metricdirectly but looking at the code there should not be a difference.I have again verified that I am running the mentioned versions of the plugin (i have removed irrelevant aws sdks from the list):
$ bin/ruby -S gem list --local | grep aws aws-sdk (3.3.0) aws-sdk-core (3.233.0) aws-sdk-opensearchservice (1.75.0) aws-sigv2 (1.3.1) aws-sigv4 (1.12.1) logstash-integration-aws (7.2.1 java) opensearch-aws-sigv4 (1.3.0)Other credential providers (such as profile assume role web id, assume role, and sso) all use the UserAgent metric method to track features, I was curious to see if you'd run into the same issue with these other providers. I tried replicating the logstash code locally but could not reproduce the error. Are you able to try a different aws-sdk-core version?
Okay I did some more investigation. There is another error happening before the mentioned one:
uninitialized constant Aws::Plugins::UserAgentstack trace:
org/jruby/RubyModule.java:4375:in `const_missing' /usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/aws-sdk-core-3.233.0/lib/aws-sdk-core/credential_provider_chain.rb:175:in `assume_role_web_identity_credentials' /usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/aws-sdk-core-3.233.0/lib/aws-sdk-core/credential_provider_chain.rb:13:in `block in resolve' org/jruby/RubyArray.java:2009:in `each' /usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/aws-sdk-core-3.233.0/lib/aws-sdk-core/credential_provider_chain.rb:12:in `resolve' /usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-opensearch-2.0.3-java/lib/logstash/outputs/opensearch/http_client/manticore_adapter.rb:87:in `aws_iam_auth_initialization' /usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-opensearch-2.0.3-java/lib/logstash/outputs/opensearch/http_client/manticore_adapter.rb:56:in `initialize'I have noticed this only happens sometimes, we have ~10 pipelines running every hour with a very similar setup and only some of them fail only sometimes with this error.
Example output pipeline configuration:
output { opensearch { hosts => ["${ES_ENDPOINT}"] ssl => true index => "${LOGSTASH_INDEX_NAME}" template => "<path-to-template>.json" template_name => "<name-of-template>" document_id => "%{id}" doc_as_upsert => true auth_type => { type => "aws_iam" region => '${AWS_REGION}' } } }I have upgraded the
aws-sdk-coreto3.240now and will check if the issue occurs again. Thank you again for taking the time to look into this 👍Thanks for your investigation! I think the uninitialized constant error for UserAgent could make more sense than the undefined method error, but the plugin is autoloaded as part of aws-sdk-core and should be defined...
You may have already looked into it, but did you notice anything different in the cases when pipelines would fail vs when they wouldn't?
Let me know whether upgrading core helps.
The issue has not appeared since upgrading to the newest version. I assume it was fixed. Thank you for your support and have a nice holiday season
Reacted by Juli Tera and Richard WangThis issue is now closed. Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one.
I am using the opensearch logstash output plugin to write data to an aws opensearch instance. Internally this gem uses the aws ruby sdk to load credentials (https://github.com/opensearch-project/logstash-output-opensearch/blob/2868d41bede5bb25ba1ef71b7c70a63ae740f445/lib/logstash/outputs/opensearch/http_client/manticore_adapter.rb#L74). I am using an assumed role web identity for authentication and thus the following code is used to load credentials from environment variables:
https://github.com/aws/aws-sdk-ruby/blame/946aefc489b6037d3fe3d680805592f178a076b9/gems/aws-sdk-core/lib/aws-sdk-core/credential_provider_chain.rb#L220-L224
This results in:
stack trace shortened.