Java permissions exception running plugin 6.2.3

[afilimonov]

I’m getting this during discovery:

java.security.AccessControlException: access denied ("java.lang.RuntimePermission" "accessDeclaredMembers")

I validated that plugin is installed correctly and plugin-security.policy file is there. I’m running the plugin with official Elasticsearch Docker image. ES starts fine without Kubernetes discovery.

[jcantrill]

Please try 6.2.3.1 which is pushed to maven via proper release workflow

[juan131]

@jcantrill I just reproduced it with 6.2.3.1:

Exception caught during discovery: Error executing: GET at: https://kubernetes.default.svc/api/v1/namespaces/default/endpoints/elasticsearch-master. Cause: access denied ("java.lang.RuntimePermission" "accessDeclaredMembers")
io.fabric8.kubernetes.client.KubernetesClientException: Error executing: GET at: https://kubernetes.default.svc/api/v1/namespaces/default/endpoints/elasticsearch-master. Cause: access denied ("java.lang.RuntimePermission" "accessDeclaredMembers")

I'm using the same RBAC configuration I was using of previous versions on the plugin.

[juan131]

It seems the src/main/plugin-metadata/plugin-security.policy was modified on commit c0404ba

Just in case it's related.

[jcantrill]

@ruromero Can you speak to why this change was made c0404ba or if you know why we just cant add this permission back?

[1] ref: https://docs.oracle.com/javase/7/docs/technotes/guides/security/permissions.html

@iocanel Can you speak to this change? I presume its coming from the kubernetes-client. Any reason we can not add this permission?

[ruromero]

@jcantrill It was mentioned but not investigated here #116
I faced this issue when upgrading k8s-client to any version > 2.0. It seems to be now using 3.0.3 since #123

[iocanel]

@jcantrill: I'm sorry. I have no idea!

[ruippeixotog]

I have just stumbled upon this issue, too. It would be great to have a hotfix release for this :)

(one unrelated question, why isn't there on Maven any 6.x version other than 6.2.3? Is there any other repo from which to install this? I am currently using Elasticsearch 6.2.1 in production, if I could avoid updating it for the time being it would be great)

[jcantrill]

CI was broken for a while. Please check the 'releases' page to see if you can get it from there in the interium

[ruippeixotog]

Unfortunately, it seems that the releases page on GitHub is also missing a 6.2.1 version... is there any chance for you to provide it somewhere? Even though publishing it on Maven would be ideal, I believe it's also easy to install a Elasticsearch plugin from an HTTP URL, so being on the releases page would be fine.

[jcantrill]

@ruippeixotog I can publish something but presume it will have the same issues described in the bz. I will provide something once we have it figured out.

[jcantrill]

@afilimonov Can you provide details of your cluster and how you exposed this issue. I locally:

• Rebuilt the 6.2.3 image mvn docker:build
• Deployed the image by creating the resources as described here
• Inserted a single record
  I don't see any errors. Possibly we have a bad build and need to republish?

# oc get services
NAME                    TYPE           CLUSTER-IP      EXTERNAL-IP                   PORT(S)          AGE
elasticsearch           LoadBalancer   172.30.166.85   172.29.90.198,172.29.90.198   9200:31223/TCP   1m
elasticsearch-cluster   ClusterIP      None            <none>                        9300/TCP         1m
logging-es              ClusterIP      172.30.44.18    <none>                        9200/TCP         11h
logging-es-cluster      ClusterIP      172.30.48.118   <none>                        9300/TCP         11h
logging-es-prometheus   ClusterIP      172.30.85.166   <none>                        443/TCP          11h
logging-kibana          ClusterIP      172.30.13.195   <none>                        443/TCP          11h
[root@openshiftdevel elasticsearch-cloud-kubernetes]# curl -k -XPOST ^C
[root@openshiftdevel elasticsearch-cloud-kubernetes]# curl -k -XPOST 172.30.166.85:9200/foo/bar/1 -d "{\"foo\":\"bar\"}"
{"error":"Content-Type header [application/x-www-form-urlencoded] is not supported","status":406}[root@openshiftdevel elasticsearch-cloud-kubernetes]# 
[root@openshiftdevel elasticsearch-cloud-kubernetes]# 
[root@openshiftdevel elasticsearch-cloud-kubernetes]# curl -k -XPOST 172.30.166.85:9200/foo/bar/1 -d "{\"foo\":\"bar\"}" -H "Content-Type: application/json"
{"_index":"foo","_type":"bar","_id":"1","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":0,"_primary_term":1}[root@openshiftdevel elasticsearch-cloud-kubernetes]# 
[root@openshiftdevel elasticsearch-cloud-kubernetes]# 
[root@openshiftdevel elasticsearch-cloud-kubernetes]# 
[root@openshiftdevel elasticsearch-cloud-kubernetes]# oc get pods
NAME                         READY     STATUS    RESTARTS   AGE
elasticsearch-client-gpn4w   1/1       Running   0          3m
elasticsearch-data-2q6vc     1/1       Running   0          3m
elasticsearch-master-p4lbs   1/1       Running   0          3m
[root@openshiftdevel elasticsearch-cloud-kubernetes]# oc logs elasticsearch-master-p4lbs
[2018-04-27T06:16:37,678][INFO ][o.e.n.Node               ] [] initializing ...
[2018-04-27T06:16:37,922][INFO ][o.e.e.NodeEnvironment    ] [Jrmu5V4] using [1] data paths, mounts [[/usr/share/elasticsearch/data (/dev/mapper/cl-root)]], net usable_space [12.4gb], net total_space [69.9gb], types [xfs]
[2018-04-27T06:16:37,923][INFO ][o.e.e.NodeEnvironment    ] [Jrmu5V4] heap size [495.3mb], compressed ordinary object pointers [true]
[2018-04-27T06:16:37,938][INFO ][o.e.n.Node               ] node name [Jrmu5V4] derived from node ID [Jrmu5V4-ShSDx9QFP63HTg]; set [node.name] to override
[2018-04-27T06:16:37,938][INFO ][o.e.n.Node               ] version[6.2.3], pid[1], build[c59ff00/2018-03-13T10:06:29.741383Z], OS[Linux/3.10.0-693.11.6.el7.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_162/25.162-b12]
[2018-04-27T06:16:37,938][INFO ][o.e.n.Node               ] JVM arguments [-Xms512m, -Xmx512m, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -XX:+HeapDumpOnOutOfMemoryError, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/usr/share/elasticsearch/config]
[2018-04-27T06:16:40,984][INFO ][o.e.p.PluginsService     ] [Jrmu5V4] loaded module [aggs-matrix-stats]
[2018-04-27T06:16:40,984][INFO ][o.e.p.PluginsService     ] [Jrmu5V4] loaded module [analysis-common]
[2018-04-27T06:16:40,984][INFO ][o.e.p.PluginsService     ] [Jrmu5V4] loaded module [ingest-common]
[2018-04-27T06:16:40,984][INFO ][o.e.p.PluginsService     ] [Jrmu5V4] loaded module [lang-expression]
[2018-04-27T06:16:40,984][INFO ][o.e.p.PluginsService     ] [Jrmu5V4] loaded module [lang-mustache]
[2018-04-27T06:16:40,984][INFO ][o.e.p.PluginsService     ] [Jrmu5V4] loaded module [lang-painless]
[2018-04-27T06:16:40,985][INFO ][o.e.p.PluginsService     ] [Jrmu5V4] loaded module [mapper-extras]
[2018-04-27T06:16:40,985][INFO ][o.e.p.PluginsService     ] [Jrmu5V4] loaded module [parent-join]
[2018-04-27T06:16:40,985][INFO ][o.e.p.PluginsService     ] [Jrmu5V4] loaded module [percolator]
[2018-04-27T06:16:40,985][INFO ][o.e.p.PluginsService     ] [Jrmu5V4] loaded module [rank-eval]
[2018-04-27T06:16:40,985][INFO ][o.e.p.PluginsService     ] [Jrmu5V4] loaded module [reindex]
[2018-04-27T06:16:40,985][INFO ][o.e.p.PluginsService     ] [Jrmu5V4] loaded module [repository-url]
[2018-04-27T06:16:40,985][INFO ][o.e.p.PluginsService     ] [Jrmu5V4] loaded module [transport-netty4]
[2018-04-27T06:16:40,985][INFO ][o.e.p.PluginsService     ] [Jrmu5V4] loaded module [tribe]
[2018-04-27T06:16:40,985][INFO ][o.e.p.PluginsService     ] [Jrmu5V4] no plugins loaded
[2018-04-27T06:16:49,688][INFO ][o.e.d.DiscoveryModule    ] [Jrmu5V4] using discovery type [zen]
[2018-04-27T06:16:51,902][INFO ][o.e.n.Node               ] initialized
[2018-04-27T06:16:51,902][INFO ][o.e.n.Node               ] [Jrmu5V4] starting ...
[2018-04-27T06:16:52,273][INFO ][o.e.t.TransportService   ] [Jrmu5V4] publish_address {172.17.0.4:9300}, bound_addresses {[::]:9300}
[2018-04-27T06:16:52,292][INFO ][o.e.b.BootstrapChecks    ] [Jrmu5V4] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2018-04-27T06:16:55,457][INFO ][o.e.c.s.MasterService    ] [Jrmu5V4] zen-disco-elected-as-master ([0] nodes joined), reason: new_master {Jrmu5V4}{Jrmu5V4-ShSDx9QFP63HTg}{IH6cSWs1TBO9kUKFJnw3RQ}{172.17.0.4}{172.17.0.4:9300}
[2018-04-27T06:16:55,461][INFO ][o.e.c.s.ClusterApplierService] [Jrmu5V4] new_master {Jrmu5V4}{Jrmu5V4-ShSDx9QFP63HTg}{IH6cSWs1TBO9kUKFJnw3RQ}{172.17.0.4}{172.17.0.4:9300}, reason: apply cluster state (from master [master {Jrmu5V4}{Jrmu5V4-ShSDx9QFP63HTg}{IH6cSWs1TBO9kUKFJnw3RQ}{172.17.0.4}{172.17.0.4:9300} committed version [1] source [zen-disco-elected-as-master ([0] nodes joined)]])
[2018-04-27T06:16:55,489][INFO ][o.e.h.n.Netty4HttpServerTransport] [Jrmu5V4] publish_address {172.17.0.4:9200}, bound_addresses {[::]:9200}
[2018-04-27T06:16:55,489][INFO ][o.e.n.Node               ] [Jrmu5V4] started
[2018-04-27T06:16:55,503][INFO ][o.e.g.GatewayService     ] [Jrmu5V4] recovered [0] indices into cluster_state
[root@openshiftdevel elasticsearch-cloud-kubernetes]# oc logs elasticsearch-data-2q6vc
[2018-04-27T06:16:35,757][INFO ][o.e.n.Node               ] [] initializing ...
[2018-04-27T06:16:35,852][INFO ][o.e.e.NodeEnvironment    ] [2_AbUu6] using [1] data paths, mounts [[/usr/share/elasticsearch/data (/dev/mapper/cl-root)]], net usable_space [12.4gb], net total_space [69.9gb], types [xfs]
[2018-04-27T06:16:35,852][INFO ][o.e.e.NodeEnvironment    ] [2_AbUu6] heap size [495.3mb], compressed ordinary object pointers [true]
[2018-04-27T06:16:35,853][INFO ][o.e.n.Node               ] node name [2_AbUu6] derived from node ID [2_AbUu68S0GWemHwU7kPtg]; set [node.name] to override
[2018-04-27T06:16:35,854][INFO ][o.e.n.Node               ] version[6.2.3], pid[1], build[c59ff00/2018-03-13T10:06:29.741383Z], OS[Linux/3.10.0-693.11.6.el7.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_162/25.162-b12]
[2018-04-27T06:16:35,854][INFO ][o.e.n.Node               ] JVM arguments [-Xms512m, -Xmx512m, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -XX:+HeapDumpOnOutOfMemoryError, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/usr/share/elasticsearch/config]
[2018-04-27T06:16:38,607][INFO ][o.e.p.PluginsService     ] [2_AbUu6] loaded module [aggs-matrix-stats]
[2018-04-27T06:16:38,607][INFO ][o.e.p.PluginsService     ] [2_AbUu6] loaded module [analysis-common]
[2018-04-27T06:16:38,607][INFO ][o.e.p.PluginsService     ] [2_AbUu6] loaded module [ingest-common]
[2018-04-27T06:16:38,607][INFO ][o.e.p.PluginsService     ] [2_AbUu6] loaded module [lang-expression]
[2018-04-27T06:16:38,608][INFO ][o.e.p.PluginsService     ] [2_AbUu6] loaded module [lang-mustache]
[2018-04-27T06:16:38,608][INFO ][o.e.p.PluginsService     ] [2_AbUu6] loaded module [lang-painless]
[2018-04-27T06:16:38,608][INFO ][o.e.p.PluginsService     ] [2_AbUu6] loaded module [mapper-extras]
[2018-04-27T06:16:38,608][INFO ][o.e.p.PluginsService     ] [2_AbUu6] loaded module [parent-join]
[2018-04-27T06:16:38,608][INFO ][o.e.p.PluginsService     ] [2_AbUu6] loaded module [percolator]
[2018-04-27T06:16:38,608][INFO ][o.e.p.PluginsService     ] [2_AbUu6] loaded module [rank-eval]
[2018-04-27T06:16:38,608][INFO ][o.e.p.PluginsService     ] [2_AbUu6] loaded module [reindex]
[2018-04-27T06:16:38,608][INFO ][o.e.p.PluginsService     ] [2_AbUu6] loaded module [repository-url]
[2018-04-27T06:16:38,608][INFO ][o.e.p.PluginsService     ] [2_AbUu6] loaded module [transport-netty4]
[2018-04-27T06:16:38,608][INFO ][o.e.p.PluginsService     ] [2_AbUu6] loaded module [tribe]
[2018-04-27T06:16:38,608][INFO ][o.e.p.PluginsService     ] [2_AbUu6] no plugins loaded
[2018-04-27T06:16:46,801][INFO ][o.e.d.DiscoveryModule    ] [2_AbUu6] using discovery type [zen]
[2018-04-27T06:16:49,813][INFO ][o.e.n.Node               ] initialized
[2018-04-27T06:16:49,814][INFO ][o.e.n.Node               ] [2_AbUu6] starting ...
[2018-04-27T06:16:50,275][INFO ][o.e.t.TransportService   ] [2_AbUu6] publish_address {172.17.0.2:9300}, bound_addresses {[::]:9300}
[2018-04-27T06:16:50,360][INFO ][o.e.b.BootstrapChecks    ] [2_AbUu6] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2018-04-27T06:16:53,620][INFO ][o.e.c.s.MasterService    ] [2_AbUu6] zen-disco-elected-as-master ([0] nodes joined), reason: new_master {2_AbUu6}{2_AbUu68S0GWemHwU7kPtg}{2kAX0hLZQ_iupLsjwyASdw}{172.17.0.2}{172.17.0.2:9300}
[2018-04-27T06:16:53,629][INFO ][o.e.c.s.ClusterApplierService] [2_AbUu6] new_master {2_AbUu6}{2_AbUu68S0GWemHwU7kPtg}{2kAX0hLZQ_iupLsjwyASdw}{172.17.0.2}{172.17.0.2:9300}, reason: apply cluster state (from master [master {2_AbUu6}{2_AbUu68S0GWemHwU7kPtg}{2kAX0hLZQ_iupLsjwyASdw}{172.17.0.2}{172.17.0.2:9300} committed version [1] source [zen-disco-elected-as-master ([0] nodes joined)]])
[2018-04-27T06:16:53,671][INFO ][o.e.h.n.Netty4HttpServerTransport] [2_AbUu6] publish_address {172.17.0.2:9200}, bound_addresses {[::]:9200}
[2018-04-27T06:16:53,671][INFO ][o.e.n.Node               ] [2_AbUu6] started
[2018-04-27T06:16:53,682][INFO ][o.e.g.GatewayService     ] [2_AbUu6] recovered [0] indices into cluster_state
[root@openshiftdevel elasticsearch-cloud-kubernetes]# oc logs elasticsearch-client-gpn4w
[2018-04-27T06:16:36,063][INFO ][o.e.n.Node               ] [] initializing ...
[2018-04-27T06:16:36,238][INFO ][o.e.e.NodeEnvironment    ] [iXBmxhu] using [1] data paths, mounts [[/usr/share/elasticsearch/data (/dev/mapper/cl-root)]], net usable_space [12.4gb], net total_space [69.9gb], types [xfs]
[2018-04-27T06:16:36,238][INFO ][o.e.e.NodeEnvironment    ] [iXBmxhu] heap size [495.3mb], compressed ordinary object pointers [true]
[2018-04-27T06:16:36,241][INFO ][o.e.n.Node               ] node name [iXBmxhu] derived from node ID [iXBmxhu5SnC4WH_VCJYSuA]; set [node.name] to override
[2018-04-27T06:16:36,241][INFO ][o.e.n.Node               ] version[6.2.3], pid[1], build[c59ff00/2018-03-13T10:06:29.741383Z], OS[Linux/3.10.0-693.11.6.el7.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_162/25.162-b12]
[2018-04-27T06:16:36,241][INFO ][o.e.n.Node               ] JVM arguments [-Xms512m, -Xmx512m, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -XX:+HeapDumpOnOutOfMemoryError, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/usr/share/elasticsearch/config]
[2018-04-27T06:16:38,138][INFO ][o.e.p.PluginsService     ] [iXBmxhu] loaded module [aggs-matrix-stats]
[2018-04-27T06:16:38,138][INFO ][o.e.p.PluginsService     ] [iXBmxhu] loaded module [analysis-common]
[2018-04-27T06:16:38,138][INFO ][o.e.p.PluginsService     ] [iXBmxhu] loaded module [ingest-common]
[2018-04-27T06:16:38,138][INFO ][o.e.p.PluginsService     ] [iXBmxhu] loaded module [lang-expression]
[2018-04-27T06:16:38,138][INFO ][o.e.p.PluginsService     ] [iXBmxhu] loaded module [lang-mustache]
[2018-04-27T06:16:38,138][INFO ][o.e.p.PluginsService     ] [iXBmxhu] loaded module [lang-painless]
[2018-04-27T06:16:38,138][INFO ][o.e.p.PluginsService     ] [iXBmxhu] loaded module [mapper-extras]
[2018-04-27T06:16:38,138][INFO ][o.e.p.PluginsService     ] [iXBmxhu] loaded module [parent-join]
[2018-04-27T06:16:38,139][INFO ][o.e.p.PluginsService     ] [iXBmxhu] loaded module [percolator]
[2018-04-27T06:16:38,143][INFO ][o.e.p.PluginsService     ] [iXBmxhu] loaded module [rank-eval]
[2018-04-27T06:16:38,143][INFO ][o.e.p.PluginsService     ] [iXBmxhu] loaded module [reindex]
[2018-04-27T06:16:38,143][INFO ][o.e.p.PluginsService     ] [iXBmxhu] loaded module [repository-url]
[2018-04-27T06:16:38,143][INFO ][o.e.p.PluginsService     ] [iXBmxhu] loaded module [transport-netty4]
[2018-04-27T06:16:38,144][INFO ][o.e.p.PluginsService     ] [iXBmxhu] loaded module [tribe]
[2018-04-27T06:16:38,144][INFO ][o.e.p.PluginsService     ] [iXBmxhu] no plugins loaded
[2018-04-27T06:16:46,863][INFO ][o.e.d.DiscoveryModule    ] [iXBmxhu] using discovery type [zen]
[2018-04-27T06:16:48,054][INFO ][o.e.n.Node               ] initialized
[2018-04-27T06:16:48,054][INFO ][o.e.n.Node               ] [iXBmxhu] starting ...
[2018-04-27T06:16:48,447][INFO ][o.e.t.TransportService   ] [iXBmxhu] publish_address {172.17.0.3:9300}, bound_addresses {[::]:9300}
[2018-04-27T06:16:48,505][INFO ][o.e.b.BootstrapChecks    ] [iXBmxhu] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2018-04-27T06:16:51,864][INFO ][o.e.c.s.MasterService    ] [iXBmxhu] zen-disco-elected-as-master ([0] nodes joined), reason: new_master {iXBmxhu}{iXBmxhu5SnC4WH_VCJYSuA}{0225hpY8Tqypm2fpkEpGmA}{172.17.0.3}{172.17.0.3:9300}
[2018-04-27T06:16:51,870][INFO ][o.e.c.s.ClusterApplierService] [iXBmxhu] new_master {iXBmxhu}{iXBmxhu5SnC4WH_VCJYSuA}{0225hpY8Tqypm2fpkEpGmA}{172.17.0.3}{172.17.0.3:9300}, reason: apply cluster state (from master [master {iXBmxhu}{iXBmxhu5SnC4WH_VCJYSuA}{0225hpY8Tqypm2fpkEpGmA}{172.17.0.3}{172.17.0.3:9300} committed version [1] source [zen-disco-elected-as-master ([0] nodes joined)]])
[2018-04-27T06:16:51,959][INFO ][o.e.h.n.Netty4HttpServerTransport] [iXBmxhu] publish_address {172.17.0.3:9200}, bound_addresses {[::]:9200}
[2018-04-27T06:16:51,959][INFO ][o.e.n.Node               ] [iXBmxhu] started
[2018-04-27T06:16:52,003][INFO ][o.e.g.GatewayService     ] [iXBmxhu] recovered [0] indices into cluster_state
[2018-04-27T06:19:49,812][INFO ][o.e.c.m.MetaDataCreateIndexService] [iXBmxhu] [foo] creating index, cause [auto(bulk api)], templates [], shards [5]/[1], mappings []
[2018-04-27T06:19:50,238][INFO ][o.e.c.m.MetaDataMappingService] [iXBmxhu] [foo/WTkENKo2QcaWG6kcee4LEA] create_mapping [bar]