mysql-exporter use invalid ip

[gimler]

I have two docker compose file one for the prometheus stuff one for my project. I share the network but the mysql-exporter use another ip address?

Error message Error pinging mysqld: dial tcp 10.67.76.55:3306: getsockopt: no route to host

When i log into the container with docker exec -it prometheus_mysqld-exporter_1 sh and call ping db i get 172.21.0.2 instate of 10.67.76.55

ping db

PING db (172.21.0.2): 56 data bytes
64 bytes from 172.21.0.2: seq=0 ttl=64 time=0.070 ms
64 bytes from 172.21.0.2: seq=1 ttl=64 time=0.055 ms

mysql-exporter docker-compose.yml

version: '2'

services:
mysqld-exporter:
    image: "prom/mysqld-exporter"
    expose:
      - "9104"
    external_links:
      - db-master:db
    environment:
      - DATA_SOURCE_NAME=root:123456@(db:3306)/
    networks:
      - default
      - project

networks:
  project:
    external:
      name: project_default

project specific docker-compose.yml

version: '2'

services:
  db-master:
    restart: always
    image: mariadb:10.1.12
    expose:
      - "3306"
    environment:
      - MYSQL_ROOT_PASSWORD=123456

[SuperQ]

Typically you run the mysqld_exporter on the localhost with your mysqld, not remotely.

[matthiasr]

But docker-compose is a valid use case. What versions of docker and docker-compose is this with?

[gimler]

Docker version 1.11.2, build b9f10c9
docker-compose version 1.7.0, build 0d7bf73

[matthiasr]

Alright I will see if I can reproduce this, although I don't have an idea how this could happen in the first place – @gimler any idea what 10.67.76.55 is? it must be getting that somewhere.

As a workaround, you could try putting the exporter into the DB's container network directly and reaching the DB over 10.127.0.0.1.

[gimler]

my actual workaround ist that i first start the project look what the ip is and than change the the compose file and use the ip instate of hostname

[SuperQ]

@gimler I'm not sure what needs to be changed in mysqld_exporter here. Is this still a problem?

[matthiasr]

I tried to reproduce this locally and could not. Since 10.67.76.55 does not simply appear out of nowhere, there must be a way that db is resolved to that IP in your environment; possibly only for a while, so that when you docker execd in it was resolvable correctly.

Try using extends instead of banging the exporter in with an external network, that way docker-compose can better understand what's going on and make sure everything is set up right from the start.

[szgaljic]

I have a very similar issue right now. I am starting a MySQL database + Prometheus exporter in the same docker-compose file and it looks like Prometheus is using the wrong IP.

This is the error I'm seeing using docker logs to inspect the Prometheus MySQL exporter container:

time="2017-03-29T03:17:19Z" level=error msg="Error pinging mysqld: Error 1045: Access denied for user '\"root'@'172.19.0.4' (using password: YES)" source="mysqld_exporter.go:252"

It's using the docker assigned IP-Address of localhost (172.19.0.4) and not that of the linked mysql_db container (as defined in the compose file).

Here is the matching docker network:

[szgaljic@orbiter docker-example]$ docker network inspect dockerexample_default
[
    {
        "Name": "dockerexample_default",
        "Id": "f618e918b40e01675386528d23a800b35b82ea37312a93611192f441753e68dc",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "172.19.0.0/16",
                    "Gateway": "172.19.0.1/16"
                }
            ]
        },
        "Internal": false,
        "Containers": {
            "ba76a61bfb9a8d38c8c6d3657783d36073520712bb02e54d6442172394fd8d2e": {
                "Name": "prometheus_mysql",
                "EndpointID": "fb3a9faaea23abbdb06a183c6c988e1fd2b44e07c28dacd3cfdbe984a99c22e7",
                "MacAddress": "[I manually removed this]",
                "IPv4Address": "172.19.0.4/16",
                "IPv6Address": ""
            },
            "d1498284404a455fc37a294edcb5c742baaf69454ff14f57485a164e01eb5a47": {
                "Name": "mysql_db",
                "EndpointID": "dccd78ce5750225b5786617ef2a3db93929d87b42c5df569b5dd293d6b53e589",
                "MacAddress": "[I manually removed this]",
                "IPv4Address": "172.19.0.2/16",
                "IPv6Address": ""
            }
        },
        "Options": {},
        "Labels": {}
    }
]

Here is the docker file:

version: '2'
services:

  mysql_db:
    image: "mysql:5.7.15"
    container_name: mysql_db
    ports:
     - "3306:3306"
    volumes:
     - ./mysql_db:/etc/mysql/conf.d
    environment:
     - MYSQL_ROOT_PASSWORD=pass
     - MYSQL_DATABASE=mydb
     - MYSQL_USER=myuser
     - MYSQL_PASSWORD=mypass
    command: --lower-case-table-names=1 --performance_schema

  #
  # Prometheus
  #

  prometheus_mysql:
    image: "prom/mysqld-exporter:v0.9.0"
    container_name: prometheus_mysql
    depends_on: 
    - mysql_db
    ports:
     - "9104:9104"
    environment:
    - DATA_SOURCE_NAME="root:pass@(mysql_db:3306)/performance_schema" 

Here are my versions:

[szgaljic@orbiter jg-mysql-replication]$ docker -v
Docker version 1.12.6, build 78d1802
[szgaljic@orbiter jg-mysql-replication]$ docker-compose -v
docker-compose version 1.6.2, build 4d72027

[fanhualei]

I have a very similar issue right now.

"Error pinging mysqld: Error 1045: Access denied for user '"exporter'@'my-monitor-pro_mysql-exporter_1.myapp_default' (using password: YES)" source="exporter.go:146"

version: '2'
services:

  mysql_db:
    image: "mysql:5.7.15"
    container_name: mysql_db
    ports:
     - "3306:3306"
    volumes:
     - ./mysql_db:/etc/mysql/conf.d
    environment:
     - MYSQL_ROOT_PASSWORD=pass
     - MYSQL_DATABASE=mydb
     - MYSQL_USER=myuser
     - MYSQL_PASSWORD=mypass
    command: --lower-case-table-names=1 --performance_schema

  #
  # Prometheus
  #

  prometheus_mysql:
    image: "prom/mysqld-exporter:v0.9.0"
    container_name: prometheus_mysql
    depends_on: 
    - mysql_db
    ports:
     - "9104:9104"
    environment:
    - DATA_SOURCE_NAME="root:pass@(mysql_db:3306)/performance_schema" 

[Shadowstrider-Jackie]

• DATA_SOURCE_NAME=root:pw123456@(mysql:3306)/
  Do not use double or single quotes