Connecting to a Service Deployed using Marathon Framework

In this article we’ll see how to connect and use a service deployed via Marathon. We’ll be using the mysql service from our previous article and leverage it in a sample web application.

The source code is available from the following link – https://github.com/bpradipt/sampleflaskapp.git

The sample code has dockerfiles for both Intel and PowerPC LE (ppc64le) architecture.

Connecting to a service using ‘docker links’

Once a service has been deployed, the next step is how to discover and connect – link to it from an application.  Note that a service in-turn can depend on another service. So linking containers becomes very important.

Few things needs to be kept in mind when ‘linking’ services via marathon.

  • Mesos/Marathon doesn’t have a way to use docker link alias. So if your application configuration is dependent on link alias name in any way it won’t work. For eg. if the web application depends on DB container and uses the environment variables with DB link prefix (DB_PORT, DB_TCP_ADDR etc) it won’t work. Ensure the application configuration doesn’t depend on link alias prefix in any way.
  • The linked application and the service needs to be deployed on the same host for them to communicate.

We need to use the constraints parameter to deploy linked containers on the same node.

Here is an example from https://mesosphere.github.io/marathon/docs/constraints.html

$ curl -X POST -H "Content-type: application/json" localhost:8080/v2/apps -d '{
   "id": "sleep-cluster",
   "cmd": "sleep 60",
   "instances": 3,
   "constraints": [["hostname", "CLUSTER", "a.specific.node.com"]]
}'

In order to use the above, remember to start the mesos slave with ‘hostname’ parameter. Here is an example:

# mesos-slave --master=zk://192.168.122.48:2181/mesos --containerizers=docker,mesos --executor_registration_timeout=10mins --ip=192.168.122.253 --hostname=ubuntu

Starting linked container using the marathon API

The setup was on an OpenPower (PowerPC arch) based environment but the instructions also applies to Intel based environment.
You need to specify the target container name as a value to the ‘link’ key. Additionally you need to use the constraints parameter to ensure that the new container gets deployed on the same host where the target container is running.

curl -X POST http://192.168.122.48:8080/v2/apps -d @flaskcontainer.json -H “Content-type: application/json”

# cat flaskcontainer.json
{
  "id": "flaskappcontainer",
  "cpus": 0.5,
  "mem": 64.0,
  "instances": 1,
  "container": {
    "type": "DOCKER",
    "docker": {
      "image": "ppc64le/flaskapp",
      "network": "BRIDGE",
      "portMappings": [
        { "containerPort": 80, "hostPort": 0, "servicePort": 0, "protocol": "tcp" }
      ],
      "parameters": [
                { "key": "link", "value": "mesos-b81f9a21-3133-49de-acf6-988226eb6874-S18.5d3dcaa7-05c6-4a5b-af68-dba32b7d1835"}
            ]
    }
  },
  "constraints": [
                [ "hostname","CLUSTER","ubuntu" ]
              ]
}

Using Mesos-DNS for service discovery and connection

Mesos-DNS creates application to IP address and port number mapping for each application running in a mesos cluster.

Mesos-DNS is available here. It requires ‘Go’ compiler and building it on any platform is straightforward as long as ‘Go’ compiler is there.

A sample configuration file is included with the source itself and can be found here. https://github.com/mesosphere/mesos-dns/blob/master/config.json.sample
Following is a sample configuration, that was used for the setup mentioned in this article.

{
  "zk": "zk://192.168.122.48:2181/mesos",
  "masters": ["192.168.122.48:5050"],
  "refreshSeconds": 60,
  "ttl": 60,
  "domain": "mesos",
  "port": 53,
  "resolvers": ["8.8.8.8"],
  "timeout": 5,
  "listener": "0.0.0.0",
  "SOAMname": "ns1.mesos",
  "SOARname": "root.ns1.mesos",
  "SOARefresh": 60,
  "SOARetry":   600,
  "SOAExpire":  86400,
  "SOAMinttl": 60,
  "dnson": true,
  "httpon": true,
  "httpport": 8125,
  "externalon": true,
  "IPSources": ["netinfo", "mesos", "host"],
  "EnforceRFC952": false
}

Some important fields in config.json:
“zk” : where zookeeper is running
“masters” : where masters are running
“domain” : the domain name for the mesos cluster
“port”   : mesos DNS port
“listener” : Ip that will bind to mesos-dns
“resolver” : External DNS servers
“httpport” : which will run mesos-dns http api

More details on mesos-DNS configuration parameters is available here.

You can either run mesos-dns directory on any host or run it via Marathon framework itself. Following is an example:
curl -X POST http://192.168.122.48:8080/v2/apps -d @mesos-dns.json -H “Content-type: application/json”
This will start mesos-dns through marathon.

# cat mesos-dns.json
{
    "cmd": "<path>/mesos-dns -config=<path>/config.json",
    "cpus": 1.0,
    "mem": 1024,
    "id": "mesos-dns",
    "instances": 1,
}

Running mesos-dns directly on the host

# mesos-dns -v=1 -config=<path_to_config_json>

The service is named using the following format – <service-name>.<framework>.<domain-name>.
So the mysql service DNS name will be mysql.marathon.mesos.

Mesos-DNS also creates DNS SRV record for the services. A SRV record associates a service name to a hostname and an IP port. Mesos-DNS generates a DNS-SRV record for service name as _task._protocol.framework.domain, where task is the application/service launched (mysql in our case), protocol is udp or tcp, framework is marathon or any other framework, domain is the cluster domain (mesos in our case).

This SRV record can be used by other marathon applications to discover services.

As an example any application requiring mysql service can lookup SRV  record for _mysql_tcp.marathon.mesos 

# docker ps
CONTAINER ID IMAGE          COMMAND    CREATED       STATUS                PORTS                        NAMES
e227390bfb3d ppc64le/mysql "/setup.sh" 3 seconds ago Up Less than a second 0.0.0.0:31172->3306/tcp   mesos-fabb6e52-064a-425a-a501-330bc772cd55-S16.85fb3e7c-b2ca-412f-ac75-1ec314bee575

# dig _mysql._tcp.marathon.mesos -t SRV
; <<>> DiG 9.9.4-RedHat-9.9.4-29.el7 <<>> _mysql._tcp.marathon.mesos -t SRV
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 2126
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; QUESTION SECTION:
;_mysql._tcp.marathon.mesos. IN SRV
;; ANSWER SECTION:
_mysql._tcp.marathon.mesos. 60 IN SRV 0 0 31172 mysql-4huw5-s16.marathon.slave.mesos.
;; ADDITIONAL SECTION:
mysql-4huw5-s16.marathon.slave.mesos. 60 IN A 192.168.122.48
;; Query time: 1 msec
;; SERVER: 192.168.122.48#53(192.168.122.48)
;; WHEN: Mon Feb 08 14:27:38 IST 2016
;; MSG SIZE rcvd: 147 

More details on mesos-dns naming is available here.
Here is an example python code leveraging dnspython module to query the SRV record and retrieve the host and port required to access the service.

import dns.resolver

a = dns.resolver.query("_mysql._tcp.marathon.mesos",dns.rdatatype.SRV)
for i in a.response.answer:
    for j in i.items:
        print j.target
        print j.port 

The output in the example setup is the following:
mysql-4huw5-s16.marathon.slave.mesos.
31172

From the above it can be inferred that the mysql service is running on the slave with hostname  mysql-4huw5-s16.marathon.slave.mesos and at port 31172.

A logic similar to the one above can be directly incorporated into the application config, or the data can be used to set relevant environment variables required by the application configuration. For example we can set MYSQL_TCP_ADDR and MYSQL_TCP_PORT to the values returned by ‘target’ and ‘port’ respectively.

Hope this will be of some help to you when using Mesos and Marathon for deploying dockerized application and services.

Pradipta Kumar Banerjee

I'm a Cloud and Linux/ OpenSource enthusiast, with 16 years of industry experience at IBM. You can find more details about me here - Linkedin

You may also like...