Deploying a Service on Mesos Cluster using Marathon Framework

Mesos is a distributed cluster manager aiming for improved resource utilization by dynamically sharing resources among multiple tasks. It was started at the University of California, Berkeley in 2009 and is in production use in many companies, including Twitter and Airbnb. Mesos provides a unified view of resources on all cluster nodes and seamless access to these resources in a manner similar to what an operating system kernel does for a single computer. Hence you’ll often see Mesos being called as the ‘kernel’ for the datacenter.  Mesos provides a core for building data center applications and its main component is a scalable two-phased scheduler.

mesos-arch

            Src: Apache Mesos Essentials Packt Publication

The key components of a Mesos cluster are the masters, slaves (or nodes) and frameworks. The master is the cluster manager and coordinates all the cluster operations. Multiple masters can be there for high availability. The slaves (or the nodes) are the cluster members where the tasks actually run. Frameworks are the actual ‘tasks’ that run in the cluster.  As a user, ‘Framework’ is the component that you’ll interface the most. There are lots of existing frameworks available allowing varied sets of applications and services to be deployed on Mesos.

You can refer to the following guide for more deeper details into the mesos architecture – http://mesos.apache.org/documentation/latest/architecture/
In this article we’ll see how to use Marathon framework to deploy applications and services on Mesos.

Marathon

Marathon is a framework used for running long-running applications or services on Mesos. These applications have high availability requirements, which means that Marathon should be able to monitor and automatically restart the application instance in case of a failure and should be able to elastically scale the application. Marathon can run other frameworks, such as Hadoop, as well as itself. A typical Marathon usage workflow would be to run N instances of an application somewhere within the cluster, and each of the application instance requires 1CPU and 1GB of memory. You submit this request to Marathon which creates N Mesos tasks that gets executed on the slaves.

Marathon provides a REST API for starting, stopping, and scaling services. There is a browser-based GUI and also a command line client. It can run in highly-available mode by running multiple Marathon instances.

In the remainder of this article we’ll see how to deploy a service via Marathon and make use of the service in a sample application. Note that the instructions mentioned here applies to both Intel and Power architecture (OpenPower) based servers. The service will be a simple mysql database service.

Introduction to services

A service is a self-contained, independently deployed and managed unit of functionality. Services oriented architectures (SOA) and recently, microservice architectures, encourage applications to be composed out of loosely coupled services. More and more modern applications are composed of multiple microservices, as they offer a number of advantages, such as code reuse, ease of scaling, independent failures, support for multiple platforms, flexibility in deployment, and greater agility.

Mesos is great at handling batch, real-time, and other processing frameworks, whose jobs are typically short-lived. Enterprise infrastructure runs a lot of applications and services that are long-running in nature and have different requirements than the data processing frameworks. These long-running services are not only critical for business, but they also consume a large portion of infrastructure resources. Thus, the ability to run services on Mesos is very important.

To run services at scale, the infrastructure needs to be able to support the following:

  1. Deployment of a service can be complex if service depends on other services and there are constraints about where the service can be deployed.
  2. Configuration management and packaging is about making sure all the dependencies for a service are met and the environment is configured properly for the service before the service starts.
  3. Service discovery and load balancing become important when multiple instances of a service are running. Service discovery answers the questions where are the instances of a particular service running and load balancing is about deciding which instance a given request should go to.
  4. Once the service is deployed, it is important to do health monitoring of the service. Health monitoring information can be used to take further actions, such as scaling a service up or down or re-launching them on a failure.
  5. Availability requirement demands that the service needs to be available in the case of high load and failures.

Deploying an application via Marathon

The source code is available from the link below –
https://github.com/bpradipt/docker-mysql.git

The source code contains Dockerfiles and related setup scripts to build the mysql docker image on both Intel and Power (ppc64le) systems.

For the examples below, 192.168.122.48 is the IP of the machine running mesos server and marathon.

One can use the Marathon UI or the REST API directly to deploy an application. For example, the below commands deploys the application leveraging Marathon’s REST API.

curl -X POST http://192.168.122.48:8080/v2/apps -d @mysqlcontainer.json -H “Content-type: application/json”

#cat mysqlcontainer.json
{
  "id": "mysql",
  "cpus": 0.5,
  "mem": 64.0,
  "instances": 1,
  "container": {
    "type": "DOCKER",
    "docker": {
      "image": "ppc64le/mysql",
      "network": "BRIDGE",
      "portMappings": [
        { "containerPort": 3306, "hostPort": 0, "servicePort": 0, "protocol": "tcp" }
      ]
    }
  },
  "env": { 
     "MYSQL_ROOT_PASSWORD" : "password",
     "MYSQL_USER" : "test",
     "MYSQL_PASSWORD" : "test", 
     "MYSQL_DB" : "BucketList"
   }
}

A non-zero hostPort will result in a random port being assigned. It’s also possible to specify explicit hostPort value. Only thing required is to ensure that the ports specified in hostPort are actually included in some resource offers. For example if port range from 7000-8000 needs to be used in addition to default port range of 31000-32000 then the following option needs to be used:
–resources=”ports(*):[7000-8000, 31000-32000]”

This is a very simple service, however this should help you to understand the basic concepts involved when using Mesos and Marathon. In the next part we’ll see how to use this service from another application deployed via Marathon.

Thanks to Abhishek Dasgupta and Vijay Sukthankar who collaborated with me to come up with these series.

Pradipta Kumar Banerjee

I'm a Cloud and Linux/ OpenSource enthusiast, with 16 years of industry experience at IBM. You can find more details about me here - Linkedin

You may also like...