How to Setup OpenStack to use Local Disks for Instances

Quite often application requirements would demand using locally attached disks (or direct attached disks) for OpenStack compute instances. One such example is running virtual hadoop clusters via OpenStack.

A virtual hadoop cluster on OpenStack leveraging local disks will look something like this:

local_disk_arch

As you can see from the diagram above, a compute node performs an additional role of a storage node, serving local disks to the instances( Virtual Machines).

The split of OpenStack services will be like the following

Compute + Storage Node

  1. Nova-compute

  2. Cinder-volume

Controller Node

  1. Nova-api

  2. Cinder-scheduler

  3. Cinder-api

  4. Glance

  5. Neutron

In summary, we need to set up a compute node as a cinder volume node as well, so as to expose the local disks as cinder volumes. There are two cinder drivers which can be made use of – LVMVolumeDriver, BlockDeviceDriver.

I’ll be showing the usage of BlockDeviceDriver which allows using plain block devices with OpenStack instances.

Additionally, the following should be kept in mind:

  1. The default cinder quota might not be sufficient for real usage, and therefore, you need to change it accordingly.

  2. As of this writing, OpenStack doesn’t have a way to automatically ensure coexistence of instance, and cinder volume, on the same node. In other words, it could happen that the cinder volume that is being attached to an instance is from a remote node, ie. from a node not running the instance. This is depicted in Scenario-1. For applications, like hadoop we would ideally like to avoid scenario-1, where a disk is served to an instance over iSCSI. Although, network bandwidths have improved significantly, and in most of the cases iSCSI should be fine, but for hadoop workloads, this could get into potential scalability issues as the number of compute and data volume grows. Instead, we might want to have Scenario-2. There is a way to achieve this manually, and I’ll describe more details in subsequent sections.

 

scen-1

 

scen-2

 

 

In the following section, I’ll show you the configuration details by taking the example of IBM PowerKVM server as a compute node. The same configuration also applies to Intel/KVM compute node.

My test setup has the following disk layout.

  1. /dev/sda : hosts the PowerKVM OS as well nova directory (/var/lib/nova)
  2. /dev/sdb : local disk to be provided to OpenStack instances.

The disks can be backed by hardware RAID or the disks can be from direct attached storage enclosures. Point to note, is that, the disks are local to the host OS.

Further, my setup as the following compute and controller nodes

1. OpenStack controller node – icmnode1

2. OpenStack compute nodes (PowerKVM) – icmhost1 and icmhost2

 

Step -1. Create separate nova availability zone per compute node

Create a host aggregate and an availability zone.

[root@icmnode1 ~]# nova aggregate-create icmhost2 icmhost2
+----+----------+-------------------+-------+------------------------------+
| Id | Name     | Availability Zone | Hosts | Metadata                     |
+----+----------+-------------------+-------+------------------------------+
| 1  | icmhost2 | icmhost2          |       | 'availability_zone=icmhost2' |
+----+----------+-------------------+-------+------------------------------+

[root@icmnode1 ~]# nova host-list
+------------------+-------------+----------+
| host_name        | service     | zone     |
+------------------+-------------+----------+
| icmnode1.ibm.com | conductor   | internal |
| icmnode1.ibm.com | scheduler   | internal |
| icmnode1.ibm.com | consoleauth | internal |
| icmhost1.ibm.com | compute     | nova     |
| icmhost2.ibm.com | compute     | nova     |
+------------------+-------------+----------+

Add the compute node to the host aggregate

[root@icmnode1 ~]# nova aggregate-add-host 1 icmhost2.ibm.com
Host icmhost2.ibm.com has been successfully added for aggregate 1
+----+----------+-------------------+--------------------+------------------------------+
| Id | Name     | Availability Zone | Hosts              | Metadata                     |
+----+----------+-------------------+--------------------+------------------------------+
| 1  | icmhost2 | icmhost2          | 'icmhost2.ibm.com' | 'availability_zone=icmhost2' |
+----+----------+-------------------+--------------------+------------------------------+

Check if availability zones are created or not.

[root@icmnode1 ~]# nova-manage service list
Binary           Host             Zone     Status      State Updated_At
nova-conductor   icmnode1.ibm.com internal enabled :-) 2014-11-02 19:18:25
nova-scheduler   icmnode1.ibm.com internal enabled :-) 2014-11-02 19:18:21
nova-consoleauth icmnode1.ibm.com internal enabled :-) 2014-11-02 19:18:23
nova-compute     icmhost1.ibm.com nova     enabled :-) 2014-11-02 19:18:25
nova-compute     icmhost2.ibm.com icmhost2 enabled :-) 2014-11-02 19:18:24

[root@icmnode1 ~]# nova aggregate-create icmhost1 icmhost1
+----+----------+-------------------+-------+------------------------------+
| Id | Name     | Availability Zone | Hosts | Metadata                     |
+----+----------+-------------------+-------+------------------------------+
| 2  | icmhost1 | icmhost1          |       | 'availability_zone=icmhost1' |
+----+----------+-------------------+-------+------------------------------+

[root@icmnode1 ~]# nova aggregate-add-host 2 icmhost1.ibm.com
Host icmhost1.ibm.com has been successfully added for aggregate 2
+----+----------+-------------------+--------------------+------------------------------+
| Id | Name     | Availability Zone | Hosts              | Metadata                     |
+----+----------+-------------------+--------------------+------------------------------+
| 2  | icmhost1 | icmhost1          | 'icmhost1.ibm.com' | 'availability_zone=icmhost1' |
+----+----------+-------------------+--------------------+------------------------------+

[root@icmnode1 ~]# nova-manage service list
Binary           Host             Zone     Status        State Updated_At
nova-conductor   icmnode1.ibm.com internal enabled :-)   2014-11-02 19:18:55
nova-scheduler   icmnode1.ibm.com internal enabled :-)   2014-11-02 19:18:51
nova-consoleauth icmnode1.ibm.com internal enabled :-)   2014-11-02 19:18:53
nova-compute     icmhost1.ibm.com icmhost1 enabled :-)   2014-11-02 19:18:55
nova-compute     icmhost2.ibm.com icmhost2 enabled :-)   2014-11-02 19:18:54

Step-2. Partition the local disk on compute node into smaller chunks

[root@icmhost1 ~]# gdisk /dev/sdb
Command (? for help): n
Partition number (1-128, default 1): 1
First sector (34-6694453214, default = 2048) or {+-}size{KMGTP}:
Last sector (2048-6694453214, default = 6694453214) or {+-}size{KMGTP}: +300G
Current type is 'Linux filesystem'
Hex code or GUID (L to show codes, Enter = 8300):
Changed type of partition to 'Linux filesystem'
Command (? for help): n
Partition number (2-128, default 2):
First sector (34-6694453214, default = 629147648) or {+-}size{KMGTP}:
Last sector (629147648-6694453214, default = 6694453214) or {+-}size{KMGTP}: +300G
Current type is 'Linux filesystem'
Hex code or GUID (L to show codes, Enter = 8300):
Changed type of partition to 'Linux filesystem'

… and so on depending on how many chunks you want.

Command (? for help): p
Disk /dev/sdb: 6694453248 sectors, 3.1 TiB
Logical sector size: 512 bytes
Disk identifier (GUID): A697C0D9-11DB-42D6-B7BC-6B5C65665049
Partition table holds up to 128 entries
First usable sector is 34, last usable sector is 6694453214
Partitions will be aligned on 2048-sector boundaries
Total free space is 402997181 sectors (192.2 GiB)
Number Start (sector) End (sector) Size Code Name
1 2048 629147647 300.0 GiB 8300 Linux filesystem
2 629147648 1258293247 300.0 GiB 8300 Linux filesystem
3 1258293248 1887438847 300.0 GiB 8300 Linux filesystem
4 1887438848 2516584447 300.0 GiB 8300 Linux filesystem
5 2516584448 3145730047 300.0 GiB 8300 Linux filesystem
6 3145730048 3774875647 300.0 GiB 8300 Linux filesystem
7 3774875648 4404021247 300.0 GiB 8300 Linux filesystem
8 4404021248 5033166847 300.0 GiB 8300 Linux filesystem
9 5033166848 5662312447 300.0 GiB 8300 Linux filesystem
10 5662312448 6291458047 300.0 GiB 8300 Linux filesystem
Command (? for help): w
Final checks complete. About to write GPT data. THIS WILL OVERWRITE EXISTING
PARTITIONS!!
Do you want to proceed? (Y/N): y
OK; writing new GUID partition table (GPT) to /dev/sdb.
The operation has completed successfully.

Step-3. Configure Cinder volume server. Ensure cinder availability zone matches with nova availability zone for the node

****</etc/cinder/cinder.conf>****

volume_driver=cinder.volume.drivers.block_device.BlockDeviceDriver
available_devices='/dev/sdb1,/dev/sdb2,/dev/sdb3,/dev/sdb4,/dev/sdb5,/dev/sdb6,/dev/sdb7,/dev/sdb8,/dev/sdb9,/dev/sdb10'
storage_availability_zone=icmhost1 

The storage availability zone and nova availability zone are the same – ‘icmhost1’

[root@icmnode1 ~]# cinder availability-zone-list
+----------+-----------+
| Name     | Status    |
+----------+-----------+
| icmhost1 | available |
| icmhost2 | available |
| nova     | available |
+----------+-----------+

Step-4. Create cinder volumes

[root@icmhost2 ~]# cinder create 300 --display-name icmhost2-vol1 --availability-zone icmhost2
+---------------------+--------------------------------------+
| Property            | Value                                |
+---------------------+--------------------------------------+
| attachments         | []                                   |
| availability_zone   | icmhost2                             |
| bootable            | false                                |
| created_at          | 2014-11-03T03:46:58.052414           |
| display_description | None                                 |
| display_name        | icmhost2-vol1                        |
| encrypted           | False                                |
| id                  | b30a6f9b-a76a-4fb2-b979-82c74352f938 |
| metadata            | {}                                   |
| size                | 300                                  |
| snapshot_id         | None                                 |
| source_volid        | None                                 |
| status              | creating                             |
| volume_type         | None                                 |
+---------------------+--------------------------------------+

[root@icmhost2 ~]# cinder list
+--------------------------------------+-----------+----------------+------+-------------+----------+-------------+
| ID                                   | Status    | Display Name   | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-----------+----------------+------+-------------+----------+-------------+
| 225514af-ee99-4ddc-b691-84ec69b34f0d | available | icmhost2-vol3  | 300  | None        | false    |             |
| 234fa4db-41fa-4c22-8768-cce32af9f70b | available | icmhost1-vol7  | 300  | None        | false    |             |
| 241f2d5d-3783-42f3-8f30-eb8aa5d930d9 | available | icmhost2-vol5  | 300  | None        | false    |             |
| 25dae4c8-e786-4586-ac0e-f33e264143cf | available | icmhost2-vol8  | 300  | None        | false    |             |
| 3292e09a-4c13-4f22-bd19-c96fb8c02d35 | available | icmhost1-vol8  | 300  | None        | false    |             |
| 3a198cb1-e5a1-401f-9f48-7f0f704d1d89 | available | icmhost1-vol9  | 300  | None        | false    |             |
| 675b2c51-e4ed-4b7e-9b58-f122a080998d | available | icmhost1-vol4  | 300  | None        | false    |             |
| 68a19e4e-c152-4b44-9039-e2c2a2eb637a | available | icmhost2-vol7  | 300  | None        | false    |             |
| 6b033e5c-5b43-48cc-9f2c-2f9f9ce923df | available | icmhost2-vol6  | 300  | None        | false    |             |
| 71f3d676-c443-4b7c-8a67-be83dbeed652 | available | icmhost2-vol9  | 300  | None        | false    |             |
| 7e6561fa-bc4f-4378-b630-0ff18a71dbf3 | available | icmhost1-vol3  | 300  | None        | false    |             |
| 7fdeb349-5452-4f63-84f1-57af8e870e16 | available | icmhost1-vol5  | 300  | None        | false    |             |
| 813b62ad-3549-47d4-a7e7-27402c30408f | available | icmhost1-vol10 | 300  | None        | false    |             |
| 91ebd346-312b-42a0-866d-5154a0ac1f29 | available | icmhost2-vol4  | 300  | None        | false    |             |
| b30a6f9b-a76a-4fb2-b979-82c74352f938 | available | icmhost2-vol1  | 300  | None        | false    |             |
| b647aa56-ddf8-42bb-ad66-0f1a62541b3f | available | icmhost2-vol10 | 300  | None        | false    |             |
| cd673599-5d08-4e7a-ae90-c5b7d5e0ac24 | available | icmhost1-vol2  | 300  | None        | false    |             |
| e0df0f74-595b-4c2e-9915-cd2eb0204f85 | available | icmhost1-vol1  | 300  | None        | false    |             |
| e570651c-fc32-4ef4-ab96-82a647197ff2 | available | icmhost2-vol2  | 300  | None        | false    |             |
| eda9cd40-9a8a-47bc-8bbc-7309970c846a | available | icmhost1-vol6  | 300  | None        | false    |             |
+--------------------------------------+-----------+----------------+------+-------------+----------+-------------+

Step-5. Boot a compute instance and attach cinder volume

[root@icmnode1 cinder]# nova boot --flavor 2 --image 2c8bf73f-2fd9-472f-8c0d-6b67a5fffe71 --block-device source=volume,id=e0df0f74-595b-4c2e-9915-cd2eb0204f85,dest=volume,shutdown=preserve --nic net-id=43b6c73b-f977-49ac-ad71-a7af6b2f05e6 --availability-zone icmhost1 hadoop1

step-5

Pradipta Kumar Banerjee

I'm a Cloud and Linux/ OpenSource enthusiast, with 16 years of industry experience at IBM. You can find more details about me here - Linkedin

You may also like...