Installing CFv2 in Openstack With BOSH

I haven’t found a decent guide on how to deploy Cloud Foundry v2 onto Openstack yet, so I thought I’d write one myself.

This post takes you through the necessary Openstack configurations, installs a micro BOSH using bosh-bootstrap, and then create/upload/deploys the cf BOSH release through it in a total time of around 4 hours.

Comments are welcome if you feel this guide could be improved, or updated to reflect the current state of things. Eventually a guide like this will probably be maintained on cloudfoundry.com, but until then you’ll have to settle with me :)

Openstack: User and Project Setup

I’m assuming you’ve already got a functioning Openstack environment to work with, configured with Quantum networking.

  • First, login to Horizon with an admin-privileged user.
  • Create a new project which has a decent quota of VCPUs, instances, disk and memory (see below for example), otherwise you’ll likely exceed these quotas and hit various errors from Openstack when trying to compile or deploy the cf-release.
  • Now create a user of your own in Admin > Users > Create User, and make this user a member of your new project.

I created a project called ‘cloudfoundry’ which has a quota as follows:

Next, log in to Horizon with your new user, and select your ‘cloudfoundry’ project from the Current Project dropdown.

Openstack: Configure Networking

I have an ‘External’ network, 172.20.217.0/24 with which I have a bank of unallocated floating IPs (in my case accessible on our corporate network).

2 floating IPs will be needed during this installation: one for microBOSH, and another to access Cloud Foundry’s gorouter. Although bosh-bootstrap (micro BOSH installation) does not need a project-allocated floating IP, the cf-release will need an IP allocated first.

Building a Micro BOSH

Dr Nic has made this whole process really easy, by creating bosh-bootstrap.

Please send him flowers and chocolates as what used to be a pretty drawn out process is now amazingly trivial!

bosh-bootstrap requires a flavor to be defined in Openstack called m1.microbosh, and later the cf-release install will require that the m1.small and m1.medium flavors have ephemeral disks. This change can be done using the Nova CLI, or in the Horizon dashboard.

If you want to clear out all your flavors before you start:

nova flavor-list | grep m1 | awk '{ print $4 }' | xargs -n 1 nova flavor-delete

And now add the flavours again, this time containing an ephemeral disk:

nova flavor-create m1.small 2 2048 20 1 --ephemeral 20 --rxtx-factor 1 --is-public true
nova flavor-create m1.medium 3 4096 20 2 --ephemeral 20 --rxtx-factor 1 --is-public true
nova flavor-create m1.microbosh 20 4096 20 2 --ephemeral 20 --rxtx-factor 1 --is-public true

Now we’ve got the Openstack side of things ready, lets get a micro BOSH installed into it.

git clone git@github.com:cloudfoundry-community/bosh-bootstrap.git
cd bosh-bootstrap
bundle install
./bin/bosh-bootstrap deploy

Now input your Openstack API endpoint, access credentials and target project you’d like to deploy a micro BOSH to, or if you’ve been using fogs REPL select from a list of fog endpoints.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
$ ./bin/bosh-bootstrap deploy
1. AWS
2. OpenStack
Choose your infrastructure: 2

Using provider OpenStack

Username: cloudfoundry
Password: ****
Tenant: cloudfoundry
Authorization Token URL: http://172.20.217.57:5000/v2.0
OpenStack Region (optional):

Confirming: Using OpenStack
Acquiring a public IP address... 172.20.217.141

Confirming: Using address 172.20.217.141
Reusing security group ssh
 -> no additional ports opened
Reusing security group bosh_nats_server
 -> no additional ports opened
Reusing security group bosh_agent_https
 -> no additional ports opened
Reusing security group bosh_blobstore
 -> no additional ports opened
Reusing security group bosh_director
 -> no additional ports opened
Reusing security group bosh_registry
 -> no additional ports opened
Destroying key pair firstbosh... done
Acquiring a key pair firstbosh... done

Confirming: Using key pair firstbosh
Determining stemcell image/file to use... curl -O 'http://bosh-jenkins-artifacts.s3.amazonaws.com/last_successful_micro-bosh-stemcell-openstack.tgz'
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  334M  100  334M    0     0   700k      0  0:08:08  0:08:08 --:--:--  702k
/root/.microbosh/deployments/firstbosh/last_successful_micro-bosh-stemcell-openstack.tgz
bundle install
Fetching source index from https://s3.amazonaws.com/bosh-jenkins-gems/
Fetching gem metadata from https://rubygems.org/........
Fetching gem metadata from https://rubygems.org/........
Resolving dependencies...
Using httpclient (2.2.4)
Using yajl-ruby (1.1.0)
Using agent_client (1.5.0.pre.721)
Using json (1.8.0)
Using nokogiri (1.5.6)
Using uuidtools (2.1.4)
Using aws-sdk (1.8.5)
Using bosh_common (1.5.0.pre.721)
Using builder (3.1.4)
Using excon (0.23.0)
Using formatador (0.2.4)
Using mime-types (1.23)
Using net-ssh (2.6.7)
Using net-scp (1.1.1)
Using ruby-hmac (0.4.0)
Using fog (1.11.1)
Using multi_json (1.7.6)
Using log4r (1.1.10)
Using ruby-atmos-pure (1.0.5)
Using blobstore_client (1.5.0.pre.721)
Using highline (1.6.19)
Using readwritesettings (3.0.1)
Using cyoi (0.5.1)
Using redcard (1.1.0)
Using thor (0.18.1)
Using bosh-bootstrap (0.11.5) from source at /apps/microbosh/bosh-bootstrap
Using bosh_cpi (1.5.0.pre.721)
Using sequel (3.43.0)
Using rack (1.5.2)
Using rack-protection (1.5.0)
Using tilt (1.4.1)
Using sinatra (1.4.3)
Using daemons (1.1.9)
Using eventmachine (1.0.3)
Using thin (1.5.1)
Using bosh_registry (1.5.0.pre.721)
Using bosh_aws_cpi (1.5.0.pre.721)
Using json_pure (1.8.0)
Using minitar (0.5.4)
Using net-ssh-gateway (1.1.0)
Using netaddr (1.5.0)
Using progressbar (0.9.2)
Using terminal-table (1.4.5)
Using bosh_cli (1.5.0.pre.721)
Using bosh_openstack_cpi (1.5.0.pre.721)
Using rest-client (1.6.7)
Using ruby_vcloud_sdk (1.5.0.pre.721)
Using bosh_vcloud_cpi (1.5.0.pre.721)
Using membrane (0.0.2)
Using ruby_vim_sdk (1.5.0.pre.721)
Using bosh_vsphere_cpi (1.5.0.pre.721)
Using sqlite3 (1.3.7)
Using bosh_cli_plugin_micro (1.5.0.pre.721)
Using bundler (1.3.4)
Your bundle is complete!
Use `bundle show [gemname]` to see where a bundled gem is installed.
bundle exec bosh micro deployment firstbosh
Deployment set to '/root/.microbosh/deployments/firstbosh/micro_bosh.yml'
bundle exec bosh -n micro deploy /root/.microbosh/deployments/firstbosh/last_successful_micro-bosh-stemcell-openstack.tgz

Verifying stemcell...
File exists and readable                                     OK
Manifest not found in cache, verifying tarball...
Read tarball                                                 OK
Manifest exists                                              OK
Stemcell image file                                          OK
Writing manifest to cache...
Stemcell properties                                          OK

Stemcell info
-------------
Name:    micro-bosh-stemcell
Version: 703


Deploy Micro BOSH
  unpacking stemcell (00:00:02)
  uploading stemcell (00:00:33)
  creating VM from 2ac8358f-5045-458d-b793-dfceb229b6b4 (00:00:26)
  waiting for the agent (00:01:25)
  create disk (00:00:04)
  mount disk (00:00:09)
  stopping agent services (00:00:01)
  applying micro BOSH spec (00:00:16)
  starting agent services (00:00:00)
  waiting for the director (00:00:45)
Done                    11/11 00:03:51
WARNING! Your target has been changed to `https://172.20.217.141:25555'!
Deployment set to '/root/.microbosh/deployments/firstbosh/micro_bosh.yml'
Deployed `firstbosh/micro_bosh.yml' to `https://firstbosh:25555', took 00:03:51 to complete

This will spawn a new Ubuntu VM and turn it into a micro BOSH (basically a BOSH condensed into a single VM).

If you need to SSH into this box, just type ./bin/bosh-bootstrap ssh (you can su to root with password c1oudc0w if you wish)

If you run into RequestEntityTooLarge errors – see the Troubleshooting section at the bottom of this guide for assistance.

Creating a DEV cf-release

Great! Now its time to install Cloud Foundry v2 through that shiny new BOSH you just deployed.

Firstly, grab the cf-release source. Then you need to grab the BOSH CLI gem, update all the submodules, and create a new DEV release as the ones in the releases/ folder are pretty old and don’t work in the bleeding edge any more.

Its recommended to use the release-candidate branch here as its deemed stable, but if you really want to hit the bleeding edge, stay on the master branch!

git clone git@github.com:cloudfoundry/cf-release.git
cd cf-release
git checkout release-candidate
bundle update
./update
bundle exec bosh create release

Grab a coffee, read a novel, or play a game of squash like I did, as this process could take up to 2 hours to complete. Creating a release downloads and prepares about 1.5Gb of source for a large number of components that will later be deployed to some VMs.

The output should end with something like:

Jobs affected by changes in this release
+---------------------------+----------+
| Name                      | Version  |
+---------------------------+----------+
| redis_node_ng             | 14.1-dev |
| uaa                       | 26.1-dev |
| mysql_node_ng             | 14.1-dev |
| syslog_aggregator         | 9.1-dev  |
| vcap_redis                | 6.1-dev  |
| cloud_controller_ng       | 6.1-dev  |
| redis_gateway             | 25.1-dev |
| dashboard                 | 9.1-dev  |
| postgresql_node_ng        | 15.1-dev |
| nats                      | 9.1-dev  |
| ccdb_postgres             | 6.1-dev  |
| serialization_data_server | 9.1-dev  |
| mysql_node                | 31.1-dev |
| postgres                  | 3.1-dev  |
| service_broker            | 8.1-dev  |
| backup_manager            | 16.1-dev |
| router_next               | 9.1-dev  |
| mongodb_node              | 27.1-dev |
| mongodb_node_ng           | 16.1-dev |
| health_manager_next       | 6.1-dev  |
| servicesmgmt              | 0.1-dev  |
| postgresql_node           | 22.1-dev |
| login                     | 10.1-dev |
| redis_node                | 24.1-dev |
| rabbit_gateway            | 20.1-dev |
| mysql_gateway             | 27.1-dev |
| collector                 | 5.1-dev  |
| postgresql_gateway        | 26.1-dev |
| dea_next                  | 11.1-dev |
| gorouter                  | 1.1-dev  |
| oauth2_gateway            | 2.1-dev  |
| mongodb_gateway           | 31.1-dev |
| rabbit_node_ng            | 14.1-dev |
| hbase_slave               | 2.1-dev  |
| opentsdb                  | 15       |
| service_utilities         | 3        |
| hbase_master              | 1        |
| rabbit_node               | 21       |
+---------------------------+----------+

Release version: 131.1-dev
Release manifest: <cf-release path>/dev_releases/cf-release-131.1-dev.yml

Uploading cf-release to BOSH

Target your Micro BOSH, and upload the release to it:

bundle exec bosh target 172.20.217.141
Target set to `firstbosh'
Your username: admin
Enter password: admin
Logged in as `admin'
bundle exec bosh upload release dev_releases/cf-release-131.1-dev.yml

Creating/uploading the release again will recognise that most of the packages haven’t changed, and only upload the ones that have in future versions of this release.

Create a cf-release deploy manifest

Next we need to tailor a deployment manifest that is suitable for our deploy target in Openstack. This manifest will create 3 VMs with a spread of the core Cloud Foundry v2 services amongst them, and give a floating IP to the VM which has gorouter and cloud controller installed.

I’ve lifted a manifest that was started originally by Ferdy in bosh-users. See this thread for more information.

You’ll notice I am using m1.microbosh as the flavor for the compilation VMs. This is because it has a 20Gb ephemeral volume on it to make sure a /dev/vdb disk would be present when spinning up new 10.04 stemcell VMs for the compilation process to complete successfully. If you don’t do this, it will attempt to mount swap space as /dev/vdb which will not be enough space to extract and compile some of the larger packages.

Save the above gist into the cf-release directory, and make any configuration changes necessary.

You will need to change the director_uuid property to be the same as the BOSH you are targetting. To find this out:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
$ bosh status

Config
             /Users/rgrenz/.bosh_config

Director
  Name       firstbosh
  URL        https://172.20.217.141:25555
  Version    1.5.0.pre.703 (release:a8e65cc1 bosh:a8e65cc1)
  User       admin
  UUID       a1ac183d-df5c-485b-ba3a-f4167c064fd4
  CPI        openstack
  dns        enabled (domain_name: microbosh)
  compiled_package_cache disabled

Deployment
  Manifest   <path to cf-release>/cf-openstack.yml

Pick the Director UUID and paste it into the manifest.

Also, make sure you put the correct Quantum network ID from Openstack into the manifest.

You can find this out using quantum CLI command:

1
2
3
4
5
6
7
quantum net-list
+--------------------------------------+-----------+------------------------------------------------------+
| id                                   | name      | subnets                                              |
+--------------------------------------+-----------+------------------------------------------------------+
| 3ad801fb-49a1-42ab-90c5-5070ca0ed97b | cr2       | 50576455-5a14-41f0-9546-f844fca75bb0 10.0.1.0/24     |
| 40ae57b9-fef2-4466-b6ff-72776b36bf2f | External  | 0f6b94da-f68f-4109-b6b2-ecdeb182d919 172.20.217.0/24 |
+--------------------------------------+-----------+------------------------------------------------------+

Take the necessary id field and paste it against the net_id property in the cf-openstack.yml manifest.

Download and upload bosh-stemcell

Before you can deploy the release, you need to download a stemcell suitable for use in Openstack. I’ve been advised that the best source of this is the latest Jenkins artifact that someone is generating:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
wget http://bosh-jenkins-artifacts.s3.amazonaws.com/last_successful_bosh-stemcell-openstack.tgz
bundle exec bosh stemcell upload last_successful_bosh-stemcell-openstack.tgz

Verifying stemcell...
File exists and readable                                     OK
Manifest not found in cache, verifying tarball...
Read tarball                                                 OK
Manifest exists                                              OK
Stemcell image file                                          OK
Writing manifest to cache...
Stemcell properties                                          OK

Stemcell info
-------------
Name:    bosh-stemcell
Version: 703

Checking if stemcell already exists...
No

Uploading stemcell...

last_successf: 100% |ooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo| 245.0MB  34.7MB/s Time: 00:00:07

Director task 4

Update stemcell
  extracting stemcell archive (00:00:04)
  verifying stemcell manifest (00:00:00)
  checking if this stemcell already exists (00:00:00)
  uploading stemcell bosh-stemcell/703 to the cloud (00:00:25)
  save stemcell bosh-stemcell/703 (c47ea09a-8a23-40a4-8d9e-547afda0b405) (00:00:00)
Done                    5/5 00:00:29

Task 4 done
Started       2013-06-11 16:56:32 UTC
Finished  2013-06-11 16:57:01 UTC
Duration  00:00:29

Stemcell uploaded and created

This downloads the latest Openstack bosh-stemcell locally, and then uploads it to your BOSH.

Deploy Cloud Foundry

First, tell the BOSH CLI that you are trying to deploy with your newly created manifest.

bundle exec bosh deployment ./cf-openstack.yml

The next part is pretty time consuming if this is the first time you’ve done it. Uploading the release triggers the individual compilation of every package in the release on upto 6 compilation VMs, then creates the final 3 VMs and installs the relevant packages specified in the manifest.

Your mileage may vary on the number of compilation VMs your Openstack can cope with at once. I periodically noticed a number of RequestEntityTooLarge errors during compilation or deploy process. See Troubleshooting below on how to handle this if you get them frequently.

If you hit any failures during deploy, BOSH CLI’s stack trace and Nova’s API logs are usually very helpful for debugging. See troubleshooting section below for more advice.

Now, lets get some magic in motion:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
bundle exec bosh deploy

Getting deployment properties from director...
Compiling deployment manifest...
Please review all changes carefully
Deploying `cf-openstack.yml' to `firstbosh' (type 'yes' to continue): yes

Director task 16

Preparing deployment
  binding deployment (00:00:00)
  binding releases (00:00:00)
  binding existing deployment (00:00:00)
  binding resource pools (00:00:00)
  binding stemcells (00:00:00)
  binding templates (00:00:00)
  binding properties (00:00:00)
  binding unallocated VMs (00:00:00)
  binding instance networks (00:00:00)
Done                    9/9 00:00:00

Preparing package compilation

Preparing DNS
  binding DNS (00:00:00)
Done                    1/1 00:00:00

Creating bound missing VMs
  small/1 (00:00:45)
  small/0 (00:00:52)
  medium/0 (00:00:53)
Done                    3/3 00:00:53

Binding instance VMs
  common1/0 (00:00:01)
  common3/0 (00:00:01)
  common2/0 (00:00:01)
Done                    3/3 00:00:01

Preparing configuration
  binding configuration (00:00:01)
Done                    1/1 00:00:01

Updating job common1
  common1/0 (canary) (00:02:00)
Done                    1/1 00:02:00

Updating job common2
  common2/0 (canary) (00:00:50)
Done                    1/1 00:00:50

Updating job common3
  common3/0 (canary) (00:01:52)
Done                    1/1 00:01:52

Task 16 done
Started       2013-06-11 17:27:06 UTC
Finished  2013-06-11 17:32:43 UTC
Duration  00:05:37

Deployed `cf-openstack.yml' to `firstbosh'

We did it!!

Use Cloud Foundry

Now that you’ve finished your Cloud Foundry deployment – lets use it!

Install the cf gem :

1
2
3
gem install cf
cf target api.mydomain.com
cf login

As you’ll see from the cf-manifest.yml, the user/pass to login is admin/c1oudc0w (see the scim: section in the manifest – thanks Nic).

Troubleshooting

If you see any RequestEntityTooLarge errors at any stage that involves interacting with Openstack, eg:

1
2
/usr/share/ruby-rvm/gems/ruby-1.9.3-p194/gems/excon-0.23.0/lib/excon/middlewares/expects.rb:10:in `response_call': Expected(200) <=> Actual(413 Request Entity Too Large) (Excon::Errors::RequestEntityTooLarge)
  response => #<Excon::Response:0x00000002c8b250 @data={:body=>"{\"overLimit\": {\"message\": \"This request was rate-limited.\", \"code\": 413, \"retryAfter\": \"5\", \"details\": \"Only 10 POST request(s) can be made to * every minute.\"}}", :headers=>{"Retry-After"=>"5", "Content-Length"=>"161", "Content-Type"=>"application/json; charset=UTF-8", "Date"=>"Tue, 11 Jun 2013 13:08:22 GMT"}, :status=>413, :remote_ip=>"172.20.217.57"}, @body="{\"overLimit\": {\"message\": \"This request was rate-limited.\", \"code\": 413, \"retryAfter\": \"5\", \"details\": \"Only 10 POST request(s) can be made to * every minute.\"}}", @headers={"Retry-After"=>"5", "Content-Length"=>"161", "Content-Type"=>"application/json; charset=UTF-8", "Date"=>"Tue, 11 Jun 2013 13:08:22 GMT"}, @status=413, @remote_ip="172.20.217.57">

This is more than likely to do with Openstack’s small default API request limit (10 reqs/min), and can be fixed by editing /etc/nova/api-paste.conf and adjusting/adding the filter:ratelimit section as follows:

1
2
3
[filter:ratelimit]
paste.filter_factory = nova.api.openstack.compute.limits:RateLimitingMiddleware.factory
limits =(POST, "*", .*, 50, MINUTE);(POST, "*/servers", ^/servers, 50, DAY);(PUT, "*", .*, 50, MINUTE);(GET, "*changes-since*", .*changes-since.*, 30, MINUTE);(DELETE, "*", .*, 100, MINUTE)

These commands may also be useful when troubleshooting a failing release:

Command What happens?
bundle exec bosh delete release this will remove the release from your BOSH (do this carefully as you will have to go thru the release upload and compilation processes all over again which could take a while)
bundle exec bosh delete deployment this will blow away all aspects of the deployment (including any running VMs) so you can start afresh
bundle exec bosh cck this carries out a cloud check and will offer to delete any orphaned VMs and detect any other inconsistencies in a deployment

Comments

Real Time Analytics