ansible balancer http https nginx role webserver

This repo is archived. You can view files and clone it, but cannot push or open issues/pull-requests.

15 KiB

Raw Blame History

Deploying a Highly Available production ready OpenShift Deployment

Requires Ansible 1.2
Expects CentOS/RHEL 6 hosts (64 bit)

A Primer into OpenShift Architecture

###OpenShift Overview

OpenShift Origin enables the users to create, deploy and manage applications within the cloud, or in other words it provies a PaaS service (Platform as a service). This aleviates the developers from time consuming processes like machine provisioning and neccesary appliaction deployments. OpenShift provides disk space, CPU resources, memory, network connectivity, and various application like JBoss, python, MySQL etc... So that the developer can spent time on coding, testing his/her new application rather than spending time on figuring out how to get/configure those resources.

###OpenShift Components

Here's a list and a brief overview of the diffrent components used by OpenShift.

Broker: is the single point of contact for all application management activities. It is responsible for managing user logins, DNS, application state, and general orchestration of the application. Customers don't contact the broker directly; instead they use the Web console, CLI tools, or JBoss tools to interact with Broker over a REST based API.
Cartridges: provide the actual functionality necessary to run the user application. Openshift currently supports many language cartridges like JBoss, PHP, Ruby, etc., as well as many DB cartridges such as Postgres, Mysql, Mongo, etc. So incase a user need to deploy or create an php application with mysql as backend, he/she can just ask the broker to deploy a php and an mysql cartridgeon seperate gears.
Gear: Gears provide a resource-constrained container to run one or more cartridges. They limit the amount of RAM and disk space available to a cartridge. For simplicity we can consider this as a seperate vm or linux container for running application for a specific tenant, but in reality they are containers created by selinux contexts and pam namespacing.
Node: are the physical machines where gears are allocated. Gears are generally over-allocated on nodes since not all applications are active at the same time.
BSN (Broker support Nodes): are the nodes which run applications for OpenShift management. for example OpenShift uses mongodb to store various user/app details, it also uses ActiveMQ for communincating with different application nodes via Mcollective. These nodes which hosts this supporing applications are called as broker support nodes.
Districts: are resource pools which can be used to seperate the application nodes based on performance or environments. so for example in a production deployment we can have two districts of nodes one which has resources with lower memory/cpu/disk requirements and another for high performance applications.

An Overview of application creation process in OpenShift.

The above figure depicts an overview of diffrent steps invovled in creating an application in OpenShift. So if a developer wants to create or deploy a JBoss & Myql application the user can request the same from diffrent client tools that are available, the choice can be an Eclipse IDE or command line tool (rhc) or even a web browser.

Once the user has instructed the client it makes a web service request to the Broker, the broker inturn check for available resources in the nodes and checks for gear and cartridge availability and if the resources are available two gears are created and JBoss and Mysql cartridges are deployed on them. The user is then notified and the user can then access the gears via ssh and start deploying the code.

Deployment Diagram of OpenShift via Ansible.

As the above diagram shows the Ansible playbooks deploys a highly available Openshift Paas environment. The deployment has two servers running lvs (piranha) for loadbalancing and ha for the brokers. Two instances of brokers also run for fault tolerence. Ansible also configures a dns server which provides name resolution for all the new apps created in the Openshift environment.

Three bsn(broker support nodes) nodes provide a replicated mongodb deployment and the same nodes three instances of higly available activeMQ cluster. There is no limitation on the number of application nodes you can add, just add the hostnames of the application nodes in the ansible inventory and ansible will configure all of them for you.

Note: As a best practise if you are deploying an actual production environemnt it is recommended to integrate with your internal DNS server for name resolution and use LDAP or integrate with an existing Active Directory for user authentication.

Deployment Steps for OpenShift via Ansible

As a first step probably you may want to setup ansible, Assuming the Ansible host is Rhel variant install the EPEL package.

	yum install http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm

Once the epel repo is installed ansible can be installed via the following command.

	http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm

It is recommended to use seperate machines for the different components of Openshift, but if you are testing it out you could combine the services but atleast four nodes are mandatory as the mongodb and activemq cluster needs atleast three for the cluster to work properly.

As a first step checkout this repository onto you ansible management host and setup the inventory(hosts) as follows.

git checkout https://github.com/ansible/ansible-examples.git

    [dns]
    ec2-54-226-116-175.compute-1.amazonaws.com

    [mongo_servers]
    ec2-54-226-116-175.compute-1.amazonaws.com
    ec2-54-227-131-56.compute-1.amazonaws.com
    ec2-54-227-169-137.compute-1.amazonaws.com

    [mq]
    ec2-54-226-116-175.compute-1.amazonaws.com
    ec2-54-227-131-56.compute-1.amazonaws.com
    ec2-54-227-169-137.compute-1.amazonaws.com

    [broker]
    ec2-54-227-63-48.compute-1.amazonaws.com
    ec2-54-227-171-2.compute-1.amazonaws.com

    [nodes]
    ec2-54-227-146-187.compute-1.amazonaws.com

    [lvs]
    ec2-54-227-176-123.compute-1.amazonaws.com
    ec2-54-227-177-87.compute-1.amazonaws.com

Once the inventroy is setup with hosts in your environment the Openshift stack can be deployed easily by issuing the following command.

    ansible-playbook -i hosts site.yml

Verifying the Installation

Once the stack has been succesfully deployed, we can check if the diffrent components has been deployed correctly.

Mongodb: Login to any bsn node running mongodb and issue the following command and a similar output should be displayed. Which displays that the mongo cluster is up with a primary node and two secondary nodes.

  [root@ip-10-165-33-186 ~]# mongo 127.0.0.1:2700/admin -u admin -p passme
  MongoDB shell version: 2.2.3
  connecting to: 127.0.0.1:2700/admin
  openshift:PRIMARY> rs.status()
  {
      "set" : "openshift",
      "date" : ISODate("2013-07-21T18:56:27Z"),
      "myState" : 1,
      "members" : [
      {
          "_id" : 0,
          "name" : "ip-10-165-33-186:2700",
          "health" : 1,
          "state" : 1,
          "stateStr" : "PRIMARY",
          "uptime" : 804,
          "optime" : {
              "t" : 1374432940000,
              "i" : 1
          },
          "optimeDate" : ISODate("2013-07-21T18:55:40Z"),
          "self" : true
      },
      {
          "_id" : 1,
          "name" : "ec2-54-227-131-56.compute-1.amazonaws.com:2700",
          "health" : 1,
          "state" : 2,
          "stateStr" : "SECONDARY",
          "uptime" : 431,
          "optime" : {
              "t" : 1374432940000,
              "i" : 1
          },
          "optimeDate" : ISODate("2013-07-21T18:55:40Z"),
          "lastHeartbeat" : ISODate("2013-07-21T18:56:26Z"),
          "pingMs" : 0
      },
      {
          "_id" : 2,
          "name" : "ec2-54-227-169-137.compute-1.amazonaws.com:2700",
          "health" : 1,
          "state" : 2,
          "stateStr" : "SECONDARY",
          "uptime" : 423,
          "optime" : {
               "t" : 1374432940000,
                "i" : 1
          },
          "optimeDate" : ISODate("2013-07-21T18:55:40Z"),
          "lastHeartbeat" : ISODate("2013-07-21T18:56:26Z"),
          "pingMs" : 0
      }   
  ],
  "ok" : 1
  }
  openshift:PRIMARY>

ActiveMQ: To verify the cluster status of activeMQ browse to the following url pointing to any one of the mq nodes and provide the credentials as user admin and password as specified in the group_vars/all file. The browser should bring up a page similar to shown below, which shows the other two mq nodes in the cluster to which this node as joined.
```
  http://ec2-54-226-116-175.compute-1.amazonaws.com:8161/admin/network.jsp
```

Broker: To check if the broker node is installed/configured succesfully, issue the following command on any broker node and a similar output should be displayed. Make sure there is a PASS at the end.

  [root@ip-10-118-127-30 ~]# oo-accept-broker -v
  INFO: Broker package is: openshift-origin-broker
  INFO: checking packages
  INFO: checking package ruby
  INFO: checking package rubygem-openshift-origin-common
  INFO: checking package rubygem-openshift-origin-controller
  INFO: checking package openshift-origin-broker
  INFO: checking package ruby193-rubygem-rails
  INFO: checking package ruby193-rubygem-passenger
  INFO: checking package ruby193-rubygems
  INFO: checking ruby requirements
  INFO: checking ruby requirements for openshift-origin-controller
  INFO: checking ruby requirements for config/application
  INFO: checking that selinux modules are loaded
  NOTICE: SELinux is Enforcing
  NOTICE: SELinux is  Enforcing
  INFO: SELinux boolean httpd_unified is enabled
  INFO: SELinux boolean httpd_can_network_connect is enabled
  INFO: SELinux boolean httpd_can_network_relay is enabled
  INFO: SELinux boolean httpd_run_stickshift is enabled
  INFO: SELinux boolean allow_ypbind is enabled
  INFO: checking firewall settings
  INFO: checking mongo datastore configuration
  INFO: Datastore Host: ec2-54-226-116-175.compute-1.amazonaws.com
  INFO: Datastore Port: 2700
  INFO: Datastore User: admin
  INFO: Datastore SSL: false
  INFO: Datastore Password has been set to non-default
  INFO: Datastore DB Name: admin
  INFO: Datastore: mongo db service is remote
  INFO: checking mongo db login access
  INFO: mongo db login successful: ec2-54-226-116-175.compute-1.amazonaws.com:2700/admin --username admin
  INFO: checking services
  INFO: checking cloud user authentication
  INFO: auth plugin = OpenShift::RemoteUserAuthService
  INFO: auth plugin: OpenShift::RemoteUserAuthService
  INFO: checking remote-user auth configuration
  INFO: Auth trusted header: REMOTE_USER
  INFO: Auth passthrough is enabled for OpenShift services
  INFO: Got HTTP 200 response from https://localhost/broker/rest/api
  INFO: Got HTTP 200 response from https://localhost/broker/rest/cartridges
  INFO: Got HTTP 401 response from https://localhost/broker/rest/user
  INFO: Got HTTP 401 response from https://localhost/broker/rest/domains
  INFO: checking dynamic dns plugin
  INFO: dynamic dns plugin = OpenShift::BindPlugin
  INFO: checking bind dns plugin configuration
  INFO: DNS Server: 10.165.33.186
  INFO: DNS Port: 53
  INFO: DNS Zone: example.com
  INFO: DNS Domain Suffix: example.com
  INFO: DNS Update Auth: key
  INFO: DNS Key Name: example.com
  INFO: DNS Key Value: *****
  INFO: adding txt record named testrecord.example.com to server 10.165.33.186: key0
  INFO: txt record successfully added
  INFO: deleteing txt record named testrecord.example.com to server 10.165.33.186: key0
  INFO: txt record successfully deleted
  INFO: checking messaging configuration
  INFO: messaging plugin = OpenShift::MCollectiveApplicationContainerProxy
  PASS

Node: To verify if the node installation/configuration has been successfull, issue the follwoing command and check for a similar output as shown below.

  [root@ip-10-152-154-18 ~]# oo-accept-node -v
  INFO: using default accept-node extensions
  INFO: loading node configuration file /etc/openshift/node.conf
  INFO: loading resource limit file /etc/openshift/resource_limits.conf
  INFO: finding external network device
  INFO: checking node public hostname resolution
  INFO: checking selinux status
  INFO: checking selinux openshift-origin policy
  INFO: checking selinux booleans
  INFO: checking package list
  INFO: checking services
  INFO: checking kernel semaphores >= 512
  INFO: checking cgroups configuration
  INFO: checking cgroups processes
  INFO: checking filesystem quotas
  INFO: checking quota db file selinux label
  INFO: checking 0 user accounts
  INFO: checking application dirs
  INFO: checking system httpd configs
  INFO: checking cartridge repository
  PASS

LVS (LoadBalancer): To check the LoadBalncer Login to the active loadbalancer and issue the follwing command, the output would show the two broker to which the loadbalancer is balancing the traffic.

  [root@ip-10-145-204-43 ~]# ipvsadm
  IP Virtual Server version 1.2.1 (size=4096)
  Prot LocalAddress:Port Scheduler Flags
   -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
  TCP  ip-192-168-1-1.ec2.internal: rr
  -> ec2-54-227-63-48.compute-1.a Route   1      0          0         
  -> ec2-54-227-171-2.compute-1.a Route   2      0          0

Creating an APP in Openshift

To create an App in openshift access the management console via any browser, the VIP specified in group_vars/all can used or ip address of any broker node can used.

	https://<ip-of-broker-or-vip>/

The page would as a login, give it as demo/passme. Once logged in follow the screen instructions to create your first Application. Note: Python2.6 cartridge is by default installed by plabooks, so choose python2.6 as the cartridge.

HA Tests

Few test's that can be performed to test High Availability are:

Shutdown any broker and try to create a new Application
Shutdown anyone mongo/mq node and try to create a new Appliaction.
Shutdown any loadbalaning machine, and the manamgement application should be available via the VirtualIP.

15 KiB Raw Blame History