diff --git a/mongodb/README.md b/mongodb/README.md index 2ab8861..e373069 100644 --- a/mongodb/README.md +++ b/mongodb/README.md @@ -5,22 +5,32 @@ In this example we demonstrate how we can orchestrate the deployment of a produc 1) Deploying a N node MongoDB cluster, which has N shards and N replication nodes. 2) Scale out capability. Expand the Cluster by adding nodes to the cluster. +3) Security, All the mongodb process are secured using the best practices. ###Deployment Architecture. To better explain the deployment architecture let's take an example where we are deploying a 3 node MongoDB cluster ( Minimum recommended by MongoDB). + The way Ansible configures the three nodes is as follows: 1) Install the mongodb software on all nodes. + 2) Creates 3 replication sets, with one primary on each node and the rest two acting as secondaries. -3) Configures MongodDB configuration servers as listed in the inventory section[mongocservers]. Recommended number is 3, so it can be the same three servers as the datanodes. + +3) Configures MongodDB configuration DB servers as listed in the inventory section[mongocservers]. Recommended number is 3, so it can be the same three servers as the datanodes. + 4) Configures a Mongos server as listed in the inventory file [mongosservers]. -5) Adds 3 Shards each belonging to individual relocation sets. + +5) Adds 3 Shards each belonging to individual replication sets. + +6) All the processes, mongod,mogos are secured using the keyfiles. Once the cluster is deployed, if we want to scale the cluster, Ansible configures it as follows: 1) Install the MongoDB application on the new node. + 2) Configure the replication set with primary as the new node and the secondaries as listed in the inventory file [replicationservers]. ( don't forget to add the new node also in the replicationservers section] + 3) Adds a new shard to the mongos service pointing to the new replication set. ###The following example deploys a three node MongoDB Cluster @@ -31,19 +41,19 @@ The inventory file looks as follows: [mongoservers] mongo1 mongo2 - mongo3 + mongo3 #The list of servers where replication should happen, by default include all servers [replicationservers] mongo1 mongo2 - mongo3 + mongo3 #The list of mongodb configuration servers, make sure it is 1 or 3 [mongocservers] mongo1 mongo2 - mongo3 + mongo3 #The list of servers where mongos servers would run. [mongosservers] @@ -53,7 +63,7 @@ Build the site with the following command: ansible-playbook -i hosts site.yml -##Verification +###Verifying the deployed MongoDB Cluster Once completed we can check replication set availibitly by connecting to individual primary replication set nodes, 'mongo --host --port and issue the command to query the status of replication set, we should get a similar output. @@ -105,10 +115,74 @@ and issue the following command to get the status of the Shards. databases: { "_id" : "admin", "partitioned" : false, "primary" : "config" } +We can also make sure the Sharding works by creating a database and collection and populate it with documents and check if the chunks of the collection are balanced equally across nodes. + +The above mentioned steps can be tested as follows: + +1) Once the Sharded cluster is ready, create a new database and an admin user for the database. This is done from the mongos machine. + + /usr/bin/mongo localhost:8888/admin -u admin -p 123456 + mongos> use test + switched to db test + + mongos> db.addUser('admin','123456') + { + "user" : "admin", + "readOnly" : false, + "pwd" : "95ec4261124ba5951720b199908d892b", + "_id" : ObjectId("51519f349cd3a93ca7e17909") + } +2) Once the DB and the user is created, create a collection and poplulate documents, This deployment add a script to the /tmp location of the mongos server which adds a new collection and 100,000 documents. + + $/usr/bin/mongo localhost:8888/test -u admin -p 123456 /tmp/testsharding.js +3) After the document's are populated, we have to enable sharding on the database and the collection. which can be done as follows: + + $/usr/bin/mongo localhost:8888/admin -u admin -p 123456 + mongos> db.runCommand( { enableSharding : "test" } ) + { "ok" : 1 } + mongos> db.runCommand( { shardCollection : "test.test_collection", key : {"number":1} }) + { "collectionsharded" : "test.test_collection", "ok" : 1 } + mongos> sh.status() + --- Sharding Status --- + sharding version: { "_id" : 1, "version" : 3 } + shards: + { "_id" : "bensible", "host" : "bensible/bensible:20103,web2:20103,web3:20103" } + { "_id" : "web2", "host" : "web2/bensible:20102,web2:20102,web3:20102" } + { "_id" : "web3", "host" : "web3/bensible:20101,web2:20101,web3:20101" } + databases: + { "_id" : "admin", "partitioned" : false, "primary" : "config" } + { "_id" : "test", "partitioned" : true, "primary" : "bensible" } + + test.test_collection chunks: + + web2 1 + bensible 19 +4) In the above example we can see the chunks being balanced across nodes. After a few minutes if we excute the same command 'sh.status()' +we will see the below output, which shows all the chunks being balanced across the three nodes. + mongos> sh.status() + --- Sharding Status --- + sharding version: { "_id" : 1, "version" : 3 } + shards: + { "_id" : "bensible", "host" : "bensible/bensible:20103,web2:20103,web3:20103" } + { "_id" : "web2", "host" : "web2/bensible:20105,web2:20105,web3:20105" } + { "_id" : "web3", "host" : "web3/bensible:20102,web2:20102,web3:20102" } + databases: + { "_id" : "admin", "partitioned" : false, "primary" : "config" } + { "_id" : "test", "partitioned" : true, "primary" : "web3" } + + test.test_collection chunks: + + bensible 7 + web2 6 + web3 7 + + + + ### Adding a new node to the Cluster To add a new node to the configured MongoDb Cluster, setup the inventory file as follows: @@ -117,20 +191,20 @@ To add a new node to the configured MongoDb Cluster, setup the inventory file as [mongoservers] mongo1 mongo2 - mongo3 + mongo3 mongo4 #The list of servers where replication should happen, by default include all servers [replicationservers] mongo4 mongo1 - mongo2 + mongo2 #The list of mongodb configuration servers, make sure it is 1 or 3 [mongocservers] mongo1 mongo2 - mongo3 + mongo3 #The list of servers where mongos servers would run. [mongosservers] @@ -140,5 +214,28 @@ Make sure you have the new node added in the replicationservers section and exec ansible-playbook -i hosts playbooks/addnode.yml -e servername=mongo4 -Verification can be done using the same steps mentioned above. +###Verification. + +The verification of the newly added node can be as easy checking the sharding status and see the chunks being rebalanced to the newly added node. + + $/usr/bin/mongo localhost:8888/admin -u admin -p 123456 + mongos> sh.status() + --- Sharding Status --- + sharding version: { "_id" : 1, "version" : 3 } + shards: + { "_id" : "bensible", "host" : "bensible/bensible:20103,web2:20103,web3:20103" } + { "_id" : "web2", "host" : "web2/bensible:20105,web2:20105,web3:20105" } + { "_id" : "web3", "host" : "web3/bensible:20102,web2:20102,web3:20102" } + { "_id" : "web4", "host" : "web4/bensible:20101,web3:20101,web4:20101" } + databases: + { "_id" : "admin", "partitioned" : false, "primary" : "config" } + { "_id" : "test", "partitioned" : true, "primary" : "bensible" } + + test.test_collection chunks: + + web4 3 + web3 6 + web2 6 + bensible 5 + diff --git a/mongodb/group_vars/all b/mongodb/group_vars/all index ae7819e..e0a6caf 100644 --- a/mongodb/group_vars/all +++ b/mongodb/group_vars/all @@ -10,10 +10,13 @@ mongos_port: 8888 mongoc_port: 7777 #The port prefix for mongod servers, the latter part is appended the playbook (the last octect of the ipaddress) -mongodb_port_prefix: 201 +mongodb_port_prefix: 20 #The directory prefix where the database files would be stored mongodb_datadir_prefix: /data/ #The interface where the mongodb process should listen on. -iface: eth0 +iface: eth1 + +#The password for admin user +mongo_admin_pass: 123456 diff --git a/mongodb/hosts b/mongodb/hosts index 3163e3f..bedc3cb 100644 --- a/mongodb/hosts +++ b/mongodb/hosts @@ -1,17 +1,23 @@ #The site wide list of mongodb servers [mongoservers] -web2 -web3 +mongo1 +mongo2 +mongo3 #The list of servers where replication should happen, by default include all servers [replicationservers] -web2 -web3 +mongo1 +mongo2 +mongo3 + #The list of mongodb configuration servers, make sure it is 1 or 3 [mongocservers] -web3 +mongo1 +mongo2 +mongo3 + #The list of servers where mongos servers would run. [mongosservers] -web3 +mongo1 diff --git a/mongodb/playbooks/addnode.yml b/mongodb/playbooks/addnode.yml index 213e1ef..5f376fc 100644 --- a/mongodb/playbooks/addnode.yml +++ b/mongodb/playbooks/addnode.yml @@ -1,11 +1,11 @@ --- #This playbook is used to add a new node the mongodb cluster -- hosts: mongoservers -- hosts: replicationservers -- hosts: mongosservers +- hosts: all + tasks: + - include: ../roles/common/tasks/main.yml + - hosts: ${servername} tasks: - - include: ../roles/common/tasks/main.yml - include: ../roles/mongod/tasks/main.yml - include: ../roles/mongod/tasks/addshard.yml diff --git a/mongodb/roles/common/tasks/main.yml b/mongodb/roles/common/tasks/main.yml index 757354d..7967d0f 100644 --- a/mongodb/roles/common/tasks/main.yml +++ b/mongodb/roles/common/tasks/main.yml @@ -12,3 +12,4 @@ with_items: - mongo-10gen - mongo-10gen-server + - bc diff --git a/mongodb/roles/mongoc/files/secret b/mongodb/roles/mongoc/files/secret new file mode 100644 index 0000000..4d77cbc --- /dev/null +++ b/mongodb/roles/mongoc/files/secret @@ -0,0 +1,3 @@ +qGO6OYb64Uth9p9Tm8s9kqarydmAg1AUdgVz+ecjinaLZ1SlWxXMY1ug8AO7C/Vu +D8kA3+rE37Gv1GuZyPYi87NSfDhKXo4nJWxI00BxTBppmv2PTzbi7xLCx1+8A1uQ +4XU0HA diff --git a/mongodb/roles/mongoc/tasks/main.yml b/mongodb/roles/mongoc/tasks/main.yml index c343724..1478a1d 100644 --- a/mongodb/roles/mongoc/tasks/main.yml +++ b/mongodb/roles/mongoc/tasks/main.yml @@ -16,6 +16,17 @@ - name: Create the mongo configuration server file template: src=../roles/mongoc/templates/mongoc.conf.j2 dest=/etc/mongoc.conf +- name: Create the script to add the admin user + template: src=../roles/mongoc/templates/adduser.j2 dest=/tmp/adduser.js + +- name: Copy the keyfile for authentication + copy: src=../roles/mongod/files/secret dest=${mongodb_datadir_prefix}/secret owner=mongod group=mongod mode=0400 + - name: Start the mongo configuration server service command: creates=/var/lock/subsys/mongoc /etc/init.d/mongoc start +- name: pause + pause: seconds=20 + +- name: add the admin user + shell: /usr/bin/mongo localhost:7777/admin /tmp/adduser.js diff --git a/mongodb/roles/mongoc/templates/adduser.j2 b/mongodb/roles/mongoc/templates/adduser.j2 new file mode 100644 index 0000000..a1fa6ae --- /dev/null +++ b/mongodb/roles/mongoc/templates/adduser.j2 @@ -0,0 +1 @@ +db.addUser('admin','{{ mongo_admin_pass }}') diff --git a/mongodb/roles/mongoc/templates/mongoc.conf.j2 b/mongodb/roles/mongoc/templates/mongoc.conf.j2 index 59c29ba..705f0c0 100644 --- a/mongodb/roles/mongoc/templates/mongoc.conf.j2 +++ b/mongodb/roles/mongoc/templates/mongoc.conf.j2 @@ -10,7 +10,7 @@ fork = true port = {{ mongoc_port }} dbpath={{ mongodb_datadir_prefix }}configdb - +keyFile={{ mongodb_datadir_prefix }}secret # location of pidfile pidfilepath = /var/run/mongoc.pid diff --git a/mongodb/roles/mongod/files/secret b/mongodb/roles/mongod/files/secret new file mode 100644 index 0000000..4d77cbc --- /dev/null +++ b/mongodb/roles/mongod/files/secret @@ -0,0 +1,3 @@ +qGO6OYb64Uth9p9Tm8s9kqarydmAg1AUdgVz+ecjinaLZ1SlWxXMY1ug8AO7C/Vu +D8kA3+rE37Gv1GuZyPYi87NSfDhKXo4nJWxI00BxTBppmv2PTzbi7xLCx1+8A1uQ +4XU0HA diff --git a/mongodb/roles/mongod/tasks/addshard.yml b/mongodb/roles/mongod/tasks/addshard.yml index 3552ac3..32d146b 100644 --- a/mongodb/roles/mongod/tasks/addshard.yml +++ b/mongodb/roles/mongod/tasks/addshard.yml @@ -12,7 +12,7 @@ with_items: ${groups.mongosservers} - name: Add the shard to the mongos - shell: /usr/bin/mongo --port ${mongos_port} /tmp/shard_init_${inventory_hostname}.js + shell: /usr/bin/mongo localhost:8888/admin -u admin -p ${mongo_admin_pass} /tmp/shard_init_${inventory_hostname}.js delegate_to: $item with_items: ${groups.mongosservers} diff --git a/mongodb/roles/mongod/tasks/main.yml b/mongodb/roles/mongod/tasks/main.yml index 619d80f..cd37fe2 100644 --- a/mongodb/roles/mongod/tasks/main.yml +++ b/mongodb/roles/mongod/tasks/main.yml @@ -21,7 +21,7 @@ with_items: ${groups.replicationservers} - name: Add the iptable rule to allow traffice dynamically - shell: iptables -I INPUT 2 -p tcp --dport ${mongodb_port_prefix}${result.stdout} -j ACCEPT + shell: sleep `echo $RANDOM/1000 | bc`; iptables -I INPUT 1 -p tcp --dport ${mongodb_port_prefix}${result.stdout} -j ACCEPT delegate_to: $item with_items: ${groups.replicationservers} @@ -30,6 +30,10 @@ delegate_to: $item with_items: ${groups.replicationservers} +- name: Copy the keyfile for authentication + copy: src=../roles/mongod/files/secret dest=${mongodb_datadir_prefix}/secret owner=mongod group=mongod mode=0400 + + - name: Start the mongodb service command: creates=/var/lock/subsys/mongod-${inventory_hostname} /etc/init.d/mongod-${inventory_hostname} start delegate_to: $item @@ -38,6 +42,9 @@ - name: Create the file to initialize the mongod replica set template: src=../roles/mongod/templates/repset_init.j2 dest=/tmp/repset_init.js +- name: Pause for a while + pause: seconds=20 + - name: Initialize the replication set shell: /usr/bin/mongo --port "$mongodb_port_prefix${result.stdout}" /tmp/repset_init.js diff --git a/mongodb/roles/mongod/templates/mongod.conf.j2 b/mongodb/roles/mongod/templates/mongod.conf.j2 index caf0de5..a8e9e17 100644 --- a/mongodb/roles/mongod/templates/mongod.conf.j2 +++ b/mongodb/roles/mongod/templates/mongod.conf.j2 @@ -12,6 +12,7 @@ fork = true port = {{ mongodb_port_prefix }}{{ result.stdout }} dbpath={{ mongodb_datadir_prefix }}mongo-{{ inventory_hostname }} +keyFile={{ mongodb_datadir_prefix }}/secret # location of pidfile pidfilepath = /var/run/mongod.pid diff --git a/mongodb/roles/mongos/files/secret b/mongodb/roles/mongos/files/secret new file mode 100644 index 0000000..4d77cbc --- /dev/null +++ b/mongodb/roles/mongos/files/secret @@ -0,0 +1,3 @@ +qGO6OYb64Uth9p9Tm8s9kqarydmAg1AUdgVz+ecjinaLZ1SlWxXMY1ug8AO7C/Vu +D8kA3+rE37Gv1GuZyPYi87NSfDhKXo4nJWxI00BxTBppmv2PTzbi7xLCx1+8A1uQ +4XU0HA diff --git a/mongodb/roles/mongos/tasks/main.yml b/mongodb/roles/mongos/tasks/main.yml index 7e020da..bb1acb7 100644 --- a/mongodb/roles/mongos/tasks/main.yml +++ b/mongodb/roles/mongos/tasks/main.yml @@ -13,6 +13,13 @@ - name: Create the mongos configuration file template: src=../roles/mongos/templates/mongos.conf.j2 dest=/etc/mongos.conf +- name: Copy the keyfile for authentication + copy: src=../roles/mongod/files/secret dest=${mongodb_datadir_prefix}/secret owner=mongod group=mongod mode=0400 + - name: Start the mongos service command: creates=/var/lock/subsys/mongos /etc/init.d/mongos start +- name: pause + pause: seconds=20 +- name: copy the file for shard test + template: src=../roles/mongos/templates/testsharding.j2 dest=/tmp/testsharding.js diff --git a/mongodb/roles/mongos/templates/mongos.conf.j2 b/mongodb/roles/mongos/templates/mongos.conf.j2 index a619110..f7ff705 100644 --- a/mongodb/roles/mongos/templates/mongos.conf.j2 +++ b/mongodb/roles/mongos/templates/mongos.conf.j2 @@ -19,4 +19,5 @@ port = {{ mongos_port }} # location of pidfile pidfilepath = /var/run/mongodb/mongos.pid +keyFile={{ mongodb_datadir_prefix }}/secret chunkSize={{ mongos_chunk_size }} diff --git a/mongodb/roles/mongos/templates/testsharding.j2 b/mongodb/roles/mongos/templates/testsharding.j2 new file mode 100644 index 0000000..87a4451 --- /dev/null +++ b/mongodb/roles/mongos/templates/testsharding.j2 @@ -0,0 +1,12 @@ +people = ["Marc", "Bill", "George", "Eliot", "Matt", "Trey", "Tracy", "Greg", "Steve", "Kristina", "Katie", "Jeff"]; + +for(var i=0; i<100000; i++){ + name = people[Math.floor(Math.random()*people.length)]; + user_id = i; + boolean = [true, false][Math.floor(Math.random()*2)]; + added_at = new Date(); + number = Math.floor(Math.random()*10001); + db.test_collection.save({"name":name, "user_id":user_id, "boolean": boolean, "added_at":added_at, "number":number }); + } +db.test_collection.ensureIndex({number:1}) +