mongodb cluster

Owned by
Gabe Castillo
Last updated: Apr 12, 2023
2 min read

The goal of this article is to describe the mongodb cluster that has been configured as the backend for peopledb3(peopledbv3). In the future, the goal is to use this for future apps that want/require a mongo backend database.

The following details are accurate as of November 6, 2018

nodes:

  • siomongo1.ucsd.edu
  • siomongo2.ucsd.edu
  • siomongo3.ucsd.edu

This trio of hosts are configured to operate as replica sets with one primary and (2) read-only replica sets. The primary goal of clustering here is for high-availability(HA), as the hosts vote for quorum on which host will serve as the master. Connection strings to this cluster should include all or a majority of nodes so that the HA functionality works as intended.


Troubleshooting:

siomongo1 was down for a few days, and it was too out-of-sync to rejoin the cluster. To fix this, remove all files from the database directory, /data/db and restart the mongod service. This will put the node into STARTUP & STARTUP2 status. Eventually, the node will rebuild, and be promoted to SECONDARY status

This should only be done on a single node, and the replication status should be checked first

Notes from bootstrapping the stage cluster

Check service status via ansible

ansible mongodb_stage -m shell -a 'systemctl status rh-mongodb36-mongod'



Check status of cluster replication

 scl enable rh-mongodb36 "mongo -u root -p'**REDACTED**' --authenticationDatabase admin --eval 'rs.status()'"
ansible mongodb_stage -m shell -a  "scl enable rh-mongodb36 $'mongo -u root -p \'REDACTED1P\' --authenticationDatabase admin --eval \'rs.status()\' '" 

[root@siomongo1-stage ~]# scl enable rh-mongodb36 "mongo -u root -p'REDACTED1P' --quiet --authenticationDatabase admin --eval 'JSON.stringify(rs.status())'"  | jq '.'


Let's find the state of the cluster nodes and which one is primary

scl enable rh-mongodb36 "mongo -u root -p'REDACTED' --quiet --authenticationDatabase admin --eval 'JSON.stringify(rs.status())'" \
| jq -C '.members[] | [.name, .stateStr]'


THIS ONE BELOW WORKS!!!!

ansible siomongo1-stage.ucsd.edu -m shell -a  "scl enable rh-mongodb36 $'mongo -u root -p \'REDACTED1P\' --quiet --authenticationDatabase admin --eval \'JSON.stringify(rs.status())\' ' | jq '.members[] | [.name, .stateStr]'"


There are some in-progress scripts on root@clortho:~/mongo-scripts. The following ENV variable must be set to see output

ANSIBLE_STDOUT_CALLBACK="minimal"