As part of this section, we will see how to setup Zookeeper using Cloudera distribution. We will also understand important concepts related to Zookeeper.
So far we have installed Cloudera manager and setup CM Services in the cluster. Here we will be adding one component after another to understand the architecture. In the production environment, we can add all the components together or at a time by planning appropriately.
- Learning Process
- Setup Zookeeper
- Review important properties
- Zookeeper Concepts
- Important Zookeeper commands
Let us start all 7 servers which are added to our cluster even though only 4 are needed. If we bring up only 4 nodes, then you will be seeing errors in Cloudera Manager web interface.
We need to follow the standard process while setting up any software.
- Downloading and Installing – already taken care as part of adding hosts to the cluster.
Architecture and Configuration – we need to understand architecture and plan for the configuration.
- Architecture – Zookeeper have all master architecture. Other services (typically masters of those services) in the cluster will act as clients. We will register services such as Namenode of HDFS, Resource Manager of YARN etc with Zookeeper later.
- Configuration Files – /etc/zookeeper/conf
- Daemon Processes – zookeeper
- Service logs – /var/log/zookeeper
- Service Data – based on the property dataDir
We are setting up cluster on 7+1 nodes. We start with 7 nodes and then we will add one more node later.
- Gateway(s) and Management Service
- bigdataserver-2 – Zookeeper
- bigdataserver-3 – Zookeeper
- bigdataserver-4 – Zookeeper
- Slaves or Worker Nodes
- We will create host group zookeeper to run commands using ansible on all nodes where Zookeeper is running.
This is the first CDH service that we are going to configure on the cluster.
- It is open source application which manages distributed applications in simple and reliable manner.
- Once setup is completed one of the servers will be elected as the leader and all other servers are followers. Followers receive message proposals from the leader and agree upon message delivery to the clients.
- It is recommended to install zookeeper server on three or more nodes (using odd number).
- In larger clusters we might have 5 or 7 Zookeeper instances
- Zookeeper cluster is also called as Zookeeper Ensemble. Quorum (2 out of 3 in 3 node cluster) is number of nodes which determine commit. In 5 node cluster, quorum will constitute 3 nodes and in 7 node cluster it is 4.
- We will be using Zookeeper for below services when we configure them as part of the cluster.
- HDFS Namenode HA
- Resource Manager HA
- HBase Masters configuration
- Kafka Brokers
- and more
Following are the steps associated with setting up of Zookeeper.
- Select hosts on which zookeeper to be installed. We will be setting up on bigdataserver-2, bigdataserver-3 and bigdataserver-4. If you are using non GCP platform make sure to choose those servers which are categorized as Masters.
- Select Data and Transaction log directory
- /var/lib/zookeeper is assigned for both Data and Transaction logs. However, they can be customized. We will be using sub directory zookeeper under /data1 . /dev/sdb which is additional disk attached to each of the servers is formatted and mounted on directory /data1 .
- Install and start zookeeper service
Validation and Troubleshooting
Once setup is done it is very important to validate the service and also to understand how to troubleshoot in case of any issues.
- Go to dataDir and check the files such as myId as well as snapshots of the services registered with Zookeeper.
- Go to /var/log/zookeeper on all the hosts where service is configured and check relevant files to make sure that there are no issues related the services configured and started.
Review important properties
Let us review some of the important properties associated with Zookeeper.
- Use Cloudera Manager and go to configs
- Also go to /etc/zookeeper/conf to review the properties
- dataLogDir – for snapshots of transaction logs.
- tickTime – used for heartbeat
- Cloudera Manager manages run time properties in unique way. When ever you restart a service or a component associated with a service, it will create sub directory under /var/run/cloudera-scm-agent/process/ to preserver configuration files with respect to service or component of a service.
- Typically latest properties file with respect to service will be available under /etc, but not in the case of Cloudera distribution.
- It is important to understand this distinction so that even if Cloudera Manager components are down we should be able to troubleshoot certain issues by getting into the appropriate locations of log files, configuration files etc.
Quite often we will have distributed services running on multiple servers (especially masters, for e.g.: Active/Passive Namenode of HDFS). Zookeeper can keep track of the multiple servers associated with a distributed service and ensure that it serves the clients of that service with High Availability and Fault Tolerance. Zookeeper service handles the following functions in a distributed environment.
- Synchronization – Typically used in case client applications should get same data from all the servers on which Zookeeper is running. If one of the follower is behind the leader and client request goes to that follower, the follower will first synchronize and then serve the client.
- Configuration Management – not used by typical Big Data Services. We can manage configuration of any application using Zookeeper.
- Grouping and Naming Registry – Used in Micro Services for Service registry or discovery.
- Leader Election – e.g.: HBase, Leader for Kafka Topic partitions etc
You can run zkCli.sh command to get details about services Zookeeper is managing. Make sure to create softlink for java executable so that zkCli.sh can run.
- ZooKeeper allows distributed processes to coordinate with each other through a shared hierarchal namespace which is organized similarly to a standard file system.
- The namespace consists of data registers – called znodes, in ZooKeeper parlance
- znodes are similar to files and directories in Linux file system.
- Information is typically stored in memory and transaction logs are generated in files, but as it grows snapshot of this information will be persisted in the file system.
- Zookeeper itself is deployed on multiple hosts and manages itself.
- Zookeeper cluster is also called as an ensemble. Typically ensemble will be either 3 nodes or 5 nodes or 7 nodes.
- The servers that make up the ZooKeeper service must all know about each other.
- They maintain an in-memory image of the state, along with transaction logs and snapshots in a persistent store.
- As long as a majority of the servers are available, the ZooKeeper service will be available.
- Clients connect to a single ZooKeeper server.
- The client maintains a TCP connection through which it sends requests, gets responses, gets watch events, and sends heartbeats.
- If the TCP connection to the server breaks, the client will connect to a different server.
- ZooKeeper stamps each update with a number that reflects the order of all ZooKeeper transactions.
- Subsequent operations can use the order to implement higher-level abstractions, such as synchronization primitives.
- It is especially fast in “read-dominant” workloads.
- ZooKeeper applications run on thousands of machines, and it performs best where reads are more common than writes, at ratios of around 10:1.
Data Model and Hierarchical Namespace
Now let us run zkCli.sh command and understand how the services are represented.
- Whenever a service is registered with Zookeeper, you will see an entry under /.
- It is called as znode
- Each znode might have children and they will have a path associated with it similar to the file system hierarchy.
- We will see these details when services such HDFS, YARN, HBase etc are added to the cluster later.
Important Zookeeper Commands
Zookeeper have bunch of commands to get the information about the services managed by Zookeeper as well as Zookeeper itself.
- Make sure to create softlink for java executable so that zkCli.sh can run.
- zkCli.sh or zookeeper-shell.sh – It is primarily to get details about services
- Here are some sample commands. Be careful while running rmr or any del commands.
- zookeeper-shell.sh comes with Zookeeper as part of Kafka. Typically we will have zkCli.sh
- Four letter commands – to get details about Zookeeper cluster
- Sample Command –
echo conf | nc kdbroker01.itversity.com 2181
- Sample Command –
Make sure you have setup Zookeeper successfully and run zkCli.sh command by the end of this section. If there are any issues, please follow up with us before going any further.
Also, if you are planning to take a break before going to next section, make sure to stop services using Cloudera Manager and then all the 7 servers by going to GCP console so that you do not waste your credits or not charged unnecessarily.