ZooKeeper is a distributed co-ordination service to manage large set of hosts. Co-ordinating and managing a service in a distributed environment is a complicated process. ZooKeeper solves this issue with its simple architecture and API.
1. What is ZooKeeper
Zookeeper is a Software project of the Apache Software Foundation, Providing an Open Source distributed Configuration service, Synchronisation service, and naming registry for large distributed system. its use to manage and co-ordinate the cluster, Like Hbase and many more… HBase is the cluster base software here zookeeper is use to manage the cluster, its not mandatory that it only works with Hbase but it manage any distributed cluster swell standalone cluster. its default port is 2181
2. Need for ZooKeeper
Controller is a critical part of the system or say heart or brain of the whole System. But its not much related to application Coding, So it doesn’t not add much value to your application but still its a critical part to manage the system.
@ Single Point of Failure
@ Not Reliable
@ Limited Scalability
So We needed Something which Redundant and High Availability. As it difficulty to implement and test. As Admin needed to know all the setup and cluster config and whole setups to recover the failures. even there were problems with the monitoring cluster.
ZOOKEEPER came in to Picture, it took control of each and everything with easy to understand and manage with monitoring the systems.
@ It gives fault tolerance Means its not a single point of failures.
@ IT High Availability means its a continuous communication with help of msg broker or transaction it keeps leader functioning alive.
@ ITs HQuorm Peer Its a forms of multiple no. of zookeeper server. Means one is the leader and if leader goes down other peers in the Quoram continuous to function of Leader. In Simple term its just like Kingdom family if king goes down his son or say designated person becomes king..
@ Can configure multiple master so if active master goes down other passive master takes its place. IT means each master is also fault tolerance.
3. How is information stored in ZooKeeper
IT store Information tree like structure it is a traditional file system like structure its file is called ZNode, its file stores data and same related to other Znode(ITs Slaves machines). Each Znode File can store data upto 1mb. ITs all data is kept in memory (RAM) and backed up to a log for reliability every end of the functioning time given. AS Zookeeper keep all data in memory its really very easy to fetch the data and it can handle the very high loads.
FoR Command Line of Zookeeper
>$ ls /
>$ create /file1/childfile fileinfo
>$ get /file1/childfile
> $ create / file1 / childfile fileinfo
> $ get / file1 / childfile
4. ZooKeeper data model
Hierarchical data model, its same like unix or and Standard file System. Nodes are known as known as Nodes, and are identified by a path. Znode can have data associated with masters and children or slave
Nodes, and are identified by a path. node can have data associated with masters and children or slave, which data size is in KBs. Every Znode are Versioned. data is read/written in its entirety. Znode can be ephemeral nodes Means as soon as the session goes down node will also vanish so its ephemeral node its created as the session is created and active. Watches can be set on Znodes so as any changes are done on Znode it will send a watcher email or alert.
5. ZNodes and types & ZooKeeper data flags
@Ephemeral mode: the Znode will be deleted when the session that created it times-out or it its forcefully deleted. ephemeral means something which is not permanent. When a HMaster server registers to Zookeeper, It creates a Znode under particular given dir like Hbase it will create a HMaster znode, that Znode is ephemeral node if Hmaster goes down this znode will also be deleted so as soon as it place is empty the HBase HA comes to know that HMaster Znode is no longer active then it does all load balancing and Other Hmaster takes its place for functioning.
@Sequence: the path name will have a monotonically increasing counter relative to the parent appended. in simple term it ll increase the no. given to particular Znode in ascending sequence or say no.
7. ZooKeeper watches
It notify the client when something is changed i. e. Configuration data or version. If a region server goes down and watch is on that Znode. if watch want to read or check some changes on that region server watch will send event to RS and client ll be notified that RS is not available or it got deleted. based on that client ll take further action.
@Onetime trigger: Have to be reset by the client if interested in future notifications.
not a full fledge notification system like RabbitMQ or ActiveMQ, but its like clients asking for recommendations. Client should verify the state after receiving the watch event. Ordering Event/ Guarantee: Client ll never see a change for which it has set a watch until it check the watch events.
i. e. if something updates or changes in Znode client ll not see any updates untill client check watch events. that guarantee is given by ZooKeeper.
Default watcher: State change, Connection Loss, Session Expire etc.
8. ZooKeeper use cases
Hadoop Use Case.
HBase Use Case
Configuration Management – Stores bootstrap
9. ZooKeeper nodes in HBase
HBASE Base dir
There are 4 main nodes or say dir when Zookeeper is configured in Hbase. Lets discuss each of it with brief description.
this Znode holds the location of the server hosting the root of all tables of Hbase.
This Znode show the current master server, if there are multiple master server, after current master goes down rest master try to grab this Znode.
This is a directory in which all Znode per Hbase server(region server) participating in the cluster. they all registers themselves as they come online. all the Znode have random number, the regions startcode. so if anything updated or restart it will be changed.
if cluster is asked or requested to shutdown.
10. ZooKeeper stat structure
The Stat Structure for each node in ZooKeeper is made up of the Following Fields this all are the transaction ID..
czxid: This Zxid is changed whenZnode is created.
mxxid: The Zxid is changed when ever Znode is Last modified.
ctime: This Time is Millisecond from epoch when the Znode was Created
mtime: this Time is millisecond from the epoch when the Znode was last modified
version: The number changed with the dat modified of the Znode
cversion: the number chenages when the children modified of the Znode
aversion: the number changes when the ACL(ACCESS CONTROL) is modefied of the Znode
ephemeralOwner: This is Session ID of the Owner of the Znode, IF its in Ephemeral node or it will be Zero
dataLength: length of the data field of Znode
numChildren: The number of children of the Znode
Zookeeper guarantees a total order of messages, and it also guarantees a total order of transaction, Zookeeper exposes the total ordering using a Zookeeper transaction id (ZXID). All Transactions will be stamped with zxid when its Transact and exactly reflects the total ordering.
whenever something gets updated its a transaction no. so Zookeeper ensure or guarantee if update comes from multiple places /node those updates are sequenced using this transaction no. (zxid).
11. ZooKeeper Quorum
HQuorum Peer: All the server will have Zookeeper Server install but only one of them will grab the leader node. Other will be in Que and they ll be called as Followers.
Client can connect to any of the Zookee per server or Followers server but all the updates or changes cordinationof the cluster will happen through the Leader (Server). So if client ask for any request to any server or follower they ll pass it to Leader server and he ll keep updates of it and fullfil the request. Its just like centralised server for update.
If Leader Goes down again the Znode will be so Follower ll try to grab the position and become the Leader.
12. ZooKeeper leader election
There are multiple Algorithm to elect the Zookeeper Leader lets discuss here.
Zookeeper Node which creates the smallest number in the znode becomes the leader and all the other nodes are followers, ZooKeeper ensemble will append the 10-digit sequence number to the path and the znode created will be /app/leader_election/guid_0000000001, /app/leader_election/guid_0000000002, etc. If the leader goes down, then its corresponding znode /app/leader_electionN gets deleted. Followers will get notification by watcher that the Leader is removed or deleted. The next in line follower node will check if there are other znodes with the smallest number. If none, then it will assume the role of the leader. Otherwise, it finds the node which created the znode with the smallest number as leader
2: All Zookeper server store a copy of data in memory and leader is elected at the 1st startup, Followers Service clients all go through the Leader, Updates are sent when a majority of server are persisted to change.
Let Asume a new rs server got registered then if majority of Zookeeper follower server ll response back that update or commit is done or transaction id are updated with them, then Leader consider this change is persisted now and send notification to client about the change.
13. HBase Master
Zookeeper is used to co-ordinate and manage multiple component in cluster to ensure all the other component in the cluster behaves in the consistence manner So that one does not step on each other, So to manage proper order zookeeper is user.
To control access to sequential resource we can user Znode. if want to find fault tolerancy we can use Zookeeper