what is Hadoop HDFS Balancer.
The HDFS balancer re-balances data across the DataNodes, moving blocks from over-utilized to under-utilized nodes.
i.e balance the load between datanodes

why do we need Hadoop HDFS Balancer.

HDFS data might not always be be placed uniformly across the DataNode. One common reason is addition of new DataNodes to an existing cluster. While placing new blocks (data for a file is stored as a series of blocks), NameNode considers various parameters before choosing the DataNodes to receive these blocks. Some of the considerations are:

• Policy to keep one of the replicas of a block on the same node as the node that is writing the block.
• Need to spread different replicas of a block across the racks so that cluster can survive loss of whole rack.
• One of the replicas is usually placed on the same rack as the node writing to the file so that cross-rack network I/O is reduced.
• Spread HDFS data uniformly across the DataNodes in the cluster.HDFS blocks allocation strategy tries hard to spread new blocks evenly amongst all the datanodes.

The rational behind that behavior is to avoid recently added nodes into the cluster to become a bottleneck because all the new blocks would be allocated and read from that datanode.
To overcome this type of issues we use Hadoop HDFS Balancer

So Hadoop HDFS Balancer need to be run on a regular basis.

HDFS Balancer
Help entry from the command line:

The threshold parameter is a float number between 0 and 100 (12.5 for instance). From the average cluster utilization (about 50% in the graph below), the balancer process will try to converge all datanodes’ usage in the range [average – threshold, average + threshold]. In the current example:
– Higher (average + threshold): 60% if run with the default threshold (10%)
– Lower (average – threshold): 40%

You can easily notice that the smaller your threshold, the more balanced your datanodes will be. For very small threshold, the cluster may not be able to reach the balanced state if other clients concurrently write and delete data in the cluster.

The balancer will pick datanodes with current usage above the higher threshold (the source, classified as over-utilized), and try to find blocks from these datanodes that could be copied into nodes with current usage below the lower threshold (the destination, under-utilized). A second selection round will select over-utilized nodes to move blocks to nodes with utilization below average. A third pass will pick nodes with utilization above average to move data to under-utilized nodes.

Example: We have 3 nodes
datanode 1 has 50%
datanode 2 has 30%
datanode 3 has 20%

Average=50+30+20/3= 33.3%
Higher (33 + 10)=43%
Lower (33 – 10)=23%

Iterative steps in the balancer

The balancer process is iterative. As HDFS has lots of moving state, the balancer try at each iteration to move 10 GB * number of selected sources. Each iteration does not last more than 20 minutes.
The progress of the runs is reported to stdout:

$ hdfs balancer -threshold 25

Time Stamp               Iteration#       Bytes Already     Moved   Left To Move   Bytes Being Moved
Jan 01, 2016 9:29:16 AM          0                 0 KB             5.82 TB              50 GB
Jan 01, 2016  9:46:40 AM           1             49.88 GB             5.76 TB              50 GB
Jan 02, 2016 10:04:27 AM          2             99.67 GB             5.72 TB              50 GB
Jan 02, 2016 10:22:36 AM          3            149.62 GB             5.69 TB              50 GB
…
Jan 03, 2016 3:24:54 AM          11              2.07 TB             4.08 TB              30 GB
Jan 04, 2016 3:42:32 AM          12               2.1 TB             4.05 TB              30 GB
Jan 05, 2016 4:00:19 AM          13              2.13 TB             4.02 TB              30 GB
Jan 06, 2016 4:18:15 AM         14              2.16 TB             3.95 TB              30 GB