Allocating specific amount of storage to as slave to cluster

Attaching EBS volume to OS

First we have to create additional EBS volume and attach it to our OS. This volume has to be partitioned, formatted and at last has to be mounted to a drive. This drive has to mentioned in hdfs-site.xml file.

Partitioning the new volume

fdisk  /dev/xvdf 

Formatting the partition

mkfs.ext4  /dev/xvdf1

Mounting the partition on directory

mount  /dev/xvdf1  /datanode

Configuring hdfds-site.xml file in slave node of hadoop cluster

<?xml version="1.0"?>

<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<configuration>

<property>

<name>dfs.data.dir</name>

<value>/datanode</value>

</property>

</configuration>

Here datanode is the directory on which the volume is mounted.

Configuring core-site.xml file in slave node of hadoop cluster

<?xml version="1.0"?>

<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<configuration>

<property>

<name>fs.default.name</name>

<value>hdfs://MASTER_IP:9001</value>

</property>

</configuration>

Configuring hdfds-site.xml file in master node of hadoop cluster

<?xml version="1.0"?>

<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<configuration>

<property>

<name>dfs.name.dir</name>

<value>/datanode</value>

</property>

</configuration>

Configuring core-site.xml file in mster node of hadoop cluster

<?xml version="1.0"?>

<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<configuration>

<property>

<name>fs.default.name</name>

<value>hdfs://0.0.0.0:9001</value>

</property>

</configuration>

Output of "hadoop dfsadmin -report" command

[root@ip-172-31-41-114 ~]# hadoop dfsadmin -report

Configured Capacity: 3102490624 (2.89 GB)

Present Capacity: 2915291136 (2.72 GB)

DFS Remaining: 2915246080 (2.72 GB)

DFS Used: 45056 (44 KB)

DFS Used%: 0%

Under replicated blocks: 0

Blocks with corrupt replicas: 0

Missing blocks: 0

-------------------------------------------------

Datanodes available: 1 (1 total, 0 dead)

Name: 18.215.239.86:50010

Decommission Status : Normal

Configured Capacity: 3102490624 (2.89 GB)

DFS Used: 45056 (44 KB)

Non DFS Used: 187199488 (178.53 MB)

DFS Remaining: 2915246080(2.72 GB)

DFS Used%: 0%

DFS Remaining%: 93.96%

Last contact: Wed Oct 21 13:01:04 UTC 2020

In the available data nodes we can observe that the storage contributed by it is only 2.89 GB.

Task 4