北屿

北屿小智障

0%

虚拟机CentOS7安装Hadoop

Posted on Edited on In

踩坑实验
主要浪费的都在网络配置，建议重置一下虚拟机网络再配置，不然容易出问题
还有就是Xshell用不了，要更新最新版本..还好我有PuTTY

安利一波刘老师的视频XD
https://www.bilibili.com/video/av69670644?from=search&seid=14704125778684531769

主机名 hadoop0
用户名 hadoop

  • modtify the hostname
    hostnamectl --static set-hostname hadoop

  • change the permission
    vi /etc/sudoers.
    hadoop ALL=(root) NOPASSWD:ALL.

  • config network
    建议第一次先不要动这步看看能不能联网，用dhcp自动获取地址康康先，然后再改静态
    VMware Edit - Virtual Network-VMnet8 - NAT config
    vmnet8
    192.168.96.0
    255.255.255.0
    192.168.96.2
1
2
3
4
5
6
7
8
9
10
11
vi /etc/sysconfig/network-scripts/ifcfg-ens33
00:0C:29:67:3D:DB
BOOTPROTU="static"
IPADDR=192.168.96.128
NETMASK=255.255.255.0
GATEWAY=192.168.96.2
DNS1=8.8.8.8
DNS2=192.168.86.2

ping baidu.com
service network restart
  • install software
1
2
sudo yum install vim
sudo yum install net-tools
1
2
3
4
5
6
7
8
9
sudo vim /etc/hosts
192.168.96.128
ssh-keygen -t rsa
cd ~/.ssh
cat id_rsa.hub > athorized_keys

sudo systemctl stop firewalld.service
sudo systemctl disable firewalld.service
sudo systemctl status firewalld.service

xshell
xftp

1
2
3
4
5
tar -zxvf jdk+tab -C ~/app
tar -zxvf hadoop+tab -C ~/app

cd ~/app/jdk
sudo vim ~/.bash_profile

1
2
3
4
5
6
7
# JAVA ENV
export JAVA_HOME=/home/hadoop/app/jdk1.8.0_211/
PATH=$JAVA_HOME/bin:$PATH

# HADOOP ENV
export HADOOP_HOME=/home/hadoop/app/hadoop-2.7.3
PATH=$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH
1
2
3
4
5
6
source ~/.bash_profile
cd ~/app/hadoop-2.7.4/etc/hadoop
vim hadoop-env.sh
vim core-site.xml

mkdir ~/app/tmp

（*）yarn-site.xml

1
2
3
4
5
6
7
8
9
10
<configuration>
<property>
<name>yarn.resourcemanager.hostname</name>
<value>hadoop00</value>
</property>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
</configuration>

（*）mapred-site.xml
这个文件一开始没有, 但是官方提供了模板, 拷贝即可
cp mapred-site.xml.template mapred-site.xml.

1
2
3
4
5
6
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
</configuration>

（*）core-site.xml 参考本机设置修改主机名, 并设置临时文件夹路径(没有要手动建立)

1
2
3
4
5
6
7
8
9
10
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://hadoop0:9000</value>
</property> 
<property>
<name>hadoop.tmp.dir</name>
<value>/home/hadoop/app/tmp</value>
</property>
</configuration>

（*）hdfs-site.xml

1
2
3
4
5
6
7
8
9
10
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
<property>
<name>dfs.permissions</name>
<value>false</value>
</property>
</configuration>

1
2
3
4
5
6
7
8
hdfs namenode -format
cd ~/app/hadoop-2.7.4/sbin
start-all.sh
jps

sudo chmod 600 ~/.ssh/authorized_keys

stop-all.sh
Related Posts
Have fun.