三个节点的Hadoop平台安装搭建教程_如何恢复360云

日期:2016-08-24 / 人气: / 来源:网络

之前介绍过在Ubuntu 12.04单机版环境中搭建hadoop详细教程,今天的教程介绍的是3个节点的Hadoop平台建设。

1.   Hadoop节点规划

本次安装规划使用三个节点,每个节点都使用centos系统。

三个节点的hostname分别规划为:centoshadoop1、centoshadoop2、centoshadoop3(此处为本教程参数,可根据实际环境情况修改)

三个节点的ip地址分别规划为:192.168.65.57、192.168.65.58、192.168.65.59(此处为本教程参数,根据实际环境情况修改)

2.   平台搭建使用的软件

下载如下软件

操作系统安装包:Centos6.3_x64

Jdk安装包:jdk-6u37-linux-x64.bin

Hadoop安装包:hadoop-1.1.2.tar.gz

3.   安装centos操作系统(可见centos频道)

安装三个节点的操作系统,安装过程省略。

4.   配置centoshadoop1节点

4.1.  修改节点hostname

[[email protected] ~]# vi /etc/sysconfig/network 

HOSTNAME=centoshadoop1

[[email protected] ~]# vi /etc/hosts

……

192.168.65.57 centoshadoop1

192.168.65.58centoshadoop2

192.168.65.59centoshadoop3

[[email protected] ~]#reboot

4.2.  关闭iptables防火墙

[[email protected]~]#service iptables stop

注意每次操作系统重启后都要操作

4.3.  建立无ssh密码登陆

生成签名文件

[[email protected]~]#cd /root

[[email protected]~]#ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa

[[email protected]~]#cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys

[[email protected]~]#

测试本地SSH无密码登录

[[email protected]~]#sshcentoshadoop1

4.4.  安装jdk

上传jdk-6u37-linux-x64.bin到/root目录下

[[email protected]~]#chmod 777 jdk-6u37-linux-x64.bin

[[email protected]~]#./jdk-6u37-linux-x64.bin

[[email protected]~]#ll

查看生成jdk-6u37-linux-x64目录

4.5.  安装hadoop软件

上传hadoop-1.1.2.tar.gz到/root目录下

[[email protected]~]#tar -zvxf hadoop-1.1.2.tar.gz

[[email protected]~]#ll

查看生成hadoop-1.1.2目录

[[email protected]~]#vi /conf/core-site.xml

<?xml version="1.0"?>

<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!-- Put site-specific property overrides in this file. -->

<configuration>

<property>

<name>fs.default.name</name>

<value>hdfs://192.168.65.57:9000</value>

</property>

<property>

<name>hadoop.tmp.dir</name>

<value>/root/hadoop-1.1.2/tmp</value>

</property>

</configuration>

[[email protected]~]#vi hdfs-site.xml

<?xml version="1.0"?>

<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!-- Put site-specific property overrides in this file. -->

<configuration>

<property>

<name>dfs.replication</name>

<value>1</value>

</property>

</configuration>

[[email protected]~]#vi mapred-site.xml

<?xml version="1.0"?>

<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!-- Put site-specific property overrides in this file. -->

<configuration>

<property>

<name>mapred.job.tracker</name>

<value>192.168.65.57:9001</value>

</property>

</configuration>

4.6.  配置hadoop数据节点

[[email protected]~]#vi /root/hadoop-1.1.2/conf/masters

192.168.65.57 #secondaryNameNode

[[email protected]~]#vi /root/hadoop-1.1.2/conf/slaves

192.168.65.58  #datanode

192.168.65.59  #datanode

4.7.  添加环境变量

[[email protected]~]#cd /root

[[email protected]~]#vi /etc/profile

export JAVA_HOME=/root/jdk1.6.0_37

export JRE_HOME=/root/jdk1.6.0_37/jre

export HADOOP_HOME=/root/hadoop-1.1.2

export HADOOP_HOME_WARN_SUPPRESS=1

export CLASSPATH=.:$JAVA_HOME/lib:$JRE_HOME/lib:$HADOOP_HOME/lib:$CLASSPATH

export PATH=$JAVA_HOME/bin:$JRE_HOME/bin:$HADOOP_HOME/bin:$PATH

[[email protected]~]#vi ~/.bashrc

export JAVA_HOME=/root/jdk1.6.0_37

export JRE_HOME=/root/jdk1.6.0_37/jre

export HADOOP_HOME=/root/hadoop-1.1.2

export HADOOP_DEV_HOME=/root/hadoop-1.1.2

export HADOOP_COMMON_HOME=/root/hadoop-1.1.2

export HADOOP_HDFS_HOME=/root/hadoop-1.1.2

export HADOOP_CONF_DIR=/root/hadoop-1.1.2/conf

[[email protected]~]#vi /root/hadoop-1.1.2/conf/hadoop-env.sh

export JAVA_HOME=/root/jdk1.6.0_37

export HADOOP_HOME_WARN_SUPPRESS=1

export HADOOP_CLASSPATH=/root/hadoop-1.1.2:/root/hadoop-1.1.2/lib

5.   配置centoshadoop2节点

5.1.  修改节点hostname

[[email protected] ~]# vi /etc/sysconfig/network 

HOSTNAME=centoshadoop2

[[email protected] ~]# vi /etc/hosts

……

192.168.65.57 centoshadoop1

192.168.65.58 centoshadoop2

192.168.65.59 centoshadoop3

[[email protected] ~]#reboot

5.2.  关闭iptables防火墙

[[email protected]~]#service iptables stop

注意每次操作系统重启后都要操作

5.3.  建立无ssh密码登陆

将centoshadoop1节点上/root/.ssh目录下的authorized_keys、known_hosts 、id_rsa.pub、id_rsa四个文件拷贝到本机的/root/.ssh目录下

5.4.  安装jdk

上传jdk-6u37-linux-x64.bin到/root目录下

[[email protected]~]#chmod 777 jdk-6u37-linux-x64.bin

[[email protected]~]#./jdk-6u37-linux-x64.bin

[[email protected]~]#ll

查看生成jdk-6u37-linux-x64目录

5.5.  安装hadoop软件

上传hadoop-1.1.2.tar.gz到/root目录下

[[email protected]~]#tar -zvxf hadoop-1.1.2.tar.gz

[[email protected]~]#ll

查看生成hadoop-1.1.2目录

[[email protected]~]#vi /conf/core-site.xml

<?xml version="1.0"?>

<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!-- Put site-specific property overrides in this file. -->

<configuration>

<property>

<name>fs.default.name</name>

<value>hdfs://192.168.65.57:9000</value>

</property>

<property>

<name>hadoop.tmp.dir</name>

<value>/root/hadoop-1.1.2/tmp</value>

</property>

</configuration>

[[email protected]~]#vi hdfs-site.xml

<?xml version="1.0"?>

<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!-- Put site-specific property overrides in this file. -->

<configuration>

<property>

<name>dfs.replication</name>

<value>1</value>

</property>

</configuration>

[[email protected]~]#vi mapred-site.xml

<?xml version="1.0"?>

<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!-- Put site-specific property overrides in this file. -->

<configuration>

<property>

<name>mapred.job.tracker</name>

<value>192.168.65.57:9001</value>

</property>

</configuration>

5.6.  添加环境变量

[[email protected]~]#cd /root

[[email protected]~]#vi /etc/profile

export JAVA_HOME=/root/jdk1.6.0_37

export JRE_HOME=/root/jdk1.6.0_37/jre

export HADOOP_HOME=/root/hadoop-1.1.2

export HADOOP_HOME_WARN_SUPPRESS=1

export CLASSPATH=.:$JAVA_HOME/lib:$JRE_HOME/lib:$HADOOP_HOME/lib:$CLASSPATH

export PATH=$JAVA_HOME/bin:$JRE_HOME/bin:$HADOOP_HOME/bin:$PATH

[[email protected]~]#vi ~/.bashrc

export JAVA_HOME=/root/jdk1.6.0_37

export JRE_HOME=/root/jdk1.6.0_37/jre

export HADOOP_HOME=/root/hadoop-1.1.2

export HADOOP_DEV_HOME=/root/hadoop-1.1.2

export HADOOP_COMMON_HOME=/root/hadoop-1.1.2

export HADOOP_HDFS_HOME=/root/hadoop-1.1.2

export HADOOP_CONF_DIR=/root/hadoop-1.1.2/conf

[[email protected]~]#vi /root/hadoop-1.1.2/conf/hadoop-env.sh

export JAVA_HOME=/root/jdk1.6.0_37

export HADOOP_HOME_WARN_SUPPRESS=1

export HADOOP_CLASSPATH=/root/hadoop-1.1.2:/root/hadoop-1.1.2/lib

6.   配置centoshadoop3节点

6.1.  修改节点hostname

[[email protected] ~]# vi /etc/sysconfig/network 

HOSTNAME=centoshadoop3

[[email protected] ~]# vi /etc/hosts

……

192.168.65.57 centoshadoop1

192.168.65.58 centoshadoop2

192.168.65.59 centoshadoop3

[[email protected] ~]#reboot

关闭iptables防火墙、建立无ssh密码登陆、安装jdk、安装hadoop软件、添加环境变量等操作跟“配置centoshadoop2节点”完全相同

7.   启动运行hadoop平台

7.1.  首先执行格式化

[[email protected]~]#hadoop namenode –format

注意:core-site.xml、hdfs-site.xml、mapred-site.xml等配置文件变化后都需要执行格式化

7.2.  启动hadoop

[[email protected]~]#start-all.sh

7.3.  停止hadoop

[[email protected]~]#stop-all.sh

7.4.  查看hadoop运行状态

查看hadoop启动的进程

[[email protected]~]#jps

查看hadoop集群状态

[[email protected]~]#hadoop dfsadmin  -report

查看hadoop中的目录情况

[[email protected]~]#hadoop fs –ls /

Hadoop2.6.0运行mapreduce之推断(speculative)执行(上)

在mapreduce中设计了Speculator接口作为推断执行的统一规范,DefaultSpeculator作为一种服务在实现了Speculator的同时继承了AbstractService,DefaultSpeculator是mapreduce的默认实现。

Hadoop,mapreduce

作者:管理员




现在致电4006-2991-90 OR 查看更多联系方式 →

Go To Top 回顶部