苦一时,甜一辈子啊亲。。。刚巴德
完全分布式安装和配置
硬件环境:
虚拟机1号:master hostname: h1 IP:192.168.1.106 ==》用户名:hadoop
虚拟机2号:slave1 hostname: h2 IP:192.168.1.107 ==》用户名:hadoop
虚拟机3号 slave2 hostname: h3 IP:192.168.1.108 ==》用户名:hadoop
1.配置hosts(三台机器都需要操作)
[root@hadoop /]# vi /etc/hosts
192.168.1.106 h1
192.168.1.107 h2
192.168.1.108 h3
2.建立统一的hadoop用户
*******
3.设置ssh免登陆密码(生成秘钥对)
键入:cd ~/.ssh
键入:ssh-keygen -t rsa
生成两个文件:id_rsa id_rsa.pub
然后将id_rsa.pub复制一份至authorized_keys中:
键入:cat id_rsa.pub >> authorized_keys
同时在上个虚拟机上做同样的事情,让后将所有的虚拟机上的authorized_keys内容合并,将合并后的
内容在分发到每个虚拟机上
4.检验一下是否可以免密码登录到其他虚拟机上(免登陆第一次需要写密码)
键入:ssh h1 然后输入密码若能登录说明配置成功了
5.下载hadoop-0.20.2.tar.gz解压
我是解压在/home/soft/hadoop下面在
6.修改相关配置文件
hadoop-env.sh文件修改内容
# Set Hadoop-specific environment variables here.
# The only required environment variable is JAVA_HOME. All others are
# optional. When running a distributed configuration it is best to
# set JAVA_HOME in this file, so that it is correctly defined on
# remote nodes.
# The java implementation to use. Required.
export JAVA_HOME=/usr/java/jdk1.8.0_45
# Extra Java CLASSPATH elements. Optional.
# export HADOOP_CLASSPATH=
# The maximum amount of heap to use, in MB. Default is 1000.
# export HADOOP_HEAPSIZE=2000
# Extra Java runtime options. Empty by default.
# export HADOOP_OPTS=-server
# Command specific options appended to HADOOP_OPTS when specified
export HADOOP_NAMENODE_OPTS="-Dcom.sun.management.jmxremote $HADOOP_NAMENODE_OPTS"
export HADOOP_SECONDARYNAMENODE_OPTS="-Dcom.sun.management.jmxremote $HADOOP_SECONDARYNAMENODE_OPTS"
export HADOOP_DATANODE_OPTS="-Dcom.sun.management.jmxremote $HADOOP_DATANODE_OPTS"
export HADOOP_BALANCER_OPTS="-Dcom.sun.management.jmxremote $HADOOP_BALANCER_OPTS"
export HADOOP_JOBTRACKER_OPTS="-Dcom.sun.management.jmxremote $HADOOP_JOBTRACKER_OPTS"
# export HADOOP_TASKTRACKER_OPTS=
# The following applies to multiple commands (fs, dfs, fsck, distcp etc)
# export HADOOP_CLIENT_OPTS
# Extra ssh options. Empty by default.
# export HADOOP_SSH_OPTS="-o ConnectTimeout=1 -o SendEnv=HADOOP_CONF_DIR"
# Where log files are stored. $HADOOP_HOME/logs by default.
# export HADOOP_LOG_DIR=${HADOOP_HOME}/logs
# File naming remote slave hosts. $HADOOP_HOME/conf/slaves by default.
# export HADOOP_SLAVES=${HADOOP_HOME}/conf/slaves
# host:path where hadoop code should be rsync'd from. Unset by default.
# export HADOOP_MASTER=master:/home/$USER/src/hadoop
# Seconds to sleep between slave commands. Unset by default. This
# can be useful in large clusters, where, e.g., slave rsyncs can
# otherwise arrive faster than the master can service them.
# export HADOOP_SLAVE_SLEEP=0.1
# The directory where pid files are stored. /tmp by default.
# export HADOOP_PID_DIR=/var/hadoop/pids
# A string representing this instance of hadoop. $USER by default.
# export HADOOP_IDENT_STRING=$USER
# The scheduling priority for daemon processes. See 'man nice'.
# export HADOOP_NICENESS=10
core-site.xml文件修改内容
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!-- Put site-specific property overrides in this file. -->
<configuration>
<property>
<name>fs.default.name</name>
<value>hdfs://localhost:9000</value>
</property>
</configuration>
hdfs-site.xml文件修改内容
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!-- Put site-specific property overrides in this file. -->
<configuration>
<property>
<name>dfs.data.dir</name>
<value>/home/soft/hadoop/hadoop-0.20.2/data</value><!-- 数据节点路径 -->
</property>
<property>
<name>dfs.replication</name>
<value>1</value> <!-- 数据块复制多少份 -->
</property>
</configuration>
mapred-site.xml文件修改内容
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!-- Put site-specific property overrides in this file. -->
<configuration>
<property>
<name>mapred.job.tracker</name>
<value>localhost:9001</value>
</property>
</configuration>
7.格式化分布式文件系统
[
root@hadoop hadoop-0.20.2]#
bin/hadoop namenode -format15/08/03 23:57:29 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG: host = hadoop/192.168.1.103
STARTUP_MSG: args = [-format]
STARTUP_MSG: version = 0.20.2
STARTUP_MSG: build =
https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20 -r 911707; compiled by 'chrisdo' on Fri Feb 19 08:07:34 UTC 2010
************************************************************/
15/08/03 23:57:30 INFO namenode.FSNamesystem: fsOwner=root,root
15/08/03 23:57:30 INFO namenode.FSNamesystem: supergroup=supergroup
15/08/03 23:57:30 INFO namenode.FSNamesystem: isPermissionEnabled=true
15/08/03 23:57:30 INFO common.Storage: Image file of size 94 saved in 0 seconds.
15/08/03 23:57:30 INFO common.Storage: Storage directory /tmp/hadoop-root/dfs/name
has been successfully formatted.15/08/03 23:57:30 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at hadoop/192.168.1.103
************************************************************/
8.启动守护进程
starting namenode, logging to /home/soft/hadoop/hadoop-0.20.2/bin/../logs/hadoop-root-namenode-hadoop.out
The authenticity of host 'localhost (::1)' can't be established.
RSA key fingerprint is 9d:53:31:22:f4:06:51:7c:9c:95:bc:91:0b:1b:8f:82.
Are you sure you want to continue connecting (yes/no)? yes
localhost: Warning: Permanently added 'localhost' (RSA) to the list of known hosts.
root@localhost's password:
localhost: starting datanode, logging to /home/soft/hadoop/hadoop-0.20.2/bin/../logs/hadoop-root-datanode-hadoop.out
root@localhost's password:
localhost: starting secondarynamenode, logging to /home/soft/hadoop/hadoop-0.20.2/bin/../logs/hadoop-root-secondarynamenode-hadoop.out
starting jobtracker, logging to /home/soft/hadoop/hadoop-0.20.2/bin/../logs/hadoop-root-jobtracker-hadoop.out
root@localhost's password:
localhost: starting tasktracker, logging to /home/soft/hadoop/hadoop-0.20.2/bin/../logs/hadoop-root-tasktracker-hadoop.out
9.观看与java有关的进程
[root@hadoop hadoop-0.20.2]# ls /usr/java/jdk1.8.0_45/bin/jps
/usr/java/jdk1.8.0_45/bin/jps
[root@hadoop hadoop-0.20.2]# /usr/java/jdk1.8.0_45/bin/jps
14112 NameNode
14384 JobTracker
14209 DataNode
14483 TaskTracker
14329 SecondaryNameNode
14638 Jps
最后要干的事情就是将刚刚解压后的hadoop-0.20.2文件夹整个拷贝到另外两个slave机器下面。
然后执行bin/stop-all.sh关闭所有的进程,再启动bin/start-all.sh
相关推荐
CentOS6.5系统下Hadoop2.6.0完全分布式环境安装与配置信息介绍
hadoop0.20.2版本在虚拟机中的安装指南,经过测试可以实现
Hadoop 2.4 完全分布式环境安装与配置
Hadoop-2.4.0分布式安装手册
hadoop与spark分布式安装,内容详细,亲自搭建成功。助于新手
Hadoop2.7.6完全分布式安装配置,亲测可用!配置文档如:https://blog.csdn.net/likunwen_001/article/details/80434579
虚拟机实现的hadoop完全分布式模式的安装和配置,有详细的操作步骤和截图,亲测成功。
Hadoop2.2.0安装配置手册!完全分布式Hadoop集群搭建过程 按照文档中的操作步骤,一步步操作就可以完全实现hadoop2.2.0版本的完全分布式集群搭建过程
Hadoop-2.7.1分布式安装手册,包含了单机、伪分布式、完全分布式方式
hbase-0.90.5.tar.gz与hadoop0.20.2版本匹配,我在我本地虚拟机已经安装成功可以使用。请放心下载!!!
hadoop2.7分布式完全安装配置手册,含hbase1.2安装配置,图文并茂,一看就会。
hadoop完全分布式安装步骤 实验环境为VirtualBOx虚拟机
redHat linux下安装hadoop 0.20.2
本文档适合初学者,指引初学者或初次接触 Hadoop 的人成功安装 Hadoop ,使用代码执行简单的 HDFS 操作,并利用 MapReduce 执行分布式计算示例代码。其中 MapReduce 示例代码为 Hadoop 官方带的 hadoop-0.20.2-...
hadoop完全分布式安装整合hive,亲自在实际服务器安装测试成功
开源分布式架构Hadoop 0.20.2版的Java API文档,使用Hadoop都需要它
Hadoop搭建完全分布式集群
本人搭建hadoop完全分布式集群时,写的文档,内有详细步骤,
Hadoop-0.21.0分布式集群配置.doc
Hadoop安装教程_单机_伪分布式配置