网站设计 中国风,温州设计集团网站建设,网站后台添加新闻,山西住房和建设厅网站1.修改Linux主机名2.修改IP3.修改主机名和IP的映射关系######注意######如果你们公司是租用的服务器或是使用的云主机#xff08;如华为用主机、阿里云主机等#xff09;/etc/hosts里面要配置的是内网IP地址和主机名的映射关系4.关闭防火墙5.ssh免登陆6.安装JDK#xff0c;配… 1.修改Linux主机名2.修改IP3.修改主机名和IP的映射关系######注意######如果你们公司是租用的服务器或是使用的云主机如华为用主机、阿里云主机等/etc/hosts里面要配置的是内网IP地址和主机名的映射关系4.关闭防火墙5.ssh免登陆6.安装JDK配置环境变量等集群规划主机名IP安装的软件运行的进程weekend01192.168.1.201jdk、hadoopNameNode、DFSZKFailoverController(zkfc)weekend02192.168.1.202jdk、hadoopNameNode、DFSZKFailoverController(zkfc)weekend03192.168.1.203jdk、hadoopResourceManagerweekend04192.168.1.204jdk、hadoopResourceManagerweekend05192.168.1.205jdk、hadoop、zookeeperDataNode、NodeManager、JournalNode、QuorumPeerMainweekend06192.168.1.206jdk、hadoop、zookeeperDataNode、NodeManager、JournalNode、QuorumPeerMainweekend07192.168.1.207jdk、hadoop、zookeeperDataNode、NodeManager、JournalNode、QuorumPeerMain说明1.在hadoop2.0中通常由两个NameNode组成一个处于active状态另一个处于standby状态。Active NameNode对外提供服务而Standby NameNode则不对外提供服务仅同步active namenode的状态以便能够在它失败时快速进行切换。hadoop2.0官方提供了两种HDFS HA的解决方案一种是NFS另一种是QJM。这里我们使用简单的QJM。在该方案中主备NameNode之间通过一组JournalNode同步元数据信息一条数据只要成功写入多数JournalNode即认为写入成功。通常配置奇数个JournalNode这里还配置了一个zookeeper集群用于ZKFCDFSZKFailoverController故障转移当Active NameNode挂掉了会自动切换Standby NameNode为standby状态2.hadoop-2.2.0中依然存在一个问题就是ResourceManager只有一个存在单点故障hadoop-2.4.1解决了这个问题有两个ResourceManager一个是Active一个是Standby状态由zookeeper进行协调安装步骤1.安装配置zooekeeper集群在weekend05上1.1解压tar -zxvf zookeeper-3.4.5.tar.gz -C /weekend/1.2修改配置cd /weekend/zookeeper-3.4.5/conf/cp zoo_sample.cfg zoo.cfgvim zoo.cfg修改dataDir/weekend/zookeeper-3.4.5/tmp在最后添加server.1weekend05:2888:3888server.2weekend06:2888:3888server.3weekend07:2888:3888保存退出然后创建一个tmp文件夹mkdir /weekend/zookeeper-3.4.5/tmp再创建一个空文件touch /weekend/zookeeper-3.4.5/tmp/myid最后向该文件写入IDecho 1 /weekend/zookeeper-3.4.5/tmp/myid1.3将配置好的zookeeper拷贝到其他节点(首先分别在weekend06、weekend07根目录下创建一个weekend目录mkdir /weekend)scp -r /weekend/zookeeper-3.4.5/ weekend06:/weekend/scp -r /weekend/zookeeper-3.4.5/ weekend07:/weekend/注意修改weekend06、weekend07对应/weekend/zookeeper-3.4.5/tmp/myid内容weekend06echo 2 /weekend/zookeeper-3.4.5/tmp/myidweekend07echo 3 /weekend/zookeeper-3.4.5/tmp/myid2.安装配置hadoop集群在weekend01上操作2.1解压tar -zxvf hadoop-2.4.1.tar.gz -C /weekend/2.2配置HDFShadoop2.0所有的配置文件都在$HADOOP_HOME/etc/hadoop目录下#将hadoop添加到环境变量中vim /etc/profileexport JAVA_HOME/usr/java/jdk1.7.0_55export HADOOP_HOME/weekend/hadoop-2.4.1export PATH$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin#hadoop2.0的配置文件全部在$HADOOP_HOME/etc/hadoop下cd /home/hadoop/app/hadoop-2.4.1/etc/hadoop2.2.1修改hadoo-env.shexport JAVA_HOME/home/hadoop/app/jdk1.7.0_552.2.2修改core-site.xmlconfiguration!-- 指定hdfs的nameservice为ns1 --propertynamefs.defaultFS/namevaluehdfs://ns1//value/property!-- 指定hadoop临时目录 --propertynamehadoop.tmp.dir/namevalue/home/hadoop/app/hadoop-2.4.1/tmp/value/property!-- 指定zookeeper地址 --propertynameha.zookeeper.quorum/namevalueweekend05:2181,weekend06:2181,weekend07:2181/value/property/configuration2.2.3修改hdfs-site.xmlconfiguration!--指定hdfs的nameservice为ns1需要和core-site.xml中的保持一致 --propertynamedfs.nameservices/namevaluens1/value/property!-- ns1下面有两个NameNode分别是nn1nn2 --propertynamedfs.ha.namenodes.ns1/namevaluenn1,nn2/value/property!-- nn1的RPC通信地址 --propertynamedfs.namenode.rpc-address.ns1.nn1/namevalueweekend01:9000/value/property!-- nn1的http通信地址 --propertynamedfs.namenode.http-address.ns1.nn1/namevalueweekend01:50070/value/property!-- nn2的RPC通信地址 --propertynamedfs.namenode.rpc-address.ns1.nn2/namevalueweekend02:9000/value/property!-- nn2的http通信地址 --propertynamedfs.namenode.http-address.ns1.nn2/namevalueweekend02:50070/value/property!-- 指定NameNode的元数据在JournalNode上的存放位置 --propertynamedfs.namenode.shared.edits.dir/namevalueqjournal://weekend05:8485;weekend06:8485;weekend07:8485/ns1/value/property!-- 指定JournalNode在本地磁盘存放数据的位置 --propertynamedfs.journalnode.edits.dir/namevalue/home/hadoop/app/hadoop-2.4.1/journaldata/value/property!-- 开启NameNode失败自动切换 --propertynamedfs.ha.automatic-failover.enabled/namevaluetrue/value/property!-- 配置失败自动切换实现方式 --propertynamedfs.client.failover.proxy.provider.ns1/namevalueorg.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider/value/property!-- 配置隔离机制方法多个机制用换行分割即每个机制暂用一行--propertynamedfs.ha.fencing.methods/namevaluesshfenceshell(/bin/true)/value/property!-- 使用sshfence隔离机制时需要ssh免登陆 --propertynamedfs.ha.fencing.ssh.private-key-files/namevalue/home/hadoop/.ssh/id_rsa/value/property!-- 配置sshfence隔离机制超时时间 --propertynamedfs.ha.fencing.ssh.connect-timeout/namevalue30000/value/property/configuration2.2.4修改mapred-site.xmlconfiguration!-- 指定mr框架为yarn方式 --propertynamemapreduce.framework.name/namevalueyarn/value/property/configuration2.2.5修改yarn-site.xmlconfiguration!-- 开启RM高可用 --propertynameyarn.resourcemanager.ha.enabled/namevaluetrue/value/property!-- 指定RM的cluster id --propertynameyarn.resourcemanager.cluster-id/namevalueyrc/value/property!-- 指定RM的名字 --propertynameyarn.resourcemanager.ha.rm-ids/namevaluerm1,rm2/value/property!-- 分别指定RM的地址 --propertynameyarn.resourcemanager.hostname.rm1/namevalueweekend03/value/propertypropertynameyarn.resourcemanager.hostname.rm2/namevalueweekend04/value/property!-- 指定zk集群地址 --propertynameyarn.resourcemanager.zk-address/namevalueweekend05:2181,weekend06:2181,weekend07:2181/value/propertypropertynameyarn.nodemanager.aux-services/namevaluemapreduce_shuffle/value/property/configuration2.2.6修改slaves(slaves是指定子节点的位置因为要在weekend01上启动HDFS、在weekend03启动yarn所以weekend01上的slaves文件指定的是datanode的位置weekend03上的slaves文件指定的是nodemanager的位置)weekend05weekend06weekend072.2.7配置免密码登陆#首先要配置weekend01到weekend02、weekend03、weekend04、weekend05、weekend06、weekend07的免密码登陆#在weekend01上生产一对钥匙ssh-keygen -t rsa#将公钥拷贝到其他节点包括自己ssh-coyp-id weekend01ssh-coyp-id weekend02ssh-coyp-id weekend03ssh-coyp-id weekend04ssh-coyp-id weekend05ssh-coyp-id weekend06ssh-coyp-id weekend07#配置weekend03到weekend04、weekend05、weekend06、weekend07的免密码登陆#在weekend03上生产一对钥匙ssh-keygen -t rsa#将公钥拷贝到其他节点ssh-coyp-id weekend04ssh-coyp-id weekend05ssh-coyp-id weekend06ssh-coyp-id weekend07#注意两个namenode之间要配置ssh免密码登陆别忘了配置weekend02到weekend01的免登陆在weekend02上生产一对钥匙ssh-keygen -t rsassh-coyp-id -i weekend012.4将配置好的hadoop拷贝到其他节点scp -r /weekend/ weekend02:/scp -r /weekend/ weekend03:/scp -r /weekend/hadoop-2.4.1/ hadoopweekend04:/weekend/scp -r /weekend/hadoop-2.4.1/ hadoopweekend05:/weekend/scp -r /weekend/hadoop-2.4.1/ hadoopweekend06:/weekend/scp -r /weekend/hadoop-2.4.1/ hadoopweekend07:/weekend/###注意严格按照下面的步骤2.5启动zookeeper集群分别在weekend05、weekend06、tcast07上启动zkcd /weekend/zookeeper-3.4.5/bin/./zkServer.sh start#查看状态一个leader两个follower./zkServer.sh status2.6启动journalnode分别在在weekend05、weekend06、tcast07上执行cd /weekend/hadoop-2.4.1sbin/hadoop-daemon.sh start journalnode#运行jps命令检验weekend05、weekend06、weekend07上多了JournalNode进程2.7格式化HDFS#在weekend01上执行命令:hdfs namenode -format#格式化后会在根据core-site.xml中的hadoop.tmp.dir配置生成个文件这里我配置的是/weekend/hadoop-2.4.1/tmp然后将/weekend/hadoop-2.4.1/tmp拷贝到weekend02的/weekend/hadoop-2.4.1/下。scp -r tmp/ weekend02:/home/hadoop/app/hadoop-2.4.1/##也可以这样建议hdfs namenode -bootstrapStandby2.8格式化ZKFC(在weekend01上执行即可)hdfs zkfc -formatZK2.9启动HDFS(在weekend01上执行)sbin/start-dfs.sh2.10启动YARN(#####注意#####是在weekend03上执行start-yarn.sh把namenode和resourcemanager分开是因为性能问题因为他们都要占用大量资源所以把他们分开了他们分开了就要分别在不同的机器上启动)sbin/start-yarn.sh到此hadoop-2.4.1配置完毕可以统计浏览器访问:http://192.168.1.201:50070NameNode weekend01:9000 (active)http://192.168.1.202:50070NameNode weekend02:9000 (standby)验证HDFS HA首先向hdfs上传一个文件hadoop fs -put /etc/profile /profilehadoop fs -ls /然后再kill掉active的NameNodekill -9 pid of NN通过浏览器访问http://192.168.1.202:50070NameNode weekend02:9000 (active)这个时候weekend02上的NameNode变成了active在执行命令hadoop fs -ls /-rw-r--r-- 3 root supergroup 1926 2014-02-06 15:36 /profile刚才上传的文件依然存在手动启动那个挂掉的NameNodesbin/hadoop-daemon.sh start namenode通过浏览器访问http://192.168.1.201:50070NameNode weekend01:9000 (standby)验证YARN运行一下hadoop提供的demo中的WordCount程序hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.4.1.jar wordcount /profile /outOK大功告成测试集群工作状态的一些指令 bin/hdfs dfsadmin -report查看hdfs的各节点状态信息bin/hdfs haadmin -getServiceState nn1获取一个namenode节点的HA状态sbin/hadoop-daemon.sh start namenode 单独启动一个namenode进程./hadoop-daemon.sh start zkfc 单独启动一个zkfc进程hive安装过程中的问题解决命令与mysql集成yum remove mysql-libs 删除linux自带的mysql lib相关文件卸载mysql rpm -qa|grep -i mysql rpm -ev MySQL-server-4.0.14-0 MySQL-client-4.0.14-0 卸载后/var/lib/mysql中的数据及/etc/my.cnf不会删除如果确定没用后就手工删除 rm -f /etc/my.cnf rm -rf /var/lib/mysql安装mysql rpm -ivh 分别接mysql的服务端和客户端安装rpm包授权mysql grant all on hive.* to rootmpc1 identified by root;相关调试命令./hive -hiveconf hive.root.loggerDEBUG,consolehive create table t_order(id int,name String,rongliang String,price double)row format delimitedfields terminated by \t; hive load data local inpath /home/hadoop/hivetestdata/xxx.data into table t_order;load data inpath;hive create external table t_order_ex(id int,name String,rongliang String,price double)row format delimited fields terminated by \tlocation /hive_ext;hive create table order_simple asselect id,name,price from table t_order;hive create table t_order_pt(id int,name String,rongliang String,price double)partitioned by (month String)row format delimited fields terminated by \t;hive load data local inpath /home/hadoop/hivetestdata/xxx.data into table t_order_pt partition(month201401);hive select count(*) from t_order_pt where month201401;
