沈阳网站建设方案,网站首页设计定位,百度百科官网入口,互联网投诉服务平台原理图 两个集群---目的#xff1a;扩容 HA联邦模式解决了单纯HA模式的性能瓶颈#xff08;主要指Namenode、ResourceManager#xff09;#xff0c;将整个HA集群划分为两个以上的集群#xff0c;不同的集群之间通过Federation进行连接#xff0c;使得HA集群拥有了横向扩… 原理图 两个集群---目的扩容 HA联邦模式解决了单纯HA模式的性能瓶颈主要指Namenode、ResourceManager将整个HA集群划分为两个以上的集群不同的集群之间通过Federation进行连接使得HA集群拥有了横向扩展的能力。理论上在该模式下能够通过增加计算节点以处理无限增长的数据。联邦模式下的配置在原HA模式的基础上做了部分调整。 配置过程 federation cp -r local/ federation 1.规划集群 ns1:nn1(s101) nn2(s102) ns2:nn3(s103) nn4(s014) 2.准备 [nn1 ~ nn4 ]ssh 所有节点. 3.停止整个集群 4.配置文件 4.1)s101和s102的hdfs-site.xml配置 [hadoop/federation/hdfs-site.xml] ?xml version1.0 encodingUTF-8? ?xml-stylesheet typetext/xsl hrefconfiguration.xsl? configuration property namedfs.nameservices/name valuens1,ns2/value /property !-- **************ns1********************* -- property namedfs.ha.namenodes.ns1/name valuenn1,nn2/value /property property namedfs.namenode.rpc-address.ns1.nn1/name values101:8020/value /property property namedfs.namenode.rpc-address.ns1.nn2/name values102:8020/value /property property namedfs.namenode.http-address.ns1.nn1/name values101:50070/value /property property namedfs.namenode.http-address.ns1.nn2/name values102:50070/value /property property namedfs.client.failover.proxy.provider.ns1/name valueorg.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider/value /property !-- **************ns2********************* -- property namedfs.ha.namenodes.ns2/name valuenn3,nn4/value /property property namedfs.namenode.rpc-address.ns2.nn3/name values103:8020/value /property property namedfs.namenode.rpc-address.ns2.nn4/name values104:8020/value /property property namedfs.namenode.http-address.ns2.nn3/name values103:50070/value /property property namedfs.namenode.http-address.ns2.nn4/name values104:50070/value /property property namedfs.client.failover.proxy.provider.ns2/name valueorg.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider/value /property !--***********************************************-- property namedfs.namenode.shared.edits.dir/name valueqjournal://s102:8485;s103:8485;s104:8485/ns1/value /property property namedfs.ha.fencing.methods/name value sshfence shell(/bin/true) /value /property property namedfs.ha.fencing.ssh.private-key-files/name value/home/centos/.ssh/id_rsa/value /property property namedfs.ha.automatic-failover.enabled/name valuetrue/value /property property namedfs.replication/name value3/value /property /configuration 4.2)s103和s104的hdfs-site.xml配置 [hadoop/federation/hdfs-site.xml] ?xml version1.0 encodingUTF-8? ?xml-stylesheet typetext/xsl hrefconfiguration.xsl? configuration property namedfs.nameservices/name valuens1,ns2/value /property !-- **************ns1********************* -- property namedfs.ha.namenodes.ns1/name valuenn1,nn2/value /property property namedfs.namenode.rpc-address.ns1.nn1/name values101:8020/value /property property namedfs.namenode.rpc-address.ns1.nn2/name values102:8020/value /property property namedfs.namenode.http-address.ns1.nn1/name values101:50070/value /property property namedfs.namenode.http-address.ns1.nn2/name values102:50070/value /property property namedfs.client.failover.proxy.provider.ns1/name valueorg.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider/value /property !-- **************ns2********************* -- property namedfs.ha.namenodes.ns2/name valuenn3,nn4/value /property property namedfs.namenode.rpc-address.ns2.nn3/name values103:8020/value /property property namedfs.namenode.rpc-address.ns2.nn4/name values104:8020/value /property property namedfs.namenode.http-address.ns2.nn3/name values103:50070/value /property property namedfs.namenode.http-address.ns2.nn4/name values104:50070/value /property property namedfs.client.failover.proxy.provider.ns2/name valueorg.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider/value /property !--***********************************************-- property namedfs.namenode.shared.edits.dir/name valueqjournal://s102:8485;s103:8485;s104:8485/ns2/value /property property namedfs.ha.fencing.methods/name value sshfence shell(/bin/true) /value /property property namedfs.ha.fencing.ssh.private-key-files/name value/home/centos/.ssh/id_rsa/value /property property namedfs.ha.automatic-failover.enabled/name valuetrue/value /property property namedfs.replication/name value3/value /property /configuration 4.3)s101 ~ s104的core-site.xml配置文件 [hadoop/federation/core-site.xml] ?xml version1.0? configuration xmlns:xihttp://www.w3.org/2001/XInclude xi:include hrefmountTable.xml / property namefs.defaultFS/name valueviewfs://ClusterX/value /property property namedfs.journalnode.edits.dir/name value/home/centos/hadoop/federation/journalnode/value /property property namehadoop.tmp.dir/name value/home/centos/hadoop/federation/value /property property nameha.zookeeper.quorum/name values102:2181,s103:2181,s104:2181/value /property /configuration 4.4)mountTable.xml 挂载表文件 [hadoop/federation/mountTable.xml] configuration property namefs.viewfs.mounttable.ClusterX.homedir/name value/home/value /property property namefs.viewfs.mounttable.ClusterX.link./home/name valuehdfs://ns1/home/value /property property namefs.viewfs.mounttable.ClusterX.link./tmp/name valuehdfs://ns2/tmp/value /property property namefs.viewfs.mounttable.ClusterX.link./projects/foo/name valuehdfs://ns1/projects/foo/value /property property namefs.viewfs.mounttable.ClusterX.link./projects/bar/name valuehdfs://ns2/projects/bar/value /property /configuration 5.操作 5.1)删除所有节点的日志和本地临时目录 $xcall.sh rm -rf /soft/hadoop/logs/* $xcall.sh rm -rf /home/centos/hadoop/federation/* 5.2)修改所有节点的hadoop软连接 $xcall.sh ln -sfT /soft/hadoop/etc/federation /soft/hadoop/etc/hadoop 5.3)对ns1集群进行格式化以及初始工作 a)启动jn集群 登录s102 ~ s104启动jounalnode进程。 $hadoop-daemon.sh start journalnode b)格式化nn1节点 [s101] $hdfs namenode -format c)复制s101的元数据到s102下. [s101] $scp -r d)在s102上执行引导过程 #s101启动名称节点 $hadoop-daemon.sh start namenode # s102执行引导,不要重格(N) $hdfs namenode -bootstrapStandby e)在s102上初始化编辑日志到jn集群(N) $hdfs namenode -initializeSharedEdits f)在s102对zookeeper格式化zkfc(选择Y). $hdfs zkfc -formatZK g)启动s101和s102的namenode和zkfc进程。 [s101] $hadoop-daemon.sh start zkfc [s102] $hadoop-daemon.sh start namenode $hadoop-daemon.sh start zkfc h)测试webui 5.4)对ns2集群进行格式化以及初始工作 a)格式nn3切记使用-clusterId属性保持和ns1的一致。 [s103] $hdfs namenode -format -clusterId CID-e16c5e2f-c0a5-4e51-b789-008e36b7289a b)复制s103的元数据到s104上。 $scp -r /home/centos/hadoop/federation centoss104:/home/centos/hadoop/ c)在s104引导 #在s103启动namenode $hadoop-daemon.sh start namenode #在s104执行引导 $hdfs namenode -bootstrapStandby d)在s104上初始化编辑日志 $hdfs namenode -initializeSharedEdits e)在s104对zookeeper格式化zkfc(选择Y). $hdfs zkfc -formatZK f)启动s103和s104的namenode和zkfc进程。 [s103] $hadoop-daemon.sh start zkfc [s104] $hadoop-daemon.sh start namenode $hadoop-daemon.sh start zkfc 5.5)停止集群 $stop-dfs.sh 5.6)重启dfs集群 $start-dfs.sh 5.7)创建目录 # 注意加p参数 $hdfs dfs -mkdir -p /home/data #上传文件,考察webui $hdfs dfs -put 1.txt /home/data 转载于:https://www.cnblogs.com/star521/p/9703171.html