生产备案号怎么查询网站,wordpress 变更中文,专业放心的企业展厅设计,个人网站空间购买作为Hadoop初学者#xff0c;自然要从安装入手。而hadoop的优势就是分布式#xff0c;所以#xff0c;也一定要安装分布式的系统。 整体安装步骤#xff0c;包括ZookeeperHDFSHbase#xff0c;为了文章简洁#xff0c;我会分三篇blog记录我的安装步骤。 本文记录的是集群…作为Hadoop初学者自然要从安装入手。而hadoop的优势就是分布式所以也一定要安装分布式的系统。 整体安装步骤包括ZookeeperHDFSHbase为了文章简洁我会分三篇blog记录我的安装步骤。 本文记录的是集群环境准备zookeeper的安装步骤。 ~~~~~~~~~~~环境准备部分~~~~~~~~~~~~~~~~~ IP部署规划准备了三个虚拟机操作系统是Centos6.7 Zookeeper主机分配 192.168.67.101 c6701192.168.67.102 c6702192.168.67.103 c6703 HDFS主机分配 192.168.67.101 c6701 --Namenodedatanode192.168.67.102 c6702 --datanode192.168.67.103 c6703 --datanode Hbase主机分配 192.168.67.101 c6701 --Master region192.168.67.102 c6702 --region192.168.67.103 c6703 --region 1. 为了分别管理各个服务我为每个服务都单独创建了用户. 创建的用户有: zk hdfs hbase 2. 为每一个用户创建ssh免密,并分发到全部节点这样我就只在C6701上执行命令通过远程的方式安装C6702和C6703 ssh-keygen -t rsa -f ~/.ssh/id_rsa 拷贝密钥到全部节点即可只有c6701访问02和03不需要密码反向访问是需要密码的 ssh-copy-id c6701 ssh-copy-id c6702 ssh-copy-id c6703 3. 强调一下/etc/hosts 需要设置 [rootc6701 ~]# more /etc/hosts 192.168.67.101 c6701.org c6701192.168.67.102 c6702.org c6702192.168.67.103 c6703.org c6703 4. 下面在C6701执行中间会调用ssh远程执行一些命令,验证ssh免密是否生效 ssh c6702 cat /proc/cpuinfo ssh c6702 hostname 5. 下载安装软件内网地址 cd /tmp/software wget http://192.21.104.48/deploy/jdk-8u144-linux-x64.tar.gz wget http://192.21.104.48/deploy/zookeeper-3.4.6.tar.gz wget http://192.21.104.48/deploy/hbase-1.1.3.tar.gz wget http://192.21.104.48/deploy/hadoop-2.6.0-EDH-0u2.tar.gz wget http://192.21.104.48/deploy/hadoop-2.7.1.tar.gz 6. 安装JDK全部节点都要安装 #tar -xzvf jdk-8u144-linux-x64.tar.gz -C /usr/local 7. 添加下面信息到.bash_profile export JAVA_HOME/usr/local/jdk1.8.0_144 export JRE_HOME/usr/local/jdk1.8.0_144/jre export CLASSPATH.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar:$JRE_HOME/lib:$CLASSPATH export PATH$JAVA_HOME/bin:$PATH source /etc/profile 8. 检查版本信息 # java -version java version 1.8.0_144 Java(TM) SE Runtime Environment (build 1.8.0_144-b01) Java HotSpot(TM) 64-Bit Server VM (build 25.144-b01, mixed mode) ~~~~~~~~~~~zookeeper的安装部分~~~~~~~~~~~~~~~~~ 1. 在C6701安装Zookeeper useradd zk echo zk:zk | chpasswd su - zk mkdir zk tar -zxvf /tmp/software/zookeeper-3.4.6.tar.gz -C /home/zk/zk ~~~~~~~~~~~~~zoo.cfg的配置~~~~~~~~~~~~~~~ $ pwd/home/zk/zk/zookeeper-3.4.6/conf/zoo.cfg $ cat /home/zk/zk/zookeeper-3.4.6/conf/zoo.cfg |grep -v ^# tickTime2000 initLimit10 syncLimit5 dataDir/data/zookeeper/data dataLogDir/data/zookeeper/log clientPort2181 autopurge.snapRetainCount3 autopurge.purgeInterval6 server.1c6701:2888:3888 server.2c6702:2888:3888 server.3c6703:2888:3888~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2. 根据zoo.cfg创建data和log两个文件夹 #mkdir -p /data/zookeeper/data#mkdir -p /data/zookeeper/log#chown -R zk:zk /data/zookeeper#chown -R zk:zk /data/zookeeper/data#chown -R zk:zk /data/zookeeper/log 3.在zookeeper的目录中创建上述两个文件夹。进入zkdata文件夹创建文件myid填入1。这里写入的1是在zoo.cfg文本中的server.1中的1。当我们把所有文件都配置完毕我们把hadoop1中yarn目录复制到其它机器中我们在修改每台机器中对应的myid文本hadoop2中的myid写入2。其余节点安照上面配置依此写入相应的数字。Zkdatalog文件夹是为了指定zookeeper产生日志指定相应的路径。 # su - zk -c echo 1 /data/zookeeper/data/myid 4. 添加环境变量,在/etc/profile目录中,将ZOOKEEPER_HOME/bin在原有的PATH后面加入:$ZOOKEEPER_HOME/bin 关于环境变量修改/etc目录下的profile文件也可以在根目录下的.bashrc目录下添加环境变量。这两者有什么区别.bashrc是对当前目录用户的环境变量profile文件是对所有用户都开放的目录。当系统加载文件中先从profile找相应的路劲如果没有会在.bashrc文件中找对应的环境变量路径。这两者大家稍至了解。 然后 source /etc/profile 5. 安装c6702的zookeeper #ssh c6702 useradd zk#ssh c6702 echo zk:zk | chpasswd 为zk用户ssh免密 #ssh-copy-id zkc6702 拷贝软件 #scp -r /tmp/software/hadoop-* rootc6702:/tmp/software#ssh c6702 chmod 777 /tmp/software/* 创建目录解压软件 #ssh zkc6702 mkdir zk#ssh zkc6702 tar -zxvf /tmp/software/zookeeper-3.4.6.tar.gz -C /home/zk/zk#ssh zkc6702 ls -al zk#ssh zkc6702 ls -al zk/zookeeper*#ssh zkc6702 rm /home/zk/zk/zookeeper-3.4.6/conf/zoo.cfg#scp -r /home/zk/zk/zookeeper-3.4.6/conf/zoo.cfg zkc6702:/home/zk/zk/zookeeper-3.4.6/conf/.#ssh zkc6702 cat /home/zk/zk/zookeeper-3.4.6/conf/zoo.cfg |grep -v ^# 根据zoo.cfg创建目录#ssh c6702 mkdir -p /data/zookeeper/data#ssh c6702 chown -R zk:zk /data/zookeeper#ssh c6702 chown -R zk:zk /data/zookeeper/data#ssh c6702 mkdir -p /data/zookeeper/log #ssh c6702 chown -R zk:zk /data/zookeeper/log 创建文件myid填入2 ssh zkc6702 echo 2 /data/zookeeper/data/myid 6. 安装c6703的zookeeper #ssh c6703 useradd zk#ssh c6703 echo zk:zk | chpasswd 为zk用户ssh免密 ssh-copy-id zkc6703 拷贝软件 #scp -r /tmp/software/hadoop-* rootc6703:/tmp/software#ssh c6703 chmod 777 /tmp/software/* 创建目录解压软件 #ssh zkc6703 mkdir zk#ssh zkc6703 tar -zxvf /tmp/software/zookeeper-3.4.6.tar.gz -C /home/zk/zk#ssh zkc6703 ls -al zk#ssh zkc6703 ls -al zk/zookeeper*#ssh zkc6703 rm /home/zk/zk/zookeeper-3.4.6/conf/zoo.cfg#scp -r /home/zk/zk/zookeeper-3.4.6/conf/zoo.cfg zkc6703:/home/zk/zk/zookeeper-3.4.6/conf/.#ssh zkc6703 cat /home/zk/zk/zookeeper-3.4.6/conf/zoo.cfg |grep -v ^# 根据zoo.cfg创建目录 #ssh c6703 mkdir -p /data/zookeeper/data#ssh c6703 chown -R zk:zk /data/zookeeper#ssh c6703 chown -R zk:zk /data/zookeeper/data#ssh c6703 mkdir -p /data/zookeeper/log #ssh c6703 chown -R zk:zk /data/zookeeper/log 创建文件myid填入3 ssh zkc6703 echo 3 /data/zookeeper/data/myid 7. 启动ZK /home/zk/zk/zookeeper-3.4.6/bin/zkServer.sh start/home/zk/zk/zookeeper-3.4.6/bin/zkServer.sh status 远程启动命令 ssh zkc6702 /home/zk/zk/zookeeper-3.4.6/bin/zkServer.sh\ start ssh zkc6703 /home/zk/zk/zookeeper-3.4.6/bin/zkServer.sh\ start ssh zkc6702 /home/zk/zk/zookeeper-3.4.6/bin/zkServer.sh\ status ssh zkc6703 /home/zk/zk/zookeeper-3.4.6/bin/zkServer.sh\ status 8. 没有全部启动三个ZK前的状态 [vagrantc7003 bin]$ ./zkServer.sh status ZooKeeper JMX enabled by defaultUsing config: /home/vagrant/zookeeper-3.4.10/bin/../conf/zoo.cfg Error contacting service. It is probably not running. 9. 需要将三个ZK全启动之后才能选举出leader $ ./zkServer.sh status ZooKeeper JMX enabled by defaultUsing config: /home/vagrant/zookeeper-3.4.10/bin/../conf/zoo.cfgMode: follower[vagrantc7002 bin]$ ./zkServer.sh status ZooKeeper JMX enabled by defaultUsing config: /home/vagrant/zookeeper-3.4.10/bin/../conf/zoo.cfgMode: leader 2018.1.13更新 客户端连接方式 客户端是随机顺序的方式连接server的无法指定或者优先读哪个一个zookeeper server。 ./zkCli.sh -server c6701:2181,c6702:2181,c6703:2181 输出如下 第一次连接连接到c6701上 [zookeeperc6702 bin]$ /usr/local/hadoop/zookeeper-3.4.6/bin/zkCli.sh -server c6701:2181,c6702:2181,c6703:2181 Connecting to c6701:2181,c6702:2181,c6703:2181......2018-01-11 21:07:30,797 [myid:] - INFO [main:ZooKeeper438] - Initiating client connection, connectStringc6701:2181,c6702:2181,c6703:2181 sessionTimeout30000 watcherorg.apache.zookeeper.ZooKeeperMain$MyWatcher4b4bc1e Welcome to ZooKeeper!2018-01-11 21:07:30,830 [myid:] - INFO [main-SendThread(c6701.python279.org:2181):ClientCnxn$SendThread975] - Opening socket connection to server c6701.python279.org/192.168.67.101:2181. Will not attempt to authenticate using SASL (unknown error) JLine support is enabled2018-01-11 21:07:30,873 [myid:] - INFO [main-SendThread(c6701.python279.org:2181):ClientCnxn$SendThread852] - Socket connection established to c6701.python279.org/192.168.67.101:2181, initiating session2018-01-11 21:07:30,916 [myid:] - INFO [main-SendThread(c6701.python279.org:2181):ClientCnxn$SendThread1235] - Session establishment complete on server c6701.python279.org/192.168.67.101:2181, sessionid 0x160e70285f70001, negotiated timeout 30000 WATCHER:: WatchedEvent state:SyncConnected type:None path:null 第二次连接连接到c6702上 Connecting to c6701:2181,c6702:2181,c6703:2181......2018-01-11 21:10:18,442 [myid:] - INFO [main:ZooKeeper438] - Initiating client connection, connectStringc6701:2181,c6702:2181,c6703:2181 sessionTimeout30000 watcherorg.apache.zookeeper.ZooKeeperMain$MyWatcher4b4bc1e Welcome to ZooKeeper!2018-01-11 21:10:18,489 [myid:] - INFO [main-SendThread(c6702.python279.org:2181):ClientCnxn$SendThread975] - Opening socket connection to server c6702.python279.org/192.168.67.102:2181. Will not attempt to authenticate using SASL (unknown error) JLine support is enabled2018-01-11 21:10:18,508 [myid:] - INFO [main-SendThread(c6702.python279.org:2181):ClientCnxn$SendThread852] - Socket connection established to c6702.python279.org/192.168.67.102:2181, initiating session2018-01-11 21:10:18,561 [myid:] - INFO [main-SendThread(c6702.python279.org:2181):ClientCnxn$SendThread1235] - Session establishment complete on server c6702.python279.org/192.168.67.102:2181, sessionid 0x260e70284650002, negotiated timeout 30000 WATCHER:: WatchedEvent state:SyncConnected type:None path:null 至此基础环境部署完。并且zookeeper也安装完成。下面我们会在下一篇文章中继续安装HDFS。 转载于:https://www.cnblogs.com/mylovelulu/p/9521984.html
