深圳高端设计网站,怎么建设百度网站,网址大全,北京托管网站正文内容一、安装jdk二、安装flume三、安装kafka1、zookeeper2、kafka四、启动测试步骤五、目录说明六、日志收集系统设计图七、大数据参考资料推荐一、安装jdk -(版本#xff1a;1.8.0_191)1.下载#xff1a;https://www.oracle.com/technetwork/java/javase/downloads/jdk8…正文内容一、安装jdk二、安装flume三、安装kafka1、zookeeper2、kafka四、启动测试步骤五、目录说明六、日志收集系统设计图七、大数据参考资料推荐一、安装jdk -(版本1.8.0_191)1.下载https://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html解压:tar -zxvf /jdk-8u191-linux-x64.tar.gz -C /home/ppgt/local/2.修改 /etc/profile 增加export JAVA_HOME/home/ppgt/local/jdk1.8.0_191export CLASSPATH.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar export PATH$PATH:$JAVA_HOME/bin3.检测java -version二、安装flume -(版本1.8.0)1.下载wget http://mirrors.hust.edu.cn/apache/flume/1.8.0/apache-flume-1.8.0-bin.tar.gz2.解压安装tar -zxvf apache-flume-1.8.0-bin.tar.gz -C /home/ppgt/local/3.验证安装bin/flume-ng version4.修改配置 conf/cp flume-env.sh.template flume-env.shvi flume-env.sh //配置java路径export JAVA_HOME/usr/local/jdk/jdk1.8.0_191-amd645.添加连接kafka的配置文件:conf/flumetokafka.conf#配置flume链接kafka# 定义这个agent中各组件的名字flume_kafka.sources exec-sourcesflume_kafka.sinks kafka-sinkflume_kafka.channels memory-channel# Describe/configure the sourceflume_kafka.sources.exec-sources.type execflume_kafka.sources.exec-sources.command tail -F /home/ppgt/tmpfile/testlogs/data.log# Describe the sinkflume_kafka.sinks.kafka-sink.type org.apache.flume.sink.kafka.KafkaSinkflume_kafka.sinks.kafka-sink.topic topiclogs01flume_kafka.sinks.kafka-sink.brokerList localhost:9092flume_kafka.sinks.kafka-sink.requiredAcks 1flume_kafka.sinks.kafka-sink.batchSize 20# Use a channel which buffers events in memoryflume_kafka.channels.memory-channel.type memoryflume_kafka.channels.memory-channel.capacity 1000flume_kafka.channels.memory-channel.transactionCapacity 100# Bind the source and sink to the channelflume_kafka.sources.exec-sources.channels memory-channelflume_kafka.sinks.kafka-sink.channel memory-channel三、安装kafka1.安装依赖zookeeper -(版本3.4.12)1)下载wget http://mirror.bit.edu.cn/apache/zookeeper/stable/zookeeper-3.4.12.tar.gz2)解压tar -zxvf zookeeper-3.4.12.tar.gz -C /home/ppgt/local/3)修改数据存储目录conf/zoo.cfgcp zoo_sample.cfg zoo.cfg修改值dataDir/home/ppgt/kafka_zk_tmp/tmp/zookeeper2.安装kafka -(版本0.9.0.0)1)下载wget https://archive.apache.org/dist/kafka/0.9.0.0/kafka_2.11-0.9.0.0.tgz2)解压tar -zxvf kafka_2.11-0.9.0.0.tgz -C /home/ppgt/local/3)修改配置文件config/server.propertiesa) zookeeper.connectlocalhost:2181 #zk服务地址b) host.namelocalhost #主机名c) log.dirs/home/ppgt/kafka_zk_tmp/tmp/kafka-logs#kafka数据的存放地址d) num.partitions1 #分区数量e) listenersPLAINTEXT://:9092 #kafka监听端口f) broker_id0 #唯一标识id四、启动测试步骤1.启动zookeeperbin/zkServer.sh start2.启动flumebin/flume-ng agent --conf conf --conf-file conf/flumetokafka.conf --name flume_kafka -Dflume.root.loggerINFO,console3.启动kafka//启动kafka服务bin/kafka-server-start.sh /home/ppgt/local/kafka_2.11-0.9.0.0/config/server.properties//创建一个topicbin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic topiclogs01//创建一个kafka消费者php /home/ppgt/www_test_ppgt_admin/syslogs_featrue_v1.0/script/cron/sysLogsConsumerRun.phpbin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic topiclogs01 --from-beginning//或者执行php消费者//生产测试日志echo 内容 exce /home/ppgt/tmpfile/testlogs/data.log五、目录说明1./home/ppgt/local/conf/zoo.cfg #配置文件conf/flumetokafka.conf #连接kafka的配置文件config/server.properties #配置文件conf/flume-env.sh#配置文件 jdk1.8.0_191 #jdk安装目录apache-flume-1.8.0-bin/#flume安装目录kafka_2.11-0.9.0.0/ #kafka安装目录zookeeper-3.4.12/ #zookeeper安装目录2./home/ppgt/kafka_zk_tmp/tmp/zookeeper/ #zookeeper数据存储目录kafka-logs/ #kafka数据的存放地址六、日志收集系统设计图七、大数据资料文章推荐flume:https://blog.csdn.net/mengfanzhundsc/article/details/81300310?fromsinglemessageisappinstalled0https://blog.csdn.net/caodaoxi/article/details/25903867https://www.cnblogs.com/tonglin0325/p/8963395.htmlhttp://www.wfuyu.com/technology/25331.htmlhttps://blog.csdn.net/Team77/article/details/44154529ahttps://blog.csdn.net/wuxintdrh/article/details/79478710https://blog.csdn.net/l1028386804/article/details/79366155https://blog.csdn.net/jy02268879/article/details/81024758http://itindex.net/detail/57323-flume-%E7%9B%91%E6%8E%A7-%E6%97%A5%E6%9C%9Fhttps://github.com/ypenglyn/locktail/blob/master/locktail_rotate.shhttps://blog.csdn.net/maoyuanming0806/article/details/79391010https://blog.csdn.net/u010316188/article/details/79905372https://blog.csdn.net/u011254180/article/details/80000763linux下flume安装flume使用taildir收集文件和文件夹taildirSource多文件监控实时采集滚动日志监控解决方案Flume使用监控文件实时采集新增数据输出到控制台Flume之——Flume读取日志数据写入Kafkaflume Source志SpoolDirflume监控flume监控参数说明flume监控指标详解大数据系列之Flume--几种不同的Sourceskafuka:https://www.jianshu.com/p/a036405f989chttps://blog.csdn.net/nankiao/article/details/78553635https://blog.csdn.net/weixin_38750084/article/details/82944759http://www.cnblogs.com/jun1019/p/6656223.htmlhttps://www.cnblogs.com/jun1019/p/6256514.htmlhttp://www.tianshouzhi.com/api/tutorials/kafka/117https://www.cnblogs.com/hei12138/p/7805475.htmlkafka介绍kafka的使用和安装kafka存储机制kafka配置说明kafka与zookeeper间的关联Zookeeper 在 Kafka 中的作用
