深圳 网站设计公司,商标在线设计,中信建设有限责任公司电话号码,wordpress关闭主题提示hadoop 环境创建
1. 八、window搭建spark IDEA开发环境
按照步骤安装完
2. windows下安装和配置hadoop
配置环境变量#xff0c;注意JAVA_HOME路径#xff0c;修改后#xff0c;重启电脑#xff0c;不重启容易报错#xff01;#xff01;#xff01;
新建dat…hadoop 环境创建
1. 八、window搭建spark IDEA开发环境
按照步骤安装完
2. windows下安装和配置hadoop
配置环境变量注意JAVA_HOME路径修改后重启电脑不重启容易报错
新建data文件夹 core-site.xml
?xml version1.0 encodingUTF-8?
?xml-stylesheet typetext/xsl hrefconfiguration.xsl?
!--Licensed under the Apache License, Version 2.0 (the License);you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an AS IS BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License. See accompanying LICENSE file.
--!-- Put site-specific property overrides in this file. --configurationpropertynamehadoop.tmp.dir/namevalue/D:/development/hadoop-3.1.1/data/value/propertypropertynamefs.defaultFS/namevaluehdfs://localhost:9000/value/property
/configuration
hdfs-site.xml
?xml version1.0 encodingUTF-8?
?xml-stylesheet typetext/xsl hrefconfiguration.xsl?
!--Licensed under the Apache License, Version 2.0 (the License);you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an AS IS BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License. See accompanying LICENSE file.
--!-- Put site-specific property overrides in this file. --configurationproperty namedfs.replication/name value1/value /property property namedfs.namenode.name.dir/name value/D:/development/hadoop-3.1.1/data/namenode/value /property property namedfs.datanode.data.dir/name value/D:/development/hadoop-3.1.1/data/datanode/value /property /configuration
mapred-site.xml
?xml version1.0?
?xml-stylesheet typetext/xsl hrefconfiguration.xsl?
!--Licensed under the Apache License, Version 2.0 (the License);you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an AS IS BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License. See accompanying LICENSE file.
--!-- Put site-specific property overrides in this file. --configurationpropertynamemapreduce.framework.name/namevalueyarn/value/property
/configuration
yarn-site.xml
?xml version1.0?
!--Licensed under the Apache License, Version 2.0 (the License);you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an AS IS BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License. See accompanying LICENSE file.
--
configuration!-- Site specific YARN configuration properties --propertynameyarn.nodemanager.aux-services/namevaluemapreduce_shuffle/value/propertypropertynameyarn.nodemanager.aux-services.mapreduce.shuffle.class/namevalueorg.apache.hahoop.mapred.ShuffleHandler/value/property
/configuration
3**.节点格式化**
正常情况下格式话结束会显示’namenode has been successfully formated’如果格式话失败原因可能是环境变量配置问题、hadoop版本和winutils版本不一致、etc中配置错误
hdfs namenode -format4. 启动hadoop
以管理员身份打开cmd使用winr快捷键打开运行输入cmd然后ctrlshiftenter选择‘是’打开cmd
执行start-all.cmd会新打开4个cmd分别是namenode、resourcemanager、nodemanager、datanode的4个进程如果这4个cmd启动没有报错则启动成功 出现四个窗口成功
5. 使用hadoop 参考
八、window搭建spark IDEA开发环境
windows下安装和配置hadoop
Windows下配置Hadoop环境全过程
hadoop启动报错JAVA_HOME is incorrectly set
