网站建设及安全管理文档广东网站建设方案

目录

1、使用xftp工具 在opt目录下创建install和soft文件

2、使用xftp工具 将压缩包上传到install文件

3、编写shell脚本

3.1、创建目录来放shell脚本

3.2、创建autoinsatll.sh文件并修改权限

3.3、编写autoinsatll.sh 文件

刷新资源

运行文件

格式化

 启动所有进程

---

Hadoop安装详细教程

1、使用xftp工具 在opt目录下创建install和soft文件

2、使用xftp工具 将压缩包上传到install文件

查看

[root@hadoop1 ~]# ls /opt/install/

 

3、编写shell脚本

3.1、创建目录来放shell脚本

 [root@hadoop3 opt]# mkdir shellfile

进入shellfile

[root@hadoop3 opt]# cd ./shellfile/
[root@hadoop3 shellfile]# pwd
/opt/shellfile

3.2、创建autoinsatll.sh文件并修改权限

[root@hadoop1 shellfile]# vim autoinsatll.sh

[root@hadoop1 shellfile]# chmod 777 ./autoinsatll.sh 

3.3、编写autoinsatll.sh 文件

#! /bin/bash
echo 'auto install hadoop begining....'#global var
hadoop=trueif [ "$hadoop" = true ];thenecho 'hadoop install set true'echo 'setup hadoop 3'tar -zxf /opt/install/hadoop-3.1.3.tar.gz -C /opt/soft/mv /opt/soft/hadoop-3.1.3 /opt/soft/hadoop313sed -i '78a\export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop' /etc/profilesed -i '78a\export HADOOP_LIBEXEC_DIR=$HADOOP_HOME/libexec' /etc/profilesed -i '78a\export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native' /etc/profilesed -i '78a\export HADOOP_INSTALL=$HADOOP_HOME' /etc/profilesed -i '78a\export HADOOP_YARN_HOME=$HADOOP_HOME' /etc/profilesed -i '78a\export HADOOP_HDFS_HOME=$HADOOP_HOME' /etc/profilesed -i '78a\export HADOOP_COMMON_HOME=$HADOOP_HOME' /etc/profilesed -i '78a\export HADOOP_MAPRED_HOME=$HADOOP_HOME' /etc/profilesed -i '78a\export YARN_NODEMANAGER_USER=root' /etc/profilesed -i '78a\export YARN_RESOURCEMANAGER_USER=root' /etc/profilesed -i '78a\export HDFS_ZKFC_USER=root' /etc/profilesed -i '78a\export HDFS_JOURNALNODE_USER=root' /etc/profilesed -i '78a\export HDFS_SECONDARYNAMENODE_USER=root' /etc/profilesed -i '78a\export HDFS_DATANODE_USER=root' /etc/profilesed -i '78a\export HDFS_NAMENODE_USER=root' /etc/profilesed -i '78a\export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$HADOOP_HOME/lib' /etc/profilesed -i '78a\export HADOOP_HOME=/opt/soft/hadoop313' /etc/profilesed -i '78a\#HADOOP_HOME' /etc/profile#core-sitesed -i '19a\  </property>' /opt/soft/hadoop313/etc/hadoop/core-site.xmlsed -i '19a\    <value>*</value>' /opt/soft/hadoop313/etc/hadoop/core-site.xmlsed -i '19a\    <name>hadoop.proxyuser.root.groups</name>' /opt/soft/hadoop313/etc/hadoop/core-site.xmlsed -i '19a\  <property>' /opt/soft/hadoop313/etc/hadoop/core-site.xmlsed -i '19a\  </property>' /opt/soft/hadoop313/etc/hadoop/core-site.xmlsed -i '19a\    <value>*</value>' /opt/soft/hadoop313/etc/hadoop/core-site.xmlsed -i '19a\    <name>hadoop.proxyuser.root.hosts</name>' /opt/soft/hadoop313/etc/hadoop/core-site.xmlsed -i '19a\  <property>' /opt/soft/hadoop313/etc/hadoop/core-site.xmlsed -i '19a\  </property>' /opt/soft/hadoop313/etc/hadoop/core-site.xmlsed -i '19a\    <description>读写队列缓存：128k</description>' /opt/soft/hadoop313/etc/hadoop/core-site.xmlsed -i '19a\    <value>131072</value>' /opt/soft/hadoop313/etc/hadoop/core-site.xmlsed -i '19a\    <name>io.file.buffer.size</name>' /opt/soft/hadoop313/etc/hadoop/core-site.xmlsed -i '19a\  <property>' /opt/soft/hadoop313/etc/hadoop/core-site.xmlsed -i '19a\  </property>' /opt/soft/hadoop313/etc/hadoop/core-site.xmlsed -i '19a\    <value>root</value>' /opt/soft/hadoop313/etc/hadoop/core-site.xmlsed -i '19a\    <name>hadoop.http.staticuser.user</name>' /opt/soft/hadoop313/etc/hadoop/core-site.xmlsed -i '19a\  <property>' /opt/soft/hadoop313/etc/hadoop/core-site.xmlsed -i '19a\  </property>' /opt/soft/hadoop313/etc/hadoop/core-site.xmlsed -i '19a\    <description>namenode上本地的hadoop临时文件夹</description>' /opt/soft/hadoop313/etc/hadoop/core-site.xmlsed -i '19a\    <value>/opt/soft/hadoop313/data</value>' /opt/soft/hadoop313/etc/hadoop/core-site.xmlsed -i '19a\    <name>hadoop.tmp.dir</name>' /opt/soft/hadoop313/etc/hadoop/core-site.xmlsed -i '19a\  <property>' /opt/soft/hadoop313/etc/hadoop/core-site.xmlsed -i '19a\  </property>' /opt/soft/hadoop313/etc/hadoop/core-site.xmlsed -i '19a\    <value>hdfs://hadoop3:9000</value>' /opt/soft/hadoop313/etc/hadoop/core-site.xmlsed -i '19a\    <name>fs.defaultFS</name>' /opt/soft/hadoop313/etc/hadoop/core-site.xmlsed -i '19a\  <property>' /opt/soft/hadoop313/etc/hadoop/core-site.xml#hadoop-env.shsed -i '53a\export JAVA_HOME=/opt/soft/jdk180'  /opt/soft/hadoop313/etc/hadoop/hadoop-env.sh#hdfs-site.xmlsed -i '19a\  </property>' /opt/soft/hadoop313/etc/hadoop/hdfs-site.xmlsed -i '19a\    <description>关闭权限验证</description>' /opt/soft/hadoop313/etc/hadoop/hdfs-site.xmlsed -i '19a\    <value>false</value>' /opt/soft/hadoop313/etc/hadoop/hdfs-site.xmlsed -i '19a\    <name>dfs.permissions.enabled</name>' /opt/soft/hadoop313/etc/hadoop/hdfs-site.xmlsed -i '19a\  <property>' /opt/soft/hadoop313/etc/hadoop/hdfs-site.xmlsed -i '19a\  </property>' /opt/soft/hadoop313/etc/hadoop/hdfs-site.xmlsed -i '19a\    <description>datanode上数据块的物理存储位置目录</description>' /opt/soft/hadoop313/etc/hadoop/hdfs-site.xmlsed -i '19a\    <value>/opt/soft/hadoop313/data/dfs/data</value>' /opt/soft/hadoop313/etc/hadoop/hdfs-site.xmlsed -i '19a\    <name>dfs.datanode.data.dir</name>' /opt/soft/hadoop313/etc/hadoop/hdfs-site.xmlsed -i '19a\  <property>' /opt/soft/hadoop313/etc/hadoop/hdfs-site.xmlsed -i '19a\  </property>' /opt/soft/hadoop313/etc/hadoop/hdfs-site.xmlsed -i '19a\    <description>namenode上存储hdfsq名字空间元数据的目录</description>' /opt/soft/hadoop313/etc/hadoop/hdfs-site.xmlsed -i '19a\    <value>/opt/soft/hadoop313/data/dfs/name</value>' /opt/soft/hadoop313/etc/hadoop/hdfs-site.xmlsed -i '19a\    <name>dfs.namenode.name.dir</name>' /opt/soft/hadoop313/etc/hadoop/hdfs-site.xmlsed -i '19a\  <property>' /opt/soft/hadoop313/etc/hadoop/hdfs-site.xmlsed -i '19a\  </property>' /opt/soft/hadoop313/etc/hadoop/hdfs-site.xmlsed -i '19a\    <description>hadoop中每一个block文件的备份数量</description>' /opt/soft/hadoop313/etc/hadoop/hdfs-site.xmlsed -i '19a\    <value>1</value>' /opt/soft/hadoop313/etc/hadoop/hdfs-site.xmlsed -i '19a\    <name>dfs.replication</name>' /opt/soft/hadoop313/etc/hadoop/hdfs-site.xmlsed -i '19a\  <property>' /opt/soft/hadoop313/etc/hadoop/hdfs-site.xml#mapred-site.xmlsed -i '19a\  </property>' /opt/soft/hadoop313/etc/hadoop/mapred-site.xmlsed -i '19a\    <value>1024</value>' /opt/soft/hadoop313/etc/hadoop/mapred-site.xmlsed -i '19a\    <name>mapreduce.reduce.memory.mb</name>' /opt/soft/hadoop313/etc/hadoop/mapred-site.xmlsed -i '19a\  <property>' /opt/soft/hadoop313/etc/hadoop/mapred-site.xmlsed -i '19a\  </property>' /opt/soft/hadoop313/etc/hadoop/mapred-site.xmlsed -i '19a\    <value>1024</value>' /opt/soft/hadoop313/etc/hadoop/mapred-site.xmlsed -i '19a\    <name>mapreduce.map.memory.mb</name>' /opt/soft/hadoop313/etc/hadoop/mapred-site.xmlsed -i '19a\  <property>' /opt/soft/hadoop313/etc/hadoop/mapred-site.xmlsed -i '19a\  </property>' /opt/soft/hadoop313/etc/hadoop/mapred-site.xmlsed -i '19a\    <value>hadoop3:19888</value>' /opt/soft/hadoop313/etc/hadoop/mapred-site.xmlsed -i '19a\    <name>mapreduce.jobhistory.webapp.address</name>' /opt/soft/hadoop313/etc/hadoop/mapred-site.xmlsed -i '19a\  <property>' /opt/soft/hadoop313/etc/hadoop/mapred-site.xmlsed -i '19a\  </property>' /opt/soft/hadoop313/etc/hadoop/mapred-site.xmlsed -i '19a\    <value>hadoop3:10020</value>' /opt/soft/hadoop313/etc/hadoop/mapred-site.xmlsed -i '19a\    <name>mapreduce.jobhistory.address</name>' /opt/soft/hadoop313/etc/hadoop/mapred-site.xmlsed -i '19a\  <property>' /opt/soft/hadoop313/etc/hadoop/mapred-site.xmlsed -i '19a\  </property>' /opt/soft/hadoop313/etc/hadoop/mapred-site.xmlsed -i '19a\    <value>/opt/soft/hadoop313/etc/hadoop:/opt/soft/hadoop313/share/hadoop/common/lib/*:/opt/soft/hadoop313/share/hadoop/common/*:/opt/soft/hadoop313/share/hadoop/hdfs/*:/opt/soft/hadoop313/share/hadoop/hdfs/lib/*:/opt/soft/hadoop313/share/hadoop/mapreduce/*:/opt/soft/hadoop313/share/hadoop/mapreduce/lib/*:/opt/soft/hadoop313/share/hadoop/yarn/*:/opt/soft/hadoop313/share/hadoop/yarn/lib/*</value>' /opt/soft/hadoop313/etc/hadoop/mapred-site.xmlsed -i '19a\   <name>mapreduce.application.classpath</name>' /opt/soft/hadoop313/etc/hadoop/mapred-site.xmlsed -i '19a\  <property>' /opt/soft/hadoop313/etc/hadoop/mapred-site.xmlsed -i '19a\  </property>' /opt/soft/hadoop313/etc/hadoop/mapred-site.xmlsed -i '19a\    <final>true</final>' /opt/soft/hadoop313/etc/hadoop/mapred-site.xmlsed -i '19a\    <description>job执行框架：local,classic or yarn</description>' /opt/soft/hadoop313/etc/hadoop/mapred-site.xmlsed -i '19a\    <value>yarn</value>' /opt/soft/hadoop313/etc/hadoop/mapred-site.xmlsed -i '19a\    <name>mapreduce.framework.name</name>' /opt/soft/hadoop313/etc/hadoop/mapred-site.xmlsed -i '19a\  <property>' /opt/soft/hadoop313/etc/hadoop/mapred-site.xml#yarn-site.xmlsed -i '17a\  </property>' /opt/soft/hadoop313/etc/hadoop/yarn-site.xmlsed -i '17a\    <value>/opt/soft/hadoop313/yarndata/log</value>' /opt/soft/hadoop313/etc/hadoop/yarn-site.xmlsed -i '17a\    <name>yarn.nodemanager.log-dirs</name>' /opt/soft/hadoop313/etc/hadoop/yarn-site.xmlsed -i '17a\  <property>' /opt/soft/hadoop313/etc/hadoop/yarn-site.xmlsed -i '17a\  </property>' /opt/soft/hadoop313/etc/hadoop/yarn-site.xmlsed -i '17a\    <value>/opt/soft/hadoop313/yarndata/yarn</value>' /opt/soft/hadoop313/etc/hadoop/yarn-site.xmlsed -i '17a\    <name>yarn.nodemanager.local-dirs</name>' /opt/soft/hadoop313/etc/hadoop/yarn-site.xmlsed -i '17a\  <property>' /opt/soft/hadoop313/etc/hadoop/yarn-site.xmlsed -i '17a\  </property>' /opt/soft/hadoop313/etc/hadoop/yarn-site.xmlsed -i '17a\    <value>mapreduce_shuffle</value>' /opt/soft/hadoop313/etc/hadoop/yarn-site.xmlsed -i '17a\    <name>yarn.nodemanager.aux-services</name>' /opt/soft/hadoop313/etc/hadoop/yarn-site.xmlsed -i '17a\  <property>' /opt/soft/hadoop313/etc/hadoop/yarn-site.xmlsed -i '17a\  </property>' /opt/soft/hadoop313/etc/hadoop/yarn-site.xmlsed -i '17a\    <value>hadoop3:8042</value>' /opt/soft/hadoop313/etc/hadoop/yarn-site.xmlsed -i '17a\    <name>yarn.nodemanager.webapp.address</name>' /opt/soft/hadoop313/etc/hadoop/yarn-site.xmlsed -i '17a\  <property>' /opt/soft/hadoop313/etc/hadoop/yarn-site.xmlsed -i '17a\  </property>' /opt/soft/hadoop313/etc/hadoop/yarn-site.xmlsed -i '17a\    <value>hadoop3:8050</value>' /opt/soft/hadoop313/etc/hadoop/yarn-site.xmlsed -i '17a\    <name>yarn.nodemanager.address</name>' /opt/soft/hadoop313/etc/hadoop/yarn-site.xmlsed -i '17a\  <property>' /opt/soft/hadoop313/etc/hadoop/yarn-site.xmlsed -i '17a\  </property>' /opt/soft/hadoop313/etc/hadoop/yarn-site.xmlsed -i '17a\    <value>hadoop3:8040</value>' /opt/soft/hadoop313/etc/hadoop/yarn-site.xmlsed -i '17a\    <name>yarn.nodemanager.localizer.address</name>' /opt/soft/hadoop313/etc/hadoop/yarn-site.xmlsed -i '17a\  <property>' /opt/soft/hadoop313/etc/hadoop/yarn-site.xmlsed -i '17a\  </property>' /opt/soft/hadoop313/etc/hadoop/yarn-site.xmlsed -i '17a\    <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler</value>' /opt/soft/hadoop313/etc/hadoop/yarn-site.xmlsed -i '17a\    <name>yarn.resourcemanager.scheduler.class</name>' /opt/soft/hadoop313/etc/hadoop/yarn-site.xmlsed -i '17a\  <property>' /opt/soft/hadoop313/etc/hadoop/yarn-site.xmlsed -i '17a\  </property>' /opt/soft/hadoop313/etc/hadoop/yarn-site.xmlsed -i '17a\    <value>20000</value>' /opt/soft/hadoop313/etc/hadoop/yarn-site.xmlsed -i '17a\    <name>yarn.resourcemanager.connect.retry-interval.ms</name>' /opt/soft/hadoop313/etc/hadoop/yarn-site.xmlsed -i '17a\  <property>' /opt/soft/hadoop313/etc/hadoop/yarn-site.xmlecho 'setup hadoop 3 success!!!'
fi

刷新资源

[root@hadoop3 shellfile]# source /etc/profile

运行文件

[root@hadoop3 shellfile]# ./hadoopautoinstall.sh

格式化

[root@hadoop3 shellfile]# hdfs namenode -format 

 启动所有进程

[root@hadoop3 shellfile]# start-all.sh

查看启动情况

[root@hadoop3 shellfile]# jps 

 出来上面六个则OK