Hadoop 3 节点集群部署从规划到启动的 7 个核心配置文件详解1. 生产级Hadoop集群规划与角色分配在构建Hadoop完全分布式集群时合理的角色规划直接影响集群性能和稳定性。我们采用1个NameNode、1个ResourceManager和3个DataNode/NodeManager的架构设计这是中小规模生产环境中最常见的配置方案。节点角色分配表主机名HDFS角色YARN角色推荐硬件配置hadoop-mnNameNode-16核CPU, 32GB内存hadoop-rmSecondaryNameNodeResourceManager16核CPU, 32GB内存hadoop-dn1DataNodeNodeManager8核CPU, 64GB内存hadoop-dn2DataNodeNodeManager8核CPU, 64GB内存hadoop-dn3DataNodeNodeManager8核CPU, 64GB内存注意NameNode和ResourceManager应部署在不同节点避免资源竞争。SecondaryNameNode建议单独部署不与NameNode同机。关键配置前检查清单所有节点已配置SSH免密登录确保/etc/hosts文件包含所有节点IP和主机名映射各节点时间同步建议配置NTP服务关闭所有节点的防火墙或开放必要端口Java环境统一推荐JDK8或JDK112. 核心配置文件解析与优化2.1 core-site.xml集群基础配置configuration !-- 定义默认文件系统URI -- property namefs.defaultFS/name valuehdfs://hadoop-mn:8020/value /property !-- 临时文件存储目录 -- property namehadoop.tmp.dir/name value/opt/hadoop/tmp/value description建议使用独立磁盘分区至少保留50GB空间/description /property !-- 启用HA时需要配置的命名空间 -- property namefs.hdfs.impl.disable.cache/name valuetrue/value /property !-- 配置HTTP静态用户 -- property namehadoop.http.staticuser.user/name valueadmin/value /property /configuration关键参数调优建议fs.trash.interval设置垃圾回收机制建议1440分钟ipc.client.connect.max.retries网络异常时连接重试次数生产环境建议30ha.zookeeper.quorum如果启用HA需配置ZooKeeper集群地址2.2 hdfs-site.xml分布式文件系统配置configuration !-- 数据块副本数 -- property namedfs.replication/name value3/value /property !-- NameNode元数据存储位置 -- property namedfs.namenode.name.dir/name valuefile:///opt/hadoop/dfs/name/value /property !-- DataNode数据存储位置 -- property namedfs.datanode.data.dir/name valuefile:///opt/hadoop/dfs/data/value /property !-- NameNode HTTP访问地址 -- property namedfs.namenode.http-address/name valuehadoop-mn:9870/value /property !-- SecondaryNameNode检查点设置 -- property namedfs.namenode.checkpoint.period/name value3600/value /property /configuration生产环境关键配置dfs.blocksize根据文件大小调整默认128MB大文件可设为256MBdfs.datanode.failed.volumes.tolerated允许失败的磁盘数dfs.namenode.handler.countNameNode并发线程数建议设为集群节点数的10倍2.3 yarn-site.xml资源管理配置configuration !-- 指定ResourceManager地址 -- property nameyarn.resourcemanager.hostname/name valuehadoop-rm/value /property !-- NodeManager上运行的附属服务 -- property nameyarn.nodemanager.aux-services/name valuemapreduce_shuffle/value /property !-- 每个容器最小内存 -- property nameyarn.scheduler.minimum-allocation-mb/name value1024/value /property !-- 单个容器最大内存 -- property nameyarn.scheduler.maximum-allocation-mb/name value16384/value /property !-- NodeManager可用物理内存 -- property nameyarn.nodemanager.resource.memory-mb/name value57344/value /property /configuration内存分配计算公式单个NodeManager可用内存 物理内存 - 系统保留(4GB) - DataNode内存(4GB) 容器内存 min(单个NodeManager可用内存, yarn.scheduler.maximum-allocation-mb)2.4 mapred-site.xmlMapReduce框架配置configuration !-- 指定MapReduce运行框架 -- property namemapreduce.framework.name/name valueyarn/value /property !-- Map任务内存设置 -- property namemapreduce.map.memory.mb/name value2048/value /property !-- Reduce任务内存设置 -- property namemapreduce.reduce.memory.mb/name value4096/value /property !-- Map任务JVM参数 -- property namemapreduce.map.java.opts/name value-Xmx1800m/value /property !-- 启用任务历史服务器 -- property namemapreduce.jobhistory.address/name valuehadoop-rm:10020/value /property /configuration2.5 workers文件DataNode节点列表hadoop-dn1 hadoop-dn2 hadoop-dn3重要文件末尾不能有空行每行结尾不能有空格2.6 hadoop-env.sh环境变量配置export JAVA_HOME/usr/lib/jvm/java-11-openjdk export HADOOP_HOME/opt/hadoop export HADOOP_LOG_DIR/var/log/hadoop export HADOOP_HEAPSIZE_MAX8g export HADOOP_NAMENODE_OPTS-XX:UseG1GC -Xmx16g export HADOOP_DATANODE_OPTS-Xmx4g2.7 yarn-env.shYARN环境配置export YARN_RESOURCEMANAGER_HEAPSIZE8g export YARN_NODEMANAGER_HEAPSIZE4g export YARN_PROXYSERVER_HEAPSIZE2g export HADOOP_JAVA_HOME/usr/lib/jvm/java-11-openjdk3. 集群启动与验证流程3.1 首次启动前的关键操作格式化NameNode仅在首次启动时执行hdfs namenode -format -clusterId hadoop-cluster同步配置文件到所有节点scp -r /opt/hadoop/etc/hadoop/* hadoop-dn1:/opt/hadoop/etc/hadoop/ scp -r /opt/hadoop/etc/hadoop/* hadoop-dn2:/opt/hadoop/etc/hadoop/ scp -r /opt/hadoop/etc/hadoop/* hadoop-dn3:/opt/hadoop/etc/hadoop/3.2 分步启动集群服务启动HDFS在NameNode节点执行start-dfs.sh启动YARN在ResourceManager节点执行start-yarn.sh启动历史服务器可选mapred --daemon start historyserver3.3 集群健康检查进程验证# NameNode节点应包含 # NameNode # SecondaryNameNode # DataNode节点应包含 # DataNode # NodeManager # ResourceManager节点应包含 # ResourceManagerWeb UI访问HDFS NameNode: http://hadoop-mn:9870YARN ResourceManager: http://hadoop-rm:8088MapReduce History Server: http://hadoop-rm:19888命令行验证# 检查HDFS状态 hdfs dfsadmin -report # 创建测试目录 hdfs dfs -mkdir /test hdfs dfs -put /etc/hosts /test # 运行WordCount测试 hadoop jar $HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-examples-*.jar wordcount /test/hosts /test/output4. 生产环境优化与维护4.1 性能调优参数HDFS调优!-- hdfs-site.xml -- property namedfs.datanode.handler.count/name value20/value /property property namedfs.namenode.handler.count/name value40/value /propertyYARN调优!-- yarn-site.xml -- property nameyarn.nodemanager.resource.cpu-vcores/name value16/value /property property nameyarn.scheduler.capacity.root.queues/name valuedefault,prod/value /property4.2 监控与日志管理关键监控指标HDFS存储利用率丢失的块数量活跃的DataNode数量YARN可用资源运行中的应用程序日志收集建议# 配置日志聚合 property nameyarn.log-aggregation-enable/name valuetrue/value /property property nameyarn.nodemanager.remote-app-log-dir/name value/tmp/logs/value /property4.3 常见故障处理NameNode无法启动检查端口冲突8020, 9870验证元数据目录权限查看日志文件中的具体错误DataNode不注册检查网络连通性验证workers文件配置检查DataNode磁盘空间资源分配问题# 查看YARN节点资源 yarn node -list -all # 检查资源预留 yarn scheduler -listReservations5. 安全配置与权限管理5.1 Kerberos集成基础配置!-- core-site.xml -- property namehadoop.security.authentication/name valuekerberos/value /property property namehadoop.security.authorization/name valuetrue/value /property5.2 HDFS权限控制# 目录权限设置示例 hdfs dfs -chmod -R 750 /data hdfs dfs -chown -R admin:hadoop /data # 启用ACL访问控制列表 property namedfs.namenode.acls.enabled/name valuetrue/value /property5.3 YARN队列权限!-- capacity-scheduler.xml -- property nameyarn.scheduler.capacity.root.prod.acl_submit_applications/name valueprod_users/value /property property nameyarn.scheduler.capacity.root.prod.acl_administer_queue/name valueprod_admins/value /property6. 备份与恢复策略6.1 NameNode元数据备份# 定期备份fsimage hdfs dfsadmin -fetchImage /backup/nn_image # 配置SecondaryNameNode检查点 property namedfs.namenode.checkpoint.period/name value3600/value /property6.2 数据备份方案DistCp跨集群复制hadoop distcp hdfs://nn1:8020/data hdfs://nn2:8020/backup/dataHDFS快照功能hdfs dfsadmin -allowSnapshot /data hdfs dfs -createSnapshot /data backup_2024056.3 灾难恢复步骤停止所有Hadoop服务恢复最新的fsimage和edits文件格式化新的NameNode启动NameNode进入安全模式执行元数据恢复退出安全模式并启动集群7. 自动化部署与运维7.1 使用Ansible部署inventory文件示例[namenode] hadoop-mn [resourcemanager] hadoop-rm [datanodes] hadoop-dn1 hadoop-dn2 hadoop-dn3playbook片段- name: 部署Hadoop配置文件 template: src: {{ item }}.j2 dest: /opt/hadoop/etc/hadoop/{{ item }} loop: - core-site.xml - hdfs-site.xml - yarn-site.xml - mapred-site.xml7.2 监控集成Prometheus监控配置scrape_configs: - job_name: hadoop static_configs: - targets: [hadoop-mn:9870, hadoop-rm:8088] metrics_path: /jmx7.3 日志分析架构Filebeat - Logstash - Elasticsearch ↓ Grafana Dashboard7.4 滚动升级流程备份所有配置和关键数据逐个节点停止服务并升级验证新版本兼容性滚动重启集群全面功能测试