argo系统内存升级操作

高俊 1年前
476次浏览 0人关注 复制链接 所属标签: Argo 升级内存

考虑一种场景,我刚开始在一个32G的低配云主机上安装了argo,使用一段时间之后,我想将机器升级到64g内存,这个时候,我该怎么做呢?

argo的installer在安装时会根据您选择的内存大小,自动进行集群参数的优化,如果您刚开始安装在32g的机器上,现在想要升级机器内存,那您需要自己调整集群各个服务组件的内存参数。下面给出所有相关的配置文件名,参数和该参数在不同内存机器上的值。您需求对比该列表进行修改,然后重启所有argo的服务。

配置文件 配置项 16G机器 32G机器 64G机器 128G机器
hadoop-env namenode_heapsize 1024 3072 8192 8192
hadoop-env nfsgateway_heapsize 128 256 256 256
hadoop-env namenode_opt_newsize 256 384 384 384
hadoop-env dtnode_heapsize 1024 1024 1536 2048
hdfs-site dfs.namenode.handler.count 100 200 400 400
yarn-env yarn_heapsize 256 512 512 512
yarn-env resourcemanager_heapsize 1024 1024 1536 2048
yarn-env nodemanager_heapsize 1024 1024 1536 2048
yarn-env apptimelineserver_heapsize 512 512 1536 512
yarn-site yarn.scheduler.minimum-allocation-mb 512 512 512 512
yarn-site yarn.scheduler.maximum-allocation-mb 4096 6144 16384 32768
yarn-site yarn.nodemanager.resource.memory-mb 4096 6144 18432 32768
yarn-site yarn.nodemanager.resource.cpu-vcores 4 6 16 24
yarn-site yarn.scheduler.maximum-allocation-vcores 4 6 16 24
mapred-site jobhistory_heapsize 1024 1024 1536 2048
mapred-site mapreduce.map.memory.mb 2048 2048 2048 2048
mapred-site mapreduce.reduce.memory.mb 2048 2048 2048 2048
mapred-site yarn.app.mapreduce.am.resource.mb 1024 1024 1024 1024
mapred-site yarn.app.mapreduce.am.command-opts -Xmx819m -Dhdp.version=${hdp.version} -Xmx819m -Dhdp.version=${hdp.version} -Xmx819m -Dhdp.version=${hdp.version} -Xmx819m -Dhdp.version=${hdp.version}
mapred-site mapreduce.map.java.opts -Xmx1638m -Xmx1638m -Xmx1638m -Xmx1638m
mapred-site mapreduce.task.io.sort.mb 1146 1146 1146 1146
mapred-site mapreduce.reduce.java.opts -Xmx1638m -Xmx1638m -Xmx1638m -Xmx1638m
hive-site hive.tez.container.size 512 512 512 512
hive-site hive.heapsize 512 512 512 2048
hive-site hive.auto.convert.join.noconditionaltask.size 73165576 143165576 143165576 143165576
hive-env hive.client.heapsize 256 512 512 512
hive-env hive.metastore.heapsize 512 512 512 2048
ams-hbase-env hbase_master_heapsize 512 512 512 1024
ams-hbase-env hbase_master_xmn_size 100 100 100 1024
ams-hbase-env hbase_regionserver_heapsize 1024 1024 1024 2048
ams-hbase-env regionserver_xmn_size 200 200 200 400
hbase-env hbase_master_heapsize 512 1024 1024 2048
hbase-env hbase_regionserver_heapsize 2048 4096 8192 16384
zookeeper-env zk_server_heapsize 1024 2048 2048 3072
jvm.config jvm.config -server\r\n-Xmx2G\r\n-verbose:gc\r\n-Xloggc:/opt/soft/log/presto/gc_presto.log\r\n-XX:+UseG1GC\r\n-XX:+UseGCOverheadLimit\r\n-XX:+ExplicitGCInvokesConcurrent\r\n-XX:+HeapDumpOnOutOfMemoryError\r\n-XX:OnOutOfMemoryError=kill -9 %p\r\n-XX:+PrintGCApplicationConcurrentTime\r\n-XX:+PrintGCApplicationStoppedTime\r\n-XX:+PrintGCCause\r\n-XX:+PrintGCDateStamps\r\n-XX:+PrintGCTimeStamps\r\n-XX:+PrintGCDetails\r\n-XX:+PrintReferenceGC\r\n-XX:+PrintClassHistogramAfterFullGC\r\n-XX:+PrintClassHistogramBeforeFullGC\r\n-XX:PrintFLSStatistics=2\r\n-XX:+PrintAdaptiveSizePolicy\r\n-XX:+PrintSafepointStatistics\r\n-XX:PrintSafepointStatisticsCount=1 -server\r\n-Xmx4G\r\n-verbose:gc\r\n-Xloggc:/var/log/presto/gc_presto.log\r\n-XX:+UseG1GC\r\n-XX:+UseGCOverheadLimit\r\n-XX:+ExplicitGCInvokesConcurrent\r\n-XX:+HeapDumpOnOutOfMemoryError\r\n-XX:OnOutOfMemoryError=kill -9 %p\r\n-XX:+PrintGCApplicationConcurrentTime\r\n-XX:+PrintGCApplicationStoppedTime\r\n-XX:+PrintGCCause\r\n-XX:+PrintGCDateStamps\r\n-XX:+PrintGCTimeStamps\r\n-XX:+PrintGCDetails\r\n-XX:+PrintReferenceGC\r\n-XX:+PrintClassHistogramAfterFullGC\r\n-XX:+PrintClassHistogramBeforeFullGC\r\n-XX:PrintFLSStatistics=2\r\n-XX:+PrintAdaptiveSizePolicy\r\n-XX:+PrintSafepointStatistics\r\n-XX:PrintSafepointStatisticsCount=1 -server\r\n-Xmx16G\r\n-verbose:gc\r\n-Xloggc:/var/log/presto/gc_presto.log\r\n-XX:+UseG1GC\r\n-XX:+UseGCOverheadLimit\r\n-XX:+ExplicitGCInvokesConcurrent\r\n-XX:+HeapDumpOnOutOfMemoryError\r\n-XX:OnOutOfMemoryError=kill -9 %p\r\n-XX:+PrintGCApplicationConcurrentTime\r\n-XX:+PrintGCApplicationStoppedTime\r\n-XX:+PrintGCCause\r\n-XX:+PrintGCDateStamps\r\n-XX:+PrintGCTimeStamps\r\n-XX:+PrintGCDetails\r\n-XX:+PrintReferenceGC\r\n-XX:+PrintClassHistogramAfterFullGC\r\n-XX:+PrintClassHistogramBeforeFullGC\r\n-XX:PrintFLSStatistics=2\r\n-XX:+PrintAdaptiveSizePolicy\r\n-XX:+PrintSafepointStatistics\r\n-XX:PrintSafepointStatisticsCount=1 -server\r\n-Xmx32G\r\n-verbose:gc\r\n-Xloggc:/var/log/presto/gc_presto.log\r\n-XX:+UseG1GC\r\n-XX:+UseGCOverheadLimit\r\n-XX:+ExplicitGCInvokesConcurrent\r\n-XX:+HeapDumpOnOutOfMemoryError\r\n-XX:OnOutOfMemoryError=kill -9 %p\r\n-XX:+PrintGCApplicationConcurrentTime\r\n-XX:+PrintGCApplicationStoppedTime\r\n-XX:+PrintGCCause\r\n-XX:+PrintGCDateStamps\r\n-XX:+PrintGCTimeStamps\r\n-XX:+PrintGCDetails\r\n-XX:+PrintReferenceGC\r\n-XX:+PrintClassHistogramAfterFullGC\r\n-XX:+PrintClassHistogramBeforeFullGC\r\n-XX:PrintFLSStatistics=2\r\n-XX:+PrintAdaptiveSizePolicy\r\n-XX:+PrintSafepointStatistics\r\n-XX:PrintSafepointStatisticsCount=1
config.properties query.max-memory-per-node 1 2 8 15
config.properties query.max-total-memory-per-node 1 2 8 15

按上面的修改各个服务组件的参数之后,按下图先stop all ,再start all即可

暂无回答