ambari-server HA

文章目录一、前言二、环境三、mysql主从配置四、同步配置文件五、实现脚本一、前言本篇文章主要讲解Ambari Server端的高可用搭建。注意，是Ambari的Server，而不是Hadoop集群的应用。截止目前为止（Ambari 2.7.x），hortonworks官方并没有给出AmbariServer的高可用的内部实现。二、环境这里我选用的纯离线安装方式，原因主要是受网络限制需要下载的依赖包

jzy3711

281545人浏览 · 2020-05-26 18:24:22

jzy3711 · 2020-05-26 18:24:22 发布

一、前言

本篇文章主要讲解Ambari Server端的高可用搭建。注意，是Ambari的Server，而不是Hadoop集群的应用。截止目前为止（Ambari 2.7.x），hortonworks官方并没有给出AmbariServer的高可用的内部实现。

二、方案和思路

1.原理

ambari 元数据存储在mysql （可在setup时自定义），所以要做mysql主从复制
ambari-server HA 就需要2个amabri-server 配置基本一致，只是一个用的是mysql主，一个用的是mysql从
原理图

2.mysql主从复制

我这里使用一主一从即可

主从的作用

实时灾备：一台主数据库宕机了，启用从数据库，用于故障切换
读写分离：主服务器可以只用于写操作，从服务器只用于读取，用于查询服务
备份：这个问题可以很好的解决数据丢失的问题，避免影响业务

主从的形式

一主多从表示只有一台主服务器，多台从服务器
主主复制表示互为主服务器，同时也互为从服务器
一主多从 ----扩展系统读取的性能，因为读是在从库读取的
多主一从 ----5.7开始支持
联级复制

主从复制原理

mysql主从复制
用户将写入的数据保存到mysql主服务器（master）上，主库将所有写的操作记录到binlog 日志中，并且生成一个log dump的线程，将 binlog 日志传给从库的I/O线程，在从服务器（slave）上生成两个线程，一个I/O线程，一个SQL线程，I/O线程去请求主库的binlog，并将得到的binlog 日志写到relay log（中继日志）文件中，然后SQL线程，会读取relay log文件的日志，并解析成具体的操作，来实现主从的操作一致，达到最终数据一致的目的

3.配置文件

/etc/ambari-server/conf/
password.dat #存放mysql密码
ambari.properties #ambari-server配置文件

#
# Copyright 2011 The Apache Software Foundation
#
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements.  See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership.  The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License.  You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

#
#Wed May 20 15:47:56 CST 2020
ulimit.open.files=65536
views.http.x-content-type-options=nosniff
server.persistence.type=remote
jdk1.8.jcpol-url=http://public-repo-1.hortonworks.com/ARTIFACTS/jce_policy-8.zip
java.releases.ppc64le=
recommendations.artifacts.lifetime=1w
http.pragma=no-cache
jdk1.8.url=http://public-repo-1.hortonworks.com/ARTIFACTS/jdk-8u112-linux-x64.tar.gz
server.execution.scheduler.misfire.toleration.minutes=480
java.home=/opt/jdk/jdk1.8.0_201
security.server.disabled.ciphers=TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384|TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384|TLS_RSA_WITH_AES_256_CBC_SHA256|TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384|TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384|TLS_DHE_RSA_WITH_AES_256_CBC_SHA256|TLS_DHE_DSS_WITH_AES_256_CBC_SHA256|TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA|TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA|TLS_RSA_WITH_AES_256_CBC_SHA|TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA|TLS_ECDH_RSA_WITH_AES_256_CBC_SHA|TLS_DHE_RSA_WITH_AES_256_CBC_SHA|TLS_DHE_DSS_WITH_AES_256_CBC_SHA|TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256|TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256|TLS_RSA_WITH_AES_128_CBC_SHA256|TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA256|TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256|TLS_DHE_RSA_WITH_AES_128_CBC_SHA256|TLS_DHE_DSS_WITH_AES_128_CBC_SHA256|TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA|TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA|TLS_RSA_WITH_AES_128_CBC_SHA|TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA|TLS_ECDH_RSA_WITH_AES_128_CBC_SHA|TLS_DHE_RSA_WITH_AES_128_CBC_SHA|TLS_DHE_DSS_WITH_AES_128_CBC_SHA|TLS_ECDHE_ECDSA_WITH_3DES_EDE_CBC_SHA|TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA|TLS_ECDH_ECDSA_WITH_3DES_EDE_CBC_SHA|TLS_ECDH_RSA_WITH_3DES_EDE_CBC_SHA|SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA|SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA|TLS_EMPTY_RENEGOTIATION_INFO_SCSV|TLS_DH_anon_WITH_AES_256_CBC_SHA256|TLS_ECDH_anon_WITH_AES_256_CBC_SHA|TLS_DH_anon_WITH_AES_256_CBC_SHA|TLS_DH_anon_WITH_AES_128_CBC_SHA256|TLS_ECDH_anon_WITH_AES_128_CBC_SHA|TLS_DH_anon_WITH_AES_128_CBC_SHA|TLS_ECDH_anon_WITH_3DES_EDE_CBC_SHA|SSL_DH_anon_WITH_3DES_EDE_CBC_SHA|SSL_RSA_WITH_DES_CBC_SHA|SSL_DHE_RSA_WITH_DES_CBC_SHA|SSL_DHE_DSS_WITH_DES_CBC_SHA|SSL_DH_anon_WITH_DES_CBC_SHA|SSL_RSA_EXPORT_WITH_DES40_CBC_SHA|SSL_DHE_RSA_EXPORT_WITH_DES40_CBC_SHA|SSL_DHE_DSS_EXPORT_WITH_DES40_CBC_SHA|SSL_DH_anon_EXPORT_WITH_DES40_CBC_SHA|TLS_RSA_WITH_NULL_SHA256|TLS_ECDHE_ECDSA_WITH_NULL_SHA|TLS_ECDHE_RSA_WITH_NULL_SHA|SSL_RSA_WITH_NULL_SHA|TLS_ECDH_ECDSA_WITH_NULL_SHA|TLS_ECDH_RSA_WITH_NULL_SHA|TLS_ECDH_anon_WITH_NULL_SHA|SSL_RSA_WITH_NULL_MD5|TLS_KRB5_WITH_3DES_EDE_CBC_SHA|TLS_KRB5_WITH_3DES_EDE_CBC_MD5|TLS_KRB5_WITH_DES_CBC_SHA|TLS_KRB5_WITH_DES_CBC_MD5|TLS_KRB5_EXPORT_WITH_DES_CBC_40_SHA|TLS_KRB5_EXPORT_WITH_DES_CBC_40_MD5
server.jdbc.hostname=<ambari-server-mysql1/2>
shared.resources.dir=/usr/lib/ambari-server/lib/ambari_commons/resources
server.jdbc.connection-pool.max-idle-time=14400
resources.dir=/var/lib/ambari-server/resources
custom.action.definitions=/var/lib/ambari-server/resources/custom_action_definitions
views.request.connect.timeout.millis=5000
jdk1.7.desc=Oracle JDK 1.7 + Java Cryptography Extension (JCE) Policy Files 7
server.jdbc.driver=com.mysql.jdbc.Driver
security.server.keys_dir=/var/lib/ambari-server/keys
server.jdbc.rca.user.name=ambari
webapp.dir=/usr/lib/ambari-server/web
views.http.pragma=no-cache
server.os_family=redhat7
server.jdbc.user.passwd=/etc/ambari-server/conf/password.dat
server.execution.scheduler.isClustered=false
views.ambari.request.connect.timeout.millis=30000
server.jdbc.database=mysql
server.jdbc.connection-pool=c3p0
server.jdbc.database_name=ambari
server.jdbc.rca.url=jdbc:mysql://<ambari-server-mysql1/2>:3306/ambari
bootstrap.script=/usr/lib/python2.6/site-packages/ambari_server/bootstrap.py
server.version.file=/var/lib/ambari-server/resources/version
jdk1.8.dest-file=jdk-8u112-linux-x64.tar.gz
server.task.timeout=1200
user.inactivity.timeout.role.readonly.default=0
server.jdbc.connection-pool.max-age=0
java.releases=jdk1.8,jdk1.7
recommendations.dir=/var/run/ambari-server/stack-recommendations
agent.stack.retry.tries=5
server.os_type=centos7
server.python.log.level=INFO
server.execution.scheduler.maxDbConnections=5
views.ambari.request.read.timeout.millis=45000
views.http.cache-control=no-store
rolling.upgrade.skip.packages.prefixes=
jdk1.8.home=/usr/jdk64/
jdk1.7.home=/usr/jdk64/
agent.task.timeout=900
bootstrap.setup_agent.script=/usr/lib/python2.6/site-packages/ambari_server/setupAgent.py
server.jdbc.rca.driver=com.mysql.jdbc.Driver
jdk1.7.dest-file=jdk-7u67-linux-x64.tar.gz
agent.package.install.task.timeout=1800
server.jdbc.port=3306
http.strict-transport-security=max-age=31536000
common.services.path=/var/lib/ambari-server/resources/common-services
agent.threadpool.size.max=25
ambari.python.wrap=ambari-python-wrap
skip.service.checks=false
server.jdbc.connection-pool.idle-test-interval=7200
ambari-server.user=root
jdk1.8.desc=Oracle JDK 1.8 + Java Cryptography Extension (JCE) Policy Files 8
views.http.strict-transport-security=max-age=31536000
http.x-content-type-options=nosniff
jdk1.7.re=(jdk.*)/jre
metadata.path=/var/lib/ambari-server/resources/stacks
jdk1.8.jcpol-file=jce_policy-8.zip
views.skip.home-directory-check.file-system.list=wasb,adls,adl
server.python.log.name=ambari-server-command.log
stackadvisor.script=/var/lib/ambari-server/resources/scripts/stack_advisor.py
http.x-xss-protection=1; mode=block
bootstrap.dir=/var/run/ambari-server/bootstrap
server.connection.max.idle.millis=900000
server.jdbc.rca.user.passwd=/etc/ambari-server/conf/password.dat
views.http.x-frame-options=SAMEORIGIN
server.jdbc.connection-pool.acquisition-size=5
http.x-frame-options=DENY
jce.download.supported=true
jdk.download.supported=true
jdk1.7.url=http://public-repo-1.hortonworks.com/ARTIFACTS/jdk-7u67-linux-x64.tar.gz
server.jdbc.user.name=ambari
mpacks.staging.path=/var/lib/ambari-server/resources/mpacks
pid.dir=/var/run/ambari-server
user.inactivity.timeout.default=0
agent.stack.retry.on_repo_unavailability=false
extensions.path=/var/lib/ambari-server/resources/extensions
views.request.read.timeout.millis=10000
jdk1.7.jcpol-file=UnlimitedJCEPolicyJDK7.zip
server.tmp.dir=/var/lib/ambari-server/data/tmp
server.execution.scheduler.maxThreads=5
server.jdbc.url=jdbc:mysql://<ambari-server-mysql1/2>:3306/ambari
server.fqdn.service.url=http://169.254.169.254/latest/meta-data/public-hostname
views.http.x-xss-protection=1; mode=block
server.http.session.inactive_timeout=1800
server.stages.parallel=true
kerberos.keytab.cache.dir=/var/lib/ambari-server/data/cache
jdk1.8.re=(jdk.*)/jre
http.cache-control=no-store
client.threadpool.size.max=25
jdk1.7.jcpol-url=http://public-repo-1.hortonworks.com/ARTIFACTS/UnlimitedJCEPolicyJDK7.zip
server.jdbc.connection-pool.max-idle-time-excess=0

/etc/ambari-agent/conf/ambari-agent.ini

# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific

[server]
hostname=<ambari-server1/2>
url_port=8440
secured_url_port=8441
connect_retry_delay=10
max_reconnect_retry_delay=30


[agent]
logdir=/var/log/ambari-agent
piddir=/var/run/ambari-agent
prefix=/var/lib/ambari-agent/data
;loglevel=(DEBUG/INFO)
loglevel=INFO
data_cleanup_interval=86400
data_cleanup_max_age=2592000
data_cleanup_max_size_MB = 100
ping_port=8670
cache_dir=/var/lib/ambari-agent/cache
tolerate_download_failures=true
run_as_user=root
parallel_execution=0
alert_grace_period=5
status_command_timeout=5
alert_kinit_timeout=14400000
system_resource_overrides=/etc/resource_overrides
; memory_threshold_soft_mb=400
; memory_threshold_hard_mb=1000
; ignore_mount_points=/mnt/custom1,/mnt/custom2

[security]
keysdir=/var/lib/ambari-agent/keys
server_crt=ca.crt
passphrase_env_var_name=AMBARI_PASSPHRASE
ssl_verify_cert=0
credential_lib_dir=/var/lib/ambari-agent/cred/lib
credential_conf_dir=/var/lib/ambari-agent/cred/conf
credential_shell_cmd=org.apache.hadoop.security.alias.CredentialShell
force_https_protocol=PROTOCOL_TLSv1_2

[network]
; this option apply only for Agent communication
use_system_proxy_settings=true

[services]
pidLookupPath=/var/run/

[heartbeat]
state_interval_seconds=60
dirs=/etc/hadoop,/etc/hadoop/conf,/etc/hbase,/etc/hcatalog,/etc/hive,/etc/oozie,
  /etc/sqoop,
  /var/run/hadoop,/var/run/zookeeper,/var/run/hbase,/var/run/templeton,/var/run/oozie,
  /var/log/hadoop,/var/log/zookeeper,/var/log/hbase,/var/run/templeton,/var/log/hive
; 0 - unlimited
log_lines_count=300
idle_interval_min=1
idle_interval_max=10


[logging]

4.更换ambari-server

首先要监控ambari-server1是否正常，如果不正常才进行更换

监控ambari-server1

curl -o /dev/null -s -m 10 --connect-timeout 10 -w %{http_code} http://<ambari-server1>:8080

更换ambari-server

在amabri-server2启动amabri-server

ambari-server start

5.重启amabri-agaent

修改配置

ambari-agent 向ambari-server提交心跳
更换ambari-agent 配置中的amabri-server IP

sed -i 's/<ambari-server1>/<ambari-server2>/g' /etc/ambari-agent/conf/ambari-agent.ini

重启ambari-agent

ambari-agent restart

三、环境

这里我选用的纯离线安装方式，原因主要是受网络限制
需要下载的依赖包：
ambari-2.5.0.3-centos7.tar.gz
HDP-2.6.5.0-centos7-rpm.tar.gz
HDP-UTILS-1.1.0.21-centos7.tar.gz
ansible-2.4-rpms.el7.tar.gz（包含安装ansible各种依赖）
mysql-5.7.27-el7-x86_64.tar.gz
mysql-connector-java-5.1.47.jar
jdk-8u201-linux-x64.tar.gz

四、mysql主从配置

参考官网
具体可参考1 2

设置从库只读

set global read_only=1;#针对普通MySQL数据库用户设置为只读
set global super_read_only=1;#针对super类MySQL数据库用户设置为只读，比如root用户
flush tables with read lock;#设定全局锁,如果只是需要只读，并不需要加锁
show global variables like "%read_only%";#查询全局变量表数据情况

我这里使用的是root 只执行第2条即可，最好都执行一下，第三条关闭客户端就失效了，可不执行。

五、同步配置文件

scp /etc/ambari-server/conf/password.dat root@<ambari-server2>:/etc/ambari-server/conf/
scp ambari.properties root@<ambari-server2>:/etc/ambari-server/conf

六、实现脚本

server_status_code="curl -o /dev/null -s -m 10 --connect-timeout 10 -w %{http_code} http://<ambari-server1>:8080"
if [ "$server_status_code" != "200" ]; then
     mysql -uroot -p123456 -h127.0.0.1 -e"set global super_read_only=0;"
     ambari-server start
     ALL_CLIENTS=`cat /etc/hosts| grep "10.20.10" | awk '{print $2}'` 
     for client in $ALL_CLIENTS
     do
     ssh $client "sed -i 's/<ambari-server1>/<ambari-server2>/g' /etc/ambari-agent/conf/ambari-agent.ini; ambari-agent restart"
     done
fi

七、缺陷

1.没有实时监控，这个可利用crontab -e解决

#*/2 * * * * /opt/ambari-serverHA.sh &

2.监控只有8080端口，应该加上pid的监控可参考ambari自定义服务解决（这个之前有写只写了一半还没发布你看看之前的文章有参考意义）

八、展望

问题

1.ambari元数据mysql备份，备份产生的问题，时效性，让HAserver切换受到影响
2.agent只能向一个server提交心跳，HAserver切换受到影响

展望

1.随着mysql 8.0 发布，MySQL NDB集群发展，第一个问题可以得到解决
2.apache对ambari-agent就赢一定的修改，或者我们有能力自己改一下也可以github上有源码
我相信解决上述2个我认为的关键问题，不久的将来官网会出ambari基于mysql8.0的一个HA方案，而且CDH HDP合并了，相信合并之后会更好，就是不知道还能不能免费使用。

7月3号补充

keepalived

解决amabri serve 和mysql 切换

ambari agent 不需要重新配置了

参考

使用keepalived 做Carbon Thrift Server HA添加链接描述

一门面向 Data 和 AI 的低代码、云原生的开源编程语言

无需安装部署，在线快速体验 Byzer

更多推荐

byzer plugin install log

Byzer 白泽

函数实现越通用越好？来看看 Byzer-LLM 的 Function Implementation 带来的编程思想大变化...

前言Function Calling 是 OpenAI 首先提出来的一个非常有用的功能，实现了大模型对函数的调用能力。Byzer-LLM 给开源模型也带来了 Function Calling 实现。在这个基础上，我们还拓展了 Respond With Class 功能，允许大模型输出标准的Python对象，进步控制了大模型的输出能力。这两个能力参看：给开源大模型带来Function Callin.