Environmental preparation
jdk8+
mysql
Node Division
centosx1 | centosx2 | centosx3 |
---|---|---|
web-server | ||
exe-server | exe-server | |
mysql |
Here, the hosts file is edited and selinux and firewalld are closed
[root@centos7x3 azkaban-exec-server]# cat /etc/hosts 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 172.16.79.101 centos7x1 172.16.79.102 centos7x2 172.16.79.103 centos7x3
Deployment steps
Build compilation
gradle is smelly and long. It's omitted for the time being
Get the three folders you need
azkaban-db-3.84.4.tar.gz #Database files azkaban-web-server-3.84.4.tar.gz #Web server file azkaban-exec-server-3.84.4.tar.gz #Exe server file
In theory, you can copy whatever service you start. It is convenient to switch nodes after copying together.
Azkaban-web-server-3.84.4 tar.gz Unzip and rename to / user / local / Azkaban / Azkaban web server
Azkaban-exec-server-3.84.4 tar.gz Unzip rename to / user / global / Azkaban / Azkaban exec server
Initialize database
Unzip azkaban-db-3.84.4 tar.gz , get the create-all-sql-3.84.4.sql
[root@centos7x1 azkaban-db-3.84.4]# ls ...... create-all-sql-3.84.4.sql ......
Enter database
# Build a library mysql> create database azkaban default character set utf8mb4; # Set up account mysql> grant all privileges on azkaban.* to 'azkaban'@'%' identified by 'azkaban'; mysql> flush privileges; mysql> exit; [root@centos7x1 azkaban-db-3.84.4]# mysql -uazkaban -p azkaban < create-all-sql-3.84.4.sql
Configure web server
Edit configuration file Azkaban web server / conf/ azkaban.properties
# Azkaban Personalization Settings azkaban.name=Test azkaban.label=My Local Azkaban azkaban.color=#FF3601 azkaban.default.servlet.path=/index web.resource.dir=/usr/local/azkaban/azkaban-web-server/web default.timezone.id=Asia/Shanghai # Azkaban UserManager class # Manage account file user.manager.class=azkaban.user.XmlUserManager user.manager.xml.file=/usr/local/azkaban/azkaban-web-server/conf/azkaban-users.xml # Loader for projects executor.global.properties=conf/global.properties azkaban.project.dir=projects # Velocity dev mode velocity.dev.mode=false # Azkaban Jetty server properties. jetty.use.ssl=false jetty.maxThreads=25 jetty.port=8081 # Azkaban Executor settings # mail settings mail.sender= mail.host= # User facing web server configurations used to construct the user facing server URLs. They are useful when there is a reverse proxy between Azkaban web servers and users. # enduser -> myazkabanhost:443 -> proxy -> localhost:8081 # when this parameters set then these parameters are used to generate email links. # if these parameters are not set then jetty.hostname, and jetty.port(if ssl configured jetty.ssl.port) are used. # azkaban.webserver.external_hostname=myazkabanhost.com # azkaban.webserver.external_ssl_port=443 # azkaban.webserver.external_port=8081 job.failure.email= job.success.email= lockdown.create.projects=false cache.directory=cache # JMX stats jetty.connector.stats=true executor.connector.stats=true # Azkaban mysql settings by default. Users should configure their own username and password. database.type=mysql mysql.port=3306 mysql.host=centos7x1 mysql.database=azkaban mysql.user=azkaban mysql.password=azkaban mysql.numconnections=100 #Multiple Executor azkaban.use.multiple.executors=true #MinimumFreeMemory only selects nodes with memory larger than 6G, so remove this policy #azkaban.executorselector.filters=StaticRemainingFlowSize,MinimumFreeMemory,CpuStatus azkaban.executorselector.filters=StaticRemainingFlowSize,CpuStatus azkaban.executorselector.comparator.NumberOfAssignedFlowComparator=1 azkaban.executorselector.comparator.Memory=1 azkaban.executorselector.comparator.LastDispatched=1 azkaban.executorselector.comparator.CpuUsage=1
Check the user account file VIM Azkaban web server / conf / Azkaban- users.xml
<azkaban-users> <user groups="azkaban" password="azkaban" roles="admin" username="azkaban"/> <user password="metrics" roles="metrics" username="metrics"/> <role name="admin" permissions="ADMIN"/> <role name="metrics" permissions="METRICS"/> </azkaban-users>
Configure exe server
Edit configuration file Azkaban exec server / conf/ azkaban.properties , similar to web server in general, but Web server has more configurations related to selecting exe node
# Azkaban Personalization Settings azkaban.name=Test azkaban.label=My Local Azkaban azkaban.color=#FF3601 azkaban.default.servlet.path=/index web.resource.dir=/usr/local/azkaban/azkaban-web-server/web default.timezone.id=Asia/Shanghai # Azkaban UserManager class user.manager.class=azkaban.user.XmlUserManager user.manager.xml.file=/usr/local/azkaban/azkaban-web-server/conf/azkaban-users.xml # Loader for projects executor.global.properties=conf/global.properties azkaban.project.dir=projects # Velocity dev mode velocity.dev.mode=false # Azkaban Jetty server properties. jetty.use.ssl=false jetty.maxThreads=25 jetty.port=8081 # Where the Azkaban web server is located azkaban.webserver.url=http://localhost:8081 # mail settings mail.sender= mail.host= # User facing web server configurations used to construct the user facing server URLs. They are useful when there is a reverse proxy between Azkaban web servers and users. # enduser -> myazkabanhost:443 -> proxy -> localhost:8081 # when this parameters set then these parameters are used to generate email links. # if these parameters are not set then jetty.hostname, and jetty.port(if ssl configured jetty.ssl.port) are used. # azkaban.webserver.external_hostname=myazkabanhost.com # azkaban.webserver.external_ssl_port=443 # azkaban.webserver.external_port=8081 job.failure.email= job.success.email= lockdown.create.projects=false cache.directory=cache # JMX stats jetty.connector.stats=true executor.connector.stats=true # Azkaban plugin settings azkaban.jobtype.plugin.dir=plugins/jobtypes # Azkaban mysql settings by default. Users should configure their own username and password. database.type=mysql mysql.port=3306 mysql.host=centos7x1 mysql.database=azkaban mysql.user=azkaban mysql.password=azkaban mysql.numconnections=100 # Azkaban Executor settings executor.maxThreads=50 executor.flow.threads=30
Copy to other nodes
[root@centos7x1 local]# scp /usr/local/azkaban/ root@centos7x2:/usr/local [root@centos7x1 local]# scp /usr/local/azkaban/ root@centos7x3:/usr/local
Starting the system
The principle is to start exe first and then web
Start the EXE server node
cd to the directory first, at start, because the system will create several files (ports, logs, etc.) in the current directory
[root@centos7x2 azkaban-exec-server]# cd /usr/local/azkaban/azkaban-exec-server/ [root@centos7x2 azkaban-exec-server]# bin/start-exec.sh [root@centos7x2 azkaban-exec-server]# ls bin conf currentpid executions executor.port executorServerLog__2020-06-27+14:27:02.out lib logs plugins projects temp
Activate exe node
Enter mysql, you can see that there are two nodes registered, change actvie=0 to 1, and activate the node
mysql> select * from executors; +----+-----------+-------+--------+ | id | host | port | active | +----+-----------+-------+--------+ | 4 | centos7x3 | 37454 | 0 | | 5 | centos7x2 | 38088 | 0 | +----+-----------+-------+--------+ 2 rows in set (0.04 sec) mysql> update executors set active=1; Query OK, 2 rows affected (0.00 sec) Rows matched: 2 Changed: 2 Warnings: 0 mysql> select * from executors; +----+-----------+-------+--------+ | id | host | port | active | +----+-----------+-------+--------+ | 4 | centos7x3 | 37454 | 1 | | 5 | centos7x2 | 38088 | 1 | +----+-----------+-------+--------+ 2 rows in set (0.02 sec)
Start web server node
Also cd to the directory first, and run start-web.sh
[root@centos7x1 azkaban-exec-server]# cd /usr/local/azkaban/azkaban-web-server/ [root@centos7x1 azkaban-web-server]# bin/start-web.sh [root@centos7x1 azkaban-web-server]# ls bin conf currentpid lib logs temp web webServerLog_2020-06-27+14:27:53.out [root@centos7x1 azkaban-web-server]#
Visit after success http://172.16.79.101:8081/
Some questions
Solve the problem that the executor is not activated
Sometimes this error is reported during startup, and the node is not activated automatically. At this time, you need to activate it manually.
2020/06/27 14:51:56.580 +0800 ERROR [ExecutorServlet] [Azkaban] executor became inactive before setting up the flow 24 azkaban.executor.ExecutorManagerException: executor became inactive before setting up the flow 24
To activate the Executor method manually:
The port number can be in the executor.port Or in the executors table in the database.
mysql> select * from executors; +----+-----------+-------+--------+ | id | host | port | active | +----+-----------+-------+--------+ | 4 | centos7x3 | 37454 | 1 | | 5 | centos7x2 | 38088 | 1 | +----+-----------+-------+--------+ 2 rows in set (0.02 sec)
curl http://${executorHost}:${executorPort}/executor?action=activate
zang@zang-MS-7A15:~$ curl http://172.16.79.103:37454/executor?action=activate {"status":"success"}