How to install CDP DC7.0.3 in RedHat 7.7

Tags (space separated): building big data platform

  • 1: Overview of CDP DC 7.0.3
  • 2: System environment initialization
  • 3: Build CDP DC 7.0.3

1: Overview of CDP DC 7.0.3

1.1 CDP dc 7.0.3

CDP DC7.0.3 is the first on premise version integrating all components of CDH and HDP after the merger of Cloudera and Hortonworks. CDP Data Center is mainly composed of Cloudera Runtime, which is composed of more than 35 open source projects. Of course, CDP Data Center also includes other functions such as Cloudera Manager, Key Management, professional support, etc., as shown in the following figure:

1.2 the main component versions of cloudera runtime are as follows:

2: System environment initialization

2.1 introduction to system environment

OS: rhel7.7x64 

cat /etc/hosts 
---
192.168.100.21  rhel01.flyfish
192.168.100.22  rhel02.flyfish
192.168.100.23  rhel03.flyfish
192.168.100.24  rhel04.flyfish
192.168.100.25  rhel05.flyfish
192.168.100.26  rhel06.flyfish
---

//The first four machines are used this time

2.2 keyless login configuration

Do root authentication without key

 SSH keygen -- always hit enter until the end

 cat id_rsa.pub >> authorized_keys 

 Import all public keys into authorized_keys and distribute them to all machines under. ssh /

 Then test

2.3 turn off firewalld, iptables and selinux

All nodes are configured:

yum install -y firewall* iptable* 

  (1) Close firewalld

 systemctl stop firewalld.service
 systemctl disable firewalld.service
 firewall-cmd --state

 (2) Close iptables 
 iptables -F
 systemctl stop iptables.service 
 service iptables save
 systemctl disable iptables.service 

 (3) Prohibit selinux 

  sed -i '/SELINUX/s/enforcing/disabled/' /etc/selinux/config  
  setenforce 0
  getenforce 0
  sestatus

  //Restart machine

  reboot

2.4 number of file handles of RHEL7.7X64 changed by all machines

vim /etc/security/limits.conf
---

*               soft    nofile          65535
*               hard    nofile          1029345
*               soft    nproc           unlimited
*               hard    nproc           unlimited
*               soft    memlock         unlimited
*               hard    memlock         unlimited

---

ansible cdh7 -m copy -a "src=/etc/security/limits.conf dest=/etc/security/" 

2.5 time synchronization:

All machines yum install chrony*
1. rhel01.flyfish:

vim /etc/chrony.conf increase
----
//Comment out the related default server
server ntp1.aliyun.com iburst

allow 192.168.100.0/24

service chronyd start
chkconfig chronyd on 

2. rhel02.flyfish----rhel04.flyfish synchronization rhel01.flyfish

vim /etc/chrony.conf
----
//Comment out the default server
server 192.168.100.21 iburst
----

service chronyd stop 
service chronyd start 
chkconfig chronyd on 

2.7 file download for CDP DC 7.0.3 installation

CM Of rpm Bag:

---
cloudera-manager-agent-7.0.3-1635136.el7.x86_64.rpm
cloudera-manager-daemons-7.0.3-1635136.el7.x86_64.rpm
cloudera-manager-server-7.0.3-1635136.el7.x86_64.rpm
cloudera-manager-server-db-2-7.0.3-1635136.el7.x86_64.rpm
enterprise-debuginfo-7.0.3-1635136.el7.x86_64.rpm
openjdk8-8.0+232_9-cloudera.x86_64.rpm
allkeys.asc

//Download address:
http://archive.cloudera.com/cm7/7.0.3/redhat7/yum/RPMS/x86_64/
http://archive.cloudera.com/cm7/7.0.3/allkeys.asc

----

cdh7.0.3 Of parcel: 

--
CDH-7.0.3-1.cdh7.0.3.p0.1635019-el7.parcel
CDH-7.0.3-1.cdh7.0.3.p0.1635019-el7.parcel.sha256
manifest.json

//Download address:
http://archive.cloudera.com/cdh7/7.0.3.0/parcels/

---

2.8 setting up swap

Will all swap Is 1 with immediate effect

ansible cdh7 -m shell -a "echo '1' > /proc/sys/vm/swappiness"

ansible cdh7 -m shell -a "sysctl -a | grep vm.swappiness"

vim /etc/sysctl.conf 
----
//Finally, add:
vm.swappiness = 1 
----
sysctl -p

2.8 set maximum transparency

ansible cdh7 -m shell -a "echo 'never' > /sys/kernel/mm/transparent_hugepage/defrag"

ansible cdh7 -m shell -a "echo 'never' > /sys/kernel/mm/transparent_hugepage/enabled"

ansible cdh7 -m shell -a "cat /sys/kernel/mm/transparent_hugepage/enabled"
ansible cdh7 -m shell -a "cat /sys/kernel/mm/transparent_hugepage/defrag"

Modify the/etc/rc.d/rc.local File permissions for boot execution
ansible cdh7 -m shell -a "chmod +x /etc/rc.d/rc.local"

vim /etc/rc.d/rc.local
---
//Finally, add:

if test -f /sys/kernel/mm/transparent_hugepage/enabled; 

then

echo never > /sys/kernel/mm/transparent_hugepage/enabled

fi

if test -f /sys/kernel/mm/transparent_hugepage/defrag; 
then

echo never > /sys/kernel/mm/transparent_hugepage/defrag

fi

---

ansible cdh7 -m copy -a "src=/etc/rc.d/rc.local dest=/etc/rc.d/"

ansible cdh7 -m shell -a "tail -f /etc/rc.d/rc.local"

2.9 configure httpd server

ansible rhel01.flyfish -m yum -a "state=present name=httpd"
ansible rhel01.flyfish -m service -a "name=httpd state=started"
ansible rhel01.flyfish -m shell -a "ps -ef |grep httpd"

modify /etc/httpd/conf/httpd.conf stay<IfModule mime_module>Modify the following in

vim /etc/httpd/conf/httpd.conf

---
AddType application/x-compress .Z
AddType application/x-gzip .gz .tgz .parcel
---

service httpd restart 

2.10 install malidb database

yum install mariadb* 

service mariadb start 

chkonfig mariadb on 

mysql_secure_installation

2.11 create tables required by CM, Hive, etc

vim cm.sql
----
create database metastore default character set utf8;
CREATE USER 'hive'@'%' IDENTIFIED BY 'password';
GRANT ALL PRIVILEGES ON metastore. * TO 'hive'@'%';
FLUSH PRIVILEGES;
create database cm default character set utf8;
CREATE USER 'cm'@'%' IDENTIFIED BY 'password';
GRANT ALL PRIVILEGES ON cm. * TO 'cm'@'%';
FLUSH PRIVILEGES;
create database rm default character set utf8;
CREATE USER 'rm'@'%' IDENTIFIED BY 'password';
GRANT ALL PRIVILEGES ON rm. * TO 'rm'@'%';
FLUSH PRIVILEGES;
create database hue default character set utf8;
CREATE USER 'hue'@'%' IDENTIFIED BY 'password';
GRANT ALL PRIVILEGES ON hue. * TO 'hue'@'%';
FLUSH PRIVILEGES;
create database oozie default character set utf8;
CREATE USER 'oozie'@'%' IDENTIFIED BY 'password';
GRANT ALL PRIVILEGES ON oozie. * TO 'oozie'@'%';
FLUSH PRIVILEGES;
create database ranger default character set utf8;
CREATE USER 'rangeradmin'@'%' IDENTIFIED BY 'password';
GRANT ALL PRIVILEGES ON ranger. * TO 'rangeradmin'@'%';
FLUSH PRIVILEGES;
----

mysql -uroot -pflyfish225

source cm.sql

2.12 install JDBC Driver

mkdir -p /usr/share/java/
tar -zxvf mysql-connector-java-5.1.44.tar.gz

mv mysql-connector-java-5.1.44-bin.jar /usr/share/java

cd /usr/share/java/

mv mysql-connector-java-5.1.44-bin.jar mysql-connector-java.jar 

chmod 777 mysql-connector-java.jar

3: Build CDP DC 7.0.3

3.1 install jdk

All nodes are unloaded from the system jdk

rpm -qa |grep jdk 
rpm -e java-1.8.0-openjdk-1.8.0.222.b03-1.el7.x86_64 copy-jdk-configs-3.3-10.el7_5.noarch java-1.8.0-openjdk-headless-1.8.0.222.b03-1.el7.x86_64 --nodeps 

yum -y install openjdk8-8.0+232_9-cloudera

vim /etc/profile
---
//Add the environment variable of jdk to the end

export JAVA_HOME=/usr/java/jdk1.8.0_232-cloudera
export CLASSPATH=.:$JAVA_HOME/jre/lib:$JAVA_HOME/lib:$JAVA_HOME/lib/tools.jar
PATH=$PATH:$HOME/bin:$JAVA_HOME/bin

----
source /etc/profile

java -version 

3.2 configure the source of cdh7.0.3

mv cdh7.0.3 cdh7
cd cdh7 

createrepo .

cd ..

mv cdh7 /var/www/html/

Make Cloudera Manager Of repo source

cd /etc/yum.reos.d/

vim cdh.repo
---
[cmrepo]
name = cm_repo
baseurl = http://192.168.100.21/cdh7/
enable = true
gpgcheck = false

---

ansible cdh7 -m copy -a "src=/etc/repos.d/cdh7.repo dest=/etc/repos.d/"
yum repolist

3.3 install Cloudera Manager Server

yum -y install cloudera-manager-server

Initialize database

/opt/cloudera/cm/schema/scm_prepare_database.sh mysql cm cm password

start-up Cloudera Manager Server

systemctl start cloudera-scm-server
systemctl status cloudera-scm-server

cd /var/log/cloudera-scm-server
tail -100f cloudera-scm-server.log

netstat -nultp |grep 7180

3.4 visit CM through http://192.168.100.21:7180/cmf/login

User name: admin
 Password: admin

Log in to CM, select 60 day trial, click continue

Cluster installation:

Cluster name

Identify the host of the cluster

Parcel Repository & Network Settings

Install cloudera SCM agent

Allocate parcel package

Click Inspect Network Performance and Inspect Hosts to check the hosts

Define services:

Click "continue" to enter the next step, test the database connection

First enter CM's user name and password admin/admin

The following 4 passwords need to contain at least one letter and one number. For the convenience of installation, enter admin@123

Installation service initialization

Tags: Big Data ansible Java shell yum

Posted on Thu, 19 Mar 2020 07:12:15 -0400 by iceman400