CDH6.3.2 deployment of big data platform

Tags (space separated): building big data platform

  • 1: Environment initialization
  • 2: Installation of CDH6.3.2

1: Environment initialization

1.1 environment introduction

System: CentOS7.5X64

cat /etc/hosts 

1.2: Keyless login configuration

Do root authentication without key

 SSH keygen -- always hit enter until the end

 cat >> authorized_keys 

chmod 600 authorized_keys
 Import all public keys into authorized_keys are distributed to all machines under. ssh /

 Then test

1.3: turn off firewalld, iptables and selinux

All nodes are configured:

yum install -y firewall* iptable* 

  (1) close firewalld

 systemctl stop firewalld.service
 systemctl disable firewalld.service
 firewall-cmd --state

 (2) close iptables 
 iptables -F
 systemctl stop iptables.service 
 service iptables save
 systemctl disable iptables.service 

 (3) Disable selinux 

  sed -i '/SELINUX/s/enforcing/disabled/' /etc/selinux/config  
  setenforce 0
  getenforce 0

  //Restart the machine


2.4: all machines change the number of file handles of CentOS7.5x64

vim /etc/security/limits.conf

*               soft    nofile          65535
*               hard    nofile          1029345
*               soft    nproc           unlimited
*               hard    nproc           unlimited
*               soft    memlock         unlimited
*               hard    memlock         unlimited
Execute commands on other nodes:

rsync -arvzP root@ /etc/security/

synchronization limits.conf   file

2.5 time synchronization:

All machines Yum install chocolate*
1.  Synchronize Alibaba cloud time servers:

vim /etc/chrony.conf  increase
Comment out the related default server
server iburst

service chronyd start
chkconfig chronyd on 

service chronyd stop 
service chronyd start 
chkconfig chronyd on 

2.6: cdh6.3.2 file download required for installation

cdh6.3.2 Offline package:


cdh6.3.1 Of CM package:


//Download address:

jdbc Drive:

2.7: install jdk

All big data nodes should be configured: jdk 

rpm -ivh oracle-j2sdk1.8-1.8.0+update181-1.x86_64.rpm

vim /etc/profile

export JAVA_HOME=/usr/java/jdk1.8.0_181-cloudera
export CLASSPATH=.:$JAVA_HOME/jre/lib:$JAVA_HOME/lib:$JAVA_HOME/lib/tools.jar
source /etc/profile 

java -version

2.8 install MariaDB server

yum install -y mariadb-server 

service mariadb start

chkconfig mariadb on 


mysql -uroot -pflyfish225

2.9 configure httpd distributor

yum install -y httpd 

vim /etc/httpd/conf/httpd.conf
//Modify / etc/httpd/conf/httpd.conf Configuration file, in <IfModule mime_ Modify the following in module >

AddType application/x-gzip .gz .tgz .parcel
service httpd start 
chkconfig httpd start 

do CM6.3.1 Source of differentiation

yum install -y createrepo

cd /var/www/html/cdh5.3.6


vim /etc/yum.repos.d/cloduera-manger.repo

name = cm_repo
baseurl =
enable = 1
gpgcheck = 0

yum makecache fast

All hosts execute command synchronization:
rsync -avrzP root@ /etc/yum.repos.d/

yum makecache fast

2.10 configure MySQL JDBC

cd /usr/share/java/

cp -p mysql-connector-java-8.0.15.jar  mysql-connector-java.jar

3: Environment required for installation of CDH6.3.2

3.1: required dependency package for installation

All big data nodes should be installed:

yum -y install chkconfig python bind-utils psmisc libxslt zlib sqlite cyrus-sasl-plain cyrus-sasl-gssapi fuse fuse-libs redhat-lsb postgresql* portmap mod_ssl openssl-devel python-psycopg2 MySQL-python

3.2 install cloudera server

yum install -y cloudera-manager-server 

3.3 create the registry required by CM in mysql

# mysql -uroot -pflyfish225
mysql> create database cmf character set utf8;
mysql> grant all privileges on cmf.* to 'cmf'@'%' identified by 'cmf_1234' with grant option;
mysql> grant all privileges on cmf.* to 'cmf'@'' identified by 'cmf_1234' with grant option;

mysql> flush privileges;

3.4 inject cm server library and cm agent server

/opt/cloudera/cm/schema/ mysql cmf cmf cmf_1234

4: Configure the cloudera Manager installation of CDH6.3.2

4.1 install and start CM

service cloudera-scm-server start 
cd /var/log/cloudera-scm-server/
tail -f cloudera-scm-server.log

5: Open web installation:

echo 10 >  /proc/sys/vm/swappiness
echo never > /sys/kernel/mm/transparent_hugepage/defrag
echo never > /sys/kernel/mm/transparent_hugepage/enabled
echo "vm.swappiness = 10" >> /etc/sysctl.conf 
sysctl -p 
vim /etc/rc.local
echo never > /sys/kernel/mm/transparent_hugepage/defrag
echo never > /sys/kernel/mm/transparent_hugepage/enabled

create database cdh_hive default character set latin1;
grant all privileges on cdh_hive.* to 'hive'@'%' identified by 'Hive_123' with grant option;
create database cdh_oozie default character set utf8;
grant all privileges on cdh_oozie.* to 'oozie'@'%' identified by 'Oozie_123' with grant option;
create database cdh_hue default character set utf8;
grant all privileges on cdh_hue.* to 'hue'@'%' identified by 'Hue_123' with grant option;
create database cdh_rm default character set utf8;
grant all privileges on cdh_rm.* to 'rm'@'%' identified by 'Rm_123' with grant option;
flush privileges;

Tags: Big Data MySQL yum RPM Java

Posted on Thu, 28 May 2020 11:01:15 -0400 by d-woo