Flink 1.9.0 source compilation on mac (using yhadoop2.6.0-cdh5.15.1)

Source code compilation and installation of flink

Preconditions:
JDK8
Maven3
(please search for jdk and maven by yourself, thank you!)

Compile by downloading Flink source code instead of downloading binary package directly
Flink GitHub address
Find the version you need in releases. I used version 1.9.0: 1.9.0

Tar - zxvf flick-release-1.9.0.tar.gz [- C can specify the directory to be placed]


If you want to use Flink packaging in other third-party manufacturers, for example, to use it on CDH and hortonworks platforms, you need to specify the Hadoop version for packaging. First, we need to package the flink-shaded-hadoop-2, and then compile it.

1. Create a new directory under the extracted flink source directory, and obtain the source code of flink shared from git

git clone https://github.com/apache/flink-shaded.git

Switch the corresponding code branches according to the missing version. What I lack here is the 7.0 version of flink-shaded-hadoop-2

git checkout  release-7.0

Modify pom.xml in the Flink shared project to join the central warehouse such as cdh. Otherwise, the package related to cdh may not be found in the compiled version.
Add in pom.xml file

<profile>
	<id>vendor-repos</id>
	<activation>
		<property>
			<name>vendor-repos</name>
		</property>
	</activation>
	<!-- Add vendor maven repositories -->
	<repositories>
		<!-- Cloudera -->
		<repository>
			<id>cloudera-releases</id>
			<url>https://repository.cloudera.com/artifactory/cloudera-repos</url>
			<releases>
				<enabled>true</enabled>
			</releases>
			<snapshots>
				<enabled>false</enabled>
			</snapshots>
		</repository>
		<!-- Hortonworks -->
		<repository>
			<id>HDPReleases</id>
			<name>HDP Releases</name>
			<url>https://repo.hortonworks.com/content/repositories/releases/</url>
			<snapshots><enabled>false</enabled></snapshots>
			<releases><enabled>true</enabled></releases>
		</repository>
		<repository>
			<id>HortonworksJettyHadoop</id>
			<name>HDP Jetty</name>
			<url>https://repo.hortonworks.com/content/repositories/jetty-hadoop</url>
			<snapshots><enabled>false</enabled></snapshots>
			<releases><enabled>true</enabled></releases>
		</repository>
		<!-- MapR -->
		<repository>
			<id>mapr-releases</id>
			<url>https://repository.mapr.com/maven/</url>
			<snapshots><enabled>false</enabled></snapshots>
			<releases><enabled>true</enabled></releases>
		</repository>
	</repositories>
</profile>

Then choose the CDH version you need to compile flink-shaded-hadoop-2! ]

mvn  clean install -DskipTests -Drat.skip=true -Pvendor-repos  -Dhadoop.version=2.6.0-cdh5.15.1

If you can see that the following figure represents the flink-shaded-hadoop-2 compilation. [but Flink, we haven't started compiling yet]

And then we're going to start with FLink
Why add Dfast: To speed up the build you can skip tests, QA plugins, and JavaDocs

mvn clean install -DskipTests -Pvendor-repos -Dfast 
					-Dhadoop.version=2.6.0-cdh5.15.1


The first compilation takes a long time, because you need to go to the central warehouse to download all the dependent packages in the flink source code, please wait patiently!

If you have any good learning ideas welcome to exchange, thank you!

Published 10 original articles, won praise 3, visited 95
Private letter follow

Tags: Hadoop git Maven github

Posted on Sun, 02 Feb 2020 08:31:38 -0500 by oneofayykind