Flink practical tutorial: Advanced 1-dimensional table Association

Introduction to flow computing Oceanus Stream computing Oceanus is a powerful tool for real-time analysis of big data product ecosystem. It is an enterprise level real-time big data analysis platform based on Apache Flink with the characteristics of one-stop development, seamless connection, sub second delay, low cost, security and stability. ...

Posted on Sat, 04 Dec 2021 21:27:12 -0500 by Placebo

Design differences of WaterMark under Flink multi parallelism

Raising questions Will the position of WaterMark design affect the normal opening and closing of the window?Next, I simulated two scenarios (source parallelism is 1 and map parallelism is 2), which are1. Set watermark after source and open the window after passing through map2. Set watermark after the map, and then open the windowps: I have se ...

Posted on Fri, 03 Dec 2021 06:38:21 -0500 by $0.05$

Flink independent cluster deployment and HA deployment

Scene description 172.19.9.202 master node JobManager master / slave 172.19.9.201 slave node TaskManager master / slave 172.19.9.203 slave node TaskManager master / slave 1, SSH master node and slave node settings should be unified ssh-keygen -t rsa -P "" Do not set password cat /root/.ssh/id_rsa.pub >> /root/.ssh/authorized_keys ...

Posted on Thu, 02 Dec 2021 16:30:34 -0500 by angelssin

Kafka Stream(KStream) vs Apache Flink

The original text is translated from DZone and translated freely according to the original text.Tencent cloud flow computing Oceanus is a powerful tool for real-time analysis of big data and is compatible with Apache Flink application. New users can 1 yuan purchase flow calculation Oceanus(Flink) cluster , readers are welcome to experience it. ...

Posted on Sat, 27 Nov 2021 23:56:32 -0500 by jf3000

[technical grass planting] I built a complete set of big data system with the money of one rougamo

How can a wool party not participate in the promotion of the Eleventh National Congress of the Communist Party of China. Then I plan to come to Tencent cloud to collect wool.Let me share how to use the money of a rougamo to build a big data platform on the cloud. After my repeated study, I found that it was too simple to collect wool. Finally, ...

Posted on Thu, 25 Nov 2021 14:40:34 -0500 by jmrothermel

wordCount, data source and Sink, Side Outputs, two-phase commit (2pc) of Flink DataStream

1. pom.xml dependency <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-streaming-scala_${scala.binary.version}</artifactId> <version>${flink.version}</version> <scope>provided</scope> </dependency> ...

Posted on Sat, 20 Nov 2021 10:04:26 -0500 by stalione

Flink+Hudi framework Lake warehouse integrated solution

This article is reproduced from the official account, and introduces the prototype construction of Flink + Hudi Lake Warehouse Integration Scheme in detail. The main contents are as follows:HudiThe new architecture is integrated with the lake warehouseBest practicesFlink on HudiFlink CDC 2.0 on Hudi1, Hudi1. IntroductionApache Hudi (pronounced ...

Posted on Thu, 04 Nov 2021 23:19:27 -0400 by oldefezziwig

Flink basic series 27 processfunction API (underlying API)

summary: The transformation operator we learned before cannot access the timestamp information and watermark information of the event. This is extremely important in some application scenarios. For example, map conversion operators such as MapFunction cannot access the timestamp or the event time of the current event. Based on this, the D ...

Posted on Tue, 02 Nov 2021 20:10:32 -0400 by twilson

Flink status management

Flink_ Status in Flink Detailed explanation of Flink state management: deep parsing of Keyed State and Operator List State  <= Good article, recommended reading Operator StateKeyed StateState Backends Status overview All data maintained by a task and used to calculate a result belong to the status of the taskIt can be considere ...

Posted on Tue, 02 Nov 2021 04:23:12 -0400 by elearnindia

Flink uses Kafka Source & Kafka Sink

FlinkKafkaConnector This connector provides access to the event flow of the Apache Kafka service. Flink provides a special Kafka connector for reading and writing data from Kafka topics. Flink Kafka Consumer is integrated with Flink's checkpoint mechanism to provide a once and only semantics. For this reason, Flink not only relies on Kafka's ...

Posted on Wed, 29 Sep 2021 19:51:38 -0400 by theoph