Programmer Help

1. Iterable 1.1 general Iteratable represents a set that can be iterated. It inherits the Traversable trait and is also the parent trait of other sets. Most importantly, it defines the method to obtain the iterator: def iterator: Iterator[A]. This is an abstract method. Its concrete implementation class needs to implement this method to reali ...

I. Overview This collection is similar to java except that scala is reimplemented according to its syntax Divided into variable and immutable sets Common list of collections   List Summary          Stores the same type of sequential storage structure        &nbsp ...

This tutorial will show you how to quickly build streaming ETL for MySQL and Postgres based on Flink CDC.Flink-CDC project address:https://github.com/ververica/flink-cdc-connectorsThis tutorial's demo is based on a Docker environment and will be done in the Flink SQL CLI, involving only SQL, without a single line of Java/Scala code, or with an ...

There are some files growing every day on HDFS. At present, Snappy compression is used. Suddenly, one day, OOM 1. Reasons: Because snappy cannot split slices, a file will be read by a task. After reading and decompressing, the data will expand many times. If the number of files is too large and your parallelism is very large, it will lead to ...

Students who have learned about Scala should know that Scala cancels the use of many keywords, such as public, static, etc; But this also brings us some puzzles. This article will answer one of them: why is the bottom layer of public or private in scala? The attribute in the class is expressed as "public" without a scope qualifier We ...

It should not be easy to master a language, but it should not be difficult to simply master some usage and use it. Scala and java have many commonalities. From the perspective of Java, it should be easy to enter Scala. Configuring the Scala environment Here I configure Scala version 2.11.8 1. First, configure the local environment variable ...

We know that RDD is a particularly important concept in spark. It can be said that all logic of spark needs to rely on RDD. In this article, we briefly talk about RDD in spark. The definition of RDD in spark is as follows: abstract class RDD[T: ClassTag]( @transient private var _sc: SparkContext, @transient private var deps: Seq[Depend ...

Pattern match ing There is a very powerful matching mechanism in Scala, such as: Judging Fixed ValuesType QueryQuick data acquisition Simple pattern matching A pattern match contains a series of alternatives, each starting with the keyword case, and each alternative contains a pattern and one or more expressions. Arrow symbol=>se ...

An action operator is an operator that triggers an action. Triggering an action means real calculation data. collect Collect is to collect data from the executor side to the driver side. For example, a simple wordcount program: object CollectAction { def main(args: Array[String]): Unit = { val conf: SparkConf = new SparkConf().set ...

There are some operators of xxxByKey in spark. Let's see. groupByKey explain Suppose we want to group some string lists: object GroupByKeyOperator { def main(args: Array[String]): Unit = { val conf = new SparkConf().setMaster("local[*]").setAppName("RDD") val context = new SparkContext(conf) val rdd: RDD[String] = contex ...