Programmer Help

[TOC] 1. Overview of spark SQL 1.1 What is spark SQL Spark SQL is a module Spark uses to process structured data. It provides a programming abstraction called DataFrame and functions as a distributed SQL query engine.Similar to what hive does. Features of 1.2 spark SQL 1. Easy to integrate: When you installed Spark, it was already integrated.N ...

1. Overview of spark-streaming 1.1 Common real-time computing engines Real-time computing engines, also known as streaming computing engines, currently have three commonly used:1. Apache Storm: True Streaming2. Spark Streaming: Strictly speaking, it's not really streaming (real-time computing)Processing continuous streaming data as discrete RD ...

Day1111 Spark Task Scheduling Several key components of Spark Spark Core Concepts and characteristics of RDD Two types of RDD generation Two types of RDD operators Operator Practice partition RDD Dependency DAG: Directed Acyclic Graph Task Submission cache checkPoint Custom Sorting Custom Partiti ...

The parameters Kafka Producer must configure when sending messages are: bootstrap.servers, key.serializer, value.serializer. Serialization is performed after the Interceptor and before the partitions are allocated. First of all, let's see how Kafka Producer can write under normal circumstances through an example code: public class ProducerJava ...

[TOC] I. overview of scala 1.1 INTRODUCTION Scala is a multi paradigm programming language. The original intention of its design is to integrate various features of object-oriented programming and functional programming. Scala runs on the Java platform (Java virtual machine) and is compatible with existing Java programs. It can also run in Jav ...

Catalog Akka overview Actor model in Akka Case 1. Send messages to yourself Case 2: ping pong Akka overview Akka is a toolkit and runtime for building highly concurrent, distributed and fault-tolerant applications on the JVM platform. Akka is written in Scala language and provides the development interface of sc ...

Topic requirement: given a floating-point base of double type and integer exponent of int type. Find the exponent power of base.   This picture is from https://blog.csdn.net/sodacoco/article/details/81606974 Well done   The difficulty of this question is to make all the situations clear and to be very intuitive. And in the fourth case, ...

The original intention of learning functional programming is to see the bleak prospect of the familiar oop programming language and sql database in the modern business society, and prepare to completely abandon the windows technology stack to the field of distributed big data technology. But in reality, the ideal is always unsatisfactory. Origi ...

Control abstraction Reduce duplicate code Duplicate code This is the most likely code to be written at the beginning. The code is very simple, which is the filtering function of file names. object FileMatcher { private def files = new File(".").listFiles() def filesEnding(query: String): Array[File] = { for (file <- files; if file ...

It is a tool for managing parallel versions of multiple software development kits on most Unix based systems. It provides a convenient command line interface (CLI) and API to install, switch, delete and list sdk related information. Here are some features: By Developers, for Developers It is no longer necessary to download, decompress and set ...