4. Principle and use of spark--sparkSQL

[TOC] 1. Overview of spark SQL 1.1 What is spark SQL Spark SQL is a module Spark uses to process structured data. It provides a programming abstraction called DataFrame and functions as a distributed SQL query engine.Similar to what hive does. Features of 1.2 spark SQL 1. Easy to integrate: When you installed Spark, it was already integrated.N ...

Posted on Sat, 16 Nov 2019 01:24:56 -0500 by HokieTracks

5. Principles and uses of spark--spark streaming

1. Overview of spark-streaming 1.1 Common real-time computing engines Real-time computing engines, also known as streaming computing engines, currently have three commonly used:1. Apache Storm: True Streaming2. Spark Streaming: Strictly speaking, it's not really streaming (real-time computing)Processing continuous streaming data as discrete RD ...

Posted on Sat, 16 Nov 2019 01:20:27 -0500 by Snart

Spark Core Knowledge Points Review-1

Day1111 Spark Task Scheduling Several key components of Spark Spark Core Concepts and characteristics of RDD Two types of RDD generation Two types of RDD operators Operator Practice partition RDD Dependency DAG: Directed Acyclic Graph Task Submission cache checkPoint Custom Sorting Custom Partiti ...

Posted on Mon, 11 Nov 2019 22:46:13 -0500 by 2oMst

Serialization and deserialization of Kafka messages

The parameters Kafka Producer must configure when sending messages are: bootstrap.servers, key.serializer, value.serializer. Serialization is performed after the Interceptor and before the partitions are allocated. First of all, let's see how Kafka Producer can write under normal circumstances through an example code: public class ProducerJava ...

Posted on Mon, 11 Nov 2019 18:18:34 -0500 by nadeem14375

I. Basic Introduction to scala

[TOC] I. overview of scala 1.1 INTRODUCTION Scala is a multi paradigm programming language. The original intention of its design is to integrate various features of object-oriented programming and functional programming. Scala runs on the Java platform (Java virtual machine) and is compatible with existing Java programs. It can also run in Jav ...

Posted on Sun, 10 Nov 2019 08:11:37 -0500 by nayone

Concurrent programming model Akka

Catalog Akka overview Actor model in Akka Case 1. Send messages to yourself Case 2: ping pong Akka overview Akka is a toolkit and runtime for building highly concurrent, distributed and fault-tolerant applications on the JVM platform. Akka is written in Scala language and provides the development interface of sc ...

Posted on Sat, 02 Nov 2019 16:06:58 -0400 by jexx

12 integer power of value

Topic requirement: given a floating-point base of double type and integer exponent of int type. Find the exponent power of base.   This picture is from https://blog.csdn.net/sodacoco/article/details/81606974 Well done   The difficulty of this question is to make all the situations clear and to be very intuitive. And in the fourth case, ...

Posted on Thu, 31 Oct 2019 19:18:06 -0400 by louisstephens

Restapi - restapi SQL: user independent services

The original intention of learning functional programming is to see the bleak prospect of the familiar oop programming language and sql database in the modern business society, and prepare to completely abandon the windows technology stack to the field of distributed big data technology. But in reality, the ideal is always unsatisfactory. Origi ...

Posted on Mon, 28 Oct 2019 07:41:00 -0400 by Fixxer

scala learning: controlling abstraction

Control abstraction Reduce duplicate code Duplicate code This is the most likely code to be written at the beginning. The code is very simple, which is the filtering function of file names. object FileMatcher { private def files = new File(".").listFiles() def filesEnding(query: String): Array[File] = { for (file <- files; if file ...

Posted on Fri, 25 Oct 2019 11:06:10 -0400 by sKunKbad

SDKMAN recommends a good one

It is a tool for managing parallel versions of multiple software development kits on most Unix based systems. It provides a convenient command line interface (CLI) and API to install, switch, delete and list sdk related information. Here are some features: By Developers, for Developers It is no longer necessary to download, decompress and set ...

Posted on Wed, 23 Oct 2019 01:49:32 -0400 by outatime88