MR is good at processing offline text files. MR+yarn extends the simple operation logic to the distributed program logic in the context of massive data.
If I write distributed programs, I will first face a problem, how to distribute my computing logic to other nodes? How to summarize the running results on all nodes?
The overall framework of MR ...
Posted on Sun, 05 Apr 2020 06:46:21 -0400 by Rohan Hill
Thrift – realize the communication between NodeJS and Java
Apache Thrift is an efficient framework implemented by Facebook that supports remote service calls in multiple programming languages.
It uses interface description language to define and create services, supports extensible cross language service deve ...
Posted on Sun, 05 Apr 2020 04:41:42 -0400 by mlin
Background: there are often some scenarios in the project that need to send messages to notifications (not message queues...).
There are many forms of messages, such as mail, such as nails.
The content of the message is determined by the business and may include dynamic parameters.
The receiver of the message may be a fixed person or a dynam ...
Posted on Sat, 04 Apr 2020 23:47:07 -0400 by dennyx
If more than one Connection is needed in a project, if the Connection is always obtained and disconnected, resources will be wasted. If a pool is created to manage the Connection, the Connection can be reused. With the pool, we don't need to create the Connection ourselves, but get the Connection object through the pool ...
Posted on Sat, 04 Apr 2020 20:09:24 -0400 by Inkybro
Add the following dependencies to the build.gradle file:
//apache log4 logs used internally by es 5.x
compile "org.apache.logging.lo ...
Posted on Sat, 04 Apr 2020 18:53:07 -0400 by Popgun
Base64 is a method of representing arbitrary binary data with 64 characters. For binary data, a group of three bytes is encoded as four bytes, that is, 3 × 8bit=4 × 6bit. As shown in the figure below, the three consecutive bytes (8bit) are divided into four 6bits, and then each 6bit high two bits ...
Posted on Sat, 04 Apr 2020 18:33:42 -0400 by Mesden
linux provides different functions for reading files, with the following references:
When a random read-write operation is performed on the same file handle (under Windows) or file descriptor (under Linux), there are two steps to locate and read/write the file pointer, but since this is not an atomic operation, ...
Posted on Sat, 04 Apr 2020 17:10:10 -0400 by MrAlaska
1. Disk management du command details
The du(disk usage) command calculates the disk space occupied by a file or directory. When no option is specified, it measures the current working directory and all subdirectories, displays the number of blocks occupied by each directory respectively, and finally displays ...
Posted on Sat, 04 Apr 2020 08:29:37 -0400 by archbeta
(1) File reading and writing
• Python has built-in functions for reading and writing files, and the usage is compatible with C.
The operating system does not allow ordinary programs to operate the disk directly, so reading and writing files is a request operation
Open a file object (also known as a file descript ...
Posted on Fri, 03 Apr 2020 15:06:42 -0400 by rajmohan
After deploying the flink cluster, my model is flink on yarn
Create a new flink processing logic code module
Make the module executable jar and put it into the whole project
Perform the submit job operation on the flink client
View the business details on the flink management page.
Flink window functions
After defining the window allocator, w ...
Posted on Fri, 03 Apr 2020 12:44:21 -0400 by RTT