Hadoop 8-day course - the third day, MapReduce details

MR is good at processing offline text files. MR+yarn extends the simple operation logic to the distributed program logic in the context of massive data. If I write distributed programs, I will first face a problem, how to distribute my computing logic to other nodes? How to summarize the running results on all nodes? The overall framework of MR ...

Posted on Sun, 05 Apr 2020 06:46:21 -0400 by Rohan Hill

Thrift -- communication between NodeJS and Java

Thrift – realize the communication between NodeJS and Java Apache Thrift is an efficient framework implemented by Facebook that supports remote service calls in multiple programming languages. It uses interface description language to define and create services, supports extensible cross language service deve ...

Posted on Sun, 05 Apr 2020 04:41:42 -0400 by mlin

Configurable message template encapsulation

Background: there are often some scenarios in the project that need to send messages to notifications (not message queues...). There are many forms of messages, such as mail, such as nails. The content of the message is determined by the business and may include dynamic parameters. The receiver of the message may be a fixed person or a dynam ...

Posted on Sat, 04 Apr 2020 23:47:07 -0400 by dennyx

DBCP C3PO Druid three database connections

If more than one Connection is needed in a project, if the Connection is always obtained and disconnected, resources will be wasted. If a pool is created to manage the Connection, the Connection can be reused. With the pool, we don't need to create the Connection ourselves, but get the Connection object through the pool ...

Posted on Sat, 04 Apr 2020 20:09:24 -0400 by Inkybro

SpringBoot integrates elastic search and CRUD operation

Configuration preparation Add the following dependencies to the build.gradle file: compile "org.elasticsearch.client:transport:5.5.2" compile "org.elasticsearch:elasticsearch:5.5.2" //apache log4 logs used internally by es 5.x compile "org.apache.logging.log4j:log4j-core:2.7" compile "org.apache.logging.lo ...

Posted on Sat, 04 Apr 2020 18:53:07 -0400 by Popgun

Base64 encoding and decoding

Base64 principle Base64 is a method of representing arbitrary binary data with 64 characters. For binary data, a group of three bytes is encoded as four bytes, that is, 3 × 8bit=4 × 6bit. As shown in the figure below, the three consecutive bytes (8bit) are divided into four 6bits, and then each 6bit high two bits ...

Posted on Sat, 04 Apr 2020 18:33:42 -0400 by Mesden

Probe into ReadType of scan in HBase

background knowledge Linux level linux provides different functions for reading files, with the following references: When a random read-write operation is performed on the same file handle (under Windows) or file descriptor (under Linux), there are two steps to locate and read/write the file pointer, but since this is not an atomic operation, ...

Posted on Sat, 04 Apr 2020 17:10:10 -0400 by MrAlaska

A practical way to find large files on linux disk

1. Disk management du command details purpose The du(disk usage) command calculates the disk space occupied by a file or directory. When no option is specified, it measures the current working directory and all subdirectories, displays the number of blocks occupied by each directory respectively, and finally displays ...

Posted on Sat, 04 Apr 2020 08:29:37 -0400 by archbeta

python learning file operation

(1) File reading and writing • Python has built-in functions for reading and writing files, and the usage is compatible with C. The operating system does not allow ordinary programs to operate the disk directly, so reading and writing files is a request operation Open a file object (also known as a file descript ...

Posted on Fri, 03 Apr 2020 15:06:42 -0400 by rajmohan

flink business use record

After deploying the flink cluster, my model is flink on yarn Create a new flink processing logic code module Make the module executable jar and put it into the whole project Perform the submit job operation on the flink client View the business details on the flink management page. Flink window functions After defining the window allocator, w ...

Posted on Fri, 03 Apr 2020 12:44:21 -0400 by RTT