I. The significance of data compression in hadoop
1. Basic overview
Compression technology can reduce the number of reading and writing sections of the underlying hdfs. And it can reduce the network bandwidth resources occupied in the process of data transmission and reduce the disk space occupied. In MapReduce, both the shuffle and merge proc ...
Posted on Tue, 29 Oct 2019 00:14:36 -0400 by Shygirl
Recently, encrypted transmission of important data is needed in the project.
We found a large number of encryption methods on the Internet and finally adopted AES encryption.
1, java terminal
* AES 128bit Encryption and decryption tool class
* @author dufy
Posted on Fri, 25 Oct 2019 12:33:00 -0400 by lelelow
When the job is submitted to yarn, it will start to schedule and run the map task. Here we start to explain the source code analysis of map input.The entry of a map task is the run() method in MapTask.class
1. First look at the MapTask.run() method
public void run(JobConf job, TaskU ...
Posted on Thu, 24 Oct 2019 07:19:58 -0400 by liquidchild
This article is based on the previous one.< Getting started with Spring Boot (6): integrating treetable and zTree to implement tree chart >, this article mainly introduces the spring boot integration swagger2. About swagger, Google. I have related video materials purchased on the Internet, and can call me if necessary.
1. Introduce relat ...
Posted on Wed, 23 Oct 2019 13:28:28 -0400 by londonjustin
There are many video codecs in the machine, such as h264/hevc/h263/mpeg4. Wait. But if you have a new standard encoder, how do you need to integrate it? Read this article and you will find the answer.
1. integration of software encoders:
Before integration, we need to understand the loading process ...
Posted on Wed, 09 Oct 2019 18:11:22 -0400 by Brian Swan
I. Realization Principle
1. Using md5 algorithm, a 32-bit signature string is generated for a long web site, which is divided into four segments, each with eight characters.
2. For each of the four cycles, 8 characters are taken as hexadecimal strings and bits and operations of 0x3fffffff(30-bit-1), and more than 30 bits are ignored. It's use ...
Posted on Wed, 09 Oct 2019 17:56:19 -0400 by v4g
1.libx264 is a free H.264 encoding library. It is part of x264 project. It is widely used. ffmpeg's H.264 implementation is libx264. ffmpeg has 264 decode and no encoding. X264 is needed.
Go to the official website to download a libx264 compressed file.I used x264-snapshot-20171212-2245.x264 official we ...
Posted on Wed, 09 Oct 2019 09:20:37 -0400 by 28rain
Compression strategies and principles
2 MR supported compression coding
hadoop comes with itself.
Is it separable
Whether the original program needs to be modified after changing to compressed format
Yes, direct use
As with text processing, no modification ...
Posted on Sun, 06 Oct 2019 22:47:40 -0400 by yepster123
I've seen that most companies have their own tool classes encapsulated, and many people have implemented the method again, not only the code quality is not guaranteed, but also the performance and concurrency will be problematic; anyway, I generally do not use the tool classes encapsulated by companies, such as apache. commons, springut ...
Posted on Sun, 06 Oct 2019 02:51:50 -0400 by skalar
The following is based on ffmpeg 2.6.9:
Several key structures:
AVFormatContext contains a stream information structure
AVCodecContext decoder context
SwsContext Pixel Information Conversion Context
AVPacket contains information about a frame
AVFrame Parses a Frame
Posted on Sat, 05 Oct 2019 07:42:29 -0400 by corylulu