XVII. hadoop compression

I. The significance of data compression in hadoop 1. Basic overview Compression technology can reduce the number of reading and writing sections of the underlying hdfs. And it can reduce the network bandwidth resources occupied in the process of data transmission and reduce the disk space occupied. In MapReduce, both the shuffle and merge proc ...

Posted on Tue, 29 Oct 2019 00:14:36 -0400 by Shygirl

vue java uses AES front and back encryption and decryption

Recently, encrypted transmission of important data is needed in the project. We found a large number of encryption methods on the Internet and finally adopted AES encryption. 1, java terminal package com.zk.web.util; /** * AES 128bit Encryption and decryption tool class * @author dufy */ im ...

Posted on Fri, 25 Oct 2019 12:33:00 -0400 by lelelow

IX. MapReduce--input source code analysis

When the job is submitted to yarn, it will start to schedule and run the map task. Here we start to explain the source code analysis of map input.The entry of a map task is the run() method in MapTask.class 1. First look at the MapTask.run() method MapTask.class //---------------------------------MapTask.java public void run(JobConf job, TaskU ...

Posted on Thu, 24 Oct 2019 07:19:58 -0400 by liquidchild

Getting started with Spring Boot: integrating swagger2

This article is based on the previous one.< Getting started with Spring Boot (6): integrating treetable and zTree to implement tree chart >, this article mainly introduces the spring boot integration swagger2. About swagger, Google. I have related video materials purchased on the Internet, and can call me if necessary. 1. Introduce relat ...

Posted on Wed, 23 Oct 2019 13:28:28 -0400 by londonjustin

Integration of Android-video-video hardware and software codec

There are many video codecs in the machine, such as h264/hevc/h263/mpeg4. Wait. But if you have a new standard encoder, how do you need to integrate it? Read this article and you will find the answer. 1. integration of software encoders: Before integration, we need to understand the loading process ...

Posted on Wed, 09 Oct 2019 18:11:22 -0400 by Brian Swan

java long link to short link code and how to jump to use

I. Realization Principle 1. Using md5 algorithm, a 32-bit signature string is generated for a long web site, which is divided into four segments, each with eight characters. 2. For each of the four cycles, 8 characters are taken as hexadecimal strings and bits and operations of 0x3fffffff(30-bit-1), and more than 30 bits are ignored. It's use ...

Posted on Wed, 09 Oct 2019 17:56:19 -0400 by v4g

Ubuntu Porting FFmpeg and SDL 2

1.libx264 is a free H.264 encoding library. It is part of x264 project. It is widely used. ffmpeg's H.264 implementation is libx264. ffmpeg has 264 decode and no encoding. X264 is needed. Go to the official website to download a libx264 compressed file.I used x264-snapshot-20171212-2245.x264 official we ...

Posted on Wed, 09 Oct 2019 09:20:37 -0400 by 28rain

Data Compression in Hadoop

1 Overview Compression strategies and principles 2 MR supported compression coding Compressed format hadoop comes with itself. algorithm File extension Is it separable Whether the original program needs to be modified after changing to compressed format DEFLATE Yes, direct use DEFLATE .deflate no As with text processing, no modification ...

Posted on Sun, 06 Oct 2019 22:47:40 -0400 by yepster123

apache-commons Common Tool Classes

Preface I've seen that most companies have their own tool classes encapsulated, and many people have implemented the method again, not only the code quality is not guaranteed, but also the performance and concurrency will be problematic; anyway, I generally do not use the tool classes encapsulated by companies, such as apache. commons, springut ...

Posted on Sun, 06 Oct 2019 02:51:50 -0400 by skalar

Android converts various encapsulated video formats into yuv format files based on ffmpeg

The following is based on ffmpeg 2.6.9: Several key structures: AVFormatContext contains a stream information structure AVCodecContext decoder context AVCodec Decoder SwsContext Pixel Information Conversion Context AVPacket contains information about a frame AVFrame Parses a Frame Process summary: av ...

Posted on Sat, 05 Oct 2019 07:42:29 -0400 by corylulu