Self-increasing ID algorithm snowflake

There are no rules for IDs generated using UUID s or GUID s Snowflake algorithm is implemented by Twitter engineers for incremental but not repetitive ID s OverviewIn distributed systems, there are scenarios where a globally unique ID is required. In order to prevent ID conflicts, a 36-bit UUID can be used, but it has some drawbacks. First, it ...

Posted on Thu, 19 Mar 2020 03:20:03 -0400 by Tea_J

scala -- pattern matching

Article directory 1. Pattern matching 1.1. It can be used in the Switch statement 1.2. guard 1.3. Variable and type patterns in patterns 1.4. Matching arrays, lists, and tuples 1.5. Sample class 1.6. Analog enumeration 1.7. Option 1. Pattern matching 1. Pattern matching is a very powerful func ...

Posted on Fri, 06 Mar 2020 01:02:14 -0500 by inni

The integration of Flink SQL client 1.10 and hive to read real-time data

Looking forward to processing streaming data in the form of pure SQL, flink 1.10 introduced Hive integration, which is available in production, and has stronger streaming SQL processing capacity. Let's try it this time~~   ​   [outline] 1. Environmental preparation 2. SQL Client and hive integrated configuration 3. Reading kafka data with ...

Posted on Wed, 04 Mar 2020 01:51:22 -0500 by xtheonex

Action action of Spark RDD operation

first def first(): T First returns the first element in RDD, without sorting. scala> var rdd1 = sc.makeRDD(Array(("A","1"),("B","2"),("C","3")),2) rdd1: org.apache.spark.rdd.RDD[(String, String)] = ParallelCollectionRDD[33] at makeRDD at :21 scala> rdd1.first res14: (String, String) = (A,1) ...

Posted on Sun, 01 Mar 2020 07:58:41 -0500 by amesowe

[actual combat] installation, deployment and simple use of zookeeper+kafka cluster

Catalog Asynchronous update: Message Queuing Message queuing: kafka zookeeper cluster construction scala build kafka cluster construction Asynchronous update: Message Queuing The multi-level cache architecture is mainly used to solve the data with low timeliness, such as the basic information ...

Posted on Sun, 01 Mar 2020 04:45:13 -0500 by tariel

Scala [12 pattern matching]

pattern matching In addition to matching values, Scala can also match types, Array and List elements, case class es, and even options. pattern matching Scala has no switch case syntax in Java. Correspondingly, Scala provides a more powerful match case syntax, that is, pattern matching, class replaci ...

Posted on Sun, 23 Feb 2020 07:04:46 -0500 by Batosi

Spark RDD creates API MySQL HBase

Generally speaking, each Spark application contains a Driver, which runs the user's main method and performs various parallel operations on the cluster. Spark provides the main abstract concept, which is the elastic distributed data set (RDD), which is an element divided across clusters Can be operated ...

Posted on Sun, 23 Feb 2020 06:15:35 -0500 by ramesh_iridium

Sparkstreaming \ updatestatebykey state calculation

Catalog 1, Theoretical basis 2, Code test wordCount 1, code 2. Test data 3. Results display 1, Theoretical basis 1. In flow computing, there is usually a need for state computing, that is, the current computing results not only depend on the current received data, but also need to merge the p ...

Posted on Sun, 16 Feb 2020 01:04:55 -0500 by godwisam

Python basic tutorial: a simple example of using map function to complete Python parallel tasks

This article mainly introduces a simple example of using map function to complete Python parallel tasks. Multithreading and multiprocess programming are always hot and difficult issues in Python. Please refer to As we all know, Python's parallel processing ability is not ideal. I think if we don't consi ...

Posted on Sat, 15 Feb 2020 09:36:11 -0500 by Rovas

Cassandra appender - distributed logging, distributed software logback appender

At the last Scala meetup of the lunar year, Liu Ying was inspired to share her professional software development experience. I suddenly realized that I haven't completely followed any standard development specifications. It is true that there will be no strict requirements for standardized operation in the process of technical research and lear ...

Posted on Wed, 12 Feb 2020 08:46:18 -0500 by eyedol