The first lesson of big data on the cloud: MaxCompute authorization and appearance operation hidden pit Guide

1, Sub account creation, AK information binding If you are the first time to log in to digital plus platform and use DataWorks with a sub account, you need to confirm the following information: • the business alias of the primary account to which the sub account belongs. • user name and password of the sub account. • AccessKey ID ...

Posted on Mon, 13 Jan 2020 02:57:38 -0500 by majik_sheff

Hive embedded mode installation (based on Hadoop)

Download and extract to the specified directory (1) Configure environment variable / etc/profile export HIVE_HOME=/usr/local/hive/ export PATH=$PATH:$HIVE_HOME/bin (2) Modify hive configuration file hive-env.sh Add hadoop path, hive path, lib path of hive #HADOOP_HOME=${bin}/../../hadoop HADOOP_HOME=/usr/local/hadoop ...

Posted on Thu, 09 Jan 2020 12:33:59 -0500 by ksimpkins

docker quick installation of Hadoop Cluster -- single node

1. Two ways to build Hadoop 1.1: the earliest installation of hadoop is the version released by the native apache. You need to download hdfs,hive,mapreduce, zookeeper and mysql from the apache official website. Each component needs to be installed separately, and the version of the component needs to be consistent, and many settings need to be ...

Posted on Tue, 17 Dec 2019 05:10:30 -0500 by brittny85

Common operations and examples of Hive DML

1. File loading Hive table Syntax: LOAD DATA [LOCAL] INPATH 'filepath' [OVERWRITE] INTO TABLE tablename [PARTITION (partcol1=val1, partcol2=val2 ...)] Example: # Create partition table create table tb_load_data( id int, name string, hobby array<string>, add map<String,string> ) partitioned by(part_tag1 strin ...

Posted on Sun, 15 Dec 2019 15:54:53 -0500 by magicrobotmonkey

Alibaba Tencent cloud hadoop+spark cluster construction

Alibaba Tencent cloud hadoop+spark cluster construction (2) linux version: centos7 hadoop version: 3.1.1 spark version: 2.3.2 hadoop has been set up in 1, and then spark. For convenience, use shell script to download spark and hive (later, run spark first). download_file.sh ------------------------------- #!/bin/bash TARGET=f ...

Posted on Sat, 14 Dec 2019 11:10:02 -0500 by icesolid

Knowledge points of jg

1. jg is divided into node table and edge table. Nodes have their own jid and related attribute fields. This jid is globally unique, and the jid of each node is different 2. The edge table will save two JIDS, one is the start node and the other is the end node, including the related attribute information of the edge 3. Becaus ...

Posted on Fri, 13 Dec 2019 16:33:55 -0500 by bosco500

Use mysql to store metadata for hive installation and configuration

Install and configure mysql Install and configure hive One mysql installation configuration I. Mysql installation 1. Install mysql command: Yum install MySQL server - y (- y parameter is the default yes for all places where you need to enter yes) 2. Open mysql service: service mysqld start 3. Set the mysql service to start a ...

Posted on Thu, 12 Dec 2019 10:31:06 -0500 by Rich464

[hadoop] hive installation practice

1. Download Hive installation package: Download on official website: http://hive.apache.org/downloads.html 2. Upload Hive's tar package and extract it:Suggestions and hadoop directory at the first level, convenient for subsequent use; Unzip: tar -zxvf apache-hive-1.2.1-bin.tar.gz -C /home/hadoop/hive Modify the name of the extracted file: mv a ...

Posted on Tue, 10 Dec 2019 16:49:08 -0500 by knsito

Step by step understanding Impala query profile

In part 3 of this blog series, I will still focus on Query Plan and Execution Summary, but will use real data( Kaggle's Flights Delay database ), to perform a more complex query. There are three tables in the test database: flights.csv airlines.csv airports.csv The query statement is as follows: SELECT a.airline as airline_name, COU ...

Posted on Mon, 09 Dec 2019 11:41:33 -0500 by Ryanz

Ubuntu 18.04.1 lts installation Hive

hive is a data warehouse tool based on Hadoop, which can map structured data files into a database table, and can convert SQL statements into MapReduce tasks to run. Its advantage is that it can quickly implement simple MapReduce statistics through SQL like statements. Install hive 1. Download and extract hive source program ...

Posted on Mon, 09 Dec 2019 11:36:11 -0500 by jeancharles