Step 1. Data source
Find the log data source of spark. In the directory / root / spark / spark-2.0.2-bin-hadoop 2.7/logs:
By reading the file, count the number of data:
val masterLog = sc.textFile("file:///root/spark/spark-2.0.2-bin-hadoop2.7/logs/spark-root-org.apache.spark.deploy.master.Master-1-hadoop000.out")
val workLog = ...
Posted on Wed, 11 Dec 2019 14:33:55 -0500 by latvaustin
There is a table partitioned, one partition per day.
There is a query on this table, often only query the data of one day in the table, but almost all the data of the whole partition is scanned every time. What can I do to optimize it?
1. Scenes to be optimized
There is a large table that generates about 1 million data per day, s ...
Posted on Wed, 11 Dec 2019 01:14:54 -0500 by exec1
1. Because the user-defined result set is used in the automatically generated Mapper.xml file (that is, the statistical function only takes a few of the fields, but other fields will still have fields after they are converted to json data, but the value is null, which makes it difficult to select the data.) The effect is not ideal.
So define a ...
Posted on Wed, 11 Dec 2019 00:52:53 -0500 by lszanto
1. Peng GE has written java project supermarket management system project in front of him, Portal
2. received many friends' personal letters to me, and I also appreciated the feedback and communication from the old fellow. The previous project was just a basic knowledge and object-oriented thinking exercise for java, but it didn't i ...
Posted on Tue, 10 Dec 2019 21:42:38 -0500 by YappyDog
In the use of EF, we often use xx. Where (P = > p.name = "Zhang San") to query data. This is because the EF framework will turn these C ා codes into Sql statements. The expression tree is mainly used. Today we will learn about the expression tree.
Understanding expression tree
Func<int,int,int> func = (a, b) => a ...
Posted on Tue, 10 Dec 2019 12:05:18 -0500 by ari_aaron
In ORACLE database, we will use some SQL statements to find out the problem of implicit conversion SQL. One of the SQL statements circulating on the Internet is as follows. Query whether there is an internal function in the filter prediction field of V $SQL plan:
Posted on Tue, 10 Dec 2019 09:14:09 -0500 by GoncaloF
The text and pictures of this article are from the Internet, only for learning and communication, not for any commercial purpose. The copyright belongs to the original author. If you have any questions, please contact us in time for handling.
Author: A prefix
PS: if you need Python learning materials, you can click the link below to get ...
Posted on Tue, 10 Dec 2019 06:01:23 -0500 by wallabee
The use of select case when in oracle novice learning
when Condition 1 then action1
when Condition 2 then action2
when Condition 3 then action3
when condition N then actionN
Judge the month
SQL> select case substr('20181118',5,2)
2 when '08' then '8yue'
3 when ' ...
Posted on Tue, 10 Dec 2019 02:47:04 -0500 by pokemon
Details of oracle connections
The default number of oracle connections is 150. When the number of connections at the same time is greater than this number, an error will be reported. The current solution is to increase the number of oracle connections from 150 to 1500. The operation is as follows
1. Problem solving before modification
If the nu ...
Posted on Mon, 09 Dec 2019 22:19:48 -0500 by 2levelsabove
As shown in the following example, there is a table t2 with the same name in different schema of the database, but only one T2 table is queried when using the \ d command.
postgres=# select oid,nspname from pg_namespace;
oid | nspname
99 | pg_toast
11736 | pg_temp_1
11737 | pg_toast_temp_ ...
Posted on Mon, 09 Dec 2019 20:29:05 -0500 by cnagra