mongoDb replica set
It mainly provides two functions
1. When data is written to the Primary node, copy the data to another replica node
2. When the primary node fails, a new replacement node is automatically selected
Typical replica set structure
A typical replication set consists of three or more nodes with voting rights. One Primary node receives write operations, read operations and voting during election, and two or more secondary nodes replicate new data on the Primary node and voting during election
How is data replicated
When a modification operation, whether insert, update or delete, reaches the master node, its operation on the data will be recorded (after some necessary transformations). These records are called oplog
The slave node continuously obtains the oplog of the new master node from the master node and plays it back on its own data, so as to keep consistent with the data of the master node.
It is very similar to the AOF method in redis persistence.
Replication set setup
1. Create data directory file
mkdir -p /data/db{1,2,3}
2. Prepare the configuration file of each database (each mongod process of the replication set should be located on a different server. We now run three instances on one server, so we need to configure them respectively)
1.Different ports: 280172801828019. 2.Different data directories data/db1,data/db2,data/db3 3. Different log file paths. Used in instances /data/db1/mongod.log /data/db2/mongod.log /data/db3/mongod.log touch mongod.log
Then edit the mongod.conf file
data/db1/mongod.conf
systemLog: destination: file path: /data/db1/mongod.log logAppend: true storage: dbPath: /data/db1 net: bindIp: 0.0.0.0 port: 28017 replication: replSetName: rs0 processManagement: fork: true
/data/db2/mongod.conf
systemLog: destination: file path: /data/db2/mongod.log logAppend: true storage: dbPath: /data/db2 net: bindIp: 0.0.0.0 port: 28018 replication: replSetName: rs0 processManagement: fork: true
/data/db3/mongod.conf
systemLog: destination: file path: /data/db3/mongod.log logAppend: true storage: dbPath: /data/db3 net: bindIp: 0.0.0.0 port: 28019 replication: replSetName: rs0 processManagement: fork: true
Start separately
mongod -f /data/db1/mongod.conf
mongod -f /data/db2/mongod.conf
mongod -f /data/db3/mongod.conf
Link the first mongo --port 28017
It is found that the three machines are still independent of each other. At this time, configure them to form a cluster
rs.initiate({
_id:"rs0",
members:[{
_id:0,
host:"localhost:28017"
},{
_id:1,
host:"localhost:28018"
},{
_id:2,
host:"localhost:28019"
}]
})View the replica set status through rs.status()
It is found that the port 28017 is the master node and the other two are the slave nodes.
By default, non primary nodes are not allowed to read data.
You can turn on the read permission by executing rs.secondaryOk()
MongoDB partitioned cluster
Composition of fragment cluster
Build a cluster of 2 slices
1. Create a data directory: it is prepared for two replica sets. Each replica set has three instances and a total of 6 data nodes.
mkdir -p /data/shard1 /data/shard1second1 /data/shard1second2 /data/shard2 /data/shard2second1 /data/shard2second2
2. Create 6 log files in total
touch /data/shard1/mongod.log /data/shard1second1/mongod.log /data/shard1second2/mongod.log /data/shard2/mongod.log /data/shard2second1/mongod.log /data/shard2second2/mongod.log
3. Start the first mongod sharding instance (three instances in total)
mongod --bind_ip 0.0.0.0 --replSet shard1 --dbpath /data/shard1 --logpath /data/shard1/mongod.log --port 27010 --fork --shardsvr
mongod --bind_ip 0.0.0.0 --replSet shard1 --dbpath /data/shard1second1 --logpath /data/shard1second1/mongod.log --port 27011 --fork --shardsvr
mongod --bind_ip 0.0.0.0 --replSet shard1 --dbpath /data/shard1second2 --logpath /data/shard1second2/mongod.log --port 27012 --fork --shardsvr
4. After the mongod instances of the first shard are started, add them to the replication set
rs.initiate(
{_id:"shard1",
"members":[
{"_id":0,"host":"192.168.120.101:27010"},
{"_id":1,"host":"192.168.120.101:27011"},
{"_id":2,"host":"192.168.120.101:27012"}
]
});Configure Config replication set: there are three instances in total
1. Create data directory:
mkdir -p /data/config /data/configsecond1 /data/configsecond2
2. Create log file
touch /data/config/mongod.log /data/configsecond1/mongod.log /data/configsecond2/mongod.log
3. Start the configuration replication set
mongod --bind_ip 0.0.0.0 --replSet config --dbpath /data/config --logpath /data/config/mongod.log --port 37010 --fork --configsvr
mongod --bind_ip 0.0.0.0 --replSet config --dbpath /data/configsecond1 --logpath /data/configsecond1/mongod.log --port 37011 --fork --configsvr
mongod --bind_ip 0.0.0.0 --replSet config --dbpath /data/configsecond2 --logpath /data/configsecond2/mongod.log --port 37012 --fork --configsvr
4. Configure the replica set for initialization
rs.initiate(
{_id:"config",
"members":[
{"_id":0,"host":"192.168.120.101:37010"},
{"_id":1,"host":"192.168.120.101:37011"},
{"_id":2,"host":"192.168.120.101:37012"}
]
});Configure mongs routing node
1. To start the mongos instance, you need to specify the address list of the configuration server
mongos --bind_ip 0.0.0.0 --logpath /data/mongos/mongos.log --port 4000 --fork --configdb config/192.168.120.101:37010,192.168.120.101:37011,192.168.120.101:37012
Where configdb is the address list of the configuration server
2. Connect to mongos and add Shards
mongo --port 4000 local direct connection
Execute script
sh.addShard("shard1/192.168.120.101:27010,192.168.120.101:27011,192.168.120.101:27012");View slice status
sh.status();
To create a shard table:
1. MongoDB sharding is set based. Even if there is a sharding cluster, the data will be automatically sharded, and the sharding table needs to be displayed
First, you need to enable database fragmentation
sh.enableSharding("library name");
For example:
sh.enableSharding("order");
sh.shardCollection("library name. Collection name", {_id: "hashed"});
sh.shardCollection("order.account",{_id: "hashed"});
Dynamic capacity expansion
Create a second replica set to implement sharding
mongod --bind_ip 0.0.0.0 --replSet shard2 --dbpath /data/shard2 --logpath /data/shard2/mongod.log --port 27013 --fork --shardsvr
mongod --bind_ip 0.0.0.0 --replSet shard2 --dbpath /data/shard2second1 --logpath /data/shard2second1/mongod.log --port 27014 --fork --shardsvr
mongod --bind_ip 0.0.0.0 --replSet shard2 --dbpath /data/shard2second2 --logpath /data/shard2second2/mongod.log --port 27015 --fork --shardsvr
Initialize replication set
rs.initiate(
{_id:"shard2",
"members":[
{"_id":0,"host":"192.168.120.101:27013"},
{"_id":1,"host":"192.168.120.101:27014"},
{"_id":2,"host":"192.168.120.101:27015"}
]
});
Join cluster partition
mongo --port 4000
sh.addShard("shard2/192.168.120.101:27013,192.168.120.101:27014,192.168.120.101:27015");MongoDB application practice
1. java Native Client
Introducing maven
<dependencies>
<dependency>
<groupId>org.mongodb</groupId>
<artifactId>mongodb-driver-sync</artifactId>
<version>4.1.1</version>
</dependency>
</dependencies>public class QuickStart {
public static void main(String[] args) {
// Mongod connecting to the local default port
// MongoClient mongoClient = MongoClients.create()
// Mongod that connects to the specified port of the remote service
// MongoClient mongoClient = MongoClients.create("mongodb://host1:27017");
// Connect to the specified port replication set
// MongoClient mongoClient = MongoClients.create("mongodb://host1:27017,host2:27017,host3:27017/?replicaSet=myReplicaSet");
// Connect mongos route: connect one
// MongoClient mongoClient = MongoClients.create( "mongodb://localhost:27017" );
// Connecting multiple mongos routes
MongoClient mongoClient = MongoClients.create("mongodb://111.229.189.98:4000");
//Get database
MongoDatabase database = mongoClient.getDatabase("productdb");
// Get collection
MongoCollection<Document> productdesc=database.getCollection( "productdesc" );
Document doc = new Document("name", "MongoDB")
.append("type", "database")
.append("count", 1)
.append("versions", Arrays.asList("v3.2", "v3.0", "v2.6"))
.append("info", new Document("x", 203).append("y", 102));
productdesc.insertOne(doc);
Bson eq = eq("name", "MongoDB");
FindIterable<Document> find = productdesc.find(eq);
Document first=find.first();
System.out.println(first);
}
}
Spring Boot integration:
1. Introduce maven
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>2.4.0</version>
<relativePath/> <!-- lookup parent from repository -->
</parent>
<groupId>com.example</groupId>
<artifactId>demo</artifactId>
<version>0.0.1-SNAPSHOT</version>
<name>demo</name>
<description>Demo project for Spring Boot</description>
<properties>
<java.version>1.8</java.version>
</properties>
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-data-mongodb</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.projectlombok</groupId>
<artifactId>lombok</artifactId>
<version>1.18.12</version>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
</plugin>
</plugins>
</build>
</project>
2. Add configuration class
For example, connect to mongos above and use MongoTemplate for database operation.
package com.example.demo.config;
import com.mongodb.client.MongoClient;
import com.mongodb.client.MongoClients;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.data.mongodb.core.MongoTemplate;
@Configuration
public class AppConfig {
public @Bean
MongoClient mongoClient() {
return MongoClients.create("mongodb://111.229.189.98:4000");
}
public @Bean
MongoTemplate mongoTemplate() {
return new MongoTemplate(mongoClient(), "productdb");
}
}
3. Testing
package com.example.demo;
public class Person {
private String id;
private String name;
private int age;
public Person(String name, int age) {
this.name = name;
this.age = age;
}
public String getId() {
return id;
}
public String getName() {
return name;
}
public int getAge() {
return age;
}
@Override
public String toString() {
return "Person [id=" + id + ", name=" + name + ", age=" + age + "]";
}
}
package com.example.demo;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.ApplicationArguments;
import org.springframework.boot.ApplicationRunner;
import org.springframework.data.mongodb.core.MongoTemplate;
import org.springframework.stereotype.Component;
import java.util.List;
import static org.springframework.data.mongodb.core.query.Criteria.where;
import static org.springframework.data.mongodb.core.query.Query.query;
import static org.springframework.data.mongodb.core.query.Update.update;
@Component
@Slf4j
public class ApplicationRunnerTest implements ApplicationRunner{
@Autowired
private MongoTemplate mongoOps;
@Override
public void run(ApplicationArguments applicationArguments) throws Exception {
Person p = new Person("Joe", 34);
// Insert document
mongoOps.insert(p);
log.info("Insert: " + p);
// consult your documentation
p = mongoOps.findById(p.getId(), Person.class);
log.info("Found: " + p);
// Update document
mongoOps.updateFirst(query(where("name").is("Joe")), update("age", 35), Person.class);
p = mongoOps.findOne(query(where("name").is("Joe")), Person.class);
log.info("Updated: " + p);
// remove document
mongoOps.remove(p);
// Check that deletion worked
List<Person> people = mongoOps.findAll(Person.class);
log.info("Number of people = : " + people.size());
mongoOps.dropCollection(Person.class);
}
}