mongoDB replica set and partitioned cluster

mongoDb replica set

It mainly provides two functions

1. When data is written to the Primary node, copy the data to another replica node

2. When the primary node fails, a new replacement node is automatically selected

Typical replica set structure

A typical replication set consists of three or more nodes with voting rights. One Primary node receives write operations, read operations and voting during election, and two or more secondary nodes replicate new data on the Primary node and voting during election

How is data replicated

When a modification operation, whether insert, update or delete, reaches the master node, its operation on the data will be recorded (after some necessary transformations). These records are called oplog

The slave node continuously obtains the oplog of the new master node from the master node and plays it back on its own data, so as to keep consistent with the data of the master node.

It is very similar to the AOF method in redis persistence.

Replication set setup

  1. Create data directory file

mkdir -p /data/db{1,2,3}

 

2. Prepare the configuration file of each database (each mongod process of the replication set should be located on a different server. We now run three instances on one server, so we need to configure them respectively)

1.Different ports: 280172801828019.
2.Different data directories
data/db1,data/db2,data/db3
3. Different log file paths. Used in instances
/data/db1/mongod.log
/data/db2/mongod.log
/data/db3/mongod.log   touch mongod.log

Then edit the mongod.conf file  

data/db1/mongod.conf

systemLog:
  destination: file
  path: /data/db1/mongod.log
  logAppend: true
storage:
  dbPath: /data/db1
net:
  bindIp: 0.0.0.0
  port: 28017
replication:
  replSetName: rs0
processManagement:
  fork: true

/data/db2/mongod.conf

systemLog:
  destination: file
  path: /data/db2/mongod.log
  logAppend: true
storage:
  dbPath: /data/db2
net:
  bindIp: 0.0.0.0
  port: 28018
replication:
  replSetName: rs0
processManagement:
  fork: true

/data/db3/mongod.conf

systemLog:
  destination: file
  path: /data/db3/mongod.log
  logAppend: true
storage:
  dbPath: /data/db3
net:
  bindIp: 0.0.0.0
  port: 28019
replication:
  replSetName: rs0
processManagement:
  fork: true

  Start separately

mongod -f /data/db1/mongod.conf

mongod -f /data/db2/mongod.conf

mongod -f /data/db3/mongod.conf

Link the first mongo --port 28017

It is found that the three machines are still independent of each other. At this time, configure them to form a cluster

rs.initiate({

_id:"rs0",
members:[{
_id:0,
host:"localhost:28017"
},{
_id:1,
host:"localhost:28018"
},{
_id:2,
host:"localhost:28019"
}]
})

View the replica set status through rs.status()

 

  It is found that the port 28017 is the master node and the other two are the slave nodes.

By default, non primary nodes are not allowed to read data.

You can turn on the read permission by executing rs.secondaryOk()

 

MongoDB partitioned cluster

Composition of fragment cluster

  Build a cluster of 2 slices

1. Create a data directory: it is prepared for two replica sets. Each replica set has three instances and a total of 6 data nodes.  

mkdir -p /data/shard1    /data/shard1second1     /data/shard1second2     /data/shard2    /data/shard2second1     /data/shard2second2   

2. Create 6 log files in total

touch  /data/shard1/mongod.log     /data/shard1second1/mongod.log    /data/shard1second2/mongod.log    /data/shard2/mongod.log     /data/shard2second1/mongod.log    /data/shard2second2/mongod.log 

  3. Start the first mongod sharding instance (three instances in total)

mongod --bind_ip 0.0.0.0 --replSet shard1 --dbpath /data/shard1 --logpath /data/shard1/mongod.log --port 27010   --fork   --shardsvr

mongod --bind_ip 0.0.0.0 --replSet shard1 --dbpath /data/shard1second1  --logpath /data/shard1second1/mongod.log --port 27011   --fork   --shardsvr
mongod --bind_ip 0.0.0.0 --replSet shard1 --dbpath /data/shard1second2 --logpath /data/shard1second2/mongod.log --port 27012   --fork   --shardsvr

4. After the mongod instances of the first shard are started, add them to the replication set

rs.initiate(
{_id:"shard1",
    "members":[ 
                {"_id":0,"host":"192.168.120.101:27010"},
                {"_id":1,"host":"192.168.120.101:27011"},
                {"_id":2,"host":"192.168.120.101:27012"}
                ]
});

Configure Config replication set: there are three instances in total

1. Create data directory:

mkdir -p /data/config    /data/configsecond1     /data/configsecond2

  2. Create log file

touch  /data/config/mongod.log     /data/configsecond1/mongod.log    /data/configsecond2/mongod.log 

3. Start the configuration replication set

mongod --bind_ip 0.0.0.0 --replSet config --dbpath /data/config  --logpath /data/config/mongod.log --port 37010   --fork   --configsvr 
mongod --bind_ip 0.0.0.0 --replSet config --dbpath /data/configsecond1 --logpath /data/configsecond1/mongod.log --port 37011   --fork   --configsvr 
mongod --bind_ip 0.0.0.0 --replSet config --dbpath /data/configsecond2  --logpath /data/configsecond2/mongod.log --port 37012   --fork   --configsvr 

4. Configure the replica set for initialization

  rs.initiate(
    {_id:"config",
        "members":[ 
                    {"_id":0,"host":"192.168.120.101:37010"},
                    {"_id":1,"host":"192.168.120.101:37011"},
                    {"_id":2,"host":"192.168.120.101:37012"}
                    ]
    });

Configure mongs routing node

1. To start the mongos instance, you need to specify the address list of the configuration server

mongos --bind_ip 0.0.0.0 --logpath /data/mongos/mongos.log --port 4000 --fork --configdb config/192.168.120.101:37010,192.168.120.101:37011,192.168.120.101:37012

  Where configdb is the address list of the configuration server

2. Connect to mongos and add Shards

mongo --port 4000 local direct connection

Execute script

sh.addShard("shard1/192.168.120.101:27010,192.168.120.101:27011,192.168.120.101:27012");

View slice status

sh.status();

To create a shard table:

  1. MongoDB sharding is set based. Even if there is a sharding cluster, the data will be automatically sharded, and the sharding table needs to be displayed

First, you need to enable database fragmentation

sh.enableSharding("library name");

For example:

sh.enableSharding("order");

sh.shardCollection("library name. Collection name", {_id: "hashed"});

sh.shardCollection("order.account",{_id: "hashed"});

Dynamic capacity expansion

  Create a second replica set to implement sharding

mongod --bind_ip 0.0.0.0 --replSet shard2 --dbpath /data/shard2 --logpath /data/shard2/mongod.log --port 27013   --fork   --shardsvr

mongod --bind_ip 0.0.0.0 --replSet shard2 --dbpath /data/shard2second1  --logpath /data/shard2second1/mongod.log --port 27014   --fork   --shardsvr
mongod --bind_ip 0.0.0.0 --replSet shard2 --dbpath /data/shard2second2 --logpath /data/shard2second2/mongod.log --port 27015   --fork   --shardsvr

Initialize replication set

rs.initiate(
{_id:"shard2",
    "members":[ 
                {"_id":0,"host":"192.168.120.101:27013"},
                {"_id":1,"host":"192.168.120.101:27014"},
                {"_id":2,"host":"192.168.120.101:27015"}
                ]
});

Join cluster partition

mongo --port 4000
sh.addShard("shard2/192.168.120.101:27013,192.168.120.101:27014,192.168.120.101:27015");

MongoDB application practice  

1. java Native Client

Introducing maven

<dependencies>
    <dependency>
        <groupId>org.mongodb</groupId>
        <artifactId>mongodb-driver-sync</artifactId>
        <version>4.1.1</version>
    </dependency>
</dependencies>
public class QuickStart {

    public static void main(String[] args) {
        // Mongod connecting to the local default port
        // MongoClient mongoClient = MongoClients.create()
        // Mongod that connects to the specified port of the remote service
        // MongoClient mongoClient = MongoClients.create("mongodb://host1:27017");
        // Connect to the specified port replication set
        // MongoClient mongoClient = MongoClients.create("mongodb://host1:27017,host2:27017,host3:27017/?replicaSet=myReplicaSet");
        // Connect mongos route: connect one
        // MongoClient mongoClient = MongoClients.create( "mongodb://localhost:27017" );
        // Connecting multiple mongos routes
        MongoClient mongoClient = MongoClients.create("mongodb://111.229.189.98:4000");


        //Get database
        MongoDatabase database = mongoClient.getDatabase("productdb");

        // Get collection
        MongoCollection<Document> productdesc=database.getCollection( "productdesc" );

        Document doc = new Document("name", "MongoDB")
                .append("type", "database")
                .append("count", 1)
                .append("versions", Arrays.asList("v3.2", "v3.0", "v2.6"))
                .append("info", new Document("x", 203).append("y", 102));

        productdesc.insertOne(doc);


        Bson eq = eq("name", "MongoDB");
        FindIterable<Document> find = productdesc.find(eq);
        Document first=find.first();
        System.out.println(first);

    }
}

 

 

Spring Boot integration:

1. Introduce maven

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
   xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
   <modelVersion>4.0.0</modelVersion>
   <parent>
      <groupId>org.springframework.boot</groupId>
      <artifactId>spring-boot-starter-parent</artifactId>
      <version>2.4.0</version>
      <relativePath/> <!-- lookup parent from repository -->
   </parent>
   <groupId>com.example</groupId>
   <artifactId>demo</artifactId>
   <version>0.0.1-SNAPSHOT</version>
   <name>demo</name>
   <description>Demo project for Spring Boot</description>

   <properties>
      <java.version>1.8</java.version>
   </properties>

   <dependencies>
      <dependency>
         <groupId>org.springframework.boot</groupId>
         <artifactId>spring-boot-starter-data-mongodb</artifactId>
      </dependency>
      <dependency>
         <groupId>org.springframework.boot</groupId>
         <artifactId>spring-boot-starter-web</artifactId>
      </dependency>

      <dependency>
         <groupId>org.springframework.boot</groupId>
         <artifactId>spring-boot-starter-test</artifactId>
         <scope>test</scope>
      </dependency>
      <dependency>
         <groupId>org.projectlombok</groupId>
         <artifactId>lombok</artifactId>
         <version>1.18.12</version>
      </dependency>


   </dependencies>

   <build>
      <plugins>
         <plugin>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-maven-plugin</artifactId>
         </plugin>
      </plugins>
   </build>

</project>

 

  2. Add configuration class

For example, connect to mongos above and use MongoTemplate for database operation.

package com.example.demo.config;

import com.mongodb.client.MongoClient;
import com.mongodb.client.MongoClients;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.data.mongodb.core.MongoTemplate;

@Configuration
public class AppConfig {

    public @Bean
    MongoClient mongoClient() {
        return MongoClients.create("mongodb://111.229.189.98:4000");
    }

    public @Bean
    MongoTemplate mongoTemplate() {
        return new MongoTemplate(mongoClient(), "productdb");
    }
}

 

3. Testing

package com.example.demo;

public class Person {

  private String id;
  private String name;
  private int age;

  public Person(String name, int age) {
    this.name = name;
    this.age = age;
  }

  public String getId() {
    return id;
  }
  public String getName() {
    return name;
  }
  public int getAge() {
    return age;
  }

  @Override
  public String toString() {
    return "Person [id=" + id + ", name=" + name + ", age=" + age + "]";
  }

}

package com.example.demo;

import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.ApplicationArguments;
import org.springframework.boot.ApplicationRunner;
import org.springframework.data.mongodb.core.MongoTemplate;
import org.springframework.stereotype.Component;

import java.util.List;

import static org.springframework.data.mongodb.core.query.Criteria.where;
import static org.springframework.data.mongodb.core.query.Query.query;
import static org.springframework.data.mongodb.core.query.Update.update;
 
@Component
@Slf4j
public class ApplicationRunnerTest implements ApplicationRunner{

    @Autowired
    private MongoTemplate mongoOps;

    @Override
    public void run(ApplicationArguments applicationArguments) throws Exception {

        Person p = new Person("Joe", 34);

          // Insert document
        mongoOps.insert(p);
        log.info("Insert: " + p);

        // consult your documentation
        p = mongoOps.findById(p.getId(), Person.class);
        log.info("Found: " + p);

        // Update document
        mongoOps.updateFirst(query(where("name").is("Joe")), update("age", 35), Person.class);
        p = mongoOps.findOne(query(where("name").is("Joe")), Person.class);
        log.info("Updated: " + p);

        // remove document
        mongoOps.remove(p);

        // Check that deletion worked
        List<Person> people =  mongoOps.findAll(Person.class);
        log.info("Number of people = : " + people.size());
        mongoOps.dropCollection(Person.class);
    }
}

 

 

 

 

 

 

Tags: MongoDB nosql

Posted on Wed, 01 Dec 2021 06:28:12 -0500 by skippy111