NIO source code analysis - FileChannel

preface:

FileChannel is a channel connected to files. We can read and write files through FileChannel. In the previous way of reading and writing files, we mainly used the streaming mode of InputStream and OutputStream.

This article will introduce the common APIs of FileChannel.

1. Basic structure of filechannel

Through its class structure diagram, we can see that FileChannel implements the read-write operation of files and is set to be interruptible. Let's learn more about its API.

2.FileChannel API

2.1 creation of filechannel

File file = new File("D:\\test.txt");

// 1. Create through RandomAccessFile
RandomAccessFile raFile = new RandomAccessFile(file, "rwd");
FileChannel channel = raFile.getChannel();

// 2. Create through FileInputStream
FileInputStream fileInputStream = new FileInputStream(file);
FileChannel inputStreamChannel = fileInputStream.getChannel();

// 3. Create through FileOutputStream
FileOutputStream fileOutputStream = new FileOutputStream(file);
FileChannel outputStreamChannel = fileOutputStream.getChannel();

  What are the specific differences between filechannels created in these three ways? We compare the source code

// RandomAccessFile.getChannel()
channel = FileChannelImpl.open(fd, path, true, rw, this);

// FileInputStream.getChannel()
channel = FileChannelImpl.open(fd, path, true, false, this);

// FileOutputStream.getChannel()
channel = FileChannelImpl.open(fd, path, false, true, append, this);

// FileChannelImpl construction method
private FileChannelImpl(FileDescriptor var1, String var2, boolean var3, boolean var4, boolean var5, Object var6) {
    this.fd = var1;
    this.readable = var3;
    this.writable = var4;
    this.append = var5;
    this.parent = var6;
    this.path = var2;
    this.nd = new FileDispatcherImpl(var5);
}

Through the private construction method of FileChannelImpl, we can know whether var3 parameter corresponds to readability and var4 corresponds to Writeability.

In combination with the parameter of filechannelimplementation passed in during fileinputstream.getchannel and fileoutputstream.getchannel, the following results can be obtained:

Acquisition methodDo you have file read and write permissions
RandomAccessFile.getChannelWhether it is readable or writable depends on the incoming mode
FileInputStream.getChannelReadable, not writable
FileOutputStream.getChannelWritable, unreadable

In addition, FileChannel also provides an open() static method, which can also be obtained through this method, but this method is not very common, and I will not elaborate.

2.2 mode of RandomAccessFile

There are two parameters in the construction method of RandomAccessFile, corresponding to file reference and mode (mode).

What are the specific values of mode? Let's look directly at the source code

public RandomAccessFile(File file, String mode)
    throws FileNotFoundException {
    
    String name = (file != null ? file.getPath() : null);
    int imode = -1;
    // Read read only mode
    if (mode.equals("r"))
        imode = O_RDONLY;
    // rw read and write mode
    else if (mode.startsWith("rw")) {
        imode = O_RDWR;
        rw = true;
        if (mode.length() > 2) {
            // And s and d, respectively, correspond to O_SYNC O_DSYNC
            if (mode.equals("rws"))
                imode |= O_SYNC;
            else if (mode.equals("rwd"))
                imode |= O_DSYNC;
            else
                imode = -1;
        }
    }
    ...
    fd = new FileDescriptor();
    fd.attach(this);
    path = name;
    open(name, imode);
}

O_ SYNC O_ What does dsync stand for?

The author directly extracts a paragraph of explanation from the Internet (from https://zhuanlan.zhihu.com/p/104994838 )

Since the read and write speed of memory is several orders of magnitude faster than that of disk, in order to make up for the disk IO Low performance, Linux The kernel introduces page caching(PageCache). We pass Linux system call(open--->write)When writing a file, the kernel will first copy the data from the user state buffer to the PageCache It returns success directly, and then the kernel returns the dirty page according to a certain policy Flush To disk, we call it write back. 

write The data written is in memory PageCache Once the kernel changes Crash Or machines Down Data loss will occur. For distributed storage, data reliability is very important, so we need to write After completion, call fsync perhaps fdatasync Persist data to disk.
write back It reduces the number of writes to the disk, but reduces the update speed of file disk data, and there is a risk of losing updated data. To ensure that disk files and data PageCache Data consistency, Linux Provided sync,fsync,msync,fdatasync,sync_file_range5 A function.
open Functional O_SYNC and O_DSYNC Parameters have and fsync and fdatasync Similar meaning: make every time write Will block the disk IO Done.

O_SYNC: Make every time write Operation blocking waiting disk IO Upon completion, the file data and file attributes are updated.
O_DSYNC: Make every time write Operation blocking waiting disk IO Complete, but if the write operation does not affect reading the data just written, you do not need to wait for the file properties to be updated.

O_DSYNC and O_SYNC Signs have subtle differences:
File to O_DSYNC When the flag is on, the flag affects file properties only when the file properties need to be updated to reflect changes in file data (for example, updating the file size to reflect that the file contains more data). When overwriting part of its existing content, the file time property is not updated synchronously.
File to O_SYNC When the flag is turned on, data and properties are always updated synchronously. For each of the files write Will be in write The time when the file was updated before returning, regardless of whether to overwrite existing bytes or append the file. be relative to fsync/fdatasync,This setting is not flexible enough and should be rarely used.

actually: Linux yes O_SYNC,O_DSYNC The same processing was done, but it was not satisfied POSIX But all of them have been realized fdatasync Semantics of.

It is precisely because of the difference in read and write speeds between memory and disk that the write method is available. It is only an optimization method to write data to pageCache. At the same time, the operating system also provides O_SYNC and O_DSYNC to ensure that the data is brushed into the disk.

2.3 write related methods

// 1. Write a single ByteBuffer to FileChannel
public abstract int write(ByteBuffer src) throws IOException;

// 2. Write batch ByteBuffer. Offset is the offset of ByteBuffer
public abstract long write(ByteBuffer[] srcs, int offset, int length)
        throws IOException;

// 3. The same as 2, offset is 0
public final long write(ByteBuffer[] srcs) throws IOException {
    return write(srcs, 0, srcs.length);
}

Standard write mode:

File file = new File("D:\\test.txt");

// 1. Create through RandomAccessFile
RandomAccessFile raFile = new RandomAccessFile(file, "rwd");
FileChannel channel = raFile.getChannel();

ByteBuffer byteBuffer = ByteBuffer.allocate(100);

String text = "When grace is lost from life, come with a burst of song";
byteBuffer.put(text.getBytes());

byteBuffer.flip();
// Write data
while (byteBuffer.hasRemaining()) {
    channel.write(byteBuffer);
}

Note: the write method is done in the while loop, because there is no guarantee how many bytes the write method writes to FileChannel at a time

2.4 read related methods

// 1. Read the file contents into a single ByteBuffer
public abstract int read(ByteBuffer dst) throws IOException;

// 2. Read the file contents into ByteBuffer [], and the offset of ByteBuffer is the specified value
public abstract long read(ByteBuffer[] dsts, int offset, int length)
        throws IOException;

// 3. The same as 2
public final long read(ByteBuffer[] dsts) throws IOException {
    return read(dsts, 0, dsts.length);
}

Standard reading method:

File file = new File("D:\\test.txt");

// 1. Create through RandomAccessFile
RandomAccessFile raFile = new RandomAccessFile(file, "rwd");
FileChannel channel = raFile.getChannel();

ByteBuffer byteBuffer = ByteBuffer.allocate(100);
// Actually read readCount bytes
int readCount = channel.read(byteBuffer);

byteBuffer.flip();
byte[] array = byteBuffer.array();
// Write the read content to the String
String s = new String(array);
// The result is the value written in the 2.3 write method just now
System.out.println(s);

2.5 force method

public abstract void force(boolean metaData) throws IOException;

As mentioned in previous 2.2, the write method may only write to the PageCache. If the system crashes at this time, the data that only exists in the PageCache but not flushed to the disk may be lost. Using the force method, we can force the file content and metadata information (the parameter boolean metaData is used to decide whether to write metadata to the disk). This method is very critical for some key operations, such as transaction operations. Using force method can ensure data integrity and reliable recovery.

2.6 lock related methods

// 1. Starting from the position position of file, the lock length is size, and the lock category is shared lock (true) or exclusive lock (false)
public abstract FileLock lock(long position, long size, boolean shared)
        throws IOException;

// 2. The same as 1, basically monopolizing the whole file
public final FileLock lock() throws IOException {
    return lock(0L, Long.MAX_VALUE, false);
}

// 3. Same as 1, try to lock the file
public abstract FileLock tryLock(long position, long size, boolean shared)
        throws IOException;

// 4. As in 2, try to lock the file
public final FileLock tryLock() throws IOException {
    return tryLock(0L, Long.MAX_VALUE, false);
}

First, we need to understand that locking is for the file itself, not the Channel or thread.

FileLock can be shared or exclusive.

The implementation of lock largely depends on the implementation of local operating system. When the operating system does not support shared locks, it will actively upgrade the shared locks to exclusive locks.

// Test FileLock through two processes
FileLock lock = null;
try {
    File file = new File("D:\\test.txt");

    // 1. Create through RandomAccessFile
    RandomAccessFile raFile = new RandomAccessFile(file, "rwd");
    FileChannel channel = raFile.getChannel();

    // Actively set exclusive lock or shared lock
    lock = channel.lock(0, Integer.MAX_VALUE, true);
    System.out.println(lock);
} catch (FileNotFoundException e) {
    e.printStackTrace();
} catch (IOException e) {
    e.printStackTrace();
} finally {
    try {
        // Active release required
        lock.release();
    } catch (IOException e) {
        e.printStackTrace();
    }
}
Based on the test results using Windows machine: Support the sharing lock of two processes on the same file; Exclusive locks of two processes on the same file are not supported (one exclusive and one share is not allowed) )

Summary:

This paper mainly introduces the common API s of FileChannel. Based on FileChannel, we can read and write files.

FileChannel also has some advanced API s, such as map(), transferTo(), transferFrom(), which we will continue to introduce in the next blog.

reference resources:

https://zhuanlan.zhihu.com/p/104994838

 

 

 

 

 

Tags: html

Posted on Sat, 20 Nov 2021 12:12:57 -0500 by CowGuy