[MIT 6.S081] Lab 9: file system

Lab 9: file system

Lab Guidance: https://pdos.csail.mit.edu/6.828/2020/labs/fs.html
Lab Code: https://github.com/VastRock-Huang/xv6-labs-2020/tree/fs

Large files (moderate)

main points

inode structure to realize double indirect block index: the first 11 elements of IP - > addrs [] are direct blocks, the 12th element is a single indirect block, and the 13th element is a double indirect block
Modify the bmap() function to fit the double indirect block

step

Modify the macro definition of direct block number in kernel/fs.h, and NDIRECT is 11
According to the experimental requirements, the original 12 direct block numbers in inode were modified to 11

#define NDIRECT 11  // lab9-1

Modify the block number array of inode related structures, including the addrs field of disk inode structure struct dinode in kernel/fs.h; And the addrs field of the memory inode structure struct inode in kernel/file.h. set the array size of the two to NDIRECT+2, because the total number of block numbers of the actual inode does not change, but NDIRECT is reduced by 1

// On-disk inode structure
struct dinode {
  // ...
  uint addrs[NDIRECT+2];   // Data block addresses  // lab9-1
};

// in-memory copy of an inode
struct inode {
  // ...
  uint addrs[NDIRECT+2];    // lab9-1
};

Add a macro definition NDOUBLYINDIRECT in kernel/fs.h to represent the total number of secondary indirect block numbers, similar to NINDIRECT. Since it is secondary, the block number that can be represented should be the square of primary indirect block number NINDIRECT

#define NDOUBLYINDIRECT (NINDIRECT * NINDIRECT)     // lab9-1

Modify bmap() function in kernel/fs.c
This function is used to return the block number in the disk corresponding to the relative block number of inode
Since the first NDIRECT block number in the inode structure is consistent with that before modification, it is only necessary to add the processing code for the secondary indirect index of the 13th NDIRECT block. The processing method is similar to that for the 1st NDIRECT block number, which only needs to be indexed twice

static uint
bmap(struct inode *ip, uint bn)
{
  uint addr, *a;
  struct buf *bp;

  if(bn < NDIRECT){
    if((addr = ip->addrs[bn]) == 0)
      ip->addrs[bn] = addr = balloc(ip->dev);
    return addr;
  }
  bn -= NDIRECT;

  if(bn < NINDIRECT){
    // Load indirect block, allocating if necessary.
    if((addr = ip->addrs[NDIRECT]) == 0)
      ip->addrs[NDIRECT] = addr = balloc(ip->dev);
    bp = bread(ip->dev, addr);
    a = (uint*)bp->data;
    if((addr = a[bn]) == 0){
      a[bn] = addr = balloc(ip->dev);
      log_write(bp);
    }
    brelse(bp);
    return addr;
  }

  // doubly-indirect block - lab9-1
  bn -= NINDIRECT;
  if(bn < NDOUBLYINDIRECT) {
    // get the address of doubly-indirect block
    if((addr = ip->addrs[NDIRECT + 1]) == 0) {
      ip->addrs[NDIRECT + 1] = addr = balloc(ip->dev);
    }
    bp = bread(ip->dev, addr);
    a = (uint*)bp->data;
    // get the address of singly-indirect block
    if((addr = a[bn / NINDIRECT]) == 0) {
      a[bn / NINDIRECT] = addr = balloc(ip->dev);
      log_write(bp);
    }
    brelse(bp);
    bp = bread(ip->dev, addr);
    a = (uint*)bp->data;
    bn %= NINDIRECT;
    // get the address of direct block
    if((addr = a[bn]) == 0) {
      a[bn] = addr = balloc(ip->dev);
      log_write(bp);
    }
    brelse(bp);
    return addr;
  }

  panic("bmap: out of range");
}

Modify itrunc() function in kernel/fs.c
This function is used to release inode data blocks
Since the structure of the secondary indirect block is added, it is also necessary to add the release code for the block in this part. The release method is the same as the structure of the primary indirect block number. It only needs two loops to traverse the secondary indirect block and the primary indirect block respectively

void
itrunc(struct inode *ip)
{
  int i, j, k;  // lab9-1
  struct buf *bp, *bp2;     // lab9-1
  uint *a, *a2; // lab9-1

  for(i = 0; i < NDIRECT; i++){
    if(ip->addrs[i]){
      bfree(ip->dev, ip->addrs[i]);
      ip->addrs[i] = 0;
    }
  }

  if(ip->addrs[NDIRECT]){
    bp = bread(ip->dev, ip->addrs[NDIRECT]);
    a = (uint*)bp->data;
    for(j = 0; j < NINDIRECT; j++){
      if(a[j])
        bfree(ip->dev, a[j]);
    }
    brelse(bp);
    bfree(ip->dev, ip->addrs[NDIRECT]);
    ip->addrs[NDIRECT] = 0;
  }
  // free the doubly-indirect block - lab9-1
  if(ip->addrs[NDIRECT + 1]) {
    bp = bread(ip->dev, ip->addrs[NDIRECT + 1]);
    a = (uint*)bp->data;
    for(j = 0; j < NINDIRECT; ++j) {
      if(a[j]) {
        bp2 = bread(ip->dev, a[j]);
        a2 = (uint*)bp2->data;
        for(k = 0; k < NINDIRECT; ++k) {
          if(a2[k]) {
            bfree(ip->dev, a2[k]);
          }
        }
        brelse(bp2);
        bfree(ip->dev, a[j]);
        a[j] = 0;
      }
    }
    brelse(bp);
    bfree(ip->dev, ip->addrs[NDIRECT + 1]);
    ip->addrs[NDIRECT + 1] = 0;
  }

  ip->size = 0;
  iupdate(ip);
}

Modify the macro definition MAXFILE of the maximum file size in kernel/fs.h. due to the addition of a secondary indirect block structure, the upper limit of the file size supported by xv6 naturally increases, so it should be modified to the correct value here

#define MAXFILE (NDIRECT + NINDIRECT + NDOUBLYINDIRECT) // lab9-1

Problems encountered

In xv6, execute bigfile to pass, but execute usertests with virtio_ disk_ The panic of intr status, as shown in the following figure:

Solution: the author is not very clear about the specific reason for the occurrence of this panic, but at the beginning, the author did not modify the NDIRECT to 11, but kept it at 12, but modified it in bmap() and itrunc(). However, after modifying the NDIRECT to 11, the panic will no longer exist

test

Execute bigfile in xv6:
Execute usertests in xv6:
. / grade lab FS bigfile single test:

Symbolic links (moderate)

main points

Add symlink (soft link) system call symlink
When the open system call is modified to handle symbolic links, and the target file of the symbolic link is still a symbolic link file, it is necessary to recursively find the target file
With O_NOFOLLOW opening symbolic links does not track to linked files
Other system calls do not track symbolic links, and then process the symbolic link file itself

step

Add the definition declaration of symlink system call, including kernel/syscall.h, kernel/syscall.c, user/usys.pl and user/user.h
Add new file type T_SYMLINK to kernel/stat.h
Add new file flag bit O_NOFOLLOW into kernel/fcntl.h
Implementing sys in kernel/sysfile.c_ Symlink() function
This function is used to generate symbolic link. Symbolic link is equivalent to a special independent file, and the stored data is the path of the target file
Therefore, in this function, first create the inode structure corresponding to the symbolic link path through create() (and use T_SYMLINK to distinguish it from ordinary files). Then write the path of the linked target file into the inode (block) through writei(). In this process, it is not necessary to judge whether the connected target path is valid
It should be noted that there are some lock release rules for the file system. The function create() will return the created inode and hold the lock of its inode at this time. The subsequent writei() needs to be written with the lock. After the operation is completed (whether successful or not), you need to call iunlockput() to release the lock of the inode and itself. The functions are iunlock() and iput() The former is to release the lock of inode; The latter is to reduce the reference of an inode (the corresponding field ref records the number of pointers to the inode in memory. The inode here is actually an inode in memory, which is allocated from the inode cache icache in memory and will be recycled to the icache when ref is 0), indicating that it is no longer necessary to hold the pointer of the inode to continue the operation

// lab9-2
uint64 sys_symlink(void) {
  char target[MAXPATH], path[MAXPATH];
  struct inode *ip;
  int n;

  if ((n = argstr(0, target, MAXPATH)) < 0
    || argstr(1, path, MAXPATH) < 0) {
    return -1;
  }

  begin_op();
  // create the symlink's inode
  if((ip = create(path, T_SYMLINK, 0, 0)) == 0) {
    end_op();
    return -1;
  }
  // write the target path to the inode
  if(writei(ip, 0, (uint64)target, 0, n) != n) {
    iunlockput(ip);
    end_op();
    return -1;
  }

  iunlockput(ip);
  end_op();
  return 0;
}

Modify sys of kernel/sysfile_ Open() function
This function is used to open the file. For symbolic links, generally, the target file of the link needs to be opened, so additional processing is required for symbolic link files
Considering that the operation of tracking symbolic links is relatively independent, the author writes an independent function follow_symlink() is used to find the target file of the symbolic link
When tracking symbolic links, we need to take into account that the target of the symbolic link may still be a symbolic link. At this time, we need to recursively track the target link until we get the real file. Two problems need to be solved: first, the symbolic link may form a ring, which will be tracked recursively all the time, so we need to detect the ring; On the other hand, it is necessary to limit the depth of links to reduce the burden of the system (these two points are also the requirements of the experiment)
For the link depth, NSYMLINK is defined in kernel/fs.h to represent the maximum symbolic link depth. If it exceeds this depth, it will not continue to track, but will return an error

// the max depth of symlinks - lab9-2
#define NSYMLINK 10

For the detection of looping, the simplest algorithm is also selected here: create an array inums [] with the size of NSYMLINK to record the inode number of the file tracked each time. After tracking the inode of the target file, its inode number will be compared with that recorded in the inums array. If there is any repetition, it will prove that the loop is formed
So follow as a whole_ The process of symlink() function is actually relatively simple, that is, it loops NSYMLINK at most times to track the symbolic link recursively: use the readi() function to read the path of the target file recorded in the symbolic link file, then use namei() to obtain the inode corresponding to the file path, and then compare it with the recorded inode number to determine whether it is ring. Until the type of the target inode is not T_SYMLINK is the symbolic link type
And in sys_ In open(), before creating the file object f=filealloc(), for symbolic links, in non No_ In the case of follow, you need to replace the inode of the current file with follow_ The inode of the target file obtained by symlink () is used for subsequent operations
Finally, consider the lock release rules in this process. For the case where symbolic links are not considered originally, in sys_ In open(), create() or namei() actually holds the lock of the inode after obtaining the inode of the current file. It will not be released through iunlock() until the end of the function (iput() is not used to release the ref reference of the inode when the execution is successful. I think it will be followed by sys_close() Before the call is executed, the inode will always be active for the operation of the file, so the reference cannot be reduced). For symbolic links, since the final open is the linked target file, the lock of the current inode will be released and the lock of the target inode will be obtained. When processing symbolic links, you need to read the IP - > type field, Naturally, you can't release the inode lock at this time, so you enter the follow_ When symlink(), the inode lock is always held. When readi() is used to read the target file path recorded in the symbolic link, the inode of the current symbolic link is no longer needed, and iunlockput() is used to release the lock and inode. When judging whether the target file is not a symbolic link, At this time, lock it again. In this way, when the function returns correctly, it will also hold the lock of the object file inode, so as to achieve the consistency before and after the function call

// recursively follow the symlinks - lab9-2
// Caller must hold ip->lock
// and when function returned, it holds ip->lock of returned ip
static struct inode* follow_symlink(struct inode* ip) {
  uint inums[NSYMLINK];
  int i, j;
  char target[MAXPATH];

  for(i = 0; i < NSYMLINK; ++i) {
    inums[i] = ip->inum;
    // read the target path from symlink file
    if(readi(ip, 0, (uint64)target, 0, MAXPATH) <= 0) {
      iunlockput(ip);
      printf("open_symlink: open symlink failed\n");
      return 0;
    }
    iunlockput(ip);
    
    // get the inode of target path 
    if((ip = namei(target)) == 0) {
      printf("open_symlink: path \"%s\" is not exist\n", target);
      return 0;
    }
    for(j = 0; j <= i; ++j) {
      if(ip->inum == inums[j]) {
        printf("open_symlink: links form a cycle\n");
        return 0;
      }
    }
    ilock(ip);
    if(ip->type != T_SYMLINK) {
      return ip;
    }
  }

  iunlockput(ip);
  printf("open_symlink: the depth of links reaches the limit\n");
  return 0;
}

uint64
sys_open(void)
{
  // ...

  if(ip->type == T_DEVICE && (ip->major < 0 || ip->major >= NDEV)){
    iunlockput(ip);
    end_op();
    return -1;
  }

  // handle the symlink - lab9-2
  if(ip->type == T_SYMLINK && (omode & O_NOFOLLOW) == 0) {
    if((ip = follow_symlink(ip)) == 0) {
      // There is no need to call iunlockput() to release the lock here, because it is in follow_ When symlink test() fails, the ip lock has been released in the function
      end_op();
      return -1;
    }
  }

  if((f = filealloc()) == 0 || (fd = fdalloc(f)) < 0){
    if(f)
      fileclose(f);
    iunlockput(ip);
    end_op();
    return -1;
  }

  // ...
}

Finally, add the compilation of the test file symlink test. C in Makefile

Problems encountered

Executing symlink test in xv6 will result in the error of Failed to open 1, as shown in the following figure:

Solution: according to the output of the above figure and the code of user / symlink test. C, you can see that the file 1 cannot be opened because of looping. According to the code, you can know that there is no looping here
Finally, the author detected the looped code and found the problem, as shown in the following code. Instead of inode number, the author began to record the pointer to execute inode and compare it through the pointer. It should be noted that the struct inode here is actually an inode cache in memory. When calling iput() When the ref reference count is 0, the inode is actually recycled. Subsequently, the inode information in the disk can be stored through iget() (equivalent to a cache pool with memory inodes)

If the pointer of struct inode is used for comparison, the inode cache in the memory may be recycled and reused, resulting in duplication. Therefore, the inode number, i.e. IP - > inums, should be recorded and compared according to the above description. This value is the disk's number of inodes and is unique. This should be used for loop detection of inodes. See the above for the correct code