Android development 5: the principle of anonymous shared memory

First, let's look at two key functions,

• Int shmget (key, size, int shmflg); this function is used to create shared memory
• Void * shmat (int SHM \ ID, const void * SHM \ addr, int shmflg); to access shared memory, you must map it to the address space of the current process

Refer to a demo on the Internet. In a simple way, the key "is the only identification of shared memory. It can be said that Linux shared memory is actually named shared memory, and the name is key. The specific usage is as follows

Read process

int main() { void *shm = NULL;//The original first address of the allocated shared memory struct shared_use_st *shared;//Pointing to shm int shmid;//Shared memory identifier //Create shared memory shmid = shmget((key_t)12345, sizeof(struct shared_use_st), 0666|IPC_CREAT); //Map shared memory to the address space of the current process shm = shmat(shmid, 0, 0); //Set shared memory shared = (struct shared_use_st*)shm; shared->written = 0; //Access shared memory while(1){ if(shared->written != 0) { printf("You wrote: %s", shared->text); if(strncmp(shared->text, "end", 3) == 0) break; }} //Detach shared memory from the current process if(shmdt(shm) == -1) { } //Delete shared memory if(shmctl(shmid, IPC_RMID, 0) == -1) { } exit(EXIT_SUCCESS); }

Writing process

int main() { void *shm = NULL; struct shared_use_st *shared = NULL; char buffer[BUFSIZ + 1];//Text to save input int shmid; //Create shared memory shmid = shmget((key_t) 12345, sizeof(struct shared_use_st), 0666|IPC_CREAT); //Connect shared memory to the address space of the current process shm = shmat(shmid, (void*)0, 0); printf("Memory attached at %X\n", (int)shm); //Set shared memory shared = (struct shared_use_st*)shm; while(1)//Write data to shared memory { //If the data has not been read, wait for the data to be read, and cannot write the text to the shared memory while(shared->written == 1) { sleep(1); } //Write data to shared memory fgets(buffer, BUFSIZ, stdin); strncpy(shared->text, buffer, TEXT_SZ); shared->written = 1; if(strncmp(buffer, "end", 3) == 0) running = 0; } //Detach shared memory from the current process if(shmdt(shm) == -1) { } sleep(2); exit(EXIT_SUCCESS); }

It can be seen that the communication efficiency of Linux shared memory is very high, and it can be accessed directly without transferring data between processes. The disadvantages are also obvious. Linux shared memory does not provide a synchronization mechanism. When using, it needs to use other means to handle the synchronization between processes. Android itself supports the function of System V in its core mindset, but the bionic library has deleted functions such as glibc's shmget, which makes Android unable to implement famous shared memory in the way of shmget. Of course, it does not want to use that. Android has created its own anonymous shared memory based on this.

Android can use all the IPC communication methods of Linux, including shared memory. However, the main way Android uses is Anonymous Shared Memory, which is different from the native one. For example, it adds a mutually exclusive lock in its driver, and realizes the transfer of shared memory through fd transfer. MemoryFile is an object encapsulated by Android for Anonymous Shared Memory. Through the analysis of MemoryFile, how to use shared memory to realize big data transfer in Android is also a means of big data transfer between processes. When developing, you can use:

IMemoryAidlInterface.aidl

package com.snail.labaffinity; import android.os.ParcelFileDescriptor; interface IMemoryAidlInterface { ParcelFileDescriptor getParcelFileDescriptor(); }

MemoryFetchService

public class MemoryFetchService extends Service { @Nullable @Override public IBinder onBind(Intent intent) { return new MemoryFetchStub(); } static class MemoryFetchStub extends IMemoryAidlInterface.Stub { @Override public ParcelFileDescriptor getParcelFileDescriptor() throws RemoteException { MemoryFile memoryFile = null; try { memoryFile = new MemoryFile("test_memory", 1024); memoryFile.getOutputStream().write(new byte[]); Method method = MemoryFile.class.getDeclaredMethod("getFileDescriptor"); FileDescriptor des = (FileDescriptor) method.invoke(memoryFile); return ParcelFileDescriptor.dup(des); } catch (Exception e) {} return null; }}}

TestActivity.java

Intent intent = new Intent(MainActivity.this, MemoryFetchService.class); bindService(intent, new ServiceConnection() { @Override public void onServiceConnected(ComponentName name, IBinder service) { byte[] content = new byte[10]; IMemoryAidlInterface iMemoryAidlInterface = IMemoryAidlInterface.Stub.asInterface(service); try { ParcelFileDescriptor parcelFileDescriptor = iMemoryAidlInterface.getParcelFileDescriptor(); FileDescriptor descriptor = parcelFileDescriptor.getFileDescriptor(); FileInputStream fileInputStream = new FileInputStream(descriptor); fileInputStream.read(content); } catch (Exception e) { }} @Override public void onServiceDisconnected(ComponentName name) { } }, Service.BIND_AUTO_CREATE);

The above is the method of using anonymous shared memory in application layer. The key point is the transfer of file descriptor, which is the main way to access and update files in Linux system. From the literal point of view of MemoryFile, shared memory is abstracted as a file, but in essence, it is the same, that is, a temporary file is created in tmpfs temporary file system (only a node is created, but the actual file is not seen). The file corresponds to the anonymous shared memory created by the Ashmem driver, which can be viewed directly under proc/pid:

Display of applied shared memory in proc.jpg

Based on MemoryFile, the following two main points are analyzed: allocation and transfer of shared memory. First, look at the constructor of MemoryFile

public MemoryFile(String name, int length) throws IOException { mLength = length; mFD = native_open(name, length); if (length > 0) { mAddress = native_mmap(mFD, length, PROT_READ | PROT_WRITE); } else { mAddress = 0; } }

It can be seen that the Java layer is just a simple encapsulation, which is implemented in the native layer. First, the shared memory is created by calling ashmem create region through the native open,

static jobject android_os_MemoryFile_open(JNIEnv* env, jobject clazz, jstring name, jint length) { const char* namestr = (name ? env->GetStringUTFChars(name, NULL) : NULL); int result = ashmem_create_region(namestr, length); if (name) env->ReleaseStringUTFChars(name, namestr); if (result < 0) { jniThrowException(env, "java/io/IOException", "ashmem_create_region failed"); return NULL; } return jniCreateFileDescriptor(env, result); }

Then mmap is called by native mmap to map the shared memory to the current process space, and then the Java layer can use FileDescriptor to access the shared memory as if it were a file.

static jint android_os_MemoryFile_mmap(JNIEnv* env, jobject clazz, jobject fileDescriptor, jint length, jint prot) { int fd = jniGetFDFromFileDescriptor(env, fileDescriptor); <!--system call mmap，Allocated memory--> jint result = (jint)mmap(NULL, length, prot, MAP_SHARED, fd, 0); if (!result) jniThrowException(env, "java/io/IOException", "mmap failed"); return result; }

How does ashmem? Create? Region apply to Linux for a piece of shared memory?

int ashmem_create_region(const char *name, size_t size) { int fd, ret; fd = open(ASHMEM_DEVICE, O_RDWR); if (fd < 0) return fd; if (name) { char buf[ASHMEM_NAME_LEN]; strlcpy(buf, name, sizeof(buf)); ret = ioctl(fd, ASHMEM_SET_NAME, buf); if (ret < 0) goto error; } ret = ioctl(fd, ASHMEM_SET_SIZE, size); if (ret < 0) goto error; return fd; error: close(fd); return ret; }

Ashmem? Device is actually an abstract shared memory device. It is a miscellaneous device (one of character devices). After the driver is loaded, it will wear an ashem file under / dev, and then the user can access the device file. Unlike general device files, it is only abstract through memory, which is different from general disk device files and serial port field device files Same:

#define ASHMEM_DEVICE "/dev/ashmem" static struct miscdevice ashmem_misc = { .minor = MISC_DYNAMIC_MINOR, .name = "ashmem", .fops = &ashmem_fops, };

Then enter the driver to see how to apply for shared memory. The open function is very common, mainly to create an ashmem? Area object

static int ashmem_open(struct inode *inode, struct file *file) { struct ashmem_area *asma; int ret; ret = nonseekable_open(inode, file); if (unlikely(ret)) return ret; asma = kmem_cache_zalloc(ashmem_area_cachep, GFP_KERNEL); if (unlikely(!asma)) return -ENOMEM; INIT_LIST_HEAD(&asma->unpinned_list); memcpy(asma->name, ASHMEM_NAME_PREFIX, ASHMEM_NAME_PREFIX_LEN); asma->prot_mask = PROT_MASK; file->private_data = asma; return 0; }

Then the size of shared memory is set by using ashmem? IOCTL,

static long ashmem_ioctl(struct file *file, unsigned int cmd, unsigned long arg) { struct ashmem_area *asma = file->private_data; long ret = -ENOTTY; switch (cmd) { ... case ASHMEM_SET_SIZE: ret = -EINVAL; if (!asma->file) { ret = 0; asma->size = (size_t) arg; } break; ... } return ret; }

As you can see, in fact, memory is not really allocated, which is also in line with the style of Linux. Only when it is actually used can memory be allocated through page missing interrupt. Then mmap function, will it allocate memory?

static int ashmem_mmap(struct file *file, struct vm_area_struct *vma) { struct ashmem_area *asma = file->private_data; int ret = 0; mutex_lock(&ashmem_mutex); ... if (!asma->file) { char *name = ASHMEM_NAME_DEF; struct file *vmfile; if (asma->name[ASHMEM_NAME_PREFIX_LEN] != '\0') name = asma->name; // The temporary file created here is actually a temporary file for backup. Some articles say that the temporary file is only visible to the kernel state and invisible to the user state. We also have no way to query it through commands. It can be seen as a hidden file and invisible to the user space!! <!--Calibrate documents for real operation--> vmfile = shmem_file_setup(name, asma->size, vma->vm_flags); asma->file = vmfile; } get_file(asma->file); if (vma->vm_flags & VM_SHARED) shmem_set_file(vma, asma->file); else { if (vma->vm_file) fput(vma->vm_file); vma->vm_file = asma->file; } vma->vm_flags |= VM_CAN_NONLINEAR; out: mutex_unlock(&ashmem_mutex); return ret; }

In fact, Linux's shared memory mechanism is reused here. Although anonymous shared memory is used, the underlying layer actually sets a name for shared memory (prefixed with the name "ashmem" name "prefix +). If the name is not set, the name" ashmem "name" prefix "is used by default. However, you don't see the function of memory allocation directly here. However, there are two functions, shmem file setup and shmem set file, which are very important and difficult to understand. Shmem file setup is the shared memory mechanism of native Linux, but Android also modifies the driver code of Linux shared memory. Anonymous shared memory is actually improved on the basis of Linux shared memory,

struct file *shmem_file_setup(char *name, loff_t size, unsigned long flags) { int error; struct file *file; struct inode *inode; struct dentry *dentry, *root; struct qstr this; error = -ENOMEM; this.name = name; this.len = strlen(name); this.hash = 0; /* will go */ root = shm_mnt->mnt_root; dentry = d_alloc(root, &this);//The allocation of dentry cat/proc/pid/maps can be found error = -ENFILE; file = get_empty_filp(); //Assign file error = -ENOSPC; inode = shmem_get_inode(root->d_sb, S_IFREG | S_IRWXUGO, 0, flags);//If the inode is allocated successfully, it is like setting up a file. Maybe there is no real file mapping d_instantiate(dentry, inode);//binding inode->i_size = size; inode->i_nlink = 0; /* It is unlinked */ // File operator, it seems that what is not created in memory??? init_file(file, shm_mnt, dentry, FMODE_WRITE | FMODE_READ, &shmem_file_operations);//Bind and specify shmem file operations as the file operation pointer ... }

Create a temporary file (maybe just an inode node in the kernel) in the tmpfs temporary file system through shmem file setup. The file corresponds to the anonymous shared memory created by the ashmem driver. However, the temporary file cannot be seen by the user state, and then the temporary file can be used. Note that the object that the shared memory mechanism actually uses the map is actually the temporary file This is an mmap file, not an ashmem device file. The main reason why this is an mmap is to replace the map object through VMA - > vm_file = ASMA - > file. When the mapped memory causes page loss interruption, the function of the object created by shmem_file_setup will be called instead of ashmem. Look at the corresponding hook function of the temporary file,

void shmem_set_file(struct vm_area_struct *vma, struct file *file) { if (vma->vm_file) fput(vma->vm_file); vma->vm_file = file; vma->vm_ops = &shmem_vm_ops; }

To return to the previous MemoryFile here, take a look at the write operation:

public void writeBytes(byte[] buffer, int srcOffset, int destOffset, int count) throws IOException { if (isDeactivated()) { throw new IOException("Can't write to deactivated memory file."); } if (srcOffset < 0 || srcOffset > buffer.length || count < 0 || count > buffer.length - srcOffset || destOffset < 0 || destOffset > mLength || count > mLength - destOffset) { throw new IndexOutOfBoundsException(); } native_write(mFD, mAddress, buffer, srcOffset, destOffset, count, mAllowPurging); }

Enter native code

static jint android_os_MemoryFile_write(JNIEnv* env, jobject clazz, jobject fileDescriptor, jint address, jbyteArray buffer, jint srcOffset, jint destOffset, jint count, jboolean unpinned) { int fd = jniGetFDFromFileDescriptor(env, fileDescriptor); if (unpinned && ashmem_pin_region(fd, 0, 0) == ASHMEM_WAS_PURGED) { ashmem_unpin_region(fd, 0, 0); return -1; } env->GetByteArrayRegion(buffer, srcOffset, count, (jbyte *)address + destOffset); if (unpinned) { ashmem_unpin_region(fd, 0, 0); } return count; }

In the kernel, the data structure corresponding to a block of memory is ashmem? Area:

struct ashmem_area { char name[ASHMEM_FULL_NAME_LEN];/* optional name for /proc/pid/maps */ struct list_head unpinned_list; /* list of all ashmem areas */ struct file *file; /* the shmem-based backing file */ size_t size; /* size of the mapping, in bytes */ unsigned long prot_mask; /* allowed prot bits, as vm_flags */ };

When Ashmem is used to allocate a block of memory and some of it is not used, the memory can be unpin. The kernel can reclaim the physical page corresponding to unpin. The reclaimed memory can also be obtained again (through the page missing handler), because the unpin operation does not change the address space of mmap. However, MemoryFile only operates the whole shared memory, but not accesses it in blocks, Therefore, pin and unpin have little significance for it. It can be regarded as the whole area is pin or unpin. For the first time, accessing through env - > getbytearrayregion will cause page missing interrupt, and then call the corresponding operation of tmpfs file to allocate the physical page. In Android's current kernel, the function in VM operations struct corresponding to page missing interrupt is fault. In the shared memory implementation, the function in VM operations struct corresponding to page missing interrupt is fault Shmem? Fault is as follows,

static struct vm_operations_struct shmem_vm_ops = { .fault = shmem_fault, #ifdef CONFIG_NUMA .set_policy = shmem_set_policy, .get_policy = shmem_get_policy, #endif };

When the tmpfs file of mmap causes a page break, the shmem? Fault function is called,

static int shmem_fault(struct vm_area_struct *vma, struct vm_fault *vmf) { struct inode *inode = vma->vm_file->f_path.dentry->d_inode; int error; int ret; if (((loff_t)vmf->pgoff << PAGE_CACHE_SHIFT) >= i_size_read(inode)) return VM_FAULT_SIGBUS; error = shmem_getpage(inode, vmf->pgoff, &vmf->page, SGP_CACHE, &ret); if (error) return ((error == -ENOMEM) ? VM_FAULT_OOM : VM_FAULT_SIGBUS); return ret | VM_FAULT_LOCKED; }

Here, you can see that the shmem ﹣ getpage function will be called to allocate the real physical page. The specific allocation strategy is complex and is not analyzed.

Pin itself means to hold, hold, ashmem? Pin? Region and ashmem? Unpin? Region. These two functions are literally used to lock and unlock anonymous shared memory, identify which memory is being used and which memory is not being used. In this way, the ashmem driver can assist memory management to a certain extent and provide certain memory optimization ability. At the beginning of creating anonymous shared memory, all the memory is in pinned state. Only when the user actively applies, can the memory be unpin. Only when the memory is in unpinned state, can the user re pin it. Now comb the driver carefully and see the implementation of pin and unpin

static int __init ashmem_init(void) { int ret; <!--Establish ahemem_area Cache--> ashmem_area_cachep = kmem_cache_create("ashmem_area_cache", sizeof(struct ashmem_area), 0, 0, NULL); ... <!--Establish ahemem_range Cache--> ashmem_range_cachep = kmem_cache_create("ashmem_range_cache", sizeof(struct ashmem_range), 0, 0, NULL); ... <!--Register miscellaneous equipment to send--> ret = misc_register(&ashmem_misc); ... register_shrinker(&ashmem_shrinker); return 0; }

When you open ashem, you will use ashmem? Area? Cachep to tell the cache to create a new ashmem? Area object, and initialize the unpinned? List, which must be null at the beginning

static int ashmem_open(struct inode *inode, struct file *file) { struct ashmem_area *asma; int ret; ret = nonseekable_open(inode, file); asma = kmem_cache_zalloc(ashmem_area_cachep, GFP_KERNEL); <!--The key is initialization unpinned_list list--> INIT_LIST_HEAD(&asma->unpinned_list); memcpy(asma->name, ASHMEM_NAME_PREFIX, ASHMEM_NAME_PREFIX_LEN); asma->prot_mask = PROT_MASK; file->private_data = asma; return 0; }

At the beginning, it is pin. Take a look at the call examples of pin and unpin:

int ashmem_pin_region(int fd, size_t offset, size_t len) { struct ashmem_pin pin = { offset, len }; return ioctl(fd, ASHMEM_PIN, &pin); } int ashmem_unpin_region(int fd, size_t offset, size_t len) { struct ashmem_pin pin = { offset, len }; return ioctl(fd, ASHMEM_UNPIN, &pin); }

Next, take a look at ashmem

static int ashmem_unpin(struct ashmem_area *asma, size_t pgstart, size_t pgend) { struct ashmem_range *range, *next; unsigned int purged = ASHMEM_NOT_PURGED; restart: list_for_each_entry_safe(range, next, &asma->unpinned_list, unpinned) { if (range_before_page(range, pgstart)) break; if (page_range_subsumed_by_range(range, pgstart, pgend)) return 0; if (page_range_in_range(range, pgstart, pgend)) { pgstart = min_t(size_t, range->pgstart, pgstart), pgend = max_t(size_t, range->pgend, pgend); purged |= range->purged; range_del(range); goto restart; } } return range_alloc(asma, range, purged, pgstart, pgend); }

The main function of this function is to create an ashmem? Range and insert it into the unpinned? List of the ashmem? Area. At this time, the original unpin ashmem? Range will be deleted first, and then a new merged ashmem? Range will be inserted into the unpinned? List.

Shared memory.jpg

Let's take a look at the implementation of the pin function. If you understand unpin, pin will be well understood. In fact, you can put a piece of shared memory into use. If it is located in unpinedlist, you can take it off:

static int ashmem_pin(struct ashmem_area *asma, size_t pgstart, size_t pgend) { struct ashmem_range *range, *next; int ret = ASHMEM_NOT_PURGED; list_for_each_entry_safe(range, next, &asma->unpinned_list, unpinned) { /* moved past last applicable page; we can short circuit */ if (range_before_page(range, pgstart)) break; if (page_range_in_range(range, pgstart, pgend)) { ret |= range->purged; if (page_range_subsumes_range(range, pgstart, pgend)) { range_del(range); continue; } if (range->pgstart >= pgstart) { range_shrink(range, pgend + 1, range->pgend); continue; } if (range->pgend <= pgend) { range_shrink(range, range->pgstart, pgstart-1); continue; } range_alloc(asma, range, range->purged, pgend + 1, range->pgend); range_shrink(range, range->pgstart, pgstart - 1); break; } } return ret; }

pin shared memory.jpg

The native Linux shared memory is handled by passing the known key, but there is no such mechanism in Android. How does Android handle it? That is to say, through binder passing file descriptors, Android binder also adapts fd passing. In fact, the principle is to convert fd in the kernel layer for the target process to be passed, because fd is only valid and unique for this process in Linux. Process A opens A file to get an fd, which cannot be used directly for process B, because that fd in B may be invalid, or It corresponds to other files. However, although the same file can have multiple file descriptors, there is only one file. In the kernel layer, there is only one inode node and file object. This is also the basis of passing fd in the kernel layer. Binder driver finds the corresponding file through the fd of the current process, creates A new fd for the target process, and passes it to the target process. The core is Convert the fd in process A to the fd in process B. take A look at the implementation of the binder in Android:

void binder_transaction(){ ... case BINDER_TYPE_FD: { int target_fd; struct file *file; <!--Key 1 can be based on fd Get to file ，Multiple processes open the same file, corresponding to the file It's the same.--> file = fget(fp->handle); <!--Key point 2,Get idle for target process fd--> target_fd = task_get_unused_fd_flags(target_proc, O_CLOEXEC); <!--Key 3 will idle the target process fd And file binding--> task_fd_install(target_proc, target_fd, file); fp->handle = target_fd; } break; ... } <!--Opened from the current process files Find in file Instances in the kernel--> struct file *fget(unsigned int fd) { struct file *file; struct files_struct *files = current->files; rcu_read_lock(); file = fcheck_files(files, fd); rcu_read_unlock(); return file; } static void task_fd_install( struct binder_proc *proc, unsigned int fd, struct file *file) { struct files_struct *files = proc->files; struct fdtable *fdt; if (files == NULL) return; spin_lock(&files->file_lock); fdt = files_fdtable(files); rcu_assign_pointer(fdt->fd[fd], file); spin_unlock(&files->file_lock); }

fd passes.jpg

Why can't Android users see the files corresponding to shared memory? Google says that tmpfs is invisible to users when the kernel doesn't define defined (config? tmpfs):

If CONFIG_TMPFS is not set, the user visible part of tmpfs is not build. But the internal mechanisms are always present.

But there is no defined (config? TMPFS) in the shmem.c driver of Android. It's just a guess here. Maybe there are other explanations. If you have any understanding, please give us some guidance.

Anonymous shared memory will not occupy Dalvik Heap and Native Heap, and will not lead to OOM. This is not only an advantage, but also a disadvantage. If it is used arbitrarily, it will lead to insufficient system resources and performance degradation,

Anonymous shared memory is not used

In addition, the calculation of the space occupied by shared memory will only be calculated to the first process that creates it, and other processes will not count ashmem.

Android anonymous shared memory is based on Linux shared memory. All files are created on tmpfs file system, and mapped to different process spaces, so as to achieve the purpose of shared memory. However, Android is modified on the basis of Linux, and the shared memory is transferred by using Binder+fd file descriptor.

Author: reading snail

Author: reading snail
Link: https://www.jianshu.com/p/d9bc9c668ba6
Source: Jianshu
The copyright belongs to the author. For commercial reprint, please contact the author for authorization. For non-commercial reprint, please indicate the source.

• Give the thumbs-up
• Collection
• share
• • Article report

6 February 2020, 01:22 | Views: 10109