swift-system icon indicating copy to clipboard operation
swift-system copied to clipboard

Implement `FileDescriptor.memoryMap`, `FileDescriptor.memoryUnmap`, and `FileDescriptor.memorySync`

Open jsflax opened this issue 3 years ago • 3 comments

Add Swift implementations for mmap, munmap, and msync:

let swiftString = "swift"
let map = try fd.memoryMap(length: swiftString.count, pageOffset: 0, kind: .shared, protection: [.write])
let ptr = map.assumingMemoryBound(to: String.self)
// Update the mapping
ptr.pointee = swiftString
// Flush changes to the file system
try fd.memorySync(memoryMap: map, length: swiftString.count, kind: .synchronous)
// Delete the mapping
try fd.memoryUnmap(memoryMap: map, length: swiftString.count)

// Read in the file and validate the string was written correctly
let readBytes = try Array<CChar>(unsafeUninitializedCapacity: swiftString.count + 1) { (buf, count) in
    count = try fd.read(into: UnsafeMutableRawBufferPointer(buf))
}
XCTAssertEqual(String(validatingPlatformString: readBytes), swiftString)

This PR adds all three methods because the ability to sync and delete mappings are sensible requirements of being able to create them.

Add FileDescriptor.MemoryProtection to describe the desired memory protection of a mapping. The memoryMap method accepts these values as an array to be bitwise OR'd by the implementation.

Add shared and private for FileDescriptor.MemoryMapKind. There are more C flags available, but this seemed like the simplest starting point.

Add synchronous and asynchronous FileDescriptor.MemorySyncKind. The memorySync method accepts either one of these values and an invalidateOtherMappings boolean that is bitwise AND'd in the implementation to represent the appropriate C flag.

Add internal constants for flags/values related to mmap, munmap, and msync.

Add additional methods for more generic mocking and error handling. This seemed like a good addition for system calls that do not return CInt.

This PR does not include a Windows implementation.

Open question: Should memory mappings be a unique type?

Arguments in favor:

  1. Memory mappings can outlive FileDescriptors.
  2. Unmapping an UnsafeMutableRawPointer leaves you with an unusable instance with no way to ascertain that it has been unmapped.

Arguments against:

  1. For the most elegant API for this, it would add a series of structs/overhead: probably a MemoryMapRaw and MemoryMap<T> that have a very similar API to UnsafeMutableRawPointer and UnsafeMutablePointer<T>. The added methods would be sync, unmap, and isUnmapped. This is more precise than what currently exists in this PR at the cost of overhead.

jsflax avatar Nov 07 '21 16:11 jsflax

There is this whole family of address space-related functionality that we can't currently group under a specific type. This includes the anonymous flavor of mmap, as well as munmap, msync, madvise etc. (It would also be great if we could expose a way to enumerate existing memory mappings, but I have no idea if there are public APIs for that, or even if that would be a good fit for System.)

@milseman are you okay with just pulling these in as top-level functions?

We could choose to instead define a namespacing enum, and add all of these as members:

enum Memory {
  static var pageSize: Int { get }
  // properties for getting the cache line size, cache sizes etc.

  func map(at:, length:, protection:, flags:, from:, offset:) throws -> UMRP
  func unmap(at:, length:) throws
  func protect(at:, length:, protection:) throws
  func sync(at:, length: flags:) throws

  // BSD/Darwinisms
  func advise(at:, length:, advice:) throws
  func inherit(at:, length:, inherit:) throws
  func isInCore(at:, length:) throws -> ??? // mincore (BSD)

  // Realtime stuff
  func lock(at:, length:) throws
  func unlock(at:, length:) throws
  func lockAll(flags:) throws
  func unlockAll() throws
}

This could be an attractive option, if for no other reason but to move these really low-level functions out of highly visible namespaces, such as FileDescriptor. (I believe (hope?) the number of Swift programs that have to use file descriptors is likely to heavily outweigh those who need to deal with memory mappings. I could be wrong, though.)

lorentey avatar Nov 14 '21 02:11 lorentey

For sysconf, we have https://github.com/apple/swift-system/pull/33, which we can pull stuff from any time. Those sketches also have the general need for a namespace for global functions. Should we have many or just one? E.g. enum FileSystem {}, enum SystemConfig {}, ...

milseman avatar Nov 14 '21 02:11 milseman

Oh, good point! I forgot we wanted to introduce SystemConfig for that. The page size etc. information feels like a good fit for it.

I think if we add one of these, we may as well add as many as we need.

(I remember being worried about adding file operations like copy/rename/remove into a FileSystem namespace rather than having them directly on FilePath, because that is such a convenient place for them. But then again, perhaps those convenience functions would be better defined in a separate module, with System restricting itself to basic syscalls like creat/unlink/rmdir/rename etc, which could just as well go under a namespace.)

lorentey avatar Nov 14 '21 02:11 lorentey