gopsutil icon indicating copy to clipboard operation
gopsutil copied to clipboard

Published 20 hours ago • verified publisher icon shirou

Fetch stats from folder mounted volumes

Open dloucasfx opened this issue 2 years ago • 3 comments

This fixes https://github.com/shirou/gopsutil/issues/1245 and https://github.com/open-telemetry/opentelemetry-collector-contrib/issues/27203

The test from OpenTelemetry

Steps to reproduce

  • Add a new disk to a windows VM
  • from Disk Management tool, right-click the new volume, and select Change drive letter and paths.
  • From there, add “Mount in the following empty NTFS folder” and select an empty dir on the C:\ drive and make sure there is NO assigned drive letter

Before the fix:

Resource SchemaURL: https://opentelemetry.io/schemas/1.9.0
Resource attributes:
     -> host.name: Str(wintestswatsplunk)
     -> os.type: Str(windows)
     -> cloud.provider: Str(azure)
     -> cloud.platform: Str(azure_vm)
     -> cloud.region: Str(eastus)
     -> host.id: Str(09f0d80d-d5a7-4082-953a-bfd10cb77cbf)
     -> cloud.account.id: Str(5db89e2c-2127-4418-9a31-dac67b04e87b)
     -> azure.vm.name: Str(win2016-swat-2021)
     -> azure.vm.size: Str(Standard_D2as_v4)
     -> azure.vm.scaleset.name: Str()
     -> azure.resourcegroup.name: Str(win2016-swat)
ScopeMetrics #0
ScopeMetrics SchemaURL:
InstrumentationScope otelcol/hostmetricsreceiver/filesystem 0.85.0-dev
Metric #0
Descriptor:
     -> Name: system.filesystem.usage
     -> Description: Filesystem bytes used.
     -> Unit: By
     -> DataType: Sum
     -> IsMonotonic: false
     -> AggregationTemporality: Cumulative
NumberDataPoints #0
Data point attributes:
     -> device: Str(C:)
     -> mode: Str(rw)
     -> mountpoint: Str(C:)
     -> type: Str(NTFS)
     -> state: Str(used)
StartTimestamp: 2023-09-26 14:11:00 +0000 UTC
Timestamp: 2023-10-03 21:22:58.5265249 +0000 UTC
Value: 19619004416
NumberDataPoints #1
Data point attributes:
     -> device: Str(C:)
     -> mode: Str(rw)
     -> mountpoint: Str(C:)
     -> type: Str(NTFS)
     -> state: Str(free)
StartTimestamp: 2023-09-26 14:11:00 +0000 UTC
Timestamp: 2023-10-03 21:22:58.5265249 +0000 UTC
Value: 116220866560
NumberDataPoints #2
Data point attributes:
     -> device: Str(D:)
     -> mode: Str(rw)
     -> mountpoint: Str(D:)
     -> type: Str(NTFS)
     -> state: Str(used)
StartTimestamp: 2023-09-26 14:11:00 +0000 UTC
Timestamp: 2023-10-03 21:22:58.5265249 +0000 UTC
Value: 1418309632
NumberDataPoints #3
Data point attributes:
     -> device: Str(D:)
     -> mode: Str(rw)
     -> mountpoint: Str(D:)
     -> type: Str(NTFS)
     -> state: Str(free)
StartTimestamp: 2023-09-26 14:11:00 +0000 UTC
Timestamp: 2023-10-03 21:22:58.5265249 +0000 UTC
Value: 15759458304

With the fix, notice the folder mounted volumes stats C:\mount4 :

2023-10-03T21:14:24.922Z        info    ResourceMetrics #0
Resource SchemaURL: https://opentelemetry.io/schemas/1.9.0
Resource attributes:
     -> host.name: Str(wintestswatsplunk)
     -> os.type: Str(windows)
     -> cloud.provider: Str(azure)
     -> cloud.platform: Str(azure_vm)
     -> cloud.region: Str(eastus)
     -> host.id: Str(09f0d80d-d5a7-4082-953a-bfd10cb77cbf)
     -> cloud.account.id: Str(5db89e2c-2127-4418-9a31-dac67b04e87b)
     -> azure.vm.name: Str(win2016-swat-2021)
     -> azure.vm.size: Str(Standard_D2as_v4)
     -> azure.vm.scaleset.name: Str()
     -> azure.resourcegroup.name: Str(win2016-swat)
ScopeMetrics #0
ScopeMetrics SchemaURL:
InstrumentationScope otelcol/hostmetricsreceiver/filesystem 0.85.0-dev
Metric #0
Descriptor:
     -> Name: system.filesystem.usage
     -> Description: Filesystem bytes used.
     -> Unit: By
     -> DataType: Sum
     -> IsMonotonic: false
     -> AggregationTemporality: Cumulative
NumberDataPoints #0
Data point attributes:
     -> device: Str(D:)
     -> mode: Str(rw)
     -> mountpoint: Str(D:)
     -> type: Str(NTFS)
     -> state: Str(used)
StartTimestamp: 2023-09-26 14:11:00 +0000 UTC
Timestamp: 2023-10-03 21:14:24.795121 +0000 UTC
Value: 1418309632
NumberDataPoints #1
Data point attributes:
     -> device: Str(D:)
     -> mode: Str(rw)
     -> mountpoint: Str(D:)
     -> type: Str(NTFS)
     -> state: Str(free)
StartTimestamp: 2023-09-26 14:11:00 +0000 UTC
Timestamp: 2023-10-03 21:14:24.795121 +0000 UTC
Value: 15759458304
NumberDataPoints #2
Data point attributes:
     -> device: Str(C:\mount4)
     -> mode: Str(rw)
     -> mountpoint: Str(C:\mount4)
     -> type: Str(NTFS)
     -> state: Str(used)
StartTimestamp: 2023-09-26 14:11:00 +0000 UTC
Timestamp: 2023-10-03 21:14:24.795121 +0000 UTC
Value: 189624320
NumberDataPoints #3
Data point attributes:
     -> device: Str(C:\mount4)
     -> mode: Str(rw)
     -> mountpoint: Str(C:\mount4)
     -> type: Str(NTFS)
     -> state: Str(free)
StartTimestamp: 2023-09-26 14:11:00 +0000 UTC
Timestamp: 2023-10-03 21:14:24.795121 +0000 UTC
Value: 4102193152
NumberDataPoints #4
Data point attributes:
     -> device: Str(C:)
     -> mode: Str(rw)
     -> mountpoint: Str(C:)
     -> type: Str(NTFS)
     -> state: Str(used)
StartTimestamp: 2023-09-26 14:11:00 +0000 UTC
Timestamp: 2023-10-03 21:14:24.795121 +0000 UTC
Value: 19642892288
NumberDataPoints #5
Data point attributes:
     -> device: Str(C:)
     -> mode: Str(rw)
     -> mountpoint: Str(C:)
     -> type: Str(NTFS)
     -> state: Str(free)
StartTimestamp: 2023-09-26 14:11:00 +0000 UTC
Timestamp: 2023-10-03 21:14:24.795121 +0000 UTC
Value: 116196978688
        {"kind": "exporter", "data_type": "metrics", "name": "logging"}

dloucasfx avatar Oct 03 '23 21:10 dloucasfx

Updated the PR with the requested changes

dloucasfx avatar Jan 08 '24 00:01 dloucasfx

@shirou would you please take a look?

atoulme avatar Jan 26 '24 22:01 atoulme

Revisiting this ; I think a way to put this to bed is to run Windows tests as part of CI so we're sure it works well. WDYT?

atoulme avatar Sep 26 '24 03:09 atoulme

@shirou @atoulme I was able to reproduce the issue that you reported with SMB drive stopped working. I have confirmed that SMB mount is not returned when using FindFirstVolume \ FindNextVolume api calls, and the only way to do that is using GetLogicalDriveStrings

I have updated the PR to: 1- Keep using the old logic, ie: using GetLogicalDriveStrings , which fetches all drives with assigned letters (including SMB) 2- Added FindFirstVolume \ FindNextVolume logic which now runs after collecting partition stats using GetLogicalDriveStrings and it then checks if a mount/drive was already processed and collect its stats otherwise. This logic guarantees that volumes with no drive letters are collected. 3- Finally, I did some refactoring to reuse some common code and make the code clear to read.

Test:

Before (Z: is the SMB):

=== RUN   TestLocal
{"device":"C:","mountpoint":"C:","fstype":"NTFS","opts":["rw","compress"]}
{"device":"Z:","mountpoint":"Z:","fstype":"NTFS","opts":["rw","compress"]}
--- PASS: TestLocal (0.01s)
PASS

After (C:\D\mount\ is the folder mount)

=== RUN   TestLocal
{"device":"C:","mountpoint":"C:","fstype":"NTFS","opts":["rw","compress"]}
{"device":"Z:","mountpoint":"Z:","fstype":"NTFS","opts":["rw","compress"]}
{"device":"C:\\D\\mount\\","mountpoint":"C:\\D\\mount\\","fstype":"NTFS","opts":["rw","compress"]}
--- PASS: TestLocal (0.00s)
PASS

dloucasfx avatar Jun 10 '25 03:06 dloucasfx

@shirou I also fixed the following lint errors: Note: the gocritic is related to a function that I have not touched IOCountersWithContext please make sure you are ok with the change

 Error: disk\disk_windows.go:168:21: type assertion on error will fail on wrapped errors. Use errors.As to check for specific errors (errorlint)
  				if errno, ok := err.(syscall.Errno); ok && errno == windows.ERROR_NO_MORE_FILES {
  				                ^
  Error: disk\disk_windows.go:260:3: nestingReduce: invert if cond, replace body with `continue`, move old body after the statement (gocritic)
  		if 'A' <= v && v <= 'Z' {
  		^
  Error: disk\disk_windows.go:22:1: File is not properly formatted (gofumpt)
  const volumeNameBufferLength = uint32(windows.MAX_PATH + 1)

dloucasfx avatar Jun 10 '25 15:06 dloucasfx

@shirou friendly reminder if you can look into this. Thanks

dloucasfx avatar Jun 13 '25 00:06 dloucasfx

merged. Thank you for your contribution!

shirou avatar Jun 19 '25 13:06 shirou