arrow icon indicating copy to clipboard operation
arrow copied to clipboard

[Benchmarking] Add portable machine and benchmark context detection

Open asfimport opened this issue 6 years ago • 6 comments

The machine detection script (make_machine_json.sh) currently looks like it will only work properly on macOS. Ideally it should work more or less correctly on all of them macOS, Linux and Windows (some values may remain undetected on some platforms). 

This probably entails:

  • extend archery benchmark (see https://jira.apache.org/jira/browse/ARROW-4827 and tentative implementation in https://github.com/apache/arrow/pull/4077/files) rather than a standalone bash script 
  • using something like psutil to grab useful machine information
  • calling nvidia-smi to query GPU characteristics (for example "nvidia-smi -q -i 0 -x")

Reporter: Antoine Pitrou / @pitrou

Related issues:

Note: This issue was originally created as ARROW-4716. Please see the migration documentation for further details.

asfimport avatar Feb 28 '19 12:02 asfimport

Antoine Pitrou / @pitrou: cc @tanyaschlusser

asfimport avatar Feb 28 '19 12:02 asfimport

Tanya Schlusser / @tanyaschlusser: (y)

asfimport avatar Feb 28 '19 13:02 asfimport

Francois Saint-Jacques / @fsaintjacques: Do we need this to be cross-platform, I think Linux should suffice first. It might be worth writing this in python for future compat with without.

asfimport avatar Mar 29 '19 15:03 asfimport

Tanya Schlusser / @tanyaschlusser: Linux first is a great choice.

Python is my favorite language and I am happy to do this. The existing shell script was only because it was easy and quick.

I am very very sorry for dropping the ball the past couple of months. My mom passed away and I have been a total wreck (and moved back home from her house and got a job) but I still want to contribute and hope you will accept me back now. You all rock very much!

asfimport avatar Apr 13 '19 14:04 asfimport

Francois Saint-Jacques / @fsaintjacques: No need to be sorry! Hope you feel better. I'll rename and re-organize this ticket.

asfimport avatar Apr 16 '19 15:04 asfimport

This issue has been marked as stale because it has had no activity in the past 365 days. Please remove the stale label or comment below, or this issue will be closed in 14 days. If this improvement is still desired but has no current owner, please add the 'Status: needs champion' label.

github-actions[bot] avatar Dec 14 '25 11:12 github-actions[bot]