tsfresh
tsfresh copied to clipboard
blue-yonder
Unable to pip install tsfresh on a Mac with M1 chip
The problem:
When trying to pip install tsfresh on a M1 Mac, the following error is encountered due to the matrixprofile sub-package :
× Running setup.py install for matrixprofile did not run successfully.
│ exit code: 1
╰─> [91 lines of output]
WARNING: The wheel package is not available.
WARNING: The wheel package is not available.
running install
running build
running build_py
creating build
creating build/lib.macosx-12.4-arm64-3.8
creating build/lib.macosx-12.4-arm64-3.8/matrixprofile
copying matrixprofile/version.py -> build/lib.macosx-12.4-arm64-3.8/matrixprofile
copying matrixprofile/preprocess.py -> build/lib.macosx-12.4-arm64-3.8/matrixprofile
copying matrixprofile/discover.py -> build/lib.macosx-12.4-arm64-3.8/matrixprofile
copying matrixprofile/__init__.py -> build/lib.macosx-12.4-arm64-3.8/matrixprofile
copying matrixprofile/core.py -> build/lib.macosx-12.4-arm64-3.8/matrixprofile
copying matrixprofile/visualize.py -> build/lib.macosx-12.4-arm64-3.8/matrixprofile
copying matrixprofile/utils.py -> build/lib.macosx-12.4-arm64-3.8/matrixprofile
copying matrixprofile/transform.py -> build/lib.macosx-12.4-arm64-3.8/matrixprofile
copying matrixprofile/exceptions.py -> build/lib.macosx-12.4-arm64-3.8/matrixprofile
copying matrixprofile/analyze.py -> build/lib.macosx-12.4-arm64-3.8/matrixprofile
copying matrixprofile/compute.py -> build/lib.macosx-12.4-arm64-3.8/matrixprofile
creating build/lib.macosx-12.4-arm64-3.8/tests
copying tests/test_pairwise_dist.py -> build/lib.macosx-12.4-arm64-3.8/tests
copying tests/test_utils.py -> build/lib.macosx-12.4-arm64-3.8/tests
copying tests/test_statistics.py -> build/lib.macosx-12.4-arm64-3.8/tests
copying tests/test_datasets.py -> build/lib.macosx-12.4-arm64-3.8/tests
copying tests/test_mpdist.py -> build/lib.macosx-12.4-arm64-3.8/tests
copying tests/test_core.py -> build/lib.macosx-12.4-arm64-3.8/tests
copying tests/test_hierarchical_clustering.py -> build/lib.macosx-12.4-arm64-3.8/tests
copying tests/__init__.py -> build/lib.macosx-12.4-arm64-3.8/tests
copying tests/test_top_k_discords.py -> build/lib.macosx-12.4-arm64-3.8/tests
copying tests/test_mpx.py -> build/lib.macosx-12.4-arm64-3.8/tests
copying tests/test_io.py -> build/lib.macosx-12.4-arm64-3.8/tests
copying tests/test_regimes.py -> build/lib.macosx-12.4-arm64-3.8/tests
copying tests/test_top_k_motifs.py -> build/lib.macosx-12.4-arm64-3.8/tests
copying tests/test_mass2.py -> build/lib.macosx-12.4-arm64-3.8/tests
copying tests/test_scrimp.py -> build/lib.macosx-12.4-arm64-3.8/tests
copying tests/test_skimp.py -> build/lib.macosx-12.4-arm64-3.8/tests
copying tests/test_cycore.py -> build/lib.macosx-12.4-arm64-3.8/tests
copying tests/test_visualize.py -> build/lib.macosx-12.4-arm64-3.8/tests
copying tests/test_analyze.py -> build/lib.macosx-12.4-arm64-3.8/tests
copying tests/test_compute.py -> build/lib.macosx-12.4-arm64-3.8/tests
copying tests/test_preprocess.py -> build/lib.macosx-12.4-arm64-3.8/tests
copying tests/test_annotation_vector.py -> build/lib.macosx-12.4-arm64-3.8/tests
copying tests/test_stomp.py -> build/lib.macosx-12.4-arm64-3.8/tests
copying tests/test_snippets.py -> build/lib.macosx-12.4-arm64-3.8/tests
creating build/lib.macosx-12.4-arm64-3.8/matrixprofile/datasets
copying matrixprofile/datasets/datasets.py -> build/lib.macosx-12.4-arm64-3.8/matrixprofile/datasets
copying matrixprofile/datasets/__init__.py -> build/lib.macosx-12.4-arm64-3.8/matrixprofile/datasets
creating build/lib.macosx-12.4-arm64-3.8/matrixprofile/io
copying matrixprofile/io/__init__.py -> build/lib.macosx-12.4-arm64-3.8/matrixprofile/io
copying matrixprofile/io/__io.py -> build/lib.macosx-12.4-arm64-3.8/matrixprofile/io
creating build/lib.macosx-12.4-arm64-3.8/matrixprofile/algorithms
copying matrixprofile/algorithms/mass2.py -> build/lib.macosx-12.4-arm64-3.8/matrixprofile/algorithms
copying matrixprofile/algorithms/skimp.py -> build/lib.macosx-12.4-arm64-3.8/matrixprofile/algorithms
copying matrixprofile/algorithms/scrimp.py -> build/lib.macosx-12.4-arm64-3.8/matrixprofile/algorithms
copying matrixprofile/algorithms/hierarchical_clustering.py -> build/lib.macosx-12.4-arm64-3.8/matrixprofile/algorithms
copying matrixprofile/algorithms/mpx.py -> build/lib.macosx-12.4-arm64-3.8/matrixprofile/algorithms
copying matrixprofile/algorithms/pairwise_dist.py -> build/lib.macosx-12.4-arm64-3.8/matrixprofile/algorithms
copying matrixprofile/algorithms/regimes.py -> build/lib.macosx-12.4-arm64-3.8/matrixprofile/algorithms
copying matrixprofile/algorithms/__init__.py -> build/lib.macosx-12.4-arm64-3.8/matrixprofile/algorithms
copying matrixprofile/algorithms/stomp.py -> build/lib.macosx-12.4-arm64-3.8/matrixprofile/algorithms
copying matrixprofile/algorithms/top_k_motifs.py -> build/lib.macosx-12.4-arm64-3.8/matrixprofile/algorithms
copying matrixprofile/algorithms/snippets.py -> build/lib.macosx-12.4-arm64-3.8/matrixprofile/algorithms
copying matrixprofile/algorithms/top_k_discords.py -> build/lib.macosx-12.4-arm64-3.8/matrixprofile/algorithms
copying matrixprofile/algorithms/statistics.py -> build/lib.macosx-12.4-arm64-3.8/matrixprofile/algorithms
copying matrixprofile/algorithms/mpdist.py -> build/lib.macosx-12.4-arm64-3.8/matrixprofile/algorithms
creating build/lib.macosx-12.4-arm64-3.8/matrixprofile/io/protobuf
copying matrixprofile/io/protobuf/protobuf_utils.py -> build/lib.macosx-12.4-arm64-3.8/matrixprofile/io/protobuf
copying matrixprofile/io/protobuf/__init__.py -> build/lib.macosx-12.4-arm64-3.8/matrixprofile/io/protobuf
copying matrixprofile/io/protobuf/proto_messages_pb2.py -> build/lib.macosx-12.4-arm64-3.8/matrixprofile/io/protobuf
running egg_info
writing matrixprofile.egg-info/PKG-INFO
writing dependency_links to matrixprofile.egg-info/dependency_links.txt
writing requirements to matrixprofile.egg-info/requires.txt
writing top-level names to matrixprofile.egg-info/top_level.txt
warning: the 'license_file' option is deprecated, use 'license_files' instead
adding license file 'LICENSE' (matched pattern 'LICENSE')
reading manifest file 'matrixprofile.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no files found matching '*.csv' under directory '.'
warning: no files found matching '*.json' under directory '.'
writing manifest file 'matrixprofile.egg-info/SOURCES.txt'
copying matrixprofile/cycore.c -> build/lib.macosx-12.4-arm64-3.8/matrixprofile
copying matrixprofile/algorithms/cympx.c -> build/lib.macosx-12.4-arm64-3.8/matrixprofile/algorithms
running build_ext
building 'matrixprofile.algorithms.cympx' extension
creating build/temp.macosx-12.4-arm64-3.8
creating build/temp.macosx-12.4-arm64-3.8/matrixprofile
creating build/temp.macosx-12.4-arm64-3.8/matrixprofile/algorithms
clang -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -I/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include -I/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include -I/Users/antonin.durieux/.pyenv/versions/3.8.13/lib/python3.8/site-packages/numpy/core/include -I/Users/antonin.durieux/.pyenv/versions/3.8.13/lib/python3.8/site-packages/numpy/core/include -I/Users/antonin.durieux/.pyenv/versions/3.8.13/include/python3.8 -c matrixprofile/algorithms/cympx.c -o build/temp.macosx-12.4-arm64-3.8/matrixprofile/algorithms/cympx.o -O2 -fopenmp
clang: error: unsupported option '-fopenmp'
error: command 'clang' failed with exit status 1
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
error: legacy-install-failure
× Encountered error while trying to install package.
╰─> matrixprofile
It seems to be related to this issue, and the coauthor of matrixprofile mentioned here that they might not be currently able to update the package, and that the STUMPY library could be a good replacement.
Anything else we need to know?:
The issue doesn't appear when installing tsfresh in a x86 virtual environnement with Rosetta, so the problem seems linked to the M1 chip.
Environment:
- Python version: 3.8.13 with pyenv
- Operating System: MacOS Monterey Version 12.4, M1 chip
- tsfresh version: 0.19.0
- Install method (conda, pip, source): pip
I'm no expert with all this stuff. Just saw a youtube thing on matrix profiles. It looks like it's using /usr/bin/clang instead of the brew-installed /usr/local/bin/gcc which has the -fopenmp flag working. Is there any way to force it to use gcc?
I'm using an M1 chip and had the same issues when trying to install it under python 3.9.
However, with Python 3.9.7 pip install worked.
You might need to
brew install llvm
I'm running into the same issue. brew install llvm did not help on Python 3.10.6
I have the same problem...brew install llvm did not help. I'm using Python 3.9.13
I used what @JoachimSchaeffer did and set my python version to exactly 3.9.7 and pip install worked.
I started taking a crack at doing what the matrixprofile author said and replacing matrix profile with stumpy. However stumpy does not provide an implementation of maximum_subsequence, but we could probably roll our own implementation of that based on what matrix profile already has.
I'm also not entirely sure that there is a 1-1 mapping between the APIs of the two packages: https://matrixprofile.docs.matrixprofile.org/api.html#matrixprofile-compute https://stumpy.readthedocs.io/en/latest/api.html#stumpy.stump
@bfdykstra I don't know anything regarding the matrixprofile API but if you'd like to submit a PR to this repo with the stumpy replacement parts then I can help review it.
Alternatively, would it make sense to simply remove the matrixprofile parts instead of replacing it given the lack of activity in that package? My concern is that we may be forcing stumpy to support matrixprofile features that are absent in stumpy and so we should be open to removing the matrixprofile bits if necessary.
@seanlaw Thank you weighing in! Not sure how critical the matrix profile stats are as features, maybe @nils-braun could provide some insight as to what would be lost if we removed them?
From the documentation of tsfresh it says:
Calculates the 1-D Matrix Profile[1] and returns Tukey’s Five Number Set plus the mean of that Matrix Profile.
Matrix profile's api documentation says it computes the exact or approximate matrix profile based on sample percent.
I see with stumpy that we can use either stumpy.stump or stumpy.scrump for an exact or approximate matrix profile.
@seanlaw Does stumpy support calculations based off of a sample percent?
Looking at the tests, might shed some light on the desired usage:
def test_matrix_profile_window(self):
# Test matrix profile output with specified window
np.random.seed(9999)
ts = np.random.uniform(size=2 ** 10)
w = 2 ** 5
subq = ts[0:w]
ts[0:w] = subq
ts[w + 100 : w + 100 + w] = subq
param = [
{"threshold": 0.98, "windows": 36, "feature": "min"},
{"threshold": 0.98, "windows": 36, "feature": "max"},
{"threshold": 0.98, "windows": 36, "feature": "mean"},
{"threshold": 0.98, "windows": 36, "feature": "median"},
{"threshold": 0.98, "windows": 36, "feature": "25"},
{"threshold": 0.98, "windows": 36, "feature": "75"},
]
self.assertAlmostEqual(matrix_profile(ts, param=param)[0][1], 2.825786727580335)
Not sure how critical the matrix profile stats are as features, maybe @nils-braun could provide some insight as to what would be lost if we removed them?
Objectively, I don't think these stats features are useful at all (save for the min of the matrix profile) as they aren't necessarily informative as an ML feature. Hence, why I suggested removing this part completely.
@seanlaw Does stumpy support calculations based off of a sample percent?
I am not too familiar with what is happening in matrixprofile but I believe that you can use stumpy.scrump and specify percentage=1.0 to obtain the exact matrix profile (see API docs).
Perhaps something like this would be equivalent for allowing the test to pass:
def test_matrix_profile_window(self):
# Test matrix profile output with specified window
np.random.seed(9999)
ts = np.random.uniform(size=2 ** 10)
w = 2 ** 5
subq = ts[0:w]
ts[0:w] = subq
ts[w + 100 : w + 100 + w] = subq
# mp = stumpy.stump(ts, m=36)[:, 0]
mp = stumpy.scrump(ts, m=36, percentage=1.0)[:, 0]
self.assertAlmostEqual(mp.min(), 2.825786727580335)
Then, to obtain the other numbers, you'd simply do:
mp.max()
mp.mean()
np.median(mp)
np.percentile(mp, 25)
np.percentile(mp, 75)
I think this is the easiest way forward without attempting to force a one-to-one match between the packages. We'll need to remove the threshold parameters or simply ignore them when passed in. Please feel free to ask more questions if I've been to vague. Does that help?
My preferred solution is to refactor the matrix profile features into a separate module, e.g. tsfresh-matrixprofiles, which can be installed on top of tsfresh.
My preferred solution is to refactor the matrix profile features into a separate module, e.g. tsfresh-matrixprofiles, which can be installed on top of tsfresh.
@kempa-liehr that's probably a good idea. Having said that, it appears that users are unable to install tsfresh right now and the solution provided above addresses this issue while preserving some level of backwards compatibility. If the goal is to split it out then I think we should simply spend the time to remove the matrixprofile package (which doesn't seem to be supported anymore) parts altogether for this PR, which would also resolve the installation issues. My suggestion would be to spend this time fixing the current issue, which is an installation problem. What do you think?
I tried to use Rosetta and it seems to work. I followed these steps:
- open terminal with Rosetta
- create and start venv (I'm using Python 3.9.13)
- install tsfresh (pip install tsfresh)
- since there is a bug with statsmodels 0.13.0 I have downgraded it to 0.12.1
- use if name == "main"
I checked with this very simple code:
from tsfresh import extract_relevant_features
from tsfresh.examples.robot_execution_failures import load_robot_execution_failures
timeseries, y = load_robot_execution_failures()
if __name__ == "__main__":
features = extract_relevant_features(timeseries, y, column_id="id", column_sort="time")
print(features)
Duplicate of #937
@giova86, thanks for your workaround.
@bfdykstra, @seanlaw, @vanbenschoten: The matrixprofile project certainly is interesting and I had approved the original pull request. On the other hand, I have not seen any of the matrix profiles features as statistically significant features in one of my projects. Therefore, I am going to remove the dependency on matrix profiles.
I'm using an M1 chip and had the same issues when trying to install it under python 3.9. However, with Python 3.9.7 pip install worked. You might need to
brew install llvm
Unfortunately this didn't work for me, but it sounds like the matrixprofile dependency is going to be removed so I will wait.
The solution I've found was to downgrade the Python version to 3.10.8 using pyenv in order to easily manage multiple Python versions (useful resource: https://opensource.com/article/19/5/python-3-default-mac).
Fixed in https://github.com/blue-yonder/tsfresh/tree/v0.20.0. Many thanks for your patience and perseverance.
Merry Christmas and a Happy New Year!