loky
loky copied to clipboard
Published
20 hours ago •
joblib
joblib
Not sure if RandomizedsearchCV() is using loky
Hello I was training my classifier using XGBoost with the following code on GCP and eventually I built my own server and now I am having difficulty. On GCP I used to see the message 'backend: using loky backend ...' but not on my own server which makes me think if this code is really taking advantage of parallelism. The server I built is far superior to what I was using on GCP. Below is some information about my code and the specs of the server. I wonder is there is something wrong with my environment.
If I use n_jobs=-1 inside the RandomizedSearchCV() code then I start getting the error below. However if I use n_jobs=-1 inside XGBClassifier() then things work albeit it is slow which makes me think that I am not taking advantage of parallelism. I also do not see the 'Backend: using loky backend ...' message either.
warnings.warn("Estimator fit failed. The score on this train-test"
/home/michael/.local/lib/python3.8/site-packages/sklearn/model_selection/_validation.py:610: FitFailedWarning: Estimator fit failed. The score on this train-test partition for these parameters will be set to nan. Details:
Traceback (most recent call last):
File "/home/michael/.local/lib/python3.8/site-packages/sklearn/model_selection/_validation.py", line 593, in _fit_and_score
estimator.fit(X_train, y_train, **fit_params)
File "/home/michael/.local/lib/python3.8/site-packages/xgboost/core.py", line 422, in inner_f
return f(**kwargs)
File "/home/michael/.local/lib/python3.8/site-packages/xgboost/sklearn.py", line 913, in fit
self._Booster = train(xgb_options, train_dmatrix,
File "/home/michael/.local/lib/python3.8/site-packages/xgboost/training.py", line 206, in train
bst = _train_internal(params, dtrain,
File "/home/michael/.local/lib/python3.8/site-packages/xgboost/training.py", line 101, in _train_internal
bst.update(dtrain, i, obj)
File "/home/michael/.local/lib/python3.8/site-packages/xgboost/core.py", line 1280, in update
_check_call(_LIB.XGBoosterUpdateOneIter(self.handle,
File "/home/michael/.local/lib/python3.8/site-packages/xgboost/core.py", line 189, in _check_call
raise XGBoostError(py_str(_LIB.XGBGetLastError()))
xgboost.core.XGBoostError: [17:24:00] ../src/c_api/../data/../common/common.h:44: ../src/c_api/../data/../common/device_helpers.cuh: 128: out of memory
Stack trace:
[bt] (0) /home/michael/.local/lib/python3.8/site-packages/xgboost/lib/libxgboost.so(+0x8d264) [0x7fccd8733264]
[bt] (1) /home/michael/.local/lib/python3.8/site-packages/xgboost/lib/libxgboost.so(+0x2fc730) [0x7fccd89a2730]
[bt] (2) /home/michael/.local/lib/python3.8/site-packages/xgboost/lib/libxgboost.so(+0x2fcde9) [0x7fccd89a2de9]
[bt] (3) /home/michael/.local/lib/python3.8/site-packages/xgboost/lib/libxgboost.so(+0x32da14) [0x7fccd89d3a14]
[bt] (4) /home/michael/.local/lib/python3.8/site-packages/xgboost/lib/libxgboost.so(+0x339bc5) [0x7fccd89dfbc5]
[bt] (5) /home/michael/.local/lib/python3.8/site-packages/xgboost/lib/libxgboost.so(+0x33aa2b) [0x7fccd89e0a2b]
[bt] (6) /home/michael/.local/lib/python3.8/site-packages/xgboost/lib/libxgboost.so(+0x431dc3) [0x7fccd8ad7dc3]
[bt] (7) /home/michael/.local/lib/python3.8/site-packages/xgboost/lib/libxgboost.so(+0x4384bd) [0x7fccd8ade4bd]
[bt] (8) /home/michael/.local/lib/python3.8/site-packages/xgboost/lib/libxgboost.so(+0x17255b) [0x7fccd881855b]
score='precision_weighted'
# score = true_edge_scorer
param_comb = 1 #number of combination to be tried
params = {
'min_child_weight': [1,3, 5,7, 10, 20, 40],
'gamma':[0, 0.25, 0.5, 1.0],
'subsample': [0.6,0.8, 1.0],
'colsample_bytree': [0.3,0.4,0.6,0.8],
'max_depth':[3,4,6,8] , #'max_depth': [2],
'eta':[ 0.001,0.01,0.1] ,
'n_estimators': [50,70, 100, 200], #'n_estimators': [20],
'reg_alpha': [0.1, 1.0, 5.0, 10.0, 50.0, 100.0],
'reg_lambda': [0.1, 1.0, 5.0, 10.0, 50.0, 100.0],
'tree_method' : ["gpu_hist"]
}
#params={'subsample': [1.0], 'min_child_weight': [5], 'max_depth': [2], 'gamma': [0.5], 'eta': [0.01], 'colsample_bytree': [0.8],'n_estimators':[100]}
model = XGBClassifier(objective='multi:softmax',n_jobs=-1, num_class=3, eval_metric='mlogloss') #base model
random_search = RandomizedSearchCV(model, param_distributions=params,n_iter=param_comb,return_train_score=True, scoring=score,cv=ps,verbose=3, random_state=1001 )
random_search.fit(Train,Train_Labels)
MY HARDWARE:
- AMD Ryzen ThreadRipper Professional Video Editing Custom PC System
- Motherboard: Asus ROG ZENITH II EXTREME Socket sTRX4/ AMD TRX40/ DDR4/ 3-Way CrossFireX & 3-Way SLI/ SATA3&USB3.2/ M.2/ WiFi/ Bluetooth
- Processor: AMD Threadripper 3990X 64-Core 2.9GHz up to 4.3GHz
- Memory: 128GB DDR4-2400/2666 PC4-19200/21300
- Video card: NVIDIA TITAN RTX 24GB GDDR6 PCI-Express
michael@haub-ubuntu:~$ pip3 list Package Version ----------------------- -------------------- altair 4.1.0 anyio 2.0.2 apturl 0.5.2 argon2-cffi 20.1.0 async-generator 1.10 attrs 20.3.0 Babel 2.9.0 backcall 0.2.0 bcrypt 3.1.7 bleach 3.2.1 blinker 1.4 Brlapi 0.7.0 certifi 2019.11.28 cffi 1.14.4 chardet 3.0.4 chrome-gnome-shell 0.0.0 Click 7.0 cloudpickle 1.6.0 colorama 0.4.3 command-not-found 0.3 cryptography 2.8 cupshelpers 1.0 cx-Freeze 6.1 cycler 0.10.0 dbus-python 1.2.16 decorator 4.4.2 defer 1.0.6 defusedxml 0.6.0 distro 1.4.0 distro-info 0.23ubuntu1 duplicity 0.8.12.0 entrypoints 0.3 fasteners 0.14.1 future 0.18.2 httplib2 0.14.0 idna 2.8 ipykernel 5.4.3 ipynb-py-convert 0.4.6 ipython 7.19.0 ipython-genutils 0.2.0 jedi 0.17.2 Jinja2 2.11.2 joblib 1.0.0 json5 0.9.5 jsonschema 3.2.0 jupyter-client 6.1.11 jupyter-core 4.7.0 jupyter-server 1.2.1 jupyterlab 3.0.5 jupyterlab-pygments 0.1.2 jupyterlab-server 2.1.2 keyring 18.0.1 kiwisolver 1.3.1 language-selector 0.1 launchpadlib 1.10.13 lazr.restfulclient 0.14.2 lazr.uri 1.0.3 lockfile 0.12.2 loky 2.9.0 louis 3.12.0 macaroonbakery 1.3.1 Mako 1.1.0 MarkupSafe 1.1.0 matplotlib 3.3.3 mistune 0.8.4 monotonic 1.5 nbclassic 0.2.6 nbclient 0.5.1 nbconvert 6.0.7 nbformat 5.1.0 nest-asyncio 1.4.3 netifaces 0.10.4 notebook 6.2.0 numpy 1.19.5 oauthlib 3.1.0 olefile 0.46 packaging 20.8 pandas 1.2.0 pandas-ta 0.2.28b0 pandocfilters 1.4.3 paramiko 2.6.0 parso 0.7.1 pexpect 4.6.0 pickleshare 0.7.5 Pillow 7.0.0 pip 20.0.2 prometheus-client 0.9.0 prompt-toolkit 3.0.10 protobuf 3.6.1 ptyprocess 0.7.0 pyaml 20.4.0 pycairo 1.16.2 pycparser 2.20 pycups 1.9.73 Pygments 2.7.4 PyGObject 3.36.0 PyJWT 1.7.1 pymacaroons 0.13.0 PyNaCl 1.3.0 pyparsing 2.4.7 pyRFC3339 1.1 pyrsistent 0.17.3 python-apt 2.0.0+ubuntu0.20.4.3 python-dateutil 2.7.3 python-debian 0.1.36ubuntu1 pytz 2019.3 pyxdg 0.26 PyYAML 5.3.1 pyzmq 20.0.0 reportlab 3.5.34 requests 2.22.0 requests-unixsocket 0.2.0 scikit-learn 0.24.0 scipy 1.6.0 screen-resolution-extra 0.0.0 seaborn 0.11.1 SecretStorage 2.3.1 Send2Trash 1.5.0 setuptools 45.2.0 simplejson 3.16.0 six 1.14.0 sklearn 0.0 sniffio 1.2.0 ssh-import-id 5.10 systemd-python 234 TA-Lib 0.4.19 terminado 0.9.2 testpath 0.4.4 threadpoolctl 2.1.0 toolz 0.11.1 tornado 6.1 traitlets 5.0.5 ubuntu-advantage-tools 20.3 ubuntu-drivers-common 0.0.0 ufw 0.36 unattended-upgrades 0.1 urllib3 1.25.8 usb-creator 0.3.7 wadllib 1.3.3 wcwidth 0.2.5 webencodings 0.5.1 wheel 0.34.2 xgboost 1.3.1 xkit 0.0.0 zoom 0.0.15
