velox
velox copied to clipboard
facebookincubator
Download source code archive directly for ubuntu and macos setup
It is better to use wget_and_untar to download the source code archive directly, instead of using git to clone the source code. Because direct download is much faster than using git clone. The setup-centos8.sh script has been consistently downloading source code archive directly.
Deploy Preview for meta-velox canceled.
| Name | Link |
|---|---|
| Latest commit | 8950ec55f70f0551add569060fd8dd810cdd16f0 |
| Latest deploy log | https://app.netlify.com/sites/meta-velox/deploys/6619db453b44e60008cd5781 |
![netlify[bot] avatar](https://avatars.githubusercontent.com/in/13473?v=4 "Published by a pub.dev verified publisher"){kind=small}
In our test environment, for boost, it took about 11 min before (from 2024-03-21T06:20:47 to 2024-03-21T06:31:13):
2024-03-21T06:20:47.3262778Z + run_and_time install_boost
2024-03-21T06:20:47.3263537Z + install_boost
2024-03-21T06:20:47.3264743Z + github_checkout boostorg/boost boost-1.84.0 --recursive
2024-03-21T06:20:47.3271539Z + local REPO=boostorg/boost
2024-03-21T06:20:47.3272490Z + shift
2024-03-21T06:20:47.3273275Z + local VERSION=boost-1.84.0
2024-03-21T06:20:47.3274005Z + shift
2024-03-21T06:20:47.3274716Z + local GIT_CLONE_PARAMS=--recursive
2024-03-21T06:20:47.3275628Z ++ basename boostorg/boost
2024-03-21T06:20:47.3276434Z + Finished running install_fmt
2024-03-21T06:20:47.3277281Z + local DIRNAME=boost
2024-03-21T06:20:47.3278123Z + SUDO='sudo --preserve-env'
2024-03-21T06:20:47.3279167Z + cd /opt/gluten/ep/build-velox/build/velox_ep
2024-03-21T06:20:47.3280153Z + '[' -z boost ']'
2024-03-21T06:20:47.3281462Z + '[' -d boost ']'
2024-03-21T06:20:47.3282170Z + '[' '!' -d boost ']'
2024-03-21T06:20:47.3283545Z + git clone -q -b boost-1.84.0 --recursive https://github.com/boostorg/boost.git
2024-03-21T06:21:00.5554970Z Note: switching to 'ad09f667e61e18f5c31590941e748ac38e5a81bf'.
2024-03-21T06:21:00.5556728Z
2024-03-21T06:21:00.5557598Z You are in 'detached HEAD' state. You can look around, make experimental
2024-03-21T06:21:00.5559211Z changes and commit them, and you can discard any commits you make in this
2024-03-21T06:21:00.5560889Z state without impacting any branches by switching back to a branch.
2024-03-21T06:21:00.5561865Z
2024-03-21T06:21:00.5562552Z If you want to create a new branch to retain commits you create, you may
2024-03-21T06:21:00.5564281Z do so (now or later) by using -c with the switch command. Example:
2024-03-21T06:21:00.5565210Z
2024-03-21T06:21:00.5565585Z git switch -c <new-branch-name>
2024-03-21T06:21:00.5667647Z
2024-03-21T06:21:00.5668525Z Or undo this operation with:
2024-03-21T06:21:00.5669179Z
2024-03-21T06:21:00.5669430Z git switch -
2024-03-21T06:21:00.5788570Z
2024-03-21T06:21:00.5869925Z Turn off this advice by setting config variable advice.detachedHead to false
2024-03-21T06:21:00.5948424Z
2024-03-21T06:31:13.8519110Z + cd boost
It took about 20s now (from 2024-03-21T08:25:15 to 2024-03-21T08:25:32)
2024-03-21T08:25:15.1901014Z + run_and_time install_boost
2024-03-21T08:25:15.1902104Z + install_boost
2024-03-21T08:25:15.1904364Z + wget_and_untar https://github.com/boostorg/boost/releases/download/boost-1.84.0/boost-1.84.0.tar.gz boost
2024-03-21T08:25:15.1907965Z + local URL=https://github.com/boostorg/boost/releases/download/boost-1.84.0/boost-1.84.0.tar.gz
2024-03-21T08:25:15.1909631Z + local DIR=boost
2024-03-21T08:25:15.1910348Z + mkdir -p boost
2024-03-21T08:25:15.1922276Z + pushd boost
2024-03-21T08:25:15.1924758Z + curl -L https://github.com/boostorg/boost/releases/download/boost-1.84.0/boost-1.84.0.tar.gz
2024-03-21T08:25:15.1927531Z /opt/gluten/ep/build-velox/build/velox_ep/fmt/boost /opt/gluten/ep/build-velox/build/velox_ep/fmt
2024-03-21T08:25:15.2047924Z % Total % Received % Xferd Average Speed Time Time Time Current
2024-03-21T08:25:15.2049463Z Dload Upload Total Spent Left Speed
2024-03-21T08:25:15.2050292Z
2024-03-21T08:25:16.2292144Z 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
2024-03-21T08:25:16.2293827Z 0 0 0 0 0 0 0 0 --:--:-- 0:00:01 --:--:-- 0
2024-03-21T08:25:16.2295320Z 0 0 0 0 0 0 0 0 --:--:-- 0:00:01 --:--:-- 0
2024-03-21T08:25:17.3928559Z
2024-03-21T08:25:18.1827510Z 0 118M 0 7421 0 0 3391 0 10:08:26 0:00:02 10:08:24 3391
2024-03-21T08:25:19.1764844Z 0 118M 0 191k 0 0 65753 0 0:31:22 0:00:02 0:31:20 232k
2024-03-21T08:25:20.1637281Z 4 118M 4 5913k 0 0 1489k 0 0:01:21 0:00:03 0:01:18 3310k
2024-03-21T08:25:21.1679483Z 15 118M 15 18.4M 0 0 3807k 0 0:00:31 0:00:04 0:00:27 6811k
2024-03-21T08:25:22.1508048Z 26 118M 26 31.2M 0 0 5364k 0 0:00:22 0:00:05 0:00:17 8469k
2024-03-21T08:25:23.1499963Z 37 118M 37 44.0M 0 0 6490k 0 0:00:18 0:00:06 0:00:12 9473k
2024-03-21T08:25:24.1504551Z 48 118M 48 56.9M 0 0 7343k 0 0:00:16 0:00:07 0:00:09 11.4M
2024-03-21T08:25:25.1513762Z 59 118M 59 69.8M 0 0 8000k 0 0:00:15 0:00:08 0:00:07 12.8M
2024-03-21T08:25:26.1492539Z 70 118M 70 82.8M 0 0 8525k 0 0:00:14 0:00:09 0:00:05 12.9M
2024-03-21T08:25:27.1490347Z 81 118M 81 95.8M 0 0 8966k 0 0:00:13 0:00:10 0:00:03 12.9M
2024-03-21T08:25:27.7938709Z 92 118M 92 108M 0 0 9332k 0 0:00:12 0:00:11 0:00:01 12.9M
2024-03-21T08:25:27.7941828Z 100 118M 100 118M 0 0 9603k 0 0:00:12 0:00:12 --:--:-- 13.1M
2024-03-21T08:25:27.8029221Z + tar -xz --strip-components=1 -f boost.tar.gz
2024-03-21T08:25:32.2022371Z + popd
2024-03-21T08:25:32.2023384Z + cd boost
@assignUser @kgpai Could you help review again? I have added the functionality to change directory to a specific path in the cmake_install method.
@liujiayi771 I havent noticed any particular slowness with git and it seems roughly equivalent to download. The advantage of using git is that its probably a little more secure than plain old downloads of tar.gz's (FYI the checksums we calculate arent really secure anyway). Do other folks see significant differences b/w git and downloading tar.gz's ? @majetideepak ?
@kgpai In my test, the time difference for Boost is quite significant, and the script for CentOS 8 does not use git currently, so both methods should work.