mcrouter
mcrouter copied to clipboard
facebook
Build Process Is Not Intuitive
I had some issues building a version of mcrouter to ship to different servers. After some amount of time trying to figure out the correct ordering of mcrouter HEAD, WANGLE_COMMIT and FOLLY_COMMIT values, I was able to create a binary, but the process seemed way too dependent on upstream OS ability to provide the correct versions of boost, automake, autoconf, ragel, double-conversion, glog and gflag.
I wanted to automate this process so I could create binaries that were less dependent on the OS packaging dependencies, and so created this Dockerfile and workflow to generate a mcrouter artifact. The ldd hack I'm not particularly proud of / confident with, but it is at least getting me to the point where I'm able to experiment with mcrouter.
Included are the Dockerfile and build scripts. Because this is currently more suited to my workflow than anything else, I'm making this an issue rather than a PR, but would promote this to a PR if any of this seems useful to the mcrouter team ( or random internet person who has the same problems I had w/ building this ).
Workflow:
mkdir target # Make the directory where artifact will be built
sudo ./build.sh # Build the docker image
sudo ./run.sh # Run the compile, mcrouter-v0.35.0.tar.bz2 will be in target/ after completion
Dockerfile
FROM centos:7
MAINTAINER mcrouter <[email protected]>
ENV BUILD_DIR /opt/facebook
ENV TARGET_DIR /target/
ENV GFLAGS_VERSION v2.2.0
ENV GLOG_VERSION v0.3.4
ENV DOUBLE_CONV_VERSION v2.0.1
ENV MCROUTER_VERSION v0.35.0
ENV FOLLY_COMMIT 81d9192f8afdf53fcbc7bf572f8ce638516380f8
ENV WANGLE_COMMIT f1704ef198062d9754e4a07857a737381d017140
ENV PATH /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:$TARGET_DIR/bin:$TARGET_DIR/sbin
ENV CFLAGS -I$TARGET_DIR/include/
ENV CXXFLAGS -I$TARGET_DIR/include/
ENV CPPFLAGS -I$TARGET_DIR/include/
ENV LDFLAGS -L$TARGET_DIR/lib/ -Wl,-R,'$ORIGIN/../lib'
RUN yum -y install git
RUN yum -y groupinstall "Development Tools"
RUN yum -y install cmake
RUN yum -y install wget
RUN yum -y install openssl openssl-devel
RUN yum -y install libevent libevent-devel
RUN yum -y install python-devel
RUN yum -y install icu libicu libicu-devel
RUN yum -y install bzip2-devel
RUN yum -y install chrpath
RUN mkdir -p $BUILD_DIR
# automake-1.14
RUN cd $BUILD_DIR && \
wget http://ftp.gnu.org/gnu/automake/automake-1.14.tar.gz && \
tar -zxvf ./automake-1.14.tar.gz
# scons 2.5.1
RUN cd $BUILD_DIR && \
wget https://iweb.dl.sourceforge.net/project/scons/scons/2.5.1/scons-2.5.1-1.noarch.rpm && \
rpm -ivh scons-2.5.1-1.noarch.rpm
# ragel 6.10
RUN cd $BUILD_DIR && \
wget "http://www.colm.net/files/ragel/ragel-6.10.tar.gz" && \
tar -zxvf ragel-6.10.tar.gz
# gflags
RUN cd $BUILD_DIR && \
git clone https://github.com/schuhschuh/gflags.git && \
cd gflags && \
git checkout $GFLAGS_VERSION
# glog
RUN cd $BUILD_DIR && \
git clone https://github.com/google/glog.git && \
cd glog && \
git checkout $GLOG_VERSION
# double_conversion
RUN cd $BUILD_DIR && \
git clone https://github.com/google/double-conversion.git && \
cd double-conversion && \
git checkout $DOUBLE_CONV_VERSION
# folly
RUN cd $BUILD_DIR && \
git clone https://github.com/facebook/folly.git && \
cd folly && \
git checkout $FOLLY_COMMIT
# wangle
RUN cd $BUILD_DIR && \
git clone https://github.com/facebook/wangle.git && \
cd wangle && \
git checkout $WANGLE_COMMIT
# mcrouter
RUN cd $BUILD_DIR && \
git clone https://github.com/facebook/mcrouter.git && \
cd mcrouter && \
git checkout $MCROUTER_VERSION
# boost
RUN cd $BUILD_DIR && \
wget "https://superb-dca2.dl.sourceforge.net/project/boost/boost/1.53.0/boost_1_53_0.tar.gz" && \
tar -zxvf boost_1_53_0.tar.gz
CMD echo "$TARGET_DIR/lib" >> /etc/ld.so.conf && \
cd $BUILD_DIR/automake-1.14 && ./configure --prefix=/usr/ && make && make install && ldconfig && \
cd $BUILD_DIR/boost_1_53_0 && ./bootstrap.sh --prefix=$TARGET_DIR && ./b2 && ./b2 install && \
cd $BUILD_DIR/ragel-6.10 && ./configure --prefix=$TARGET_DIR && make && make install && \
cd $BUILD_DIR/gflags && cmake -DBUILD_SHARED_LIBS=ON -DBUILD_STATIC_LIBS=ON -DCMAKE_INSTALL_PREFIX:PATH=$TARGET_DIR . && make && make install && ldconfig && \
cd $BUILD_DIR/glog && ./configure --enable-static --enable-shared --prefix=$TARGET_DIR && make && make install && ldconfig && \
cd $BUILD_DIR/double-conversion && scons prefix=$TARGET_DIR && scons install prefix=$TARGET_DIR && cmake -DBUILD_SHARED_LIBS=ON -DBUILD_STATIC_LIBS=ON -DCMAKE_INSTALL_PREFIX:PATH=$TARGET_DIR . && make install && ldconfig && \
cd $BUILD_DIR/folly/folly && autoreconf -ivf && ./configure --enable-static --enable-shared --prefix=$TARGET_DIR --with-boost=$TARGET_DIR && make && make install && ldconfig && \
cd $BUILD_DIR/wangle/wangle && cmake -DBUILD_SHARED_LIBS=ON -DCMAKE_INSTALL_PREFIX:PATH=$TARGET_DIR -DFOLLY_INCLUDE_DIR=$TARGET_DIR/include/ -DFOLLY_LIBRARYDIR=$TARGET_DIR/lib/ . && make && make install && ldconfig && \
cd $BUILD_DIR/mcrouter/mcrouter && AUTOMAKE=automake-1.14 ACLOCAL="aclocal-1.14 -I/usr/share/aclocal/" ./autogen.sh && ./configure --enable-static --enable-shared --prefix=$TARGET_DIR --with-boost=$TARGET_DIR && make && make install && \
for i in $(ldd $TARGET_DIR/bin/mcrouter | egrep "(icu|crypto|ssl)" | egrep -o "/[^ ]+"); do cp $i /$TARGET_DIR/lib/; done && \
chrpath -r "\$ORIGIN/../lib/" $TARGET_DIR/bin/mcrouter && \
chrpath -r "\$ORIGIN/../lib/" $TARGET_DIR/bin/mcpiper && \
cd $TARGET_DIR && tar -cjvf /tmp/mcrouter-$MCROUTER_VERSION.tar.bz2 bin/mcrouter bin/mcpiper lib/*.so lib/*.so.* && \
mv /tmp/mcrouter-$MCROUTER_VERSION.tar.bz2 $TARGET_DIR
VOLUME ["/target"]
build.sh
#!/bin/bash
docker build . -t mcrouter:latest
run.sh
#!/bin/bash
docker run -v $(pwd)/target:/target mcrouter:latest
Thanks for posting this Dockerfile! The "official" Dockerfile seems to be failing I assume because the folly dependency isn't right for some reason:
from ./../folly/Format.h:27,
from detail/CacheLocality.cpp:28:
./../folly/FBVector.h: In lambda function:
./../folly/FBVector.h:1432:49: error: parameter packs not expanded with '...':
M_construct(start, std::forward<Args>(args)...);
^
./../folly/FBVector.h:1432:49: note: 'args'
make[2]: *** [detail/CacheLocality.lo] Error 1```
Thanks for posting this @netshade BUT be aware that the binary built using your method was VERY SLOW in our tests.
The test is 3 runs of 10k loops of 10 operations (get, set, delete) done by 10-30 threads to mcrouter using a single pool of 3 memcache server (written in Java and Python to make sure we don't hit any client library differences, but they gave very similar results).
That test was finished after ALMOST 2 TIMES LONGER TIME using your binary than a binary built using https://github.com/gdubicki/mcrouter/blob/master/mcrouter/scripts/install_centos_7.2.sh .
Have you done any performance tests of your binary? Or perhaps even tried to use it production already?
@gdubicki Wow, that is surprising to me; not in that I have evidence to the contrary ( we never used Mcrouter beyond prototyping ), just in that I'm surprised that there is a difference and it's that extreme.
I have no great ideas afa what might actually be contributing to the slowdown ( Different gcc version between Dockerfile and your build machine? CFLAGS on your build machine possibly have optimizations enabled? etc. ), but as a fair warning to anyone coming to this issue later, I would note that the above build script is absolutely hacked together and has not been used in production by myself, at the very least.