hpx Explore feasibility of removing all dependencies on Boost

A discussion at CSCS took place where the anti-boost argument went as follows

Q: The idea is that eventually the features in HPX will be available via the c++ standard? A: Ideally, yes most of the HPX features will be in c++ Q: Will the c++ standard have a dependency on boost then? A: Of course not. Q: Well then, you need to get rid of the boost dependency in HPX before it will be adopted by users X,Y,Z

Currently we make use of program options which could be replaced (given a lot of effort), by cxxopt or an alternative.

other boost libraries that we use are

atomic - do we need it other than for our compat layer?
filesystem - this should be in c++17?
program_options - could be replaced by alternative in principle
regex - why not use std::regex? (some work converting expressions)
system - no idea if we really need this
chrono - should be in std::
context ?
thread ?
spirit ?

It would seem on casual inspection that we rely on context/thread and others could be removed apart from spirit that is used in parsing thread binding options (yes? elsewhere?).

Are there other parts of our code that rely on boost that are not listed here? How much effort would it take to absorb context/thread into HPX? (and do it in such a way that we could still benefit from upstream fixes)

Does anyone feel strongly about this? (either in favour or against using boost).

Boost dependency tracker:

Accumulators

These can be made optional

[ ] boost/accumulators/accumulators.hpp
[ ] boost/accumulators/framework/accumulator_base.hpp
[ ] boost/accumulators/framework/depends_on.hpp
[ ] boost/accumulators/framework/extractor.hpp
[ ] boost/accumulators/framework/parameters/sample.hpp
[ ] boost/accumulators/numeric/functional.hpp
[ ] boost/accumulators/statistics/count.hpp
[ ] boost/accumulators/statistics_fwd.hpp
[ ] boost/accumulators/statistics/max.hpp
[ ] boost/accumulators/statistics/mean.hpp
[ ] boost/accumulators/statistics/median.hpp
[ ] boost/accumulators/statistics/min.hpp
[ ] boost/accumulators/statistics/rolling_mean.hpp
[ ] boost/accumulators/statistics/rolling_variance.hpp
[ ] boost/accumulators/statistics/rolling_window.hpp
[ ] boost/accumulators/statistics/stats.hpp
[ ] boost/accumulators/statistics/variance.hpp

(string) algorithms

These are genuinely useful, and don't have a good replacement. Are they small enough to copy over?

Removed in #4434

[x] boost/algorithm/string/case_conv.hpp
[x] boost/algorithm/string/classification.hpp
[x] boost/algorithm/string.hpp
[x] boost/algorithm/string/predicate.hpp
[x] boost/algorithm/string/replace.hpp
[x] boost/algorithm/string/split.hpp
[x] boost/algorithm/string/trim.hpp

Serialization

These are leftovers and can be removed

[x] boost/archive/basic_archive.hpp
[x] boost/archive/text_iarchive.hpp
[x] boost/archive/text_oarchive.hpp

ASIO thread pools (#5158)

The service pools can already be disabled at compile time.

[x] boost/asio/basic_waitable_timer.hpp
[x] boost/asio/buffer.hpp
[x] boost/asio/detail/winsock_init.hpp
[x] boost/asio/error.hpp
[x] boost/asio/io_service.hpp
[x] boost/asio/ip/address_v4.hpp
[x] boost/asio/ip/address_v6.hpp
[x] boost/asio/ip/host_name.hpp
[x] boost/asio/ip/tcp.hpp
[x] boost/asio/placeholders.hpp
[x] boost/asio/posix/stream_descriptor.hpp
[x] boost/asio/read.hpp
[x] boost/asio/windows/stream_handle.hpp
[x] boost/asio/write.hpp

Context switching

This one is optional (generic coroutines)

[x] boost/context/detail/fcontext.hpp

Exception

These are used in networking (made optional in #4365), needed only if the user enables ASIO_HAS_BOOST_THROW_EXCEPTION)

[x] boost/exception/diagnostic_information.hpp
[x] boost/exception/exception.hpp

Filesystem

These need a compatibility layer. In progress: #4036.

[x] boost/filesystem/convenience.hpp
[x] boost/filesystem/exception.hpp
[x] boost/filesystem.hpp
[x] boost/filesystem/operations.hpp
[x] boost/filesystem/path.hpp

Spirit/fusion/phoenix

Can be made optional by making performance counters and thread affinity optional. Needs to be replaced with an alternative in other places.

[ ] boost/spirit/home/qi/auxiliary/lazy.hpp
[ ] boost/spirit/home/qi/detail/enable_lit.hpp
[ ] boost/spirit/home/qi/detail/string_parse.hpp
[ ] boost/spirit/home/qi/domain.hpp
[ ] boost/spirit/home/qi/meta_compiler.hpp
[ ] boost/spirit/home/qi/parser.hpp
[ ] boost/spirit/home/qi/skip_over.hpp
[ ] boost/spirit/home/support/char_class.hpp
[ ] boost/spirit/home/support/common_terminals.hpp
[ ] boost/spirit/home/support/detail/get_encoding.hpp
[ ] boost/spirit/home/support/handles_container.hpp
[ ] boost/spirit/home/support/info.hpp
[ ] boost/spirit/home/support/modify.hpp
[ ] boost/spirit/home/support/string_traits.hpp
[ ] boost/spirit/home/support/unused.hpp
[ ] boost/spirit/include/phoenix_core.hpp
[ ] boost/spirit/include/phoenix.hpp
[ ] boost/spirit/include/phoenix_object.hpp
[ ] boost/spirit/include/phoenix_operator.hpp
[ ] boost/spirit/include/qi_action.hpp
[ ] boost/spirit/include/qi_alternative.hpp
[ ] boost/spirit/include/qi_auxiliary.hpp
[ ] boost/spirit/include/qi_char.hpp
[ ] boost/spirit/include/qi_directive.hpp
[ ] boost/spirit/include/qi.hpp
[ ] boost/spirit/include/qi_nonterminal.hpp
[ ] boost/spirit/include/qi_numeric.hpp
[ ] boost/spirit/include/qi_operator.hpp
[ ] boost/spirit/include/qi_parse.hpp
[ ] boost/spirit/include/qi_sequence.hpp
[ ] boost/spirit/include/qi_string.hpp
[ ] boost/spirit/include/qi_uint.hpp
[ ] boost/spirit/include/support_istream_iterator.hpp
[ ] boost/fusion/include/adapt_struct.hpp
[ ] boost/fusion/include/at.hpp
[ ] boost/fusion/include/define_struct.hpp
[ ] boost/fusion/include/io.hpp
[ ] boost/fusion/include/pair.hpp
[ ] boost/fusion/include/std_pair.hpp
[ ] boost/phoenix/bind.hpp
[ ] boost/phoenix/core.hpp

Iostreams

Used in iostreams and some components.

[x] boost/io/ios_state.hpp (#4365)
[ ] boost/iostreams/device/file_descriptor.hpp
[ ] boost/iostreams/filter/bzip2.hpp
[ ] boost/iostreams/filter/zlib.hpp
[ ] boost/iostreams/stream.hpp

Iterator

Mostly used in tests and examples, also libfabric.

[x] boost/iterator/counting_iterator.hpp (#4365)
[x] boost/iterator/iterator_adaptor.hpp (#4365)
[x] boost/iterator/iterator_categories.hpp (#5183)
[x] boost/iterator/iterator_facade.hpp (#4365)

Queues

Deque is already copied, why not the others? (see #6098)

[x] boost/lockfree/detail/prefix.hpp
[x] boost/lockfree/detail/tagged_ptr.hpp
[x] boost/lockfree/policies.hpp
[x] boost/lockfree/queue.hpp
[x] boost/lockfree/stack.hpp

Log

Used in parcelports. Can be replaced with our logging.

[ ] boost/log/trivial.hpp

Pool

Used in verbs parcelport.

[x] boost/pool/poolfwd.hpp (#5093)
[x] boost/pool/simple_segregated_storage.hpp (#5093)

Program options

Can be copied over (#4040).

[x] boost/program_options.hpp
[x] boost/program_options/options_description.hpp
[x] boost/program_options/parsers.hpp
[x] boost/program_options/variables_map.hpp

Range

Mostly used in examples, some uses in the core library (#6095).

[x] boost/range/adaptor/filtered.hpp
[x] boost/range/algorithm/copy.hpp
[x] boost/range/algorithm/for_each.hpp
[x] boost/range/begin.hpp
[x] boost/range/counting_range.hpp
[x] boost/range/end.hpp
[x] boost/range/functions.hpp
[x] boost/range/irange.hpp
[x] boost/range/iterator.hpp
[x] boost/range/mutable_iterator.hpp
[x] boost/range/numeric.hpp

SIMD

To be deleted (#4028).

[x] boost/simd/function/aligned_load.hpp
[x] boost/simd/function/aligned_store.hpp
[x] boost/simd/function/load.hpp
[x] boost/simd/function/store.hpp
[x] boost/simd/function/sum.hpp
[x] boost/simd.hpp

SmartPtr

Used in concurrency module.

[x] boost/smart_ptr/detail/spinlock.hpp (#5008, #5137)
[x] boost/smart_ptr/shared_array.hpp (#4801)

System

Header only since 1.69. Do we want to completely get rid of it?

[x] boost/system/error_code.hpp
[x] boost/system/system_error.hpp

Miscellaneous

[x] boost/any.hpp hpx::util::any
[ ] boost/array.hpp We add serialization support for it; tuple has overloads for it but just use C++11 std::array?
[x] boost/assert.hpp HPX_ASSERT
[x] boost/assign/std/vector.hpp Not necessary (#4843)
[x] boost/bimap.hpp No real replacement? (#5252)
[x] boost/bind/arg.hpp C++11; can remove?
[x] boost/bind.hpp Only in test; replace with hpx::util::bind
[ ] boost/config.hpp ...
[x] boost/config/warning_disable.hpp ... (#5137)
[x] boost/container/small_vector.hpp libfabric parcelport, future_data (#5621)
[ ] boost/crc.hpp parcelport logging
[x] boost/cstdint.hpp uint128_t?
[x] boost/current_function.hpp hpx/assertion/current_function.hpp
[x] boost/detail/sp_typeinfo.hpp used in hpx::util::any
[x] boost/detail/workaround.hpp ...
[x] boost/dynamic_bitset.hpp optional for cpu mask (#5851)
[x] boost/foreach.hpp only used in inspect
[ ] boost/functional/hash.hpp used in hpx::util::any
[ ] boost/function.hpp replace with hpx::util::function (perf test only)
[x] boost/generator_iterator.hpp used in PAPI perf counter (#5137)
[x] boost/integer/common_factor.hpp (perf test only) (#6174)
[x] boost/integer.hpp partitioned vector/unordered map iterators (#4843)
[x] boost/intrusive_ptr.hpp everywhere (#4138)
[x] boost/intrusive/slist.hpp everywhere (#6077)
[x] boost/lexical_cast.hpp many places (#4288)
[x] boost/math/constants/constants.hpp 1d wave equation (#4843)
[ ] boost/multi_array.hpp only for serialization support
[ ] boost/next_prior.hpp inspect
[x] boost/operators.hpp a couple of utilities (#4843)
[ ] boost/parameter/keyword.hpp histogram
[x] boost/predef/other/endian.h serialization, networking (#5137)
[ ] boost/preprocessor.hpp parcelport logging, replace with ours?
[x] boost/ref.hpp only for compatibility? only support std::reference_wrapper?
[x] boost/shared_array.hpp std::shared_ptr<T[]> (#4801)
[x] boost/shared_ptr.hpp std::shared_ptr (#4050)
[x] boost/swap.hpp std::swap, only used in iostreams
[x] boost/thread/locks.hpp tests, examples, and unordered map (#4843)
[x] boost/thread/tss.hpp not needed if native tls is available (#4843)
[x] boost/tokenizer.hpp inspect, command line and configuration handling (#6096)
[x] boost/utility/addressof.hpp not used (we define addressof in std namespace!), use std
[x] boost/utility/binary.hpp binary literals from C++14? copy over?
[x] boost/utility/string_ref.hpp std::string&? std::string_view? C++17... (#6093)
[x] boost/utility/swap.hpp std::swap
[x] boost/variant.hpp Needs replacement
[ ] boost/version.hpp ...

Sep 03 '18 17:09 biddisco

Current status:

boost::chrono is not used anymore,
boost::context is used on certain platforms only.
boost::thread is not be used anymore

You forgot to mention boost::asio in your list above, we depend on it for the TCP parcel-port, the system timer functionality, and for general kernel thread scheduling on the main thread.

Easy targets for removal are indeed boost::regex and boost::atomic. @K-ballo has been working on both but got stuck for various reasons. boost::filesystem could be added through a compatibility layer to keep supporting older platforms.

The elephant in the room (in terms of header only libraries) is boost::spirit which I don't even begin understanding how to remove without major effort. It however depends on a rather large list of other Boost libraries (fusion, phoenix, mpl, type traits, variant, to name a few). Spirit alone may prevent us from becoming fully independent from Boost.

Let me however add that I think the discussion cited above is inapproproate at best. For instance, what benefit would we have from replacing boost::program_options with some other external dependency? From my understanding, the main reason for people being against Boost is that they are not able/willing to read documentation and/or properly configure dependencies in general. How would yet another external dependency help? All of those arguments are mostly an attempt to find excuses not to have to learn something new...

Sep 05 '18 10:09 hkaiser

boost::atomic is mostly gone, only one use case remains. Some deprecated executor is making use of an invalid atomic<> specialization which standard implementations diagnose, we are waiting for that executor to simply be removed.
boost::regex is a tricky one, std::regex lacks multiline support before 17, and standard implementations aren't capable of dealing with big inputs as boost one is (matches that take a couple seconds to boost take up to 20mins for std, when it doesn't bail out early due to complexity/limits). Many inspect checks rely on matching regexes with lot of backtracking on entire source files, switching to std::regex ruins those checks completely. There's been an attempt to replace inspect checks regexes with token-based matchers, it was eventually abandoned in favor of clang's ast matcher but it could be revived.
boost::system is the real tricky one. Supporting boost::error_code alongside std::error_code (not to mention the weird hpx::error_code) is a nightmare, as it is an out parameter, any function taking one has to be rewraped to perform the mapping. Supporting boost::error_category alongside std::error_category is doable (recent versions should already be doing it). Since both asio and filesystem use it all over their interfaces, and filesystem has to remain in a compatibility layer for a while, it is simply not possible to just replace everything with std::error_code at once.

These are the current boost #inclusion numbers in core HPX, ignoring compatibility:

COUNT | LINE
-----------------------------------------------------
  31 | boost/intrusive_ptr.hpp
  17 | boost/version.hpp
  14 | boost/system/error_code.hpp
  13 | boost/filesystem/path.hpp
  11 | boost/shared_array.hpp
  10 | boost/program_options/options_description.hpp
  10 | boost/ref.hpp
   9 | boost/program_options/variables_map.hpp
   8 | boost/asio/io_service.hpp
   8 | boost/asio/ip/tcp.hpp
   8 | boost/dynamic_bitset.hpp
   7 | boost/algorithm/string/classification.hpp
   7 | boost/algorithm/string/split.hpp
   7 | boost/asio/ip/host_name.hpp
   7 | boost/current_function.hpp
   7 | boost/io/ios_state.hpp
   7 | boost/lexical_cast.hpp
   7 | boost/regex.hpp
   7 | boost/smart_ptr/detail/spinlock.hpp
   7 | boost/tokenizer.hpp
   6 | boost/assign/std/vector.hpp
   6 | boost/config.hpp
   6 | boost/iostreams/device/file_descriptor.hpp
   6 | boost/program_options.hpp
   5 | boost/filesystem.hpp
   5 | boost/shared_ptr.hpp
   5 | boost/simd.hpp
   4 | boost/accumulators/statistics_fwd.hpp
   4 | boost/asio/basic_waitable_timer.hpp
   4 | boost/detail/endian.hpp
   4 | boost/exception/exception.hpp
   4 | boost/filesystem/convenience.hpp
   4 | boost/filesystem/operations.hpp
   4 | boost/spirit/include/qi_numeric.hpp
   4 | boost/spirit/include/qi_parse.hpp
   4 | boost/system/system_error.hpp
   3 | boost/accumulators/accumulators.hpp
   3 | boost/accumulators/framework/accumulator_base.hpp
   3 | boost/accumulators/framework/depends_on.hpp
   3 | boost/accumulators/framework/extractor.hpp
   3 | boost/accumulators/framework/parameters/sample.hpp
   3 | boost/accumulators/numeric/functional.hpp
   3 | boost/accumulators/statistics/count.hpp
   3 | boost/accumulators/statistics/max.hpp
   3 | boost/accumulators/statistics/mean.hpp
   3 | boost/accumulators/statistics/min.hpp
   3 | boost/accumulators/statistics/stats.hpp
   3 | boost/algorithm/string.hpp
   3 | boost/algorithm/string/case_conv.hpp
   3 | boost/intrusive/slist.hpp
   3 | boost/range/irange.hpp
   3 | boost/spirit/include/qi_char.hpp
   3 | boost/spirit/include/qi_operator.hpp
   3 | boost/spirit/include/qi_string.hpp
   3 | boost/spirit/include/qi.hpp
   3 | boost/utility/swap.hpp
   3 | boost/variant.hpp
   2 | boost/accumulators/statistics/median.hpp
   2 | boost/accumulators/statistics/rolling_variance.hpp
   2 | boost/accumulators/statistics/rolling_window.hpp
   2 | boost/accumulators/statistics/variance.hpp
   2 | boost/any.hpp
   2 | boost/array.hpp
   2 | boost/asio/buffer.hpp
   2 | boost/asio/error.hpp
   2 | boost/asio/placeholders.hpp
   2 | boost/asio/read.hpp
   2 | boost/asio/write.hpp
   2 | boost/bimap.hpp
   2 | boost/config/warning_disable.hpp
   2 | boost/cstdint.hpp
   2 | boost/detail/interlocked.hpp
   2 | boost/exception/diagnostic_information.hpp
   2 | boost/fusion/include/adapt_struct.hpp
   2 | boost/fusion/include/define_struct.hpp
   2 | boost/fusion/include/io.hpp
   2 | boost/generator_iterator.hpp
   2 | boost/integer.hpp
   2 | boost/iostreams/filter/zlib.hpp
   2 | boost/lockfree/queue.hpp
   2 | boost/operators.hpp
   2 | boost/program_options/parsers.hpp
   2 | boost/range/algorithm/for_each.hpp
   2 | boost/smart_ptr/scoped_ptr.hpp
   2 | boost/spirit/include/phoenix_core.hpp
   2 | boost/spirit/include/phoenix_object.hpp
   2 | boost/spirit/include/phoenix_operator.hpp
   2 | boost/spirit/include/qi_auxiliary.hpp
   2 | boost/spirit/include/qi_nonterminal.hpp
   2 | boost/spirit/include/qi_uint.hpp
   2 | boost/thread/tss.hpp
   2 | boost/utility/string_ref.hpp
   1 | boost/accumulators/statistics/moment.hpp
   1 | boost/accumulators/statistics/rolling_mean.hpp
   1 | boost/accumulators/statistics/sum.hpp
   1 | boost/algorithm/string/predicate.hpp
   1 | boost/algorithm/string/replace.hpp
   1 | boost/asio/detail/winsock_init.hpp
   1 | boost/asio/ip/address_v4.hpp
   1 | boost/asio/ip/address_v6.hpp
   1 | boost/asio/posix/stream_descriptor.hpp
   1 | boost/asio/windows/stream_handle.hpp
   1 | boost/assert.hpp
   1 | boost/atomic.hpp
   1 | boost/bind/arg.hpp
   1 | boost/container/small_vector.hpp
   1 | boost/context/all.hpp
   1 | boost/context/detail/fcontext.hpp
   1 | boost/detail/sp_typeinfo.hpp
   1 | boost/filesystem/exception.hpp
   1 | boost/functional/hash.hpp
   1 | boost/fusion/include/at.hpp
   1 | boost/fusion/include/pair.hpp
   1 | boost/fusion/include/std_pair.hpp
   1 | boost/iostreams/filter/bzip2.hpp
   1 | boost/iostreams/stream.hpp
   1 | boost/iterator/counting_iterator.hpp
   1 | boost/iterator/iterator_adaptor.hpp
   1 | boost/iterator/iterator_categories.hpp
   1 | boost/iterator/iterator_facade.hpp
   1 | boost/lockfree/detail/prefix.hpp
   1 | boost/lockfree/detail/tagged_ptr.hpp
   1 | boost/lockfree/policies.hpp
   1 | boost/multi_array.hpp
   1 | boost/parameter/keyword.hpp
   1 | boost/phoenix/bind.hpp
   1 | boost/phoenix/core.hpp
   1 | boost/range/adaptor/filtered.hpp
   1 | boost/range/algorithm/copy.hpp
   1 | boost/range/begin.hpp
   1 | boost/range/counting_range.hpp
   1 | boost/range/end.hpp
   1 | boost/range/numeric.hpp
   1 | boost/scoped_array.hpp
   1 | boost/scoped_ptr.hpp
   1 | boost/shmem/shmem_named_shared_object.hpp
   1 | boost/signals2.hpp
   1 | boost/simd/function/aligned_load.hpp
   1 | boost/simd/function/aligned_store.hpp
   1 | boost/simd/function/load.hpp
   1 | boost/simd/function/store.hpp
   1 | boost/simd/function/sum.hpp
   1 | boost/spirit/home/qi/auxiliary/lazy.hpp
   1 | boost/spirit/home/qi/detail/enable_lit.hpp
   1 | boost/spirit/home/qi/detail/string_parse.hpp
   1 | boost/spirit/home/qi/domain.hpp
   1 | boost/spirit/home/qi/meta_compiler.hpp
   1 | boost/spirit/home/qi/parser.hpp
   1 | boost/spirit/home/qi/skip_over.hpp
   1 | boost/spirit/home/support/char_class.hpp
   1 | boost/spirit/home/support/common_terminals.hpp
   1 | boost/spirit/home/support/detail/get_encoding.hpp
   1 | boost/spirit/home/support/handles_container.hpp
   1 | boost/spirit/home/support/info.hpp
   1 | boost/spirit/home/support/modify.hpp
   1 | boost/spirit/home/support/string_traits.hpp
   1 | boost/spirit/home/support/unused.hpp
   1 | boost/spirit/include/qi_alternative.hpp
   1 | boost/spirit/include/qi_directive.hpp
   1 | boost/spirit/include/qi_sequence.hpp
   1 | boost/swap.hpp
   1 | boost/type_traits/integral_constant.hpp
   1 | boost/type_traits/is_same.hpp
   1 | boost/type_traits/remove_reference.hpp
   1 | boost/type_traits/type_with_alignment.hpp
   1 | boost/utility.hpp
   1 | boost/utility/addressof.hpp
   1 | boost/utility/binary.hpp

Sep 05 '18 13:09 K-ballo

Let me however add that I think the discussion cited above is inapproproate at best. For instance, what benefit would we have from replacing boost::program_options with some other external dependency? From my understanding, the main reason for people being against Boost is that they are not able/willing to read documentation and/or properly configure dependencies in general. How would yet another external dependency help? All of those arguments are mostly an attempt to find excuses not to have to learn something new...

program_options was just an example. the idea was not to introduce a new external dependency, but to simply add an include file to hpx::util::cxxopts or something of that kind.

The problem is simply that we have a massive dependency on a set of libraries that are poorly maintained and we frequently have issues with breakages or inconsistencies between different versions of boost. Other people out there have the same problem and they have stopped using boost. They won't touch HPX as long as it depends on boost. I disagree with them and their reasoning. I wanted to get some feedback on just how hard it would be to remove boost.

Where is spirit used? I seem to recall it's in the parsing of thread bindings. Where else?

Sep 05 '18 14:09 biddisco

@K-ballo Thanks for compiling that list of #include files. It's a worryingly long one.

Sep 05 '18 14:09 biddisco

program_options was just an example. the idea was not to introduce a new external dependency, but to simply add an include file to hpx::util::cxxopts or something of that kind.

If you're suggesting to add things to HPX so easily, why are you opposed to adding a simple thread-local allocator arguing about it being 'a shame to see the code base grow in this way'?

The problem is simply that we have a massive dependency on a set of libraries that are poorly maintained and we frequently have issues with breakages or inconsistencies between different versions of boost.

This is quite a mouthful and mostly intended to be spreading FUD. Spirit for instance is a very well supported library with unmatched performance and functionality anywhere else. Same is true for many other Boost libraries we work with (program_options, regex, and others to name a few).

Where is spirit used? I seem to recall it's in the parsing of thread bindings. Where else?

Spirit is used for parsing:

thread bindings
performance counter names
various smaller parsers here and there (SLURM configuration, general runtime configuration, some examples)

Sep 05 '18 14:09 hkaiser

If you're suggesting to add things to HPX so easily, why are you opposed to adding a simple thread-local allocator arguing about it being 'a shame to see the code base grow in this way'?

Your criticism is valid, however, I was willing to include a single header parser, but remove a dependency on a larger library. The thread allocator is one that you found on the web and is not a supported library with users reporting bugs and supplying fixes, which the suggested program options replacement is. My objection was mostly that this functionality might already be present in jemalloc and it'd be nice to know if we can make use of something we already use.

For performance counter names - does having a parser buy much more than a regex would give us?

Sep 05 '18 17:09 biddisco

For performance counter names - does having a parser buy much more than a regex would give us?

I don't think the performance counter names could be 'parsed' using a regex.

Sep 05 '18 18:09 hkaiser

The thread allocator is one that you found on the web and is not a supported library

What I proposed as a PR is a major rewrite of an idea found on the web. It is not supported in the same way as the rest of HPX is not supported, however.

Sep 05 '18 18:09 hkaiser

@sithhell wrote on IRC

We should just submodule the boost libs we need and include them in our build system. Then it looks like we're free of boost...

And this is very much what I am hoping becomes possible with boost eventually. Do we have any idea of when the modularization work in boost is likely to make this feasible?

Sep 06 '18 05:09 biddisco

We should just submodule the boost libs we need and include them in our build system. Then it looks like we're free of boost...

And this is very much what I am hoping becomes possible with boost eventually.

@biddisco Git submodules are generally a PITA, in my experience.

Do we have any idea of when the modularization work in boost is likely to make this feasible?

vcpkg (https://github.com/Microsoft/vcpkg) relies on modularized Boost. I'm not sure how much effort they have put into making this happen.

Sep 06 '18 12:09 hkaiser

git submodules are a PITA but we currently use a slight variation for APEX with our cmake git_external/submodule script that hides some of the pain and that could be implemented for those parts of boost that we wanted to embed.

Sep 06 '18 13:09 biddisco

I'd like to go ahead and remove the thread compatibility layer as there should be support for it everywhere (and it's off by default). Are there actually platforms where we know that this is still needed?

Apr 12 '19 12:04 msimberg

I think we can all agree that this is not going to be straightforward. This is never going to happen if we have to find full replacements for all the boost functionality. It can happen with compatibility layers and making features optional. And while we can tell users "just install boost", I think HPX still becomes more approachable with an option of not depending on Boost. It's just one less thing to worry about.

How about actually copying program_options into our repo. It ~hasn't had updates in a few years and~ (not quite true...) has only a few other boost dependencies that I think we can replace with our implementations (shared_ptr, any, function, iterator_facade). The question is if we still have to support e.g. hpx::init overloads with boost::program_options or if we can go directly to hpx::program_options.
Boost.System is everywhere, but only because of the service pools. The service pools are already optional and we could make Boost.System optional if all of the service pools are disabled.
Filesystem in a compatibility layer as @K-ballo already suggested.

These three are the only compulsory compiled libraries that we depend on. That still leaves the header-only dependencies.

Boost.Spirit could be made optional by making the performance counters and the thread affinity syntax optional.
Boost.Accumulator could be made optional (those counters would just not be available).
The rest I think are mostly single-header utilities that we could move over to our repo, if we don't already have the replacement in place.

Remember that e.g. the logging functionality is already Boost functionality copied over and customized for our needs. I don't see how copying the other utilities that we need would be any different.

PS. I'm aware that this would add complexity to testing with a bunch of new optional features, but I think that's acceptable. Ideally many of these options would be batched up into bigger meta-options (HPX_WITH_BOOST=OFF).

@kshitij12345

Aug 21 '19 10:08 msimberg

How about actually copying program_options into our repo. It ~hasn't had updates in a few years and~ (not quite true...) has only a few other boost dependencies that I think we can replace with our implementations (shared_ptr, any, function, iterator_facade). The question is if we still have to support e.g. hpx::init overloads with boost::program_options or if we can go directly to hpx::program_options.

I'd be fine with that. We'll have to add variant to the list of things to have in HPX, though. FWIW, we use Park's implementation of variant in Phylanx - this could be moved over (including serialization support, etc.).

Boost.System is everywhere, but only because of the service pools. The service pools are already optional and we could make Boost.System optional if all of the service pools are disabled.

Disabling the timer-pools, while they are already optional (are they?), would disable all timed operations (sleep_until, etc.). But we might be able to rig up some alternative timer support.

Filesystem in a compatibility layer as @K-ballo already suggested.

Yes, agreed.

These three are the only compulsory compiled libraries that we depend on. That still leaves the header-only dependencies.

Boost.Spirit could be made optional by making the performance counters and the thread affinity syntax optional.

Additionally, the SLURM batch environment handling, some runtime parameter handling, and some examples depend on Spirit. Those uses are fairly simple and could be rewritten, however.

Boost.Accumulator could be made optional (those counters would just not be available).

Ok. Fair enough.

The rest I think are mostly single-header utilities that we could move over to our repo, if we don't already have the replacement in place.

In principle I'd agree. We would have to create a list of things to see how many additional support facilities this would entail.

Aug 21 '19 13:08 hkaiser

Disabling the timer-pools, while they are already optional (are they?), would disable all timed operations (sleep_until, etc.). But we might be able to rig up some alternative timer support.

HPX_WITH_TIMER_POOL etc. They're compulsory if you want timed operations but if you don't need them they can be disabled.

In principle I'd agree. We would have to create a list of things to see how many additional support facilities this would entail.

Indeed. I was going by @K-ballo's list and didn't see too many headers outside of the big libraries, but I'll put together a new list.

Aug 22 '19 06:08 msimberg

Additionally, the SLURM batch environment handling, some runtime parameter handling, and some examples depend on Spirit. Those uses are fairly simple and could be rewritten, however.

Most of the stuff like slurm parsing, affinity argument handling etc is not 'critical' in the sense that if regexes were used, it's a one off cost that would not impact performance. True?

Does the performance counter syntax incur a runtime cost that is significantly improved by using boost spirit? This seems like the one place where it might make a noticable runtime impact. Is this correct and are there other places where it matters?

Aug 22 '19 07:08 biddisco

A complete list of Boost dependencies with incomplete annotations: https://gist.github.com/msimberg/3a1b4273297ce931717262bcf094f011.

Aug 22 '19 09:08 msimberg

Does the performance counter syntax incur a runtime cost that is significantly improved by using boost spirit? This seems like the one place where it might make a noticable runtime impact. Is this correct and are there other places where it matters?

@biddisco I don't think replacing Spirit will incur a measurable runtime overhead. While I agree with using regexes (or manual parsing using string facilities) for the smaller usages of Spirit is feasible, I don't think its possible to replace the perf-counter name parser with some regexes. The syntax is fairly complex. Same is probably true for the affinity description.

However I'd be fine with optionally disabling those two things if somebody doesn't want to have Boost as a dependency. Performance counters and the affinity bindings are important but not mission critical.

Aug 22 '19 12:08 hkaiser

I added the list of boost headers to to main issue description where we can keep track of what to do and what has been done about the different dependencies. Please update as you see fit.

@kshitij12345 I pinged you to let you know that if you're still interested in this there would be some good starting points here. If you want to go easy first you could look at some of the headers from the miscellaneous section. If you want to start at the deep end you can have a look at spirit. If you're unsure about something please check with us first.

Aug 23 '19 08:08 msimberg

@msimberg Oh Thank You Very Much. Sorry have been a bit busy. Will start with it around October. Will check with you once before starting with it. Thank you again :)

Aug 26 '19 06:08 kshitij12345

Hi folks: Can I PR or is the ticket for internal folks?

Sep 04 '19 12:09 tapaswenipathak

@tapaswenipathak feel free to work on this. My comment above to @kshitij12345 applies to you as well. The miscellaneous section is a good place to start.

Sep 04 '19 12:09 msimberg

Taken from here: https://gist.github.com/K-ballo/7b551d71487bdf34d0b25aa034e399a5

COUNT    | LINE
-----------------------------------------------------
      11 | <boost/asio/io_service.hpp>
      11 | <boost/shared_array.hpp>
      10 | <boost/tokenizer.hpp>
       9 | <boost/asio/ip/tcp.hpp>
       9 | <boost/utility/string_ref.hpp>
       8 | <boost/system/error_code.hpp>
       8 | <boost/version.hpp>
       6 | <boost/dynamic_bitset.hpp>
       6 | <boost/iostreams/device/file_descriptor.hpp>
       5 | <boost/asio/ip/host_name.hpp>
       5 | <boost/smart_ptr/detail/spinlock.hpp>
       4 | <boost/accumulators/statistics_fwd.hpp>
       4 | <boost/config.hpp>
       4 | <boost/exception/exception.hpp>
       4 | <boost/predef/other/endian.h>
       4 | <boost/spirit/include/qi_numeric.hpp>
       4 | <boost/spirit/include/qi_parse.hpp>
       4 | <boost/system/system_error.hpp>
       3 | <boost/accumulators/accumulators.hpp>
       3 | <boost/accumulators/framework/accumulator_base.hpp>
       3 | <boost/accumulators/framework/depends_on.hpp>
       3 | <boost/accumulators/framework/extractor.hpp>
       3 | <boost/accumulators/framework/parameters/sample.hpp>
       3 | <boost/accumulators/numeric/functional.hpp>
       3 | <boost/accumulators/statistics/max.hpp>
       3 | <boost/accumulators/statistics/min.hpp>
       3 | <boost/lockfree/queue.hpp>
       3 | <boost/range/irange.hpp>
       3 | <boost/spirit/include/qi_char.hpp>
       3 | <boost/spirit/include/qi_operator.hpp>
       3 | <boost/spirit/include/qi_string.hpp>
       2 | <boost/accumulators/statistics/count.hpp>
       2 | <boost/accumulators/statistics/mean.hpp>
       2 | <boost/accumulators/statistics/median.hpp>
       2 | <boost/accumulators/statistics/rolling_variance.hpp>
       2 | <boost/accumulators/statistics/rolling_window.hpp>
       2 | <boost/accumulators/statistics/stats.hpp>
       2 | <boost/accumulators/statistics/variance.hpp>
       2 | <boost/asio/basic_waitable_timer.hpp>
       2 | <boost/asio/buffer.hpp>
       2 | <boost/asio/error.hpp>
       2 | <boost/asio/placeholders.hpp>
       2 | <boost/asio/read.hpp>
       2 | <boost/asio/write.hpp>
       2 | <boost/exception/diagnostic_information.hpp>
       2 | <boost/fusion/include/adapt_struct.hpp>
       2 | <boost/intrusive/slist.hpp>
       2 | <boost/program_options/detail/cmdline.hpp>
       2 | <boost/program_options/options_description.hpp>
       2 | <boost/program_options/parsers.hpp>
       2 | <boost/program_options/value_semantic.hpp>
       2 | <boost/range/algorithm/for_each.hpp>
       2 | <boost/shared_ptr.hpp>
       2 | <boost/spirit/include/qi_auxiliary.hpp>
       2 | <boost/spirit/include/qi_nonterminal.hpp>
       2 | <boost/spirit/include/qi.hpp>
       2 | <boost/variant.hpp>
       1 | <boost/accumulators/statistics/rolling_mean.hpp>
       1 | <boost/any.hpp>
       1 | <boost/array.hpp>
       1 | <boost/asio/detail/winsock_init.hpp>
       1 | <boost/asio/ip/address_v4.hpp>
       1 | <boost/asio/ip/address_v6.hpp>
       1 | <boost/asio/posix/stream_descriptor.hpp>
       1 | <boost/asio/windows/stream_handle.hpp>
       1 | <boost/bimap.hpp>
       1 | <boost/bind/arg.hpp>
       1 | <boost/config/warning_disable.hpp>
       1 | <boost/container/small_vector.hpp>
       1 | <boost/context/detail/fcontext.hpp>
       1 | <boost/filesystem.hpp>
       1 | <boost/functional/hash.hpp>
       1 | <boost/fusion/include/at.hpp>
       1 | <boost/fusion/include/define_struct.hpp>
       1 | <boost/fusion/include/io.hpp>
       1 | <boost/fusion/include/pair.hpp>
       1 | <boost/fusion/include/std_pair.hpp>
       1 | <boost/intrusive_ptr.hpp>
       1 | <boost/iostreams/stream.hpp>
       1 | <boost/iterator/iterator_categories.hpp>
       1 | <boost/lockfree/detail/prefix.hpp>
       1 | <boost/lockfree/detail/tagged_ptr.hpp>
       1 | <boost/lockfree/policies.hpp>
       1 | <boost/multi_array.hpp>
       1 | <boost/optional.hpp>
       1 | <boost/parameter/keyword.hpp>
       1 | <boost/phoenix/bind.hpp>
       1 | <boost/phoenix/core.hpp>
       1 | <boost/program_options/cmdline.hpp>
       1 | <boost/program_options/config.hpp>
       1 | <boost/program_options/detail/config_file.hpp>
       1 | <boost/program_options/detail/convert.hpp>
       1 | <boost/program_options/detail/parsers.hpp>
       1 | <boost/program_options/detail/utf8_codecvt_facet.hpp>
       1 | <boost/program_options/detail/value_semantic.hpp>
       1 | <boost/program_options/environment_iterator.hpp>
       1 | <boost/program_options/eof_iterator.hpp>
       1 | <boost/program_options/errors.hpp>
       1 | <boost/program_options/option.hpp>
       1 | <boost/program_options/positional_options.hpp>
       1 | <boost/program_options/variables_map.hpp>
       1 | <boost/program_options/version.hpp>
       1 | <boost/range/adaptor/filtered.hpp>
       1 | <boost/range/algorithm/copy.hpp>
       1 | <boost/range/begin.hpp>
       1 | <boost/range/counting_range.hpp>
       1 | <boost/range/end.hpp>
       1 | <boost/range/numeric.hpp>
       1 | <boost/spirit/home/qi/auxiliary/lazy.hpp>
       1 | <boost/spirit/home/qi/detail/enable_lit.hpp>
       1 | <boost/spirit/home/qi/detail/string_parse.hpp>
       1 | <boost/spirit/home/qi/domain.hpp>
       1 | <boost/spirit/home/qi/meta_compiler.hpp>
       1 | <boost/spirit/home/qi/parser.hpp>
       1 | <boost/spirit/home/qi/skip_over.hpp>
       1 | <boost/spirit/home/support/char_class.hpp>
       1 | <boost/spirit/home/support/common_terminals.hpp>
       1 | <boost/spirit/home/support/detail/get_encoding.hpp>
       1 | <boost/spirit/home/support/handles_container.hpp>
       1 | <boost/spirit/home/support/info.hpp>
       1 | <boost/spirit/home/support/modify.hpp>
       1 | <boost/spirit/home/support/string_traits.hpp>
       1 | <boost/spirit/home/support/unused.hpp>
       1 | <boost/spirit/include/phoenix_core.hpp>
       1 | <boost/spirit/include/phoenix_object.hpp>
       1 | <boost/spirit/include/phoenix_operator.hpp>
       1 | <boost/spirit/include/qi_alternative.hpp>
       1 | <boost/spirit/include/qi_directive.hpp>
       1 | <boost/spirit/include/qi_sequence.hpp>
       1 | <boost/spirit/include/qi_uint.hpp>
-----------------------------------------------------
     280 | TOTAL LINES

Aug 20 '20 20:08 hkaiser

Here is an updated list:

#include <boost/accumulators/accumulators.hpp>
#include <boost/accumulators/framework/accumulator_base.hpp>
#include <boost/accumulators/framework/depends_on.hpp>
#include <boost/accumulators/framework/extractor.hpp>
#include <boost/accumulators/framework/parameters/sample.hpp>
#include <boost/accumulators/numeric/functional.hpp>
#include <boost/accumulators/statistics/count.hpp>
#include <boost/accumulators/statistics/max.hpp>
#include <boost/accumulators/statistics/mean.hpp>
#include <boost/accumulators/statistics/median.hpp>
#include <boost/accumulators/statistics/min.hpp>
#include <boost/accumulators/statistics/rolling_mean.hpp>
#include <boost/accumulators/statistics/rolling_variance.hpp>
#include <boost/accumulators/statistics/rolling_window.hpp>
#include <boost/accumulators/statistics/stats.hpp>
#include <boost/accumulators/statistics/variance.hpp>
#include <boost/accumulators/statistics_fwd.hpp>
#include <boost/archive/iterators/base64_from_binary.hpp>
#include <boost/archive/iterators/binary_from_base64.hpp>
#include <boost/archive/iterators/transform_width.hpp>
#include <boost/array.hpp>
#include <boost/bind/arg.hpp>
#include <boost/config.hpp>
#include <boost/context/detail/fcontext.hpp>
#include <boost/crc.hpp>
#include <boost/exception/diagnostic_information.hpp>
#include <boost/exception/exception.hpp>
#include <boost/filesystem.hpp>
#include <boost/functional/hash.hpp>
#include <boost/fusion/include/adapt_struct.hpp>
#include <boost/fusion/include/define_struct.hpp>
#include <boost/fusion/include/io.hpp>
#include <boost/fusion/include/std_pair.hpp>
#include <boost/fusion/include/vector.hpp>
#include <boost/integer/common_factor.hpp>
#include <boost/intrusive_ptr.hpp>
#include <boost/iostreams/device/file_descriptor.hpp>
#include <boost/iostreams/filter/bzip2.hpp>
#include <boost/iostreams/filter/zlib.hpp>
#include <boost/iostreams/stream.hpp>
#include <boost/iterator/iterator_categories.hpp>
#include <boost/log/expressions.hpp>
#include <boost/log/expressions/formatter.hpp>
#include <boost/log/expressions/formatters.hpp>
#include <boost/log/expressions/formatters/stream.hpp>
#include <boost/log/sources/record_ostream.hpp>
#include <boost/log/sources/severity_logger.hpp>
#include <boost/log/trivial.hpp>
#include <boost/log/utility/formatting_ostream.hpp>
#include <boost/log/utility/manipulators/to_log.hpp>
#include <boost/log/utility/setup/common_attributes.hpp>
#include <boost/log/utility/setup/console.hpp>
#include <boost/multi_array.hpp>
#include <boost/next_prior.hpp>
#include <boost/optional.hpp>
#include <boost/parameter/keyword.hpp>
#include <boost/phoenix.hpp>
#include <boost/preprocessor.hpp>
#include <boost/regex.hpp>
#include <boost/shared_array.hpp>
#include <boost/shared_ptr.hpp>
#include <boost/spirit/home/support/iterators/istream_iterator.hpp>
#include <boost/spirit/home/x3/auxiliary.hpp>
#include <boost/spirit/home/x3/auxiliary.hpp>
#include <boost/spirit/home/x3/char.hpp>
#include <boost/spirit/home/x3/core.hpp>
#include <boost/spirit/home/x3/core/parser.hpp>
#include <boost/spirit/home/x3/core/skip_over.hpp>
#include <boost/spirit/home/x3/directive.hpp>
#include <boost/spirit/home/x3/nonterminal.hpp>
#include <boost/spirit/home/x3/numeric.hpp>
#include <boost/spirit/home/x3/operator.hpp>
#include <boost/spirit/home/x3/string.hpp>
#include <boost/spirit/home/x3/support/traits/move_to.hpp>
#include <boost/spirit/home/x3/support/unused.hpp>
#include <boost/spirit/include/phoenix_core.hpp>
#include <boost/spirit/include/phoenix_object.hpp>
#include <boost/spirit/include/phoenix_operator.hpp>
#include <boost/spirit/include/qi.hpp>
#include <boost/spirit/include/qi_action.hpp>
#include <boost/spirit/include/qi_auxiliary.hpp>
#include <boost/spirit/include/qi_char.hpp>
#include <boost/spirit/include/qi_numeric.hpp>
#include <boost/spirit/include/qi_operator.hpp>
#include <boost/spirit/include/qi_parse.hpp>
#include <boost/spirit/include/qi_string.hpp>
#include <boost/spirit/include/qi_uint.hpp>
#include <boost/spirit/include/support_istream_iterator.hpp>
#include <boost/system/error_code.hpp>
#include <boost/version.hpp>

Dec 04 '22 15:12 hkaiser

hpx hpx copied to clipboard

Explore feasibility of removing all dependencies on Boost

Accumulators

(string) algorithms

Serialization

ASIO thread pools (#5158)

Context switching

Exception

Filesystem

Spirit/fusion/phoenix

Iostreams

Iterator

Queues

Log

Pool

Program options

Range

SIMD

SmartPtr

System

Miscellaneous

hpx
hpx copied to clipboard