Arc

Programming language for data stream analysis.

Requirements

OCaml (and dune), Rust (and cargo), and C++ (and CMake and Ninja).

Examples

A basic streaming word-count application can be written in functional-style as follows:

val wordcounts = lines
  .flatmap(_.split(" "))
  .keyby(_)
  .window(
    length = 10min,
    stride = 3min
  )
  .count()

The same code can also be written using a more declarative, relational-style, syntax. This concept is borrowed from Morel and applied to streaming data.

val wordcounts =
  from
    line in lines,
    word in line.split(" ")
  keyby word
  window
    length = 10min
    stride = 3min
  reduce count
    identity 1;

Feature highlights

• Statically typed with global type inference.
• Parametric polymorphism (generics and rows) and ad-hoc polymorphism (type classes).
• Mix of functional syntax, imperative control-flow/mutation, and relational operators.
• Algebraic data types.
• First-class data streams.
• Complex event processing using tasks.
• Window-based computation.
• Low-level compilation and distributed execution.
• Command-line interface for data ingestion.

Note: All features have not yet been implemented :)

Installation

git clone [email protected]:cda-group/arc.git
cd arc/
git submodule update --init --recursive
./build

Documentation

• Getting started
• Language Reference
• Developers

Project Structure

• arc-lang - A compiler for Arc-Lang.
• arc-mlir - An optimizer for Arc-Lang.
• arc-runtime - A local runtime which supports the execution of Arc-Lang programs.
• arc-python - A Python library for writing Arc-Lang applications.

Related Projects

• arcon - A distributed runtime which will support execution of Arc-Lang.
• kompact - A component-actor middleware which Arc-Runtime and Arcon are both implemented in.

Other

Arc-Lang ain't done until the fat lady sings. - Peter Van-Roy