amoro

amoro copied to clipboard

Welcome to arctic, arctic is a streaming lake warehouse system open sourced by NetEase. Arctic adds more real-time capabilities on top of iceberg and hive, and provides stream-batch unified, out-of-the-box metadata services for dataops, allowing Data lakes much more usable and practical.

What is arctic

Arctic is a streaming lakehouse service built on top of apache iceberg table format. Through arctic, users could benefit optimized CDC、streaming update、fresh olap etc. on engines like flink, spark, and trino. Combined with efficient offline processing capabilities of data lakes, arctic can serve more scenarios where streaming and batch are fused. At the same time, the function of self-optimization、concurrent conflict resolution and standard management tools could effectively reduce the burden on users in data lake management and optimization.

Arctic services are presented by deploying AMS, which can be considered as a replacement for HMS (Hive Metastore), or HMS for iceberg. Arctic uses iceberg as the base table format, but instead of hacking the iceberg implementation, it uses iceberg as a lib. Arctic's open overlay architecture can help large-scale offline data lakes quickly upgraded to real-time data lakes, without worrying about compatibility issues with the original data lakes, allowing data lakes to meet more real-time analysis, real-time risk control, Real-time training, feature engineering and other scenarios.

Arctic features

• Efficient streaming updates based on primary keys
• Data auto bucketing and self-optimized for performance and efficiency
• Encapsulating data lake and message queue into a unified table to achieve lower-latency computing
• Provide standardized metrics, dashboard and related management tools for streaming lakehouse
• Support spark and flink to read and write data, support trino to query data
• 100% compatible with iceberg / hive table format and syntax
• Provide transactional guarantee for streaming and batch concurrent writing

Modules

Arctic contains modules as below:

• arctic-core contains core abstractions and common implementions for other modules
• arctic-flink is the module for integrating with Apache Flink (use arctic-flink-runtime for a shaded version)
• arctic-spark is the module for integrating with Apache Spark (use arctic-spark-runtime for a shaded version)
• arctic-trino now provides query integrating with apache trino, built on JDK11
• arctic-optimizing exposes optimizing container/group api and provides default implemetion
• arctic-ams is arctic meta service module
  • ams-api contains ams thrift api
  • ams-dashboard is the dashboard frontend for ams
  • ams-server is the backend server for ams

Building

Arctic is built using Maven with Java 1.8 and Java 11(only for trino module).

• To build Trino module need config toolchains.xml in ${user.home}/.m2/ dir, the content is

<?xml version="1.0" encoding="UTF-8"?>
<toolchains>
    <toolchain>
        <type>jdk</type>
        <provides>
            <version>11</version>
            <vendor>sun</vendor>
        </provides>
        <configuration>
            <jdkHome>${yourJdk11Home}</jdkHome>
        </configuration>
    </toolchain>
</toolchains>

• To invoke a build and run tests: mvn package -P toolchain
• To skip tests: mvn -DskipTests package -P toolchain

Engines supported

Arctic support multiple processing engines as below:

Quickstart

Visit https://arctic.netease.com/ch/docker-quickstart/ to quickly explore what arctic can do.

Join Community

If you are interested in Lakehouse, Data Lake Format, welcome to join our community, we welcome any organizations, teams and individuals to grow together, and sincerely hope to help users better use Data Lake Format through open source.

Join the Arctic WeChat Group: Add " kllnn999 " as a friend on WeChat and specify "Arctic lover".