AR based on mobile 101

[ChenYilong]

关于移动端 AR 的一些思考与调研

今年 Apple 发布了其历史上两款史上最强（gui）的移动类型设备，其中附加值最高的，不是屏幕、尺寸，应属 A12 和 A12X Bionic 芯片。全程讲硬件，于是发布会并不是像国内的相声专场式发布会那样好看。但这两种发布会重心不同，一个是志在解决未来的问题，一个努力在解决当下的问题。有人说曾几何时苹果发布会是不屑于炫耀自己的芯片是多么强大，可是现如今苹果发布会上不停的说这枚芯片有多么厉害，多么智能。这是 Apple 软件创新乏力，于是用硬件凑。而从开发者角度看，从 2017、2018 的 WWDC 中，机器学习和 AR 的 session 每年都超过10个。说明硬件升级，是 Apple 在为软件层面布局，下一个风口，应该是 Core ML 和 AR。而机器学习在移动端落地，除了图像处理、拍照应用、语音处理，最大的入口就是 AR，其中移动端 AR 落地最好就是如抖音、line等视频直播应用。

巨头新宠

Android 阵营，Google 准备了 TensorFlow Lite，但 Google 不做硬件，这个推广的短板。即使是 Pixel，单就图像识别一项上，iPhone 上经过优化，速度能比它快6倍以上。同时 AR 也成为 BAT 的新战场，吴恩达负责的百度研究院也成立第四大实验室——增强现实实验室(AR Lab)，应用在了百度 APP 上，应用场景有 AR 在手机上复原古建筑，实用性有待提升，运用的是完全自研的 AR 技术。阿里巴巴有支付宝 AR 实景红包，淘宝 VR 购物，还投资了 AR 初创公司 Magic Leap;　腾讯有 QQ AR 实景红包、 AR 火炬，据说还要做一个 QAR 的 AR 开放平台。但都没有硬件优势，Apple 是唯一“软硬兼施”，假设 Apple 的目标是击退 Android，引流到自己平台上，那么可以说 Apple 拿得一手好牌，似乎并没打好。

而 Apple ，在发布会上演示的多人对抗 AR 游戏，以及自家应用比如 iMessage 里的换脸功能， AR 技术上的积累就可见一斑，但却没有带来足够的流量，反倒是在抖音被“带”火起来的，不过今年库克还专门去了国内的在 AR 领域已经落地、视频流量之王的抖音、头条访问，可见对 AR 技术落地的关注。明年发布会上 iMessage 换脸一键分享到抖音、或者抖音唤起系统的换脸摄像头、甚至介绍 ARKit 在抖音上的应用，都不是不可能。这个反应出 ARKit 推广还是非常难。

我在近期做的一个项目也使用了 AR 技术，但是由于无法摆脱兼容版本的限制、无法跨平台，于是第一个排除的竟是最强大的 ARKit。于是我想在这里做下调研，想看下国内做的 AR 技术现状。稍后也会基于此写几篇关于 AR 的博文，供大家在做技术选型时做参考。

调研两条内容：

1. 你正在应用是否涉及 AR 功能，如果涉及，应用场景属于哪种，或者说被训练物体的种类属于哪一类？投票地址微博下方： 【你的APP中AR应用场景属于哪类？】http://t.cn/Ew4TM1b
2. 你的 APP 中的AR是用什么SDK实现的？微博投票选项有限制，故放在金数据上： https://jinshuju.net/f/uYXFOA

应用场景属于哪一种？

说明：分类方案是引用自《iPhone Game Blueprints》一书：

• Symbol-based: 也即基准标记物(fiducial markers)，是最初级的一类训练物体方式，这类的物体识别基于类似于非常明显的标记，比如边框特征，二维、三维物体的边角。QR 二维码就是非常典型的Symbol-based，所有二维码都有三个角是边框状，以此作为是否是二维码的标准来进行物体追踪。但本次调研，二维码不属于AR范畴。
• Image-based: 基于图片，比如支付宝 AR 红包。
• Object-based: 基于实物，比如抖音特效里的川剧变脸、iMessage里的换脸。
• Map-based: 基于地图；比如采用 SLAM 技术(Simultaneous Localization And Mapping，即时定位与地图构建技术)，SLAM 已大量运用在无人机与无人车上用于建立周围环境的三维地图，在 APP 上的应用比如百度APP，镜头对准实物建筑物，在手机上复原古建筑。
• Location-based: 基于地理位置，比如 2016 年大火的Pokémon Go，实际仅仅是运用地理位置信息信，并未采用其他信息作为marker。
• other：欢迎留言补充。

原文如下：

• Symbol-based: This is a basic method. It is the recognition and tracking of special graphic markers that are officially known as fiducial markers or fiducials (from the Latin fiducia, which means trust), but are sometimes referred to as AR tags, tracking marks, artificial landmarks, and so on. In some cases, slightly modified QR code markers can be used as well.
• Image-based: Graphic images such as photos or illustrations are split into reference points (the simplest way is to find a contrast frame around a picture). For instance, some applications use the one-dollar bill as the real-world basis that AR content will be created on, arguing that bills are the most common pieces of paper with graphics on them that an ordinary person can get their hands on. Such practice is used by programs such as Shimmer Augmented Reality Viewer for iOS (http://ionreality.com/shimmer/), which allows you to view 3D models in the AR space. In some cases, when very accurate recognition tasks are the goal, the image-based method may require very powerful CPU resources, because graphic images are split into thousands of markers that are being tracked in real time. This is why there is a service called Xloudia (http://www.lm3labs.com/xloudia/) that makes such calculations in the cloud computing to recognize complicated images on the server side.
• Object-based: Special real objects are used as markers. They can be passive (only their geometry and contrast color will be used as markers) or active (they will emit signals as light ones). Usually, they come in the shape of small cubes or spheres. There are also some applications that track a user's face and use their movements as input data.
• Map-based: A 3D map of the surroundings is created and the elements of a landscape are used as references. For instance, AR applications can use simultaneous localization and mapping (SLAM) tracking, a technology that is utilized by autonomous, movable robots. This is one of the most advanced methods of orientation and navigation in space. It allows us to use real surroundings, and virtual elements can interact with real-world objects in a more natural way.
• Location-based: This algorithm does not require any graphic markers. It uses the advantages of GPS (or its competitors), the compass, and accelerometer to determine the position and tilt of a device. Usually, such technology is used to add some virtual features to AR geo-information about real physical objects, such as city landmarks, in order to give the virtual representation of the information some textual and graphic description, interactive functionality, and so on. There are also some interesting massively multiplayer online game projects based on accurate location detection.

你的 APP 中的AR是用什么SDK实现的？

列表中的SDK，我并没有全部用过，信息来自于格哈德Reitmayr创建和共享在1inkedin上。