getting-started icon indicating copy to clipboard operation
getting-started copied to clipboard

Published 20 hours ago • verified publisher icon timvideos

[streaming-system] Automated build and configuration of full streaming system

Open mithro opened this issue 11 years ago • 4 comments

Brief explanation

System to build and configure all of the machines needed to record and stream a multi track conference.

Full automated deployment of both the collecting infrastructure (parts inside a conference venue / user group venue) and the off site infrastructure (parts in the cloud).

Expected results

Convert existing scripts and manual processes to a consistent and modular deployment system templates. Demonstrate it working in VMs.

Detailed Explanation

Problem

A 25 track conference needs about 60 physical and 30 virtual machines. Each needs an os installed, host name assigned, networking that plays well with the conference infrastructure, various applications installed and configured.

Setup

For production there will be a physical box with Debian stable, access to a standard Debian repository, sudo user, a master config file, file of secrets and a script. The script will install and configure the needed services to provide automated pxe install for the remaining machines and provision and configure the vm's.

For testing, a single physical box running lots of vms that replicates the production setup. This adds a layer of complexity and such, but is the only sensible way to test and maintain and enhance such a system.

Configuration

Everything possible will be contained in the master config file. There should be no editing additional config files by hand. Part of this project will be to determine what all needs to be included in the master config file. The file will likely be created by the veyepar system which will be responsible for transforming the conference schedule into a consistent format and providing a UI to manage such things as: Talks in the room that the attendees know as “UB2.252A (Lameere)” will have a url of www.timvideos.com/fosdem2014/lameere ad an irc channel of #fosdem-lameere; and the video mixer machine has a mac address of 00:11:22..., content source machines: [22:33:44... 44:55:66...], host names: lameere-mixer, lameere-source-1 and -2, and somehow specifying if the encoders will be EC2, RackSpace or local vm's.

Pieces

  • collecting infrastructure
    • dvswitch or gst-switch
    • flumotion collector (watchdog, register, etc)
  • streaming infrastructure (EC2)

Further reading

Existing build systems that do quite a bit of what is needed: https://github.com/yoe/fosdemvideo/blob/master/doc/README https://github.com/CarlFK/veyepar/blob/master/setup/nodes/pxe/README.txt https://github.com/timvideos/streaming-system/blob/master/tools/setup/runall.sh

Knowledge Prerequisites

  • Good understanding of deploying machines and systems such as PXE boot.

Contacts

mithro avatar Feb 17 '14 06:02 mithro

@maxstr Does your work with #21 make some of this simplier?

mithro avatar Aug 01 '14 23:08 mithro

@mithro i would like to do this as my gsoc'16 project

rahulroxx avatar Mar 01 '16 18:03 rahulroxx

Please read our Summer of Code page at https://code.timvideos.us/summer-of-code/, especially the FAQ section.

mithro avatar Mar 07 '16 12:03 mithro

This is mostly complete: https://salsa.debian.org/debconf-video-team/ansible

It lacks a few of the described features, but I think those should be logged as bugs against the respective projects.

It doesn't have the build a farm of VMs tests, but I think that's ok. you can use qemu to test nodes, and some of the key pieces need real hardware to really test (hdmi2usb, usb audio.)

GitLab
Ansible playbooks for the DebConf Video Team's machines - https://debconf-video-team.pages.debian.net/ansible/

CarlFK avatar Mar 01 '18 23:03 CarlFK