Timeout when linking CMAF muxer to sink while resuming pipeline

[zcesur]

Seeing intermittent timeouts during pad linking between CMAF muxer and sink. This eventually leads to a pattern match crash in the CMAF muxer. Hard to reproduce consistently but affects both audio/video tracks. Seems to be caused by the GenServer call in ‎Membrane.Core.Element.PadController.do_handle_link‎ which times out after 5 seconds

Could there be a deadlock or race condition during playback state transitions that's causing the sink to not respond to link requests? :thinking:

[debug] <0.788.0>/:sink/{:cmaf_muxer, "audio_master"} Tried to link pad {Membrane.Pad, :output, #Reference<0.1819401566.4031774723.74705>}, but neighbour :sink
is not alive.

[debug] <0.788.0>/:sink/ Failed to establish link between {:cmaf_muxer, "audio_master"} via {Membrane.Pad, :output, #Reference<0.1819401566.4031774723.74705>} and
:sink via {Membrane.Pad, :input, "audio_master"} because {:cmaf_muxer, "audio_master"} is down.

[debug] <0.788.0>/:sink/ Spec #Reference<0.1819401566.4031774722.86411> status changed to linked internally
[debug] <0.788.0>/:sink/ Sending link response, link_id: #Reference<0.1819401566.4031774724.83906>, pad: {Membrane.Pad, :input, "audio_master"}
[debug] <0.788.0>/:sink/{:cmaf_muxer, "audio_master"} Ignoring unlinking pad {Membrane.Pad, :output, #Reference<0.1819401566.4031774723.74705>} that hasn't been successfully linked
[debug] <0.788.0>/:sink/ Spec #Reference<0.1819401566.4031774722.86411> status changed to linking externally
[debug] <0.788.0>/:sink/ Proceeding spec #Reference<0.1819401566.4031774723.74707> startup: initializing, dependent specs: []
[debug] <0.788.0>/:sink/ Proceeding spec #Reference<0.1819401566.4031774723.74707> startup: initializing, dependent specs: []
[debug] <0.788.0>/:sink/ Spec #Reference<0.1819401566.4031774723.74707> status changed to initialized
[debug] <0.788.0>/:sink/ Got internal link request, pad ref {Membrane.Pad, :input, "video_master"}, child {:parser, "video_master"}, spec #Reference<0.1819401566.4031774723.74707>
[debug] <0.788.0>/:sink/ Spec #Reference<0.1819401566.4031774723.74707> status changed to linking internally
[debug] <0.788.0>/:sink/{:parser, "video_master"} Element handle link on pad :output with pad {Membrane.Pad, :input, #Reference<0.1819401566.4031774723.74731>} of child {:cmaf_muxer, "video_master"}
[debug] <0.788.0>/:sink/{:cmaf_muxer, "video_master"} Element handle link on pad {Membrane.Pad, :input, #Reference<0.1819401566.4031774723.74731>} with pad :output of child {:parser, "video_master"}
[debug] <0.788.0>/:sink/{:cmaf_muxer, "video_master"} Element handle link on pad {Membrane.Pad, :output, #Reference<0.1819401566.4031774723.74732>} with pad {Membrane.Pad, :input, "video_master"} of child :sink
[debug] <0.788.0>/:sink/ Failed to establish link between {:cmaf_muxer, "video_master"} via {Membrane.Pad, :output, #Reference<0.1819401566.4031774723.74732>} and
:sink via {Membrane.Pad, :input, "video_master"} because {:cmaf_muxer, "video_master"} is down.

[debug] <0.788.0>/:sink/ Spec #Reference<0.1819401566.4031774723.74707> status changed to linked internally
[debug] <0.788.0>/:sink/ Sending link response, link_id: #Reference<0.1819401566.4031774724.83907>, pad: {Membrane.Pad, :input, "video_master"}
[debug] <0.788.0>/:sink/ Spec #Reference<0.1819401566.4031774723.74707> status changed to linking externally
[debug] <0.788.0>/:tee_audio Element handle link on pad {Membrane.Pad, :output, #Reference<0.1819401566.4031774724.83908>} with pad {Membrane.Pad, :input, "audio_master"} of child :sink
[debug] <0.788.0>/:sink/{:cmaf_muxer, "video_master"} Tried to link pad {Membrane.Pad, :output, #Reference<0.1819401566.4031774723.74732>}, but neighbour :sink
is not alive.

[debug] <0.788.0>/:sink/{:cmaf_muxer, "video_master"} Ignoring unlinking pad {Membrane.Pad, :output, #Reference<0.1819401566.4031774723.74732>} that hasn't been successfully linked
[debug] <0.788.0>/:sink/ Handle link %Membrane.Core.Parent.Link.Endpoint{
  child: :sink,
  pad_spec: {Membrane.Pad, :input, "audio_master"},
  pad_ref: {Membrane.Pad, :input, "audio_master"},
  pid: #PID<0.824.0>,
  pad_props: %{
    options: [
      track_name: "audio_master",
      encoding: :AAC,
      segment_duration: 6000000000,
      partial_segment_duration: 1000000000
    ],
    toilet_capacity: nil,
    target_queue_size: nil,
    min_demand_factor: nil,
    auto_demand_size: nil,
    throttling_factor: nil
  },
  pad_info: %{
    name: :input,
    options: [
      encoding: [
        spec: ":AAC | :H264 | :H265",
        description: "Encoding type determining which parser will be used for the given stream.\n"
      ],
      track_name: [
        spec: "String.t() | nil",
        default: nil,
        description: "Name that will be used to name the media playlist for the given track, as well as its header and segments files.\nIt must not contain any URI reserved characters\n"
      ],
      segment_duration: [
        spec: "Membrane.Time.t()",
        description: "The minimal segment duration of the regular segments.\n"
      ],
      partial_segment_duration: [
        spec: "Membrane.Time.t() | nil",
        default: nil,
        description: "The segment duration of the partial segments.\nIf not set then the bin won't produce any partial segments.\n"
      ],
      max_framerate: [
        spec: "float() | nil",
        default: nil,
        description: "The maximal framerate of video variant. This information is used in master playlist.\n\nWhen set to nil then this information won't be added to master playlist. For audio it should be set to nil.\n"
      ]
    ],
    direction: :input,
    accepted_formats_str: ["Membrane.AAC", "Membrane.H264", "Membrane.H265"],
    availability: :on_request
  },
  child_spec_ref: #Reference<0.1819401566.4031774725.64144>
}
[debug] <0.788.0>/:sink/{:parser, "audio_master"} Element handle link on pad :input with pad {Membrane.Pad, :output, #Reference<0.1819401566.4031774724.83908>} of child :tee_audio
[debug] <0.788.0>/:sink/ Spec #Reference<0.1819401566.4031774722.86411> status changed to ready
[debug] <0.788.0>/:sink/{:parser, "audio_master"} Got play request
[debug] <0.788.0>/:sink/ Cleaning spec %{status: :linking_externally, children_names: MapSet.new([cmaf_muxer: "audio_master", parser: "audio_master"]), awaiting_responses: MapSet.new([]), links_ids: [#Reference<0.1819401566.4031774722.86412>, #Reference<0.1819401566.4031774722.86413>, #Reference<0.1819401566.4031774722.86414>], dependent_specs: MapSet.new([])}
[debug] <0.788.0>/:sink/{:cmaf_muxer, "audio_master"} Got play request
[debug] <0.788.0>/:tee_audio Sending stream format through pad {Membrane.Pad, :output, #Reference<0.1819401566.4031774724.83908>}
Stream format: %Membrane.AAC{sample_rate: 48000, channels: 2, profile: :LC, mpeg_version: 4, samples_per_frame: 1024, frames_per_buffer: 1, encapsulation: :none, config: nil}

[debug] <0.788.0>/:tee_video Element handle link on pad {Membrane.Pad, :output, #Reference<0.1819401566.4031774724.83909>} with pad {Membrane.Pad, :input, "video_master"} of child :sink
[debug] <0.788.0>/:sink/{:parser, "audio_master"} Sending stream format through pad :output
Stream format: %Membrane.AAC{sample_rate: 48000, channels: 2, profile: :LC, mpeg_version: 4, samples_per_frame: 1024, frames_per_buffer: 1, encapsulation: :none, config: {:esds, <<3, 128, 128, 128, 34, 0, 1, 0, 4, 128, 128, 128, 20, 64, 21, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 5, 128, 128, 128, 2, 17, 144, 6, 128, 128, 128, 1, 2>>}}

[debug] <0.788.0>/:sink/ Handle link %Membrane.Core.Parent.Link.Endpoint{
  child: :sink,
  pad_spec: {Membrane.Pad, :input, "video_master"},
  pad_ref: {Membrane.Pad, :input, "video_master"},
  pid: #PID<0.824.0>,
  pad_props: %{
    options: [
      track_name: "video_master",
      encoding: :H264,
      segment_duration: 6000000000,
      partial_segment_duration: 1000000000
    ],
    toilet_capacity: nil,
    target_queue_size: nil,
    min_demand_factor: nil,
    auto_demand_size: nil,
    throttling_factor: nil
  },
  pad_info: %{
    name: :input,
    options: [
      encoding: [
        spec: ":AAC | :H264 | :H265",
        description: "Encoding type determining which parser will be used for the given stream.\n"
      ],
      track_name: [
        spec: "String.t() | nil",
        default: nil,
        description: "Name that will be used to name the media playlist for the given track, as well as its header and segments files.\nIt must not contain any URI reserved characters\n"
      ],
      segment_duration: [
        spec: "Membrane.Time.t()",
        description: "The minimal segment duration of the regular segments.\n"
      ],
      partial_segment_duration: [
        spec: "Membrane.Time.t() | nil",
        default: nil,
        description: "The segment duration of the partial segments.\nIf not set then the bin won't produce any partial segments.\n"
      ],
      max_framerate: [
        spec: "float() | nil",
        default: nil,
        description: "The maximal framerate of video variant. This information is used in master playlist.\n\nWhen set to nil then this information won't be added to master playlist. For audio it should be set to nil.\n"
      ]
    ],
    direction: :input,
    accepted_formats_str: ["Membrane.AAC", "Membrane.H264", "Membrane.H265"],
    availability: :on_request
  },
  child_spec_ref: #Reference<0.1819401566.4031774725.64144>
}
[debug] <0.788.0>/:sink/{:parser, "video_master"} Element handle link on pad :input with pad {Membrane.Pad, :output, #Reference<0.1819401566.4031774724.83909>} of child :tee_video
[debug] <0.788.0>/:sink/ Spec #Reference<0.1819401566.4031774723.74707> status changed to ready
[debug] <0.788.0>/:sink/{:parser, "video_master"} Got play request
[debug] <0.788.0>/:sink/{:cmaf_muxer, "video_master"} Got play request
[debug] <0.788.0>/:sink/{:parser, "video_master"} Transiting `flow_control: :auto` pads to :pull effective flow control
[debug] <0.788.0>/:sink/ Cleaning spec %{status: :linking_externally, children_names: MapSet.new([cmaf_muxer: "video_master", parser: "video_master"]), awaiting_responses: MapSet.new([]), links_ids: [#Reference<0.1819401566.4031774721.97285>, #Reference<0.1819401566.4031774721.97286>, #Reference<0.1819401566.4031774721.97287>], dependent_specs: MapSet.new([])}
[debug] <0.788.0>/:tee_video Sending stream format through pad {Membrane.Pad, :output, #Reference<0.1819401566.4031774724.83909>}
Stream format: %Membrane.H264{width: 1920, height: 1080, profile: :high, alignment: :au, nalu_in_metadata?: true, framerate: nil, stream_structure: {:avc3, <<1, 100, 0, 40, 255, 224, 0>>}}

[debug] <0.788.0>/ Spec #Reference<0.1819401566.4031774724.83905> status changed to linked internally
[debug] <0.788.0>/ Spec #Reference<0.1819401566.4031774724.83905> status changed to ready
[debug] <0.788.0>/ Cleaning spec %{status: :linking_internally, children_names: MapSet.new([]), awaiting_responses: MapSet.new([]), links_ids: [#Reference<0.1819401566.4031774724.83906>, #Reference<0.1819401566.4031774724.83907>], dependent_specs: MapSet.new([])}
[error] GenServer #PID<0.2353.0> terminating
** (MatchError) no match of right hand side value: %{input: [{Membrane.Pad, :input, #Reference<0.1819401566.4031774723.74731>}]}
    (membrane_mp4_plugin 0.35.2) lib/membrane_mp4/muxer/cmaf.ex:227: Membrane.MP4.Muxer.CMAF.handle_playing/2
    (membrane_core 1.0.1) lib/membrane/core/callback_handler.ex:139: Membrane.Core.CallbackHandler.exec_callback/4
    (membrane_core 1.0.1) lib/membrane/core/callback_handler.ex:69: Membrane.Core.CallbackHandler.exec_and_handle_callback/5
    (membrane_core 1.0.1) lib/membrane/core/element/lifecycle_controller.ex:84: Membrane.Core.Element.LifecycleController.handle_playing/1
    (membrane_core 1.0.1) lib/membrane/core/element.ex:249: Membrane.Core.Element.handle_info/2
    (stdlib 6.1) gen_server.erl:2345: :gen_server.try_handle_info/3
    (stdlib 6.1) gen_server.erl:2433: :gen_server.handle_msg/6
    (stdlib 6.1) proc_lib.erl:329: :proc_lib.init_p_do_apply/3
Last message: {Membrane.Core.Message, :play, [], []}
State: %Membrane.Core.Element.State{module: Membrane.MP4.Muxer.CMAF, name: {:cmaf_muxer, "video_master"}, parent_pid: #PID<0.824.0>, playback: :stopped, type: :filter, internal_state: %{segment_min_duration: 6000000000, video_pad: nil, pad_to_track_data: %{}, all_input_pads_ready?: false, buffers_awaiting_init: [], sample_queues: %{}, finish_current_segment?: false, awaiting_stream_formats: %{}, pads_registration_order: [{Membrane.Pad, :input, #Reference<0.1819401566.4031774723.74731>}], chunk_duration_range: %Membrane.MP4.Muxer.CMAF.DurationRange{min: 50000000, target: 1000000000}}, pad_refs: [{Membrane.Pad, :input, #Reference<0.1819401566.4031774723.74731>}], pads_info: %{input: %{name: :input, options: [], direction: :input, accepted_formats_str: ["%AAC{config: {:esds, _esds}}", "%Opus{self_delimiting?: false}", "%H264{stream_structure: structure, alignment: :au} when H264.is_avc(structure)", "%H265{stream_structure: structure, alignment: :au} when H265.is_hvc(structure)"], availability: :on_request, flow_control: :manual, demand_unit: :buffers}, output: %{name: :output, options: [tracks: [spec: "[Membrane.Pad.dynamic_id()] | :all", default: :all, description: "A list of the input pad ids that should be muxed together into a single output track.\n\nIf not specified the pad will include all unreferenced input pads.\n"]], direction: :output, accepted_formats_str: ["Membrane.CMAF.Track"], availability: :on_request, flow_control: :manual, demand_unit: nil}}, synchronization: %{timers: %{}, clock: nil, stream_sync: :membrane_no_sync, parent_clock: #PID<0.825.0>, latency: 0}, delayed_demands: MapSet.new([]), effective_flow_control: :push, initialized?: true, terminating?: false, setup_incomplete?: false, supplying_demand?: false, handling_action?: false, stalker: %Membrane.Core.Stalker{pid: #PID<0.789.0>, ets: #Reference<0.1819401566.4031905793.76364>}, resource_guard: #PID<0.2354.0>, subprocess_supervisor: #PID<0.2352.0>, handle_demand_loop_counter: 0, demand_size: nil, pads_to_snapshot: MapSet.new([]), playback_queue: [], pads_data: %{{Membrane.Pad, :input, #Reference<0.1819401566.4031774723.74731>} => %Membrane.Element.PadData{availability: :on_request, stream_format: nil, direction: :input, flow_control: :manual, name: :input, ref: {Membrane.Pad, :input, #Reference<0.1819401566.4031774723.74731>}, options: %{}, pid: #PID<0.2347.0>, other_ref: :output, input_queue: %Membrane.Core.Element.InputQueue{q: #Qex<[]>, log_tag: "{Membrane.Pad, :input, #Reference<0.1819401566.4031774723.74731>}", target_size: 40, atomic_demand: %Membrane.Core.Element.AtomicDemand{counter: %Membrane.Core.Element.AtomicDemand.DistributedAtomic{worker: #PID<0.2435.0>, atomic_ref: #Reference<0.1819401566.4031905793.98144>}, receiver_status: %Membrane.Core.Element.AtomicDemand.DistributedAtomic{worker: #PID<0.2436.0>, atomic_ref: #Reference<0.1819401566.4031905793.98146>}, receiver_process: #PID<0.2353.0>, sender_status: %Membrane.Core.Element.AtomicDemand.DistributedAtomic{worker: #PID<0.2437.0>, atomic_ref: #Reference<0.1819401566.4031905793.98148>}, sender_process: #PID<0.2347.0>, sender_pad_ref: :output, throttling_factor: 1, toilet_capacity: 200, receiver_demand_unit: :buffers, buffered_decrementation: 0, toilet_overflowed?: false}, inbound_metric: Membrane.Buffer.Metric.Count, outbound_metric: Membrane.Buffer.Metric.Count, pad_ref: {Membrane.Pad, :input, #Reference<0.1819401566.4031774723.74731>}, stalker_metrics: %{size: #Reference<0.1819401566.4031905793.98151>}, size: 0, demand: 40}, demand: 0, incoming_demand: nil, demand_unit: :buffers, start_of_stream?: false, end_of_stream?: false, auto_demand_size: nil, sticky_messages: [], atomic_demand: %Membrane.Core.Element.AtomicDemand{counter: %Membrane.Core.Element.AtomicDemand.DistributedAtomic{worker: #PID<0.2435.0>, atomic_ref: #Reference<0.1819401566.4031905793.98144>}, receiver_status: %Membrane.Core.Element.AtomicDemand.DistributedAtomic{worker: #PID<0.2436.0>, atomic_ref: #Reference<0.1819401566.4031905793.98146>}, receiver_process: #PID<0.2353.0>, sender_status: %Membrane.Core.Element.AtomicDemand.DistributedAtomic{worker: #PID<0.2437.0>, atomic_ref: #Reference<0.1819401566.4031905793.98148>}, sender_process: #PID<0.2347.0>, sender_pad_ref: :output, throttling_factor: 1, ...}, manual_demand_size: 0, associated_pads: [], sticky_events: [], stream_format_validation_params: [], other_demand_unit: :buffers, other_effective_flow_control: :push, stalker_metrics: %{...}, ...}}}
[error] GenServer #PID<0.2341.0> terminating
** (MatchError) no match of right hand side value: %{input: [{Membrane.Pad, :input, #Reference<0.1819401566.4031774723.74704>}]}
    (membrane_mp4_plugin 0.35.2) lib/membrane_mp4/muxer/cmaf.ex:227: Membrane.MP4.Muxer.CMAF.handle_playing/2
    (membrane_core 1.0.1) lib/membrane/core/callback_handler.ex:139: Membrane.Core.CallbackHandler.exec_callback/4
    (membrane_core 1.0.1) lib/membrane/core/callback_handler.ex:69: Membrane.Core.CallbackHandler.exec_and_handle_callback/5
    (membrane_core 1.0.1) lib/membrane/core/element/lifecycle_controller.ex:84: Membrane.Core.Element.LifecycleController.handle_playing/1
    (membrane_core 1.0.1) lib/membrane/core/element.ex:249: Membrane.Core.Element.handle_info/2
    (stdlib 6.1) gen_server.erl:2345: :gen_server.try_handle_info/3
    (stdlib 6.1) gen_server.erl:2433: :gen_server.handle_msg/6
    (stdlib 6.1) proc_lib.erl:329: :proc_lib.init_p_do_apply/3
Last message: {Membrane.Core.Message, :play, [], []}
State: %Membrane.Core.Element.State{module: Membrane.MP4.Muxer.CMAF, name: {:cmaf_muxer, "audio_master"}, parent_pid: #PID<0.824.0>, playback: :stopped, type: :filter, internal_state: %{segment_min_duration: 6000000000, video_pad: nil, pad_to_track_data: %{}, all_input_pads_ready?: false, buffers_awaiting_init: [], sample_queues: %{}, finish_current_segment?: false, awaiting_stream_formats: %{}, pads_registration_order: [{Membrane.Pad, :input, #Reference<0.1819401566.4031774723.74704>}], chunk_duration_range: %Membrane.MP4.Muxer.CMAF.DurationRange{min: 50000000, target: 1000000000}}, pad_refs: [{Membrane.Pad, :input, #Reference<0.1819401566.4031774723.74704>}], pads_info: %{input: %{name: :input, options: [], direction: :input, accepted_formats_str: ["%AAC{config: {:esds, _esds}}", "%Opus{self_delimiting?: false}", "%H264{stream_structure: structure, alignment: :au} when H264.is_avc(structure)", "%H265{stream_structure: structure, alignment: :au} when H265.is_hvc(structure)"], availability: :on_request, flow_control: :manual, demand_unit: :buffers}, output: %{name: :output, options: [tracks: [spec: "[Membrane.Pad.dynamic_id()] | :all", default: :all, description: "A list of the input pad ids that should be muxed together into a single output track.\n\nIf not specified the pad will include all unreferenced input pads.\n"]], direction: :output, accepted_formats_str: ["Membrane.CMAF.Track"], availability: :on_request, flow_control: :manual, demand_unit: nil}}, synchronization: %{timers: %{}, clock: nil, stream_sync: :membrane_no_sync, parent_clock: #PID<0.825.0>, latency: 0}, delayed_demands: MapSet.new([]), effective_flow_control: :push, initialized?: true, terminating?: false, setup_incomplete?: false, supplying_demand?: false, handling_action?: false, stalker: %Membrane.Core.Stalker{pid: #PID<0.789.0>, ets: #Reference<0.1819401566.4031905793.76364>}, resource_guard: #PID<0.2342.0>, subprocess_supervisor: #PID<0.2340.0>, handle_demand_loop_counter: 0, demand_size: nil, pads_to_snapshot: MapSet.new([]), playback_queue: [], pads_data: %{{Membrane.Pad, :input, #Reference<0.1819401566.4031774723.74704>} => %Membrane.Element.PadData{availability: :on_request, stream_format: nil, direction: :input, flow_control: :manual, name: :input, ref: {Membrane.Pad, :input, #Reference<0.1819401566.4031774723.74704>}, options: %{}, pid: #PID<0.2335.0>, other_ref: :output, input_queue: %Membrane.Core.Element.InputQueue{q: #Qex<[]>, log_tag: "{Membrane.Pad, :input, #Reference<0.1819401566.4031774723.74704>}", target_size: 40, atomic_demand: %Membrane.Core.Element.AtomicDemand{counter: %Membrane.Core.Element.AtomicDemand.DistributedAtomic{worker: #PID<0.2361.0>, atomic_ref: #Reference<0.1819401566.4031905793.97380>}, receiver_status: %Membrane.Core.Element.AtomicDemand.DistributedAtomic{worker: #PID<0.2362.0>, atomic_ref: #Reference<0.1819401566.4031905793.97383>}, receiver_process: #PID<0.2341.0>, sender_status: %Membrane.Core.Element.AtomicDemand.DistributedAtomic{worker: #PID<0.2363.0>, atomic_ref: #Reference<0.1819401566.4031905793.97386>}, sender_process: #PID<0.2335.0>, sender_pad_ref: :output, throttling_factor: 1, toilet_capacity: 200, receiver_demand_unit: :buffers, buffered_decrementation: 0, toilet_overflowed?: false}, inbound_metric: Membrane.Buffer.Metric.Count, outbound_metric: Membrane.Buffer.Metric.Count, pad_ref: {Membrane.Pad, :input, #Reference<0.1819401566.4031774723.74704>}, stalker_metrics: %{size: #Reference<0.1819401566.4031905793.97390>}, size: 0, demand: 40}, demand: 0, incoming_demand: nil, demand_unit: :buffers, start_of_stream?: false, end_of_stream?: false, auto_demand_size: nil, sticky_messages: [], atomic_demand: %Membrane.Core.Element.AtomicDemand{counter: %Membrane.Core.Element.AtomicDemand.DistributedAtomic{worker: #PID<0.2361.0>, atomic_ref: #Reference<0.1819401566.4031905793.97380>}, receiver_status: %Membrane.Core.Element.AtomicDemand.DistributedAtomic{worker: #PID<0.2362.0>, atomic_ref: #Reference<0.1819401566.4031905793.97383>}, receiver_process: #PID<0.2341.0>, sender_status: %Membrane.Core.Element.AtomicDemand.DistributedAtomic{worker: #PID<0.2363.0>, atomic_ref: #Reference<0.1819401566.4031905793.97386>}, sender_process: #PID<0.2335.0>, sender_pad_ref: :output, throttling_factor: 1, ...}, manual_demand_size: 0, associated_pads: [], sticky_events: [], stream_format_validation_params: [], other_demand_unit: :buffers, other_effective_flow_control: :pull, stalker_metrics: %{...}, ...}}}
[debug] <0.788.0>/:sink/{:cmaf_muxer, "audio_master"} subprocess supervisor got exit request from parent, reason: :shutdown, shutting down children
[debug] <0.788.0>/:sink/ Child {:cmaf_muxer, "audio_master"} crashed but was not a member of any crash group.
Terminating.

[debug] <0.788.0>/:sink/{:cmaf_muxer, "video_master"} subprocess supervisor got exit request from parent, reason: :shutdown, shutting down children
[error] <0.788.0>/:sink/ Terminating with reason: {:membrane_child_crash, {:cmaf_muxer, "audio_master"}, {{:badmatch, %{input: [{Membrane.Pad, :input, #Reference<0.1819401566.4031774723.74704>}]}}, [{Membrane.MP4.Muxer.CMAF, :handle_playing, 2, [file: ~c"lib/membrane_mp4/muxer/cmaf.ex", line: 227]}, {Membrane.Core.CallbackHandler, :exec_callback, 4, [file: ~c"lib/membrane/core/callback_handler.ex", line: 139]}, {Membrane.Core.CallbackHandler, :exec_and_handle_callback, 5, [file: ~c"lib/membrane/core/callback_handler.ex", line: 69]}, {Membrane.Core.Element.LifecycleController, :handle_playing, 1, [file: ~c"lib/membrane/core/element/lifecycle_controller.ex", line: 84]}, {Membrane.Core.Element, :handle_info, 2, [file: ~c"lib/membrane/core/element.ex", line: 249]}, {:gen_server, :try_handle_info, 3, [file: ~c"gen_server.erl", line: 2345]}, {:gen_server, :handle_msg, 6, [file: ~c"gen_server.erl", line: 2433]}, {:proc_lib, :init_p_do_apply, 3, [file: ~c"proc_lib.erl", line: 329]}]}}
[debug] <0.788.0>/:sink/ subprocess supervisor got exit request from parent, reason: :shutdown, shutting down children
[debug] <0.788.0>/:sink/{:parser, "video_master"} subprocess supervisor got exit request from parent, reason: :shutdown, shutting down children
[debug] <0.788.0>/:sink/:sink subprocess supervisor got exit request from parent, reason: :shutdown, shutting down children
[info] Elixir.Algora.Pipeline.Storage.Manifest terminating because of :shutdown
[debug] <0.788.0>/ Child :sink crashed but was not a member of any crash group.
Terminating.

[debug] <0.788.0>/:sink/{:parser, "audio_master"} subprocess supervisor got exit request from parent, reason: :shutdown, shutting down children
[error] <0.788.0>/ Terminating with reason: {:membrane_child_crash, :sink, {:membrane_child_crash, {:cmaf_muxer, "audio_master"}, {{:badmatch, %{input: [{Membrane.Pad, :input, #Reference<0.1819401566.4031774723.74704>}]}}, [{Membrane.MP4.Muxer.CMAF, :handle_playing, 2, [file: ~c"lib/membrane_mp4/muxer/cmaf.ex", line: 227]}, {Membrane.Core.CallbackHandler, :exec_callback, 4, [file: ~c"lib/membrane/core/callback_handler.ex", line: 139]}, {Membrane.Core.CallbackHandler, :exec_and_handle_callback, 5, [file: ~c"lib/membrane/core/callback_handler.ex", line: 69]}, {Membrane.Core.Element.LifecycleController, :handle_playing, 1, [file: ~c"lib/membrane/core/element/lifecycle_controller.ex", line: 84]}, {Membrane.Core.Element, :handle_info, 2, [file: ~c"lib/membrane/core/element.ex", line: 249]}, {:gen_server, :try_handle_info, 3, [file: ~c"gen_server.erl", line: 2345]}, {:gen_server, :handle_msg, 6, [file: ~c"gen_server.erl", line: 2433]}, {:proc_lib, :init_p_do_apply, 3, [file: ~c"proc_lib.erl", line: 329]}]}}}
[debug] <0.788.0>/ Pipeline supervisor got exit from pipeline with reason {:membrane_child_crash, :sink, {:membrane_child_crash, {:cmaf_muxer, "audio_master"}, {{:badmatch, %{input: [{Membrane.Pad, :input, #Reference<0.1819401566.4031774723.74704>}]}}, [{Membrane.MP4.Muxer.CMAF, :handle_playing, 2, [file: ~c"lib/membrane_mp4/muxer/cmaf.ex", line: 227]}, {Membrane.Core.CallbackHandler, :exec_callback, 4, [file: ~c"lib/membrane/core/callback_handler.ex", line: 139]}, {Membrane.Core.CallbackHandler, :exec_and_handle_callback, 5, [file: ~c"lib/membrane/core/callback_handler.ex", line: 69]}, {Membrane.Core.Element.LifecycleController, :handle_playing, 1, [file: ~c"lib/membrane/core/element/lifecycle_controller.ex", line: 84]}, {Membrane.Core.Element, :handle_info, 2, [file: ~c"lib/membrane/core/element.ex", line: 249]}, {:gen_server, :try_handle_info, 3, [file: ~c"gen_server.erl", line: 2345]}, {:gen_server, :handle_msg, 6, [file: ~c"gen_server.erl", line: 2433]}, {:proc_lib, :init_p_do_apply, 3, [file: ~c"proc_lib.erl", line: 329]}]}}}, stopping subprocess supervisor
[debug] <0.788.0>/ subprocess supervisor got exit request from parent, reason: :shutdown, shutting down children
[debug] <0.788.0>/:funnel_video subprocess supervisor got exit request from parent, reason: :shutdown, shutting down children
[debug] <0.788.0>/{:audio_reconnect, 2} subprocess supervisor got exit request from parent, reason: :shutdown, shutting down children
[debug] <0.788.0>/{:src, 2}/ subprocess supervisor got exit request from parent, reason: :shutdown, shutting down children
[debug] <0.788.0>/{:video_reconnect, 2} subprocess supervisor got exit request from parent, reason: :shutdown, shutting down children
[debug] <0.788.0>/{:src, 0}/ subprocess supervisor got exit request from parent, reason: :shutdown, shutting down children
[debug] <0.788.0>/{:src, 2}/:funnel_audio subprocess supervisor got exit request from parent, reason: :shutdown, shutting down children
[debug] <0.788.0>/{:src, 2}/:funnel_video subprocess supervisor got exit request from parent, reason: :shutdown, shutting down children
[debug] <0.788.0>/{:video_reconnect, 0} subprocess supervisor got exit request from parent, reason: :shutdown, shutting down children
[debug] <0.788.0>/{:src, 0}/:src subprocess supervisor got exit request from parent, reason: :shutdown, shutting down children
[debug] <0.788.0>/{:src, 0}/:audio_parser subprocess supervisor got exit request from parent, reason: :shutdown, shutting down children
[debug] <0.788.0>/{:src, 2}/:audio_parser subprocess supervisor got exit request from parent, reason: :shutdown, shutting down children
[debug] <0.788.0>/:funnel_audio subprocess supervisor got exit request from parent, reason: :shutdown, shutting down children
[debug] <0.788.0>/{:audio_reconnect, 0} subprocess supervisor got exit request from parent, reason: :shutdown, shutting down children
[debug] <0.788.0>/{:src, 1}/ subprocess supervisor got exit request from parent, reason: :shutdown, shutting down children
[debug] <0.788.0>/:tee_audio subprocess supervisor got exit request from parent, reason: :shutdown, shutting down children
[debug] <0.788.0>/{:audio_reconnect, 1} subprocess supervisor got exit request from parent, reason: :shutdown, shutting down children
[debug] <0.788.0>/{:src, 2}/:src subprocess supervisor got exit request from parent, reason: :shutdown, shutting down children
[debug] <0.788.0>/{:src, 1}/:src subprocess supervisor got exit request from parent, reason: :shutdown, shutting down children
[debug] <0.788.0>/{:src, 0}/:video_parser subprocess supervisor got exit request from parent, reason: :shutdown, shutting down children
[debug] <0.788.0>/{:src, 0}/:funnel_audio subprocess supervisor got exit request from parent, reason: :shutdown, shutting down children
[debug] <0.788.0>/{:src, 2}/:video_parser subprocess supervisor got exit request from parent, reason: :shutdown, shutting down children
[debug] <0.788.0>/:tee_video subprocess supervisor got exit request from parent, reason: :shutdown, shutting down children
[debug] <0.788.0>/{:video_reconnect, 1} subprocess supervisor got exit request from parent, reason: :shutdown, shutting down children
[debug] <0.788.0>/{:src, 0}/:funnel_video subprocess supervisor got exit request from parent, reason: :shutdown, shutting down children
[debug] <0.788.0>/{:src, 1}/:demuxer subprocess supervisor got exit request from parent, reason: :shutdown, shutting down children
[debug] <0.788.0>/{:src, 0}/:demuxer subprocess supervisor got exit request from parent, reason: :shutdown, shutting down children
[debug] <0.788.0>/{:src, 2}/:demuxer subprocess supervisor got exit request from parent, reason: :shutdown, shutting down children
[debug] <0.788.0>/{:src, 1}/:funnel_video subprocess supervisor got exit request from parent, reason: :shutdown, shutting down children
[debug] <0.788.0>/{:src, 1}/:audio_parser subprocess supervisor got exit request from parent, reason: :shutdown, shutting down children
[debug] <0.788.0>/{:src, 1}/:funnel_audio subprocess supervisor got exit request from parent, reason: :shutdown, shutting down children
[debug] <0.788.0>/{:src, 1}/:video_parser subprocess supervisor got exit request from parent, reason: :shutdown, shutting down children
[debug] <0.788.0>/ Pipeline supervisor got exit from subprocess supervisor, exiting
[error] GenServer #PID<0.785.0> terminating
** (stop) {:membrane_child_crash, :sink, {:membrane_child_crash, {:cmaf_muxer, "audio_master"}, {{:badmatch, %{input: [{Membrane.Pad, :input, #Reference<0.1819401566.4031774723.74704>}]}}, [{Membrane.MP4.Muxer.CMAF, :handle_playing, 2, [file: ~c"lib/membrane_mp4/muxer/cmaf.ex", line: 227]}, {Membrane.Core.CallbackHandler, :exec_callback, 4, [file: ~c"lib/membrane/core/callback_handler.ex", line: 139]}, {Membrane.Core.CallbackHandler, :exec_and_handle_callback, 5, [file: ~c"lib/membrane/core/callback_handler.ex", line: 69]}, {Membrane.Core.Element.LifecycleController, :handle_playing, 1, [file: ~c"lib/membrane/core/element/lifecycle_controller.ex", line: 84]}, {Membrane.Core.Element, :handle_info, 2, [file: ~c"lib/membrane/core/element.ex", line: 249]}, {:gen_server, :try_handle_info, 3, [file: ~c"gen_server.erl", line: 2345]}, {:gen_server, :handle_msg, 6, [file: ~c"gen_server.erl", line: 2433]}, {:proc_lib, :init_p_do_apply, 3, [file: ~c"proc_lib.erl", line: 329]}]}}}
Last message: {:EXIT, #PID<0.786.0>, {:membrane_child_crash, :sink, {:membrane_child_crash, {:cmaf_muxer, "audio_master"}, {{:badmatch, %{input: [{Membrane.Pad, :input, #Reference<0.1819401566.4031774723.74704>}]}}, [{Membrane.MP4.Muxer.CMAF, :handle_playing, 2, [file: ~c"lib/membrane_mp4/muxer/cmaf.ex", line: 227]}, {Membrane.Core.CallbackHandler, :exec_callback, 4, [file: ~c"lib/membrane/core/callback_handler.ex", line: 139]}, {Membrane.Core.CallbackHandler, :exec_and_handle_callback, 5, [file: ~c"lib/membrane/core/callback_handler.ex", line: 69]}, {Membrane.Core.Element.LifecycleController, :handle_playing, 1, [file: ~c"lib/membrane/core/element/lifecycle_controller.ex", line: 84]}, {Membrane.Core.Element, :handle_info, 2, [file: ~c"lib/membrane/core/element.ex", line: 249]}, {:gen_server, :try_handle_info, 3, [file: ~c"gen_server.erl", line: 2345]}, {:gen_server, :handle_msg, 6, [file: ~c"gen_server.erl", line: 2433]}, {:proc_lib, :init_p_do_apply, 3, [file: ~c"proc_lib.erl", line: 329]}]}}}}

[kaf-lamed-beyt]

How can i reproduce this @zcesur?

I'm running the project locally now and nothing seems to be broken.

[zcesur]

Yeah it's hard to reproduce consistently. Try start streaming -> stop streaming -> start streaming back to back a handful of times. Eventually it should crash

[kaf-lamed-beyt]

Oh okay. I'll let you know how it goes

[kaf-lamed-beyt]

Does this make any sense @zcesur?

[error] Failed to upload partial 97b9043b-4613-4296-96e4-a8cf87b4df0f/audio_segment_6_audio_master_0_part.m4s {:error, {:http_error, 403, %{body: "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>AccessDenied</Code><Message>Access Denied.</Message><Resource>/algora-tv-test/97b9043b-4613-4296-96e4-a8cf87b4df0f/audio_segment_6_audio_master_0_part.m4s</Resource><RequestId>1734459622955147919</RequestId><Key>97b9043b-4613-4296-96e4-a8cf87b4df0f/audio_segment_6_audio_master_0_part.m4s</Key><BucketName>algora-tv-test</BucketName></Error>", headers: [{"Access-Control-Allow-Headers", "*"}, {"Access-Control-Allow-Methods", "*"}, {"Access-Control-Allow-Origin", "*"}, {"Access-Control-Expose-Headers", "*"}, {"Content-Length", "393"}, {"Content-Type", "application/xml"}, {"Server", "Tigris OS"}, {"Server-Timing", "total;dur=310"}, {"Strict-Transport-Security", "max-age=63072000; includeSubDomains; preload"}, {"X-Amz-Request-Id", "1734459622955147919"}, {"Date", "Tue, 17 Dec 2024 18:20:23 GMT"}], status_code: 403}}}
[error] Task #PID<0.1713.0> started from #PID<0.866.0> terminating
** (Protocol.UndefinedError) protocol String.Chars not implemented for {1138173, {:error, {:http_error, 403, %{body: "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>AccessDenied</Code><Message>Access Denied.</Message><Resource>/algora-tv-test/97b9043b-4613-4296-96e4-a8cf87b4df0f/video_segment_4_video_master.m4s</Resource><RequestId>1734459623926262055</RequestId><Key>97b9043b-4613-4296-96e4-a8cf87b4df0f/video_segment_4_video_master.m4s</Key><BucketName>algora-tv-test</BucketName></Error>", headers: [{"Access-Control-Allow-Headers", "*"}, {"Access-Control-Allow-Methods", "*"}, {"Access-Control-Allow-Origin", "*"}, {"Access-Control-Expose-Headers", "*"}, {"Content-Length", "379"}, {"Content-Type", "application/xml"}, {"Server", "Tigris OS"}, {"Server-Timing", "total;dur=0"}, {"Strict-Transport-Security", "max-age=63072000; includeSubDomains; preload"}, {"X-Amz-Request-Id", "1734459623926262055"}, {"Date", "Tue, 17 Dec 2024 18:20:23 GMT"}, {"Connection", "close"}], status_code: 403}}}} of type Tuple. This protocol is implemented for the following type(s): Atom, BitString, Complex, Date, DateTime, Decimal, Float, Image.Exif.Gps, Image.Exif.Thumbnail, Integer, List, NaiveDateTime, OpenApiSpex.Cast.Error, Phoenix.LiveComponent.CID, Postgrex.Copy, Postgrex.INET, Postgrex.MACADDR, Postgrex.Query, Time, URI, Version, Version.Requirement
    (elixir 1.17.3) lib/string/chars.ex:3: String.Chars.impl_for!/1
    (elixir 1.17.3) lib/string/chars.ex:22: String.Chars.to_string/1
    (algora 0.1.0) lib/algora/pipeline/storage.ex:310: anonymous fn/2 in Algora.Pipeline.Storage.store_content/6
    (elixir 1.17.3) lib/task/supervised.ex:101: Task.Supervised.invoke_mfa/2
Function: #Function<5.16005492/0 in Algora.Pipeline.Storage.store_content/6>
    Args: []
[error] Failed to upload partial 97b9043b-4613-4296-96e4-a8cf87b4df0f/video_segment_4_video_master_6_part.m4s {:error, {:http_error, 403, %{body: "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>AccessDenied</Code><Message>Access Denied.</Message><Resource>/algora-tv-test/97b9043b-4613-4296-96e4-a8cf87b4df0f/video_segment_4_video_master_6_part.m4s</Resource><RequestId>1734459623671184396</RequestId><Key>97b9043b-4613-4296-96e4-a8cf87b4df0f/video_segment_4_video_master_6_part.m4s</Key><BucketName>algora-tv-test</BucketName></Error>", headers: [{"Access-Control-Allow-Headers", "*"}, {"Access-Control-Allow-Methods", "*"}, {"Access-Control-Allow-Origin", "*"}, {"Access-Control-Expose-Headers", "*"}, {"Content-Length", "393"}, {"Content-Type", "application/xml"}, {"Server", "Tigris OS"}, {"Server-Timing", "total;dur=0"}, {"Strict-Transport-Security", "max-age=63072000; includeSubDomains; preload"}, {"X-Amz-Request-Id", "1734459623671184396"}, {"Date", "Tue, 17 Dec 2024 18:20:25 GMT"}], status_code: 403}}}
[error] Failed to upload partial 97b9043b-4613-4296-96e4-a8cf87b4df0f/audio_segment_6_audio_master_1_part.m4s {:error, {:http_error, 403, %{body: "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>AccessDenied</Code><Message>Access Denied.</Message><Resource>/algora-tv-test/97b9043b-4613-4296-96e4-a8cf87b4df0f/audio_segment_6_audio_master_1_part.m4s</Resource><RequestId>1734459625878422398</RequestId><Key>97b9043b-4613-4296-96e4-a8cf87b4df0f/audio_segment_6_audio_master_1_part.m4s</Key><BucketName>algora-tv-test</BucketName></Error>", headers: [{"Access-Control-Allow-Headers", "*"}, {"Access-Control-Allow-Methods", "*"}, {"Access-Control-Allow-Origin", "*"}, {"Access-Control-Expose-Headers", "*"}, {"Content-Length", "393"}, {"Content-Type", "application/xml"}, {"Server", "Tigris OS"}, {"Server-Timing", "total;dur=0"}, {"Strict-Transport-Security", "max-age=63072000; includeSubDomains; preload"}, {"X-Amz-Request-Id", "1734459625878422398"}, {"Date", "Tue, 17 Dec 2024 18:20:25 GMT"}], status_code: 403}}}
[error] Failed to upload partial 97b9043b-4613-4296-96e4-a8cf87b4df0f/video_segment_5_video_master_0_part.m4s {:error, {:http_error, 403, %{body: "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>AccessDenied</Code><Message>Access Denied.</Message><Resource>/algora-tv-test/97b9043b-4613-4296-96e4-a8cf87b4df0f/video_segment_5_video_master_0_part.m4s</Resource><RequestId>1734459626294458855</RequestId><Key>97b9043b-4613-4296-96e4-a8cf87b4df0f/video_segment_5_video_master_0_part.m4s</Key><BucketName>algora-tv-test</BucketName></Error>", headers: [{"Access-Control-Allow-Headers", "*"}, {"Access-Control-Allow-Methods", "*"}, {"Access-Control-Allow-Origin", "*"}, {"Access-Control-Expose-Headers", "*"}, {"Content-Length", "393"}, {"Content-Type", "application/xml"}, {"Server", "Tigris OS"}, {"Server-Timing", "total;dur=0"}, {"Strict-Transport-Security", "max-age=63072000; includeSubDomains; preload"}, {"X-Amz-Request-Id", "1734459626294458855"}, {"Date", "Tue, 17 Dec 2024 18:20:26 GMT"}, {"Connection", "close"}], status_code: 403}}}
[error] Failed to upload partial 97b9043b-4613-4296-96e4-a8cf87b4df0f/audio_segment_6_audio_master_2_part.m4s {:error, {:http_error, 403, %{body: "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>AccessDenied</Code><Message>Access Denied.</Message><Resource>/algora-tv-test/97b9043b-4613-4296-96e4-a8cf87b4df0f/audio_segment_6_audio_master_2_part.m4s</Resource><RequestId>1734459627847008240</RequestId><Key>97b9043b-4613-4296-96e4-a8cf87b4df0f/audio_segment_6_audio_master_2_part.m4s</Key><BucketName>algora-tv-test</BucketName></Error>", headers: [{"Access-Control-Allow-Headers", "*"}, {"Access-Control-Allow-Methods", "*"}, {"Access-Control-Allow-Origin", "*"}, {"Access-Control-Expose-Headers", "*"}, {"Content-Length", "393"}, {"Content-Type", "application/xml"}, {"Server", "Tigris OS"}, {"Server-Timing", "total;dur=0"}, {"Strict-Transport-Security", "max-age=63072000; includeSubDomains; preload"}, {"X-Amz-Request-Id", "1734459627847008240"}, {"Date", "Tue, 17 Dec 2024 18:20:27 GMT"}], status_code: 403}}}
[error] Failed to upload partial 97b9043b-4613-4296-96e4-a8cf87b4df0f/video_segment_5_video_master_1_part.m4s {:error, {:http_error, 403, %{body: "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>AccessDenied</Code><Message>Access Denied.</Message><Resource>/algora-tv-test/97b9043b-4613-4296-96e4-a8cf87b4df0f/video_segment_5_video_master_1_part.m4s</Resource><RequestId>1734459628244613692</RequestId><Key>97b9043b-4613-4296-96e4-a8cf87b4df0f/video_segment_5_video_master_1_part.m4s</Key><BucketName>algora-tv-test</BucketName></Error>", headers: [{"Access-Control-Allow-Headers", "*"}, {"Access-Control-Allow-Methods", "*"}, {"Access-Control-Allow-Origin", "*"}, {"Access-Control-Expose-Headers", "*"}, {"Content-Length", "393"}, {"Content-Type", "application/xml"}, {"Server", "Tigris OS"}, {"Server-Timing", "total;dur=0"}, {"Strict-Transport-Security", "max-age=63072000; includeSubDomains; preload"}, {"X-Amz-Request-Id", "1734459628244613692"}, {"Date", "Tue, 17 Dec 2024 18:20:28 GMT"}], status_code: 403}}}
[error] Failed to upload partial 97b9043b-4613-4296-96e4-a8cf87b4df0f/audio_segment_6_audio_master_3_part.m4s {:error, {:http_error, 403, %{body: "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>AccessDenied</Code><Message>Access Denied.</Message><Resource>/algora-tv-test/97b9043b-4613-4296-96e4-a8cf87b4df0f/audio_segment_6_audio_master_3_part.m4s</Resource><RequestId>1734459629266372104</RequestId><Key>97b9043b-4613-4296-96e4-a8cf87b4df0f/audio_segment_6_audio_master_3_part.m4s</Key><BucketName>algora-tv-test</BucketName></Error>", headers: [{"Access-Control-Allow-Headers", "*"}, {"Access-Control-Allow-Methods", "*"}, {"Access-Control-Allow-Origin", "*"}, {"Access-Control-Expose-Headers", "*"}, {"Content-Length", "393"}, {"Content-Type", "application/xml"}, {"Server", "Tigris OS"}, {"Server-Timing", "total;dur=0"}, {"Strict-Transport-Security", "max-age=63072000; includeSubDomains; preload"}, {"X-Amz-Request-Id", "1734459629266372104"}, {"Date", "Tue, 17 Dec 2024 18:20:29 GMT"}], status_code: 403}}}
[error] Failed to upload partial 97b9043b-4613-4296-96e4-a8cf87b4df0f/video_segment_5_video_master_2_part.m4s {:error, {:http_error, 403, %{body: "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>AccessDenied</Code><Message>Access Denied.</Message><Resource>/algora-tv-test/97b9043b-4613-4296-96e4-a8cf87b4df0f/video_segment_5_video_master_2_part.m4s</Resource><RequestId>1734459629646876342</RequestId><Key>97b9043b-4613-4296-96e4-a8cf87b4df0f/video_segment_5_video_master_2_part.m4s</Key><BucketName>algora-tv-test</BucketName></Error>", headers: [{"Access-Control-Allow-Headers", "*"}, {"Access-Control-Allow-Methods", "*"}, {"Access-Control-Allow-Origin", "*"}, {"Access-Control-Expose-Headers", "*"}, {"Content-Length", "393"}, {"Content-Type", "application/xml"}, {"Server", "Tigris OS"}, {"Server-Timing", "total;dur=0"}, {"Strict-Transport-Security", "max-age=63072000; includeSubDomains; preload"}, {"X-Amz-Request-Id", "1734459629646876342"}, {"Date", "Tue, 17 Dec 2024 18:20:29 GMT"}, {"Connection", "close"}], status_code: 403}}}
[error] Failed to upload partial 97b9043b-4613-4296-96e4-a8cf87b4df0f/audio_segment_6_audio_master_4_part.m4s {:error, {:http_error, 403, %{body: "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>AccessDenied</Code><Message>Access Denied.</Message><Resource>/algora-tv-test/97b9043b-4613-4296-96e4-a8cf87b4df0f/audio_segment_6_audio_master_4_part.m4s</Resource><RequestId>1734459630931241384</RequestId><Key>97b9043b-4613-4296-96e4-a8cf87b4df0f/audio_segment_6_audio_master_4_part.m4s</Key><BucketName>algora-tv-test</BucketName></Error>", headers: [{"Access-Control-Allow-Headers", "*"}, {"Access-Control-Allow-Methods", "*"}, {"Access-Control-Allow-Origin", "*"}, {"Access-Control-Expose-Headers", "*"}, {"Content-Length", "393"}, {"Content-Type", "application/xml"}, {"Server", "Tigris OS"}, {"Server-Timing", "total;dur=0"}, {"Strict-Transport-Security", "max-age=63072000; includeSubDomains; preload"}, {"X-Amz-Request-Id", "1734459630931241384"}, {"Date", "Tue, 17 Dec 2024 18:20:30 GMT"}], status_code: 403}}}
[error] Failed to upload partial 97b9043b-4613-4296-96e4-a8cf87b4df0f/video_segment_5_video_master_3_part.m4s {:error, {:http_error, 403, %{body: "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>AccessDenied</Code><Message>Access Denied.</Message><Resource>/algora-tv-test/97b9043b-4613-4296-96e4-a8cf87b4df0f/video_segment_5_video_master_3_part.m4s</Resource><RequestId>1734459631314149893</RequestId><Key>97b9043b-4613-4296-96e4-a8cf87b4df0f/video_segment_5_video_master_3_part.m4s</Key><BucketName>algora-tv-test</BucketName></Error>", headers: [{"Access-Control-Allow-Headers", "*"}, {"Access-Control-Allow-Methods", "*"}, {"Access-Control-Allow-Origin", "*"}, {"Access-Control-Expose-Headers", "*"}, {"Content-Length", "393"}, {"Content-Type", "application/xml"}, {"Server", "Tigris OS"}, {"Server-Timing", "total;dur=0"}, {"Strict-Transport-Security", "max-age=63072000; includeSubDomains; preload"}, {"X-Amz-Request-Id", "1734459631314149893"}, {"Date", "Tue, 17 Dec 2024 18:20:32 GMT"}], status_code: 403}}}
[error] Failed to upload 97b9043b-4613-4296-96e4-a8cf87b4df0f/index.m3u8: {:error, {:http_error, 403, %{body: "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>AccessDenied</Code><Message>Access Denied.</Message><Resource>/algora-tv-test/97b9043b-4613-4296-96e4-a8cf87b4df0f/index.m3u8</Resource><RequestId>1734459633039790985</RequestId><Key>97b9043b-4613-4296-96e4-a8cf87b4df0f/index.m3u8</Key><BucketName>algora-tv-test</BucketName></Error>", headers: [{"Access-Control-Allow-Headers", "*"}, {"Access-Control-Allow-Methods", "*"}, {"Access-Control-Allow-Origin", "*"}, {"Access-Control-Expose-Headers", "*"}, {"Content-Length", "335"}, {"Content-Type", "application/xml"}, {"Server", "Tigris OS"}, {"Server-Timing", "total;dur=0"}, {"Strict-Transport-Security", "max-age=63072000; includeSubDomains; preload"}, {"X-Amz-Request-Id", "1734459633039790985"}, {"Date", "Tue, 17 Dec 2024 18:20:33 GMT"}], status_code: 403}}}
[error] Failed to upload partial 97b9043b-4613-4296-96e4-a8cf87b4df0f/audio_segment_6_audio_master_5_part.m4s {:error, {:http_error, 403, %{body: "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>AccessDenied</Code><Message>Access Denied.</Message><Resource>/algora-tv-test/97b9043b-4613-4296-96e4-a8cf87b4df0f/audio_segment_6_audio_master_5_part.m4s</Resource><RequestId>1734459633035161809</RequestId><Key>97b9043b-4613-4296-96e4-a8cf87b4df0f/audio_segment_6_audio_master_5_part.m4s</Key><BucketName>algora-tv-test</BucketName></Error>", headers: [{"Access-Control-Allow-Headers", "*"}, {"Access-Control-Allow-Methods", "*"}, {"Access-Control-Allow-Origin", "*"}, {"Access-Control-Expose-Headers", "*"}, {"Content-Length", "393"}, {"Content-Type", "application/xml"}, {"Server", "Tigris OS"}, {"Server-Timing", "total;dur=0"}, {"Strict-Transport-Security", "max-age=63072000; includeSubDomains; preload"}, {"X-Amz-Request-Id", "1734459633035161809"}, {"Date", "Tue, 17 Dec 2024 18:20:33 GMT"}], status_code: 403}}}
[error] Failed to upload partial 97b9043b-4613-4296-96e4-a8cf87b4df0f/video_segment_5_video_master_4_part.m4s {:error, {:http_error, 403, %{body: "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>AccessDenied</Code><Message>Access Denied.</Message><Resource>/algora-tv-test/97b9043b-4613-4296-96e4-a8cf87b4df0f/video_segment_5_video_master_4_part.m4s</Resource><RequestId>1734459633513959520</RequestId><Key>97b9043b-4613-4296-96e4-a8cf87b4df0f/video_segment_5_video_master_4_part.m4s</Key><BucketName>algora-tv-test</BucketName></Error>", headers: [{"Access-Control-Allow-Headers", "*"}, {"Access-Control-Allow-Methods", "*"}, {"Access-Control-Allow-Origin", "*"}, {"Access-Control-Expose-Headers", "*"}, {"Content-Length", "393"}, {"Content-Type", "application/xml"}, {"Server", "Tigris OS"}, {"Server-Timing", "total;dur=0"}, {"Strict-Transport-Security", "max-age=63072000; includeSubDomains; preload"}, {"X-Amz-Request-Id", "1734459633513959520"}, {"Date", "Tue, 17 Dec 2024 18:20:33 GMT"}, {"Connection", "close"}], status_code: 403}}}
[error] Failed to upload partial 97b9043b-4613-4296-96e4-a8cf87b4df0f/audio_segment_6_audio_master_6_part.m4s {:error, {:http_error, 403, %{body: "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>AccessDenied</Code><Message>Access Denied.</Message><Resource>/algora-tv-test/97b9043b-4613-4296-96e4-a8cf87b4df0f/audio_segment_6_audio_master_6_part.m4s</Resource><RequestId>1734459634215247516</RequestId><Key>97b9043b-4613-4296-96e4-a8cf87b4df0f/audio_segment_6_audio_master_6_part.m4s</Key><BucketName>algora-tv-test</BucketName></Error>", headers: [{"Access-Control-Allow-Headers", "*"}, {"Access-Control-Allow-Methods", "*"}, {"Access-Control-Allow-Origin", "*"}, {"Access-Control-Expose-Headers", "*"}, {"Content-Length", "393"}, {"Content-Type", "application/xml"}, {"Server", "Tigris OS"}, {"Server-Timing", "total;dur=0"}, {"Strict-Transport-Security", "max-age=63072000; includeSubDomains; preload"}, {"X-Amz-Request-Id", "1734459634215247516"}, {"Date", "Tue, 17 Dec 2024 18:20:34 GMT"}], status_code: 403}}}
[info] <0.827.0>/ Finalized manifests for video "97b9043b-4613-4296-96e4-a8cf87b4df0f"
[debug] <0.827.0>/ Removing children: [:funnel_video, :funnel_audio, :tee_audio, :tee_video, :sink, {:audio_reconnect, 0}, {:src, 0}, {:video_reconnect, 0}]
[debug] <0.827.0>/:funnel_video Received terminate request
[debug] <0.827.0>/:funnel_video Terminating with reason :normal
[debug] <0.827.0>/:funnel_audio Received terminate request
[debug] <0.827.0>/:tee_video Received terminate request
[debug] <0.827.0>/{:audio_reconnect, 0} Received terminate request
[debug] <0.827.0>/{:src, 0}/ Removing children: [:src, :funnel_video, :funnel_audio, :demuxer, :audio_parser, :video_parser]
[debug] <0.827.0>/{:video_reconnect, 0} Received terminate request
[debug] <0.827.0>/{:src, 0}/:demuxer Received terminate request
[debug] <0.827.0>/:tee_audio Received terminate request
[debug] <0.827.0>/{:src, 0}/:src Received terminate request
[debug] <0.827.0>/:sink/ Removing children: [:sink, {:cmaf_muxer, "audio_master"}, {:cmaf_muxer, "video_master"}, {:parser, "audio_master"}, {:parser, "video_master"}]
[debug] <0.827.0>/{:audio_reconnect, 0} Terminating with reason :normal
[debug] <0.827.0>/{:video_reconnect, 0} Terminating with reason :normal
[debug] <0.827.0>/{:src, 0}/:funnel_video Received terminate request
[debug] <0.827.0>/{:src, 0}/:video_parser Received terminate request
[debug] <0.827.0>/:tee_audio Received end of stream on pad :input
[debug] <0.827.0>/{:src, 0}/:funnel_video Terminating with reason :normal
[debug] <0.827.0>/{:src, 0}/:video_parser Terminating with reason :normal
[debug] <0.827.0>/:sink/{:parser, "audio_master"} Received terminate request
[debug] <0.827.0>/:sink/{:cmaf_muxer, "audio_master"} Received terminate request
[debug] <0.827.0>/:tee_audio Terminating with reason :normal
[debug] <0.827.0>/:sink/{:parser, "audio_master"} Received end of stream on pad :input
[debug] <0.827.0>/:sink/{:cmaf_muxer, "audio_master"} Received end of stream on pad {Membrane.Pad, :input, #Reference<0.3109971441.2874146817.124247>}
[debug] <0.827.0>/:sink/:sink Received terminate request
[debug] <0.827.0>/{:src, 0}/:demuxer Terminating with reason :normal
[debug] <0.827.0>/:sink/{:parser, "audio_master"} Terminating with reason :normal
[debug] <0.827.0>/:sink/{:cmaf_muxer, "audio_master"} Terminating with reason :normal
[debug] <0.827.0>/:funnel_audio Terminating with reason :normal
[debug] <0.827.0>/:tee_video Received end of stream on pad :input
[debug] <0.827.0>/:tee_video Terminating with reason :normal
[debug] <0.827.0>/{:src, 0}/:funnel_audio Received terminate request
[debug] <0.827.0>/{:src, 0}/:funnel_audio Terminating with reason :normal
[debug] <0.827.0>/{:src, 0}/:audio_parser Received terminate request
[debug] <0.827.0>/{:src, 0}/:audio_parser Terminating with reason :normal
[debug] <0.827.0>/:sink/{:cmaf_muxer, "video_master"} Received terminate request
[debug] <0.827.0>/:sink/{:cmaf_muxer, "video_master"} Received end of stream on pad {Membrane.Pad, :input, #Reference<0.3109971441.2874146819.80156>}
[debug] <0.827.0>/:sink/{:parser, "video_master"} Received terminate request
[debug] <0.827.0>/:sink/{:parser, "video_master"} Received end of stream on pad :input
[debug] <0.827.0>/:sink/{:parser, "video_master"} Terminating with reason :normal
[debug] <0.827.0>/{:src, 0}/:src Terminating with reason :normal
[debug] <0.827.0>/:sink/:sink Received end of stream on pad {Membrane.Pad, :input, "audio_master"}
[debug] <0.827.0>/:sink/{:cmaf_muxer, "video_master"} Terminating with reason :normal
[debug] <0.827.0>/:tee_audio subprocess supervisor got exit request from parent, reason: :shutdown, shutting down children
[debug] <0.827.0>/{:src, 0}/:demuxer subprocess supervisor got exit request from parent, reason: :shutdown, shutting down children
[debug] <0.827.0>/:funnel_video subprocess supervisor got exit request from parent, reason: :shutdown, shutting down children
[debug] <0.827.0>/:sink/{:cmaf_muxer, "video_master"} subprocess supervisor got exit request from parent, reason: :shutdown, shutting down children
[debug] <0.827.0>/:sink/{:cmaf_muxer, "audio_master"} subprocess supervisor got exit request from parent, reason: :shutdown, shutting down children
[debug] <0.827.0>/{:src, 0}/:audio_parser subprocess supervisor got exit request from parent, reason: :shutdown, shutting down children
[debug] <0.827.0>/{:src, 0}/:src subprocess supervisor exiting
[debug] <0.827.0>/:tee_video subprocess supervisor got exit request from parent, reason: :shutdown, shutting down children
[debug] <0.827.0>/:sink/{:parser, "audio_master"} subprocess supervisor got exit request from parent, reason: :shutdown, shutting down children
[debug] <0.827.0>/:funnel_audio subprocess supervisor got exit request from parent, reason: :shutdown, shutting down children
[debug] <0.827.0>/{:audio_reconnect, 0} subprocess supervisor got exit request from parent, reason: :shutdown, shutting down children
[debug] <0.827.0>/:sink/{:parser, "video_master"} subprocess supervisor got exit request from parent, reason: :shutdown, shutting down children
[debug] <0.827.0>/{:src, 0}/:funnel_audio subprocess supervisor got exit request from parent, reason: :shutdown, shutting down children
[debug] <0.827.0>/{:src, 0}/:funnel_video subprocess supervisor got exit request from parent, reason: :shutdown, shutting down children
[debug] <0.827.0>/{:src, 0}/:video_parser subprocess supervisor got exit request from parent, reason: :shutdown, shutting down children
[debug] <0.827.0>/{:video_reconnect, 0} subprocess supervisor got exit request from parent, reason: :shutdown, shutting down children
[debug] <0.827.0>/:sink/ Not calling the handle_element_end_of_stream callback with the following arguments:
        {:parser, "audio_master"}, :input, %{name: :sink, children: %{:sink => %Membrane.ChildEntry{name: :sink, module: Algora.Pipeline.Sink, options: %Algora.Pipeline.Sink{manifest_config: %Algora.Pipeline.Sink.ManifestConfig{video_uuid: "97b9043b-4613-4296-96e4-a8cf87b4df0f", module: Algora.Pipeline.HLS, name: "index"}, track_config: %Algora.Pipeline.Sink.TrackConfig{target_window_duration: :infinity, mode: :live, header_naming_fun: &Membrane.HTTPAdaptiveStream.Manifest.Track.default_header_naming_fun/2, segment_naming_fun: &Membrane.HTTPAdaptiveStream.Manifest.Track.default_segment_naming_fun/1, partial_naming_fun: &Membrane.HTTPAdaptiveStream.Manifest.Track.default_partial_naming_fun/2, persist?: true}, storage: %Algora.Pipeline.Storage{directory: "/tmp/97b9043b-4613-4296-96e4-a8cf87b4df0f", video: %Algora.Library.Video{__meta__: #Ecto.Schema.Metadata<:loaded, "videos">, id: 149, duration: 0, title: "kaf-lamed-beyt's channel", description: nil, type: :livestream, format: :hls, corrupted: false, is_live: true, thumbnail_url: nil, vertical_thumbnail_url: nil, og_image_url: nil, url: "http://localhost:4000/hls/97b9043b-4613-4296-96e4-a8cf87b4df0f/index.m3u8", url_root: "http://localhost:4000/hls/97b9043b-4613-4296-96e4-a8cf87b4df0f", uuid: "97b9043b-4613-4296-96e4-a8cf87b4df0f", filename: "index.m3u8", channel_handle: "kaf-lamed-beyt", channel_name: "null (for now)", channel_avatar_url: "https://avatars.githubusercontent.com/u/48127194?v=4", messages_count: 0, visibility: :public, remote_path: "97b9043b-4613-4296-96e4-a8cf87b4df0f/index.m3u8", local_path: nil, deleted_at: nil, tags: ["typescript", "zig", "lua", "sol", "makin"], user_id: 1, user: #Ecto.Association.NotLoaded<association :user is not loaded>, show_id: nil, show: #Ecto.Association.NotLoaded<association :show is not loaded>, transmuxed_from_id: nil, transmuxed_from: #Ecto.Association.NotLoaded<association :transmuxed_from is not loaded>, content_metrics: #Ecto.Association.NotLoaded<association :content_metrics is not loaded>, messages: #Ecto.Association.NotLoaded<association :messages is not loaded>, appearances: #Ecto.Association.NotLoaded<association :appearances is not loaded>, product_reviews: #Ecto.Association.NotLoaded<association :product_reviews is not loaded>, inserted_at: ~N[2024-12-17 18:19:08], updated_at: ~N[2024-12-17 18:19:08]}, sequences: %{}, partials_in_ets: %{}, video_header: "", video_segment: "", setup_completed?: false, manifest_uploader: nil, partial_uploaders: %{}}, cleanup_after: nil}, component_type: :element, pid: #PID<0.866.0>, clock: nil, sync: :membrane_no_sync, spec_ref: #Reference<0.3109971441.2874146817.124000>, group: nil, initialized?: true, ready?: true, terminating?: true}, {:cmaf_muxer, "audio_master"} => %Membrane.ChildEntry{name: {:cmaf_muxer, "audio_master"}, module: Membrane.MP4.Muxer.CMAF, options: %Membrane.MP4.Muxer.CMAF{segment_min_duration: 6000000000, chunk_target_duration: 1000000000}, component_type: :element, pid: #PID<0.900.0>, clock: nil, sync: :membrane_no_sync, spec_ref: #Reference<0.3109971441.2874146818.81337>, group: nil, initialized?: true, ready?: true, terminating?: true}, {:cmaf_muxer, "video_master"} => %Membrane.ChildEntry{name: {:cmaf_muxer, "video_master"}, module: Membrane.MP4.Muxer.CMAF, options: %Membrane.MP4.Muxer.CMAF{segment_min_duration: 6000000000, chunk_target_duration: 1000000000}, component_type: :element, pid: #PID<0.906.0>, clock: nil, sync: :membrane_no_sync, spec_ref: #Reference<0.3109971441.2874146819.80122>, group: nil, initialized?: true, ready?: true, terminating?: true}, {:parser, "audio_master"} => %Membrane.ChildEntry{name: {:parser, "audio_master"}, module: Membrane.AAC.Parser, options: %Membrane.AAC.Parser{samples_per_frame: 1024, out_encapsulation: :none, audio_specific_config: nil, output_config: :esds}, component_type: :element, pid: #PID<0.897.0>, clock: nil, sync: :membrane_no_sync, spec_ref: #Reference<0.3109971441.2874146818.81337>, group: nil, initialized?: true, ready?: true, terminating?: true}, {:parser, "video_master"} => %Membrane.ChildEntry{name: {:parser, "video_master"}, module: Membrane.H264.Parser, options: %Membrane.H264.Parser{spss: [], ppss: [], output_alignment: :au, skip_until_keyframe: true, repeat_parameter_sets: true, output_stream_structure: :avc1, generate_best_effort_timestamps: false}, component_type: :element, pid: #PID<0.903.0>, clock: nil, sync: :membrane_no_sync, spec_ref: #Reference<0.3109971441.2874146819.80122>, group: nil, initialized?: true, ready?: true, terminating?: true}}, clock: nil, playback: :playing, resource_guard: #PID<0.864.0>, utility_supervisor: #PID<0.861.0>, pads: %{{Membrane.Pad, :input, "audio_master"} => %Membrane.Bin.PadData{ref: {Membrane.Pad, :input, "audio_master"}, options: %{encoding: :AAC, partial_segment_duration: 1000000000, segment_duration: 6000000000, track_name: "audio_master", max_framerate: nil}, availability: :on_request, direction: :input, name: :input, link_id: #Reference<0.3109971441.2874146818.81319>, endpoint: %Membrane.Core.Parent.Link.Endpoint{child: {:parser, "audio_master"}, pad_spec: :input, pad_ref: :input, pid: #PID<0.897.0>, pad_props: %{options: [], toilet_capacity: nil, target_queue_size: nil, min_demand_factor: nil, auto_demand_size: nil, throttling_factor: nil}, pad_info: %{name: :input, options: [], direction: :input, accepted_formats_str: ["AAC", "Membrane.RemoteStream"], availability: :always, flow_control: :manual, demand_unit: :buffers}, child_spec_ref: #Reference<0.3109971441.2874146818.81337>}, linked?: true, response_received?: false, spec_ref: #Reference<0.3109971441.2874146818.81337>}, {Membrane.Pad, :input, "video_master"} => %Membrane.Bin.PadData{ref: {Membrane.Pad, :input, "video_master"}, options: %{encoding: :H264, partial_segment_duration: 1000000000, segment_duration: 6000000000, track_name: "video_master", max_framerate: nil}, availability: :on_request, direction: :input, name: :input, link_id: #Reference<0.3109971441.2874146818.81320>, endpoint: %Membrane.Core.Parent.Link.Endpoint{child: {:parser, "video_master"}, pad_spec: :input, pad_ref: :input, pid: #PID<0.903.0>, pad_props: %{options: [], toilet_capacity: nil, target_queue_size: nil, min_demand_factor: nil, auto_demand_size: nil, throttling_factor: nil}, pad_info: %{name: :input, options: [], direction: :input, accepted_formats_str: ["RemoteStream", "H264"], availability: :always, flow_control: :auto}, child_spec_ref: #Reference<0.3109971441.2874146819.80122>}, linked?: true, response_received?: false, spec_ref: #Reference<0.3109971441.2874146819.80122>}}, parent_clock: #PID<0.830.0>, start_of_stream_received?: true}, %{original_module: Algora.Pipeline.SinkBin}
        because the bin is in the zombie mode
[debug] <0.827.0>/:sink/ Not calling the handle_element_end_of_stream callback with the following arguments:
        {:cmaf_muxer, "audio_master"}, {Membrane.Pad, :input, #Reference<0.3109971441.2874146817.124247>}, %{name: :sink, children: %{:sink => %Membrane.ChildEntry{name: :sink, module: Algora.Pipeline.Sink, options: %Algora.Pipeline.Sink{manifest_config: %Algora.Pipeline.Sink.ManifestConfig{video_uuid: "97b9043b-4613-4296-96e4-a8cf87b4df0f", module: Algora.Pipeline.HLS, name: "index"}, track_config: %Algora.Pipeline.Sink.TrackConfig{target_window_duration: :infinity, mode: :live, header_naming_fun: &Membrane.HTTPAdaptiveStream.Manifest.Track.default_header_naming_fun/2, segment_naming_fun: &Membrane.HTTPAdaptiveStream.Manifest.Track.default_segment_naming_fun/1, partial_naming_fun: &Membrane.HTTPAdaptiveStream.Manifest.Track.default_partial_naming_fun/2, persist?: true}, storage: %Algora.Pipeline.Storage{directory: "/tmp/97b9043b-4613-4296-96e4-a8cf87b4df0f", video: %Algora.Library.Video{__meta__: #Ecto.Schema.Metadata<:loaded, "videos">, id: 149, duration: 0, title: "kaf-lamed-beyt's channel", description: nil, type: :livestream, format: :hls, corrupted: false, is_live: true, thumbnail_url: nil, vertical_thumbnail_url: nil, og_image_url: nil, url: "http://localhost:4000/hls/97b9043b-4613-4296-96e4-a8cf87b4df0f/index.m3u8", url_root: "http://localhost:4000/hls/97b9043b-4613-4296-96e4-a8cf87b4df0f", uuid: "97b9043b-4613-4296-96e4-a8cf87b4df0f", filename: "index.m3u8", channel_handle: "kaf-lamed-beyt", channel_name: "null (for now)", channel_avatar_url: "https://avatars.githubusercontent.com/u/48127194?v=4", messages_count: 0, visibility: :public, remote_path: "97b9043b-4613-4296-96e4-a8cf87b4df0f/index.m3u8", local_path: nil, deleted_at: nil, tags: ["typescript", "zig", "lua", "sol", "makin"], user_id: 1, user: #Ecto.Association.NotLoaded<association :user is not loaded>, show_id: nil, show: #Ecto.Association.NotLoaded<association :show is not loaded>, transmuxed_from_id: nil, transmuxed_from: #Ecto.Association.NotLoaded<association :transmuxed_from is not loaded>, content_metrics: #Ecto.Association.NotLoaded<association :content_metrics is not loaded>, messages: #Ecto.Association.NotLoaded<association :messages is not loaded>, appearances: #Ecto.Association.NotLoaded<association :appearances is not loaded>, product_reviews: #Ecto.Association.NotLoaded<association :product_reviews is not loaded>, inserted_at: ~N[2024-12-17 18:19:08], updated_at: ~N[2024-12-17 18:19:08]}, sequences: %{}, partials_in_ets: %{}, video_header: "", video_segment: "", setup_completed?: false, manifest_uploader: nil, partial_uploaders: %{}}, cleanup_after: nil}, component_type: :element, pid: #PID<0.866.0>, clock: nil, sync: :membrane_no_sync, spec_ref: #Reference<0.3109971441.2874146817.124000>, group: nil, initialized?: true, ready?: true, terminating?: true}, {:cmaf_muxer, "audio_master"} => %Membrane.ChildEntry{name: {:cmaf_muxer, "audio_master"}, module: Membrane.MP4.Muxer.CMAF, options: %Membrane.MP4.Muxer.CMAF{segment_min_duration: 6000000000, chunk_target_duration: 1000000000}, component_type: :element, pid: #PID<0.900.0>, clock: nil, sync: :membrane_no_sync, spec_ref: #Reference<0.3109971441.2874146818.81337>, group: nil, initialized?: true, ready?: true, terminating?: true}, {:cmaf_muxer, "video_master"} => %Membrane.ChildEntry{name: {:cmaf_muxer, "video_master"}, module: Membrane.MP4.Muxer.CMAF, options: %Membrane.MP4.Muxer.CMAF{segment_min_duration: 6000000000, chunk_target_duration: 1000000000}, component_type: :element, pid: #PID<0.906.0>, clock: nil, sync: :membrane_no_sync, spec_ref: #Reference<0.3109971441.2874146819.80122>, group: nil, initialized?: true, ready?: true, terminating?: true}, {:parser, "audio_master"} => %Membrane.ChildEntry{name: {:parser, "audio_master"}, module: Membrane.AAC.Parser, options: %Membrane.AAC.Parser{samples_per_frame: 1024, out_encapsulation: :none, audio_specific_config: nil, output_config: :esds}, component_type: :element, pid: #PID<0.897.0>, clock: nil, sync: :membrane_no_sync, spec_ref: #Reference<0.3109971441.2874146818.81337>, group: nil, initialized?: true, ready?: true, terminating?: true}, {:parser, "video_master"} => %Membrane.ChildEntry{name: {:parser, "video_master"}, module: Membrane.H264.Parser, options: %Membrane.H264.Parser{spss: [], ppss: [], output_alignment: :au, skip_until_keyframe: true, repeat_parameter_sets: true, output_stream_structure: :avc1, generate_best_effort_timestamps: false}, component_type: :element, pid: #PID<0.903.0>, clock: nil, sync: :membrane_no_sync, spec_ref: #Reference<0.3109971441.2874146819.80122>, group: nil, initialized?: true, ready?: true, terminating?: true}}, clock: nil, playback: :playing, resource_guard: #PID<0.864.0>, utility_supervisor: #PID<0.861.0>, pads: %{{Membrane.Pad, :input, "audio_master"} => %Membrane.Bin.PadData{ref: {Membrane.Pad, :input, "audio_master"}, options: %{encoding: :AAC, partial_segment_duration: 1000000000, segment_duration: 6000000000, track_name: "audio_master", max_framerate: nil}, availability: :on_request, direction: :input, name: :input, link_id: #Reference<0.3109971441.2874146818.81319>, endpoint: %Membrane.Core.Parent.Link.Endpoint{child: {:parser, "audio_master"}, pad_spec: :input, pad_ref: :input, pid: #PID<0.897.0>, pad_props: %{options: [], toilet_capacity: nil, target_queue_size: nil, min_demand_factor: nil, auto_demand_size: nil, throttling_factor: nil}, pad_info: %{name: :input, options: [], direction: :input, accepted_formats_str: ["AAC", "Membrane.RemoteStream"], availability: :always, flow_control: :manual, demand_unit: :buffers}, child_spec_ref: #Reference<0.3109971441.2874146818.81337>}, linked?: true, response_received?: false, spec_ref: #Reference<0.3109971441.2874146818.81337>}, {Membrane.Pad, :input, "video_master"} => %Membrane.Bin.PadData{ref: {Membrane.Pad, :input, "video_master"}, options: %{encoding: :H264, partial_segment_duration: 1000000000, segment_duration: 6000000000, track_name: "video_master", max_framerate: nil}, availability: :on_request, direction: :input, name: :input, link_id: #Reference<0.3109971441.2874146818.81320>, endpoint: %Membrane.Core.Parent.Link.Endpoint{child: {:parser, "video_master"}, pad_spec: :input, pad_ref: :input, pid: #PID<0.903.0>, pad_props: %{options: [], toilet_capacity: nil, target_queue_size: nil, min_demand_factor: nil, auto_demand_size: nil, throttling_factor: nil}, pad_info: %{name: :input, options: [], direction: :input, accepted_formats_str: ["RemoteStream", "H264"], availability: :always, flow_control: :auto}, child_spec_ref: #Reference<0.3109971441.2874146819.80122>}, linked?: true, response_received?: false, spec_ref: #Reference<0.3109971441.2874146819.80122>}}, parent_clock: #PID<0.830.0>, start_of_stream_received?: true}, %{original_module: Algora.Pipeline.SinkBin}
        because the bin is in the zombie mode
[debug] <0.827.0>/:sink/ Not calling the handle_element_end_of_stream callback with the following arguments:
        {:parser, "video_master"}, :input, %{name: :sink, children: %{:sink => %Membrane.ChildEntry{name: :sink, module: Algora.Pipeline.Sink, options: %Algora.Pipeline.Sink{manifest_config: %Algora.Pipeline.Sink.ManifestConfig{video_uuid: "97b9043b-4613-4296-96e4-a8cf87b4df0f", module: Algora.Pipeline.HLS, name: "index"}, track_config: %Algora.Pipeline.Sink.TrackConfig{target_window_duration: :infinity, mode: :live, header_naming_fun: &Membrane.HTTPAdaptiveStream.Manifest.Track.default_header_naming_fun/2, segment_naming_fun: &Membrane.HTTPAdaptiveStream.Manifest.Track.default_segment_naming_fun/1, partial_naming_fun: &Membrane.HTTPAdaptiveStream.Manifest.Track.default_partial_naming_fun/2, persist?: true}, storage: %Algora.Pipeline.Storage{directory: "/tmp/97b9043b-4613-4296-96e4-a8cf87b4df0f", video: %Algora.Library.Video{__meta__: #Ecto.Schema.Metadata<:loaded, "videos">, id: 149, duration: 0, title: "kaf-lamed-beyt's channel", description: nil, type: :livestream, format: :hls, corrupted: false, is_live: true, thumbnail_url: nil, vertical_thumbnail_url: nil, og_image_url: nil, url: "http://localhost:4000/hls/97b9043b-4613-4296-96e4-a8cf87b4df0f/index.m3u8", url_root: "http://localhost:4000/hls/97b9043b-4613-4296-96e4-a8cf87b4df0f", uuid: "97b9043b-4613-4296-96e4-a8cf87b4df0f", filename: "index.m3u8", channel_handle: "kaf-lamed-beyt", channel_name: "null (for now)", channel_avatar_url: "https://avatars.githubusercontent.com/u/48127194?v=4", messages_count: 0, visibility: :public, remote_path: "97b9043b-4613-4296-96e4-a8cf87b4df0f/index.m3u8", local_path: nil, deleted_at: nil, tags: ["typescript", "zig", "lua", "sol", "makin"], user_id: 1, user: #Ecto.Association.NotLoaded<association :user is not loaded>, show_id: nil, show: #Ecto.Association.NotLoaded<association :show is not loaded>, transmuxed_from_id: nil, transmuxed_from: #Ecto.Association.NotLoaded<association :transmuxed_from is not loaded>, content_metrics: #Ecto.Association.NotLoaded<association :content_metrics is not loaded>, messages: #Ecto.Association.NotLoaded<association :messages is not loaded>, appearances: #Ecto.Association.NotLoaded<association :appearances is not loaded>, product_reviews: #Ecto.Association.NotLoaded<association :product_reviews is not loaded>, inserted_at: ~N[2024-12-17 18:19:08], updated_at: ~N[2024-12-17 18:19:08]}, sequences: %{}, partials_in_ets: %{}, video_header: "", video_segment: "", setup_completed?: false, manifest_uploader: nil, partial_uploaders: %{}}, cleanup_after: nil}, component_type: :element, pid: #PID<0.866.0>, clock: nil, sync: :membrane_no_sync, spec_ref: #Reference<0.3109971441.2874146817.124000>, group: nil, initialized?: true, ready?: true, terminating?: true}, {:cmaf_muxer, "video_master"} => %Membrane.ChildEntry{name: {:cmaf_muxer, "video_master"}, module: Membrane.MP4.Muxer.CMAF, options: %Membrane.MP4.Muxer.CMAF{segment_min_duration: 6000000000, chunk_target_duration: 1000000000}, component_type: :element, pid: #PID<0.906.0>, clock: nil, sync: :membrane_no_sync, spec_ref: #Reference<0.3109971441.2874146819.80122>, group: nil, initialized?: true, ready?: true, terminating?: true}, {:parser, "video_master"} => %Membrane.ChildEntry{name: {:parser, "video_master"}, module: Membrane.H264.Parser, options: %Membrane.H264.Parser{spss: [], ppss: [], output_alignment: :au, skip_until_keyframe: true, repeat_parameter_sets: true, output_stream_structure: :avc1, generate_best_effort_timestamps: false}, component_type: :element, pid: #PID<0.903.0>, clock: nil, sync: :membrane_no_sync, spec_ref: #Reference<0.3109971441.2874146819.80122>, group: nil, initialized?: true, ready?: true, terminating?: true}}, clock: nil, playback: :playing, resource_guard: #PID<0.864.0>, utility_supervisor: #PID<0.861.0>, pads: %{{Membrane.Pad, :input, "video_master"} => %Membrane.Bin.PadData{ref: {Membrane.Pad, :input, "video_master"}, options: %{encoding: :H264, partial_segment_duration: 1000000000, segment_duration: 6000000000, track_name: "video_master", max_framerate: nil}, availability: :on_request, direction: :input, name: :input, link_id: #Reference<0.3109971441.2874146818.81320>, endpoint: %Membrane.Core.Parent.Link.Endpoint{child: {:parser, "video_master"}, pad_spec: :input, pad_ref: :input, pid: #PID<0.903.0>, pad_props: %{options: [], toilet_capacity: nil, target_queue_size: nil, min_demand_factor: nil, auto_demand_size: nil, throttling_factor: nil}, pad_info: %{name: :input, options: [], direction: :input, accepted_formats_str: ["RemoteStream", "H264"], availability: :always, flow_control: :auto}, child_spec_ref: #Reference<0.3109971441.2874146819.80122>}, linked?: true, response_received?: false, spec_ref: #Reference<0.3109971441.2874146819.80122>}}, parent_clock: #PID<0.830.0>, start_of_stream_received?: true}, %{original_module: Algora.Pipeline.SinkBin}
        because the bin is in the zombie mode
[debug] <0.827.0>/:sink/ Not calling the handle_element_end_of_stream callback with the following arguments:
        {:cmaf_muxer, "video_master"}, {Membrane.Pad, :input, #Reference<0.3109971441.2874146819.80156>}, %{name: :sink, children: %{:sink => %Membrane.ChildEntry{name: :sink, module: Algora.Pipeline.Sink, options: %Algora.Pipeline.Sink{manifest_config: %Algora.Pipeline.Sink.ManifestConfig{video_uuid: "97b9043b-4613-4296-96e4-a8cf87b4df0f", module: Algora.Pipeline.HLS, name: "index"}, track_config: %Algora.Pipeline.Sink.TrackConfig{target_window_duration: :infinity, mode: :live, header_naming_fun: &Membrane.HTTPAdaptiveStream.Manifest.Track.default_header_naming_fun/2, segment_naming_fun: &Membrane.HTTPAdaptiveStream.Manifest.Track.default_segment_naming_fun/1, partial_naming_fun: &Membrane.HTTPAdaptiveStream.Manifest.Track.default_partial_naming_fun/2, persist?: true}, storage: %Algora.Pipeline.Storage{directory: "/tmp/97b9043b-4613-4296-96e4-a8cf87b4df0f", video: %Algora.Library.Video{__meta__: #Ecto.Schema.Metadata<:loaded, "videos">, id: 149, duration: 0, title: "kaf-lamed-beyt's channel", description: nil, type: :livestream, format: :hls, corrupted: false, is_live: true, thumbnail_url: nil, vertical_thumbnail_url: nil, og_image_url: nil, url: "http://localhost:4000/hls/97b9043b-4613-4296-96e4-a8cf87b4df0f/index.m3u8", url_root: "http://localhost:4000/hls/97b9043b-4613-4296-96e4-a8cf87b4df0f", uuid: "97b9043b-4613-4296-96e4-a8cf87b4df0f", filename: "index.m3u8", channel_handle: "kaf-lamed-beyt", channel_name: "null (for now)", channel_avatar_url: "https://avatars.githubusercontent.com/u/48127194?v=4", messages_count: 0, visibility: :public, remote_path: "97b9043b-4613-4296-96e4-a8cf87b4df0f/index.m3u8", local_path: nil, deleted_at: nil, tags: ["typescript", "zig", "lua", "sol", "makin"], user_id: 1, user: #Ecto.Association.NotLoaded<association :user is not loaded>, show_id: nil, show: #Ecto.Association.NotLoaded<association :show is not loaded>, transmuxed_from_id: nil, transmuxed_from: #Ecto.Association.NotLoaded<association :transmuxed_from is not loaded>, content_metrics: #Ecto.Association.NotLoaded<association :content_metrics is not loaded>, messages: #Ecto.Association.NotLoaded<association :messages is not loaded>, appearances: #Ecto.Association.NotLoaded<association :appearances is not loaded>, product_reviews: #Ecto.Association.NotLoaded<association :product_reviews is not loaded>, inserted_at: ~N[2024-12-17 18:19:08], updated_at: ~N[2024-12-17 18:19:08]}, sequences: %{}, partials_in_ets: %{}, video_header: "", video_segment: "", setup_completed?: false, manifest_uploader: nil, partial_uploaders: %{}}, cleanup_after: nil}, component_type: :element, pid: #PID<0.866.0>, clock: nil, sync: :membrane_no_sync, spec_ref: #Reference<0.3109971441.2874146817.124000>, group: nil, initialized?: true, ready?: true, terminating?: true}, {:cmaf_muxer, "video_master"} => %Membrane.ChildEntry{name: {:cmaf_muxer, "video_master"}, module: Membrane.MP4.Muxer.CMAF, options: %Membrane.MP4.Muxer.CMAF{segment_min_duration: 6000000000, chunk_target_duration: 1000000000}, component_type: :element, pid: #PID<0.906.0>, clock: nil, sync: :membrane_no_sync, spec_ref: #Reference<0.3109971441.2874146819.80122>, group: nil, initialized?: true, ready?: true, terminating?: true}}, clock: nil, playback: :playing, resource_guard: #PID<0.864.0>, utility_supervisor: #PID<0.861.0>, pads: %{}, parent_clock: #PID<0.830.0>, start_of_stream_received?: true}, %{original_module: Algora.Pipeline.SinkBin}
        because the bin is in the zombie mode
[debug] <0.827.0>/{:src, 0}/ subprocess supervisor got exit request from parent, reason: :shutdown, shutting down children
[debug] <0.827.0>/ Not calling the handle_element_end_of_stream callback with the following arguments:
        :tee_audio, :input, %{children: %{:funnel_audio => %Membrane.ChildEntry{name: :funnel_audio, module: Algora.Pipeline.Funnel, options: %Algora.Pipeline.Funnel{end_of_stream: :notify}, component_type: :element, pid: #PID<0.856.0>, clock: nil, sync: :membrane_no_sync, spec_ref: #Reference<0.3109971441.2874146817.123689>, group: nil, initialized?: true, ready?: true, terminating?: true}, :tee_audio => %Membrane.ChildEntry{name: :tee_audio, module: Membrane.Tee.Parallel, options: nil, component_type: :element, pid: #PID<0.859.0>, clock: nil, sync: :membrane_no_sync, spec_ref: #Reference<0.3109971441.2874146817.123689>, group: nil, initialized?: true, ready?: true, terminating?: true}, :tee_video => %Membrane.ChildEntry{name: :tee_video, module: Membrane.Tee.Parallel, options: nil, component_type: :element, pid: #PID<0.850.0>, clock: nil, sync: :membrane_no_sync, spec_ref: #Reference<0.3109971441.2874146817.123689>, group: nil, initialized?: true, ready?: true, terminating?: true}, :sink => %Membrane.ChildEntry{name: :sink, module: Algora.Pipeline.SinkBin, options: %Algora.Pipeline.SinkBin{video_uuid: "97b9043b-4613-4296-96e4-a8cf87b4df0f", manifest_name: "index", manifest_module: Algora.Pipeline.HLS, storage: %Algora.Pipeline.Storage{directory: "/tmp/97b9043b-4613-4296-96e4-a8cf87b4df0f", video: %Algora.Library.Video{__meta__: #Ecto.Schema.Metadata<:loaded, "videos">, id: 149, duration: 0, title: "kaf-lamed-beyt's channel", description: nil, type: :livestream, format: :hls, corrupted: false, is_live: true, thumbnail_url: nil, vertical_thumbnail_url: nil, og_image_url: nil, url: "http://localhost:4000/hls/97b9043b-4613-4296-96e4-a8cf87b4df0f/index.m3u8", url_root: "http://localhost:4000/hls/97b9043b-4613-4296-96e4-a8cf87b4df0f", uuid: "97b9043b-4613-4296-96e4-a8cf87b4df0f", filename: "index.m3u8", channel_handle: "kaf-lamed-beyt", channel_name: "null (for now)", channel_avatar_url: "https://avatars.githubusercontent.com/u/48127194?v=4", messages_count: 0, visibility: :public, remote_path: "97b9043b-4613-4296-96e4-a8cf87b4df0f/index.m3u8", local_path: nil, deleted_at: nil, tags: ["typescript", "zig", "lua", "sol", "makin"], user_id: 1, user: #Ecto.Association.NotLoaded<association :user is not loaded>, show_id: nil, show: #Ecto.Association.NotLoaded<association :show is not loaded>, transmuxed_from_id: nil, transmuxed_from: #Ecto.Association.NotLoaded<association :transmuxed_from is not loaded>, content_metrics: #Ecto.Association.NotLoaded<association :content_metrics is not loaded>, messages: #Ecto.Association.NotLoaded<association :messages is not loaded>, appearances: #Ecto.Association.NotLoaded<association :appearances is not loaded>, product_reviews: #Ecto.Association.NotLoaded<association :product_reviews is not loaded>, inserted_at: ~N[2024-12-17 18:19:08], ...}, sequences: %{}, partials_in_ets: %{}, video_header: "", video_segment: "", setup_completed?: false, manifest_uploader: nil, partial_uploaders: %{}}, target_window_duration: :infinity, persist?: true, mode: :live, hls_mode: :separate_av, header_naming_fun: &Membrane.HTTPAdaptiveStream.Manifest.Track.default_header_naming_fun/2, segment_naming_fun: &Membrane.HTTPAdaptiveStream.Manifest.Track.default_segment_naming_fun/1, mp4_parameters_in_band?: false, cleanup_after: nil}, component_type: :bin, pid: #PID<0.862.0>, clock: nil, sync: :membrane_no_sync, spec_ref: #Reference<0.3109971441.2874146817.123689>, group: nil, initialized?: true, ready?: true, terminating?: true}, {:audio_reconnect, 0} => %Membrane.ChildEntry{name: {:audio_reconnect, 0}, module: Membrane.Tee.Parallel, options: nil, component_type: :element, pid: #PID<0.853.0>, clock: nil, sync: :membrane_no_sync, spec_ref: #Reference<0.3109971441.2874146817.123689>, group: nil, initialized?: true, ready?: true, terminating?: true}, {:src, 0} => %Membrane.ChildEntry{name: {:src, 0}, module: Algora.Pipeline.SourceBin, options: %Algora.Pipeline.SourceBin{client_ref: #PID<0.816.0>, url: nil}, component_type: :bin, pid: #PID<0.834.0>, clock: #PID<0.835.0>, sync: :membrane_no_sync, spec_ref: #Reference<0.3109971441.2874146817.123689>, group: nil, initialized?: true, ready?: true, terminating?: true}, {:video_reconnect, 0} => %Membrane.ChildEntry{name: {:video_reconnect, 0}, module: Membrane.Tee.Parallel, options: nil, component_type: :element, pid: #PID<0.844.0>, clock: nil, sync: :membrane_no_sync, spec_ref: #Reference<0.3109971441.2874146817.123689>, group: nil, initialized?: true, ready?: true, terminating?: true}}, clock: #PID<0.830.0>, playback: :playing, resource_guard: #PID<0.829.0>, utility_supervisor: #PID<0.826.0>, start_of_stream_received?: true}, %{original_module: Algora.Pipeline}
        because the pipeline is in the zombie mode
[debug] <0.827.0>/ Not calling the handle_element_end_of_stream callback with the following arguments:
        :tee_video, :input, %{children: %{:funnel_audio => %Membrane.ChildEntry{name: :funnel_audio, module: Algora.Pipeline.Funnel, options: %Algora.Pipeline.Funnel{end_of_stream: :notify}, component_type: :element, pid: #PID<0.856.0>, clock: nil, sync: :membrane_no_sync, spec_ref: #Reference<0.3109971441.2874146817.123689>, group: nil, initialized?: true, ready?: true, terminating?: true}, :tee_video => %Membrane.ChildEntry{name: :tee_video, module: Membrane.Tee.Parallel, options: nil, component_type: :element, pid: #PID<0.850.0>, clock: nil, sync: :membrane_no_sync, spec_ref: #Reference<0.3109971441.2874146817.123689>, group: nil, initialized?: true, ready?: true, terminating?: true}, :sink => %Membrane.ChildEntry{name: :sink, module: Algora.Pipeline.SinkBin, options: %Algora.Pipeline.SinkBin{video_uuid: "97b9043b-4613-4296-96e4-a8cf87b4df0f", manifest_name: "index", manifest_module: Algora.Pipeline.HLS, storage: %Algora.Pipeline.Storage{directory: "/tmp/97b9043b-4613-4296-96e4-a8cf87b4df0f", video: %Algora.Library.Video{__meta__: #Ecto.Schema.Metadata<:loaded, "videos">, id: 149, duration: 0, title: "kaf-lamed-beyt's channel", description: nil, type: :livestream, format: :hls, corrupted: false, is_live: true, thumbnail_url: nil, vertical_thumbnail_url: nil, og_image_url: nil, url: "http://localhost:4000/hls/97b9043b-4613-4296-96e4-a8cf87b4df0f/index.m3u8", url_root: "http://localhost:4000/hls/97b9043b-4613-4296-96e4-a8cf87b4df0f", uuid: "97b9043b-4613-4296-96e4-a8cf87b4df0f", filename: "index.m3u8", channel_handle: "kaf-lamed-beyt", channel_name: "null (for now)", channel_avatar_url: "https://avatars.githubusercontent.com/u/48127194?v=4", messages_count: 0, visibility: :public, remote_path: "97b9043b-4613-4296-96e4-a8cf87b4df0f/index.m3u8", local_path: nil, deleted_at: nil, tags: ["typescript", "zig", "lua", "sol", "makin"], user_id: 1, user: #Ecto.Association.NotLoaded<association :user is not loaded>, show_id: nil, show: #Ecto.Association.NotLoaded<association :show is not loaded>, transmuxed_from_id: nil, transmuxed_from: #Ecto.Association.NotLoaded<association :transmuxed_from is not loaded>, content_metrics: #Ecto.Association.NotLoaded<association :content_metrics is not loaded>, messages: #Ecto.Association.NotLoaded<association :messages is not loaded>, appearances: #Ecto.Association.NotLoaded<association :appearances is not loaded>, product_reviews: #Ecto.Association.NotLoaded<association :product_reviews is not loaded>, inserted_at: ~N[2024-12-17 18:19:08], updated_at: ~N[2024-12-17 18:19:08]}, sequences: %{}, partials_in_ets: %{}, video_header: "", video_segment: "", setup_completed?: false, manifest_uploader: nil, partial_uploaders: %{}}, target_window_duration: :infinity, persist?: true, mode: :live, hls_mode: :separate_av, header_naming_fun: &Membrane.HTTPAdaptiveStream.Manifest.Track.default_header_naming_fun/2, segment_naming_fun: &Membrane.HTTPAdaptiveStream.Manifest.Track.default_segment_naming_fun/1, mp4_parameters_in_band?: false, cleanup_after: nil}, component_type: :bin, pid: #PID<0.862.0>, clock: nil, sync: :membrane_no_sync, spec_ref: #Reference<0.3109971441.2874146817.123689>, group: nil, initialized?: true, ready?: true, terminating?: true}, {:src, 0} => %Membrane.ChildEntry{name: {:src, 0}, module: Algora.Pipeline.SourceBin, options: %Algora.Pipeline.SourceBin{client_ref: #PID<0.816.0>, url: nil}, component_type: :bin, pid: #PID<0.834.0>, clock: #PID<0.835.0>, sync: :membrane_no_sync, spec_ref: #Reference<0.3109971441.2874146817.123689>, group: nil, initialized?: true, ready?: true, terminating?: true}}, clock: #PID<0.830.0>, playback: :playing, resource_guard: #PID<0.829.0>, utility_supervisor: #PID<0.826.0>, start_of_stream_received?: true}, %{original_module: Algora.Pipeline}
        because the pipeline is in the zombie mode
[debug] <0.827.0>/:sink/ Ignoring unlinking pad {Membrane.Pad, :input, "audio_master"} that hasn't been successfully linked
[debug] <0.827.0>/:sink/ Ignoring unlinking pad {Membrane.Pad, :input, "video_master"} that hasn't been successfully linked
[error] GenServer #PID<0.866.0> terminating
** (RuntimeError) Cannot discontinue finished track
    (membrane_http_adaptive_stream_plugin 0.18.5) lib/membrane_http_adaptive_stream/manifest/track.ex:318: Membrane.HTTPAdaptiveStream.Manifest.Track.discontinue/1
    (algora 0.1.0) lib/algora/pipeline/sink.ex:282: Algora.Pipeline.Sink.handle_end_of_stream/3
    (membrane_core 1.0.1) lib/membrane/core/callback_handler.ex:139: Membrane.Core.CallbackHandler.exec_callback/4
    (membrane_core 1.0.1) lib/membrane/core/callback_handler.ex:69: Membrane.Core.CallbackHandler.exec_and_handle_callback/5
    (membrane_core 1.0.1) lib/membrane/core/element/event_controller.ex:117: Membrane.Core.Element.EventController.exec_handle_event/4
    (elixir 1.17.3) lib/enum.ex:2531: Enum."-reduce/3-lists^foldl/2-0-"/3
    (membrane_core 1.0.1) lib/membrane/core/element/lifecycle_controller.ex:103: Membrane.Core.Element.LifecycleController.handle_terminate_request/1
    (membrane_core 1.0.1) lib/membrane/core/element.ex:299: Membrane.Core.Element.handle_info/2
    (stdlib 6.1) gen_server.erl:2345: :gen_server.try_handle_info/3
    (stdlib 6.1) gen_server.erl:2433: :gen_server.handle_msg/6
    (stdlib 6.1) proc_lib.erl:329: :proc_lib.init_p_do_apply/3
Last message: {Membrane.Core.Message, :terminate, [], []}
State: %Membrane.Core.Element.State{module: Algora.Pipeline.Sink, name: :sink, parent_pid: #PID<0.862.0>, playback: :playing, type: :sink, internal_state: %{manifest: %Algora.Pipeline.Manifest{video_uuid: "97b9043b-4613-4296-96e4-a8cf87b4df0f", name: "index", module: Algora.Pipeline.HLS, tracks: %{"audio_master" => %Membrane.HTTPAdaptiveStream.Manifest.Track{id: "audio_master", mode: :live, encoding: %{mp4a: %{channels: 2, aot_id: 2, frequency: 48000}}, content_type: :audio, resolution: nil, max_framerate: nil, segment_duration: 6870000000, target_window_duration: :infinity, track_name: "audio_master", partial_segment_duration: 1000000000, partial_naming_fun: &Membrane.HTTPAdaptiveStream.Manifest.Track.default_partial_naming_fun/2, header_extension: ".mp4", segment_extension: ".m4s", header_naming_fun: &Membrane.HTTPAdaptiveStream.Manifest.Track.default_header_naming_fun/2, segment_naming_fun: &Membrane.HTTPAdaptiveStream.Manifest.Track.default_segment_naming_fun/1, persist?: true, header_name: "audio_header_audio_master_part_0.mp4", current_seq_num: 0, current_discontinuity_seq_num: 0, segments: #Qex<[%Membrane.HTTPAdaptiveStream.Manifest.Segment{name: "audio_segment_0_audio_master.m4s", duration: 6869000000, size: 143593, attributes: [creation_time: ~U[2024-12-17 18:19:10.969933Z]], type: :full, parts: [%{name: "audio_segment_0_audio_master_0_part.m4s", size: 20099, payload: nil, duration: 981000000, independent?: true}, %{name: "audio_segment_0_audio_master_1_part.m4s", size: 20573, payload: nil, duration: 981000000, independent?: true}, %{name: "audio_segment_0_audio_master_2_part.m4s", size: 20560, payload: nil, duration: 982000000, independent?: true}, %{name: "audio_segment_0_audio_master_3_part.m4s", size: 20609, payload: nil, duration: 981000000, independent?: true}, %{name: "audio_segment_0_audio_master_4_part.m4s", size: 20606, payload: nil, duration: 981000000, independent?: true}, %{name: "audio_segment_0_audio_master_5_part.m4s", size: 20587, payload: nil, duration: 982000000, independent?: true}, %{name: "audio_segment_0_audio_master_6_part.m4s", size: 20559, payload: nil, duration: 981000000, ...}]}, %Membrane.HTTPAdaptiveStream.Manifest.Segment{name: "audio_segment_1_audio_master.m4s", duration: 6869000000, size: 144323, attributes: [creation_time: ~U[2024-12-17 18:19:20.702083Z]], type: :full, parts: [%{name: "audio_segment_1_audio_master_0_part.m4s", size: 20593, payload: nil, duration: 981000000, independent?: true}, %{name: "audio_segment_1_audio_master_1_part.m4s", size: 20656, payload: nil, duration: 982000000, independent?: true}, %{name: "audio_segment_1_audio_master_2_part.m4s", size: 20631, payload: nil, duration: 981000000, independent?: true}, %{name: "audio_segment_1_audio_master_3_part.m4s", size: 20700, payload: nil, duration: 981000000, independent?: true}, %{name: "audio_segment_1_audio_master_4_part.m4s", size: 20539, payload: nil, duration: 982000000, independent?: true}, %{name: "audio_segment_1_audio_master_5_part.m4s", size: 20608, payload: nil, duration: 981000000, ...}, %{name: "audio_segment_1_audio_master_6_part.m4s", size: 20596, payload: nil, ...}]}, %Membrane.HTTPAdaptiveStream.Manifest.Segment{name: "audio_segment_2_audio_master.m4s", duration: 6870000000, size: 144154, attributes: [creation_time: ~U[2024-12-17 18:19:31.426836Z]], type: :full, parts: [%{name: "audio_segment_2_audio_master_0_part.m4s", size: 20626, payload: nil, duration: 982000000, independent?: true}, %{name: "audio_segment_2_audio_master_1_part.m4s", size: 20630, payload: nil, duration: 981000000, independent?: true}, %{name: "audio_segment_2_audio_master_2_part.m4s", size: 20564, payload: nil, duration: 981000000, independent?: true}, %{name: "audio_segment_2_audio_master_3_part.m4s", size: 20605, payload: nil, duration: 982000000, independent?: true}, %{name: "audio_segment_2_audio_master_4_part.m4s", size: 20588, payload: nil, duration: 981000000, ...}, %{name: "audio_segment_2_audio_master_5_part.m4s", size: 20562, payload: nil, ...}, %{name: "audio_segment_2_audio_master_6_part.m4s", size: 20579, ...}]}, %Membrane.HTTPAdaptiveStream.Manifest.Segment{name: "audio_segment_3_audio_master.m4s", duration: 6869000000, size: 144209, attributes: [creation_time: ~U[2024-12-17 18:19:44.456007Z]], type: :full, parts: [%{name: "audio_segment_3_audio_master_0_part.m4s", size: 20606, payload: nil, duration: 981000000, independent?: true}, %{name: "audio_segment_3_audio_master_1_part.m4s", size: 20661, payload: nil, duration: 981000000, independent?: true}, %{name: "audio_segment_3_audio_master_2_part.m4s", size: 20535, payload: nil, duration: 982000000, independent?: true}, %{name: "audio_segment_3_audio_master_3_part.m4s", size: 20598, payload: nil, duration: 981000000, ...}, %{name: "audio_segment_3_audio_master_4_part.m4s", size: 20603, payload: nil, ...}, %{name: "audio_segment_3_audio_master_5_part.m4s", size: 20637, ...}, %{name: "audio_segment_3_audio_master_6_part.m4s", ...}]}, %Membrane.HTTPAdaptiveStream.Manifest.Segment{name: "audio_segment_4_audio_master.m4s", duration: 6869000000, size: 144173, attributes: [creation_time: ~U[2024-12-17 18:19:54.256500Z]], type: :full, parts: [%{name: "audio_segment_4_audio_master_0_part.m4s", size: 20619, payload: nil, duration: 981000000, independent?: true}, %{name: "audio_segment_4_audio_master_1_part.m4s", size: 20551, payload: nil, duration: 982000000, independent?: true}, %{name: "audio_segment_4_audio_master_2_part.m4s", size: 20626, payload: nil, duration: 981000000, ...}, %{name: "audio_segment_4_audio_master_3_part.m4s", size: 20585, payload: nil, ...}, %{name: "audio_segment_4_audio_master_4_part.m4s", size: 20612, ...}, %{name: "audio_segment_4_audio_master_5_part.m4s", ...}, %{...}]}, %Membrane.HTTPAdaptiveStream.Manifest.Segment{name: "audio_segment_5_audio_master.m4s", duration: 6870000000, size: 144129, attributes: [creation_time: ~U[2024-12-17 18:20:09.394897Z]], type: :full, parts: [%{name: "audio_segment_5_audio_master_0_part.m4s", size: 20591, payload: nil, duration: 982000000, independent?: true}, %{name: "audio_segment_5_audio_master_1_part.m4s", size: 20568, payload: nil, duration: 981000000, ...}, %{name: "audio_segment_5_audio_master_2_part.m4s", size: 20584, payload: nil, ...}, %{name: "audio_segment_5_audio_master_3_part.m4s", size: 20645, ...}, %{name: "audio_segment_5_audio_master_4_part.m4s", ...}, %{...}, ...]}, %Membrane.HTTPAdaptiveStream.Manifest.Segment{name: "audio_segment_6_audio_master.m4s", duration: 6869000000, size: 144289, attributes: [creation_time: ~U[2024-12-17 18:20:21.629086Z]], type: :full, parts: [%{name: "audio_segment_6_audio_master_0_part.m4s", size: 20624, payload: nil, duration: 981000000, ...}, %{name: "audio_segment_6_audio_master_1_part.m4s", size: 20591, payload:  (truncated)
[debug] <0.827.0>/:sink/:sink subprocess supervisor got exit request from parent, reason: :shutdown, shutting down children
[debug] <0.827.0>/:sink/ Child :sink crashed but was not a member of any crash group.
Terminating.

[info] Elixir.Algora.Pipeline.Storage.Manifest terminating because of {%RuntimeError{message: "Cannot discontinue finished track"}, [{Membrane.HTTPAdaptiveStream.Manifest.Track, :discontinue, 1, [file: ~c"lib/membrane_http_adaptive_stream/manifest/track.ex", line: 318, error_info: %{module: Exception}]}, {Algora.Pipeline.Sink, :handle_end_of_stream, 3, [file: ~c"lib/algora/pipeline/sink.ex", line: 282]}, {Membrane.Core.CallbackHandler, :exec_callback, 4, [file: ~c"lib/membrane/core/callback_handler.ex", line: 139]}, {Membrane.Core.CallbackHandler, :exec_and_handle_callback, 5, [file: ~c"lib/membrane/core/callback_handler.ex", line: 69]}, {Membrane.Core.Element.EventController, :exec_handle_event, 4, [file: ~c"lib/membrane/core/element/event_controller.ex", line: 117]}, {Enum, :"-reduce/3-lists^foldl/2-0-", 3, [file: ~c"lib/enum.ex", line: 2531]}, {Membrane.Core.Element.LifecycleController, :handle_terminate_request, 1, [file: ~c"lib/membrane/core/element/lifecycle_controller.ex", line: 103]}, {Membrane.Core.Element, :handle_info, 2, [file: ~c"lib/membrane/core/element.ex", line: 299]}, {:gen_server, :try_handle_info, 3, [file: ~c"gen_server.erl", line: 2345]}, {:gen_server, :handle_msg, 6, [file: ~c"gen_server.erl", line: 2433]}, {:proc_lib, :init_p_do_apply, 3, [file: ~c"proc_lib.erl", line: 329]}]}
[error] <0.827.0>/:sink/ Terminating with reason: {:membrane_child_crash, :sink, {%RuntimeError{message: "Cannot discontinue finished track"}, [{Membrane.HTTPAdaptiveStream.Manifest.Track, :discontinue, 1, [file: ~c"lib/membrane_http_adaptive_stream/manifest/track.ex", line: 318, error_info: %{module: Exception}]}, {Algora.Pipeline.Sink, :handle_end_of_stream, 3, [file: ~c"lib/algora/pipeline/sink.ex", line: 282]}, {Membrane.Core.CallbackHandler, :exec_callback, 4, [file: ~c"lib/membrane/core/callback_handler.ex", line: 139]}, {Membrane.Core.CallbackHandler, :exec_and_handle_callback, 5, [file: ~c"lib/membrane/core/callback_handler.ex", line: 69]}, {Membrane.Core.Element.EventController, :exec_handle_event, 4, [file: ~c"lib/membrane/core/element/event_controller.ex", line: 117]}, {Enum, :"-reduce/3-lists^foldl/2-0-", 3, [file: ~c"lib/enum.ex", line: 2531]}, {Membrane.Core.Element.LifecycleController, :handle_terminate_request, 1, [file: ~c"lib/membrane/core/element/lifecycle_controller.ex", line: 103]}, {Membrane.Core.Element, :handle_info, 2, [file: ~c"lib/membrane/core/element.ex", line: 299]}, {:gen_server, :try_handle_info, 3, [file: ~c"gen_server.erl", line: 2345]}, {:gen_server, :handle_msg, 6, [file: ~c"gen_server.erl", line: 2433]}, {:proc_lib, :init_p_do_apply, 3, [file: ~c"proc_lib.erl", line: 329]}]}}
[debug] <0.827.0>/:sink/ subprocess supervisor got exit request from parent, reason: :shutdown, shutting down children
[debug] <0.827.0>/ Child :sink crashed but was not a member of any crash group.
Terminating.

[error] <0.827.0>/ Terminating with reason: {:membrane_child_crash, :sink, {:membrane_child_crash, :sink, {%RuntimeError{message: "Cannot discontinue finished track"}, [{Membrane.HTTPAdaptiveStream.Manifest.Track, :discontinue, 1, [file: ~c"lib/membrane_http_adaptive_stream/manifest/track.ex", line: 318, error_info: %{module: Exception}]}, {Algora.Pipeline.Sink, :handle_end_of_stream, 3, [file: ~c"lib/algora/pipeline/sink.ex", line: 282]}, {Membrane.Core.CallbackHandler, :exec_callback, 4, [file: ~c"lib/membrane/core/callback_handler.ex", line: 139]}, {Membrane.Core.CallbackHandler, :exec_and_handle_callback, 5, [file: ~c"lib/membrane/core/callback_handler.ex", line: 69]}, {Membrane.Core.Element.EventController, :exec_handle_event, 4, [file: ~c"lib/membrane/core/element/event_controller.ex", line: 117]}, {Enum, :"-reduce/3-lists^foldl/2-0-", 3, [file: ~c"lib/enum.ex", line: 2531]}, {Membrane.Core.Element.LifecycleController, :handle_terminate_request, 1, [file: ~c"lib/membrane/core/element/lifecycle_controller.ex", line: 103]}, {Membrane.Core.Element, :handle_info, 2, [file: ~c"lib/membrane/core/element.ex", line: 299]}, {:gen_server, :try_handle_info, 3, [file: ~c"gen_server.erl", line: 2345]}, {:gen_server, :handle_msg, 6, [file: ~c"gen_server.erl", line: 2433]}, {:proc_lib, :init_p_do_apply, 3, [file: ~c"proc_lib.erl", line: 329]}]}}}
[debug] <0.827.0>/ Pipeline supervisor got exit from pipeline with reason {:membrane_child_crash, :sink, {:membrane_child_crash, :sink, {%RuntimeError{message: "Cannot discontinue finished track"}, [{Membrane.HTTPAdaptiveStream.Manifest.Track, :discontinue, 1, [file: ~c"lib/membrane_http_adaptive_stream/manifest/track.ex", line: 318, error_info: %{module: Exception}]}, {Algora.Pipeline.Sink, :handle_end_of_stream, 3, [file: ~c"lib/algora/pipeline/sink.ex", line: 282]}, {Membrane.Core.CallbackHandler, :exec_callback, 4, [file: ~c"lib/membrane/core/callback_handler.ex", line: 139]}, {Membrane.Core.CallbackHandler, :exec_and_handle_callback, 5, [file: ~c"lib/membrane/core/callback_handler.ex", line: 69]}, {Membrane.Core.Element.EventController, :exec_handle_event, 4, [file: ~c"lib/membrane/core/element/event_controller.ex", line: 117]}, {Enum, :"-reduce/3-lists^foldl/2-0-", 3, [file: ~c"lib/enum.ex", line: 2531]}, {Membrane.Core.Element.LifecycleController, :handle_terminate_request, 1, [file: ~c"lib/membrane/core/element/lifecycle_controller.ex", line: 103]}, {Membrane.Core.Element, :handle_info, 2, [file: ~c"lib/membrane/core/element.ex", line: 299]}, {:gen_server, :try_handle_info, 3, [file: ~c"gen_server.erl", line: 2345]}, {:gen_server, :handle_msg, 6, [file: ~c"gen_server.erl", line: 2433]}, {:proc_lib, :init_p_do_apply, 3, [file: ~c"proc_lib.erl", line: 329]}]}}}, stopping subprocess supervisor
[debug] <0.827.0>/ subprocess supervisor got exit request from parent, reason: :shutdown, shutting down children
[debug] <0.827.0>/ Pipeline supervisor got exit from subprocess supervisor, exiting
[error] GenServer #PID<0.824.0> terminating
** (stop) {:membrane_child_crash, :sink, {:membrane_child_crash, :sink, {%RuntimeError{message: "Cannot discontinue finished track"}, [{Membrane.HTTPAdaptiveStream.Manifest.Track, :discontinue, 1, [file: ~c"lib/membrane_http_adaptive_stream/manifest/track.ex", line: 318, error_info: %{module: Exception}]}, {Algora.Pipeline.Sink, :handle_end_of_stream, 3, [file: ~c"lib/algora/pipeline/sink.ex", line: 282]}, {Membrane.Core.CallbackHandler, :exec_callback, 4, [file: ~c"lib/membrane/core/callback_handler.ex", line: 139]}, {Membrane.Core.CallbackHandler, :exec_and_handle_callback, 5, [file: ~c"lib/membrane/core/callback_handler.ex", line: 69]}, {Membrane.Core.Element.EventController, :exec_handle_event, 4, [file: ~c"lib/membrane/core/element/event_controller.ex", line: 117]}, {Enum, :"-reduce/3-lists^foldl/2-0-", 3, [file: ~c"lib/enum.ex", line: 2531]}, {Membrane.Core.Element.LifecycleController, :handle_terminate_request, 1, [file: ~c"lib/membrane/core/element/lifecycle_controller.ex", line: 103]}, {Membrane.Core.Element, :handle_info, 2, [file: ~c"lib/membrane/core/element.ex", line: 299]}, {:gen_server, :try_handle_info, 3, [file: ~c"gen_server.erl", line: 2345]}, {:gen_server, :handle_msg, 6, [file: ~c"gen_server.erl", line: 2433]}, {:proc_lib, :init_p_do_apply, 3, [file: ~c"proc_lib.erl", line: 329]}]}}}
Last message: {:EXIT, #PID<0.825.0>, {:membrane_child_crash, :sink, {:membrane_child_crash, :sink, {%RuntimeError{message: "Cannot discontinue finished track"}, [{Membrane.HTTPAdaptiveStream.Manifest.Track, :discontinue, 1, [file: ~c"lib/membrane_http_adaptive_stream/manifest/track.ex", line: 318, error_info: %{module: Exception}]}, {Algora.Pipeline.Sink, :handle_end_of_stream, 3, [file: ~c"lib/algora/pipeline/sink.ex", line: 282]}, {Membrane.Core.CallbackHandler, :exec_callback, 4, [file: ~c"lib/membrane/core/callback_handler.ex", line: 139]}, {Membrane.Core.CallbackHandler, :exec_and_handle_callback, 5, [file: ~c"lib/membrane/core/callback_handler.ex", line: 69]}, {Membrane.Core.Element.EventController, :exec_handle_event, 4, [file: ~c"lib/membrane/core/element/event_controller.ex", line: 117]}, {Enum, :"-reduce/3-lists^foldl/2-0-", 3, [file: ~c"lib/enum.ex", line: 2531]}, {Membrane.Core.Element.LifecycleController, :handle_terminate_request, 1, [file: ~c"lib/membrane/core/element/lifecycle_controller.ex", line: 103]}, {Membrane.Core.Element, :handle_info, 2, [file: ~c"lib/membrane/core/element.ex", line: 299]}, {:gen_server, :try_handle_info, 3, [file: ~c"gen_server.erl", line: 2345]}, {:gen_server, :handle_msg, 6, [file: ~c"gen_server.erl", line: 2433]}, {:proc_lib, :init_p_do_apply, 3, [file: ~c"proc_lib.erl", line: 329]}]}}}}
State: %{pid: #PID<0.827.0>, params: %{app: "live", stream_key: "1iPRahtVUBV7sknB2d_ytAZIqw9mPbHcuz6Ltq_-H1g", client_ref: #PID<0.816.0>, video_uuid: nil}}
[error] Task #PID<0.1858.0> started from #PID<0.1857.0> terminating
** (MatchError) no match of right hand side value: {:error, :eaddrinuse}
    (membrane_rtmp_plugin 0.27.3) lib/membrane_rtmp_plugin/rtmp_server/listener.ex:41: Membrane.RTMPServer.Listener.run/1
    (elixir 1.17.3) lib/task/supervised.ex:101: Task.Supervised.invoke_mfa/2
Function: &Membrane.RTMPServer.Listener.run/1
    Args: [%{name: nil, port: 9006, server: #PID<0.1857.0>, listen_options: [:binary, {:packet, :raw}, {:active, false}, {:ip, {0, 0, 0, 0}}], handle_new_client: &Algora.Pipeline.Manager.handle_new_client/3, use_ssl?: false, client_timeout: 5000000000}]
[debug] FLAME.Terminator.system_stop: idle shutdown
[error] Task #PID<0.1818.0> started from #PID<0.866.0> terminating
** (Protocol.UndefinedError) protocol String.Chars not implemented for {894744, {:error, {:http_error, 403, %{body: "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>AccessDenied</Code><Message>Access Denied.</Message><Resource>/algora-tv-test/97b9043b-4613-4296-96e4-a8cf87b4df0f/video_segment_5_video_master.m4s</Resource><RequestId>1734459634667870533</RequestId><Key>97b9043b-4613-4296-96e4-a8cf87b4df0f/video_segment_5_video_master.m4s</Key><BucketName>algora-tv-test</BucketName></Error>", headers: [{"Access-Control-Allow-Headers", "*"}, {"Access-Control-Allow-Methods", "*"}, {"Access-Control-Allow-Origin", "*"}, {"Access-Control-Expose-Headers", "*"}, {"Content-Length", "379"}, {"Content-Type", "application/xml"}, {"Server", "Tigris OS"}, {"Server-Timing", "total;dur=0"}, {"Strict-Transport-Security", "max-age=63072000; includeSubDomains; preload"}, {"X-Amz-Request-Id", "1734459634667870533"}, {"Date", "Tue, 17 Dec 2024 18:20:34 GMT"}, {"Connection", "close"}], status_code: 403}}}} of type Tuple. This protocol is implemented for the following type(s): Atom, BitString, Complex, Date, DateTime, Decimal, Float, Image.Exif.Gps, Image.Exif.Thumbnail, Integer, List, NaiveDateTime, OpenApiSpex.Cast.Error, Phoenix.LiveComponent.CID, Postgrex.Copy, Postgrex.INET, Postgrex.MACADDR, Postgrex.Query, Time, URI, Version, Version.Requirement
    (elixir 1.17.3) lib/string/chars.ex:3: String.Chars.impl_for!/1
    (elixir 1.17.3) lib/string/chars.ex:22: String.Chars.to_string/1
    (algora 0.1.0) lib/algora/pipeline/storage.ex:310: anonymous fn/2 in Algora.Pipeline.Storage.store_content/6
    (elixir 1.17.3) lib/task/supervised.ex:101: Task.Supervised.invoke_mfa/2
Function: #Function<5.16005492/0 in Algora.Pipeline.Storage.store_content/6>
    Args: []
[error] Failed to upload 97b9043b-4613-4296-96e4-a8cf87b4df0f/audio_master.m3u8: {:error, {:http_error, 403, %{body: "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>AccessDenied</Code><Message>Access Denied.</Message><Resource>/algora-tv-test/97b9043b-4613-4296-96e4-a8cf87b4df0f/audio_master.m3u8</Resource><RequestId>1734459635719820351</RequestId><Key>97b9043b-4613-4296-96e4-a8cf87b4df0f/audio_master.m3u8</Key><BucketName>algora-tv-test</BucketName></Error>", headers: [{"Access-Control-Allow-Headers", "*"}, {"Access-Control-Allow-Methods", "*"}, {"Access-Control-Allow-Origin", "*"}, {"Access-Control-Expose-Headers", "*"}, {"Content-Length", "349"}, {"Content-Type", "application/xml"}, {"Server", "Tigris OS"}, {"Server-Timing", "total;dur=0"}, {"Strict-Transport-Security", "max-age=63072000; includeSubDomains; preload"}, {"X-Amz-Request-Id", "1734459635719820351"}, {"Date", "Tue, 17 Dec 2024 18:20:35 GMT"}], status_code: 403}}}
[error] Task #PID<0.1805.0> started from #PID<0.866.0> terminating
** (Protocol.UndefinedError) protocol String.Chars not implemented for {2134272, {:error, {:http_error, 403, %{body: "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>AccessDenied</Code><Message>Access Denied.</Message><Resource>/algora-tv-test/97b9043b-4613-4296-96e4-a8cf87b4df0f/audio_segment_6_audio_master.m4s</Resource><RequestId>1734459635062321122</RequestId><Key>97b9043b-4613-4296-96e4-a8cf87b4df0f/audio_segment_6_audio_master.m4s</Key><BucketName>algora-tv-test</BucketName></Error>", headers: [{"Access-Control-Allow-Headers", "*"}, {"Access-Control-Allow-Methods", "*"}, {"Access-Control-Allow-Origin", "*"}, {"Access-Control-Expose-Headers", "*"}, {"Content-Length", "379"}, {"Content-Type", "application/xml"}, {"Server", "Tigris OS"}, {"Server-Timing", "total;dur=0"}, {"Strict-Transport-Security", "max-age=63072000; includeSubDomains; preload"}, {"X-Amz-Request-Id", "1734459635062321122"}, {"Date", "Tue, 17 Dec 2024 18:20:35 GMT"}], status_code: 403}}}} of type Tuple. This protocol is implemented for the following type(s): Atom, BitString, Complex, Date, DateTime, Decimal, Float, Image.Exif.Gps, Image.Exif.Thumbnail, Integer, List, NaiveDateTime, OpenApiSpex.Cast.Error, Phoenix.LiveComponent.CID, Postgrex.Copy, Postgrex.INET, Postgrex.MACADDR, Postgrex.Query, Time, URI, Version, Version.Requirement
    (elixir 1.17.3) lib/string/chars.ex:3: String.Chars.impl_for!/1
    (elixir 1.17.3) lib/string/chars.ex:22: String.Chars.to_string/1
    (algora 0.1.0) lib/algora/pipeline/storage.ex:310: anonymous fn/2 in Algora.Pipeline.Storage.store_content/6
    (elixir 1.17.3) lib/task/supervised.ex:101: Task.Supervised.invoke_mfa/2
Function: #Function<5.16005492/0 in Algora.Pipeline.Storage.store_content/6>
    Args: []
[error] Failed to upload 97b9043b-4613-4296-96e4-a8cf87b4df0f/index.m3u8: {:error, {:http_error, 403, %{body: "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>AccessDenied</Code><Message>Access Denied.</Message><Resource>/algora-tv-test/97b9043b-4613-4296-96e4-a8cf87b4df0f/index.m3u8</Resource><RequestId>1734459636048204865</RequestId><Key>97b9043b-4613-4296-96e4-a8cf87b4df0f/index.m3u8</Key><BucketName>algora-tv-test</BucketName></Error>", headers: [{"Access-Control-Allow-Headers", "*"}, {"Access-Control-Allow-Methods", "*"}, {"Access-Control-Allow-Origin", "*"}, {"Access-Control-Expose-Headers", "*"}, {"Content-Length", "335"}, {"Content-Type", "application/xml"}, {"Server", "Tigris OS"}, {"Server-Timing", "total;dur=0"}, {"Strict-Transport-Security", "max-age=63072000; includeSubDomains; preload"}, {"X-Amz-Request-Id", "1734459636048204865"}, {"Date", "Tue, 17 Dec 2024 18:20:36 GMT"}], status_code: 403}}}
[error] Failed to upload 97b9043b-4613-4296-96e4-a8cf87b4df0f/video_master.m3u8: {:error, {:http_error, 403, %{body: "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>AccessDenied</Code><Message>Access Denied.</Message><Resource>/algora-tv-test/97b9043b-4613-4296-96e4-a8cf87b4df0f/video_master.m3u8</Resource><RequestId>1734459636454218634</RequestId><Key>97b9043b-4613-4296-96e4-a8cf87b4df0f/video_master.m3u8</Key><BucketName>algora-tv-test</BucketName></Error>", headers: [{"Access-Control-Allow-Headers", "*"}, {"Access-Control-Allow-Methods", "*"}, {"Access-Control-Allow-Origin", "*"}, {"Access-Control-Expose-Headers", "*"}, {"Content-Length", "349"}, {"Content-Type", "application/xml"}, {"Server", "Tigris OS"}, {"Server-Timing", "total;dur=0"}, {"Strict-Transport-Security", "max-age=63072000; includeSubDomains; preload"}, {"X-Amz-Request-Id", "1734459636454218634"}, {"Date", "Tue, 17 Dec 2024 18:20:36 GMT"}], status_code: 403}}}
[error] GenServer #PID<0.868.0> terminating
** (RuntimeError) Cannot discontinue finished track
    (membrane_http_adaptive_stream_plugin 0.18.5) lib/membrane_http_adaptive_stream/manifest/track.ex:318: Membrane.HTTPAdaptiveStream.Manifest.Track.discontinue/1
    (algora 0.1.0) lib/algora/pipeline/sink.ex:282: Algora.Pipeline.Sink.handle_end_of_stream/3
    (membrane_core 1.0.1) lib/membrane/core/callback_handler.ex:139: Membrane.Core.CallbackHandler.exec_callback/4
    (membrane_core 1.0.1) lib/membrane/core/callback_handler.ex:69: Membrane.Core.CallbackHandler.exec_and_handle_callback/5
    (membrane_core 1.0.1) lib/membrane/core/element/event_controller.ex:117: Membrane.Core.Element.EventController.exec_handle_event/4
    (elixir 1.17.3) lib/enum.ex:2531: Enum."-reduce/3-lists^foldl/2-0-"/3
    (membrane_core 1.0.1) lib/membrane/core/element/lifecycle_controller.ex:103: Membrane.Core.Element.LifecycleController.handle_terminate_request/1
    (membrane_core 1.0.1) lib/membrane/core/element.ex:299: Membrane.Core.Element.handle_info/2
    (stdlib 6.1) gen_server.erl:2345: :gen_server.try_handle_info/3
    (stdlib 6.1) gen_server.erl:2433: :gen_server.handle_msg/6
    (stdlib 6.1) proc_lib.erl:329: :proc_lib.init_p_do_apply/3
Last message: {:EXIT, #PID<0.866.0>, {%RuntimeError{message: "Cannot discontinue finished track"}, [{Membrane.HTTPAdaptiveStream.Manifest.Track, :discontinue, 1, [file: ~c"lib/membrane_http_adaptive_stream/manifest/track.ex", line: 318, error_info: %{module: Exception}]}, {Algora.Pipeline.Sink, :handle_end_of_stream, 3, [file: ~c"lib/algora/pipeline/sink.ex", line: 282]}, {Membrane.Core.CallbackHandler, :exec_callback, 4, [file: ~c"lib/membrane/core/callback_handler.ex", line: 139]}, {Membrane.Core.CallbackHandler, :exec_and_handle_callback, 5, [file: ~c"lib/membrane/core/callback_handler.ex", line: 69]}, {Membrane.Core.Element.EventController, :exec_handle_event, 4, [file: ~c"lib/membrane/core/element/event_controller.ex", line: 117]}, {Enum, :"-reduce/3-lists^foldl/2-0-", 3, [file: ~c"lib/enum.ex", line: 2531]}, {Membrane.Core.Element.LifecycleController, :handle_terminate_request, 1, [file: ~c"lib/membrane/core/element/lifecycle_controller.ex", line: 103]}, {Membrane.Core.Element, :handle_info, 2, [file: ~c"lib/membrane/core/element.ex", line: 299]}, {:gen_server, :try_handle_info, 3, [file: ~c"gen_server.erl", line: 2345]}, {:gen_server, :handle_msg, 6, [file: ~c"gen_server.erl", line: 2433]}, {:proc_lib, :init_p_do_apply, 3, [file: ~c"proc_lib.erl", line: 329]}]}}
State: %{manifests: %{"audio_master.m3u8" => {{:send_local, #Reference<0.3109971441.2874146819.89759>}, {"#EXTM3U\n#EXT-X-VERSION:7\n#EXT-X-TARGETDURATION:7\n#EXT-X-SERVER-CONTROL:CAN-BLOCK-RELOAD=YES,PART-HOLD-BACK=3.0,CAN-SKIP-UNTIL=27.477\n#EXT-X-PART-INF:PART-TARGET=1.0\n#EXT-X-MEDIA-SEQUENCE:0\n#EXT-X-DISCONTINUITY-SEQUENCE:0\n#EXT-X-MAP:URI=\"https://fly.storage.tigris.dev/algora-tv-test/97b9043b-4613-4296-96e4-a8cf87b4df0f/audio_header_audio_master_part_0.mp4\"\n#EXT-X-PROGRAM-DATE-TIME:2024-12-17T18:19:10.969Z\n#EXTINF:6.869,\nhttps://fly.storage.tigris.dev/algora-tv-test/97b9043b-4613-4296-96e4-a8cf87b4df0f/audio_segment_0_audio_master.m4s\n#EXT-X-PROGRAM-DATE-TIME:2024-12-17T18:19:20.702Z\n#EXTINF:6.869,\nhttps://fly.storage.tigris.dev/algora-tv-test/97b9043b-4613-4296-96e4-a8cf87b4df0f/audio_segment_1_audio_master.m4s\n#EXT-X-PROGRAM-DATE-TIME:2024-12-17T18:19:31.426Z\n#EXTINF:6.87,\nhttps://fly.storage.tigris.dev/algora-tv-test/97b9043b-4613-4296-96e4-a8cf87b4df0f/audio_segment_2_audio_master.m4s\n#EXT-X-PART:DURATION=0.981,URI=\"audio_segment_3_audio_master_0_part.m4s\",INDEPENDENT=YES\n#EXT-X-PART:DURATION=0.981,URI=\"audio_segment_3_audio_master_1_part.m4s\",INDEPENDENT=YES\n#EXT-X-PART:DURATION=0.982,URI=\"audio_segment_3_audio_master_2_part.m4s\",INDEPENDENT=YES\n#EXT-X-PART:DURATION=0.981,URI=\"audio_segment_3_audio_master_3_part.m4s\",INDEPENDENT=YES\n#EXT-X-PART:DURATION=0.981,URI=\"audio_segment_3_audio_master_4_part.m4s\",INDEPENDENT=YES\n#EXT-X-PART:DURATION=0.982,URI=\"audio_segment_3_audio_master_5_part.m4s\",INDEPENDENT=YES\n#EXT-X-PART:DURATION=0.981,URI=\"audio_segment_3_audio_master_6_part.m4s\",INDEPENDENT=YES\n#EXT-X-PROGRAM-DATE-TIME:2024-12-17T18:19:44.456Z\n#EXTINF:6.869,\nhttps://fly.storage.tigris.dev/algora-tv-test/97b9043b-4613-4296-96e4-a8cf87b4df0f/audio_segment_3_audio_master.m4s\n#EXT-X-PART:DURATION=0.981,URI=\"audio_segment_4_audio_master_0_part.m4s\",INDEPENDENT=YES\n#EXT-X-PART:DURATION=0.982,URI=\"audio_segment_4_audio_master_1_part.m4s\",INDEPENDENT=YES\n#EXT-X-PART:DURATION=0.981,URI=\"audio_segment_4_audio_master_2_part.m4s\",INDEPENDENT=YES\n#EXT-X-PART:DURATION=0.981,URI=\"audio_segment_4_audio_master_3_part.m4s\",INDEPENDENT=YES\n#EXT-X-PART:DURATION=0.982,URI=\"audio_segment_4_audio_master_4_part.m4s\",INDEPENDENT=YES\n#EXT-X-PART:DURATION=0.981,URI=\"audio_segment_4_audio_master_5_part.m4s\",INDEPENDENT=YES\n#EXT-X-PART:DURATION=0.981,URI=\"audio_segment_4_audio_master_6_part.m4s\",INDEPENDENT=YES\n#EXT-X-PROGRAM-DATE-TIME:2024-12-17T18:19:54.256Z\n#EXTINF:6.869,\nhttps://fly.storage.tigris.dev/algora-tv-test/97b9043b-4613-4296-96e4-a8cf87b4df0f/audio_segment_4_audio_master.m4s\n#EXT-X-PART:DURATION=0.982,URI=\"audio_segment_5_audio_master_0_part.m4s\",INDEPENDENT=YES\n#EXT-X-PART:DURATION=0.981,URI=\"audio_segment_5_audio_master_1_part.m4s\",INDEPENDENT=YES\n#EXT-X-PART:DURATION=0.981,URI=\"audio_segment_5_audio_master_2_part.m4s\",INDEPENDENT=YES\n#EXT-X-PART:DURATION=0.982,URI=\"audio_segment_5_audio_master_3_part.m4s\",INDEPENDENT=YES\n#EXT-X-PART:DURATION=0.981,URI=\"audio_segment_5_audio_master_4_part.m4s\",INDEPENDENT=YES\n#EXT-X-PART:DURATION=0.981,URI=\"audio_segment_5_audio_master_5_part.m4s\",INDEPENDENT=YES\n#EXT-X-PART:DURATION=0.982,URI=\"audio_segment_5_audio_master_6_part.m4s\",INDEPENDENT=YES\n#EXT-X-PROGRAM-DATE-TIME:2024-12-17T18:20:09.394Z\n#EXTINF:6.87,\nhttps://fly.storage.tigris.dev/algora-tv-test/97b9043b-4613-4296-96e4-a8cf87b4df0f/audio_segment_5_audio_master.m4s\n#EXT-X-PART:DURATION=0.981,URI=\"audio_segment_6_audio_master_0_part.m4s\",INDEPENDENT=YES\n#EXT-X-PART:DURATION=0.981,URI=\"audio_segment_6_audio_master_1_part.m4s\",INDEPENDENT=YES\n#EXT-X-PART:DURATION=0.982,URI=\"audio_segment_6_audio_master_2_part.m4s\",INDEPENDENT=YES\n#EXT-X-PART:DURATION=0.981,URI=\"audio_segment_6_audio_master_3_part.m4s\",INDEPENDENT=YES\n#EXT-X-PART:DURATION=0.981,URI=\"audio_segment_6_audio_master_4_part.m4s\",INDEPENDENT=YES\n#EXT-X-PART:DURATION=0.982,URI=\"audio_segment_6_audio_master_5_part.m4s\",INDEPENDENT=YES\n#EXT-X-PART:DURATION=0.981,URI=\"audio_segment_6_audio_master_6_part.m4s\",INDEPENDENT=YES\n#EXT-X-PROGRAM-DATE-TIME:2024-12-17T18:20:21.629Z\n#EXTINF:6.869,\nhttps://fly.storage.tigris.dev/algora-tv-test/97b9043b-4613-4296-96e4-a8cf87b4df0f/audio_segment_" <> ..., [content_type: "application/x-mpegURL", cache_control: "no-cache, no-store, private"]}}, "index.m3u8" => {{:send_local, #Reference<0.3109971441.2874146819.89758>}, {"#EXTM3U\n#EXT-X-VERSION:7\n#EXT-X-INDEPENDENT-SEGMENTS\n#EXT-X-MEDIA:TYPE=AUDIO,NAME=\"audio_default_name\",GROUP-ID=\"audio_default_id\",AUTOSELECT=YES,DEFAULT=YES,URI=\"audio_master.m3u8\",CODECS=\"mp4a.40.2\"\n#EXT-X-STREAM-INF:BANDWIDTH=2519765,AVERAGE-BANDWIDTH=2501029,RESOLUTION=1280x720,CODECS=\"avc1.64001f\",AUDIO=\"audio_default_id\"\nvideo_master.m3u8", [content_type: "application/x-mpegURL", cache_control: "no-cache, no-store, private"]}}, "video_master.m3u8" => {{:send_local, #Reference<0.3109971441.2874146819.89760>}, {"#EXTM3U\n#EXT-X-VERSION:7\n#EXT-X-TARGETDURATION:12\n#EXT-X-SERVER-CONTROL:CAN-BLOCK-RELOAD=YES,PART-HOLD-BACK=3.0,CAN-SKIP-UNTIL=36.0\n#EXT-X-PART-INF:PART-TARGET=1.0\n#EXT-X-MEDIA-SEQUENCE:0\n#EXT-X-DISCONTINUITY-SEQUENCE:0\n#EXT-X-MAP:URI=\"https://fly.storage.tigris.dev/algora-tv-test/97b9043b-4613-4296-96e4-a8cf87b4df0f/video_header_video_master_part_0.mp4\"\n#EXT-X-PROGRAM-DATE-TIME:2024-12-17T18:19:12.310Z\n#EXTINF:11.966,\nhttps://fly.storage.tigris.dev/algora-tv-test/97b9043b-4613-4296-96e4-a8cf87b4df0f/video_segment_0_vide (truncated)

This is my first attempt at streaming and this is an excerpt of error (I can't paste everything because of GitHub's limit), as opposed to trying it multiple times like you suggested. Is this the same error?

[kaf-lamed-beyt]

Because I keep forgetting the commands in the setup command, I tried adding aliases so it is easier for new contributors to use.

I opened a new PR (#132) for it, perhaps you could take a look and let me know what you think.

[zcesur]

Are you using a private bucket? Looks like a permission error

Just a heads up - this particular issue is pretty deep in the video processing pipeline 😁

[kaf-lamed-beyt]

Oh! I'm not so sure. But, the last time I did a bucket setup was with tigris — I think it was with the guide you shared with me a while ago.

And thanks for letting me know the scale of the issue 😅.

I had already gone to read about transmuxing. From the issue description, I think it's a bit related, no?

[lastcanal]

@kaf-lamed-beyt your S3 bucket is not properly configured. Feel free to reach out on Discord if you have questions about getting setup.

@zcesur I added crash_group_mode: :transient to the group with the crashing muxer in SinkBin and that gave me this very helpful error message:

[error] <0.785.0>/ Error occured in Membrane Pipeline:
** (Membrane.CallbackError) Bin :sink removed its pad {Membrane.Pad, :input, "video_master"}, but callback `c:Membrane.Pipeline.handle_child_pad_removed/4` is not implemented in Algora.Pipeline.

This means, that `:sink removed the pad on its own, without knowledge of its parent. For example it could happen due to removing :sink's child linked to the :sink's inner pad or by removing link between :sink and its child.

If you want to handle this scenario, implement `c:Membrane.Pipeline.handle_child_pad_removed/4` callback in Algora.Pipeline.

Might be possible to recover by adding Algora.Pipeline.handle_child_pad_removed/4, which will allow us to re-initialize the HLS tracks, the HLS component itself, or at least do an orderly shutdown of the pipeline.

[kaf-lamed-beyt]

Aha! I see. Just sent you a DM now