Compositing

RevueSDK has no server to mix a call for you, so the device does it. Two tools combine several tracks into one:

• AudioCompositor mixes several audio tracks into one audio track.
• VideoCompositor draws several video tracks into one picture, laid out by a composition template (CompositeTemplate).

Neither belongs to a room or a participant. You create them, give them their tracks, and use their output: publish it to friends, or record it.

Threads. Do not call either compositor from more than one thread at a time. Call them on the main thread. AudioCompositor.isRunning, VideoCompositor.isRunning and VideoCompositor.currentTemplate send their change notifications on the thread that made the call.

Mixing audio

Start the mixer with AudioCompositor.start(), then add tracks with AudioCompositor.addTrack(_:) and take them out with AudioCompositor.removeTrack(_:). The mix comes out of AudioCompositor.outputTrack.

Start before you add tracks. Always call AudioCompositor.start() first. A track added before AudioCompositor.start() is never mixed, and adding the same track again afterwards does not help.

Restarting. AudioCompositor.stop() releases the mixer and drops every track. After AudioCompositor.start(), add the tracks again; AudioCompositor.outputTrack is then a new track, see Restarting a compositor.

Silence. Set AudioCompositor.autoOutputSilentBuffer to true to keep the output going with silence while no track is sending. Turn it on when you record the mix, so the sound in the file has no gaps.

Example: a mix for each guest

In a meeting where you are the host and send everyone's audio, each guest should hear everybody except themselves. Create one AudioCompositor for each guest. Each one mixes your microphone and every other guest, leaves that guest out, and is published only to that guest:

Mixer	Mixes	Published to
Guest A's	You, B, C	A
Guest B's	You, A, C	B
Guest C's	You, A, B	C

When a guest joins, add their voice to every other guest's mixer and create a mixer for them. When a guest leaves, stop their mixer and remove their voice from the others.

Swift

import RevueSDK

/// MCU host: one AudioCompositor per guest, mixing the host and every other guest but not that guest, so
/// nobody hears their own voice back. Start each mixer before adding tracks: a track added before start()
/// is never mixed. Each mix is published to its own guest only; it is not played on this device.
@MainActor
final class PerGuestMixers {
    private var mixers: [String: AudioCompositor] = [:]  // guest identity → the mix that guest hears
    private var guestAudio: [String: AudioTrack] = [:]   // guest identity → that guest's voice

    /// Call when a guest's audio track arrives (RoomDelegate didSubscribeTrack), once per guest.
    func guestAudioArrived(_ audio: AudioTrack, from guest: RemoteParticipant) async throws {
        guard let local = Room.shared.localParticipant, guestAudio[guest.identity] == nil else { return }

        // The new guest's voice goes into every other guest's mix.
        for mixer in mixers.values {
            mixer.addTrack(audio)
        }
        guestAudio[guest.identity] = audio

        // The new guest's own mix: the host and the other guests.
        let mixer = AudioCompositor()
        try mixer.start()  // start first, then addTrack
        mixer.addTrack(local.microphoneTrack)
        for (identity, other) in guestAudio where identity != guest.identity {
            mixer.addTrack(other)
        }
        mixers[guest.identity] = mixer

        guard let output = mixer.outputTrack else { return }
        _ = try await local.publish(audioTrack: output, to: guest)
    }

    /// Call when a guest leaves (RoomDelegate participantDidDisconnect).
    func guestLeft(_ guest: RemoteParticipant) {
        mixers.removeValue(forKey: guest.identity)?.stop()
        if let audio = guestAudio.removeValue(forKey: guest.identity) {
            for mixer in mixers.values {
                mixer.removeTrack(audio)
            }
        }
    }
}

Composing video

Start the compositor with VideoCompositor.start(), then give it a composition template with VideoCompositor.setTemplate(_:). The picture comes out of VideoCompositor.outputTrack.

• VideoCompositor.resolutionSize is the size of the picture (default 1280×720).
• VideoCompositor.fps is how many pictures it makes per second (default 30).

You can change both while the compositor is running: the pictures after the change use the new value.

Restarting. VideoCompositor.stop() clears the composition template. After VideoCompositor.start(), call VideoCompositor.setTemplate(_:) again; VideoCompositor.outputTrack is then a new track, see Restarting a compositor.

Composition templates

A CompositeTemplate lists where each track goes in the picture, one VideoTrackPositionInfo per track:

• Position. VideoTrackPositionInfo.normalizedRect is a fraction of VideoCompositor.resolutionSize, from 0 to 1, with (0, 0) at the top left. CGRect(x: 0.5, y: 0, width: 0.5, height: 1) is the right half of the picture, whatever its size.
• Layers. The order of CompositeTemplate.trackPositions is the drawing order. Later entries are drawn over earlier ones.
• Fit or fill. VideoTrackPositionInfo.contentMode decides how a track that does not match its area is scaled: VideoContentMode.fill (the default) fills the area and crops the rest, VideoContentMode.fit shows the whole picture with black bars.
• Video only. Only video tracks are drawn. An entry whose track is not a VideoTrack is skipped: its area stays empty and there is no error.

The tracks in a composition template do not have to be published. Call VideoCompositor.setTemplate(_:) again at any time to switch to another composition template; the next picture uses it. VideoCompositor.currentTemplate is the composition template in use, or nil.

Swift

import RevueSDK

import CoreGraphics

/// Two cameras side by side in a 1280×720 picture. Each position is a fraction of resolutionSize, with
/// (0, 0) at the top left.
@MainActor
func startSideBySideComposite(left: VideoTrack, right: VideoTrack) -> VideoCompositor {
    let compositor = VideoCompositor()
    compositor.fps = 30
    compositor.resolutionSize = CGSize(width: 1280, height: 720)
    compositor.start()
    compositor.setTemplate(CompositeTemplate(trackPositions: [
        VideoTrackPositionInfo(track: left, normalizedRect: CGRect(x: 0, y: 0, width: 0.5, height: 1)),
        VideoTrackPositionInfo(track: right, normalizedRect: CGRect(x: 0.5, y: 0, width: 0.5, height: 1)),
    ]))
    return compositor
}

/// One camera fills the picture and a second one sits in the bottom-right corner. The order of the array
/// is the drawing order: the second entry is drawn over the first.
func cornerOverlayTemplate(background: VideoTrack, corner: VideoTrack) -> CompositeTemplate {
    CompositeTemplate(trackPositions: [
        VideoTrackPositionInfo(track: background, normalizedRect: CGRect(x: 0, y: 0, width: 1, height: 1)),
        VideoTrackPositionInfo(
            track: corner,
            normalizedRect: CGRect(x: 0.7, y: 0.7, width: 0.25, height: 0.25),
            contentMode: .fit
        ),
    ])
}

Publishing and recording the output

AudioCompositor.outputTrack and VideoCompositor.outputTrack are ordinary local tracks:

• Publish them to friends with LocalParticipant.publish(audioTrack:to:options:) and LocalParticipant.publish(videoTrack:to:options:). See Publishing tracks.
• Record them by passing them to a TrackRecorder. See Recording.
• Show the composed video in a VideoView.

The mix is not played on this device. AudioCompositor.outputTrack only goes where you send it.

Both output tracks are nil until start().

Swift

import RevueSDK

/// Send the composed video and the mix to a friend. outputTrack is nil until start(), and a new track
/// after every stop() and start().
@MainActor
func publishCompositeOutput(
    video: VideoCompositor,
    audio: AudioCompositor,
    to friend: RemoteParticipant
) async throws {
    guard let local = Room.shared.localParticipant,
          let videoOutput = video.outputTrack,
          let audioOutput = audio.outputTrack else { return }
    _ = try await local.publish(videoTrack: videoOutput, to: friend)
    _ = try await local.publish(audioTrack: audioOutput, to: friend)
}

/// Record the whole meeting: the composed video and the mix in one file.
@MainActor
func recordCompositeOutput(
    video: VideoCompositor,
    audio: AudioCompositor,
    fileName: String
) throws -> TrackRecorder? {
    guard let videoOutput = video.outputTrack, let audioOutput = audio.outputTrack else { return nil }
    let recorder = TrackRecorder(videoTrack: videoOutput, audioTrack: audioOutput)
    try recorder.startRecording(fileName: fileName)
    return recorder
}

/// After stop() and start() the compositor has a new outputTrack and no template. Set the template
/// again, unpublish the old track and publish the new one. A recorder fed by the old track has to be
/// replaced the same way.
@MainActor
func restartVideoCompositor(
    _ compositor: VideoCompositor,
    template: CompositeTemplate,
    replacing oldPublication: LocalTrackPublication,
    to friend: RemoteParticipant
) async throws -> LocalTrackPublication? {
    compositor.stop()
    compositor.start()
    compositor.setTemplate(template)
    guard let local = Room.shared.localParticipant, let output = compositor.outputTrack else { return nil }
    local.unpublish(publication: oldPublication)
    return try await local.publish(videoTrack: output, to: friend)
}

Restarting a compositor

Every start() creates a new output track. After you stop a compositor and start it again, the old output track sends nothing more, so:

1. Read outputTrack again.
2. Unpublish the old track and publish the new one.
3. Stop the recorder that used the old track and create a new TrackRecorder with the new one.

Also put back what stop() cleared: the tracks of an AudioCompositor, and the composition template of a VideoCompositor.