No remote audio when `setManualRenderingMode(true)`

[ynnekj]

I have my own audio capture hardware that collects audio data externally, so I don't want to use the system microphone for recording.

However, I couldn't find a way to inject my custom audio data into the LiveKit SDK.

What I would like to know is:
If I set enableMicrophone to false, how can I feed my own audio data into the SDK in order to replace the functionality of a LocalAudioTrack?

Is there an officially supported way to create a custom audio track using external audio sources?

[ynnekj]

I have used setManualRenderingMode, capturePostProcessingDelegate to solve my need to input custom audio.
However, because setManualRenderingMode(true) is set, the remote audio cannot be played normally.
How can I start playing the remote audio normally?

try AudioManager.shared.setManualRenderingMode(true)
AudioManager.shared.capturePostProcessingDelegate = CustomAudio()

[hiroshihorie]

With webrtc-sdk/webrtc#188 and #758, AudioManager.shared.add(remoteAudioRenderer:) will work now, so you can attach an audio player. Final code will look something like :

let audioPlayer = AudioPlayerRenderer() // Keep instance

try! AudioManager.shared.setManualRenderingMode(true)
AudioManager.shared.add(remoteAudioRenderer: audioPlayer)

Task {
    try await audioPlayer.start()
}

[ynnekj]

Thank you very much. Using AudioPlayerRenderer works, but we have encountered a new problem:

1. We are currently using LiveKit SDK version 2.7.1.
2. Based on your previous feedback, we added AudioManager.shared.add(remoteAudioRenderer: audioPlayer), and it successfully plays remote audio.
3. However, we found that when executing try await audioPlayer!.start(), the AudioSession Category is modified, and it seems to enable HPF. This causes our device’s custom recording (via our own BLE recording protocol, not iOS’s built-in recording) to stop working.
4. We believe that when playing remote audio, only AVAudioSessionCategoryPlayback (A2DP) is needed. Using AVAudioSessionCategoryPlayAndRecord is unnecessary, and it triggers HPF, which disables our custom BLE recording.
5. We tried manually setting AVAudioSessionCategoryPlayback, but then encountered another issue: "Audio Engine Error (Audio engine returned error code: -4102)".

Is there any way to solve this problem?
In our scenario, we actually do not need to enable the system recording feature or request microphone permission.

Our device differs from regular headphones in that it includes a custom BLE recording protocol. While it also supports HPF and A2DP, when we start custom BLE recording, if the system attempts to enable HPF (e.g., when preparing to use the microphone), our BLE recording is automatically stopped.

[ynnekj]

This is what we tried: when the app is in the foreground, it can solve our problem. However, when the app is in the background, it fails to set session.setCategory twice successfully, even though Background Modes – Audio is enabled.

private func connectWithVoice() async throws {
    let connectionDetails = try await getConnection()
    guard let connectionDetails else { return }

    // We don’t need the recording feature, we only need simple remote audio playback
    // To prevent the system from enabling HPF when calling audioPlayer!.start(),
    // we changed it to manually configure the AudioSession
    AudioManager.shared.audioSession.isAutomaticConfigurationEnabled = false
    
    do {
        /* The first time we need to set .playAndRecord, otherwise when connecting we get the following error:
            Error Domain=io.livekit.swift-sdk Code=801 "Audio Engine Error(Audio engine returned error code: -4102)"
            UserInfo={NSLocalizedDescription=Audio Engine Error(Audio engine returned error code: -4102)}
         */
        let session = AVAudioSession.sharedInstance()
        try session.setActive(false)
        try session.setCategory(.playAndRecord,
                                mode: .videoChat,
                                options: [.mixWithOthers, .allowBluetooth, .allowAirPlay])
        try session.setActive(true)
        
    } catch {
        print("⚠️ 1Failed to reset AudioSession: \(error)")
    }

    audioPlayer = AudioPlayerRenderer() // Keep instance
    
    try AudioManager.shared.setManualRenderingMode(true)
    AudioManager.shared.capturePostProcessingDelegate = CustomAudio()
    AudioManager.shared.add(remoteAudioRenderer: audioPlayer!)
    
    try await room.connect(
        url: connectionDetails.serverUrl,
        token: connectionDetails.participantToken,
        connectOptions: .init(enableMicrophone: true)
    )
    
    Task {
        // The second time, we switch to .playback to avoid the system enabling HPF,
        // and then call audioPlayer!.start()
        // When the app is in the foreground, this approach works and solves my problem.
        // However, if the app goes into the background, setting setCategory twice like this
        // results in the following error and no audio is played:
        // ⚠️ 2Failed to reset AudioSession: Error Domain=NSOSStatusErrorDomain Code=560557684
        //  "Session activation failed" UserInfo={NSLocalizedDescription=Session activation failed}
        do {
            let session = AVAudioSession.sharedInstance()
            try session.setActive(false)
            try session.setCategory(.playback,
                                    mode: .default,
                                    options: [])
            try session.setActive(true)
            try await self.audioPlayer!.start()
            
        } catch {
            print("⚠️ 2Failed to reset AudioSession: \(error)")
        }
    }
}

[hiroshihorie]

Yes in your case you should:

AudioManager.shared.audioSession.isAutomaticConfigurationEnabled = false

And I'm not sure why the engine fails if category is not .playAndRecord, .playback should work.
Will make a ticket and investigate that one.

[hiroshihorie]

The SDK was checking for .playAndRecord & mic perms even for manual rendering mode which doesn't access the device, so I patched this in this PR: #780
I've tested it and it works without touching the AVAudioSession (if you set AudioManager.shared.audioSession.isAutomaticConfigurationEnabled = false). So you don't need to set .playAndRecord for audio engine to start.