Speech to speech: Mute functionality

[pranavjoshi001]

Changelog Entry

• Added mute/unmute functionality for speech-to-speech with silent chunks to keep server connection alive, in PR #5688, by @pranavjoshi

Description

This PR adds mute/unmute functionality for the Speech-to-Speech (S2S) feature as core API only, without UI changes. When muted, the microphone is turned off (browser indicator disappears) but silent audio chunks continue to be sent to keep the server connection alive. This prevents connection timeouts while allowing consumers to implement their own mute UI.

Design

The mute functionality works at multiple levels:

1. AudioWorklet Level:

   • Added MUTE and UNMUTE commands
   • When muted, the processor generates silent (all zeros) Int16 audio chunks instead of real audio data
   • This keeps chunks flowing at the same interval to maintain the server connection

2. useRecorder Hook (useRecorder.ts):

   • Added mute() function that:
     • Sends MUTE command to the worklet
     • Disconnects the source node from the audio graph
     • Stops all MediaStream tracks (turns off browser mic indicator)
   • Returns an unmute() function that:
     • Sends UNMUTE command to the worklet
     • Re-acquires the microphone via getUserMedia
     • Reconnects the source node to the audio graph

3. VoiceRecorderBridge (VoiceRecorderBridge.tsx):

   • Wires the mute function to the voice state machine
   • When voice state transitions to muted, calls mute() and stores the unmute function
   • When voice state transitions back to listening, calls the stored unmute function

4. Redux Actions & Hooks:

   • Added muteVoiceRecording and unmuteVoiceRecording Redux actions
   • Exposed useVoiceRecordingMuted hook for consumers that gives value and setter function.

Specific Changes

• Added MUTE and UNMUTE command handling in AudioWorklet processor to generate silent chunks when muted
• Added mute function to useRecorder.ts hook that disconnects audio and stops MediaStream while continuing to send silent chunks
• Updated VoiceRecorderBridge.tsx to handle mute/unmute based on voice state changes
• Added muteVoiceRecording.ts and unmuteVoiceRecording.ts Redux actions
• Updated voiceActivity.ts reducer to handle VOICE_MUTE_RECORDING and VOICE_UNMUTE_RECORDING actions
• Added useVoiceRecordingMuted` hook and exported them from API, component, and bundle packages
• Added unit tests for mute functionality in useRecorder.spec.tsx

• I have added tests and executed them locally
• I have updated CHANGELOG.md
• I have updated documentation

Review Checklist

This section is for contributors to review your work.

• Accessibility reviewed (tab order, content readability, alt text, color contrast)
• Browser and platform compatibilities reviewed
• CSS styles reviewed (minimal rules, no z-index)
• Documents reviewed (docs, samples, live demo)
• Internationalization reviewed (strings, unit formatting)
• package.json and package-lock.json reviewed
• Security reviewed (no data URIs, check for nonce leak)
• Tests reviewed (coverage, legitimacy)

[compulim]

Add end-to-end tests.

[pranavjoshi001]

Add end-to-end tests.

done

[compulim]

Aren't they simple negation of each other? 🤣

[compulim]

It's okay to work on it when we touch it.

[compulim]

Nice work, love the code, well done.

[Copilot]

Pull request overview

Adds core (non-UI) mute/unmute support for Speech-to-Speech recording by introducing a muted voice state and keeping the server stream alive via silent audio chunks while the physical microphone is stopped.

Changes:

• Add VOICE_MUTE_RECORDING / VOICE_UNMUTE_RECORDING actions and reducer handling, plus muted in VoiceState.
• Extend S2S recorder/worklet to support MUTE/UNMUTE and generate silent frames while muted; wire it via VoiceRecorderBridge.
• Expose a consumer-facing hook useVoiceRecordingMuted, export it through component/bundle entrypoints, and add unit + HTML harness tests.

Reviewed changes

Copilot reviewed 16 out of 16 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
packages/core/src/reducers/voiceActivity.ts	Adds mute/unmute action handling and transitions voice state to/from muted.
packages/core/src/index.ts	Exports new mute/unmute actions (and reorders related voice exports).
packages/core/src/actions/muteVoiceRecording.ts	New Redux action creator/constants for muting voice recording.
packages/core/src/actions/unmuteVoiceRecording.ts	New Redux action creator/constants for unmuting voice recording.
packages/core/src/actions/setVoiceState.ts	Extends VoiceState union to include muted.
packages/api/src/providers/SpeechToSpeech/private/useRecorder.ts	Adds AudioWorklet mute logic + recorder mute/unmute flow with silent chunks.
packages/api/src/providers/SpeechToSpeech/private/VoiceRecorderBridge.tsx	Bridges voiceState === 'muted' to recorder mute/unmute behavior.
packages/api/src/providers/SpeechToSpeech/private/useRecorder.spec.tsx	Adds unit tests for mute/unmute behavior and commands.
packages/api/src/hooks/useVoiceRecordingMuted.ts	New public hook for consumers to read/set muted state via Redux actions.
packages/api/src/hooks/index.ts	Exports useVoiceRecordingMuted.
packages/api/src/boot/hook.ts	Re-exports useVoiceRecordingMuted from the boot hook entry.
packages/component/src/boot/hook.ts	Surfaces useVoiceRecordingMuted through component hook entry.
packages/bundle/src/boot/actual/hook/minimal.ts	Surfaces useVoiceRecordingMuted through minimal bundle hook entry.
tests/assets/esm/speechToSpeech/mockMediaDevices.js	Updates test audio mock to support muted vs non-muted chunk generation.
tests/html2/speechToSpeech/mute.unmute.html	Adds HTML harness test validating idle/listening/muted transitions and chunk content.
CHANGELOG.md	Adds changelog entry for S2S mute/unmute functionality.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

VOICE_UNMUTE_RECORDING currently forces voiceState: 'listening' even when the current state is not muted (it only logs a warning). This means calling useVoiceRecordingMuted()[1](false) while voiceState is idle (or any other state) will incorrectly transition into listening. Consider returning state (no-op) unless state.voiceState === 'muted', similar to the guard used for VOICE_MUTE_RECORDING.

[Copilot]

Console warning message uses "transit" (e.g., "Should not transit...") which reads like a typo/grammar issue in a user-visible log message. Consider changing it to "transition" for clarity and consistency.

[Copilot]

The unmute closure calls acquireAndConnectMediaStream() without awaiting or handling errors. If getUserMedia() rejects, this becomes an unhandled promise rejection; and if recording stops while the promise is in-flight, the resolved stream can be assigned to streamRef without being connected/stopped, potentially turning the mic indicator back on while not recording. Consider making the unmute path async (or explicitly catch) and adding a cancellation/guard so newly acquired streams are immediately stopped if the worklet/recording has been torn down.

Suggested change

	// Restart MediaStream and reconnect source (fire and forget)
	acquireAndConnectMediaStream();
	// Restart MediaStream and reconnect source (handle errors and teardown race)
	(async () => {
	try {
	await acquireAndConnectMediaStream();
	} catch {
	// Swallow errors to avoid unhandled promise rejections.
	// Optionally log if needed.
	} finally {
	// If recording/worklet have been torn down while acquiring the stream,
	// immediately stop any newly acquired media stream to avoid leaving
	// the mic active when not recording.
	if (!audioCtxRef.current || !workletRef.current) {
	stopMediaStream();
	}
	}
	})();

[Copilot]

In the mock sine-wave generator, audioData[+i] is equivalent to audioData[i] but the unary + is unnecessary and makes the intent harder to read. Consider using audioData[i] here.

Suggested change

	audioData[+i] = Math.sin(i * 0.1) * 0.5;
	audioData[i] = Math.sin(i * 0.1) * 0.5;

[Copilot]

Changelog entry says mute/unmute was added "via useRecorder hook", but useRecorder is an internal/private implementation detail under providers/SpeechToSpeech/private. For consumers, the new public surface appears to be useVoiceRecordingMuted (and/or the muteVoiceRecording/unmuteVoiceRecording actions). Consider rewording this entry to reference the public API to avoid confusing integrators.

Suggested change

	- Added core mute/unmute functionality for speech-to-speech via `useRecorder` hook (silent chunks keep server connection alive), in PR [#5688](https://github.com/microsoft/BotFramework-WebChat/pull/5688), by [@pranavjoshi](https://github.com/pranavjoshi001)
	- Added core mute/unmute functionality for speech-to-speech via `useVoiceRecordingMuted` hook and `muteVoiceRecording` / `unmuteVoiceRecording` actions (silent chunks keep server connection alive), in PR [#5688](https://github.com/microsoft/BotFramework-WebChat/pull/5688), by [@pranavjoshi](https://github.com/pranavjoshi001)