This documentation is for reference only. We are no longer onboarding new customers to Programmable Video. Existing customers can continue to use the product until December 5, 2026.
We recommend migrating your application to the API provided by our preferred video partner, Zoom. We've prepared this migration guide to assist you in minimizing any service disruption.
Simulcast is a standardized technique used to retain media quality when some subscribers have limited bandwidth. It is a mechanism for providing scalability to non-scalable video codecs such as VP8.
With simulcast, a client sends multiple versions of the same video simultaneously. Each version is encoded independently at a different resolution and frame rate; this way, a subscriber with limited bandwidth can receive a lower quality version of the media, but subscribers with more bandwidth can receive a higher quality video and their media quality is not degraded.
Twilio Video provides the option to use VP8 simulcast in Group Rooms via Twilio's Selective Forwarding Unit (SFU). To learn more about SFUs and media exchange in Peer-to-Peer vs. Group Rooms, see Understanding Video Rooms.
SFUs simply forward media and can neither transcode nor modify the video. When sending media with unicast (only sending one version of the video track), publishers need to reduce quality to adapt to the worst of their subscribers' bandwidths so that no subscriber is congested.
With VP8 simulcast, the SFU can forward higher quality videos to higher bandwidth subscribers and lower quality videos to lower bandwidth ones. The track publisher sends different track qualities and the SFU selects the most optimal quality for each subscriber. Then, the subscriber receives a single VP8 encoded video that is most suited for their network conditions.
The following illustration shows the difference between unicast and simulcast in a Group Room.
The SFU is critical to enabling simulcast. Simulcast is not recommended in Peer-to-Peer rooms, because there is no SFU mediating the data being sent. Simulcast in a Peer-to-Peer room means that each client is sending multiple differently encoded versions to every other participant in the room, which will consume more resources for both senders and receivers.
Simulcast offers the following benefits for Group Room participants:
On the other hand, simulcast also has some drawbacks:
Twilio's standard VP8 simulcast sends up to three layers of video at different resolutions. See the approximate resolutions of each layer here. In some video conferencing contexts, the higher resolution layers that consume the most resources to encode may not be needed.
Twilio Video offers adaptive simulcast, which enables and disables simulcast layers dynamically to improve bandwidth and CPU usage. This helps save device resources in cases such as presentation and grid modes, when the application does not need a Participant's highest resolution video. Adaptive simulcast ensures that publishers are only encoding the spatial layers needed at a given moment.
For example, when someone is presenting in a video conference, you will frequently only display the presenter's video in a large format, and will display only thumbnails of other participants' video. The participants who are not presenting do not need to encode and send higher resolution video layers because their video is not highlighted. The same might also be true in a video conference in grid mode, where each participant's video is the same size and no one's video needs to be the highest quality.
In these situations, adaptive simulcast can detect which layers are being used by subscribers and automatically turn off encoding on the publisher's side for higher spatial layers that are not being used. As speakers change, adaptive simulcast will dynamically turn on or turn off the appropriate spatial layers, based on what subscribers in the room are using.
Note that adaptive simulcast will not disable any video layers when the Room is being recorded, to help produce high quality recordings.
Adaptive simulcast is currently available in the Twilio Video JavaScript SDK, Android SDK, and iOS SDK. Please review:
for more information. If your application is currently using VP8 simulcast, we recommend that you switch to adaptive simulcast.
Twilio SDKs encode up to three spatial layers when simulcast is enabled. The following table illustrates which layers are typically generated given a particular capture resolution. Remark that this is just an approximation and that the real behavior may be slightly different. In the table, disabled
means that that layer is not sent in those conditions. (Video of the specified resolution is not generated by the
publisher and is not available at the SFU to be forwarded to subscribers).
Capture resolution | Layer 1 | Layer 2 | Layer 3 |
---|---|---|---|
352x288 | 352x288 | disabled | disabled |
480x360 | 240x180 | 480x360 | disabled |
640x480 | 320x240 | 640x480 | disabled |
640x480 (with crop) | 240x240 | 480x480 | disabled |
960x540 | 240x135 | 480x270 | 960x540 |
1024x768 | 256x192 | 512x384 | 1024x768 |
1024x768 (with crop) | 240x192 | 480x384 | 960x768 |
1280x720 | 320x180 | 640x360 | 1280x720 |
1280x720 (with crop) | 225x180 | 450x360 | 900x720 |
1920x1080 | 480x270 | 960x540 | 1920x1080 |
Simulcast can be enabled in Group Rooms. The following table illustrates Twilio's current support for simulcast:
Twilio Video SDK | Browser (or N/A) | VP8 Simulcast Support (only Group Rooms) |
---|---|---|
JavaScript | Chrome | Yes (SDK v1.7.0+) |
JavaScript | Firefox | No |
JavaScript | Safari | Yes (Safari 12.1+ with SDK 1.17.0+) |
Android | N/A | Yes (SDK v2.1.0+) |
iOS | N/A | Yes (SDK v2.1.0+) |
Simulcast is disabled by default. You can enable simulcast on a per-Participant basis when connecting to a Room.
To enable adaptive simulcast, set preferredVideoCodecs="auto"
in ConnectOptions
when connecting to a video Room. The SDK will use VP8 simulcast, and will enable/disable simulcast layers dynamically, thus improving bandwidth and CPU usage.
Adaptive simulcast works best when used along with Client Track Switch Off Control
and Video Content Preferences
. These two flags allow the SFU to determine which simulcast layers are needed, thus allowing it to disable the layers not needed on publisher side.
_11const { connect } = require('twilio-video');_11_11const room = await connect(token, {_11 preferredVideoCodecs: 'auto',_11 bandwidthProfile: {_11 video: {_11 contentPreferencesMode: 'auto',_11 clientTrackSwitchOffControl: 'auto'_11 }_11 }_11});
Please note the following limitations with adaptive simulcast in the JavaScript SDK:
preferredVideoCodecs="auto"
will revert to unicast in the following cases:
You can enable standard simulcast by setting simulcast: true
in ConnectOptions
when connecting to a video Room.
_10// Web JavaScript_10// Remember that simulcast only needs to be enabled in media publishers_10// See compatibility table above with supported browsers and required SDK versions_10_10const room = await connect(token, {_10 preferredVideoCodecs: [_10 { codec: 'VP8', simulcast: true }_10 ]_10});
Any Group Room Participant with VP8 simulcast enabled publishes all their video tracks using VP8 simulcast. Once this is done, Twilio's video infrastructure leverages simulcast tracks to provide the best possible quality to any subscriber without requiring any additional action from you.
By default, simulcast is disabled. You can enable simulcast on a per-Participant basis when connecting to a Room. This is done using the ConnectOptions
as shown in the following code snippet:
_10// Swift code_10// Remember that simulcast only need to be enabled in media publishers_10// See compatibility table above to with required SDK versions_10_10let connectOptions = ConnectOptions(token: accessToken) { (builder) in_10 builder.preferredVideoCodecs = [Vp8Codec(simulcast: true)]_10}
Any Group Room Participant with VP8 simulcast enabled publishes all its video tracks using VP8 simulcast. Once this is done, Twilio's video infrastructure leverages simulcast tracks to provide the best possible quality to any subscriber without requiring any additional action from you.
By default, simulcast is disabled. You can enable simulcast on a per-Participant basis when connecting to a Room. This is done using the ConnectOptions
as shown in the following code snippet:
_10// Java code_10// Remember that simulcast only need to be enabled in media publishers_10// See compatibility table above to with required SDK versions_10_10ConnectOptions connectOptions = new ConnectOptions.Builder(accessToken).preferVideoCodecs(Collections.singletonList(new Vp8Codec(true))).build();
Any Group Room Participant with VP8 simulcast enabled publishes all its video tracks using VP8 simulcast. Once this is done, Twilio's video infrastructure leverages simulcast tracks to provide the best possible quality to any subscriber without requiring any additional action from you.
To optimize video quality while minimizing CPU usage and bandwidth, it is recommended to use VP8 simulcast with the capture settings suggested below on each mobile platform.
24 FPS. When simulcasting, this will result in 3 temporal layers of 24 FPS, 12 FPS, and 6 FPS. Selecting 24 frames / second instead of the default of 30 reduces the CPU load on the VP8 software encoder.
iOS devices support high resolution capture formats with ratios of 1.33:1 and 1.77:1. When simulcasting, it is often desirable to produce a squarish ratio (1.25:1) that can be viewed by subscribers in landscape or portrait, and as smaller thumbnails. Cropping is performed at the source by using a format request. Besides changing the ratio of the captured video, cropping also reduces the number of pixels that need to be processed by the software encoder. Using 1280x720 or 1024x768 for video capture will result in 3-layer simulcast with the layer structure as shown in the table above. Using 640x480 is recommended on older iPhones and will result in 2-layer simulcast.
If a Group Room is being used, it is recommended to remove the rotation tags using hardware acceleration using this API. Also, it is recommended to reduce the audio bitrate tuned for speech content.
The above recommendations are implemented in this code snippet:
_79struct CaptureDeviceUtils {_79_79 // Produce 3 spatial layers ~ {960x768, 480x384, 240x192}. 1024x768 is captured on most phones_79 // Produce 3 spatial layers ~ {900x720, 450x360, 225x180}, 1280x720 is captured on on iPhone X_79 static let kSimulcastVideoDimensions = CMVideoDimensions(width: 900, height: 720)_79 static let kSimulcastVideoFrameRate = UInt(24)_79 static let kSimulcastVideoBitrate = UInt(1800)_79_79 /*_79 * @brief Finds the smallest format that is suitably close to the ratio requested._79 *_79 * @param device The AVCaptureDevice to query._79 * @param targetRatio The ratio that is preferred._79 *_79 * @return A format that satisfies the request._79 */_79 static func selectFormatBySize(device: AVCaptureDevice,_79 targetSize: CMVideoDimensions) -> VideoFormat {_79 // Arranged from smallest to largest._79 let formats = CameraSource.supportedFormats(captureDevice: device)_79 var selectedFormat = formats.firstObject as? VideoFormat_79 for format in formats {_79 guard let videoFormat = format as? VideoFormat else {_79 continue_79 }_79 if videoFormat.pixelFormat != PixelFormat.formatYUV420BiPlanarFullRange {_79 continue_79 }_79 let dimensions = videoFormat.dimensions_79 // Cropping might be used if there is not an exact match._79 if (dimensions.width >= targetSize.width && dimensions.height >= targetSize.height) {_79 selectedFormat = videoFormat_79 break_79 }_79 }_79 return selectedFormat!_79}_79_79let options = CameraSourceOptions { (builder) in_79 // Stripping rotation tags using hardware acceleration_79 builder.rotationTags = .remove_79}_79camera = CameraSource(options: options, delegate: self)_79_79// Assume front camera is available_79let frontCamera = CameraSource.captureDevice(position: .front)_79if let camera = camera {_79 localVideoTrack = LocalVideoTrack(source: camera, enabled: true, name: "Camera")_79_79 // Discover a simulcast format for the front camera_79 let format = CaptureDeviceUtils.selectFormatBySize(device: frontCamera!,_79 targetSize: CaptureDeviceUtils.kSimulcastVideoDimensions)_79_79 // Lower the frame rate to reduce CPU load, but still produce 3 temporal layers (f, f/2, f/4)_79 format.frameRate = CaptureDeviceUtils.kSimulcastVideoFrameRate_79_79 // Apply slight cropping to reduce CPU load, and provide square-ish video_79 let croppedFormat = VideoFormat.init()_79 croppedFormat.dimensions = CaptureDeviceUtils.kSimulcastVideoDimensions_79 camera.requestOutputFormat(croppedFormat)_79_79 camera.startCapture(device: device, format:format) { (captureDevice, videoFormat, error) in_79 if let error = error {_79 self.logMessage(messageText: "Capture failed with error.\ncode = \((error as NSError).code) error = \(error.localizedDescription)")_79 }_79 }_79}_79_79let connectOptions = ConnectOptions(token: accessToken) { (builder) in_79 if let localVideoTrack = localVideoTrack {_79 builder.videoTracks = [localVideoTrack]_79 }_79 builder.isNetworkQualityEnabled = true_79 builder.networkQualityConfiguration =_79 NetworkQualityConfiguration(localVerbosity: .minimal, remoteVerbosity: .minimal)_79 // Enable Vp8 simulcast, and cap the bitrate at 1.8 Mbps to reduce strain on the sender. Reduce audio bitrate for speech content._79 builder.encodingParameters = EncodingParameters(audioBitrate:16, videoBitrate:1800)_79 builder.preferredVideoCodecs = [Vp8Codec(simulcast: true)]_79}
24 FPS. When simulcasting, this will result in 3 temporal layers of 24 FPS, 12 FPS, and 6 FPS. Selecting 24 frames / second instead of the default of 30 reduces the CPU load on the VP8 encoder.
Using 1280x720 or 1024x768 for video capture will result in 3-layer simulcast with the layer structure as shown in the table above. Using 640x480 for video capture will result in a 2-layer simulcast.
It is recommended to reduce the audio bitrate tuned for speech content.
The above settings are specified as part of the Video Format API as shown in the code snippet below:
_25import tvi.webrtc.MediaCodecVideoEncoder;_25_25VideoDimensions videoDimensions = VideoDimensions.VGA_VIDEO_DIMENSIONS;_25if (MediaCodecVideoEncoder.isVp8HwSupported()) {_25 videoDimensions = VideoDimensions.HD_720P_VIDEO_DIMENSIONS;_25}_25VideoFormat videoFormat = new VideoFormat(videoDimensions, 24);_25_25LocalVideoTrack localVideoTrack = LocalVideoTrack.create(context, true, videoCapturer, videoFormat);_25_25// Enable network quality information for local and remote participants_25NetworkQualityConfiguration configuration =_25 new NetworkQualityConfiguration(_25 NetworkQualityVerbosity.NETWORK_QUALITY_VERBOSITY_MINIMAL,_25 NetworkQualityVerbosity.NETWORK_QUALITY_VERBOSITY_MINIMAL);_25_25ConnectOptions connectOptions = new ConnectOptions.Builder(accessToken)_25 .enableNetworkQuality(true)_25 .networkQualityConfiguration(configuration)_25 .videoTracks(Collections.singletonList(localVideoTrack))_25 // Cap the bitrate at 1.8 Mbps to reduce strain on the sender. Reduce audio bitrate for speech content._25 .encodingParameters(new EncodingParameters(16, 1800)_25 // Enable Vp8 simulcast_25 .preferVideoCodecs(Collections.singletonList(new Vp8Codec(true))) // Enable simulcast_25 .build();