Brightcove QoE Insights

Introduction

Quality of Experience (QoE) Insights incorporate QoE analytics data and user engagement data to allow you to track and determine your audience's QoE tolerance and abandonment factors. These insights lead to actionable steps to improve user experience.

QoE insights provide Brightcove customers with easy-to-understand correlations from QoE Metrics, potential business impacts, and recommended interventions. These insights are categorized as follows:

• QoE Tolerance:
  • Help you understand when QoE metrics have an impact on user engagement
• QoE Benchmarks:
  • Provide a sense of how your QoE metrics compare with other customers
• QoE Recommendations:
  • Suggest areas for improvement based on QoE Metric thresholds and a heat map of error codes.

QoE Insights incorporate:

• QOE Analytics
• Brightcove Analytics
• Audience Insights data

Setup

QoE Insights are available to Media Video Cloud customers for an additional charge. Contact your Customer Success Manager to enable your account.

Navigation

Once your account is enabled for QoE Insights, you will find them in the Analytics module in Studio.

1. Open Video Cloud Studio.
2. In the primary navigation, click Analytics.

3. In the panel navigation, click QOE. At the top, you will see a Quality Score for your account. This feature is a part of QoE Insights

   The Quality Score provides a quick reference for the user experience related to video playback.

   

Quality Score

The Quality Score is a quick way to view customer engagement. The score is based on correlations between video playback and viewer dropoff. The colors indicate the following:

• Green - Everything looks good
• Yellow - Areas to watch
• Red - Action may be taken to improve

Metrics are weighted to calculate the Quality Score:

Quality Score = Video start time / 50
              + Rebuffering per min / 30
              + Upscale duration / 20
              - Percentage error rate
    

Time range and dimensions

Use time range and dimension to filter the data displayed in the charts below.

Time range selector

Click DATE to expand the dropdown menu. Select the time range for your reporting data. The charts below trend over time, based on the time range.

Dimension selector

Click DIMENSION to expand the dropdown menu. Select the measurement for your reporting data. When you select a dimension, it will be reflected in all of the charts described below.

Dimensions

Account

The Account view provides an overall number for all streams in your account across all platforms.

Device

The Device view makes it easier to see device-specific issues and trends over time. The example below shows a spike in error rate for Android devices that could correlate to a recent app update.

This graph shows the top 5 devices by view count, and the account level.

Stream type

The Stream type view compares all content in your account with VOD and Live metrics.

Player

The Player view shows specific players' performance and allows you to isolate and troubleshoot players where QoE is underperforming. Modifications to chunk size and buffer size can be measured using this metric.

This graph shows the top 5 players by view count, and the account level.

Country

For international customers, viewing QoE metrics by country can assist in isolating regional issues. In the fictional example below, the customer adopted a new content delivery network (CDN) for their Asian consumers. The chart shows that the CDN underperformed when looking at the stall rate.

This graph shows the top 5 countries by view count, and the account level.

Error rate

The error rate is the number of errors reported by the player divided by the number of play requests.

Here we focus on the errors that prevent playback since some errors occur in the background without affecting the viewer experience. Brightcove Player reports errors that prevent video playback, but some of these errors happen before playback begins. Low error rates mean that viewers can watch their selected content.

This example shows the top 5 devices by view count, and the account level.

Video start time

The video start time is the average time between play request and stream start.

This metric is helpful because long load times cause abandonment before streaming starts. Long load times can point to issues with CDN, player plug-ins, and initial stream bitrate where intervention makes sense. Low video start times mean your viewers can start watching their videos quickly, which is what they expect.

This example shows the top 5 devices by view count, and the account level.

Stall rate

The stall rate is the total number of stalls divided by the total number of hours streamed.

We focus on actual video stalls instead of all rebuffering events because rebuffering may or may not impact playback, depending on the platform. Stalls affect the viewer experience, including single stalls of significant length or frequent stalls of varying lengths. Low stall rates provide smoother playback and a better viewer experience.

This example shows the top 5 devices by view count, and the account level.

Upscale time

The upscale time is the average number of seconds per hour of viewing that is spent in an upscaled state (seconds per hour).

Upscaling occurs when a video rendition is streamed in a lower resolution than the current player size. Upscaling can result in fuzziness or video pixelation. One area where upscaling is common is when content is streamed to a 4K TV. In this case, the stream may be available at a high resolution but not encoded at 4k. While technically, this is upscaling, most viewers will not notice. Low upscaling time is associated with smooth, crisp video playback.

This example shows the top 5 countries by view count, and the account level.

Summary table

The summary table makes it easy to explore and identify anomalies.

The date range and dimensions that you set for the page, also applies to the data in the summary table. For example, if you select the Device dimension, you will see a Summary by Device.

Summary by Device

Each column is sortable. The default sort order is video views descending.

In addition to the key metrics, the summary table includes the following data:

• View count
• Video resolution breakdown
• Rebuffering time
• Avg Bitrate

For details about the column data, see the Glossary below.

Insights time range

Use time range to filter the data displayed in the charts below.

Click 3 months to expand the dropdown menu. Select the time range for your reporting data. The charts below trend over time, based on the time range.

Abandonment rate

This chart shows the abandonment rate. The chart shows the percent of play requests where users exited before play start.

Dimension

Click Device to expand the Abandonment Rate dimension dropdown menu. Select a dimension to change the data displayed.

Error codes

This chart provides a clear view of the errors generated by the players and granular aggregations that will allow you to troubleshoot and isolate situations or specific configurations that could be worked out to improve your audience’s Quality of Experience.

The Error Code chart will allow you to visually measure the impact of the policies implemented to protect the streams and how those policies contribute to the Error rate.

This example shows the top 5 devices by view count overlayed with the average abandonment rate.

Each column is sortable. Below, you can review the Error codes data:

• Play requests- Number of play requests
• Error count- Number of counting erros
• Type- Type of error code
• Sessions effected- Percent of all sessions with a fatal error
• Top country- Countries with the highest error count
• Top device- Devices with the highest error count
• Top platform- Platforms with the highest error count
• Top OS- OS with the highest error count
• Top video- Videos with the highest error count
• Stream type- Live or VOD

Actions

Here are the actions that are available for this graph.

• Click to download data as a CSV file.

  

Dimension

Click Device to expand the Error codes dimension dropdown menu. Select a dimension to change the data displayed.

Error trends

The following graphs show the error trends filtered by device.

Actions

Here are the actions that are available for this graph.

• Click to view the full graphs.

Dimension

Click Error Types to expand the Error trends dimension dropdown menu. Select a dimension to change the data displayed.

Error code types

The table below lists the types of error codes.

Engagement Funnel

The Engagement Funnel shows viewer engagement through various stages of video consumption. The funnel shows viewer engagement (those watching and those lost) from the play request to video view through the point of 75% video completion. The funnel lets you visualize the magnitude of lost viewers for the chosen dimension.

Dimension

Click Device to expand the Engagement Funnel dimension dropdown menu. Select a dimension to change the data displayed.

Metric

The Metric selector lets you control the buckets, which define how the data is grouped.

The example below has the Rebuffer metric selected. The colors in the funnel shows the rebuffering rate for each group:

• 5+ minutes
• 30 seconds to 5 minutes
• 5 to 30 seconds
• Less than 5 seconds
• Number of errors

If you look at the Rebuffer metric and viewer drop-off is large, you can look at the bitrate you are using and adjust it to minimize buffering.

Group by

Another way to look at the data is to group by the selected metric.

Completion Rate

The Completion Rate charts show what happens over time to viewers once they are watching a video. When video start time is high, most viewers will drop off.

Dimension

Click Video start time to expand the Completion Rate dimension dropdown menu. Select a dimension to change the data displayed.

Example

In this example, with Rebuffering time selected, we see that most viewers decide to continue watching within the first 20 percent of the video length. Also, most viewers (97%) continue watching when the rebuffering time is less than 5 seconds.

Interest Phase

The highlighted area in the graph shows that the highest abandonment rate happens within the first 20% of the video. Knowing this, you can be intelligent with your ad strategy during this timeframe. Consider varying the number of ads and the ad length to see how it affects viewer completion.

Video start time

The bar graph below shows the total number of video views based on video start time.

Completion details

The completion details allow you to dig into what is happening within a metric time period. This example shows the following:

• When the re-buffer rate is between 1 and 5 minutes, only 14.1% of viewers watch at least 75% of the video length (considered complete)
• There are 76.6K total views
• Average error rate is 6.9%
• Average stall rate is 12.5/hr
• Upscale time is 3.7min/hr
• Top views by country, device, and content duration

Glossary

Error rate

Percent of all play requests with errors that prevented playback experience.

Errors are often caused by player or player plugin misconfiguration. Viewing error rate by device or player can provide direction on where to focus on reducing errors. You can review our Brightcove Player Error Reference for more information on player errors.

Video start time

Average time between the play request and the stream start per video view; this time does not include ad pre-roll time.

The number of player plugins that load, initial stream bitrate, and viewer bandwidth can impact start time. Reviewing players, bitrate ladder, and device/video resolution breakdown can provide direction on improving this.

Counting errors

Distinct fatal errors per session, per viewer. If multiple errors occur in a session, they are de-duplicated.

Video resolution

Percent of engagement events issued by categories of video resolution.

Reviewing resolution breakdown by device in combination with video start time, rebuffering time, and stall rate can show ways to optimize playback.

Rebuffering time

Average seconds of buffering per hour of video viewed.

This often increases when the bandwidth is too low to consume the bitrate of the stream that is playing. Looking at bitrate and video resolution by device or country can provide insights into video resolution adjustments that could reduce rebuffering.

Stall Rate

Stalls per hours viewed in the time range.

Stall rates often increase when the bandwidth is too low to consume the bitrate of the stream that is playing. Looking at bitrate and video resolution by device or country can both provide insights into video resolution adjustments that could reduce rebuffering.

Avg bitrate

Average bitrate (kbps) across all video views in the time range.

Upscale time

Average seconds per hour spent in an upscaled state.

Upscaling has a real impact on picture clarity when video resolution is significantly lower than screen resolution. Review screen resolutions by device to understand the potential impact of upscaling better. Consumers using 4K TVs will often have upscaling occur, but the viewing experience can still be high quality.