Insert SEI into H.264 (AVC) Streams | v0.18.0.0+

OvenMediaEngine Enterprise supports SEI insertion starting from version 0.18.0.0-1.

OvenMediaEngine can insert Supplemental Enhancement Information (SEI) into live streams to deliver custom data with video content at frame-level precision.

Overview

• You can insert SEI dynamically through the OvenMediaEngine's Send Event API or continuously through XML configuration.

• When inserting SEI, you can add custom data, and the inserted SEI automatically includes UUID and Timestamp.

  • The inserted SEI is generated in OvenMediaEngine-specific format containing UUID, Timestamp, and custom data.

• OvenPlayer can receive SEI inserted by OvenMediaEngine and provides functionality to automatically parse the SEI specification defined by OvenMediaEngine.

Inserting SEI via the Send Event API

API Interface

You can use OvenMediaEngine's SendEvent REST API to dynamically trigger events that insert SEI into streams.

Request

POST /v1/vhosts{vhost}/apps/{app}/streams/{stream}:sendEvent

Header

Authorization: Basic {credentials}

# Authorization
Credentials for HTTP Basic Authentication created with <AccessToken>

Body

{
    "eventFormat": "sei",
    "eventType": "video",
    "events": [
        {
            "seiType": "UserDataUnregistered",
            "data": "OvenMediaEngine"
        }
    ]
}

POST /v1/vhosts{vhost}/apps/{app}/streams/{stream}:sendEvents

Header

Authorization: Basic {credentials}

# Authorization
Credentials for HTTP Basic Authentication created with <AccessToken>

Body

[
  {
      "eventFormat": "sei",
      "eventType": "video",
      "events": [
          {
              "seiType": "UserDataUnregistered",
              "data": "OvenMediaEngine"
          }
      ]
  }
]

Parameter	Required	Description
eventFormat	Y	Specifies the event format (use sei format).
eventType	N	Specifies the event type. Default: video
events	Y	Contains event data values.
event.seiType	N	Specifies SEI type. Default: UserDataUnregistered
event.data	Y	Enter the actual data to be transmitted.

Response

200 Ok

Header

Content-Type: application/json

Body

{
    "message": "OK",
    "statusCode": 200
}

400 Bad Request

Header

Content-Type: application/json

Body

{
    "message": "eventFormat(string) and events(array) are required",
    "statusCode": 400
}

{
    "message": "eventFormat is not supported: [XXX]",
    "statusCode": 400
}

{
    "message": "Could not make events data",
    "statusCode": 400
}

{
    "message": "eventType must be string",
    "statusCode": 400
}

{
    "message": "eventType is not supported: [XXX]",
    "statusCode": 400
}

500 Internal Server Error

Header

Content-Type: application/json

Body

{
    "message": "Could not inject event: [XXX]",
    "statusCode": 500
}

Inserting SEI through XML Configuration

For scenarios requiring continuous SEI insertion, you can configure this behavior through XML configuration. Create an XML file defining SEI insertion events and enable EventGenerator in Server.xml.

Configuration Example

Server.xml: You can enable the EventGenerator functionality for SEI insertion by adding <Application><EventGenerator>.

<?xml version="1.0" encoding="UTF-8"?>
<Server version="8">
  ...
  <VirtualHosts>
    <VirtualHost>
      <Applications>
        <Application>
          ...
          <EventGenerator>
            <Enable>true</Enable>
            <Path>events/send_event_info.xml</Path>
          </EventGenerator>
        </Application>
      </Applications>
    </VirtualHost>
  </VirtualHosts>
</Server>

Element	Required	Description
<Enable>	Y	Sets activation status to true or false. Default: false
<Path>	Y	Sets the path to the XML file defining SEI insertion details. If a relative path is specified, the directory containing the Server.xml file is used as the base.

XML Defining SEI Insertion Events: Create an XML file defining SEI insertion events at the path specified in Server.xml. In this example, it's send_event_info.xml.

<?xml version="1.0" encoding="UTF-8"?>
<EventInfo>
  <Event>
    <Enable>true</Enable>
    <SourceStreamName>stream*</SourceStreamName>
    <Interval>2000</Interval>
    <EventFormat>sei</EventFormat>
    <EventType>video</EventType>
    <Values>
      <SeiType>UserDataUnregistered</SeiType>
      <Data>Hi! OvenMediaEngine! CurrentTime:${EpochTime}</Data>
      <KeyframeOnly>true</KeyframeOnly>
    </Values>
  </Event>
</EventInfo>

Parameter	Required	Description
<Enable>	Y	Sets activation status to true or false. Default: false
<SourceStreamName>	Y	Specifies the target stream name. It supports wildcards (*) for pattern matching.
<Interval>	Y	Sets event occurrence interval in milliseconds (ms).
<EventFormat>	Y	Specifies event format (use sei format).
<EventType>	N	Specifies the event type. Default: video
<Values>	Y	Contains the value of the event data.
<Values><SeiType>	N	Specifies SEI type. Default: UserDataUnregistered
<Values><Data>	Y	Specifies custom data to be inserted into SEI. It supports the macro ${EpochTime}, which is replaced with the server's current epoch time in milliseconds when transmitted (e.g., 1747147513056).
<Values><KeyframeOnly>	N	Specifies the target for event insertion. If set to true, the event will be inserted into the first keyframe after the specified interval has passed (Supported from version 0.18.2.0+). Default: false

Changes made to the event definition XML file are immediately applied without needing to restart OvenMediaEngine.

OvenMediaEngine-Specific SEI Payload Format

The payload of SEI generated by OvenMediaEngine always includes UUID and Timestamp values in the following structure:

0                   1                   2                   3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| uuid_iso_iec_11578(128)                                       |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                                                               |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                                                               |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                                                               |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Timestamp (64)                                                |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                                                               |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Data (Payload Size - UUID(128) - Timestamp(64))               |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ...                                                           |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Field	Size (bits)	Description
UUID	128	Indicates that the SEI Payload follows OvenMediaEngine's specification. Always set to 464d4c47-5241-494e-434f-4c4f554201
Timestamp	64	Epoch time in milliseconds
Data	Varies depending on custom data	Custom data

Receiving SEI Data

OvenPlayer can parse SEI inserted by OvenMediaEngine and pass the included UUID, Timestamp, and custom data to the application.

Code Example

var player = OvenPlayer.create('player', {
  sources: [
    {
      type: 'webrtc', // Playing WebRTC stream
      file: 'wss://[YOUR_OvenMediaEngine]:3333/app/stream'
    }
  ],
  parseStream: {
    enabled: true // Enable H.264 NAL parsing
  }
});

function toAsciiString(byteArray) {
  return String.fromCharCode.apply(null, byteArray);
}

player.on('metaData', function (metadata) {
  console.log('MetaData:', metadata);
  /* Output:
    {
      type: 'sei',
      nalu: Unit8Array(33),
      sei: {
        type: 5,
        size: 39,
        payload: Unit8Array(39)
      },
      registered: true,
      uuid: '464d4c47-5241-494e-434f-4c4f-55524201',
      timecode: 1739851602778,
      userdata: Unit8Array(15)
    }
  */

  console.log(`Convert user data to string: ${toAsciiString(metadata.userdata)}`);
  /* Output:
    Convert user data to string: OvenMediaEngine
  */
});

Player Initialization:

• Call OvenPlayer.create() function to create a player in the specified div.

• Specify the stream type and URL in the sources array. SEI is only supported in WebRTC streams.

• Enable H.264 Network Abstraction Layer (NAL) parsing through parseStream.enabled: true setting. This is required for SEI metadata processing.

SEI Data Processing:

• Register a player.on('metaData', callback) event listener to process SEI whenever it is received.

• You can obtain the UUID, Timestamp, and custom data inserted by OvenMediaEngine from the event callback parameter.

metaData Event Callback Parameters

Field	Description
type	Always set to sei, indicating this is SEI metadata
nalu	Uint8Array containing raw Network Abstraction Layer Unit (NALU) data of the SEI
sei	SEI parsing result containing the following sub-fields: type: SEI type size: Payload size payload: Raw SEI payload data (Uint8Array)
registered	Indicates whether the SEI was generated in the format defined by OvenMediaEngine. If true, the following additional fields are included
uuid	(when registered=true) Unique identifier inserted by OvenMediaEngine into the SEI
timecode	(when registered=true) Timestamp (milliseconds) when the SEI was inserted
userdata	(when registered=true) Uint8Array containing custom data. This data should be parsed according to the application's requirements

Appendix: Installing OvenPlayer

OvenPlayer version 0.10.39 and later can receive SEI data included in the video during WebRTC stream playback.

1. OvenPlayer can be downloaded from the OvenPlayer GitHub.
2. After extracting the downloaded file, copy all the files in the dist directory to your project's library folder:

├─.github
├─demo
├─dist
│      ovenplayer.js
│      ovenplayer.js.map
│      RTCTransformWorker.worker.worker.js
│      RTCTransformWorker.worker.worker.js.map
├─docs
├─packages
└─src

Key files to copy:

• ovenplayer.js: OvenPlayer core library

• RTCTransformWorker.worker.worker.js: Worker script for WebRTC processing

Due to Web Worker CORS policy, ovenplayer.js must be self-hosted, and the RTCTransformWorker.worker.worker.js file must exist in the same path as ovenplayer.js.