C#-Uniform Render Streaming with WebRTC for Bidirectional Text Messaging from Unity Client to JavaScript Server

Enabling Real-Time Communication Between Unity and JavaScript Using WebRTC

With the growing demand for real-time communication in applications, developers are leveraging WebRTC to transmit video, audio, and data seamlessly. In this context, Unity Render Streaming offers a powerful framework to stream video from a JavaScript server to a Unity client. However, establishing bidirectional communication—such as sending text messages back from Unity to the JavaScript server—can be a challenge.

In this project, we aim to build a streaming app by using the Unity Render Streaming plugin along with the WebRTC protocol. While the video streaming part has been configured successfully, the next step involves enabling data exchange through an RTCDataChannel. This will allow text messages to be sent from the Unity client during active video streams.

We’ll explore how to modify existing code samples to meet these requirements. The JavaScript-based web app will manage the video streaming, while the Unity client will attempt to send a sample message such as "Hello World" back to the server. We’ll walk through the configuration challenges encountered and their potential solutions.

This article also discusses issues such as missing namespaces and unresolved references in Unity. We’ll assess if the current code is on the right track and provide solutions to fix common problems, like the error related to 'SingleConnection'. By the end, we aim to establish a smooth, bi-directional text messaging pipeline between Unity and JavaScript using WebRTC.

Building a Unity-to-JavaScript Messaging System with WebRTC

The goal of this project is to use the Unity Render Streaming plugin with WebRTC to transmit video from a JavaScript server to a Unity client, while also sending data messages back from Unity. The first step in the JavaScript script involves setting up a media stream. We use the `captureStream()` method to stream video content from an HTML5 element, and an RTCPeerConnection object manages the connection between the two peers. A DataChannel is created within this connection, which enables non-media data (like text) to be transmitted alongside the video stream.

On the Unity side, we initialize the WebRTC connection in the `ReceiverSample.cs` script by setting up a RTCDataChannel. This channel is responsible for both sending and receiving text data to the JavaScript server. A coroutine function is used to repeatedly send the message “Hello World” every two seconds, only if the channel state is "Open." When the DataChannel on the JavaScript server receives a message, it logs the content to the console to confirm the successful connection.

A critical issue highlighted during this process is the missing reference to the `SingleConnection` namespace in the Unity code, causing compilation errors. This suggests that either the required package is missing or there’s an incorrect dependency in the project configuration. To resolve this, we recommend verifying that all necessary dependencies, such as the Unity WebRTC package, are correctly installed and referenced. If the package is unavailable, the class may need to be replaced with a standard RTCPeerConnection object.

Finally, the error-handling mechanism for both scripts ensures that failed connections are reported in the console, helping with debugging. The Unity script includes a `StartCoroutine()` function to manage asynchronous messaging, which is useful in maintaining real-time data exchange without blocking other processes. We also use `OnOpen` and `OnClose` events to monitor the state of the DataChannel, ensuring that messages are only sent when the connection is stable. This modular setup ensures that the code can be extended or modified easily, and it provides a solid starting point for building more advanced streaming applications using Unity and WebRTC.

// sendvideo.js - JavaScript Server-Side Code
import * as Logger from "../../module/logger.js";
export class SendVideo {
  constructor(localVideoElement, remoteVideoElement) {
    this.localVideo = localVideoElement;
    this.remoteVideo = remoteVideoElement;
    this.peerConnection = new RTCPeerConnection();
    this.dataChannel = this.peerConnection.createDataChannel("myDataChannel");
  }
  async startLocalVideo() {
    const stream = document.createElement('video').captureStream();
    this.localVideo.srcObject = stream;
    await this.localVideo.play();
    this.peerConnection.createOffer().then(offer => {
      this.peerConnection.setLocalDescription(offer);
    });
  }
}

// ReceiverSample.cs - Unity Client Code
using System.Collections;
using UnityEngine;
using Unity.WebRTC;
public class ReceiverSample : MonoBehaviour {
  private RTCDataChannel dataChannel;
  void Start() { StartCoroutine(InitializeConnection()); }
  IEnumerator InitializeConnection() {
    var connection = new RTCPeerConnection();
    dataChannel = connection.CreateDataChannel("myDataChannel");
    dataChannel.OnOpen += OnChannelOpen;
    yield return null;
  }
  void OnChannelOpen() { StartCoroutine(SendMessageLoop()); }
  IEnumerator SendMessageLoop() {
    while (dataChannel.ReadyState == RTCDataChannelState.Open) {
      dataChannel.Send("Hello World");
      yield return new WaitForSeconds(2);
    }
  }
}

// Adjustments to avoid missing reference issues in ReceiverSample.cs
using Unity.WebRTC;
public class FixedReceiverSample : MonoBehaviour {
  private RTCPeerConnection peerConnection;
  private RTCDataChannel dataChannel;
  void Start() { InitializeWebRTC(); }
  void InitializeWebRTC() {
    peerConnection = new RTCPeerConnection();
    dataChannel = peerConnection.CreateDataChannel("myDataChannel");
    dataChannel.OnOpen += () => Debug.Log("DataChannel open!");
    peerConnection.CreateOffer().ContinueWith(offer => {
      peerConnection.SetLocalDescription(offer.Result);
    });
  }
}

// DataChannelTest.cs - Unity Unit Test
using NUnit.Framework;
using Unity.WebRTC;
public class DataChannelTest {
  [Test]
  public void TestDataChannelCommunication() {
    var connection = new RTCPeerConnection();
    var channel = connection.CreateDataChannel("testChannel");
    bool messageReceived = false;
    channel.OnMessage += message => {
      messageReceived = message == "Hello World";
    };
    channel.Send("Hello World");
    Assert.IsTrue(messageReceived);
  }
}

Exploring Data Transmission Challenges in Unity and WebRTC Streaming

In addition to video streaming, enabling data communication between a Unity client and a JavaScript server using WebRTC opens up new possibilities. However, implementing such communication isn't always straightforward, especially when you integrate the Unity Render Streaming plugin. A common issue encountered is properly setting up and managing the RTCDataChannel for seamless text communication. In our example, the Unity client should be able to send a "Hello World" message back to the JavaScript server. This step requires careful handling of both Unity’s scripting environment and WebRTC’s protocol nuances.

One important challenge involves dependency management in Unity. Errors like the missing `SingleConnection` namespace in our `ReceiverSample.cs` code highlight the need to ensure that all necessary plugins, including WebRTC, are installed correctly. A good practice here is to check for compatibility between the Unity version and the plugin version in use. The issue could also stem from outdated or missing Unity Render Streaming components, which need to be configured with the correct connection objects.

Beyond technical issues, an additional aspect to explore is the latency between Unity and JavaScript over WebRTC. While WebRTC provides low-latency communication, network conditions can still affect message delivery. Using coroutines within Unity allows for non-blocking message delivery, ensuring that sending text data (e.g., via `StartCoroutine`) doesn’t interrupt video streaming. Testing the stability of the connection through `RTCDataChannelState` is another key practice to ensure that messages are only sent when the channel is active. These strategies help optimize performance and ensure a better user experience in real-time applications.