wiki:104hackathon/webrtc

Version 7 (modified by lorenzo@…, 9 months ago) (diff)

--

WebRTC Interop (KITE) and Compliance (WPT) testing

  • Goals
    • Fix browsers to support Simulcast 1.0 better,
    • thus passing more compliance tests,
    • and interoperate better with other browser, and here specifically with SFUs
  • Champion(s)
    • Harald Alvestrand <harald at alvestrand.no>,
    • "Dr Alex" Gouaillard <alex.gouaillard at cosmosoftware.io>
  • Browser vendors with Staff on site to fix on-the-fly
    • Google (Harald, Henrik, Marina ...)
    • Apple (Youenn F.)
    • Mozilla (Nils O., ...)
    • Microsoft (Nikita)
  • SFU Vendors with staff on site to fix on-the-fly
    • Janus (lorenzo)
    • Jitsi (emil, boris, or saul)
    • meedoze (Sergio)
    • others ?
  • Project(s)
    • Add more tests to the Web Platform Tests suite for webrtc 1.0 Simulcast API Compliance
    • Add more tests to the KITE webrtc 1.0 simulcast interop test suite
    • Specifically Test browser interoperability with SFUs and MCUs in Simulcast Mode
    • FIx browsers, rince, repeat
    • fix SFUs, rince repeat

Specifically for this event, we created a gitHub repository with two automated kite interoperability tests. https://github.com/ManuCosmo/KITE-Hackathon

One test is a "typical" SFU test: KITE-Janus-Test is provided, which can be easily adapted to test any SFU, and should be the starting point for SFU developers wanting to automatically test against all the browser configuration CoSMo will provide for testing that week end.

One test, "KITE-Simulcast-Test", was specifically developed to test browser compliance. An app which sends simulcast to a special SFU based on meedoze, and receives back a given stream back depending on interaction with the UI is provided. It allows to test in loopback mode with a single browser (since simulcast is only sender side anyway), and all stream selection logic is made through UI button which makes it easily instrumented through KITE. The different simulcast features are not tested directly but indirectly (if I can select and receive a stream at a specific configuration it means the SFU receives them well and can differentiate them). The medooze simulcast loopback page, used by e.g. Apple for testing is hosted here: https://playground.cosmosoftware.io/index.html

To test Janus, a couple of servers are available:

The easier way to test simulcasting is the EchoTest?, which will allow you to choose which layers to send back. A couple of query strings are available to enable simulcast and force a specific codec: simulcast=true will enable old-style simulcasting (SDP munging for Chrome and Safari, rid-based for Firefox), while simulcast2=true will enable the new rid-based simulcasting on Chrome M74 and M75; vcodec=X forces a specific codec (e.g., vcodec=h264). Here's a couple of examples:

The videoconferencing demo can be used as well: visit the videoroomtest.html page instead. The same query strings for simulcast apply; there's no changing the video codec though (defaults to VP8, we can add a new room for other codecs).

We plan to extend and complete this document over the hackathon with each and every new tests the SFU vendors will propose, or modified browsers the browsers vendor will provide: https://docs.google.com/spreadsheets/d/1F5iJ1Qs8AgVTlgdC1ZRhSVHRvTM46AW3r1_9oQhZiMc/edit#gid=0

Interesting links and prerequisite reading: