Open-source IP camera libraries for Kotlin and Swift

Baby Zzz's talks to IP cameras straight from the phone: it finds them on your Wi-Fi, signs in over ONVIF, plays their RTSP stream, steers pan and tilt, and sends a parent's voice to the camera's speaker. We took that code out of our Android and iOS apps and published it as two libraries anyone can use, in their own apps or tools.

go2rtc-style camera protocols, as libraries

go2rtc is a popular streaming server written in Go that speaks a long list of camera protocols. kotlin2rtc and swift2rtc bring go2rtc-style camera protocols to Kotlin and Swift as embeddable libraries: there is no server to run, the code lives inside your app and talks to the camera directly.

They are independent projects. go2rtc and pytapo (a Python library for TP-Link Tapo cameras) were valuable protocol references while we wrote them, and we are grateful to their authors. Neither project is affiliated with Baby Zzz's or endorses it.

kotlin2rtc: ONVIF and RTSP for Android and the JVM

Plain Kotlin with coroutines, no Android dependency, so it runs in an Android app as well as on any JVM (bytecode for Java 17). Four modules:

• kotlin2rtc-onvif: WS-Discovery, the ONVIF device, media and PTZ client (WS-Security UsernameToken, HTTP Digest when the camera asks for it), the camera's speaker level, and a camera probe that checks a camera answers and is the same device before a login is sent.
• kotlin2rtc-rtsp: the RTSP audio backchannel (ONVIF two-way audio).
• kotlin2rtc-codec: G.711 (µ-law and A-law), G.726, AAC framing, L16, RTP and RTCP packetizing.
• kotlin2rtc-core: HTTP Digest, credentials that never print their password, and a local-network guard: SOAP and RTSP go in cleartext only to addresses on the device's own network.

Install

The first release (0.1.0) is being prepared for Maven Central; until it is out, build it from the repository.

// settings.gradle.kts: until 0.1.0 is on Maven Central, build it from a checkout next to your project
includeBuild("../kotlin2rtc")

// build.gradle.kts (group io.github.mafischer)
dependencies {
    implementation("io.github.mafischer:kotlin2rtc-onvif:0.1.0-SNAPSHOT")
    implementation("io.github.mafischer:kotlin2rtc-rtsp:0.1.0-SNAPSHOT")  // audio backchannel
    implementation("io.github.mafischer:kotlin2rtc-codec:0.1.0-SNAPSHOT") // G.711, G.726, AAC, L16, RTP
}

Find a camera and get its stream

import io.github.mafischer.kotlin2rtc.core.*
import io.github.mafischer.kotlin2rtc.onvif.*

// Only addresses on this device's subnets may be called (no cleartext SOAP to the internet)
val network = LocalNetwork(listOf(LinkSubnet.parse("192.168.1.20/24")))

// WS-Discovery: the ONVIF cameras that answer on this network
val cameras = WsDiscovery(accept = network::allows).discover(timeoutMillis = 3_000)

val onvif = OnvifClient(
    transport = LanHttpTransport(allows = network::allowsCamera),
    deviceUrl = cameras.first().deviceUrl,
    credentials = Credentials("camera-user", "camera-password"),
)
onvif.syncClock() // WS-Security timestamps follow the camera's clock
val endpoints = onvif.endpoints()
val profile = onvif.profiles(endpoints.mediaUrl).first { it.hasVideo }
val rtspUrl = onvif.streamUri(endpoints.mediaUrl, profile.token)

ONVIF two-way audio on Android and iOS

Many ONVIF cameras with a speaker accept sound from the app over RTSP: the audio backchannel of the ONVIF Streaming Specification, required by ONVIF Profile T and offered by many other cameras too. The client asks with a DESCRIBE carrying Require: www.onvif.org/ver20/backchannel; the camera lists its speaker as a sendonly audio media; the client sets it up over TCP interleaved, starts it with PLAY, then sends RTP audio in the camera's codec.

kotlin2rtc does all of that today, on Android and any JVM: it picks the best codec the camera offers (G.711, AAC-LC, G.726 at 40 to 16 kbit/s, L16), keeps the session alive and closes it cleanly. It is what drives push to talk and sleep sounds on IP cameras in the Baby Zzz's Android app. On iOS, the backchannel is planned for swift2rtc.

Push to talk in Kotlin (RTSP backchannel)

import io.github.mafischer.kotlin2rtc.codec.BackchannelEncoder
import io.github.mafischer.kotlin2rtc.rtsp.RtspBackchannelClient

// Blocking calls: run them on an IO thread. The login rides in the URL (kept in memory only).
val client = RtspBackchannelClient(
    url = "rtsp://camera-user:[email protected]:554/stream1",
    allows = network::allowsCamera,
)
// DESCRIBE with "Require: www.onvif.org/ver20/backchannel"
val media = client.describe().media ?: return // no speaker kotlin2rtc can send to
val encoder = BackchannelEncoder(media)        // G.711, G.726, AAC or L16, as the camera asked
encoder.microphoneRate(16_000)
val link = client.open()                       // SETUP (TCP interleaved) + PLAY
try {
    while (talking) {
        val block: FloatArray = readMicrophone(16_000 / 50) // one 20-millisecond block, -1..1
        encoder.packets(sound = null, microphone = block).forEach(link::sendRtp)
    }
} finally {
    link.close() // TEARDOWN
}

swift2rtc: ONVIF and RTSP for iOS and macOS

A Swift package (Swift 6, iOS 17 and macOS 14) with three products:

• Swift2RTCOnvif: WS-Discovery, the ONVIF device, media and PTZ client, PTZ control with optical zoom, and the camera probe.
• Swift2RTCRTSP: an RTSP client: RTSP messages and SDP, RTP depacketizing of H.264, H.265, AAC and G.711, and decoding with VideoToolbox and AudioToolbox.
• Swift2RTCCore: HTTP Digest, credentials, a small socket HTTP transport and the same local-network guard.

The audio backchannel (two-way audio) is planned.

Install with Swift Package Manager

// Package.swift
dependencies: [
    .package(url: "https://github.com/mafischer/swift2rtc.git", branch: "main"), // a version tag comes with 0.1.0
],
targets: [
    .target(name: "MyApp", dependencies: [
        .product(name: "Swift2RTCOnvif", package: "swift2rtc"),
        .product(name: "Swift2RTCRTSP", package: "swift2rtc"),
    ]),
]

Find a camera and get its stream

import Swift2RTCOnvif

// Only private (local) addresses may be called
let isLocal: @Sendable (InetAddress) -> Bool = { $0.isPrivate }

let cameras = try await WSDiscovery(accept: isLocal).discover(timeout: 3)
let onvif = OnvifClient(
    transport: SocketLanTransport(allows: isLocal),
    deviceURL: cameras[0].deviceURL,
    credentials: OnvifCredentials(username: "camera-user", password: "camera-password")
)
_ = try await onvif.syncClock()
let endpoints = try await onvif.endpoints()
let profiles = try await onvif.profiles(mediaURL: endpoints.mediaURL)
if let profile = profiles.first(where: { $0.hasVideo }) {
    let rtspURL = try await onvif.streamURI(mediaURL: endpoints.mediaURL, profileToken: profile.token)
}

From RTP packets to pictures

import Swift2RTCRTSP

let sdp = SDP.parse(describeAnswer)           // the camera's DESCRIBE body
var video = VideoDepacketizer(codec: .h264)    // or .h265
if let packet = RTPPacket.parse(interleavedPayload) {
    for unit in video.push(packet) {
        // unit.nalUnits: one picture, ready for a VideoToolbox decompression session
    }
}

Tapo two-way audio without the Tapo app (coming)

TP-Link Tapo cameras such as the Tapo C210 offer ONVIF and RTSP for video, but their speaker does not take sound over the ONVIF backchannel: the Tapo app uses TP-Link's own protocol for it. The Baby Zzz's app for Android and Baby Zzz's Desktop already talk to Tapo speakers with that protocol, in their own code, and we plan to add it to both libraries.

Why libraries instead of a server

Servers like go2rtc are great when you run a home lab: one box on your network talks to every camera and restreams them. A baby monitor app has different needs:

• No extra box for parents to set up: the phone talks to the camera itself.
• Camera logins stay on the device: they are not handed to another server.
• The app controls the session: it opens the stream when a parent watches and closes it when they stop, which matters for battery and for the camera's limited number of connections.
• A small attack surface: nothing listens on the network, and the local-network guard refuses to send cleartext to any address outside the device's own network.

Used in Baby Zzz's

The Baby Zzz's Android app uses this code to add IP cameras, play them next to phone cameras, move PTZ cameras, and talk or play sleep sounds through the camera's speaker. The iOS app, coming soon, uses the Swift code. See how to use a Tapo or ONVIF camera as a baby monitor and the app's features.

Licence and contributions

Both libraries are released under the ISC licence. Bug reports, camera compatibility notes and pull requests are welcome on GitHub: kotlin2rtc issues and swift2rtc issues.