January 26, 2023
Protocols are nice
Protocols are pretty awesome. When developing Kompetansia we ran into a particular problem that was solved very elegantly via protocols.
Figure 1. Protocols are forever
For one of the apps, we needed an app that could run on both iPads and Chromebooks. And this is where the protocols come in and shine. While the app is written in Clojurescript, the container it runs in differs between iPad (Cordova based) and Chromebook (HTML5 based). So we need to capture sound, but the APIs to do so are different.
The protocol to capture sound
For Cordova we needed two protocols, while HTML5 Audio API only required one.
(defprotocol IAudio
(start-recording [this])
(stop-recording [this])
(cancel-recording [this])
(recording? [this]))
(defprotocol IRecording
(save-recording! [this data]))Cordova implementation
For Cordova we used a plugin, which handle the recording as a separate thing compared to the HTML5 version. There are a lot of callbacks, event listeners and other good stuff that is part of the js eco system, with all its glorious complexity hidden away by a nice, clean protocol.
(ns audio.cordova
(:require [audio.protocol]
[promesa.core :as p]
[re-frame.core :as rf]
[taoensso.timbre :as log]))
(defn on-audio-input-capture [recording]
(fn [event]
(try
(if (and event (.-data event))
(audio.protocol/save-recording! recording (.-data event))
(log/error ::on-audio-input-capture-invalid-event event))
(catch js/Error e
(log/error ::on-audio-input-capture e)))))
(defn on-audio-input-error [error]
(log/error error)
(rf/dispatch [:audio/status :error error]))
(defn cordova-start-capture [{:keys [audio-input recording config status] :as cordova-audio}]
(try
(if (and audio-input (not (.isCapturing ^js audio-input)))
(do
(reset! status ::started)
(log/debug "Starting sound capture")
(.addEventListener js/window "audioinput" (:on-audio-input-capture recording) false)
(.addEventListener js/window "audioinputerror" (:on-audio-input-error recording) false)
(device-ready audio-input config))
(log/warn "audio-input is capturing or is non-existant"))
(catch js/Error e
(log/error ::cordova-start-capture e))))
(defn cordova-stop-capture [{:keys [audio-input recording done-fn status] :as cordova-audio} canceled?]
(try
(if (and audio-input (.isCapturing ^js audio-input))
(do
(if canceled?
(log/debug "Canceling sound capture")
(log/debug "Stopping sound capture"))
(reset! status ::stopped)
(.stop ^js audio-input)
(.removeEventListener js/window "audioinput" (:on-audio-input-capture recording))
(.removeEventListener js/window "audioinputerror" (:on-audio-input-error recording))
(let [encoder (js/WavAudioEncoder.
(.-sampleRate (.getCfg audio-input))
(.-channels (.getCfg audio-input)))
blob (atom nil)
reader (js/FileReader.)]
(set! (.-onloadend reader) #(do (reset!
;; we reset the atom for capturing
;; base64 audio/wav data
(:wav-data recording)
;; the result in the reader starts with
;; data:audio/wav;base64, followed by the
;; base64 data. Here we cut off the
;; everything but the base64 audio data
(subs (.-result reader) 22))
(when (and done-fn
(not canceled?))
(done-fn cordova-audio))))
(p/do!
(.encode encoder #js [@(:data recording)])
(reset! blob (.finish encoder "audio/wav"))
(.readAsDataURL reader @blob))))
(log/warn "audio-input is not capturing or is non-existant"))
(catch js/Error e
(log/error ::cordova-stop-capture e))))
(defrecord Recording [data data-received-tally on-audio-input-capture on-audio-input-error wav-data]
audio.protocol/IRecording
(save-recording! [this chunk]
(swap! data-received-tally + (.-length chunk))
(let [new-data (js/Float32Array. @data-received-tally)]
(.set new-data @data 0)
(.set new-data chunk (- @data-received-tally (.-length chunk)))
(reset! data new-data))))
(defn recording [settings]
(let [r (map->Recording (assoc settings :wav-data (atom nil)))
-on-audio-input-capture (on-audio-input-capture r)]
(assoc r
:on-audio-input-capture -on-audio-input-capture
:on-audio-input-error on-audio-input-error)))
(defrecord Audio [status audio-input config recording container done-fn]
audio.protocol/IAudio
(start-recording [this]
(log/info "Starting audio recording")
(cordova-start-capture this))
(stop-recording [this]
(log/info "Stopping audio recording")
(cordova-stop-capture this false))
(cancel-recording [this]
(log/info "Canceling audio recording")
(cordova-stop-capture this true))
(recording? [this]
(and audio-input (.isCapturing ^js audio-input))))
(defn audio [config done-fn]
(let [-recording (recording {:data (atom (js/Float32Array. 0)) :data-received-tally (atom 0)})]
(map->Audio {:audio-input js/window.audioinput
:recording -recording
:container "wav"
:config config
:status (atom ::not-started)
:done-fn done-fn})))HTML5 implementation
The HTML5 implementation was a bit simpler to implement than Cordova.
(ns audio.browser
(:require [audio.protocol]
[goog.object :as gobj]
[re-frame.core :as rf]
[taoensso.timbre :as log]))
(defonce state (atom {:stream nil}))
(defn ->media-stream []
(let [constraints #js {:audio true
:video false}]
(-> (js/Promise.resolve (.getUserMedia (-> js/window .-navigator .-mediaDevices) constraints))
(.then #(if-not (:stream @state)
(swap! state assoc :stream %)
(log/warn ::get-media-stream "Stream already acquired")))
(.catch #(log/error %)))))
(defn- start-recording*
[{:keys [recorder config recording done-fn canceled?] :as audio}]
(log/debug ::start-recording* recording)
(try
(set! (.-ondataavailable recorder) (fn [e]
(let [reader (js/FileReader.)]
(set! (.-onloadend reader)
#(do
(reset! (:wav-data recording) (subs (.-result reader) 35))
(when (and done-fn
(not @canceled?))
(done-fn audio))))
(.readAsDataURL reader (-> e .-data)))))
(.start recorder)
(catch js/Error e (log/error ::start-recording e))))
(defn- stop-recording*
[{:keys [recorder canceled?] :as recorder}]
(try
(reset! canceled? false)
(.stop recorder)
(catch js/Error e (log/error ::stop-recording e))) )
(defn- cancel-recording*
[{:keys [recorder canceled?] :as recorder}]
(try
(reset! canceled? true)
(.stop recorder)
(catch js/Error e (log/error ::cancel-recording e))) )
(defrecord MediaRecorder [config recorder container done-fn canceled?]
audio.protocol/IAudio
(start-recording [this]
(log/info "Starting audio recording")
(start-recording* this))
(stop-recording [this]
(log/info "Stopping audio recording")
(stop-recording* this))
(cancel-recording [this]
(log/info "Canceling audio recording")
(cancel-recording* this))
(recording? [this]
(case (.-state recorder)
"recording" true
false)))
(defn media-recorder [config done-fn]
(let [-recorder (js/MediaRecorder. (get-in @state [:stream]) config)]
(map->MediaRecorder {:recorder -recorder
:config config
:canceled? (atom false)
:recording {:wav-data (atom nil)}
:container "webm"
:done-fn done-fn})))Dummy implementation
And since it’s so cheap, and we sometimes just want to work with the logic, without actually recording anything, we can do a dummy implementation.
(ns audio.dummy
(:require [audio.protocol]
[taoensso.timbre :as log]))
(defrecord Recording []
audio.protocol/IRecording
(save-recording! [this chunk]))
(defn recording [settings]
(map->Recording {}))
(defrecord Audio [status config recording done-fn]
audio.protocol/IAudio
(start-recording [this]
(log/info "Starting dummy audio recording")
(reset! status ::started))
(stop-recording [this]
(log/info "Stopping dummy audio recording")
(reset! status ::stopped))
(recording? [this]
(= @status ::started)))
(defn audio [config done-fn]
(let [-recording (recording {})]
(map->Audio {:recording -recording
:config config
:status (atom ::not-started)
:done-fn done-fn})))Helper namespace
With 3 namespaces (and maybe more to come), we can further hide the implementation details behind a helper namespace.
(ns audio
(:require [audio.browser]
[audio.cordova]
[audio.dummy]
[audio.protocol]
[common.config :refer [config]]
[taoensso.timbre :as log]))
(defn recorder
([type] (recorder :dummy
nil
(fn [audio]
(log/info "Audio is finished"))))
([type config done-fn]
(case (keyword type)
:cordova (audio.cordova/audio config done-fn)
:browser (audio.browser/media-recorder config done-fn)
(audio.dummy/audio config done-fn))))
(defn start-recording [recorder]
(audio.protocol/start-recording recorder))
(defn stop-recording [recorder]
(audio.protocol/stop-recording recorder))
(defn cancel-recording [recorder]
(audio.protocol/cancel-recording recorder))
(defn recording? [recorder]
(audio.protocol/recording? recorder))
(defn browser? []
(= "browser" (get-in @config [:audio :type])))
(defn cordova? []
(= "cordova" (get-in @config [:audio :type])))Usage
And with this we have now hidden most of the complexity and can now do this. We can extend the protocol with other implementations should the need ever arise, and the code using the implementation would be none the wiser.
A similar protocol for playing sound was also done, as it turns out the HTML5 tag for sound is a bit limited in controlling how and when the sound is played.
Anway… awesome, right?
(defn send-off-audio [url id]
(fn [audio]
(let [payload {:id id
:audio/base64 @(get-in audio [:recording :wav-data])
:audio/container (get-in audio [:container])}]
(POST url
{... other parms here
:body payload
... other params here}))))
(def audio-type "cordova")
(def recorder (audio/recorder audio-type
(case audio-type
"cordova" #js {:audioSourceType 1}
"browser" #js {:mimeType "audio/webm;codecs=opus"})
(send-off-audio "https://example.com" 1)))
(audio/start-recording recorder)
(js/setTimeout (fn [_]
(audio/stop-recording recorder))
3000)
(log/info "Are we recording yet?" {:recording? (audio/recording? recorder)})