Multimodal System for Audio Scene Source Counting and Analysis

<p>This thesis explores audio scene analysis (ASA) for determining the number of active sources in an audio scene, a task that is defined as audio source counting. A first of its kind dataset called SARdB is produced with audio and text modalities, and annotations for the number of speakers and the number of sound events present in an audio recording. For speaker counting, an audio-based ResNet-34 and text-based Bidirectional Long Short-Term Memory (BLSTM) network set a baseline prediction accuracy of 46.03% and 89.57% when considering a margin of error of one speaker, while outperforming various state-of-the-art systems in speaker counting. Another audio-based ResNet-34 model demonstrates the optimal result for sound event counting at 50.55% prediction accuracy and 86.59% accuracy with a margin of error of one sound event. The proposed method for source counting is also shown to perform in real-time with an overall processing time of ∼0.4614s.</p>