Category Archives: Research

These posts are dedicated to my research.

Exploring how speech air flow may impact the spread of airborne diseases

I am participating on an American Association for the Advancement of Science (AAAS) 2022 meeting panel on “Transmission of Airborne Pathogens through Expiratory Activities” on Friday, February 18th from 6:00 to 6:45 AM Greenwich mean time. You can register for the meeting by clicking here. In advance of that meeting, the University of British Columbia asked me some Q&A questions exploring how speech air flow may impact the spread of airborne diseases.

The AAAS meeting itself is hosted by Prof. Bryan Gick of the University of British Columbia. It has individual talks by Dr. Sima Asadi on “Respiratory behavior and aerosol particles in airborne pathogen transmission”, Dr. Nicole M. Bouvier on “Talking about respiratory infectious disease transmission”, and myself on “Human airflow while breathing, speaking, and singing with and without masks”.

Dr. Sima Asadi’s talk focuses on the particles emitted during human speech, and the efficacy of masks in controlling their outward emission. For this work, Sima received the Zuhair A. Munir Award for the Best Doctoral Dissertation in Engineering from UC Davis in 2021. She is currently a postdoctoral associate in Chemical Engineering at MIT (Boston).

Dr. (Prof) Nicole M. Bouvier is an associate professor of Medicine and Infectious Diseases and Microbiology at the Icahn School of Medicine at Mount Sinai (New York). Nichole discusses how we understand the roots by which respiratory microorganisms, like viruses and bacteria, transmit between humans, which is fundamental in how we develop both medical and public health countermeasures to reduce or prevent their spread. However, much of what we think we know is based on evidence that is incomplete at best, and full of confusing terminology, as the current COVID-19 pandemic has made abundantly clear.

I myself am new to airborne transmission research, coming instead from the perspective that visual and aero-tactile speech help with speech perception, and so masks would naturally interfere with clear communication. They would do this by potentially muffling some speech sounds, but mostly by cutting off the perceiver form visual and even tactile speech signals.

However, since my natural interests involve speech air flow, I was ideally suited to move into research studying how these same air flows may be reduced or eliminated by face masks. I conduct this research with a Mechanical Engineering team at the University of Canterbury, and some of their results are featured in my individual presentation. Our most recent publication on Speech air flow with and without face masks was highlighted in previous posts on Maps of Speech, and in a YouTube video found here.

Speech air flow with and without face masks

It took a while due to the absolutely shocking amount of work required for the “Gait change in tongue movement” article, but Natalia Kabaliuk, Luke Longworth, Peiman Pishyar‑Dehkordi, Mark Jermy and I were able to get our article on “Speech air flow with and without face masks” accepted to Scientific Reports (Nature Publishing Group). The article is now out (though a pre-review version had been available since we submitted this article to Sci Rep). You can also watch my YouTube video describing many of the results.

Here is an example of a low-stiffness air-flow from a porous mask, which allows leaks from the tops, bottoms, and sides, and forward flow prevention, as taken from Figure 5 of the article.

Figure 5. Audio and Schlieren of speech through a porous face mask (Frame 621, 1st block, CORI Supermask). Image from 88 ms after the release burst for the [kh ] in “loch”. Note that the k’s puff is smoother and less well defined than the one in Fig. 2, but still has eddies that change air-density across the span of the puff. The red-dashed line in the audio waveform indicates the timing of the schlieren frame.

And here is an example of typical higher-stiffness flow from a less porous mask from Figure 8.

Figure 8. Audio and Schlieren of speech with a tightly fitting surgical mask (Frame 334, 1st block, Henry Schlein surgical mask [level 2]). Air slowly flows out above the eyes, floating out and upward continuously. The red-dashed line in the audio waveform indicates the timing of the schlieren frame.

Masks can be made to fit tighter, as in well-designed KN95/N95 masks and masks with metal strips at the nose to prevent upward-escaping air flow. However, for all the masks we studied, the tradeoff was not entirely avoided. And with that, here is our abstract:

Face masks slow exhaled air flow and sequester exhaled particles. There are many types of face masks on the market today, each having widely varying fits, filtering, and air redirection characteristics. While particle filtration and flow resistance from masks has been well studied, their effects on speech air flow has not. We built a schlieren system and recorded speech air flow with 14 different face masks, comparing it to mask-less speech. All of the face masks reduced air flow from speech, but some allowed air flow features to reach further than 40 cm from a speaker’s lips and nose within a few seconds, and all the face masks allowed some air to escape above the nose. Evidence from available literature shows that distancing and ventilation in higher-risk indoor environment provide more benefit than wearing a face mask. Our own research shows all the masks we tested provide some additional benefit of restricting air flow from a speaker. However, well-fitted mask specifically designed for the purpose of preventing the spread of disease reduce air flow the most. Future research will study the effects of face masks on speech communication in order to facilitate cost/benefit
analysis of mask usage in various environments.

Gait Change in Tongue Movement

Bryan Gick and I recently published an article on “Gait Change in Tongue Movement” in Scientific Reports (Nature Publishing Group). Below is the abstract, with images alongside. However, if you want an easy-to-follow walkthrough of the paper, I also published a YouTube video on the paper on my YouTube Channel for Maps of Speech.

During locomotion, humans switch gaits from walking to running, and horses from walking to trotting to cantering to galloping, as they increase their movement rate. It is unknown whether gait change leading to a wider movement rate range is limited to locomotive-type behaviours, or instead is a general property of any rate-varying motor system. The tongue during speech provides a motor system that can address this gap. In controlled speech experiments, using phrases containing complex tongue-movement sequences, we demonstrate distinct gaits in tongue movement at different speech rates. As speakers widen their tongue-front displacement range, they gain access to wider speech-rate ranges.

At the widest displacement ranges, speakers also produce categorically different patterns for their slowest and fastest speech. Speakers with the narrowest tongue-front displacement ranges show one stable speech-gait pattern, and speakers with widest ranges show two. Critical fluctuation analysis of tongue motion over the time-course of speech revealed these speakers used greater effort at the beginning of phrases—such end-state-comfort effects indicate speech planning.

Based on these findings, we expect that categorical motion solutions may emerge in any motor system, providing that system with access to wider movement-rate ranges.

Evidence for active control of tongue lateralization in Australian English /l/

Jia Ying, Jason A. Shaw, Christopher Carignan, Michael Proctor, myself, and Catherine T. Best just published Evidence for active control of tongue lateralization in Australian English /l/. Most research on /l/ articulation has looked at motion timing along the midline, or midsagittal plane. This study compares that information to motion on the sides of the tongue. It focuses on Australian English (AusE), using three-dimensional electromagnetic articulography (3D EMA).

Fig. 11. Temporal dynamics of tongue curvature in the coronal plane over the entire V-/l/ interval. The brackets indicate onset (red) and coda (blue) /l/ intervals. Each bracket extends from the /l/ onset to its peak. For onset /l/s, the peak occurs earlier (at about 200 ms) than coda /l/s (at about 450 ms). A time of zero indicates the vowel onset. The 800-interval window captures the entire V-/l/ articulation in every token.

The articulatory analyses show: 1) consistent with past work, the timing lag between mid-sagittal tongue tip and tongue body gestures differs for syllable onsets and codas, and for different vowels.

2) The lateral channel is formed by tilting the tongue to the left/right side of the oral cavity as opposed to curving the tongue within the coronal plane

3) the timing of lateral channel formation relative to the tongue body gesture is consistent across syllable positions and vowel contexts – even though temporal lag between tongue tip and tongue body gestures varies.

This last result suggests that lateral channel formation is actively controlled as opposed to resulting as a passive consequence of tongue stretching. These results are interpreted as evidence that the formation of the lateral channel is a primary articulatory goal of /l/ production in AusE.

Locating de-lateralization in the pathway of sound changes affecting coda /l/

Patrycja Strycharczuk, Jason Shaw, and I just published Locating de-lateralization in the pathway of sound changes affecting coda /l/, in which we analyze New Zealand English /l/ using Ultrasound and Articulometry. You can find the article here. Put in the simplest English terms, the article shows the process by which /l/-sounds in speech can change over time from a light /l/ (like the first /l/ in ‘lull’) to a darker /l/ (like the second /l/ in ‘lull’). This darkening is the result of the upper-back, or dorsum, of the tongue moving closer to the back of the throat. This motion in turn reduces lateralization, or the lowering of the sides of the tongue away from the upper teeth. This is followed, over time, by the tongue tip no longer connecting to the front of the hard palate – the /l/ becomes a back vowel or vocalizes.

Two subcategories identified in the distribution of TT raising for the Vl#C context. Red = vocalized.

If you want a more technical description, Here is the abstract:

‘Vocalization’ is a label commonly used to describe an ongoing change in progress affecting coda /l/ in multiple accents of English. The label is directly linked to the loss of consonantal constriction observed in this process, but it also implicitly signals a specific type of change affecting manner of articulation from consonant to vowel, which involves loss of tongue lateralization, the defining property of lateral sounds. In this study, we consider two potential diachronic pathways of change: an abrupt loss of lateralization which follows from the loss of apical constriction, versus slower gradual loss of lateralization that tracks the articulatory changes to the dorsal component of /l/. We present articulatory data from seven speakers of New Zealand English, acquired using a combination of midsagittal and lateral EMA, as well as midsagittal ultrasound. Different stages of sound change are reconstructed through synchronic variation between light, dark, and vocalized /l/, induced by systematic manipulation of the segmental and morphosyntactic environment, and complemented by comparison of different individual articulatory strategies. Our data show a systematic reduction in lateralization that is conditioned by increasing degrees of /l/-darkening and /l/-vocalization. This observation supports the idea of a gradual diachronic shift and the following pathway of change: /l/-darkening, driven by the dorsal gesture, precipitates some loss of lateralization, which is followed by loss of the apical gesture. This pathway indicates that loss of lateralization is an integral component in the changes in manner of articulation of /l/ from consonantal to vocalic.

Native language influence on brass instrument performance

Matthias Heyne, myself, and Jalal Al-Tamimi recently published Native language influence on brass instrument performance: An application of generalized additive mixed models (GAMMs) to midsagittal ultrasound images of the tongue. The paper contains the bulk of the results form Matthias’ PhD Dissertation. The study is huge, with ultrasound tongue recordings of 10 New Zealand English (NZE) and 10 Tongan trombone players. There are 12,256 individual tongue contours of vowel tokens (7,834 for NZE, 4,422 for Tongan) and 7,428 individual tongue contours of sustained note production (3,715 for NZE, 3,713 for Tongan).

Figure 4 in the paper.

The results show that native language influences tongue position during Trombone note production. This includes tongue position and note variability. The results also support Dispersion Theory (Liljencrants and Lindblom 1972; Lindblom, 1986; Al-Tamimi and Ferragne, 2005) in that vowel production is more variable in Tongan, which has few vowels, then in NZE, which has many.

The results also show that note production at the back of the tongue maps to low-back vowel production (schwa and ‘lot’ for NZE, /o/ and /u/ for schwa). These two result sets support an analysis of local optimization with semi-independent tongue regions (Ganesh et al., 2010, Loeb, 2012).

The results do not, however, support the traditional brass pedagogy hypothesis that higher notes are played with a closer (higher) tongue position. However, Matthias is currently working with MRI data that *does* support the brass pedagogy hypothesis, and that we might not have seen this because of the ultrasound transducer stabilization system needed to keep the ultrasound probe aligned to the participant’s head.

Liljencrants, Johan, and Björn Lindblom. 1972. “Numerical Simulation of Vowel Quality Systems: The Role of Perceptual Contrast.” Language, 839–62.

Lindblom, Björn. 1963. Spectrographic study of vowel reduction. The Journal of the Acoustical Society of America 35(11): 1773–1781.

Al-Tamimi, J., and Ferragne, E. 2005. “Does vowel space size depend on language vowel inventories? Evidence from two Arabic dialects and French,” in Proceedings of the Ninth European Conference on Speech Communication and Technology, Lisbon, 2465–2468.

Ganesh, Gowrishankar, Masahiko Haruno, Mitsuo Kawato, and Etienne Burdet. 2010. “Motor Memory and Local Minimization of Error and Effort, Not Global Optimization, Determine Motor Behavior.” Journal of Neurophysiology 104 (1): 382–90.

Loeb, Gerald E. 2012. “Optimal Isn’t Good Enough.” Biological Cybernetics 106 (11–12): 757–65.

Tri-modal speech: Audio-visual-tactile integration in speech perception

Figure 3 in paper.

Myself, Doreen Hansmann, and Catherine Theys just published our article on “Tri-modal Speech: Audio-visual-tactile Integration in Speech Perception” in the Journal of the Acoustical Society of America. This paper was also presented as a poster at the American Speech-Language-Hearing Association (ASHA) Annual Convention in Orlando, Florida, November 21-22, 2019, winning a meritorious poster award.

TL-DR; People use auditory, visual, and tactile speech information to accurately identify syllables in noise. Auditory speech information is the most important, then visual information, and lastly aero-tactile information – but we can use them all at once.

Abstract

Speech perception is a multi-sensory experience. Visual information enhances (Sumby and Pollack, 1954) and interferes (McGurk and MacDonald, 1976) with speech perception. Similarly, tactile information, transmitted by puffs of air arriving at the skin and aligned with speech audio, alters (Gick and Derrick, 2009) auditory speech perception in noise. It has also been shown that aero-tactile information influences visual speech perception when an auditory signal is absent (Derrick, Bicevskis, and Gick, 2019a). However, researchers have not yet identified the combined influence of aero-tactile, visual, and auditory information on speech perception. The effects of matching and mismatching visual and tactile speech on two-way forced-choice auditory syllable-in-noise classification tasks were tested. The results showed that both visual and tactile information altered the signal-to-noise threshold for accurate identification of auditory signals. Similar to previous studies, the visual component has a strong influence on auditory syllable-in-noise identification, as evidenced by a 28.04 dB improvement in SNR between matching and mismatching visual stimulus presentations. In comparison, the tactile component had a small influence resulting in a 1.58 dB SNR match-mismatch range. The effects of both the audio and tactile information were shown to be additive.

Derrick, D., Bicevskis, K., and Gick, B. (2019a). “Visual-tactile speech perception and the autism quotient,” Frontiers in Communication – Language Sciences 3(61), 1–11, doi: http://dx.doi.org/10.3389/fcomm.2018.00061

Gick, B., and Derrick, D. (2009). “Aero-tactile integration in speech perception,” Nature 462, 502–504, doi: https://doi.org/10.1038/nature08572.

McGurk, H., and MacDonald, J. (1976). “Hearing lips and seeing voices,” Nature 264, 746–748, doi: http://dx.doi.org/https://doi.org/10.1038/264746a0

“Articulatory Phonetics” Resources

Back in 2013, Bryan Gick, Ian Wilson and myself published a textbook on “Articulatory Phonetics”. This book contained many assignments at the end of each chapter and in supplementary resources. After many years of using those assignments, Bryan, Ian, and many colleagues figured out that they needed some serious updating.

These updates have been completed, and are available here. The link includes recommended lab tools to use while teaching from this book, as well as links and external resources.

P.S. Don’t get too excited students – I’m not posting the answers here or anywhere else 😉

Aero-tactile integration during speech perception: Effect of response and stimulus characteristics on syllable identification

Jilcy Madappallimattam, Catherine Theys and I recently published an article demonstrating that aero-tactile stimuli does not enhance speech perception during open-choice experiments the way it does during two-way forced-choice experiments.

Abstract

Integration of auditory and aero-tactile information during speech perception has been documented during two-way closed-choice syllable classification tasks (Gick and Derrick, 2009), but not during an open-choice task using continuous speech perception (Derrick et al., 2016). This study was designed to compare audio-tactile integration during open-choice perception of individual syllables. In addition, this study aimed to compare the effects of place and manner of articulation. Thirty-four untrained participants identified syllables in both auditory-only and audio-tactile conditions in an open-choice paradigm. In addition, forty participants performed a closed-choice perception experiment to allow direct comparison between these two response-type paradigms. Adaptive staircases, as noted by Watson (1983). Were used to identify the signal-to-noise ratio for identification accuracy thresholds. The results showed no significant effect of air flow on syllable identification accuracy during the open-choice task, but found a bias towards voiceless identification of labials, and towards voiced identification of velars. Comparison of the open-choice results to those of the closed-choice task show a significant difference between both response types, with audio-tactile integration shown in the closed-choice task, but not in the open-choice task. These results suggest that aero-tactile enhancement of speech perception is dependent on response type demands.

Derrick, D., O’Beirne, G. A., De Rybel, T., Hay, J., and Fiasson, R. (2016). “Effects of aero-tactile stimuli on continuous speech perception,” Journal of the Acoustical Society of America, 140(4), 3225.

Gick, B., and Derrick, D. (2009). “Aero-tactile integration in speech perception,” Nature 462, 502–504.

Watson, A. B. (1983). “QUEST: A Bayesian adaptive psychometric method,” Perceptual Psychophysics, 33(2), 113–120.

Ultrasound Transducer Stabilizer for Children.

Our three-dimensional printable ultrasound transducer stabilizer has been a huge success. It is in use here at the University of Canterbury, as well as the University of Michigan, Hiroshima University, University of California, Los Angeles, and soon at the University of British Columbia. (And it is available at Western Sydney University).

However, Phil Hoole at Ludwig Maximilian University of Munich figured out that the transducer stabilizer does *not* work with Children. He developed a solution to that problem, and I am making it available here. Within this zip file, there is a new probe holder. The base and clip-holder should be printed as is. Each remaining file needs to be scaled to 75% of their size and then printed. Each file marked with X2 needs to be printed *twice*.

I will put photos of this version of the probe-holder online once I have printed new copies and sewn all the pieces together sometime in October.