Hampshire Student Seeks to Improve Accessibility in Basketball with a Speech–to–Sign Language Algorithm

For his Div II, Rupert Tawiah-Quashie F22 designed a speech–to–sign language algorithm. We spoke with him about his project’s genesis, the mentors who guided him, the grants that supported his vision, and his commitment to greater inclusivity in sports.

How did you first get interested in this project?

My uncle and cousin are devoted Boston Celtics fans who would get hyped during games over plays and fouls I didn’t grasp. As a basketball novice, I knew little beyond the goal of getting the ball in the hoop. Curious to understand their passion, I researched the sport’s origins, rules, and how they were created. I stumbled upon an article titled “Basketball and the Deaf,” by Hunter Clark. In it, Clark highlighted the challenges deaf players and referees face during fast-paced games. 

I dove into articles on communication techniques and refereeing tactics, becoming engrossed in this obstacle to accessibility. I realized that one major challenge that referees faced during a game like basketball was calling the fouls on time. Their motto is “See it and call it,” but how can you blow the whistle if the players can’t hear? Sometimes they have to chase players down the court to stop the action, and that disrupts the whole game. Basketball is a game that promotes sportsmanship, teamwork, and togetherness, so I had the idea for this project.

What steps did you take?

When I first embarked on this project, I was woefully ignorant about sign language. I immersed myself in learning the nuances of the hand shapes, palm orientations, movements, and facial expressions that make up this complex visual language. I watched videos of interpreters demonstrating basketball terms in sign language, envisioning how this could help bridge the communication divide. I spent countless hours creating a dataset of snippets of basketball sign-language videos and their meaning from the internet. I then motion-captured the hand movements to animate an avatar for uniformity. 

When it finally worked seamlessly, I was overwhelmed with gratitude and amazement. The algorithm took the voice, converted it into text, broke it down into word tokens, and then the avatar — based on my sister’s face — demonstrated the corresponding sign language for each word in succession. Now a referee’s voice could instantly be translated to signing. It was incredibly rewarding. 

Were any faculty members particularly inspiring/helpful along the way?

Visiting Assistant Professor of Psychology Ethan Ludwin-Peery, my Division II chair, was instrumental in nurturing the algorithm from an embryonic idea into a fully realized tool. When I first shared the concept with him, unsure if it was too ambitious, he encouraged me to dream big. His faith in my potential gave me the confidence to pursue it. 

Did you need/receive any funding?

One of the aspects of Hampshire College that I appreciate most is the generous grant program, which empowers students to pursue meaningful work. When conceptualizing my project, I was able to align my goals with several relevant grants and make a compelling case for support. The process of researching appropriate funding sources and articulating my vision in applications taught me invaluable skills.

The Culture, Brain, Development Program (CBD) grant I received was foundational, underwriting the ambitious scope of the project and validating its potential for impact. With CBD support, I purchased subscriptions to help animate and run the models. 

The Cognitive Science Research Fund was essential for exploring the intersection of language, technology, and accessibility. Perhaps most meaningful was the Ruth Moore Fund for ASL/Deaf Studies. Knowing this project honored Ruth Moore’s legacy by advancing inclusion inspired me daily.

What are your plans for the algorithm?

My purpose in designing this algorithm was to make basketball more inclusive, not to make a profit. And this project wouldn’t have been possible without open-source resources, repositories, grants, and visionaries who share their gifts. So, I intend to make the code freely available to build upon. My dream is for this technology to grow beyond my contributions and improve through many caring hearts and hands.

I’m filled with gratitude for the mentors, funders, and pioneers who paved the way for this project. Now I hope to pay that generosity forward. The algorithm is just the beginning; there are so many possibilities to increase accessibility that have yet to be imagined.