For the world’s 430 million deaf and hard of hearing people, communicating with hearing individuals can be challenging. Researchers are now exploring a solution that uses machine learning to translate sign language into text or speech instantly. The solution could make dialogue easier and help ensure that deaf and hard of hearing people are better included in society.
AI suggested collaborating with SINTEF
The idea of using AI technology to translate sign language into text or speech came from Tone Ervik and Pål Rudshavn at Statped in Trondheim. They noticed that AI was getting better at speech to text translation. Could AI and new language models also be used to translate sign language?
“I asked ChatGPT how we could move forward with this idea, and it suggested contacting SINTEF — so that’s what we did,” says Tone Ervik.
The combination of societal benefit and technological challenge made SINTEF researchers immediately interested.
“We at SINTEF saw this as a fantastic opportunity. With the rapid advances in AI, we wanted to use this technology to make a meaningful difference in society,” says Kostas Boletsis, supported by colleague Zia Uddin.
With support from Stiftelsen Dam, the project “AI Driven Norwegian Sign Language Translator” began in February 2024.
Researchers in the U.S. have already made progress with real time sign language interpretation tools using machine learning. But Norwegian Sign Language is unique, so a dedicated model must be developed in Norway.
Boletsis and Uddin began by training a computer to automatically recognize NTS signs for the numbers 0 to 10.
“We focused on numbers 0–10 because we needed to start somewhere, and Norwegian Sign Language differs from other sign languages. It could have been any other 11 gestures. We can expand with additional analysis, but the basic approach remains the same — just scaled up with more complex algorithms,” says Zia Uddin.
Tested in real time
Their system achieved a 95 per cent test accuracy. According to SINTEF researchers, this shows that the solution handles variations in signing style, speed, and angle.
A year after the project began, it was time to test the AI based system in real time. Twelve sign language users visited Statped, where they stood in front of the computer and performed the signs from 0 to 10. The program uses hand and mouth markers to distinguish between signs with similar hand shapes, such as 3 and 8. Some misclassifications occurred, and this information will help improve the model.
“The goal is to develop a learning app for real time recognition and evaluation of NTS, where users immediately receive a translation via an avatar. This will help sign language users communicate with hearing people in shops, at the hairdresser, at the airport, and so on. Today’s test results point to great future potential,” says Kostas Boletsis.
Researchers say further development should focus on:
- expanding the vocabulary
- testing in different environments (lighting, camera angles, speeds)
- using more sensor data for better spatial understanding
The goal: an app
“Given the scope of what we want to achieve, the work will naturally take several years. At the same time, AI will continue to evolve. The core of these projects is data — we need to develop a dataset, a corpus, with lots of information and many videos for each sign,” says Zia Uddin.
Once that is in place, large scale AI models can be trained to handle a much broader range of expressions.
Researchers will now seek funding to take the project to the next level. The dream is an app or software — for example, installed on a mobile phone — that can translate key words and phrases from sign language simultaneously.
“Sign language is incredibly important for deaf and hard of hearing people, and with the advances in AI, especially in image and video analysis, we believe we can develop a tool that truly makes a difference,” says SINTEF researcher Zia Uddin.
The research is published in the article Real Time Norwegian Sign Language Recognition Using MediaPipe and LSTM.
