Making React apps talk

This summer holiday started with sad news. My friend Frode (51) had died from ALS. ALS is a terrible disease that eats away your muscles one by one and speech is one of the first to suffer. Frode knew this, soa year ago he asked me to help make something that could make him talk once his own ability to do so disappeared.

In the spirit of Frode I’d like to share this project by making the Github repository public and explain in this blog post how I solved it. It wasn’t hard, but I hope the files will prove to be a good starting point for others with a need to make something similar.

Brief background on Frode and ALS

I had known Frode since we were both active in the community around the Flash and Director multimedia tools. If you’re curious about that, dig through the archives of this blog dating back to 2005. Frode was a regular at the FUGN meetups that I co-organized and more often than not, we’d be the last few in the bar. When Flash died down, I switched to hardware and Frode stuck with software, but we stayed in touch over the years. Usually over beers.

When Frode learned that he had ALS, he started planning for days to come, but he remained extremely positive until the very end. He was like that before ALS and he just decided that there was no need to change just because of a stupid disease. That is just so impressive? I’m sure that I could not have sucked it up like that without resorting to feeling pity for myself. Such a great guy even though he knew that he’d die from this within less than a year!

Frode knew that all the speech software offered by the Norwegian (and probably any other) state was terrible, so he asked me if I could help him make something that would work from his phone. Frode’s greatest wish was to be able to use the voice of the former Prime Minister in Norway - Erna Solberg. That didn’t pan out, but she sent him a personal message. praising his courage and odd choice. Frode had researched and found that Play.ht was the only service that offered voices that could speak anything other than English. They use AI generated voices that sound really convincing. If you ask the service to use a Norwegian voice and say something in English, it basically sounds like a Norwegian that is shitty at speaking English?

You can try this out for yourself on the Chatter App homepage.

Making your React app talk

First of all a disclaimer - I only took this as far as was useful for Frode. One could do so much more and you could certainly make this in many other ways. To me it was a fun project to help a friend in need. All the application logic is in a single file.

You first need an account with Play.ht. If you’re using the service a lot, you’ll need a paid account. I purchased the lowest commercial tier they offer, just to be sure that Frode would never run into the problem of it not working.

The app is quite simple. The app is just a text input that is focused upon load, plus some shortcut buttons for some common simple phrases (yes/no/maybe). As soon as a user is pressing Enter/Send, the text is sent to the play.ht API.

Using Play.ht is an async operation, so you’ll need to poll to check if your conversion is finished. I use “useInterval” for the state toggling. I’m using Axios to call the API every second. Based on the return value, I’ll change the state of the app. Once you get a file back, you set it as the source of the audio player. The Play.ht service is good overall, but it can take time to generate long passages of text. In my case, this was not a problem. One thing I wish that I did, but never got around to was to add a caching mechanism. This could be done with local storage but you could also check if the previous link for the same request was still valid. I found that Play.ht will normally let the generated files live for a day or two.

I also set this up to auto-deploy to a Droplet on Digital Ocean. Very smooth process and I’ll keep using this setup on future projects.Very convenient to just commit the code you tested locally and get it published automatically.

Limitations and taking this further

Playback is done using React Audio Player and native browser support will decide if this will work on your specific device. The current setup worked for Frode, but I did experience that some phones do not play back the audio. If it fails, try using a different browser. In my experience, this works on older iPhones but fails on newer ones. This code snippet explains what is changed and why. So to work around this, you’ll have to first create the audio file and then present a new Play button that the user must click again to play it. Annoying for this usecase, but probably a good reason for it?

After the first version, I got some help from my friend Erik who also knew Frode. There are so many things that could have been done to make this a full blown app that was super easy to use, but Frode didn’t need more than this. He was a modest guy, so even though I asked him what I could to improve it, he said that he was happy. I hope the code will make it possible/easier for someone else to help their friends.

Post mortem (about Frode)

If you knew Frode, but didn’t hear about his disease, check out this video made by the national broadcaster in Norway. Frode and I both grew up working in local radio. While I was educated as a sound engineer and later ditched the entire radio business, Frode stayed active at Radio Nova with Kvegpels as his main project. Frode was such a lovable person and an institution that the radio station named a prize after him. We both followed a similar path with radio, multimedia programming and loving Richard Feynman for his curiosity and ability to explain things in a way ppl can understand.

Not everyone has someone that will put on a festival for you, but Frode’s best friend Kristoffer organized Frodivalen - a festival of rock and fun at John Dee in Oslo in support of Frode and ALS research. We all hoped that Frode would live to see the day, but fate would not have it like that. It was an amazing evening and I got to hang out with lots of nice people that knew Frode. I also discovered the band Blomst - one of my new favorite rock bands!

Rest in peace my friend! You would have laughed well if you saw the combo of news stories along with your eulogy. Also, I love what was on your iPad when you passed.