A screenshot of the NowWithFeeling main site. The goal is to have more activities in the future. Click here to visit the website
Project Description
NowWithFeeling is a capstone project with the Robots with Social Intelligence and Empathy (rosie) lab to create a citizen science platform. The platform is created to act as a collaboration hub with different activities to conduct research.
After creating the main site, we created an activity that processed voices called Teacher Voice. The user is tasked to record scripts in their most engaging voice. The audio was processed and results were displayed. The site was deployed and roughly 50 people participated in the activity. The goal of this activity was to collect engaging voices and extract their features to improve virtual teaching agents and reduce the burden on overworked teachers.
I was responsible for leading front-end development, designing the prototypes and project management.
The Team
Lina Dang-Nguyen, Julie Jen, Eric Liu
The Stack
-
React
-
AWS DynamoDB
-
AWS Lambda
-
AWS S3
-
React Testing
Key Challenges
Processing Audio Realtime
Processing audio is a key feature in the application. Users are required to record audio, upload it and receive results. I’ve never worked with audio before so it was a challenge designing an interface and figuring out how to implement it.
Implemented Recording Page
Recording Page Prototype
Designing the flow to minimize user wait time
Processing audio takes time. My team and I decided that 30 seconds was a reasonable time to wait based on our research and user testing. Our stakeholders wanted 12 audio features to be extracted and there was a risk that extracting all these features would take more than 30 seconds. I designed the flow so that there are two recordings done separately. Above is the prototype of the audio upload flow. As soon as the first recording was completed, we sent the data and waited in the background for the results. Since the second recording was designed to take 20-30s, by the end of this recording the results of the first one would be ready. As a result, users didn’t have to wait as long since some of the work was being done in the background.
Building an architecture to process audio
My team and I never built a whole system from scratch before. We knew how individual pieces like creating endpoints for the backend and using react for the front-end but deploying it together was new. We decided to create a system with AWS Lambda, AWS DynamoDB and AWS S3 because these are industry standard products with detailed documentation. AWS also had data processing pipelines, which provided us a framework to solve our audio processing problem.
These are a lot of products to learn all at once and we still had to create a MVP to present to the stakeholders and get user feedback. So we split the project into two iterations.
Version 1: Endpoints via Node-JS
To reduce the team's burden and allow for time to learn the new tools, I wrote the first iteration of the backend. This allowed the team to explore and investigate the AWS integrations, while I coded a NodeJS backend to send the audio to S3. Node-JS is a backend server environment and I used ExpressJS to create the endpoints. Within our two week sprint, I got the backend and front-end working together and presented it.
Version 2: Endpoints via AWS Lambdas
Since I already had the front-end set up, migrating to the Lambdas was trivial. All I had to do was update the api endpoints that the front-end called. I had originally set up all the api links in one file, which made it easier to change.
Key Takeaways
Plan for the unexpected
Creating timelines is hard when you’ve never created a deployed application before. We had a tight timeline at first but people got sick and unexpected things happened so I had to re-prioritize and assess what was possible. As the project manager, it was my job to help the client understand what is achievable and be as transparent as possible. For example, we planned to have a second activity on the website but the scope was too large due to the delays with launching the first activity so I worked with the client to prioritize what was absolutely necessary.
Front-end tests are time consuming but make it easy to make changes
It was my first time writing UI tests because I only tested my UI manually so it was tough at first. After writing my first component test, I learned how to simulate and search for specific events, which made it easier to write the following tests. When we had to make a patch to our deployed app, these tests gave me confidence that the patch wouldn’t break the deployed front-end.
Improvements
Improved Accessibility
After taking a class on web accessibility, I realized that the website I developed was lacking in accessibility. For example, you cannot use the keyboard to select the form inputs. I would also utilize tools like google lighthouse to audit the existing website and prioritize the accessibility fixes. Improving the accessibility can improve user experience and search engine optimization.