The problem:
A craftsman never blames his tools, but what if there are no tools to blame?

The situation:
When I moved to the People team from the Retail team, I lost access to a lot of reporting tools that helped manage that chatbot. All I had to work with was the raw chat transcripts.

Generating an intent distribution report took half a day. And testing how the models responded to changes was done by feeding utterances into the test chat interface one by one.

The chat transcript dashboard had a bug in it that created duplicate rows in the output file. So I had to figure out how to deduplicate 230,000 rows of transcript data. Excel couldn’t handle it.

I also needed a way to run the utterances through the test chat interface after making changes to the model to measure their effects. Manually entering and recording the output for hundreds of interactions was not practical. And in 2022, LLMs were not generally available to assist me.

The solutions:
After learning that there was no way Excel could deduplicate all of the transcript data I needed, I turned to Python. I followed along with freecodecamp.com‘s Introduction to Python video, then signed up for Coursera and took the University of Michigan’s Python for Everybody courses. This enabled me to write a script that opened the transcript CSV file with the built-in csv library, loop through the file and save the odd-numbered rows to a new CSV file.

For utterance testing, I used AppleScript to automate the Mac OS GUI. With the test chat web page open in Safari, and the list of test utterances open in Excel, the script would navigate to the correct cell, copy the utterance, activate Safari, then paste it into the test chat input field and press Return to send it to the chatbot. After a two-second delay to let the response appear in the chat window, it screen-captured the interaction, complete with intent and confidence score courtesy of the test chat window. I’d planned to add OCR, but just having the screen shots with all of the information I needed in one shot made reviewing model performance much easier.

The outcomes:
These scripts, and the skills I learned to write them, enabled me to monitor intent performance before and after model changes by reading the interaction in Preview, then recording the results in the original utterance CSV.