The problem:
How to contribute to a project as an outsider.

The situation:
The People Assistant chatbot was designed to use the content from the People site as a content repository. This solves the content creation problem because the platform vectorizes anything published to the website, saves it to a vector store, and generates responses from that database. But it created two shortcomings: as a platform, it had no predefined workflows, so it could only answer FAQs or provide a long, multi-step process in one response in the chat window. And from a content standpoint, there are processes and policies that are intentionally withheld from the People site for reasons that were never shared with me. 

These were solvable problems, but they required rethinking the solution architecture.

Having come from smaller, more nimble companies, I didn’t understand that no changes to the architecture were going to happen because those decisions had already been made. Once I realized that my input wasn’t helpful, I used my natural stoicism to figure out where I could be helpful, then I pivoted.

The solutions:
The best thing I could do to improve the chatbot experience was to create tools that helped measure its performance. We had a team reviewing response quality, but no way to measure intent distribution because the foundation model we used doesn’t provide intents. We also needed a way to redact PII from the utterances before we sent them to an external LLM for analysis.

The outcomes:
I used my Python skills and two public names databases (one from DATA.gov) and created a script that indexes each utterance, extracts matches from the names list, and puts them in the next column in order of appearance, separated by commas. This allows you to skim with great accuracy vs. reading utterances, which can introduce its own errors.

Now that I had a way to redact PII, I used the $100 in LLM tokens per day Apple provides to create a tagging agent. When I had that working well, I converted it to a Python script that made the solution more deterministic, feeding instructions to the LLM and giving it less room to misinterpret its commands. It’s essentially a RAG workflow, but instead of accepting an utterance and generating a response, it sends the utterance to the bot and asks it to write a rationale for why it selected the intent. The human reviewer only needs to agree or disagree with the LLM’s prediction and rationale.

The result is a digital version of myself so my boss will be able to manage the People Assistant without me.