Cleaning UX research transcripts
Published on
This ETL workflow converts noisy, raw Product Discovery transcripts into a highly efficient "Semantic Memory" for AI agents. By automating the removal of timestamps, source tags, and filler interruptions, the method structures dialogue into concise blocks that preserve the user's voice and essential details. The result is a much more accurate RAG (Retrieval-Augmented Generation) system that saves tokens and streamlines the extraction of requirements and user journeys without the clutter of raw data.
From Noisy Transcripts to Semantic Memory: How I optimize huge volumes of UX Discovery data for AI Agents (without blowing the token budget). 🧠✨**
Anyone working in Product Discovery knows the drill: conducting 10 or 20 exploratory interviews generates immense strategic value, but also a mountain of raw data.
A single 1-hour interview recorded on tools like read.ai often results in massive transcripts clutter, full of noise, timestamps, and filler words ("um," "ah," "yeah") that burn through tokens and pollute data vectorization (embeddings).
To solve this and optimize my RAG (Retrieval-Augmented Generation) pipelines, I developed a custom pre-processing tool (ETL) that performs heavy-duty "hygiene" on this data before it becomes long-term memory for my AI agents.
What this automation does in practice:
🔇 Noise Removal: Automatically strips out source tags, timestamps, and mid-sentence speaker names.
🔄 Smart Merge: Eliminates phatic interruptions (short agreements like "I see," "right," "cool") and fuses user speech into fluid, cohesive paragraphs.
📦 JSON Structuring: Transforms raw dialogue into clean Context (Question) and Memory (Answer) pairs, preserving the user's first-person tone and specific details (apps mentioned, sentiments, pains).
The result? A lean, clean final file focused solely on high-value information. This doesn't just save significant processing resources; it makes AI information retrieval far more accurate when extracting user pains, functional requirements, and user journeys later on.
Strategic design today also means engineering how we consume data. 🛠️
For anyone interested in testing this logic, I created a Google Gem that applies this specific hygiene prompt. Just upload your raw transcript and watch the magic happen.
👇 Link to the Gem: https://bit.ly/4b6k7rG