Slack Question Analyzer

It finds the questions a Slack support channel keeps asking, then checks whether the answer you wrote up made people stop asking. The transcripts never leave your computer.

The Slack Question Analyzer’s Learned topics window, open over its dashboard. The made-up sample topic SFTP Timeouts has a saved answer and a live frequently asked questions (FAQ) page. A red line says the page isn’t moving the needle, at 0.4 asks a week before it was published and 1 a week after, so the doc may be hard to find or the answer incomplete. Under it, a note says 1 ask was answered in a thread after the answer was saved, and a chart of asks from 11 May to 6 July 2026 has a dashed line on 15 June, where the FAQ was published. A second topic, Partner Onboarding, is listed below

The numbers

405tests, none of which needs a language model installed to run
2.62version, with a written specification of the pipeline kept beside the code
0transcripts sent off the computer it runs on

My part

  1. Builtthe whole thing: the parser for Slack exports and pasted threads, the grouping, the topic bank, the Flask server and the React dashboard.
  2. Designedthe dashboard in light and dark: the ranked topics, which open into every way a question was asked, and a Week in Review that steps through calendar weeks.
  3. Testedthe grouping on labeled transcripts with answer keys. A change to a threshold stays only if the score holds.321 of 331 checks passing at the last scored run

Why I built it

A support channel answers the same questions over and over, and nobody sees it happening. One person asks how to raise the SFTP timeout for large transfers. (SFTP is a protocol for sending files securely.) Weeks later someone says transfers keep aborting on big files and asks if there’s a wait setting, and then someone else asks what controls the SFTP wait time. It’s one question asked three ways, and each thread gets answered on its own. One entry on a frequently asked questions (FAQ) page would have covered all three.

The made-up topic at the top of the dashboard's ranked list, opened. Under the File Operations theme, SFTP Timeouts was asked 5 times. Its main question reads How do I raise the SFTP timeout for large transfers?, tagged sftp, timeout and transfers, and a line says the wording matches 88 percent, 3 asks were answered, it was first asked on 12 May 2026 and last asked on 6 July. Three of its phrasings follow, each with its date: How do I raise the SFTP timeout for large transfers?, answered, 12 May; Transfers keep aborting on big files, is there a wait setting?, answered, 2 June; and What controls the SFTP wait time?, unanswered, 20 June
A made-up topic, the top one in the ranked list, opened. It was asked five times, and each way of asking it keeps its date and whether a thread answered it.

Keyword search can’t catch that, because the three people don’t use the same words. Sending the channel’s history to a hosted service wasn’t an option either, since support threads are full of customer names and internal details. So while I was interning at IBM, I built a tool that groups questions by meaning and runs on one computer. A Python pipeline drives local language models through Ollama, and a Flask server hands the results to a React dashboard.

What I decided

  • Paste a thread straight in. The default input is a paste box. You copy a thread out of Slack and paste it, names, times and the “2 replies” line included. The parser also reads Slack’s own export files (zipped or not), comma-separated files and plain text, and it handles dates with no year. It keeps each message’s text, date and replies and drops the names, so it never records who asked.
  • A close call needs a shared subject word. The tool scores how close two questions are in meaning, where 1 means identical, and a question joins a group when it averages at least 0.8 against everything in it. Take two made-up questions, “How do I configure the antivirus scanner?” and “How do I configure the certificate vault?” They have the same shape, so they score close. A real rewording almost always shares a word that names its subject, like “timeout”, so the tool looks for one, and words that turn up all over the channel don’t count. Two groups that land just under the bar can still merge if a model agrees they match. With no subject word in common, they have to clear the full bar, and a lone question that shares none with the nearest group stays on its own.
  • Topics keep their names. A topic bank carries topics from one analysis to the next. Each recurring topic with a proper name is saved there as the average of its questions’ meanings, and a later group that lands close to it takes the same name. “SFTP Timeouts” is still “SFTP Timeouts” ten weeks later, so the weeks can be compared. Every ask is fingerprinted by its wording and date, so if you export the last 90 days every month, the bank doesn’t count an ask twice. Grouping is never perfect, so topics can be renamed, merged or deleted from the dashboard.
  • Check whether the FAQ worked. When the tool drafts an answer, it can use only what people replied in the threads. It checks each draft against those replies, and a draft that fails quotes the replies instead. Once someone approves the wording, every later FAQ export uses it. Marking the FAQ published stamps the date, and from then on the tool compares asks per week before and after it. A drop of half or more means it’s working, and a drop of 15 percent or more means it’s helping. Anything else, a rise included, reads as not working. The made-up SFTP topic went from 0.4 asks a week to 1, so the tool says the doc may be hard to find or the answer incomplete. With less than two weeks of data after the date it says it’s too early to tell, and with less than two weeks before, that there’s no baseline.
The Add a transcript window. A note says to paste a Slack thread straight from the channel or upload an export file. In the box is a pasted thread: a made-up person, Maya Chen, asks at 9 AM how to raise the SFTP timeout for large transfers, a line says 2 replies, and another made-up person, Sam Ortiz, answers at 9 AM to set transfer.timeout to 300 in the partner settings. Below are a link to upload export files instead, Cancel and Analyze
The paste box, with a sample thread laid out the way Slack copies one. Maya Chen and Sam Ortiz are made-up names.

From questions to topics

  1. 23 questions loggedScreening had already moved 3 pieces of product feedback, the “it would be nice if” kind, to a list of their own.

  2. 17 belong to a question asked more than onceThe other 6 were asked once, and they’re listed under the ranked topics.

  3. 6 topics, ranked by how often they come upEach one opens back up into every way it was asked.

The sample run in the project’s own screenshots. Each bar is as wide as its count.

Did the FAQ work?

In my tool you mark a topic’s FAQ as published, and from then on it compares how often the question comes up before and after that day. Set the asks and the dates here and see what it says.

Start from an example

Not working

“FAQ not moving the needle: 0.4/wk before to 1/wk after. The doc may be hard to find, or the answer incomplete.”

Asks went from 0.4 a week to 1 a week, up 143%. Anything short of a 15% fall counts as not working, rises included.

Before the FAQ

The tool counts back 8 weeks at most.

2 asks in the 34 days it counts

Since the FAQ

The newest question in any saved analysis, on any topic. The publish day counts as after.

3 asks in the 21 days since

This runs in your browser. It’s the tool’s verdict rule on its own, with its thresholds and its wording. The real tool reads the dates from every analysis you’ve saved; here you set the counts and the spans, and the asks are spread evenly over them. The tool works out the change from the rates before it rounds them, which is why 0.4 to 1 a week in the sample comes out as up 143%. The sample topic is SFTP Timeouts, a question about secure file transfers from the tool’s made-up sample (the one in its README screenshot), and the test topic comes from the tool’s own tests.

The controls need JavaScript, so with it turned off this shows the sample topic as the tool judges it.