There is a common annoyance among every Discord moderator: people posting in the wrong channels.
In the DSPy community Discord, this happens most frequently with job-related postings: people looking to hire and people looking to be hired. To be clear: we like job posts! We just don't want listings and resumes cluttering the main channel.
It doesn't take long to delete the message and send a DM asking them to move it, but it certainly gets annoying.
The good news is that getting AI to detect and take the action is pretty easy.
All of my code is available on GitHub: https://github.com/cmpnd-ai/dspy-cli/tree/main/examples/discord-mod
Most Errors Come From Underspecification, Not LLM Mistakes
We expect most modern LLMs to do a good job at this. Like a REALLY good job. So I expect any errors to come from underspecification rather than LLM errors.
Underspecification is a deep term here and it could mean: (1) not knowing the distribution of the problem space you will solve until you see how models interact with it, (2) lacking detail about the actual task you want to solve, or (3) lacking detail on what the model you are working with needs in order to solve the problem. Understanding the levels of specification for smaller LMs today, the frontier LMs today, and what the frontier LMs will be in 3 years is incredibly important.
This bot isn't meant to ban users or be a catch-all spam detector. If it happens to catch and delete a stray non-job spam message, that's upside, but really the use case is about moving job postings into the correct channel.
Detect Intent First, Decide Action Second
The main goal of this system is to detect the primary intent of a message. The intent can be job posting, job seeking, or neither. We also want the system to suggest an action, between deleting, moving, flagging, or allowing, depending on context.
We also want to be notified as mods if there is an action taken, so that we can be aware if things start to break.
Primary intent is a fuzzy line to draw. There are messages that have self-promotion in them, but the primary intent is not to promote. This line was chosen by talking to the team during the creation of this project.
For example:
Oh yeah I see why GEPA is cool. In my last job, I used MIPRO to improve our LLM-as-a-judge for insurance document relevance inside a chat by 20%, and my manager + our customers were very happy. Ultimately, I'm now looking for the next challenge and I'm excited about applying GEPA to whatever my next problem is (P.S. pls DM if you'd like to chat about hiring me!).
In this case, the primary intent of the message is to validate that they've used GEPA at a company, and share the project they worked on. Getting hired is secondary. As a team we talked about this, and it's about the primary intent, rather than the presence of anything hiring-related.
We will want to use the message, author, and the channel it was posted in. All of these could provide relevant information. If a user named dm_me_for_free_crypto sends a sketchy message, the LLM should take the username into account. Same with channel.
It's because of this contextual nuance that things look a little fuzzy, rather than a pure message: str -> is_job_related: bool pipeline.
DSPy Makes the Code Remarkably Simple
Let's start to put that into a DSPy signature:
class JobPostingSignature(dspy.Signature):
"""Classify a Discord message and determine moderation action.
A job posting or job seeking message will express the primary
intent of introducing the user, their qualifications/openness
to work, and their availability for hire, or availability to
hire others"""
message: str = dspy.InputField(desc="The Discord message content")
author: str = dspy.InputField(desc="The message author's username")
channel_name: str = dspy.InputField(desc="The channel where posted")
intent: Literal["post_job", "seek_job", "other"] = dspy.OutputField(
desc="The user's intent"
)
action: Literal["allow", "move", "flag", "delete"] = dspy.OutputField(
desc="Action to take"
)
reason: str = dspy.OutputField(
desc="Brief explanation of the classification"
)Note that there are two reasoning fields. This is a design choice.
The first model reasoning happens when we call the signature with dspy.ChainOfThought(JobPostingSignature). This is to give the model time to deliberate. We are using gpt-5-nano with no reasoning built in, so this will explicitly give the model a chance to discuss the message if there is any nuance.
The second reasoning field is meant to be user/moderator facing. After the model has committed to a course of action, we want to show the user and us as the mods, "Why did you get banned?" which is different reasoning than the model's internal debate about what the intent of the message is.
Part of why I use DSPy is that everything is built in already. Structured outputs, retries, plumbing between different LLMs and providers if I ever want to update, are all for free.
The module which takes in the signature above looks like:
class ClassifyJobPosting(dspy.Module):
gateway = JobPostingGateway
def __init__(self):
super().__init__()
self.classifier = dspy.ChainOfThought(JobPostingSignature)
def forward(self, message: str, author: str, channel_name: str):
return self.classifier(
message=message,
author=author,
channel_name=channel_name
)It's important to note that I am not doing any prompt optimization here. I am just using DSPy for the LLM orchestration aspects. It's an incredibly concise tool to express your task that you want the LLM to solve without having to write large string prompts.
A Cronjob is Simpler Than Real-time Streaming
The most complex part is integrating with Discord and implementing the business logic for taking actions based on the LLM outputs.
We opt for a cron approach rather than real-time streaming. Mostly because the real-time streaming requires us to maintain a websocket connection. The DSPy server averages way less than 20 messages every 5 minutes.
Based on historical analysis of DSPy server traffic over the last 6 months, the p99 number of messages per 5 minutes is 11, so 20 messages is more than enough.
The behavior is as follows: every 5 minutes, gather the last 20 messages, skip any that have already been processed, classify as "move", "flag", "delete", or "allow", then take the relevant action.
To implement this, I use dspy-cli to handle the routing and scheduling. dspy-cli is a command line tool to help you create, scaffold, and deploy DSPy applications.
Fly.io is a Great Match for dspy-cli Projects
When you create a dspy-cli project, there is a generated Dockerfile that you can deploy onto any service. We deploy this onto Fly.io. We want a single permanently running machine that triggers itself every 5 minutes.
Fly automatically scales your workload to 2 machines and will pause if there is not enough external traffic. For this deployment, there will by definition be 0 API-based traffic, so we need to turn off auto-scaling while keeping a minimum of one machine running:
auto_stop_machines = false
auto_start_machines = true
min_machines_running = 1Learnings
Fly.io + dspy-cli Make Hobbyist Deployment Easy. This was the first real problem I wanted to solve with it. Figuring out how to have a single, permanently running machine on Fly did have its quirks, but I feel good about where I arrived.
Spending Time on the Signature Clarified Our Intent. I also got feedback from other people by just sending them the signature and asking if this describes the moderation ideology we want to have, which is how we arrived at the 'primary intent' framing. The alignment of the team on the actual problem we are solving is incredibly important.
As Edge Cases Arrive, Document Them In Your Evals. It initially thought some of the introductions that got posted in #general should stay. This was wrong! As some people in the Discord pointed out, it's pretty obvious that anyone using the word "blockchain" in their post is likely not a legitimate user. I added this into the evaluation framework.
LLM Inference + Fly.io Costs Under $1/Month. This is shockingly cheap to run! The LLM inference is not expensive (using gpt-5-nano), and the server costs less than $1 per month.
The Blog Post Took Longer Than The Code. This bot did not take long to implement. I spent much longer writing this blog post than coding.
If you want to join the DSPy Discord and see the bot in action, here's the invite: https://discord.gg/f5DJ778ZnK
The full source code is available on GitHub: https://github.com/cmpnd-ai/dspy-cli/tree/main/examples/discord-mod
If you have used DSPy or are thinking about using it, I'd love to chat! My username on the DSPy Discord is @Isaac. Feel free to email me at isaac@cmpnd.ai or DM me on Twitter/X @Isaacbmiller1.
Appendix
DSPy Signature to Prompt Example
If you are curious about how the DSPy signature gets turned into a prompt, here's how it works.
DSPy has a concept of an "adapter" which is a class that determines how a signature gets turned into a prompt, and also how the answers get extracted on the other end. The rough outline for adapters is:
System prompt:
1. Inputs
2. Outputs
3. Task
User prompt:
1. Field name: value for field, value in inputs
2. Desired assistant output listThe exact prompt that gets sent to the LLM for our signature is:
System message:
Your input fields are:
1. `message` (str): The Discord message content
2. `author` (str): The message author's username
3. `channel_name` (str): The channel where the message was posted
Your output fields are:
1. `intent` (Literal['post_job', 'seek_job', 'other']): The user's intent
2. `action` (Literal['allow', 'move', 'flag', 'delete']): Action to take
3. `reason` (str): Brief explanation of the classification
All interactions will be structured in the following way:
[[ ## message ## ]]
{message}
[[ ## author ## ]]
{author}
[[ ## channel_name ## ]]
{channel_name}
[[ ## intent ## ]]
{intent}
[[ ## action ## ]]
{action}
[[ ## reason ## ]]
{reason}
[[ ## completed ## ]]And the LM responds with:
[[ ## intent ## ]]
seek_job
[[ ## action ## ]]
move
[[ ## reason ## ]]
The message expresses the user's intent to seek job opportunities,
which is more appropriate for a jobs channel rather than general chat.
[[ ## completed ## ]]Batch Processing
For higher throughput, CronGateway supports batch mode which processes all inputs in parallel using DSPy's module.batch():
class JobPostingGateway(CronGateway):
schedule = "*/5 * * * *"
use_batch = True # Enable parallel processing
num_threads = 4 # Number of concurrent threads (optional)
max_errors = 10 # Stop batch if this many errors occur (optional)With batch mode enabled, the scheduler will call your module's batch() method instead of running forward() sequentially. This is useful when you have many inputs and want to parallelize LLM calls. The on_complete callback still runs once per input after all batch results return.
Full fly.toml
app = 'discord-mod'
primary_region = 'ewr'
[build]
[http_service]
internal_port = 8000
force_https = true
auto_stop_machines = false
auto_start_machines = true
min_machines_running = 1
processes = ['app']
[[vm]]
memory = '1gb'
cpu_kind = 'shared'
cpus = 1
memory_mb = 1024
[mounts]
source = "dspy_discord_data"
destination = "/data"