Joel Horwitz has written 7 million lines of code with agents since April, and he put about half of a $500,000 raise into real ad accounts that he ran himself, so that he could find out what breaks. We talked last week. These are the 7 things from that conversation that I think will change how you build your own GTM stack.
Wednesday morning in Koh Samui, and half of my MCP connections had been failing since I sat down. There was an email about it in my inbox that I hadn’t opened yet. I’d spent a good part of that morning going back through my own configuration, looking for whatever I had broken.
Then, about 30 minutes into our call, Joel Horwitz mentioned almost in passing that Anthropic had updated their MCP connector and appended a forward slash to the end of everyone’s MCP route.
My whole outage came down to one character.
Joel already knew about it, because one of his customers had hit the same thing that morning and he’d gone in and added aliases so that it wouldn’t happen to them again. And Joel’s response there, honestly, is the entire argument for the way he builds.
Joel runs Synter, an agent platform for paid media, and before that he ran growth for Amp at Sourcegraph. Between April and now he’s written 7 million lines of code with agents, tried more or less every coding agent that exists, open source and closed source both, and put roughly half of a $500,000 raise through live ad accounts to find where the whole thing fell over.
I wanted the 5 months of expensive mistakes that produced that product, and he was generous with them.
We got through about half of my questions before we ran out of time, so the full founder feature is still coming. In the meantime, these 7 learnings are too useful to sit on, and the first one is actively costing people money this week.
1. Your skill files are an attack surface
Joel put it to me plainly:
“Skills are malware. Like, remember that.”
Here’s what I mean. A skill file is a markdown file full of instructions that your agent will follow. Someone posts on LinkedIn offering their Google Ads skill file, you download it, you install it, and whatever is written inside it is now something that your agent does on your behalf. Joel has been watching people do exactly that and then upload the results into his own platform. That’s why he ended up building a scanner for skill files.
My business partner Jonathan brought up a case where this already happened. A skill file sat on a product’s own homepage, and that file carried instructions to find a crypto wallet and move whatever was in it to a different address. Everyone who installed that file ran those instructions.
Joel’s read on the news coverage is that the coverage got the story backwards:
“All these news reports of, oh, the agent escaped. It didn’t escape. Someone uploaded a skill that was malware.”
Two companies have now put real money behind that same read. Snyk and Vercel have scanned close to 4,000 agent skills across the major marketplaces, and roughly 147 new agent skills get published every day, so they stood that scanning program up because the agent skill supply chain became a real target for attackers. Joel himself starts at skills.sh, which is Vercel’s open skills ecosystem, and he treats whatever he finds there as a starting point rather than as something to install as-is.
Key Takeaway: Treat a third-party skill file the way that you’d treat a package from an author that you’ve never heard of on any public code registry. Read it before you install it, then run it through a scanner. Joel has open-sourced one of his own, and he calls it Agent Shield.
2. Tools are Python scripts. Skills are markdown files.
Joel decoded the vocabulary for me in about 30 seconds…
“Tools are just words for Python scripts. Just so you guys all know, like we say Python scripts, we say tools… We say skills. What we’re saying is a markdown file. It’s a pre-prompt.”
I’ve watched a lot of tech founders stall out on agent architecture, because the vocabulary makes it sound like a different category from the software that they already know how to reason about. A tool is a script that your agent can call. A skill is a text file that your agent reads before it starts working. Once you’re thinking about it in those terms, you can make architectural decisions about your own agent stack instead of deferring them.
3. Build the tool once, then stop re-prompting
The failure mode that Joel described is specific. You’ve probably been paying for it without ever naming it: a coding agent writes a brand new Python script every single time that it touches an ad platform, and it burns context relearning the same platform standard operating procedure on every single run.
Joel’s fix is a two-agent pattern, and you can port it into whatever agent stack you’re already running.
- A builder agent attempts the campaign over and over until it works.
- An architecture agent then reads what the builder actually did, finds the repeating pattern, and writes it into a Python script that lives permanently on the backend.
“So I build, build, build, build. And then the architect goes in. Okay, I see the patterns here. I’m going to turn that into a Python script, literally, and drop that onto your backend system and call that a tool.”
The result that he describes is a full-funnel LinkedIn campaign that launches in one shot, in under an hour, because the tools already exist and nothing is being relearned.
A bigger context window does not fix this, and that’s where most people get it wrong. “You can go to a one-million context window with Opus… Agents degrade. They degrade in the window.” Every rudimentary instruction that you leave sitting inside the prompt is real capacity that you’re spending.
Key Takeaway: Anything that your agent works out twice should become a tool. Your context window is working memory, not storage, so your job is to move settled knowledge out of the prompt and into something permanent that the agent can call.
4. Your real cost per acquisition includes your token bill
Joel described meeting a founder who had launched one LinkedIn campaign and burned through an enormous number of turns getting there. (A turn is one exchange between you and the model.)
“People are like, oh, I launched this ad campaign and I got a CAC of like $45. I’m like, but how much did you spend on AI to do it?”
Joel’s accounting has three components where yours probably has one: your paid media cost, your model cost, and your operating cost.
You can recalculate this today, and almost nobody has. If you’re running agents anywhere in your growth stack, your customer acquisition cost (CAC) is understated by whatever you’re paying Anthropic or OpenAI to produce the result.
In fact, Joel spent more on ads than he did on tokens. He says that’s deliberate, because he runs a lot of open-source models and he told me he’s fine-tuned his own model on opted-in, anonymized data. He also runs a masking layer between his ad platform data and his model provider, because Google’s terms wouldn’t let him pass that data to Anthropic in a form that Anthropic could read.
Key Takeaway: Add the model-spend line to your CAC calculation this week, before you decide whether the agent is earning its place in your stack.
5. Get your measurement working before you spend
Joel’s first agent project at Sourcegraph was maintenance work, and he’s emphatic that the boring quality of it is exactly why it mattered.
“I always start with like the most mundane tasks first, like not the fancy stuff. And if I gave you a list of the use cases that I’m seeing, they’re not that glamorous.”
What he actually built was server-side conversion tracking, because a developer audience wouldn’t accept cookie-based tracking. He wired up Google Ads, Google Analytics, Google Tag Manager, and PostHog, with an application programming interface (API) key for each of those four. Then he ran the conversions API server side and hashed every email address with SHA-256, which is a one-way hash, so that no raw email address was ever exposed. Then he ran the whole build past legal. He says that he’s not a data engineer and had never done any of it before.
His position on spending money without server-side tracking in place is absolute:
“Anyone who comes to you ever and says, oh, I’m going to do a brand awareness campaign, I’m like, no, no, no. Like not unless you have a pixel on my page or on my server side.”
Build what Joel describes in this section before you put another dollar into ads.
Joel also gave a founder on our call a launch sequence for a brand new product, and the sequence generalizes cleanly to anything that you’re launching:
- Get your server-side conversion tracking correct before you spend a single advertising dollar.
- Pick one human face for the brand, and build your organic content around that person.
- Watch which organic content actually converts, and put your paid spend behind that content.
Key Takeaway: Most of marketing operations is coding, and coding agents are good at writing code. Point your agent at the unglamorous measurement work first, because your conversion tracking is what makes every dollar you spend afterward measurable.
6. Never hand an agent a raw API key
This one is short, and you can fix it today.
“A lot of people just like give the agents the key. Don’t do that. Don’t ever do that, because now you need to rotate it.”
Put your credentials in a secrets manager instead. Joel named 1Password and Doppler. Then give your agent access to the manager rather than to the key itself, so that rotation stays possible and no key sits in a configuration file where it can leak.
You can also have your agent pull the change log out of your ad platform and read exactly what your agency has been doing inside your account. It costs you nothing, and almost nobody does it. In Joel’s words, “You can audit them today. Nobody knows that.”
7. The Apache Spark lesson, which is really the production lesson
This learning is as much mine as it is Joel’s, because I lived the same decade in data.
In 2015, IBM ran a campaign called 100 data products in 100 days on Apache Spark, and Joel wrote the launch speech for it. Labs in Switzerland, Italy, India and the United States built genetics applications, a traffic application out of China, and shipping work with Maersk, because Spark had collapsed the barrier to entry. Anyone could suddenly work with the same data model.
Then almost none of it reached production.
“They build these products. Then they couldn’t put them into production. Why? Because, you know, you still need an orchestration layer. Like it’s really, it works really well on your laptop. But then we want to actually put it in production. That’s when things can go sideways.”
Joel’s argument is that you’re watching the same pattern run again right now. People are building beautiful applications with Lovable and Replit, and those applications have no tracking and no distribution attached to them, so they stay demos.
Key Takeaway: The barrier to building has collapsed twice in 10 years, and both times the constraint moved immediately downstream into production and distribution. Build for the orchestration layer from the start, or plan on rebuilding.
What’s harder than the architecture…
I asked what’s actually limiting adoption, and Joel pointed straight past the technology at the org chart.
“I think our limitation right now is not Synter. It’s change management.”
Change management is what I run into on nearly every engagement that I take. Most people won’t make a structural change because they could lose their job over it, so they automate the reporting and leave the operating model exactly where it was.
Joel flips that fear around, and I think his version is the correct one:
“I would be scared not to make those type of changes.”
Reproduce This Win
- Scan every third-party skill file before you install it, so that you know what your agent has been told to do.
- Move your API keys into a secrets manager, so that you can rotate them when you need to.
- Turn anything that your agent works out twice into a tool, so that it stops relearning the same job.
- Add your model spend to your CAC calculation this week, so that you know your real cost per customer.
- Fix your server-side conversion tracking before you spend another advertising dollar, so that the spend is measurable.
- Pull your ad platform change log, so that you can read what your agency actually did last quarter.
- Build for the orchestration layer from day one, so that your demo can reach production.
We ran out of time at roughly the halfway mark on my questions. I’d call that a good sign about the density of the conversation. Joel and I are picking it up again on the 21st while he’s in London, and the full founder feature will follow from that.
Until then, take this with you: your advantage comes from everything around the model, and that layer is still built by hand.
If you’re in the middle of wiring agents into your own growth stack and you hit something in here that doesn’t map onto your setup, hit reply and tell me where it breaks. I read every one of these, and the sharpest questions tend to become the next brief.
