Workers AI gives you access to ML models running on Cloudflare's GPU infrastructure, callable as a Worker binding. No model hosting, no GPU bills, no infrastructure to manage. The promise is compelling. Here's what it actually looks like after running it in production.
What I'm Using
Two models:
@cf/facebook/bart-large-cnn— summarizes post content into a 2-3 sentence abstract@cf/meta/m2m100-1.2b— translates English posts to Spanish when no human translation exists
Both are free tier. Both have limits. Both require a caching strategy to be usable.
BART: The First-Visit Latency Problem
BART takes roughly 1-2 seconds to summarize a 500-word post. On a Worker, that latency blocks the entire response.
The first time someone visits a post, the Worker fetches the content from KV, runs BART, renders the page with the summary, and caches the result back to KV. Subsequent visits read from KV and pay no AI cost.
javascriptaiSummary = await env.POSTS_KV.get(`summary:${slug}`, "text");
if (!aiSummary) {
const r = await env.AI.run("@cf/facebook/bart-large-cnn", {
input_text: content.slice(0, 2000),
max_length: 100,
});
const summary = r.summary || r.text || "";
if (summary) {
aiSummary = summary;
ctx.waitUntil(env.POSTS_KV.put(`summary:${slug}`, summary));
}
}
The await before AI.run is intentional — the first visitor sees the summary on the same page load, not on a reload. The KV write uses ctx.waitUntil because we already have the value and don't need to block on the write.
The content.slice(0, 2000) truncates input. BART has a token limit, and sending a 3000-word post in full causes errors. 2000 characters covers most of an introduction and first section — enough for a useful summary.
M2M100: Good at Short Text, Unreliable at Long Text
The translation model is m2m100-1.2b, a 1.2 billion parameter multilingual model. It handles short to medium posts well. For longer posts (1500+ words), the output quality degrades — sentences start to drift, technical terms get mistranslated.
The mitigation: slice to 5000 characters and present the result as "AI translation" rather than a human translation. Users know they're getting an approximation.
javascriptconst result = await env.AI.run("@cf/meta/m2m100-1.2b", {
text: content.slice(0, 5000),
source_lang: "en",
target_lang: "es",
});
If a human translation exists (a post:<slug>-es key in KV), it takes priority. AI translation is a fallback for posts where I haven't written the Spanish version yet.
Rate Limits Are Real
The free tier has model-specific rate limits. I've hit them during testing when triggering AI generation on multiple posts in quick succession. In production with organic traffic, a single post per session, and KV caching, I haven't hit limits.
The try/catch around all AI calls is not optional:
javascripttry {
const r = await env.AI.run(...);
// use result
} catch(e) {
// fail silently — page still loads without summary
}
A rate limit error from Workers AI throws. Without the catch, it would 500 the entire page request. With it, the page loads normally and the summary block shows a "generating..." placeholder.
The Caching Strategy
Both models use the same pattern: check KV first, generate on miss, write back to KV.
KV reads are ~5ms. AI runs are 500ms-2000ms. The ratio means that after the first generation, the cost is negligible.
For BART summaries, the cache key is summary:<slug>. For translations, it's post:<slug>-es-ai (separate from human translations at post:<slug>-es).
There's no TTL on the cache. A summary generated once is valid indefinitely — the post content doesn't change. If I edit a post and want a new summary, I delete the KV key manually.
What I'd Do Differently
Pre-generate summaries on publish. Right now the first visitor pays the 1-2s latency. The publish script could run BART for each post and seed the KV cache during CI. The first visitor would see a cached summary like everyone else.
Streaming for translations. The translate endpoint currently blocks until the full translation is ready. For long posts, that's 3-5 seconds. A streaming response that sends translated chunks as they arrive would feel much faster.
Both are improvements I haven't made yet because the current behavior is acceptable. First-visit latency for summaries is invisible to most users — they're reading the post content while BART runs.