Server-Side Syntax Highlighting Without a Library

Syntax highlighting on the web usually works one of two ways: a client-side library (Prism, highlight.js) runs after the page loads and colorizes code blocks, or a server-side library (Shiki, Prism with SSR) does it at render time.

On a Cloudflare Worker, both approaches have problems. I ended up writing a custom tokenizer. Here's why, and how it works.

Why Not Prism or highlight.js Client-Side#

Client-side highlighting loads a JavaScript library in the browser — Prism is ~30KB minified, highlight.js is ~30-900KB depending on how many languages you include. The library runs after the page loads, parses the code blocks, and applies color spans.

For a static blog, this means:

1. HTML arrives with plain, uncolored code blocks
2. JS loads and executes
3. Code blocks get colored

There's a flash of unstyled code. On slow connections it's visible. On fast connections it's just wasted work — the server could have done this.

Why Not Shiki Server-Side#

Shiki is a server-side highlighter that uses VS Code's TextMate grammars. The output quality is excellent — it handles every language correctly and matches editor themes precisely.

The problem: Shiki's full bundle is 7-8MB. Cloudflare Workers have a 1MB script size limit. Shiki doesn't fit.

There are ways around this (dynamic imports, separate Worker for highlighting, CDN-hosted WASM), but they add complexity I didn't want. The goal was a single-file Worker with no build step.

The Custom Tokenizer#

The approach: write a minimal tokenizer that handles the languages I actually use (JavaScript, TypeScript, shell, YAML, SQL, and a few others) and produces reasonable output for everything else.

The tokenizer works left-to-right through the source code, matching the highest-priority token at each position:

javascriptconst PATTERNS = [
  // Strings first — prevents false positives inside string content
  { type: "str", re: /("(?:[^"\\]|\\.)*"|'(?:[^'\\]|\\.)*'|`(?:[^`\\]|\\.)*`)/y },
  // Block comments
  { type: "cmt", re: /(\/\*[\s\S]*?\*\/)/y },
  // Line comments (// and #)
  { type: "cmt", re: /(\/\/[^\n]*|#[^\n]*)/y },
  // Config keys: word at line start followed by ":"
  { type: "key", re: /(?:^|\n)([\w-]+)\s*:/y },
  // Shell variables $VAR or ${VAR}
  { type: "var", re: /(\$\{?[\w]+\}?)/y },
  // Keywords
  { type: "kw", re: /\b(const|let|var|function|async|await|return|if|else|for|while|class|import|export|from|default|new|typeof|instanceof|null|undefined|true|false|void)\b/y },
  // Numbers
  { type: "num", re: /\b(\d+\.?\d*(?:e[+-]?\d+)?)\b/y },
];

The y flag on each regex is the sticky flag — it only matches at the current lastIndex position. This is what makes the left-to-right scan work without backtracking.

For each position in the string, the tokenizer tries each pattern in order. First match wins. Unmatched characters are treated as plain text.

javascriptlet pos = 0;
while (pos < code.length) {
  let matched = false;
  for (const { type, re } of PATTERNS) {
    re.lastIndex = pos;
    const m = re.exec(code);
    if (m && m.index === pos) {
      output += `<span class="hl-${type}">${escapeHtml(m[1])}</span>`;
      pos += m[0].length;
      matched = true;
      break;
    }
  }
  if (!matched) {
    output += escapeHtml(code[pos]);
    pos++;
  }
}

Overlapping tokens are handled by the priority order. Strings have highest priority — a keyword inside a string literal doesn't get colored as a keyword.

The Color Mapping#

The token types map to the site's color palette:

css.hl-kw  { color: var(--accent); }        /* yellow-green: keywords */
.hl-key { color: var(--accent2); }       /* cyan: config keys */
.hl-var { color: var(--accent2); }       /* cyan: shell variables */
.hl-str { color: #ff9f47; }              /* orange: strings */
.hl-num { color: #ff9f47; }              /* orange: numbers */
.hl-cmt { color: var(--text-dim); }      /* grey: comments */

The colors match the terminal animation on the landing page, so code blocks feel visually consistent with the rest of the site.

Limitations#

The tokenizer doesn't parse. It pattern-matches. It doesn't understand scope, type information, or language-specific semantics. A regex that looks like a string in JavaScript gets colored as a string. A keyword used as a property name (.return, .class) gets colored as a keyword.

For a blog, these edge cases are rare and low-stakes. Readers follow the logic, not the exact coloring. A tokenizer that's 60 lines and works for 95% of real code in blog posts is better than a 7MB library that works perfectly.

If I were building a code editor or an IDE plugin, I'd use a proper grammar-based approach. For a Worker that needs to highlight code blocks in blog posts, this is exactly enough.