jfo
— archive — feed — contact

An Unnamed Static Site Generator

Jul 13, 2022

I wrote a static site generator; to dispense with the "why."

This blog has been, in order, wordpress, middleman, jekyll, and hugo. Each time I become comfortable with one, inevitably they will become too slow for my needs (jekyll) or introduce subtle, minute breaking changes into some version or another, offering me an opportunity to jump ship (middleman and hugo) or just be not fit for purpose / total overkill (wordpress). I am tired of migrating every few years to get a random feature or please the platform. So I'll write my own. How hard can it be? Not that hard, really, as it turned out.

This post is about the most basic iteration with which this blog is currently being deployed. There are no bells and whistles, and though I might add some conveniences as I go along, it's current form is more than adequate for my needs at the moment, so I count that as a win. I spent part of the first week of my batch fiddling around with it, so I also gave a short presentation that first friday about it, which was recorded.

What are the benefits of rolling my own?

Configurability. I will no longer need to appease changelogs driven by someone(s) else's priorities, whatever they are. I can include the features I need or want and not the ones that I don't. You can't get much more configurable than "roll your own" after all.

Stability. The generator will change when I want it to. I'll have to keep up with platform changes, like version bumps in the runtime, but that happens with much more pomp and circumstance and more infrequently than package updates.

The learns. As always.

Let's get started, then.

What is a static site generator?

A static site generator is basically a compiler, right? We take an input (the source) transform it somehow and output it in some other form. This is a loose usage of "compile" but I'll stand by the general idea. Usually this is markdown -> html.

I love markdown, in spite of its troubled past. It's easy to read and write in its raw form, and as long as you don't expect it to do everything html can do seamlessly and can live with the occasional weirdness, it's fine.

You could sit down and write a whole post directly in html, but there are some pretty obvious downsides to that.

  1. It sucks. It's hard to do, error prone, and adds extra work. Writing is already a lot of work. I don't want to be farting around with mysteriously unclosed divs.

  2. It's not flexible. Markdown is a type of (semi)-standardized source, and it can thus be turned into differently formatted targets. Sure, so can html, but we've established that writing in html sucks, so why not use markdown?

  3. I'm used to it. YMMV.

So start with this: I want a program that takes as input some markdown and outputs some html.

What am I going to write it in? I've been wanting to try Deno out for a few years, so I'm going to try something new.

import { Marked } from "https://deno.land/x/[email protected]/mod.ts";

const sourcetext = "This is a string of markdown.";
const output = Marked.parse(sourcetext);

console.log(output);
// { content: "<p>This is a string of markdown.</p>\n", meta: {} }

So here is a program that takes in a source and outputs markdown. I am pulling my only dependency directly from https://deno.land/x. This is an officially hosted site for third party packages, of which there are quite a few high quality ones already, many but not all of them being ports of npm packeges.

You'll notice that the output turns out to be an object with the expected content, but what is meta? It turns out this is quite useful for a blog generator, as out of the box this markdown implementation supports parsing YAML frontmatter.

I ran this from the CLI by using deno run filename.js.

import { Marked } from "https://deno.land/x/[email protected]/mod.ts";

const sourcetext = `
---
title: 'This is a title'
---

This is a string of markdown.
`;
const output = Marked.parse(sourcetext);

console.log(output);
// {
//  content: "<p>This is a string of markdown.</p>\n",
//  meta: { title: "This is a title" }
// }

Frontmatter is at this point a pretty standard method for getting metadata attached to source markdown files in a static generator, all the way back to at least Jekyll. This obviates any need to keep some kind of index or central database of that metadata, and since it's completely arbitrary what I put in there, this will offer a lot of flexibility when it comes time to add f.ex tagging or syndication etc.

Next step is fairly obvious, I'd like to read in a file and apply this parsing to it. Let's pretend that file contains the string in sourcetext, above.

import { Marked } from "https://deno.land/x/[email protected]/mod.ts";

const input = Deno.readTextFileSync('./sourcetext.md');
const output = Marked.parse(input);

console.log(output);

This breaks.

error: Uncaught PermissionDenied: Requires read access to "./sourcetext.md", run again with the --allow-read flag
const input = Deno.readTextFileSync("./sourcetext.md");
                   ^
    at deno:core/01_core.js:106:46
    at unwrapOpResult (deno:core/01_core.js:126:13)
    at Object.opSync (deno:core/01_core.js:140:12)
    at openSync (deno:runtime/js/40_files.js:37:22)
    at Object.readTextFileSync (deno:runtime/js/40_read_file.js:30:18)

Deno is "secure by default". It takes a sandbox model of security in the same way that browser runtimes do, so if you want to give it access to your filesystem, you have to do so explicitly.

deno run --allow-read filename.js

This works.

I'm using the sync version of the file reading function here, for simplicity. I'm used to Node, where for scripting it's easiest to just use synchronous I/O for smaller scripts. I'll come back to this later to discuss top level-awaits, which are genuinely exciting to me.

Of course, I don't just want to print the output to the console, I want to produce some html from the source.

import { Marked } from "https://deno.land/x/[email protected]/mod.ts";
import { paramCase } from "https://deno.land/x/case/mod.ts";

const input = Deno.readTextFileSync("./sourcetext.md");
const output = Marked.parse(input);

Deno.writeTextFileSync(
  `./${paramCase(output.meta.title)}.html`,
  output.content
);

Notice again I'm importing a helper module, this time to do some case manipulation so that I can produce a url ready filename from the metadata.

This also breaks. I need explicit write access, too, and it is separate.

deno run --allow-read --allow-write filename.js

This works.

If you're thinking that we're already frighteningly close to a full-fledged static site generator, well then you'd be mostly right.

import {
  ensureDirSync,
  expandGlobSync,
} from "https://deno.land/[email protected]/fs/mod.ts";
import { paramCase } from "https://deno.land/x/case/mod.ts";
import { Marked } from "https://deno.land/x/[email protected]/mod.ts";

// We create the output directory:
ensureDirSync("./build");

const inputFiles = expandGlobSync("**/*.md");
for (const file of inputFiles) {
  const input = Deno.readTextFileSync(file.path);

  // We write the file processed through markdown:
  const markup = Marked.parse(input);
  Deno.writeTextFileSync(
    `./build/${paramCase(markup.meta.title)}.html`,
    markup.content,
  );
}

This script (which is basically all it is at this point) will grab all of the markdown files in the directory it is run in and create a build folder with the processed output of each one in its corresponding html file. For a pretty stripped down idea of what a static site generator is, this ~20 line script basically fits the bill. Pretty neat! Let me add one more thing.

import {
  ensureDirSync,
  expandGlobSync,
} from "https://deno.land/[email protected]/fs/mod.ts";
import { paramCase } from "https://deno.land/x/case/mod.ts";
import { Marked } from "https://deno.land/x/[email protected]/mod.ts";
import { render } from "https://deno.land/x/mustache_ts/mustache.ts";

// We create the output directory:
ensureDirSync("./build");

const inputFiles = expandGlobSync("**/*.md");
for (const file of inputFiles) {
  const input = Deno.readTextFileSync(file.path);

  // We write the file processed through markdown:
  const markup = Marked.parse(input);
  const renderedContent = render(markup.content, {
     img: () => (text) => `<image src="/${text}" />`,
  });

  Deno.writeTextFileSync(
    `./b/${paramCase(markup.meta.title)}.html`,
      render(Deno.readTextFileSync("./templates/post.mustache"), {
        content: renderedContent,
        title: markup.meta.title,
        date: markup.meta.date,
      }),
  );
}

I'm introducing a mustache rendering pass here, where ./templates/post.mustache maybe looks, oversimplified, something like:

<!DOCTYPE html>
<html>
  <head>
    <title>{{title}}</title>
  </head>

  <body>
   <article>
     <h1>{{title}}</h1>
     <sub>{{date}}</sub>
     <div>{{content}}</div>
   </article>
  </body>
</html>

Mustache lets me pass in variables that can be in scope during rendering, which allows each post to define its own title and date and any other metadata I want to interpolate in the yaml frontmatter. I can also do neat things like the img helper lambda passed into the first pass render above:

const renderedContent = render(markup.content, {
  img: () => (text) => `<image src="/${text}" />`,
});

Inside a post, now I can use this like a function like so:

{{#img}}name-of-file.jpg{{/img}}

I hope you can see how flexible and easy this could be for extending things!

This is basically all it does. The code I've ended up with has a couple more steps that flow straightforwardly from what I've already shown above... I render an archive page, and an RSS feed, and shuffle the logic around a bit to be more modular and also have a set of options that can be passed in. You can see the current source here.

Why was this so easy?

I'm using libraries that fit my use case really well. If I had to start from "write a markdown parser/generator" this would be a longer first post. The fact that it supports frontmatter out of the box like that even is quite the stroke of luck.

Deno is much more "batteries included" than node ever was. ensureDirSync and expandGlobSync are just part of the standard library in deno. In node, the former requires a tiny bit of ceremony, but the latter necessitates a whole-ass library. Classic node. I am not 100 sold on Deno but it's quite pleasant to work with on such a low stakes project.

I may add more features to this generator and/or write more about it in the future but I hope this has demonstrated how easy it was to cobble something together that genuinely does fulfill my modest needs and provides me with a lot of benefit in terms of writing comfortably.