← back

A ((code generator) generator)

This is a ((code generator) generator).

a (code generator)

A code generator takes some sort of configuration and produces code from it:

config + generator -> code

as an example:

open api spec + generator -> API Client

or

markdown files + generator -> static website files

code generators are neat because they allow us to impart high degrees of structure in our app layout, save ourselves from writing boilerplate, and ship changes across a huge surface in a programatic way. They are also high leverage. if you build a reasonable code generator, you can share it across many projects.

for example, we can change the codegen step to switch between fetch and axios:

open api spec + fetch-based generator -> API Client using fetch

open api spec + axios-based generator -> API Client using axios
                ^ i changed

if all of your projects use the same generator, this can be an "easy" migration in all of your projects.

most of these are pet examples, but you can do arbitarily complex things with code generation if you are so inclined.

Note: A downside of code generators is that your abstraction can be bad, and you have to "leak" around it to express the code you are actually after.

so what goes into a (code generator)?

First, we have to have a defined "shape" of the config we are going to recieve:

config + generator -> code

^ we must be able to understand this config

such that we can accept a config and know how to read it. an example shape might be:

posts:
    title: Backward Compat Guidelines,
    title: Securing an API

and our generator itself may take the following steps:

  1. read in the config
  2. create a file for each post based on title
  3. create a blank post template for the post
  4. save the files onto disc in some location on our computer

and that's it! this would be a fairly simply but effective code generator.

can we make a generator for these code generators?

so then I wondered if I could abstract out this pattern one "layer" higher.

of our steps, 1 and 4 are going to be shared with ~all code generators.

as I go around creating code generators for my projects, it woild be nice if they all shared the same "primative" so that as I learn more, i can share the learning across different code generators

so lets work on a new machine:

  • this machine provides a language for us tod express a configuration shape
  • it gives us for free the utilities to parse and validate that shape
  • it specifies an interface for converting the config into outputs
  • it provides machinery to write those files to disc

and some psudo-code:

-- I AM NOT SHARED
makeShape :: Shape
makeShape =
 -- specify the shape we expect from the config

-- I AM NOT SHARED
produceOutfiles :: Config Shape -> OutFiles Shape
-- this code is unique to each `generator`


configParser :: File -> Shape -> Either (Config Shape) Errors
configParser configFile shape =
    -- read in the file and convert it to a config object, aka an "instance" of a Shape
    -- return errrors if we cant parse it correctly

writeFiles :: OutFiles -> Either FilesWritten Errors
 -- do the actual IO of writing files.
 -- something, something monad

emit :: Config Shape -> Either FilesWritten Errors
emit config =
     productOutfiles . writeFiles $ config

the unshared code is makeShape and produceOutputFiles.

everything else can be shared. ok, so to specify a code generator we say:

CodeGenerator = {
  shape: Shape;
  produceOutfiles: Config Shape -> OutFiles

}

and we have:

doEverything :: File -> CodeGenerator -> FilesWritten
doEverything providedConfig generator =
  -- use above to generate

this is more or less a ((code generator) generator)!

we can build up various code generators with this primative that share a lot of code.

limitations

I wrote the psudo-code in haskell, because i wrote the actual code in haskell as well.

One limitation, is that we have to define all the code generators at compile time, since they are just defined in the actual haskell codeblocks.

This also makes this tool hard to "share" since it requires one to recompile the project with their code.

We may also want to run more than once code-generator on a specific config. For example we have both generate an API client and Documentation from an API specification. This would require us to invoke generator more than once. This is not really a problem, since the output sets are distinct:

my_project/
    api_cient/
         {files go here for the client}
    docs/
         {files go here for the docs}

But what if we want to run more than one generator on the same set of files:

personal site config -> generate static site -> proxy image links through CDN

It's not really clear if we should ever this. "Second order" sounds kind of scary and hard to reason about if you did anything with substantive complexity.

Maybe there is a good reason to do this, but it evades me.

why did i do this

One of my motivations for this project was to be able to spin up really quick microsites:

  • A disc golf score keeper / stat tracker
  • A score keeper for a card game my partner and I play
  • A TODO list app
  • A packing list app for a trip
  • A warmup routine for my climbing sessions
  • etc

Of course LLM's are really good at spinning sites like this up, but I want to host them all the same, build them all the same, and have the same "dev flow" for them.

So I want to be able to specify 90% of the boilerplate for these sites and get the exact same thing out the other side "for free".

(I made a "hosted" sqlite site that lets me spin up tables and gives me a CRUD API for those tables.)

  • an auth flow to grab a token from an auth server for the DB API
  • a generated "Client" for the frontend to talk to backend DB.
  • static site forms and pages that are optimized for easy deploy, mobile, my preferences, etc.
  • a narrow band of "custom" code per app that builds whatever we need for that specific app. (which i will largely delegate to an LLM)

Then if I change my auth server in the future, i simply regenerate all the projects at once and its updated. If i change the apps to work with a local copy of Sqlite instead of my normal backend, i just need to configure this once. etc.

I'm sure this will have a ton of downsides, but for now it does seem to be a pretty interesting approach!