Hugo Data Files

I started using Hugo to generate this site in mid-2017. While I've only added ten new posts in the year and a half since, I have also been keeping local copies of all of my Twitter interactions. My original implementation had every single Tweet as its own file.  This resulted in over ten thousand files, which in turn resulted in increasingly slow processing times for Hugo every time I added a new post or Tweet.

In my case, "slow" was about 15 seconds. That's still pretty impressive, since this site currently runs on a dual CPU virtual machine with 2GB RAM.  The majority of the processing time was clearly spent in file I/O operations.

Today I've revamped how I store all my Tweets, making use of Hugo data files. Tweets are now stored in YAML files organized by year and month. This has dramatically reduced the number of files involved in generating the site. Current build times are now down to about 1.5 seconds!

While updating my micropub library and theme files to deal with this, I also extended them to finally properly support Indieweb post types like bookmarks and favorites.  I don't use favorites, myself, but the addition of bookmarks means I can pretty trivially move over the hundred or so bookmarks I've had sitting in Pinboard for the last five years.

As usual, it's been a fun and frustrating experience dealing with the Hugo docs, the Micropub specification, and my own paltry abilities with PHP. I don't think I'll ever be done tinkering with things here, but that's okay with me.