How my blogroll gets generated (now completely automatic!)

Yesterday I teased a new post about how I automated my blogroll generation by writing a Go script and using the Miniflux API. Here it is.

Up until yesterday, I always updated my blogroll manually. I wrote about that process here. It was a bit of automation but not completely automated. I fixed that and now do what Robert van Bregt suggested or what Jan Boddez does with his WordPress plugin.

To generate my site I use a self-hosted instance of Drone CI (I already wrote about that a few times) with a custom Docker image. That image includes Hugo (to build the site), but it also includes Go itself instead of just the Hugo binary, because Hugo modules require Go to be installed.

Because Go is already installed and I’m way more fluent in Go than in shell-scripting, I decided to write a small Go tool. First I wanted to use the official Miniflux Go client but it contains a lot of stuff that I don’t need and I wanted to limit the script to only use the Go standard library (so it doesn’t always need to download dependencies first – and the standard library already contains tools to parse and produce XML - OPML is XML - and JSON). The Go script gets called by a simple go run tools/blogroll/blogroll.go, I also didn’t want to commit any binaries to my blog’s Git repository.

You want to see the script? Here it is (at the time of this article’s publication): (Update: I updated the script!)

package main

import (
"encoding/json"
"encoding/xml"
"fmt"
"io/ioutil"
"net/http"
"time"
)

type Opml struct {
Body struct {
OpmlOutline
} `xml:"body" json:"body"`
}

type OpmlOutline struct {
Text string `xml:"text,attr" json:"_text,omitempty"`
Title string `xml:"title,attr" json:"_title,omitempty"`
XmlUrl string `xml:"xmlUrl,attr" json:"_xmlUrl,omitempty"`
HtmlUrl string `xml:"htmlUrl,attr" json:"_htmlUrl,omitempty"`
Outlines []OpmlOutline `xml:"outline" json:"outline,omitempty"`
}

func main() {

// Get OPML
minifluxClient := http.Client{
Timeout: time.Second * 5,
}
req, err := http.NewRequest(http.MethodGet, "https://MY-MINIFLUX-DOMAIN/v1/export", nil)
if err != nil {
fmt.Println(err)
return
}
req.Header.Set("Content-Type", "application/json")
req.Header.Set("Accept", "application/json")
req.Header.Set("X-Auth-Token", "API-TOKEN")
res, getErr := minifluxClient.Do(req)
if getErr != nil {
fmt.Println(err)
return
}
if res.Body != nil {
defer func() { _ = res.Body.Close() }()
}
if res.StatusCode != http.StatusOK {
fmt.Println("Status code: ", res.StatusCode)
return
}
opmlBytes, err := ioutil.ReadAll(res.Body)
if err != nil {
fmt.Println(err)
return
}

// Unmarshal
opml := &Opml{}
err = xml.Unmarshal(opmlBytes, opml)
if err != nil {
fmt.Println(err)
return
}

// Filter
newOpml := &Opml{}
for _, outline := range opml.Body.Outlines {
if outline.Text == "Blogs" || outline.Text == "Comics" {
newOpml.Body.Outlines = append(newOpml.Body.Outlines, outline)
}
}

// Marshal as JSON
jsonOpmlBytes, err := json.MarshalIndent(newOpml.Body, "", " ")
if err != nil {
fmt.Println(err)
return
}

// Write to file
err = ioutil.WriteFile("data/opml.json", jsonOpmlBytes, 0644)
if err != nil {
fmt.Println(err)
return
}

}

My goal was to create a data/opml.json file that was as close to the one I generated manually before. Nevertheless I had to slightly modify the blogroll Hugo shortcode from the previous post. It currently looks like this:

{{ $opmlJson := index .Site.Data.opml "outline" }}
{{ range $opmlJson }}
{{ (printf "%s %s" "##" ._text) | markdownify }}
<ul>
{{ range sort (index . "outline") "_title" "asc" }}
<li><a href="{{ ._htmlUrl }}" target="_blank">{{ ._title }}</a> (<a href="{{ ._xmlUrl }}" target="_blank">Feed</a>)</li>
{{ end }}
</ul>
{{ end }}

It now enables displaying multiple categories of feeds (“Blogs” and “Comics” in my case) and is adapted to the new JSON format.

Most things (API key or filtered categories) in the Go script are hard coded, because I didn’t want to make it generic. If someone uses it or takes inspiration from my code, they would probably modify it anyway. It also made things easier. After all it’s just a simple HTTP request, parsing, filtering, and printing to a file. Most of the lines are basic error handling, so that an error in updating the opml.json file doesn’t result in a failure of the blog build pipeline.

So feel free to checkout the blogroll and the featured blogs!

Interactions & Comments