Building Out A Dev Blog
Here we go again! If you've been following these posts, welcome back, and if not welcome aboard! This time we'll be continuing our work to extend the basic code we put together to create a static site generator. We actually jump quite a bit forward this week. We've now are at a point where we can convert an entire directory of well-formatted markdown files into HTML ready to host. Anyway, let's keep on with our attempts to learn Go...
Diving In
I'm still toying around with a different way to show our updated code but, for now, I think the code base is small enough I can just start from the top and work my way down. I'm going to gloss over the bits that haven't changed since last week so, if you haven't read that I suggest you do!
Now then, let's jump down to our first real changes.
package main
import (
"bufio"
"bytes"
"fmt"
"html/template"
"io/ioutil"
"os"
"regexp"
"strings"
"github.com/microcosm-cc/bluemonday"
"github.com/russross/blackfriday"
yaml "gopkg.in/yaml.v2"
)
const delim = "---"
type post struct {
Title string
Published bool
Description string
Tags []string
CoverImage string
Series string
PostBody template.HTML
}
var templ = `<!DOCTYPE html>
<html lang="en">
<head>
<title>{{.Title}}</title>
<meta charset="utf-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="HandheldFriendly" content="True">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<meta name="referrer" content="no-referrer-when-downgrade" />
<meta name="description" content="{{.Description}}" />
</head>
<body>
<div class="post">
<h1>{{.Title}}</h1>
{{.PostBody}}
</div>
</body>
</html>
`
Naming Stuff Is Hard
Here we get to the first (small) change, the function is the same as last week but I've renamed it from loadFile() to getContents(). It seems to more accurately capture what it does.
func getContents(f *string) ([]byte, error) {
b, err := ioutil.ReadFile(*f)
if err != nil {
return nil, err
}
return b, nil
}
One thing you should notice throughout this section is I've tried to make sure we're are breaking out code into standalone functions. This should keep our main() nice and make it so we can write tests. Below we have parseFM() which wraps up our un-marshaling and returning the first stage of our front matter. It may be more correctly called parseYAML() but, we'll leave it as is for now. isNil() has not changed and still is not really used, I will see to that before the next post, possibly.
func parseFM(b *[]byte) (map[string]interface{}, error) {
m := make(map[string]interface{})
err := yaml.Unmarshal(*b, &m)
if err != nil {
msg := fmt.Sprintf("error: %v\ninput:\n%s", err, b)
return nil, fmt.Errorf(msg)
}
return m, nil
}
func isNil(i interface{}) bool {
if i != nil {
return false
}
return true
}
As before we need to split up the data we loaded from the markdown file. A call to splitData() will take care of it. I've fed in various bits of malformed data and the if statement seemed to work as expected each time. I also ran my current set of markdown dev.to posts through and each was converted without a problem. They can all be found over at dev.shindakun.net.
func splitData(f *[]byte) ([][]byte, error) {
b := bytes.Split(*f, []byte(delim))
if len(b) < 3 || len(b[0]) != 0 {
return nil, fmt.Errorf("Front matter is damaged")
}
return b, nil
}
Once we've split the data and parsed out the front matter we call makePost() to build the post struct and the template. I'm passing back the *post because we're currently executing the template just before we write the file to disk, not in this function itself. It will also be handy when I build out something to create an index.html. Again, we're not really using isNil() as intended mostly because I don't like the if/else jumble that it will create. We'll get that taken care of one of these days.
// makePost creates the post struct, returns that and the template HTML
func makePost(fm map[string]interface{}, contents []byte,
s [][]byte) (*template.Template, *post) {
p := &post{}
if isNil(fm["title"]) {
panic("isNil tripped at title")
} else {
p.Title = fm["title"].(string)
}
p.Published = fm["published"].(bool)
p.Description = fm["description"].(string)
// TODO: Strip space after comma prior to parse?
tmp := fm["tags"].(string)
p.Tags = strings.Split(tmp, ", ")
p.CoverImage = fm["cover_image"].(string)
p.Series = fm["series"].(string)
pBody := contents[len(s[1])+(len(delim)*2):]
out := blackfriday.Run(pBody)
bm := bluemonday.UGCPolicy()
bm.AllowAttrs("class").Matching(regexp.MustCompile("^language-[a-zA-Z0-9]+$")).OnElements("code")
p.PostBody = template.HTML(bm.SanitizeBytes(out))
tm := template.Must(template.New("msg").Parse(templ))
return tm, p
}
There you go, we're down to our main() function! Right of the bat, you'll see that we're now using ioutil.ReadDir() to get a list of the contents of the current directory.
func main() {
d, err := ioutil.ReadDir(".")
if err != nil {
panic(err)
}
We then range through the list of the current directory and immediately check to see if the current file ends with .md. If it does the address of our string gets passed into getContents() and if all goes well we'll get the contents back.
for _, f := range d {
if t := f.Name(); strings.HasSuffix(t, ".md") {
contents, err := getContents(&t)
if err != nil {
panic(err)
}
We'll pass a pointer to our contents into splitData() and then pass the second chunk of s, that is s[1], into parseFM(). This gives us the base building blocks for our post. Both splitData() and parseFM() could probably be moved into makePost(), then we could pass the contents directly into it and do all the work there. That's something to consider for when refactoring.
s, err := splitData(&contents)
if err != nil {
panic(err)
}
fm, err := parseFM(&s[1])
if err != nil {
msg := fmt.Sprintf("error: %v\ninput:\n%s", err, s[1])
panic(msg)
}
tm, p := makePost(fm, contents, s)
In this current version of the code, we're going to take the current filename and lop off the extension. We'll then create a new file with the same name as the original and append .html. Again, this is all currently happening in the same directory and we're being really sloppy and not checking for existing files or anything. os.Create() uses 0666 (or equivalent permissions in Windows) for the file permissions so in theory we should be able to write and overwrite with no issues. This is another case where we might not be doing what may be considered "correct" - instead, we are just going through an iterative process. This lets everyone see how the code has changed, warts and all.
File created, o, we then use bufio.NewWriter() to create a "Writer" with a buffer. We can then execute our template and write directly into the file which is closed after we flush the buffer.
fin := strings.TrimSuffix(t, ".md")
o, err := os.Create(fin + ".html")
if err != nil {
panic(err)
}
defer o.Close()
buf := bufio.NewWriter(o)
err = tm.Execute(buf, p)
if err != nil {
panic(err)
}
buf.Flush()
}
}
}
If we have any files left we jump back up to the top of our loop and do the whole things again!
Next Time
We have just about everything in place! But first, we need to create and write out index.html. I think I'm going implement that over the next day or so and put up a quick follow up post. After that, I'll need to decide if we want to do one more post in the "dev site" series and extend the code to use external templates. It shouldn't be that much more work, I suppose I'll wait and see how busy the next week looks on Monday. See you next time!
You can find the code for this and most of the other Attempting to Learn Go posts in the repo on GitHub.
shindakun
/
atlg
Source repo for the "Attempting to Learn Go" posts I've been putting up over on dev.to
Attempting to Learn Go
Here you can find the code I've been writing for my Attempting to Learn Go posts that I've been writing and posting over on Dev.to.
Post Index
| Enjoy this post? |
|---|
| How about buying me a coffee? |
Top comments (8)
About this snippet:
You're taking in
*stringwhenstringis better for what you want to do.But more importantly, you are not checking your pointer:
I'll make the assumption that you are doing it this way because you are already checking before the function, but I would say that this would be better done inside so you never have to think about it again.
On another note: I don't think the function expresses what it's doing correctly. Consider the following alternative signatures:
func getFileContents(filename string) ([]byte, error) {}func contentsOf(f *os.File) ([]byte, error) {}func contentsOf(r io.Reader) ([]byte, error) { return ioutil.ReadAll(r) }And in that last case, if you're operating on readers, why not just... not use a function at all!
As an
*os.Fileis aReader, you would eliminate the boilerplate code, and be able to leverage one of the most powerful interfaces in the standard library.Thanks for the comments! I really should be passing in just the string as you've pointed out, there is no reason to use the pointer. My thinking in making this a function on its own was to be able to write tests for it later, which I suppose means I had better be checking for that pointer. I'm not sure just wrapping reading a file into a function is worth it just to be able to write a test down the road though so it might be worth doing it away with it entirely.
Definitely, something for me to consider. Thanks again!
Worth considering as well that using a
Readerrather than a concrete file here will make testing this easier later anyway, since you'll be able to use anyReader. not just a file, you can mock your input using buffers or any oldstrings.NewReader("thing")!Thanks for taking the time to respond.
That is a fantastic point! I think using a
Readeris going to be the way I go.Hey, nice post series. I always learn something new from it.
What would you think of renaming some functions to make them more unambiguous? I know, Go is known for making names as short as possible, but ... 🤔
For example
The code within explains itself, as it is short enough, but if you read only the function signature, it is not quit clear. Does it get the content of a string and returns it as a byte slice? What, if the function gets longer? I would have to skim a lot more code to understand it.
How about
or
And why do you use a reference to the ˋfileˋ string instead of a simple value?
Another one would be
Abbreviations are hard to follow, it not within the closed context of a function. Maybe
would make it instantly clear?!
Looking forward to your next post.
Thanks for the comment! It would definitely make it clearer. I always go back and forth on names, I've been thinking about learning more toward descriptive functions like
getFileContent(f string)especially for stuff I'm going to be posting. Now, is probably a good time to try and commit to that. I can't remember why I wrote it that way and didn't just send in a string. It's not like we're memory constrained it doesn't need to be that way and the passed in string would just be garbage collected eventually.Hi, good job :)
Try avoiding things like
f *[]byteas slices and maps are already pointers by themselves. Also, it's a good practice to check whether entries exist in the map before converting them to types:Could be:
While it does seem redundant in personal/toy projects, I have seen some applications panic and crash due to such simple assumptions when someone later modifies the input map and misses some values :)
P.S. It would be a lot more robust if you parsed that yaml into a struct!
Thanks for the comment! I've been putting off dealing with checking on the front matter map for two posts. 😅
isNil()basically only exists for checking the map but I keep putting off implementing the full thing because I was working with "known good" files. But I kind of like the pattern in your example I will probably be using that. I think at some point it might go into a struct if I keep extending the code.