DEV Community

Sérgio Araújo
Sérgio Araújo

Posted on • Updated on

Combining awk and vim to get real power

#awk #gawk #vim #neovim

Foreword

After the initial post of this little, "how to", @Sundeep commented:

why not use awk itself to insert the snippet? since you are using gawk, you can also use -i inplace if you want inplace editing

I don't remember exactly, but I think you can use silent to avoid having to use Enter manually, check these: ...

I will explain the scenario, my old solution and then the gawk solution

The Scenario

I have about 500 markdown files on my personal vimwiki, for years I have been making notes to help myself and others through Linux/vim commands. Recently I started using pandoc to convert my markdown files to html and LaTeX.

using vim + gawk

I have two mappings on my ~/.config/init.vim that makes it easy to convert my current documment to html and pdf. During the process of convertion pandoc warns about the lack of the title, and this brought me here.

if !exists('*PandocMappings')
    function! PandocMappings()
        " NOTE: <F16> equals to Shift-F4
        noremap <F4> :! pandoc '%:p' -o /tmp/'%:p:t'.html --template=easy_template.html --toc && firefox /tmp/'%:p:t'.html &<CR><CR>
        noremap <F16> :! pandoc '%:p' -o /tmp/'%:p:t'.pdf --pdf-engine=pdflatex --toc && evince /tmp/'%:p:t'.pdf &<CR><CR>
    endfun
endif

augroup my_markdown
    autocmd!
    autocmd FileType markdown call PandocMappings()
augroup END
Enter fullscreen mode Exit fullscreen mode

My markdown files lack metadata, something like this:

--------
file: the file name
lang: pt-BR
author: Sergio Araujo
date: abr 03, 2019
created: abr 03, 2019
abstrac: This aims...
--------
Enter fullscreen mode Exit fullscreen mode

After adding manually the metadata in about eight of them I realized that I had better tools and the knowledge (including web search) to solve my problem. The metadata should start at first line.

Finding out the files with no metadata

Using awk, more specificly gnu awk "gwak" printing the list of files who have the metadata is simple

gawk '(FNR==1 && $1 ~ /^---/) {print FILENAME}' *.md

FNR==1  ...................... the first line equals to the first register
&& ........................... it allows us to test a second condition
~  ........................... match regular expression
/^---/ ....................... the regular expression itself
Enter fullscreen mode Exit fullscreen mode

The command above showed me the files I already had inserted the metadata, inverting the regex is very simple, just use exclamation. So the list of files that did not had the metatada can be printed with: 

gawk '(FNR==1 && $1 !~ /^---/) {print FILENAME}' *.md
Enter fullscreen mode Exit fullscreen mode

Combining the awk result with vim

Using the shell Command Substitution we can deliver to vim the list we wanted.

vim $(gawk '(FNR==1 && $1 !~ /^---/) {print FILENAME}' *.md)
Enter fullscreen mode Exit fullscreen mode

The vim/neovim argdo command

When you open vim with something like:

vim *.md
Enter fullscreen mode Exit fullscreen mode

All the opened files are in the arglist, so we can use something like:

:argdo command
Enter fullscreen mode Exit fullscreen mode

A UltiSnippet snippet to insert the header 

snippet headermd "markdown header" w
--------
file: `!p snip.rv = snip.fn`
lang: pt-BR
author: `!v g:snips_author`
date: `!v strftime("%b %d, %Y")`
created: `!v strftime("%b %d, %Y")`
--------
endsnippet
Enter fullscreen mode Exit fullscreen mode

the vim function that can be called by argdo to inserts the snippet 

fun! InsertHeaderMd()
    0pu_
    exe "normal! iheadermd\<C-r>=UltiSnips#ExpandSnippet()\<CR>"
endfun
Enter fullscreen mode Exit fullscreen mode 
0pu_  ............................ On the line zero insert black hole register (blank line)
exe normal! ...................... normal command
iheadermd ........................ enters insert mode and types headermd
\<C-r>= .......................... expression register
UltiSnips#ExpandSnippet()\<CR> ... the UltiSnippet trigger to call the snippet
Enter fullscreen mode Exit fullscreen mode

Finishing up everything

We do not even have put the function (just the snippet) on the configuration files, you can copy the function to the clipboard and type this on vim:

:@+
Enter fullscreen mode Exit fullscreen mode

Now with all files opened you can do:

:silent argdo call InsertHeaderMd() 
:silent argdo update
Enter fullscreen mode Exit fullscreen mode

Thanks to @Sundeep I added the silent option which allowed me run the command without hiting Enter a bunch of times. And he also suggested awk as a solution

Using a vim macro with argdo

Opening the files without metadata

vim $(gawk '(FNR==1 && $1 !~ /^---/) {print FILENAME}' *.md)
Enter fullscreen mode Exit fullscreen mode

Creating a vim macro

:let @a=''
:let @a="ggO---\<Return>title: \<C-r>=expand('%:t')\<CR>\<Return>author: Sergio Araújo\<Return>abstract:\<Return>---\<Esc>"
Enter fullscreen mode Exit fullscreen mode

Explanation

let @a='' ............................ delete any marcro 'a'
ggO .................................. got to the first line and open a line above
---\<Return> ......................... insert three dashes and a new line
\<C-r>=  ............................. starts a expression register
expand('%:t') ........................ expands the filename "t = tail" 
\<Esc> ............................... goes back to the normal mode
Enter fullscreen mode Exit fullscreen mode

Running the argdo command plus a macro

:set hidden ............................ allows to go to the next file without saving

:silent argdo execute "normal! @a" | update 

:silent ................................. does not show any messages
Enter fullscreen mode Exit fullscreen mode

The gawk solution

This solution came from a @Sundeep comment

  var=$(date "+%b %m, %Y")
  awk -i inplace -v date="$var" 'BEGINFILE {print "--\nfile: "FILENAME"\ndate: "date"\nabstract: \n---"}; (FNR==1 && $1 !~ /^---/) {print}' *.md
Enter fullscreen mode Exit fullscreen mode

Job done!

Explaining the awk long command... 

-i inplace ..................... makes possible edit files directly in awk
-v date="$var" ................. captures the content of var=$(date "+%b %m, %Y")
BEGINFILE ...................... insets some strings "between quotes"
FILENAME ....................... gawk variable that returns the file name
FNR==1  ........................ the first line equals to the first register
&& ............................. it allows us to test a second condition
~  ............................. match regular expression
/^---/ ......................... the regular expression itself
Enter fullscreen mode Exit fullscreen mode

The next step will be using a variable using shell Here Documents. If you guys can help me with this I would thank you a lot!

Could it be done with sed?

I know that sed can also insert a header in a bunch of files, but I should have write a shell script to get each file name or use -V sed option. The solution I figured out was good for my problem, and it also inspired me enough to share this ideas with you. Feel free to comment and suggest alternatives or improvements. (Please fix my typing, I am not a native English speaker).

References:

Top comments (3)

Collapse
 
learnbyexample profile image
Sundeep •

why not use awk itself to insert the snippet? since you are using gawk, you can also use -i inplace if you want inplace editing

I don't remember exactly, but I think you can use silent to avoid having to use Enter manually, check these:

Collapse
 
voyeg3r profile image
Sérgio Araújo •

Thanks @Sundeep for your tips! I will update my post. But actually the real purpuse of it was reached, since many people will learn more about awk, vim, and of course, I will learn more with your great suggestions.

Collapse
 
voyeg3r profile image
Sérgio Araújo • • Edited

Hi @Sundeep, I have made some changes on my article, now it shows a pure awk solution, a vim macro solution + awk. Both brings us more ideas for the future needs.

Read next

stormsidali2001 profile image

Next.js 15 Deep Dive: Building a Notes App with Advanced Features

Sidali Assoul -

i_am_vesh profile image

Shell Scripting in DevOps: A Complete Guide

Avesh -

abhijeetx1 profile image

Building a Scalable Real-Time Job Board with React, Node.js, and Google Authentication

Abhijeet -

pucinsk profile image

Ruby exceptions: tips & tricks

Jokūbas -