GSoC 20: Week 8-9

#gsoc #opensource #python

Hello everyone,
I am Niraj and sorry for skipping blog of last week. I got carried away in other tasks. Anyway, I am sharing work I have done during both Week 8 and 9 today.

What did I do in past two weeks?

I have done research on various configuration file formats and compiled outcomes of it in the Discussion: Configuration file format. Some users recommended INI files because it is very old and still popular among masses but INI file does not have any built-in type support and It also lacks formal specification. It parses everything as string. So, we have to process data parsed by configparser to convert it into something usable.

For example if INI file contains following data:

[checker]
; list of checkers you want to run
runs = [curl, binutils]

It will be parsed as following dictionary by configparser:

{
    "checker": {
        "runs": "[curl,binutils]",  # This has to be transformed into list 
    },
}

So, parsing INI file won't be as easy as TOML or YAML which supports complex datatypes by default. It is also not easy to parse other datatypes like integer, float etc.

TOML is very similar to INI file and TOML also supports complex data types by default. As you can see our example config file has been parsed correctly by toml parser:

[checker]
# list of checkers you want to run
runs = ["curl", "binutils"]

{
    'checker': {
        'runs': ['curl', 'binutils'],  # this is correctly parsed as list
    },
}

I concluded that TOML and YAML are both very easy to read and write by both machine and human. So, we should use one of them. We have discussed which format to use in meeting and my mentors had various opinions on it. Summary of our discussion was:

The top contenders among our team seem to be TOML (readable, familar to python folk and close enough to INI for skill transfer for windows folk) and YAML (which might be a better fit for the dev-ops community that we hope will be among the biggest users of cve-bin-tool).

Since parsers for both formats produce similar Python structures, I have created ConfigParser class which can parse both YAML and TOML file format. I have also added basic tests for it. I have also changed architecture of main function of cli.py to add support for config files and To make sure that option given from terminal get preference over config option, I have also written tests for that. I have also fixed quiet mode bugs.

Since, it's final phase of the GSoC and most of my project is completed, I am going to focus on documentation of the project in this third and final phase.

I have changed user manual and project readme. I am also going to change other documentation. I have created user manual for new input engine features and config file feature.

What am I doing this week?

I have talked with a user and we come to conclusion that our documettion lacks some important How-to guides which are necessary as mentioned by Daniele Procida in his amazing PyCon talk. So, I am going to create a How-to directory inside our doc directory which will contain interesting recipes for different usecases. Ex: