How to create raw bytecode in python

Introduction

Well, first I had started doing this on github.io, but then a group of virtual gangsters beat me up and made me realize that there was no point in using that having this fantastic website. Well, there it is, i use this, as the title says, this is a raw python bytecode tutorial, i hope you enjoy it (Because there is a second part...)

Pre requirements

• Basic knowledge of Python
• Know what a bytes object is
• Know the concept of stack

What is Python?

Python is a multiparadigm interpreted programming language, it supports polymorphism, object-oriented programming (OOP / OOP) and imperative programming.

How does it work?

Python, as already named, is an interpreted language, this means that it passes through an interpreter that connects what the computer is going to do, with what you write. Python does not generate machine code as a C or C ++ program would generate, but rather works more or less like Java, it has a virtual machine that interprets bytecode. This default interpreter is CPython, which is responsible for executing the bytecode on your computer. Here we are not going to use compilers, but rather we are going to handle language implementations, basically interpreters that interprets (forgive the redundancy) the written code after translating it into bytecode. There is a wide variety of these, e.g. IronPython (C # implementation), Jython (pure Java implementation), Micropython (C version optimized to run on microcontrollers).
Here is a schematic of how Python works and the steps that the interpreter takes to run the code that you wrote.

How to create USABLE bytecode

Well, we have two things, first, stripped bytecode, that is, bytes in hexadecimal representing opcodes and parameters, and secondly, we have CodeType, a data type in Python that helps us to create ByteCode that SUITABLE AND USABLE. Also to assemble, you have to know how to disassemble, we are going to use the module dis, this module is used to disassemble functions, files and code.

import dis

def sum (x, y):
    return x + y
dis.dis (sum)

The output of that snippet of code is as follows

1. 4   0 LOAD_FAST    0 (x)
2.     2 LOAD_FAST    1 (y)
3.     4 BINARY_ADD
4.     6 RETURN_VALUE
>>>

As we can see, all of that is bytecode, now the explanation.

As you may have noticed, I listed the lines in the output in order to make this explanation easier.
Each instruction in Python has a specific OPCODE (Operation Code), in this case we use 3, LOAD_FAST BINARY_ADD RETURN_VALUE, we will explain what each one does.

• LOAD_FAST: Loads a variable to the top of the stack (Top Of Stack).
• BINARY_ADD: Add the two values ​​at the top of the stack and return them to the top of the stack.
• RETURN_VALUE: Returns the value that is in TOS.

Well, now that we've explain the opcodes, we can get an idea of ​​how our code works internally, but there are still doubts, annoying but necessary doubts, like these, "What is the 4 on the left side, the 4 that is at the beginning of the first line?", "What are the numbers to the left of the OPCODES? "Why does a 0 appear to the right of LOAD_FAST?, And the 1?", "We wouldn't want to loadx and y to add them instead of 0 and 1?".

Well, I will answer in order.

• The 4 is the line where the disassembled bytecode begins.
• These numbers represent the offset of the bytes.
• The 0 and the 1 correspond to an index, since the variables of the code are stored in a list (array), the 0 and 1 represent the index, however, the module dis tells us which variable is to the right of this number (hence the 0 (x) and 1 (y)). *

How do we re-create our function to make it bytecode?

Well, the first thing we do is import CodeType andFunctionType (To pass it to function) from the [types] module (https://docs.python.org/3/library/types.html#module-types)

import dis
from types import CodeType, FunctionType

def sum (x, y):
    return x + y

After this, we are going to create our object code
python

import dis
from types import CodeType, FunctionType

def sum (x, y):
    return x + y

# This will be explained later, these are flags
CO_OPTIMIZED = 0x0001
CO_NEWLOCALS = 0x0002
CO_NOFREE = 0x0002

my_code = CodeType (
    2, #argcount
    0, #kwonlyargcount
    2, #nlocals
    2, #stacksize
    (CO_OPTIMIZED | CO_NEWLOCALS | CO_NOFREE), #flags
    bytes ([124, 0, 124, 1, 23, 0, 83, 0]), #codestring
    (0,), #constants
    (), # names of constants or global (names)
    ('x', 'y',), #variable names (varnames)
    'blog_no_name', #filename
    'crafted_sum', #name (code name / function)
     9, #Firstlineno (First line where this code appears)
     b'', #lnotab
     (), #freevars
     (), # freecellvars
     )

_sum = FunctionType (my_code, {})
result = _sum (213,3)
print (result)

# Expected output
# 216

Well well ... Many new things appear, we will explain these arguments right now.

CodeType: argcount, kwonlyargcount, nlocals, stacksize, flags, codestring, constants, names, varnames, filename, name, firstlineno, lnotab, freevars, freecellvars

One last two things to note before moving on to FunctionType, the first is that the 0s that follow the opcodes * eg [124, 0, ...] * are the argument, and the second is that each bytecode can vary from version to version, to know or orient yourself about the codestring, you can use the following snippet

def sum (x, y):
    return x + y
sum.__ code __.co_code

# Expected output in Python 3.7.9 (The version I use)
# b '|\x00|\x01\x17\x00S\x00'
# The bytes are interpreted as characters, probably to make it more readable. (If we put chr (124) it will print the character |)

"Crafting" the function

We are going to use FunctionType now.
FunctionType: code, globals, name, argdefs, closure

Well, once this is clear, now we would only have to add a FunctionType with our object code (my_code) and call it.

import dis
from types import CodeType, FunctionType

def sum (x, y):
    return x + y

After this, we are going to create our object code

import dis
from types import CodeType, FunctionType

def sum (x, y):
    return x + y

# This I will explain later, they are flags
CO_OPTIMIZED = 0x0001
CO_NEWLOCALS = 0x0002
CO_NOFREE = 0x0002

my_code = CodeType (
    2, #argcount
    0, #kwonlyargcount
    2, #nlocals
    2, #stacksize
    (CO_OPTIMIZED | CO_NEWLOCALS | CO_NOFREE), #flags
    bytes ([124, 0, 124, 1, 23, 0, 83, 0]), #codestring
    (0,), #constants
    (), # names of constants or global (names)
    ('x', 'y',), #variable names (varnames)
    'blog_no_name', #filename
    'crafted_sum', #name (code / function name)
    9, #Firstlineno (First line where this code appears)
    b '', #lnotab
    (), #freevars
    (), # freecellvars
    )

_sum = FunctionType (my_code, {})
result = _sum (213,3)
print (result)

# Expected output
# 216

This is all for now, later I will upload another tutorial explaining the closures

Sources

• Module dis
• How does Python work?
• Flags and other useful data
• Freevar, Cellvars and local variables
• Scott Sanderson and Joe Jevnik "Playing with Python bytecode"