Introduction
I love Python and I work with Odoo since 10 years.
In all these years I wrote a lot of code for Odoo, both modules and migrations scripts. Every time you need to move a lot of data from a source to a destination, time is an important point.
For example, I once wrote a script to copy more than 20 millions of products from an external software to Odoo database. Odoo executes a lot of checks every time you create a new record, so it's important to understand that every seconds is vital.
With so many records, every millisecond is a resource.
When we're writing the code, what we do is create the necessary data and run the tests. These data are often some products, partners, orders and invoices. When everything works you can create automatic tests.
I don't think every time we develop something we create a stress test for a large amount of data.
So, I always study and try new tips and tricks to increase speed of my code. A code is never perfect and it can be improved, always!
Using this basic script I want to show you some snippets to speed up code effortlessly.
import timeit
# Here an exemple of common but slow code
<CODE SLOW>
# Here an exemple of fast code
<CODE FAST>
setup = "records = env['res.partner'].search([], limit=1000)"
repetitions = (1, 10, 100)
globals_vals = {'env': env}
for repetition in repetitions:
slow_time = timeit.repeat(
code_slow,
setup=setup,
number=repetition,
globals={'env': env}
)
fast_time = timeit.repeat(
code_fast,
setup=setup,
number=repetition,
globals=globals_vals
)
print(f'=============== loops {repetition}')
print(slow_time)
print(fast_time)
set vs recordset
Working with Odoo means using a lots of recordsets. A recordset is an object with datas, functions and methods. It is a set of data, either from a database or calculated, on steroids. If you know other ORMs, you know what I'm talking about.
Sometimes, however, using the operations that the framework makes available is not convenient and it's more advantageous to use directly what python offers.
Code
code_slow = """
result = env['res.partner'].browse()
for record in records:
result |= record
"""
code_fast = """
result = set()
for record in records:
result.add(record.id)
env['res.partner'].browse(result)
"""
Results
=============== loops 1
[0.13306364299933193, 0.10838474300544476, 0.10796952999953646, 0.10946986099588685, 0.10623473400482908]
[0.0008965670058387332, 0.000925485001062043, 0.0009349410029244609, 0.0009159730034298263, 0.0009303710030508228]
=============== loops 10
[1.106517462998454, 1.1065070289987489, 1.1934943989981548, 1.1571569709994947, 1.1170753799960949]
[0.009077082999283448, 0.009004210005514324, 0.009018140997795854, 0.009551764000207186, 0.009227231996192131]
=============== loops 100
[11.516509660999873, 11.74818367199623, 11.85345141900325, 12.856167154001014, 13.63239839200105]
[0.09894232600345276, 0.09653759599314071, 0.09699099600402405, 0.10923265099700075, 0.10307022200140636]
write vs field settings
When you have a recordset, it's possible to change a value with a simple assignment. This is handy but every time you use it, Odoo prepare a query and change this value in cache. When you need to write more values in the same recordset or the same value on more recordsets, you must use write function.
Code
code_slow = """
for record in records:
record.ref = 'test_speed'
"""
code_fast = """
records.write({'ref': 'test_speed'})
"""
Results
=============== loops 1
[2.9913442569959443, 0.9559976780001307, 0.900353463999636, 0.8353457880002679, 1.0130259670040687]
[0.13637410899536917, 0.1379215750057483, 0.14272761800384615, 0.12462920800317079, 0.13321589499537367]
=============== loops 10
[10.402880988003744, 11.000012251999578, 11.697913458003313, 11.106917288998375, 11.169360947998939]
[2.0335192759957863, 2.4452823780011386, 2.2474471349996747, 2.5181185900000855, 2.776447412004927]
list comprehension vs mapped
Among the recordset methods, we have mapped(). It creates a set of values from a recordset. Values got from mapped() can be navigated through Odoo relations. In simple cases, mapped() is as convenient as list comprehension. In complex cases (for example, a navigation of relations of 3 levels) mapped() is more convenient to use.
It gives always better readability and this is good for maintainability.
Code
code_slow = """
states_set = set([r.state_id.id for r in records if r.state_id])
states = env['res.country.state'].browse(states_set)
"""
code_fast = """
states = records.mapped('state_id')
"""
Results
=============== loops 1
[0.31179145100031747, 0.004711384994152468, 0.004819055000552908, 0.004665278996981215, 0.0044039049971615896]
[0.0036316699988674372, 0.003607042999647092, 0.003627748999861069, 0.0036114609974902123, 0.003590111999073997]
=============== loops 10
[0.047915923001710325, 0.04522175599413458, 0.044370734998665284, 0.04409689400199568, 0.04411724000237882]
[0.035760099002800416, 0.0357158859987976, 0.03603034999832744, 0.036463224998442456, 0.03686660800303798]
=============== loops 100
[0.45273306199669605, 0.44231640599900857, 0.43908101499982877, 0.44457509399944684, 0.4423183210019488]
[0.36031395699683344, 0.3654527219987358, 0.38619487999676494, 0.3619795150007121, 0.3626547140011098]
loop vs filtered
Another recordset function is filtered(). You can use to filter recordset with a function or a value. Like mapped(), filtered() is as convenient as loop for simple cases but can be more convenient for complex case.
It gives always better readability and this is good for maintainability.
Code
code_slow = """
partners_start_with_a = set()
for record in records:
if record.name.lower().startswith('a'):
partners_start_with_a.add(record.id)
env['res.partner'].browse(partners_start_with_a)
"""
code_fast = """
records.filtered(lambda r: r.name.lower().startswith('a'))
"""
Results
=============== loops 1
[0.3316459549969295, 0.002682621001440566, 0.0026461579982424155, 0.002589060997706838, 0.002434752997942269]
[0.002538305998314172, 0.002414796006632969, 0.0024050630017882213, 0.0024614639987703413, 0.002459411000018008]
=============== loops 10
[0.02490358999784803, 0.027465703002235387, 0.02418504800152732, 0.023117996999644674, 0.023439353004505392]
[0.02508747999672778, 0.024697505999938585, 0.02418924599624006, 0.024244852000265382, 0.024165777998859994]
=============== loops 100
[0.23826791400642833, 0.22961928599397652, 0.23434631299460307, 0.2344872630055761, 0.2336325560027035]
[0.24336347499775002, 0.24536530500336085, 0.2900283189956099, 0.24434635799843818, 0.24427578799804905]
single browse vs multi browse
In some cases we have a list of ids and we want to get the relative recordsets. In Odoo we use browse(). browse() require an id or a list of ids and return a single recordset or a recordset of recordsets. If you need to iterate them you must loop directly on browse() return.
Code
code_slow = """
record_ids = records.ids
for record_id in record_ids:
record = env['res.partner'].browse(record_id)
record.name
"""
code_fast = """
record_ids = records.ids
for record in env['res.partner'].browse(record_ids):
record.name
"""
Results
=============== loops 1
[1.6325390929996502, 0.004575414001010358, 0.0046123150023049675, 0.005507702997419983, 0.004666433000238612]
[0.0024330550004378892, 0.002370230002270546, 0.0023654250035178848, 0.0024355540008400567, 0.0023724880011286587]
=============== loops 10
[0.049219961001654156, 0.04274192699813284, 0.0399551490045269, 0.039992154997889884, 0.04065825999714434]
[0.020162856999377254, 0.02014190399495419, 0.021769888000562787, 0.021047277994512115, 0.020693995997135062]
=============== loops 100
[0.41175051799655193, 0.4071878250033478, 0.4059581019973848, 0.40895022099721245, 0.40670431699982146]
[0.20786928899906343, 0.2082131749994005, 0.2086394680009107, 0.23048232900328003, 0.20772191799915163]
loop vs map
This is a python pure tip but I use it in my script especially with math calculations. map() call a function on an iterable object and return an object with results.
Code
code_slow = """
def increment_id(record_id):
return record_id + 1
for record_id in records.ids:
increment_id(record_id)
"""
code_fast = """
def increment_id(record_id):
return record_id + 1
map(increment_id, records.ids)
"""
Results
=============== loops 10
[0.0014327949975267984, 0.0014315969965537079, 0.0014401990047190338, 0.0015307379944715649, 0.0014989250048529357]
[0.0004444369988050312, 0.00044178399548400193, 0.000439752999227494, 0.0004845689982175827, 0.0004615380021277815]
=============== loops 100
[0.016216568001254927, 0.013593563002359588, 0.012723464002192486, 0.01124695000180509, 0.011159162997500971]
[0.003178275001118891, 0.0032086909995996393, 0.003190825998899527, 0.0031917309970594943, 0.003197609999915585]
=============== loops 1000
[0.11258809099672362, 0.11201509100646945, 0.11202597500232514, 0.11326640100014629, 0.11274601000332041]
[0.034097837000445, 0.03186929500225233, 0.031756801006849855, 0.031629047996830195, 0.0315622540001641]
Conclusions
I know you're thinking that 0.07 seconds on a line of code aren't so important but if you remember my introduction you can see that on 20 millions of repetations we can earn a lot of time!
See you soon. I'm going away... very fast!
Support
Top comments (4)
What would have been interesting in this topic is to actually see your approach and comparasion of batch creation of records > 1000 records , this seems like a real bottleneck at least for odoo10, interesting article <3
Hey, am starting out in odoo development and I want to pick your thoughts on something.
How would you track changes in ir.attachment from form view, Like the moment somebody uploads a file, a record is updated in the form. Thank you in advance
Hi Eric. I think that other sites can be used to ask questions. This isn't the right place. I don't want to generate noise under a post about other arguments. Please, consider to use official odoo help site. Thanks.
It'a a simple example to filter recordset. Of course, there is a better way to write this but my focus is on "Odoo code" and not "Pure Python Code" ;)
Thank you :)