Optimised code is essential because it directly impacts the efficiency, performance, and scalability of software. Well-written code runs faster, co...
For further actions, you may consider blocking this person and/or reporting abuse
While it's certainly good practice to keep some things "private" that are not intended to be used by external callers, this does nothing for speed or efficiency. Prefixing with an underscore as in the example is really just a convention anyway and doesn't prevent calling the function.
Thanks for the feedback.
Actually, optimized code is harder to mantain. That's why it is much better to use an appropriate algorithm than use, say, arrays instead of lists. Moreover, the use of these techniques before improving the algorithm in use would be an example of premature optimization.
consider lazy evaluation (
map,filter,reduce, and the entireitertoolsmodule). I have often seen folks comprehend a large list into memory for a loop only to break the loop early meaning all members of the comprehended list with indices higher than the member where the list was broken were wasted CPU time and memory. I have also seen folks comprehend an original dataset into a second, separate variable (possibly for clarity's sake?) to use once and then never again but the GC never collects it because a variable still refers to it. It would have been better to either assign the second variable as a map of the original data variable or simply iterate directly over the comprehended list without assigning a variable name to it so that the GC cleans it up immediately after the loop is complete rather than having it stick around in memory for no reason.tl;dr there is a time and place for mappings vs comprehensions. Both have pros and cons. Please consider why and how you are using comprehended variables!
Yes that’s right, thanks for the insight.
loops are the most unoptimized when it comes to python. many devs overlook this.
great read james!
I can tell you that the loop problem is dev specific, not Python. You need to design good logic or program flow to have optimized loops. I've seen 2-3 loops in places where changing the flow will leave just 1 in place
Agreed!
@roshan_khan_28 yes that’s right, thanks for the feedback.
Interesting, and similar advice was given over 20 years for other languages. In any case, keep in mind Knuth's advice:
“The real problem is that programmers have spent far too much time worrying about efficiency in the wrong places and at the wrong times; premature optimization is the root of all evil (or at least most of it) in programming.”
stackify.com/premature-optimizatio...
Section 10 Optimizing Loops, you introduce numpy without a single mention of it. I find that a tad surprising and would fix it. Numpy is indeed great for such operations as you're illustrated but it is not a small library, nor the only way to do that (jax is another example), what you are introducing here is literally a suggestion to use numpy for efficient processing of large data sets which is great advice, but that is how it should be introduce IMHO - it's not a Python thing so much as a numpy thing.
Thanks for the notice, I have added some explanation about numpy.
While short logs could seems as good idea from the pov of an SRE it would be better to use a logger with configurable format so you can not only have control over format but also over log level
This increases application observability, log readability and shorten debug times!
@juanitomint Thank you for the addition.
Thanks a lot 🪴
I guess I would have to bookmark this now because some of my code are so memory tasking and would immensely benefit from these tips.
Good Article! Insightful and loved it!
Thanks @ddebajyati, I’m glad you enjoyed it.
I think the memory mapped file does not speed up the code. It just let you treat a file as a bytearray in memory . It still needs disk I/O to read the data in a file.
Yeah maybe it's a good strategy for allocate that time at startup do frequently accessed files 🤔
is the python going to be the future ???
OR
google doesn't need the python developers???
Nice!
Great list, James! Avoiding unnecessary work is another great one ;)
Very helpful
Very good post, thanks.
Thanks @sc0v0ne, glad you liked it.
This is great! I especially find useful your points with regards to memory and loops. Thank you....
Best, Anna
I don't think the
mmapexample is a great one - wheremmapreally shines is when treating the file as a largebytesobject (think aarrayofstruct.packs) you can then access by index.