Python has become one of the most popular programming languages out there, particularly for beginners and those new to the hacker/maker world. Unfortunately, while it’s easy to get something up and running in Python, it’s performance compared to other languages is generally lacking. Often, when starting out, we’re just happy to have our code run successfully. Eventually, though, performance always becomes a priority. When that happens for you, you might like to check out the nifty tips from [Evgenia Verbina] on how to make your Python code faster.
Many of the tricks are simple common sense. For example, it’s useful to avoid creating duplicates of large objects in memory, so altering an object instead of copying it can save a lot of processing time. Another easy win is using the Python math module instead of using the exponent (**) operator since math calls some C code that runs super fast. Others may be unfamiliar to new coders—like the benefits of using sets instead of lists for faster lookups, particularly when it comes to working with larger datasets. These sorts of efficiency gains might be merely useful, or they might be a critical part of making sure your project is actually practical and fit for purpose.
It’s worth looking over the whole list, even if you’re an intermediate coder. You might find some easy wins that drastically improve your code for minimal effort. We’ve explored similar tricks for speeding up code on embedded platforms like Arduino, too. If you’ve got your own nifty Python speed hacks, don’t hesitate to notify the tipsline!
The best trick to make your Python code thousands times faster is to write it in C/C++, Pascal, Rust or any other compiled language…
Aw you beat me too it. This is the answer though.
Python is a reasonable tool and all but sometimes I do wonder if I will ever reach for it outside of a scripting context.
It really depends most of the time the code will be run once or twice. In which case dev time costs from writing C/Rust will massively outweigh the gain.
So long as the heavy lifting is in some library like Numpy.
If it’ll be running repeatedly or in an ongoing fashion then it’s deffo worth the devtime costs.
I don’t like the idea of language that makes white spaces part of syntax just because the creators never heard about development environments that include automated indentation and collapsing functions as part of GUI.
I used it for writing simple scripts that performed iterative calculations, and only because I didn’t want to use scripting in a spreadsheet program. But it’s used for just about everything. Imagine using 80’s BASIC for writing a modern CAD software or something…
That will make your program run faster, but not your python code as you didn’t modify your python code or the way it is interpreted.
Python has many advantages over compiled languages. You can test things in real time using interactive terminal and Jupyter notebooks. There are many libraries available that are easy to integrate. Python has many optimized libraries. So even if the interpreted code is slow it can delegate performance critical parts to those libraries.
Porting python code to C/C++ can take a few hours to a few months.
I use both C/C++ and python. C/C++ for embedded devices. Python for PC tools (code analysis, processing log files, generating PDF’s and XLSX documents, regression analysis, code generation, web scraping, build scripts, etc.).
I have used micropython on embedded devices, but I’ve never used that in any serious project.
Every advantage you just wrote also applies to Julia, which is JIT-compiled and much faster than Python. Give it a try some time, it’s a really nice language. :)
I am a beginner in Julia, still, and like it more than Python (in which i’m a beginner too and tend to stay that way for multitude of reasons, not the least of the which is the whitespace significance), but i do follow the Julia discussion board and there are from time to time people who come from Python and redo their code in Julia and wonder why it’s not significally faster as promised. Mostly it’s because they do not use the strenghts of Julia.
In Julia you also have to follow certain rules that make your code faster (avoid dynamic allocations, use functions, do not use global variables etc). Just writing it similary to Python might not do much speedwise. But with Julia you have the chance to write normal Julia code without resorting to things like Numby et co, which to my understanding, is somewhat to like using yet another language.
Personally Python speed is all i’ll ever need. I do not have the amount of data or other speed requirements, that i couldn’t do with Python, but i just dislike it and Julia interests me.
That whole thing about how python can be high performance because the libraries are written in c++ is pure BS. I’ve experienced it first hand. Yes it can be fast, but if any real python code touches your data the whole thing grids to a halt. 1. It takes an insane amount of care and experimentation to avoid this, and 2. If you can’t use any python operators or data-structures, what is the point of using python at all.
C?! If I’m using Python it’s because I’m lazy – I don’t want to be bothered with memory management. Go would be a suitable substitute.
This is the answer. I built a whole computer vision app with python and later rewrote it in C++ for performance reasons. Best decision I ever made. Trying to write high performance code in python is an absolute nightmare.
Or use pypy to compile the Python code.
The advice in the article seems useless: either the savings are minimal, or they are generic programming concepts like hash lookup being faster than linear lookup.
Nah, dude. C/C++ is still going to be several times faster than anything pypy can produce.
Sure, and if you are skilled enough, hand-optimized assembler is even faster. Pypy is however a viable way to remove a lot of the Python overhead with minimal effort.
I checked out the list, and two things are clear to me:
It’s a JetBrains ad, hawking their AI “tools”
Calling time.sleep one time instead of one thousand makes the code sleep one thousandth as much
For a real Python performance tip: Every time you use the . operator to look up a member of an object, that’s a hash table lookup. Make local references to repeatedly-accessed members before loops, don’t do the hash table lookup more than you need to. This can make a big difference, and is used all over the place in the standard library.
object.attr is a hash lookup? Now this is hacking … rewriting code to meticulously avoid a core feature.
It’s not avoiding a core feature, you just do object_attr = object.attr before the loop.
Exactly. And that is an optimizing tip for any decent language.
it feels like it, attribute lookups are basically free in other OOP languages
Most of these results are marginal and since time.time()-start was used to measure run times it is highly inaccurate. Use timeit to get results that are believable.
all the operators “call some C code”
presumably you meant using math.pow instead of ** since “the math module” can’t be used in the same way as an operator
math.pow and ** do NOT do the same thing and are not universally interchangeable. “Unlike the built-in ** operator, math.pow() converts both its arguments to type float. Use ** or the built-in pow() function for computing exact integer powers.”
I don’t have much antipathy towards python becasue i never have to use it. i don’t hate it the way i hate C++, for example :)
But i have an unusually low amount of respect for it. It’s unusually slow even for an interpretted language. It’s also unusually unreadable. I haven’t yet run into a bit of python code, no matter how simple, that doesn’t start out with a lecture on how unsuitable its object model is for the way it’s actually used. Especially for a ‘beginner’ language, i can’t believe people jump through the hoops of its object model just for hello worlds! But mostly, it’s just dog slow because it’s implementation is slow and then the idioms that people actually use in it are dog slow idioms in any language.
I do generally believe you can improve performance within any language by actually learning how to use it. For example, java is faster if you use byte[] or char[] for string parsing instead of java.lang.String, right? Languages with garbage collection are faster if you give just a little mind to when you are doing your allocations. People haev made a lot of noise about the fact that clean languages that lean towards functional programming, like Scheme and OCaml, can be just as fast as any other language. To my surprise, the straightforward functional way to use OCaml was very fast (faster than C, because there was a mistake in a part of my C code that was completely elided by OCaml) in the one experiment i did.
And that’s where Python fails…anyone who knows better isn’t using Python. Python is only used by people who don’t know any better, who get suckered in by the slowest idioms. By contrast to Perl, which is nearly as slow but much more expressive and much more suited to one-off hacks, i see larger programs written in Python. To where you ‘apt install xxx’ and then use some unusably slow piece of garbage and discover it was written in Python. In Perl, everything i write is so simple that i don’t mind that it’s slow.
You can’t improve Python’s performance by improving the way you use it because the culture of how it’s used is its problem. And that’s why “just use a different language” is the effective advice, because anyone who isn’t willing to use a different language won’t be willing to confront that culture of using it for its weaknesses either.
As someone who does almost zero programming and has been meaning to get into it for like 30 years this was a really insightful comment and probably saved me a lot of time and frustration. Thanks.
Greg A is giving a pretty biased take. Not saying he’s wrong, but not all of us have a whole lifetime available to learn the intricacies of C++
No one in their right mind is going to suggest that python is fast at execution. But execution speed is not a primary design consideration for most hobbyist/ learning projects, and development speed is usually of primary concern (where python shines, and C++ really does not, even if you have good skills).
Nevermind the fact that c++ is stuffed to the gills with footguns and ways to write horribly memory-insecure code if you didn’t have a really good education or are rushing. Don’t listen to me, there are huge stakeholders in tech who have realized this is a massive problem:
https://www.memorysafety.org/docs/memory-safety/
TL;DR: 60-90% of ALL bugs are estimated to be memory safety related (ios/mac os, Microsoft, android). Modern languages like python/rust/go etc etc do not have this issue.
Python is a lovely (if seriously flawed) language, particularly for getting started.
If you never feel held back by python then cool, you’ve got a language that will serve you well. Otherwise, grab a ‘real’ language like Go, Rust, or C++ – at that point you should be able to decide what’s best for you
You are someone that misses the point of python. I use it often.
I also have C/C++ pulls accepted into numerous major radio projects and a radio tinygo lib 90% done.
Python is great if you don’t need speed. The “python isn’t readable,” comment is just crazy. Python is easily readable.
If you don’t like Python, thats fine, but its absolutely useful.
I think it’s something more than just missing the point but that’s their problem, they don’t have to make everyone’s problem or scare off beginners.
“anyone who knows better isn’t using Python.” Nonsense. Python is a fine solution for many problems. Yes, it is slow. But in many cases, it doesn’t matter. As long as the solution is fast enough, I don’t need to invest the considerable extra effort to use C++. Processor cycles are cheap, programmer cycles are expensive.
Totally agree. Our company uses Python for lots of tasks. Slow is relative. If you task tasks a second to run in Python, and takes 1/10 in C. …. So what? Fast ‘enough’ is the key…. I use it a lot at home too. Again it is fast enough for (throwing out a number) 90% of all the tasks that we do. The other beauty is the source is the runtime. So fast turn-around on changes. And Engineers to Software people can understand and make changes to the code. Fast development also. Wins all around.
A speed up tip. At work I had a situation where I had a list of files that I needed to sort by file modified time. My original solution was to to ‘glob’ it (get the files), then go get the modified time on each file, and finally sort it. With a lot of files this was ‘slow’ . So found a solution by using ‘ os.scandir() ‘. This is much much faster as the file info is returned with the file name. Just an FYI….
Should note the use case. The list of files are shown in a QT dialog box for our dispatchers to pick from. So ‘time’ opening the dialog is quite noticeable when operator presses button and no dialog for 5-10 seconds…. With the new implementation, click button, and dialog presents with file list in less than a second. Big change.
This !
Forget the seconds, if the job runs overnight.
Forget them when the script saves you hours.
Be thankful for a python-caused coffeebreak, b/c in it you can rethink if the right business problem is solved or the “I did not mean that” variant.
And DO remember the time gained 6 months into the future, when you or (shudder..) your colleague needs to actually understand and adapt that code.
Want fast code changes ? documentation and readable code, babe !
“Want fast code changes ? documentation and readable code, babe !” . Yep! And, as I said, the source is the runtime. You don’t have to login into your/a workstation, find the source, make changes, compile, copy executable to the machines (in our case VMs) where you are using it. Test, rinse, repeat. It is nice that a person can just open the file, make a change (and document change in the file) and test/run. Save’s a lot of time if called in the middle of the night, or if the normal maintainers are on vacation, etc. No compilers to keep track of. There are a lot of plusses when Python is used in the appropriate situations.
The comments section looks like I expected when I read the title of the article.
“I would like my bicycle to go faster.”
“Get a car.”
That’s kind of good advice though.
The cost is time: time to learn the language, time to develop, build and distribute code, time to debug and iterate
Bicycle to car is a pretty good comparison in cost to the time spent on c++ or python.
Is your code going to be ‘going down the street’? If so, spending that much on learning and development is crazy, you could use that time for so many better things.
But if you are going a long distance and speed is super important, then a car is a great choice, but it costs WAY more.
Just pick the right language for the job. When you are tasked with a project, it is part of the evaluation of where the project is going to be used and by whom, time constraints, resources, knowledge base, budget, etc. Language will fall out of the analysis.
With a car analogy …. Driving a Ferrari in town with a 25mph speed limit, is a big waste of money. But if you are using it out at the track for racing… Maybe not.
Enzo hated his customers.
Because they drive Ferraris at 25mph, to impress other idiots with their bad taste in cars for cruising.
That said, Enzo also said a lot of stupid things.
e.g. ‘I like American cars, I just don’t understand why they put truck engines in them.’
Ferrari street cars were slow AF, until they started putting truck engines in them…
See also:
‘Hot Rod Magazine’ racing a Pontiac GTO against a Ferrari GTO…at Monza, after the 1/4 mile.
They’ve been effectively mocking Italian trash for many decades.
The only thing Enzo really cared about was F1 racing.
The car company was a means to an end.
He hated his customers and wanted them to suffer and PAY.
If the ‘right tool for the job’ is python, refuse the job.
A shell script or a real language is always better.
High or low there is always a better option.
First Python is a ‘real’ language and very very useful. It is just interpreted. Like Java, Basic, Perl, and some other languages. And a compiled language is not always better. For one of my jobs, I spent a couple of years rewriting all the batch, VB, Excel VB, Access VB applications in Python for a particular process the company had. Maintenance went to close to zero, and call-outs are a thing of the past for that system. No, I am a believer in the easy to use, very useful Python language…. You can’t go wrong with it … unless misusing it. Note I used Pascal, C/C++ and assembly for most of my career with some Fortran support for scientists… I sure wish I had a tool like Python to program the ‘boring stuff’ back then.
I really dislike shell (such as Bash) programming except for very short repetitive tasks (like just calling rsync to backup a partition). Much harder to read than Python after the fact if you start using the control structures.
Real languages don’t break backward compatibility.
You don’t have to run real languages on specific versions of interpreters.
Python is not a language you use to get job done quick. Here is one example I stumbled upon recently while writing Sigrok decoders (limited to python 3.4) – Enums are EXTREMELY SLOW. It’s been “fixed” in later versions such that enum attribute lookup is “only” 3-6x slower (3.14) than string instead of 25-70x!
Python 3.4, big loop:
Direct string Access d[‘key’]: 3.37 seconds
Direct Enum Access d[enum.key]: 157.99 seconds
SimpleNamespace class.key access is almost same speed at string lookup tho, so at least not all is lost.