Python has become one of the most popular programming languages out there, particularly for beginners and those new to the hacker/maker world. Unfortunately, while it’s easy to get something up and running in Python, it’s performance compared to other languages is generally lacking. Often, when starting out, we’re just happy to have our code run successfully. Eventually, though, performance always becomes a priority. When that happens for you, you might like to check out the nifty tips from [Evgenia Verbina] on how to make your Python code faster.
Many of the tricks are simple common sense. For example, it’s useful to avoid creating duplicates of large objects in memory, so altering an object instead of copying it can save a lot of processing time. Another easy win is using the Python math module instead of using the exponent (**) operator since math calls some C code that runs super fast. Others may be unfamiliar to new coders—like the benefits of using sets instead of lists for faster lookups, particularly when it comes to working with larger datasets. These sorts of efficiency gains might be merely useful, or they might be a critical part of making sure your project is actually practical and fit for purpose.
It’s worth looking over the whole list, even if you’re an intermediate coder. You might find some easy wins that drastically improve your code for minimal effort. We’ve explored similar tricks for speeding up code on embedded platforms like Arduino, too. If you’ve got your own nifty Python speed hacks, don’t hesitate to notify the tipsline!
The best trick to make your Python code thousands times faster is to write it in C/C++, Pascal, Rust or any other compiled language…
Aw you beat me too it. This is the answer though.
Python is a reasonable tool and all but sometimes I do wonder if I will ever reach for it outside of a scripting context.
It really depends most of the time the code will be run once or twice. In which case dev time costs from writing C/Rust will massively outweigh the gain.
So long as the heavy lifting is in some library like Numpy.
If it’ll be running repeatedly or in an ongoing fashion then it’s deffo worth the devtime costs.
I don’t like the idea of language that makes white spaces part of syntax just because the creators never heard about development environments that include automated indentation and collapsing functions as part of GUI.
I used it for writing simple scripts that performed iterative calculations, and only because I didn’t want to use scripting in a spreadsheet program. But it’s used for just about everything. Imagine using 80’s BASIC for writing a modern CAD software or something…
That was always a pretty worthless reason to dislike python, there were tab versus space and 4 space versus 8 space code style wars before python was a twinkle in Guido’s eye. Finding a UI or a plugin for a UI that supports python syntax is not hard.
I disagree. You indent your code anyway, so brackets are redundant.
It makes code more consistent and therefore makes merging easier (fewer style changes so fewer line diffs).
That will make your program run faster, but not your python code as you didn’t modify your python code or the way it is interpreted.
Python has many advantages over compiled languages. You can test things in real time using interactive terminal and Jupyter notebooks. There are many libraries available that are easy to integrate. Python has many optimized libraries. So even if the interpreted code is slow it can delegate performance critical parts to those libraries.
Porting python code to C/C++ can take a few hours to a few months.
I use both C/C++ and python. C/C++ for embedded devices. Python for PC tools (code analysis, processing log files, generating PDF’s and XLSX documents, regression analysis, code generation, web scraping, build scripts, etc.).
I have used micropython on embedded devices, but I’ve never used that in any serious project.
Every advantage you just wrote also applies to Julia, which is JIT-compiled and much faster than Python. Give it a try some time, it’s a really nice language. :)
I am a beginner in Julia, still, and like it more than Python (in which i’m a beginner too and tend to stay that way for multitude of reasons, not the least of the which is the whitespace significance), but i do follow the Julia discussion board and there are from time to time people who come from Python and redo their code in Julia and wonder why it’s not significally faster as promised. Mostly it’s because they do not use the strenghts of Julia.
In Julia you also have to follow certain rules that make your code faster (avoid dynamic allocations, use functions, do not use global variables etc). Just writing it similary to Python might not do much speedwise. But with Julia you have the chance to write normal Julia code without resorting to things like Numby et co, which to my understanding, is somewhat to like using yet another language.
Personally Python speed is all i’ll ever need. I do not have the amount of data or other speed requirements, that i couldn’t do with Python, but i just dislike it and Julia interests me.
That whole thing about how python can be high performance because the libraries are written in c++ is pure BS. I’ve experienced it first hand. Yes it can be fast, but if any real python code touches your data the whole thing grids to a halt. 1. It takes an insane amount of care and experimentation to avoid this, and 2. If you can’t use any python operators or data-structures, what is the point of using python at all.
C?! If I’m using Python it’s because I’m lazy – I don’t want to be bothered with memory management. Go would be a suitable substitute.
This is the answer. I built a whole computer vision app with python and later rewrote it in C++ for performance reasons. Best decision I ever made. Trying to write high performance code in python is an absolute nightmare.
Or use pypy to compile the Python code.
The advice in the article seems useless: either the savings are minimal, or they are generic programming concepts like hash lookup being faster than linear lookup.
Nah, dude. C/C++ is still going to be several times faster than anything pypy can produce.
Sure, and if you are skilled enough, hand-optimized assembler is even faster. Pypy is however a viable way to remove a lot of the Python overhead with minimal effort.
If C statements for bit manipulation, GPIO, and DMA (or Device Tree) were not a mass of characters from the shifted top row of my KB, maybe it would be OK. MicroPython has several levels of optimization from using locals declared outside a loop because access on the stack is fast, to including assembly code within your Python. Speed versus textbook Python can be 80 to 1.
Yes, Python can use some of the great C and Fortran libraries. But so does C/C++. C is nearly unusable without the “pre-processor” and standard library. In the Python world, the IDE for R&D is Jupyterlab, which is pretty dang cool.
While technically true in some cases, I think people tend to strongly overestimate runtime differences between Python and compiled languages in most of the practical applications.
Working almost 10 years in Python, Rust and C++ for engineering applications, I’ve seen “let’s rewrite this in Rust/C++” scenarios dozens of times and I am yet to see one that actually resulted in a massive improvement.
For almost all serious math/computer vision/ML calculations Python utilizes blazing-fast, beautifully optimized compiled libraries such as numpy, scipy, opencv or torch anyway. For other low-level custom functions, you just throw numba decorator for JiT compilation, and often get performance comparable with Rust/C++. Higher-level configuration/logic/glue code rarely has significant impact on the performance anyway, and is very easy and fast to develop, test and iterate on.
Personally I’m more of a Rust enthusiast, but days where Python was just some slow scripting language for hobby projects are long gone in engineering, and it’s hard to find other language that would offer such vast choice of robust, mature and fast libraries, which is often crucial.
This.
You don’t need to write the fast libraries yourself. They’re already there, and wrapping them in your Python code logic / intent doesn’t cost you much performance, but gives you a lot in terms of readability, interfacing with data sources, etc. It really can be the best of both worlds: high-level coding with low-level speed.
I checked out the list, and two things are clear to me:
It’s a JetBrains ad, hawking their AI “tools”
Calling time.sleep one time instead of one thousand makes the code sleep one thousandth as much
For a real Python performance tip: Every time you use the . operator to look up a member of an object, that’s a hash table lookup. Make local references to repeatedly-accessed members before loops, don’t do the hash table lookup more than you need to. This can make a big difference, and is used all over the place in the standard library.
object.attr is a hash lookup? Now this is hacking … rewriting code to meticulously avoid a core feature.
It’s not avoiding a core feature, you just do object_attr = object.attr before the loop.
Exactly. And that is an optimizing tip for any decent language.
it feels like it, attribute lookups are basically free in other OOP languages
Most of these results are marginal and since time.time()-start was used to measure run times it is highly inaccurate. Use timeit to get results that are believable.
all the operators “call some C code”
presumably you meant using math.pow instead of ** since “the math module” can’t be used in the same way as an operator
math.pow and ** do NOT do the same thing and are not universally interchangeable. “Unlike the built-in ** operator, math.pow() converts both its arguments to type float. Use ** or the built-in pow() function for computing exact integer powers.”
I don’t have much antipathy towards python becasue i never have to use it. i don’t hate it the way i hate C++, for example :)
But i have an unusually low amount of respect for it. It’s unusually slow even for an interpretted language. It’s also unusually unreadable. I haven’t yet run into a bit of python code, no matter how simple, that doesn’t start out with a lecture on how unsuitable its object model is for the way it’s actually used. Especially for a ‘beginner’ language, i can’t believe people jump through the hoops of its object model just for hello worlds! But mostly, it’s just dog slow because it’s implementation is slow and then the idioms that people actually use in it are dog slow idioms in any language.
I do generally believe you can improve performance within any language by actually learning how to use it. For example, java is faster if you use byte[] or char[] for string parsing instead of java.lang.String, right? Languages with garbage collection are faster if you give just a little mind to when you are doing your allocations. People haev made a lot of noise about the fact that clean languages that lean towards functional programming, like Scheme and OCaml, can be just as fast as any other language. To my surprise, the straightforward functional way to use OCaml was very fast (faster than C, because there was a mistake in a part of my C code that was completely elided by OCaml) in the one experiment i did.
And that’s where Python fails…anyone who knows better isn’t using Python. Python is only used by people who don’t know any better, who get suckered in by the slowest idioms. By contrast to Perl, which is nearly as slow but much more expressive and much more suited to one-off hacks, i see larger programs written in Python. To where you ‘apt install xxx’ and then use some unusably slow piece of garbage and discover it was written in Python. In Perl, everything i write is so simple that i don’t mind that it’s slow.
You can’t improve Python’s performance by improving the way you use it because the culture of how it’s used is its problem. And that’s why “just use a different language” is the effective advice, because anyone who isn’t willing to use a different language won’t be willing to confront that culture of using it for its weaknesses either.
As someone who does almost zero programming and has been meaning to get into it for like 30 years this was a really insightful comment and probably saved me a lot of time and frustration. Thanks.
Greg A is giving a pretty biased take. Not saying he’s wrong, but not all of us have a whole lifetime available to learn the intricacies of C++
No one in their right mind is going to suggest that python is fast at execution. But execution speed is not a primary design consideration for most hobbyist/ learning projects, and development speed is usually of primary concern (where python shines, and C++ really does not, even if you have good skills).
Nevermind the fact that c++ is stuffed to the gills with footguns and ways to write horribly memory-insecure code if you didn’t have a really good education or are rushing. Don’t listen to me, there are huge stakeholders in tech who have realized this is a massive problem:
https://www.memorysafety.org/docs/memory-safety/
TL;DR: 60-90% of ALL bugs are estimated to be memory safety related (ios/mac os, Microsoft, android). Modern languages like python/rust/go etc etc do not have this issue.
Python is a lovely (if seriously flawed) language, particularly for getting started.
If you never feel held back by python then cool, you’ve got a language that will serve you well. Otherwise, grab a ‘real’ language like Go, Rust, or C++ – at that point you should be able to decide what’s best for you
100% – i am biased as heck.
But i think when you say “particularly for getting started” you allude to my point: getting started is the thing you do before you are going. But Python users never go. If your second step isn’t abandoning Python, then you are “getting started” all your life. Other faith traditions don’t have this problem, at least not in this way.
I started with BASIC and it was a great way to get started but even before i found out other languages exist, i was dreaming of other languages instead of building giant unusably slow end-user programs like pyslsk :)
On the contrary, I think it’s one of the worst advises that a person who wants to try programming may receive.
Situations where python’s performance really matters are quite rare, specific, and limited to only certain industries/applications. In most cases, you either don’t need amazing performance, or get the performance anyway, since for most of the compute-intense stuff Python utilizes fast compiled libraries anyway.
What you get from Python in return is amazing amount of materials, mature and well-documented libraries, and simplicity rarely matched by other languages.
You want to plot some graphs in python? No problem, you can find a great standard way to do this in 2 minutes googling, and it runs out of the box.
You want to do this in Rust? Well, there are 15 different libraries, each of them have some caveats, none of them really have great documentation, and 2 hours later you find yourself reading obscure comparison articles, documentations and tutorials trying to figure out what will really support all your needs.
You want to do this in C++? Well, let’s start with Makefiles, or better, CMake, LD_LIBRARY_PATH environmental variables, dynamic/static linking caveats and resolving dependency hells before we even start to look for a proper library….
You are someone that misses the point of python. I use it often.
I also have C/C++ pulls accepted into numerous major radio projects and a radio tinygo lib 90% done.
Python is great if you don’t need speed. The “python isn’t readable,” comment is just crazy. Python is easily readable.
If you don’t like Python, thats fine, but its absolutely useful.
I think it’s something more than just missing the point but that’s their problem, they don’t have to make everyone’s problem or scare off beginners.
The thing is, if you want to make a one-off hack, like to tease relational data out of a text file, perl is better than python. I agree with the idea that every language doesn’t have to be C. But i don’t know what Python is actually good at, i just know that an awful lot of people use it for things outside of whatever realm that is.
I figure Python is good for the things Perl is good for if you don’t know perl. But you should learn Perl. :)
I do C all day long, and have done enough C++, PHP and JS to make things happen, and every time I do stuff with python (which I do use a lot) I am left wondering what possible reason there is for abandoning perfectly sensible things like boring old arrays and instead using list/dict/tuple none of which ever seem to be the right answer for anything.
My two possible conclusions are:
1. I’m clearly not smart enough to understand the genius at play here
2. Someone was trying to be clever and different and has made a rod for their own back, and for all our backs, forever more.
At least with their seemingly unnecessary reinvention of the print statement they have now rowed back to pretty much the C version with print(f “…”), almost as if there was nothing wrong with the old way in the first place…
“anyone who knows better isn’t using Python.” Nonsense. Python is a fine solution for many problems. Yes, it is slow. But in many cases, it doesn’t matter. As long as the solution is fast enough, I don’t need to invest the considerable extra effort to use C++. Processor cycles are cheap, programmer cycles are expensive.
Totally agree. Our company uses Python for lots of tasks. Slow is relative. If you task tasks a second to run in Python, and takes 1/10 in C. …. So what? Fast ‘enough’ is the key…. I use it a lot at home too. Again it is fast enough for (throwing out a number) 90% of all the tasks that we do. The other beauty is the source is the runtime. So fast turn-around on changes. And Engineers to Software people can understand and make changes to the code. Fast development also. Wins all around.
A speed up tip. At work I had a situation where I had a list of files that I needed to sort by file modified time. My original solution was to to ‘glob’ it (get the files), then go get the modified time on each file, and finally sort it. With a lot of files this was ‘slow’ . So found a solution by using ‘ os.scandir() ‘. This is much much faster as the file info is returned with the file name. Just an FYI….
Should note the use case. The list of files are shown in a QT dialog box for our dispatchers to pick from. So ‘time’ opening the dialog is quite noticeable when operator presses button and no dialog for 5-10 seconds…. With the new implementation, click button, and dialog presents with file list in less than a second. Big change.
This !
Forget the seconds, if the job runs overnight.
Forget them when the script saves you hours.
Be thankful for a python-caused coffeebreak, b/c in it you can rethink if the right business problem is solved or the “I did not mean that” variant.
And DO remember the time gained 6 months into the future, when you or (shudder..) your colleague needs to actually understand and adapt that code.
Want fast code changes ? documentation and readable code, babe !
“Want fast code changes ? documentation and readable code, babe !” . Yep! And, as I said, the source is the runtime. You don’t have to login into your/a workstation, find the source, make changes, compile, copy executable to the machines (in our case VMs) where you are using it. Test, rinse, repeat. It is nice that a person can just open the file, make a change (and document change in the file) and test/run. Save’s a lot of time if called in the middle of the night, or if the normal maintainers are on vacation, etc. No compilers to keep track of. There are a lot of plusses when Python is used in the appropriate situations.
^ this, for me Python sits between a bash script and an actual C program when I need to do something that is too much for Bash but doesn’t quite warrant a full C executable.
It’s perfectly fine 99% of the time although it has its quirks, what language doesn’t?
The comments section looks like I expected when I read the title of the article.
“I would like my bicycle to go faster.”
“Get a car.”
That’s kind of good advice though.
The cost is time: time to learn the language, time to develop, build and distribute code, time to debug and iterate
Bicycle to car is a pretty good comparison in cost to the time spent on c++ or python.
Is your code going to be ‘going down the street’? If so, spending that much on learning and development is crazy, you could use that time for so many better things.
But if you are going a long distance and speed is super important, then a car is a great choice, but it costs WAY more.
Lots of anti-car people will tell you a car is not just a better bike.
I’m pro-car and pro-bike. I use C/C++ but also python.
A car is not a “better bike”.
Higher cost of ownership: Paid parking, road taxes, insurance, fuel.
For trips under 15 minutes a bike is usually faster.
I think a bike is like python and a car is like C/C++.
Just pick the right language for the job. When you are tasked with a project, it is part of the evaluation of where the project is going to be used and by whom, time constraints, resources, knowledge base, budget, etc. Language will fall out of the analysis.
With a car analogy …. Driving a Ferrari in town with a 25mph speed limit, is a big waste of money. But if you are using it out at the track for racing… Maybe not.
Enzo hated his customers.
Because they drive Ferraris at 25mph, to impress other idiots with their bad taste in cars for cruising.
That said, Enzo also said a lot of stupid things.
e.g. ‘I like American cars, I just don’t understand why they put truck engines in them.’
Ferrari street cars were slow AF, until they started putting truck engines in them…
See also:
‘Hot Rod Magazine’ racing a Pontiac GTO against a Ferrari GTO…at Monza, after the 1/4 mile.
They’ve been effectively mocking Italian trash for many decades.
The only thing Enzo really cared about was F1 racing.
The car company was a means to an end.
He hated his customers and wanted them to suffer and PAY.
If the ‘right tool for the job’ is python, refuse the job.
A shell script or a real language is always better.
High or low there is always a better option.
First Python is a ‘real’ language and very very useful. It is just interpreted. Like Java, Basic, Perl, and some other languages. And a compiled language is not always better. For one of my jobs, I spent a couple of years rewriting all the batch, VB, Excel VB, Access VB applications in Python for a particular process the company had. Maintenance went to close to zero, and call-outs are a thing of the past for that system. No, I am a believer in the easy to use, very useful Python language…. You can’t go wrong with it … unless misusing it. Note I used Pascal, C/C++ and assembly for most of my career with some Fortran support for scientists… I sure wish I had a tool like Python to program the ‘boring stuff’ back then.
I really dislike shell (such as Bash) programming except for very short repetitive tasks (like just calling rsync to backup a partition). Much harder to read than Python after the fact if you start using the control structures.
Real languages don’t break backward compatibility.
You don’t have to run real languages on specific versions of interpreters.
I’ve been running Python since 3.3 . I’ve only ran into one deprecated method since then. So you must be talking about the 2.xx to 3.xx series which I didn’t have to deal with. Running on 3.12 now and no issues with any home code, or work code.
BTW, I wrote a Rust application a few years ago and tried to recompile it. Had issues with new compiler. Rust is considered by some as ‘the’ language.
I’ve tried to compile some old ‘C’ code. Had to massage it for a new compiler will compile it.
Same with old Fortran code.
Basic had no standard. Commodore was different than other brands, etc…
So in my brain…. Python isn’t the only language that is changed over time…. And I think it was for the better.
Therefore by your definition: There is no ‘real’ language :) . Yet.
Post didn’t show up…. So here is a second one :) .
C, Fortran, Rust, etc. Have all broken backwards compatibility. Programs I tried to compile in C no longer work without changes using modern C compiler. A Rust program I had written a few years ago, needed changes too. Same with Fortran. Modern Fortran is hardly recognizable to Fortran of the 50s, 60s. So while Python 3 broke Python 2 it was for the better. In are company we started with Python 3.3 and haven’t had any problems through 3.12 . And I still say, all the languages above are ‘real’ languages :) .
I built a fairly large website in PHP 5. Guess what? A rare case in which I am the victim, the victim of no goshdarn backwards compatibility. Imagine my disappointment.
“A shell script or a real language is always better.”
A shell script? Really?
It’s not portable.
It’s slow.
Can’t debug it.
Limited features, can’t import libraries.
If we’d asked the customers what they wanted, they would have said “faster horses”
Henry Ford
Python is not a language you use to get job done quick. Here is one example I stumbled upon recently while writing Sigrok decoders (limited to python 3.4) – Enums are EXTREMELY SLOW. It’s been “fixed” in later versions such that enum attribute lookup is “only” 3-6x slower (3.14) than string instead of 25-70x!
Python 3.4, big loop:
Direct string Access d[‘key’]: 3.37 seconds
Direct Enum Access d[enum.key]: 157.99 seconds
SimpleNamespace class.key access is almost same speed at string lookup tho, so at least not all is lost.
Well I can certainly write my Python script faster and better in Perl. If I want it to run faster I write in C.
But can you read anything written in that write only language?
LUTs are all you need.
I second that.
Regularly run into things that could be computed with 8-bit processor using LUTs for definite/fixed/forced states and nearly zero error checking. Some of these could even be handled by a 4-bit microcontroller, they are THAT simple to start with (basic business logic that goes “do this when/if such and such conditions are met”).
will python or micro python ever support RISC-V?