Hacking The Python For Loop

In the early days of C, you’d occasionally see someone — probably a former Pascal programmer — write something like this:

#define BEGIN {
#define END }

This would usually initiate complaints about abusing the preprocessor and generally being anti-C. Surely no modern language would permit such things, right? Perhaps not. Consider [Tushar Sadhwani] who wanted to create a classic C-style for loop inside of Python. He did it, and the journey is perhaps more interesting than the result.

First, you can’t just transport straight C for loops into Python. There has to be some concession to Python syntax. The initial attempt was clever but not clever enough. However, the disassembly of the Python code was telling. The second attempt, however, was particularly interesting.

That attempt used an odd feature to examine the interpreter’s tree structure for the code and then modify it. This is sort of like a very painful C preprocessor but more powerful. That version works although it is pretty convoluted.

Ironically, [Tushar] then set up a third attempt after seeing code that tries to replace Python indentation with braces using a codec. In Python-speak, a codec lets you convert different text encodings. However, you can do other things than text encoding conversion. This is closest in spirit to the C preprocessor method. You can wade through the source code ahead of processing and make whichever changes you see fit.

Is any of this really useful? Probably not as it is. But you never know when you might need to do something exotic and one of these techniques could save the day. You probably couldn’t get away with some of this on MicroPython, of course. Your mileage may vary depending on where you find your Python running — like the Web.

33 thoughts on “Hacking The Python For Loop

  1. Good mustard, someone is wrong on the internet, and despite your self aggrandising title; it’s you…

    ‘whatever’ is used when there is no restriction on choice. e.g.
    The point was past his comprehension whatever we tried.

    There are a limited set of correct manipulations, therefore “which” is applicable.
    ‘what’ say you now?

  2. For loops are a bit of a nuisance in Python.
    My other, and bigger problems with Python is the horrible significance of using indentation to create loops, and that variables get added mysteriously by just mistyping a var. (Which is quite annoying to debug). I once debugged a Python program written by someone else, and it turned out a single tab from the last line of what should have been in a for loop was missing.

    I’m also quite missing the compile step, which does a lot of syntax checking, although “modern” python always seems to compile scripts before it runs them (and then hides the compiled results somewhere).

    Another annoyance is that you can’t add to an non existent list, which breaks “pythonesque syntax”. If you want to add items to a list in a loop, then you do have to declare an empty list beforehand.

    Don’t understand me wrong, there are a lot of nice Ideas in Python, but the overall language seems sloppy and badly designed. I have a quite strong suspicion that people who use and like python are also “sloppy” in their behavior.

    I’m longing for Python 3 or 5 which at least has an option to make it mandatory to declare variables (I like the Idea of the “strict” keyword in Perl. The turnover from Python V2 to V3 also took about 10 years too long. Partly because the break of compatibility was not handled in any decent way (such as that “strict” keyword) and therefore scripts depending on libraries written in an older version is… complicated. But also, python users apparently simply just don’t care.

    I quite like C and C++ (although neither are flawless), but to see how these languages have evolved over the last 50 years without loosing compatibility is quite amazing.

    Maintaining python programs is apparently also a dependency hell. KiCad V7 was scheduled for the end of January, but at the moment it is being delayed because of some wxPython problem that should never have existed in the first place.

    1. “I have a quite strong suspicion that people who use and like python are also “sloppy” in their behavior…. 50 years without loosing compatibility is quite amazing.”

      It’s amusing that such a careful person would conflate loosing and losing.

      Joking aside, I don’t think that C/C++ is so “compatible.” How many different C/C++ compilers are there right now? Clang, gcc, Microsoft, etc. I know just enough C/C++ to be dangerous but I’ve had many experiences where a compile would fail because of which compiler was used, the architecture slightly different, etc. The core functionality of C 50 years ago was so limited that it’s not too surprising that the core is still supported. Yes, python did change print from a statement to a function, the OOP model changed in subtle ways, etc but overall the feel of the language has remained.

      And C/C++ programmers talking about python’s dependency hell is the pot calling the kettle black.

    2. I’ve gotten used to indentation instead of brackets but I agree with not declaring variables (lack of strict keyword) and static code analysis being a hindrance. I don’t pretend to be an expert in language design but isn’t it funny how all the trendy dynamic languages (read python and JavaScript) have come around to specifying variable types. It is cool to just bang out code and watch it run (my most recent fun experience was playing with micropython and the prototyping is just so much quicker) but for a bigger project you probably want as much error checking as possible before you actually run the code.

      1. Here’s a short lesson in language design:

        Separation of concerns means that you shouldn’t expect language features like variable declaration to double as a spell checker. That’s not the language’s job. Any half-decent development environment will have tools to help with that.

          1. C without the training wheels. Actually, I love python for how quick I can fail and iterate. With C it just happens in the compiler first. You have to test your code either way.

        1. It’s impossible to solve this with a spell checker, especially in case of one letter variables. (And yes they do make sense in loops or in mathematical expressions).
          Only declaring them beforehand can solve this.

          Type inference is another issue, but allowing to use before declaration one or another way, makes it only possible for tools to guess. And guessing leads to annoying messages.

    3. > Another annoyance is that you can’t add to an non existent list, which breaks “pythonesque syntax”. If you want to add items to a list in a loop, then you do have to declare an empty list beforehand.

      I don’t get this point. How would you like to add an item to a non-existent list? You have to define the variable before operating on it.
      And if you mean something like `d[a].append` where d may or may not have element a, you may want to look at collections.defaultdict.

    4. > but the overall language seems sloppy and badly designed
      Tell that to the C designers, with all the “undefined behaviour.”

      Tell that to the Javascript people.
      >console.log(‘1’ == 1);
      >// Expected output: true

      Seriously?

      Python is great for smaller programs. I use it all the time.

      Once you get above some level of complexity, you need to switch to a statically typed, compiled language.

      Where that limit lies depends on the tasks at hand.

      The stuff I do at work? Not going to use Python, no way. The code base is enormous. You can’t get along with dynamic typing in that environment.

      Stufff I do at home? Yeah, Python is fine. The projects are small enough that you can readily check what type of object a function expects – by hand, as needed.

    5. What’s wrong with for-loops in Python? They’re way more declarative than using indexes. There are tons of built-in tools that allow for powerful for-loops.

      As for white space and variable typos: get better tools. Using static declaration as a spell checker is silly. It’s not the language’s job to catch your typos: that’s the job of a linter. Separation of concerns is a thing.

      1. A linter is historically a fix to compensate weaknesses of compilers. Later on their functionality was built-in for good reason. Separation of concerns makes a cut at well defined and reasonable boundaries, this is not one.

        Checking if a coding style is followed, yes, that is a separate concern from compiling. But checking if a variable has a clearly defined function throughout its scope is at the core of every compiler.

        This is goes far beyond “spell checking”, it’s completely changing the meaning of code, not just become slightly less readable (what a spell checker fixes).

  3. People here are funny! Some of these comments just show how many of you are just playing the Mr. know it all. How many of you have built large scale applications, used by millions of people on a daily basis? Do you know how many Python(version 2 and 3) applications are there that match that description? Come on people! Some of you here just sound jealous about Python being preferred over your C/C++ for a ton of projects. The language design some here are complaining about is what mathematicians and co. like and so far Python has been more successful than any other language at supporting those most of the related use cases. Talking about small projects , have you ever read the source code of the AWS CLI? People here talk like they were some Bjarne Stroustrup or a Guido. Some comments here are crazy!

  4. When I code for Arduino in C, I also use some funny preprocessor definitions for time calculations. The basic time unit is milliseconds, so to transform the values into something more meaningful, I use these definitions:

    #define seconds *1000
    #define minutes *60*1000
    #define hour *3600*1000
    #define hours *3600*1000

    Then, in the sketch that controls my garden watering system, I can use lines like these:

    unsigned long autoWaterRunningTime = (unsigned long) 16 seconds;
    unsigned long maxRunningTime = (unsigned long) 4 minutes + 15 seconds;

    I don’t know if is bad or good style, but I don’t care cause it works for me :)

Leave a Reply to Eugeny (@eugeeeeny)Cancel reply

Please be kind and respectful to help make the comments section excellent. (Comment Policy)

This site uses Akismet to reduce spam. Learn how your comment data is processed.