COBOL Isn’t The Issue: A Misinterpreted Crisis

Is history doomed to repeat itself? Or rather, is there really any doubt that it isn’t, considering recent events that made the news? I am of course talking about New Jersey’s call for COBOL programmers to fix their ancient unemployment system, collapsing under the application spikes caused by the COVID-19 lockdown. Soon after, other states joined in, and it becomes painfully apparent that we have learned absolutely nothing from Y2K: we still rely on the same old antiques running our infrastructure, and we still think people want to voluntarily write COBOL.

Or maybe they do? Following the calls for aid, things went strangely intense. IBM announced to offer free COBOL trainings and launched a forum where programmers can plug their skills and availability. The Open Mainframe Project’s COBOL programming course suddenly tops the list of trending GitHub projects, and Google Trends shows a massive peak for COBOL as well. COBOL is seemingly on its way to be one of the hottest languages of 2020, and it feels like it’s only a matter of time until we see some MicroCOBOL running on a Teensy.

However, the unemployment systems in question are unfortunately only a tiny selection of systems relying on decades old software, written in a language that went out of fashion a long time ago, which makes it difficult to find programmers in today’s times. Why is that?

In the end, COBOL is “just another language”, regardless of its age, and nothing stops anyone from learning it or developing with it. Just because it’s an old language doesn’t mean COBOL is entirely outdated and irrelevant either. Computer languages don’t “die”, they just go to sleep.

Or in the case of COBOL, it’s more that the areas where it makes sense to use the language are limited and highly specific. COBOL’s natural domain is reliable high volume data processing and it’s a perfect language for that use. COBOL is solely outdated as a general-purpose language.

So maybe the issue of finding COBOL programmers isn’t really about the language, but rather what its use implies about the systems using it, and the work environment one can expect. The thing is, those systems are a product of their times, where development processes and methodologies as we know them today simply didn’t exist, embedded in industries where software is merely seen as a tool, if not a nuisance. But COBOL itself is irrelevant here, we’d be in the same situation with any other language. Worse yet, if we don’t start to finally tackle the actual issues at hand, or at least learn from them, and continue to blame everything on an old language, history is going to repeat itself, and we will end up in this exact situation with other languages as well.

But what issues am I talking about then? Well, let’s start from the very beginning, and the lack of love for COBOL.

An Ode To COBOL

While virtually everyone has heard of COBOL in one way or another, and stating that someone you know has once worked with it gets you some serious street cred, the language doesn’t really get much attention here on Hackaday. If it wasn’t for honoring the life and works of those involved in its earliest development and paving the way for it, we’d barely see anything at all about it. But it shouldn’t really come as a surprise either, after all it is the Common Business-Oriented Language. Could a name be more hacker repellent? And unlike with BASIC, nostalgia won’t be of much help either, considering its main application areas.

As a result, COBOL has a sour side taste as a boring language that requires you to bring a tie and a broom stick to work. Nevertheless, COBOL is a definite and important part of computer history, and as we can see from the recent news, still plays a big role 61 years after the initial meeting that kicked off its existence. Depending where we look at, it might even be the dominating language in its field.

As rule of thumb, the more disastrous a system failure could be to society, the more likely it is to find COBOL in it, with banking usually serving as prime example, along with insurance, health sector, civil infrastructure, and of course government administration such as the aforementioned unemployment systems. Looking at some numbers, 43% of all banking systems, 80% of point-of-sale, and 95% of ATM transactions in the US used and relied on COBOL code in 2017, and it won’t be much different when we look at other countries.

So all in all, programming in COBOL gets you to work in incredibly sexy environments that can easily keep up with the excitement and coolness of, say, game or web development — not.

The Generation Conflict

Yes, yes, not everyone cares about that or wants to do gimmicky web or game development, apart from those spoiled youngsters who don’t even know how to change a register value with a screwdriver anymore — but that’s kinda the point. Every time a COBOL shortage is mentioned, the main issue seems that the old generation that knew the language has left the active workforce. Meanwhile, we advanced an entire generation since Y2K, and it’s still the same issue all over again. Yet those born during the last big COBOL crisis are now slowly entering the job market as new developers, in higher numbers than ever before, and their mindset, skills, and interests couldn’t be further away from the language and the systems using it.

And why would it? Software development and the opportunities within it is nothing like it was in the 90’s, not to mention the 60’s. Keep in mind we’re talking about a generation where the computer isn’t a miraculous wonder device, but a commodity. What was revolutionary once, has become a bare minimum at best.

But attractiveness and generational differences of interests and fascinations are really just the seagull sitting on top the tip of the iceberg. And no, compulsory COBOL courses in elementary school won’t be the solution. Plus, it’s not like you couldn’t run COBOL from Node.js or the other way around to cater to those youngster web developers.

On the other hand, different strokes for different folks and all, so the business field or its attractiveness might not necessarily be a problem. I mean, who am I to question people’s preferences, and developing for mainframes can definitely sound thrilling. But here’s the thing: these urgent calls for COBOL programmers only ever appear when there’s a crisis to be averted, so you’ll likely just end up as a code janitor whose usefulness ends as soon as the job is done. In other words: lack of job security as a blessing in disguise, since your work is likely going to be frustratingly dull and uninspiring anyway.

That leaves the people who either don’t care about any of this, and those who counter the job security situation by publicly claiming how they’ll do the job for a 2+ year full-time contract at $400 an hour and a complementary banana. In a way, they have a point, but the first group is definitely going to win that bidding. Why? Because this generational mismatch of values, interests, and expectations goes both ways, and we’re talking about industries that, for the most part, aren’t native to software development.

A Revealing Number

Let’s take another look at those numbers from earlier and focus on the one I didn’t mention yet: 220 billion lines of code in existence. Pick any random article about COBOL, and it’s guaranteed you will find this same number (±20 billion, depending on the year), presumably to impress by demonstrating how omnipresent the language and the systems its running on are.

Does it though? Or does it rather expose the zeitgeist of the industries where COBOL is actually in use, and the dissociation between today’s software world and those crying out for its help? Which would imply that this is an issue of obsolescence after all, but it’s still not the language, but the way software is perceived.

I mean, when was the last time lines of code had any relevance for software developers, or even their supervisors, especially where more was better? If anything, people pride themselves nowadays how few lines it took them to solve a common problem. Think of the millions of hours wasted in flame wars that could have been saved by simply showing how Linux’s ~50 million lines makes it clearly superior to FreeBSD’s puny ~25 million lines, and why OpenBSD still beats them all. (By this metric, Emacs would beat Vim, hence proving my point that lines of codes are clearly meaningless.)

But alright, maybe that number is just meant as a fun fact and I’m seriously overinterpreting it. Then again, maybe I’m not, and it really provides some unintended insight on what to expect in a COBOL work environment, where software development is seen as mere manual labour, like punching metal sheets in a factory. Don’t forget to add your words-per-minute count to your CV.

Anyway, that’s still not the whole story. Let’s talk about the code, then.

Ghosts In The Machine

How often do you look at someone else’s code and become convinced you could have done it better? And how often did you have to deal with legacy code and curse its author? If you’re lucky, you can just refactor it and move on with your life. If not, you might realize halfway through that everything has happened for a reason, despite how weird it looks, and you eventually find a less elegant way to work around it, introducing new peculiar code the next person may find questionable.

Now imagine that was the norm. Not only that, but imagine how decades and decades of workaround after workaround piling up in the codebase has become the very core of the system itself. No doubt, every single patch had its reason and validity to exist, and whoever wrote it surely knew what they were doing. Maybe even the next person working on it, adding their own workarounds, has been around long enough to remember everything about some specific quirk that had to be handled carefully. But that was ten iterations and 30 years ago. The original business has been acquired twice and restructured thrice since then, and each time it was integrated patched together with another system that has grown the exact same way.

Image CC BY-ND 2.0 Jason Tester Guerrilla Futures

Remember that we are talking about systems that are critical to our infrastructure. Their undisrupted operation is crucial, and some decades ago, you couldn’t just try out a few solutions on your commute to the office and run some automated tests during lunch. Whatever did the job had to be good enough, and we know very well that nothing is more permanent than a temporary solution that works. Considering that Y2K became an issue in the first place, people clearly didn’t expect that those systems remain relevent for decades to come.

Well, they did, and the point of no return was passed long ago. Mix in the mentality towards software and the replaceability of programmers, and things could have only gotten exponentially worse since. If Y2K was a problem because the people that initially developed the system had left the workforce, what can we expect a generation later, when those who had to painstakingly figure out all the quirks to add new workarounds themselves are retiring as well?

But once again, this has nothing to do with COBOL, and throwing COBOL courses at people won’t be a silver bullet. If anything, it takes a frustration-resistant mindset, perseverance, and a high pain threshold to dig through the code — something you can’t teach, and repeating the same old “we need COBOL programmers!” chants will hardly increase the number or availability of people with those abilities. Sure, it requires COBOL skills to fix the system, but COBOL skills alone won’t get you far.

Towards A Brighter Tomorrow

So what is the solution then? I hope it’s obvious enough that the problem is too complex in its entirety to have a single solution, and each aspect I mentioned, which I’m sure doesn’t even cover half of the entire picture, will require separate considerations.

One thing is sure, handwringing in hindsight that everything should have been handled differently back in the 60’s or 90’s won’t cut it, and neither will writing everything from scratch. Getting there little by little is probably a good start, but the complexity that makes the codebase a problem in the first place won’t be fixed this way. If anything, it risks complicating things even more  because it requires to fully analyze and understand every little detail and quirk, with zero room for errors or guesswork. It will take years and years of planning, and who is willing to invest that time and money when the majority of systems seem to work “just fine” right now?

As for the other issues, well, you cannot change people. But you can try to understand them — which goes both ways. Yes, the current generation makes software engineering probably one of the most spoiled professions ever, but just because you had it worse and didn’t whine doesn’t mean everyone else is willing to go through that themselves — just like technological basics and their history that seem obsolete in practice are still valuable and worth knowing.

We do live in a different age than the systems that are running COBOL. The ubiquitousness that software has reached doesn’t only provide entirely new and exciting work areas, but hopefully shows that it’s more than just a cost factor to those that have otherwise no direct relation to the field. And if not, then let’s at least take a lesson from it for ourselves. We can hardly blame the developers or their supervisors who didn’t know better at the time for this situation, but if we don’t start to finally learn from it, next time is definitely going to be on us. And that applies to areas both outside and inside the R&D departments.

Unlike 60 or even 20 years ago, we have a plethora of tools at our disposal today that allow us to track changes, progress, issues, and requirements in our software, most predominantly in form of issue tracking and version control. But none of that has any use with a halfhearted adaption for the sake of just having it done, and "fixed stuff." commit messages — and don’t even get me started on code documentation! The way we look at software development may have changed from 60 years ago, but we still have a long way ahead to adjust our mindset until we stop making the same mistakes all over again.

Well, at least next time the crisis will involve some more common languages, and everyone fixing this year’s unemployment systems can hopefully retire in peace by then. And I’m convinced, the headlines won’t call for Pascal or C developers then, so maybe we can finally stop pretending this is all a COBOL issue, when it’s really a “dreadful work expectation” issue.

On a final note, I’d be thrilled to hear stories from the trenches of those who were involved in preventing Y2K at the time!

142 thoughts on “COBOL Isn’t The Issue: A Misinterpreted Crisis

  1. I haven’t used COBOL since 1982 and I declined offers to return to it during the countdown to Y2K hitting, mainly because I was perfectly happy where I was and the extra cash wasn’t sufficient to draw me in. That doesn’t mean I regard it as an inferior language; when I used it, a well designed data division could reduce a program to read an input record MOVE it to an output record and print it, all of the formatting of the data was taken care of within the MOVE. Calculation could be similarly tailored for precision. For its intended purposes it was a fine language, the many tricks that COBOL programmers got up to can make maintaining and modifying it a nightmare; one of the reasons I stayed where I was (though I could have retired 10 years earlier had I succumbed).

    1. Owlman touches on the thing that distinguishes COBOL from modern languages… sometimes as much as half of a COBOL program is describing data formats…the actual procedural part that modern programmers would consider the program is often a very small part of the program. Once the data format has been described, doing something with it becomes comparatively easy.

      The problem, as I see it, is that actual computer people, people who are interested in how computers work inside and in the process of programming, seldom today are interested in big MIS/ADP/IT business projects. Finding programmers is easy; you just have to start paying a living wage and people will be lining up to do your programming. The hard part is doing the systems analysis, understanding the business and how to make the software work for the business.

      What is wrong in NJ is that they have a big mainframe database and they have a web front end and they have people printing out data from the web front end systems and punching them by hand into the mainframe database. Because when the people designed the web front end, they didn’t understand the mainframe or even care about it. Their goal was to provide the -appearance- of automation without actually automating anything.

      1. Bingo. All to often developers skip the WHAT questions and jump straight to HOW. In the above example, no doubt, the wanted to show their slick new new, state of the art UI. Never considered WHAT that UI was supposed to be doing.

    2. If I was a young software developer I would design and build a new system that would run parallel to the current system for 13 weeks (unemployment only runs for 13 weeks and then you have to re-apply in most states) and then retire the current systems on week 14. When maintaining a car you have to recognize that at some point it is simply cheaper and more reliable to buy a new car. Shiny web front end and the back-end running on Amazon AWS or similar. Let each state customize the business rules that determine eligibility, rate of pay, etc (middleware). Sell it to the states. Job done.

      1. You know just as I do that designing and building a new system that does exactly, down to all the details, the same the original system is to take you more like 150 weeks. Of which you’ll probably spend 50 weeks to memorize ALL the documentation that was ever written for the system, and another 50 weeks analyzing the code to find ALL the differences against that documentation, and generate new and accurate documentation. Then you’ll then spend 50 weeks programming a new system based off your new and fully accurate documentation.

        And what you’re then left with is a system that is EXACTLY the same as the old system, except it uses a more modern programming language.

        And then one executive says to you: “Guy, we payed you 2 million dollars over the past 150 weeks. Is your system better than the old system?”.

        And you’d have to say “no”, because it is EXACTLY the same as the old system. Not even better maintainable, because you couldn’t implement that more modern scalable architecture, as such an architecture would break your fully accurate requirements, and would lead to connected systems going on the fritz because events happen in a different order, or are being delayed less or more than previously, or whatever.

        You’ll simply never get to that 13 weeks. And if you’re unlucky, they’ll send the lawyers after you to get their money back. And you’ll end up sleeping under a bridge.

        And even if you DID get to that 13 weeks, you’ll spend another 20 weeks finding out why all the dependent systems are going on the fritz all the time, and patching your software to make the whole ecosystem work nicely with each other again.

          1. You are building a new system and tossing the old system in the trash. You do not read the old code and try to document and find all the differences in the old code. You do not try to replicate all the intricacies of what what the old code is doing. When you buy a new car you don’t measure and compare everything to the old car.
            The problem is simple. Design a system that lets people enter their information (millions of web pages do it every day), let an employee of the state review the information and OK the claim (send an email to the applicant requesting further information if needed), each week run the electronic payments. Everything runs on the cloud. If someone does not have a debit card give them one. Electronic payments and debits have been on my desktop for 20+ years.

        1. “And even if you DID get to that 13 weeks, you’ll spend another 20 weeks finding out why all the dependent systems are going on the fritz all the time, and patching your software to make the whole ecosystem work nicely with each other again.”

          Isn’t this exactly what software developers do? knock out something that does most of what you want.. then fix/patch the rest? …. time for a windows update….

          People don’t want change, change must be required or mandated.

      2. problem is that while running in parallel over 13 weeks, dosnt mean that every one applies at the same time. they dont. so a 13 week parallel would end very much longer, and even then wont be complete, at the very most perfect, you would have to do it for a year, and not have any remaining issues in the new system at least a few months before you cut over to the new one. on of the most common results of a new scratch built system, is complete failure, because every on involved doesnt know every thing the current software does, since its been automated for years if not decades. and they are looking almost exclusively at the shinny new thing

    3. I think this entire thread catches the core issues spot on. 1) Working with dying languages (both in terms of people and technical support)
      2) lack of long term planning (>10 years)
      3) the fundamentally fragmented nature of technology/compilation stacks
      4) inability to map old pipelines/interfaces to newer ones

      So this is not only a problem that plagued the industry but also research and quite literally every single kind of system development. We need to think about common data/process interfaces for everything we do from now on. I personally face a lot of these problems and I think python started doing the right way about this(when it comes to mapping C, C++ code to python) we need to start thinking building C/C++-struct interfaces for every piece of code we build.

  2. I’ll just say this. The root of the problem is no planning for maintenance. Getting the initial system installed and set up is all that matters or is budgeted for. On big telescopes I work with (my old day job), I saw the same mindset. The job was done when the system started up with success. Now those systems are obsolete hardware, software and hardware technologies that nobody on the current staff understands, and maybe even who knows where the source code is. The truth is that big systems need to be re-implemented on some kind of schedule, but try selling that to management. So nothing is done until some drastic failure happens and then the finger pointing begins.

    1. More specific than that, Government Managers are rewarded for saving money today, not tomorrow much less 10 years from now. I’ve seen it too much myself. If they can cut the cost of an effort today by $100k, they don’t care if it increases costs 5 or 10 years from now by $10M. Total lifecycle costs (long-term costs) are not a consideration.

      1. More specific than that, Politicians are elected for cutting taxes today, not tomorrow much less 10 years from now. I’ve seen it too much myself. If they can cut the taxes today by 10%, they don’t care if it increases costs 5 or 10 years from now by $10B. Total lifecycle costs (long-term costs) are not a consideration.

        1. More specific than that, Politicians are elected for spending today, not tomorrow much less 10 years from now. I’ve seen it too much myself. If they can make get elected, they don’t care about costs 5 or 10 years from now. Total lifecycle costs (long-term costs) are not a consideration.

          There, fixed that for you.

    2. There’s some interesting arguments in this. First, the mainframe isn’t some ancient piece of hardware. While IBM may be selling less hardware, it is in fact fare more powerful. The platform itself offers end-to-end encryption, runs linux, and is even making it easier to run tasks off host. For example, you can now use something like Zowe (zowe.org) to use modern testing frameworks and deployment tools like Jenkins.

      While I have had my prejudices against the platform, here’s one thing to consider. While Google handles 66K web searches a second, CICS on the mainframe handles 1.2 million.

      And there’s some 240 billion lines of COBOL code, growing about 1.5 billion a year.

      Is the platform dead? No.

      It is a hot platform to be on? No.

      But it is critical.

    3. “The truth is that big systems need to be re-implemented on some kind of schedule, but try selling that to management. So nothing is done until some drastic failure happens and then the finger pointing begins.”

      Reality is that your business invested a lot of money in that system, and re-implementing means to reinvest a similar sum of money. But that can only be done if the old system made the company at least twice as much as it cost at the time. And often it didn’t, because most of the money that the old system made was put into salaries and other investments necessary to make the company grow. So there is no budget for a new system, and the old system needs to be kept running because it pays for the salaries. Catch-22.

      At the same time there are startup companies who somehow pried a nice investment sum of someone, and don’t have a big business yet, so don’t have those high running costs. But what they do have is the lump of money, and the time to do something with it. So those startups have the chance to make a whole new system from scratch, which probably works better, and is more scalable, etc., etc., than the system of your own business.

      A really, really smart boss might invest in such a startup company, help them create their system according to their views (after all, they are experts), but nudge here and there so that the new system at some point comes close to being used to replace your old system. And then you buy the whole startup, pay them money to get rid of anyone who is too adventurous, and then tailor the system to your own company and slowly start phasing it in, replacing your old system wherever possible.

      Most management really DOES see that reimplementation once in a while is necessary. But most often their hands are tied by things that you’re not considering.

  3. I had an internship converting COBOL into Java to run on the mainframe JVM we were developing around 2005. Self taught for the most part, it’s not a super difficult language but has some really strange quirks. I haven’t used it since. Over the course of my career I’ve learned a bunch of obscure/dying/dead languages including FORTRAN and Ada. I worked for an engineering firm where we frequently built new hardware/software as a drop in to support legacy systems. There’s $$$s to be had in those old systems and sometimes its more about being adaptable than anything else. Maybe I should see if I can revive my COBOL for side cash.

      1. You don’t automatically get scalability or maintainability by translating an unscalable, unmaintainable program into a modern language. Scalability and maintainability have to be designed in.

    1. You have an interesting view of Ada. It’s neither obscure nor dying. Translation to a newer language is not the solution to the present problem, if the COBOL code is indeed at fault. If you miss COBOL problems while you’re translating you’re just translating errors. If you catch the errors, have someone on the COBOL temp team fix them. A solution next month beats a solution next year.

      1. Yes. At last, someone who is thinking straight. COBOL is a Language designed for Common-Business Oriented systems. It is not an Object Oriented language. It is not a Games Orientated language. It is not a Scientific Orientated language. It is not a Mathematical Orientated Language. And irrespective of how old ANY language is, logic is logic – even if it is a million years old.

      1. COBOL crash isn’t as problem.. it can be fixed.
        Helicopter crash IS a problem. for the pilot and passengers.
        if it isn’t a problem for you, do flight test yourself, alone. risk you life as you want. just yours..
        or just change to manage a plush shop.

        no one as his place in aeronautical industry if security isn’t his priority..
        look boeing..
        they haven’t finished paying it !!!

        1. > COBOL crash isn’t as problem.. it can be fixed.
          > Helicopter crash IS a problem. for the pilot and passengers.

          Tell that to the 20 million people who didn’t get their unemployment benefits that month and are grabbing rakes and torches to make you pay…

          1. but is that because of the front end (which isnt likely to be COBOL is it?) or the interface from the front end (might be cobol…but doubtful)..or is it because of the back end (maybe…but doubtful after banks by themselves seem to be able to support a million transactions per second). so did they under estimate the front end ( likely…whether is a cloud or what ever….going from maybe 100k applications per week to about 6-8 million a week….will break just about any thing)…or was it the interface (same problem)….or backend(same problem)
            cloud doesnt mean you dont have to worry at all about scale. just means you pay more for it when needed. but unless you knew you might need that, you might have set a limit on how high you wanted to pay. as mentioned its not the language, its not even the hardware, its the change in volume. and unexpected change of volume

      1. I’m skeptical that COBOL code is the source of their problems. Programs aren’t physical machines. They don’t wear out. If a program runs a 1,000 times running it 1,000,000 times won’t break it. Could be a data or data entry problem. My suspicion is “training issues”.

        1. They said it’s because of the massive influx of new applicants so I’m guessing simultaneous users, probably a matter of adequate load sharing and infrastructure stuff only all that is most likely ancient codebase, hence the problem

          1. I agree. It is probably less about the code than the system it is running on. My guess is the platform itself was pretty ancient along with the code. I bet a modern system running the COBOL code would have been OK.

          2. Steven Naslund: I disagree. Scalability is less about how fast the hardware runs or how much memory it has, and more about how the software handles simultaneous access.

        2. THAT depends on whether there’s some sort of counting mechanisms in the program that alter its behavior between runs.

          Suppose for example that the program keeps count on how many times it has experienced a power outage, to alert the administration to check that everything is fine. There’s a simple counter that keeps tally of how many times the program has started, and subtracts one each time it was shut down properly. Therefore if the number is greater than 1, the program knows it had a power outage.

          So, as time goes by, the program experiences a number of random power-outages and the administration keeps ignoring the warning messages because it’s part of a scripted system that nobody really looks at. Uh oh, the variable that loads up the boot count is only 8 bits, so after 255 blackouts, the program has an overflow and eats its own database.

        3. Actually, that 1,000,000th execution could be the first time that a particular path through the code was executed, and uncover the bug that only occurs with that particular sequence of decisions and actions within the program.

          1. That reminds me of Division bug that was in the Intel chip back in the ’90’s.
            Intel said something to the effect “Don’t worry, it would only happen maybe once in 200 million calculations.”
            Uh, wait, doesn’t that chip do 200 million calculations a second?

        4. But IIR in COBOL you do have to specify the size of everything you create & use and the systems I’ve seen based that size on the maximum use case as specified in the specs. So if they never expected more than x people per week to apply for benefits, every storage structure from buffers to files would be allocated for…x records, no more. In that respect it’s very much like Y2k; they don’t wear out physically but the size of the numbers grow past what they thought would be the lifespan of the code.

          1. THAT is the problem. Or at least part of it. If it was only that, all you’d have to do is increase the array sizes, and job done. But it’s never that simple, because in the days of buck-a-bit memory, programmers went to great lengths to take as little memory as possible for each record. Which was where the Y2K problem came from – programmers said, “Look, we’re spending two digits of memory on EVERY TRANSACTION, that NEVER CHANGE. I can fix that.”

  4. I think the problem is they were asking for “volunteers” to write COBOL well um good luck with that, as it turns out software development is hard and you generally have to pay people a market rate salary to do that type of work. NJ would rather spend all that tax revenue on… whatever it is they spend it on, I couldn’t tell you after living in NJ for a year what it is they do with all that money but yeah….

    1. I laughed when I read “volunteer” At least i got paid well for Y2K remediation (on non-COBOL systems) even if no one today seems to think there was a real problem because we did a fairy decent job of preventing a disaster.

        1. Heh. Story of my life. This is how i once lost a job. Boss:’What are you doing the whole day? There is nothing productive i see here! You better look out for a new job till the end of the month.’
          Me, spending 10 hour a day to keep everything running, because the ‘smart, productive people’ keep on breaking everything.
          Called me weeks after leaving: ‘Umm, can you help me about… .’ Nope. Nope, nope, nope!
          I usually do not hold a grudge, but this one gave me an evil smirk.

      1. So did I. Laughable that they are looking for volunteers to fix a problem caused by their IT staff who apparently failed to scale their systems properly. Were the IT staff volunteers? Doubt it.

    2. That mainframe forum linked above rapidly filled with volunteer offers, but no projects actually needing/accepting help.

      There is a surplus, not a shortage. The organizations do not have the ability to oversee the changes they want made, so they’re not able to make use of the volunteers that they imagined they needed.

      What they need is competent project managers that they know and trust; well, it is hard to find volunteers for that one. You’ll have a line of applicants, but isn’t feasible to select any. They’d have to hire a hiring consultant to process the volunteers, and we all know what the quality results of that would be.

      You can’t have it be an important system that handles large amounts of money, with an organization unwilling to spend the money on the structural oversight needed for maintenance programming, and then make changes. Forget about changes. Think work-arounds.

      1. Nah, first they need incompetent project managers to screw up so badly that the system is irretrievably trashed, then they need competent project managers to start over from scratch.

      2. “Volunteers” are a synonym for “we don’t have a budget so we can’t pay you the going rate” Remember this is Jersey, one of the most corrupt and badly run states in the U.S.

  5. I find that when evaluating and reversing code, that I don’t just need to understand the code, but I also need to understand the culture it was written in. Very often divisions in the code matching the silos in the workplace. I spend so much time figuring out why one module encoded everything into a datatype mess then hand off like an internal interface to another module where the datatypes are rehydrated. It makes sense when you realize the modules were written separately and 3 years apart to replace paper processes that nobody has thought about in years since that division was eliminated because the code was more efficient.

    Spending significant time in efficient design, the original devs wrote like they were bullet proof because processing time, storage, and memory was gold and so was the guarantee of a pension. Old problems solved in COBOL became commonplace, legends repeated over and over and are now crystalized in silicon in every proc in the last 30 years. They became native processor calls, but they are so old and antiquated that recent compilers don’t include them even when they are exactly the right thing for the problem at hand.

  6. So you wanted a COBOL “hacking” story:

    Back in the day (1982) I heard that someone on the IBM 360 Mainframe at Uni, figures out that if you assigned a value to a constant, it overwrote the constant’s value in core memory. Some of the constants that were “cached” as it were, were the literal values for 2, 3, 4 etc.

    So he (for we only had one female Computer Science student in 1982, and I don’t think she would have stooped to this) submitted a punched card job assigning funny values to some of the constants. And then sat back to observe the general confusion when the following programs failed to work, or terminate, or gave strange results!

    I think it was COBOL, but I might be wrong, it might have been in assembler, which then broke the COBOL batch jobs.

    1. Doesn’t make sense, or I’m not understanding. If I submit a deck with a constant a=1 and the next person submits a deck with a constant a=2, that should work every trip of the train.

      1. It should. Unless there was some disparity between the machine’s native character set and the input set that you needed to set 1=1 etc, and that was handled globally, so you didn’t expect to have to redefine it each program.

      2. Same in Forth;
        on evaluation of source, words are tested if they are function and then if they are number.
        Since you can define function with any name, defining function ‘1’ that returns ‘2’ make things go funny :)

    1. For me it was the mindset of the people maintaining the mainframes. I asked what version of a particular message queue software we were running, since some recent versions could be spun up in a container for rapid local development cycles, and the answer was a suspicious “why do you want to know?” as if I’d just asked for someone’s mother’s maiden name. It took me three days to get a straight answer out of someone.

      From then on I stuck to the documentation.

  7. FORTRAN H had the property that because it is entirely “pass by reference”, you could change the value of constants. A friend learned this the hard way debugging a program over what should have been Christmas break at work. However, I doubt this persisted from one program run to another. It’s a natural result of passing a constant argument as a variable in a subroutine or function call. The called routine gets the address of the constant, as there was no way to pass by value in FORTRAN H. So if the called routine modifies the argument, the constant is changed. VAX FORTRAN introduced “pass by value” to preclude such problems.

    It’s simply an example of why you need to understand a lot more than the syntax of a language to use it properly. I did a lot of mixed C & FORTRAN 77 programming to provide dynamic memory allocation for old scientific codes which no one but a complete fool would want to rewrite. FORTRAN called a C routine which allocated the memory and called the old FORTRAN routine. When it returned, the C code freed the memory before returning to the calling FORTRAN code. I also threw exceptions in C++ passed back from FORTRAN. And as an afternoon amusement wrote a program that passed values between functions without any arguments.

    COBOL was designed for *business* data processing requirements. FORTRAN was designed for *scientific* programming. C was designed for *systems* programming. I never have understood why people insist upon using a language for tasks for which it was not intended. Nor why some people want to rewrite a well tested and debugged algorithm in the language de jour. Though I suspect that writing GUI codes rather than scientific codes ahs a lot to do with it.

  8. My software career began in 1970 with a 1 unit FORTRAN class. I asked someone, “what the heck is FORTRAN?” The answer was, “some kind of computer class”. I signed up and the rest is history. I gained the reputation as the computer whiz in the dorm. Some guy brought his programming project to me, and I confidently flipped open the printed listing. The same question resulted, “what the heck is this?” The answer was, “COBOL”. I told him I was sorry, very sorry.

    There is my COBOL hacking story.

    The same thing I say to good old New Jersey. I am very sorry. For them to exploit the current crisis to do something about an outdated system they should have done something about long long ago borders on the evil, the criminal, the insane!
    I am sorry, very sorry.

  9. I had a ‘play’ updating some COBOL business code – back around 1980,;and found that while it was well suited to the task, it was a ‘boring,’ language that didn’t allow any flair or creativity in presenting output or handling data in some ‘other wat’. Very bow tie, and cardigan. I enjoyed the experience,but would never spec it for a current solution.
    FORTRAN had similar issues, great for what it was used for, but very inflexible as a general purpose language.

  10. Not that I’m one to repeat unsubstantiated scuttlebutt… well much… or unless it’s amusing… or it supports my point of view… you know important reasons… but I heard a rumour that the Army quartermaster corps investigated training chimps to program COBOL in the 1970s, to help handle their record keeping. However the project ended in a stalemate between the corps of veterinarians who maintained that it was inhumane to lobotomise chimpanzees for any purpose and the animal behaviourists who having taken a good look at COBOL insisted that it was inhumane not to.

  11. The real difficulty is not COBOL itself (it’s just another programming language) but the environment it generally runs on: CICS on z/OS mainframes. The various CICS extensions to COBOL can be arcane and tend to have a steep learning curve. Also the environment is not: clone source from repo, load into favorite IDE, compile and debug. For the most part one would probably be running a 3270 terminal emulator somewhere (Windows, Linux, MacOS) and interacting with the mainframe that way. Most of the application software is 40+ years old and has been hacked on so many times that calling it spaghetti code would be a compliment.

    Not sure I would ever want to go back to that environment again. The one project I was on, interfacing a JAVA based web service with a general ledger written in COBOL/CICS may have caused permanent drain bamage.

  12. When I was learning FORTRAN and such back at the tail end of the keypunch era, I noticed that at some point during the semester someone would hang out a shingle: “COBOL Help Desk”.

    Never one for FORTRAN, PL/1, or anything else. This said something about either the language or those forced to program in it.

  13. I think COBOL is just one part of the whole “big business, big computers” mindset.

    When I was little in the late 1970, my Dad worked in the IT department of one of the biggest American banks. When I got excited that there were “computers” at his work, he brought home (besides old punched cards to play with) an outdated “Systems Standards Manual” in a binder with replaceable pages. Dry, way over my head, but I kept at it trying to read it. I distinctly remember “Identification Division, Records Section” which (in retrospect) were clearly COBOL standards to use. But there was much more in the manual: flowchart standards, and a whole section on the proper method of “walkthrus.” Wish I still had that manual.

    The whole mindset was a long, carefully documented standard of EVERY step in the creation of a business program, down to the final printouts on 132-column paper. Can you imagine today having to plan your entire program ahead of time, flowchart chart it, have the committee sit down and walkthru every step carefully, before finalizing the code in COBOL format? That’s what I imagine is the COBOL mentality.

    1. People wrote complete programs on paper. This sounds crazy for the current generation but we really did. I learned BASIC from a book and wrote my first programs with pen & paper and transferred that to punch cards. You could send in one batch of cards per week and had to wait another week for the results. One error was a disaster so you quickly learned or you went under.

    2. A little more planning and a little fix-as-you-go is not a bad thing. I sorta planned my PHP pages for CRUD on a database, got them working (FAYG yes, but it was my first attempt at PHP), then memorized them. Don’t remember them in detail 10 years later, but I do remember the plan.

      Can you imagine storyboarding a screenplay? Crazy talk, amirite?

  14. Ah, COBOL. I voluntarily took a COBOL class back around 1970, despite COBOL being seriously looked down upon by us FORTRAN, etc programmers even way back then. The thing was, on the CDC 6400 computer our university had, COBOL was the first language which allowed one to read from and write to the Teletype terminals! Our FORTAN etc required you to submit a punchcard deck (entered on an 026 or 029 card punch and edited by physically moving cards) wrapped in a rubber band, then come back half an hour later to sift though a pile of decks wrapped in 132 column line printer paper outputs and find your own results.

    COBOL normally was the same, but the new release also allowed your program to interact in clickey clackey real time with a human. That interactivity, where you type and then the program types in return, was really exciting in a way that few today can imagine. I was quickly hooked!

    Not on COBOL (which I’ve not used since), but on the interactivity. Luckily, other languages later supported the teletypes, and later video terminals, and onward to graphic screens and mice, and handheld smartphones. But it was COBOL strangely enough, which in my case first demonstrated the excitement of interactive computing. I was seduced from being a math major into becomng a programmer – my reward circuits responded more actively to creating abstractions that could play with me, than to the abstract beauty of a well constructed theorum that just sits on the page.

  15. I have a book on COBOL from 1970’s, the unique version of the language made by and for my country – the entire syntax was translated into polish. The only thing the language lacked were polish diacritics. The book also covered FORTRAN – its syntax for polish computers was not translated for some reason. Probably because there was much more useful imperialist software in FORTRAN, than in COBOL…

  16. I disagree with a lot of comments on here. COBOL is the problem. Yes, just another language, and no, not at all hard to learn. For a lot of reasons it is actually quite a reasonable language, and particularly usable for beginners too. Downside is it is utterly uninteresting – mind numbingly boring just doesn’t do justice to it. Such a long winded ****ed up idea of a programming language.

    My brief encounter with it was enough to almost destroy a life long love of programming and forced a serious career reevaluation. SInce that time I’ve refused multiple job offers as a cobol programmer, and while they have all been very well paid it has never been enough to make me want to have my brains dribble out my ears from boredom every working day.

    My advice if some poor unfortunate offers you a COBOL job: run. If they haven’t managed to move away from it yet then not only is the work going to be soul destroying but the mismanagement will drive you insane. There is no good reason for COBOL in this day and age.

  17. I think the reasons those systems are still running, after 40 years (or more) of being patched is that many of them are much better designed in the first place than some of the crap that gets written today – irrespective of language..

    Some of that design effort was forced because of things like CICS – where you have to make everything into independent functional units that could work in large transaction systems – and if you did it wrong 200K people couldn’t log in on Monday..

    So it’s that – coupled with the languages minimum scope – that has made the progs so (relatively) reliable

  18. Yeah, I wrote my share of COBOL back in the day. And yeah, it is just another language. The real challenge in maintaining legacy code is that things were different back then.

    The structured coding conventions (sequence, if-then-else, do-while, do-until etc…) that we take for granted today didn’t come into wide use until the 1980s. A lot of code wasn’t organized into a nicely nested cascade of Squeak blocks. And objects flat didn’t exist. The three-branch IF and the GOTO were still a thing. Everybody that coded for a living became familiar with spaghetti code.

    I once had to deconvolve someones BASIC program that had logic so complex, that I had to draw the logic flows in the margins of the listings, using colored pencils. And it was only about two pages long. And this was a typical day at work.

    Moral of the story: Learning the language is easy. Figuring out how the code works so you don’t break it when you try to fix it is a -whole- lot harder. Also, the GOTO can burn in Hell.

    1. This!

      Today, a click opens a compiler, a click opens a code editor and a click executes the compiled code.

      You can mod code and compile/execute so fast that it’s like working with an interpreted language.

      COBOL was designed for the workflow of that era.

      You started planning out with flow charts.

      You spent a lot of time with that phase as the resulting flowchart is you source code as opposed to COBOL being your source code.

      COBOL had poor support for strong structure and often the structure was unintelligible from reading complex COBOL. It was the flowchart that you looked at to understand the structure.

      Next you would start writing out code on paper. Then compare your code to the flowchart to confirm your code.

      Then transfer the code to punch cards and send it off to the computer site.

      Your code was expected to run without bugs (let alone crashing) the very first time. As you were expected to debug at the written paper stage.

      Any runtime bug was seen as complete incompetence.

      As time went on the programming paradigms shifted and people didn’t bother to update flowcharts as that was less a requirement for other newer languages.

      COBOL really suffered for this. Getting COBOL instead of the underlaying flowchart is like getting an bin file instead of C.

      There other problems back then to. Early compilers were single pass so had to place things in the right order for the compiler. Memory was limited on smaller systems so you couldn’t just put all your compiler directives at the top and expect them to be stored till needed. You had to place then immediately before they were needed and unset them when your finished with them. Like a manual garbage collection.

      Also, I wouldn’t say that COBOL was just another language.

      It’s a very early language that had poor structure and was procedural only, no classes, no OOP, no first class citizens, no inheritance.

      Just try and find something similar between COBOL and a modern language like Lua.

      1. “Any runtime bug was seen as complete incompetence.”
        – As you can see that part right there is a terrible, nay, stupid system.

        “COBOL really suffered for this. Getting COBOL instead of the underlaying flowchart is like getting an bin file instead of C.”
        – can your find the runtime vug or bufs in the abobe sentence?

  19. As part of my Computer Science courses, I had to write a few applications in Cobol. Early 80s. That is as much as I used it, as my field was Real-time control and C and Assembly were the languages then. I have downloaded Open Cobol for ‘fun’ over the years on Linux, but never wrote a ‘serious’ application with it. To late in my career to ‘start’ over. I worked with scientists for awhile and the language was Fortran. The control system I currently support still has Fortran code, although in the next version all the Fortran has been converted to C/C++.

    Good luck on finding programmers. Probably pays well though and built in job security if one cares about that!

    1. That would be a worthy project. Doing the two in parallel, it would be interesting to see which was actually cheaper and/or faster.

      I still don’t think the code is the root problem in this instance. Search this page for “1,000,000” for my reasoning.

    2. The FAA tried what you suggested and gave up. The modern coders couldn’t replace the old software. What they produced was buggy and unreliable. Not something you want in a ATC system. The new coders were too incompetent.

      In case you missed it, it’s not that that the COBOL program in question is buggy, it’s just that the entire system is being overwhelmed by the amount of applications., I bet their mainframe is maxed out.

      Generally rewriting code in some new cool language is silly. if the current version works and is dead reliable. Sadly reliable is not something you can say today about modern softwer.

  20. There’s a lot of food for thought in this article and I enjoyed it. But it starts from a dubious premise: that the problems (whatever they are) that New Jersey is experiencing are COBOL-based rather than due to human factors. “We processed so many applications (for unemployment) that the software broke” just doesn’t sound likely to be true.

    1. I don’t think that is the problem either. The problem that I think it is. Is the CARES act requires too many changes to the system to be done in a very short time frame.

  21. The problem is more that large software companies have found that failing to deliver very large software projects is far more profitable than actually delivering them. These projects are managed so as to keep the customer moving the requirements so the company can keep claiming project variation money. As a result these long obsolete legacy systems continue in operation because no-one can ever bring a replacement system to completion. Most likely it will be abandoned because of poor functionality, performance or massive cost overruns.

  22. Back in high school (1979-1982) the vocational school offered “Data Processing” on an IBM System 34 (i think been
    a few decades). Once I was done the course, I never used RPG II again. I didn’t really get into computers until about
    1983-1984 with the Commodore 64 and the Apple //e. Hence, the language of choice was BASIC on the C64
    and Applesoft Basic on the //e. Then I managed to find out about the wonderful world of telecommunications and
    the blazing speed of 300 baud. BBS’s were everywhere. Then fate hit. One program for the Apple //e written by
    a gentleman from Rockford Illinois. Diversi-Dial, or D-Dial for short. A program that loaded through the Apple //e’s
    cassette port that allowed 7 300 baud modems to talk to each other. RPG II and the very beginnings of COBOL were
    forgotten. The phone lines connected D-Dial stations together creating one of the earliest forms of multi-network
    chat and e-mail. It was an exciting time. Early packet networks like Sprint offered a service called PC-Pursuit.
    This allowed stations to call other stations in distant cities without incurring long distance charges. (Remember those?)
    Now we have a worldwide network that can pull information from practically anywhere in the world in less than a few
    minutes. I miss those early days, what would have happened had I kept up with RPG II and finished learning COBOL?
    I’ll never know. My path went down a different road, so I will never know.

      1. Glad you liked it. It was meant to show that not everyone heads down the same path.
        I could’ve kept up with RPG II and COBOL, but as you read, I didn’t as did a lot of other people.
        If you were around during the early 80’s you know the Commodore 64 was one of the hottest
        selling machines of the time. Also, I DID mention COBOL and RPG II in my previous comment.

  23. My father is a COBOL programmer and therefore I’m quite familiar with the language and the development environments that they use. At least in his company, version control and documentation generators have been used during the last 30 years and they enforce its use, so the system is indeed well documented. The problem is that it is not possible to look for code on github and use third party libraries, protocols need to be programmed to follow international standards and, indeed, that’s the problem they have when hiring new programmers: they need to be taught about those protocols and the way that system works and managers want to find someone that already knows that to avoid investing in training them. Consequently, several years ago they tried to migrate the system to distributed servers and modern languages. After several crashes due to unmantained third party software and libraries, they decided to keep using COBOL and their IBM mainframe and nowadays even the software that runs on their linux servers is programmed in COBOL. They even invested in modern COBOL IDEs that include autocompletion.

    I think that the problem is a consequence of considering programmers sustituible without training them and having obscure unmantained code due to having less programmers than required. Changing the language and the tools won’t fix that and rewriting huge systems each time a new language can be riskier

    1. True, changing the language won’t fix things. I don’t think the problem is not enough programmers who know
      COBOL, but programmers who are well versed in COBOL. It comes down to the hardware sometimes.
      For instance, I know Applesoft BASIC. How many people still use an Apple //e?
      A machine well maintained will run for quite a long time, and a lot of these mini-computers from back in the
      day are still maintained and running quite well. As long as parts can be sourced for the equipment, a mini
      computer can run pretty much forever. As with my comment above, things change. Back in my day, we had
      the C64 and the //e as well as Atari, the Amiga etc. The hardware changes. Nowadays everyone pretty much
      has a PC. Although the C64 and Apple have their following, the mini computers were designed at a time when
      there was no internet, and they were pretty much designed to do one thing whereas a PC can do multiple
      things, run different languages etc. I’m of the thought, if something is working, and isn’t broken, don’t fix it.
      These mini computers run COBOL well, and probably will continue to run COBOL as long as they are
      maintained and kept running. I come from an era as described above so for me, COBOL was before my time.
      Could I learn COBOL? Sure, but is it worth my time to do so? It comes down to the almighty dollar.
      If I can support myself by learning COBOL and getting that kind of work, it would make sense.
      However, staying in my current field currently meets my needs so no need to change.

  24. Isn´t that COBOL thing just an example of the deliquescent state of the whole country ?
    with crumbing infrastructures, increasing poverty, no free education, no health insurance for most, 40% of the population who still have no broadband Internet access, and so and so ?
    The pandemic is just the trigger pushing the country off the precipice. The “invisible hand of the market” doctrine is a disaster, yet no one has to wait for it to happen: it´s already there, and the rest of the world can see how the leader of the third world countries (a first class sh++++le) is heading.

    1. The “invisible hand of the market” does not describe the situation when more than a third of the GDP is government spending. The free market is eclipsed by the state and the handful or megacorporations that are in bed with the state.

      The New Deal re-defined the role of the government as an economic stabilizer that was supposed to prevent large swings in the economy, but that theory never worked right and simply created a continuous spiral of debt and inflation.

    2. Third world country is a political term. The first world is the countries aligned with the USA, the second the USSR, and the third are unaligned. It is by definition impossible for the USA to be leader of third world. Now these unaligned countries were typically the unwanted very poor countries hence the slang derogatory use found often today. Your use of it shows ignorance though as does most of the rest of your premises including the big one of conflating government actions with the free market.

  25. I can’t believe nobody caught on the “ancient unemployment system” bit.

    From the perspective of its original programmers, I think the system is doing what it was supposed to do.

  26. My mum wrote COBOL. She complained about 2-digit dates when she wrote it: some people were born in the previous century, and worse still, some live over 100 years.
    She always spoke of COBOL with a sneer. Real programmers, she told me, wrote FORTRAN. (Yes, real programmers write assembly, and she did that too). Why FORTRAN? Because it showed you had some competence beyond being a code monkey, and people who managed FORTRAN projects didn’t do stupid stuff like 2-digit dates.

  27. Wow, the NJ governor letting literally anyone work with mainframes and datasheets with highly sensitive information. I predict that 2021 is going to see a mysteriously huge spike in identity theft specifically for anyone applying for unemployment in NJ.

  28. The answer is obvious to me.

    Step One is to replace all those old mainframes with OpenVMS systems; the OS is old but HP (the last big party to acquire it in full) still sells it, and there’s a company outside Boston that’s still selling-and-supporting it, or duplicating it from scratch based on information bought-or-licensed from HP, or something like that. But I digress. The point is this: on OpenVMS, anything that can be written in one language can be written in any of several others. There’s none of the “you can’t do _____ in [language]” that has pervaded all subsequent operating systems (and certainly the Unix/Linux and Windows worlds; I don’t know Apple).

    Step Two is to transfer all that mainframe COBOL code to those OpenVMS systems, and attempt to compile it. Make whatever changes are necessary to get it to compile, and to operate. (Verify proper operation by running it side-by-side with the mainframe version for a year or two; after forty years of doing nothing, one more year hardly matters.)

    Step Three is to start rewriting small pieces of the COBOL code in modern languages, whichever of them is/are supported on OpenVMS today. Just make sure it’s an actual compiled language, producing code that runs directly on the hardware, not an interpreted language that produces pseudocode that runs an order of magnitude more slowly in a virtual machine. (That last bit will probably be the hardest pill for today’s web-and-mobile developers to swallow.) Continue running the ever-evolving system side-by-side with the legacy one, as above.

    Step Four is to repeat Step Three, module-by-module, until the whole thing has been converted.

    Step Five, which I consider optional, is to alter the front end to a more modern user interface, and the back end to more modern data-storage technology (databases, cloud, whatever) — but only to the extent that it doesn’t break anything inside.

    Seems straightforward enough. I could do it in my basement, if I had more drive space and an extra forty years of lifespan.

    1. Or simply put, those who are now applying for unemployment welfare have plenty of time to die of old age before getting any help ? Well, that would spare quite some costs, just pulling the plug and let them die of hunger, and reorient that money to some big companies who need it. There are plenty around, who have bad numbers because of the current crisis. Workers are replaceable, but big corporations not so much.

      Some of those people applying for unemployment money might even be proficient COBOL coders, why not isolate them into private-run jails where they would produce the code for few cents an hour ? that would be efficient, and cost-cutting !

    2. VMS was a complete environment. The CISC instruction set helping to enforce the calling convention and the single source for most of the language did make it easy to mix languages for a single executable easily. Other envirments had paths for that though, and really for these types of systems you can have the interface be the dB and multi processes. No need to use VMS itself to use those ideas. However if you get Owen to be sysadmin, sure why not.

  29. The majority opinion seems to be this:
    Better code is more maintainable code, and more maintainable code costs more money…UP FRONT. If the project life cycle is even worth mentioning, “Minimum Viable Product” code (i.e. less well designed and less maintainable) will cost more money, and a whole lot more grief.

    My first office gig was writing internal use business programs in Visual BASIC. At $20/hr, I was still cheaper than dealing with corp. IT. Was my extremely junior code the best long term investment?–not at all. You get what you pay for. (And don’t tell me all languages are equivalent…VB should rot.)

    What can we do? I don’t know. Seems like money is the root of all evil, or at least all bad code. Tell your project managers, politely, when the long term might be suffering for the short term. Document your code, even when you don’t have time–“pay it forward,” but for the brave engineers that will follow you. (And consider languages that offer runtime error handling…maybe?)

  30. “Whatever did the job had to be good enough, and we know very well that nothing is more permanent than a temporary solution that works. ”

    This is how I feel about makefiles. Friggin makefiles.

  31. From 1995 to 2000, a COBOL programmer by the name of Ed Lein, unmarried and without children, made a fortune revising legacy code to handle the Y2K problem. He was employed as a contractor, mostly working at IBM and others. He did his job well, and his revisions ensured his applications would handle Y2K, 2038 and beyond. Basically Ed was set for a life of luxury and freedom.

    As luck would have it, in 2005 he developed acute myeloid leukemia. Instead of undergoing radiation and chemotherapies, he opted for an experimental procedure pioneered by Dr. Ken Storey, whereby he had his blood replaced by a biologically safe antifreeze, then had his body temperature reduced to slightly below freezing. (This is the same approach used by certain amphibians in Arctic climates. The idea is to slow metabolism to essentially zero, and limit freezing to extracellular fluid, but not intracellular tissue.)

    Ed’s hope was that in the future, advances in medical science could have him reanimated and cured of his leukemia. He put all of his fortune in the scheme, but with no dependents, his money was his to use as he saw fit and he paid for maintenance for fifty years. As the years passed Ed remained in stasis. Analysis of microscopic tissue samples taken each year showed negligible degradation.

    Many more years had passed. Eventually, medical science had advanced somewhat to realize Ed’s hopes. When he woke up in this wondrous new world, he was immediately struck with how familiar his surroundings were. It was if he were back in 2005, with his old desk and reference manuals. He asked someone who looked in charge, “Am I cured?”, the reply was in the negative. Ed sighed a bit and asked “What year is it?”, the shocking reply came with an explanation:

    “Please let me elucidate”, he said, then continued in a halting manner, as if not totally familiar with the language, “In 2015, your equipment was saved with you as a precaution. You can thank the people at HP for their foresight. Anyway, we really admire your work and we still haven’t gotten around to replacing a lot of apps in some critical systems. We’re wondering if you can help us out with the Y10K problem coming up? Sorry, we can’t do anything about your leukemia, but given your past performance, the modifications shouldn’t take you long, and we’ll pay you enough so you can go back in the freezer. Who knows, maybe in the future they can cure you!”

    As Ed listened, the horror of revisiting his COBOL after eight millennia of neglect struck home: “Where do you guys keep the coffee?”

  32. I was tasked with building a new computer system for a financial institution. Spent a lot of time shaddowing tellers, then the telephone operators, etc. At the end of that time/pay period I got my ass chewed out for all the time I was wasting with no measurable work completed. My reply, “ah, if I’m to be designing a computer system that tellers (and really each department) use, shouldn’t I have some functional idea of how they ACTUALLY do their jobs and what the workers need from the software?”

    “oh well yes, guess that is true, very well. continue.”

    My system was in use for about ten years and once replace (I was long gone by then =), former co-workers that I’d bump into would tell me how superior my software was and how the new graphical web based system actually generated more work (management: thats a huge loss in FTE).

    Thus the problem is lack of big picture, ego and arrogance in management.

  33. Sorry if this has been mentioned before. Lots of great comments, but I didn’t read them all. COBOL is a file based language. (in the end they all are). But, COBOL requires a different way of thinking about the solution. Like reading Shakespeare. You can read the words, know what most of them are. But understanding what they are saying takes a bit of study. And, in the case of COBOL, understanding where your grandfather came from. I’ve programmed BASIC, C, Java, and Python. But, I can easily work with C#, C++, Go, any number of modern languages. They are all similar enough. But, COBOL …

  34. I was one of the Oracle DBAs on our state DMV Y2K complete replacement of the mainframe system. The old system used a database engine that could only interface with COBOL-I. I was mostly self taught in numerous languages but had managed to avoid COBOL up till then. I had to learn to structure IF statement logic with no ELSE clause in order to extract the data. I use the techniques to this day and my code runs faster and is easier to understand.

  35. As a 40+ year veteran of old and new tech, I am compelled to make several short responses:
    1) The COBOL programs in the days of old were developed to handle the expected loads of then, probably plus 20%. They were not the problem, the unexpected 500% spike in unemployment claims were. New technology stacks would have the same problem if sized and designed for expected loads plus nominal spikes. No one would design a system to handle a spike not expected in their lifetime. It would be a waste of tax dollars.
    2) COBOL code and accompanying WFL (Work Flow Language) or JCL (Job Control Language) was relatively simple, as the mainframe of the day handled the comms, firewall access, load balancing, autoscaling, DB management, and all of the accompanying network/resource utilization without what now is a nightmare with all of the disjointed and constantly changing open source technology that is required using a DevOps model out of necessity, and will require many more times the manpower to keep things running to perform the same task.
    3) In my current organization, we are constantly tweaking and tuning the new-tech infrastructure to make sure our containers and JVM / Core have the right amount of memory, ports are opened up, DB instance types are configured correctly, etc. etc. We will be hiring more O&M developers and platform teams to keep this application up and running as we add more features and troubleshoot for years (5-10?) , when the old COBOL system it replaced was barely even remembered, as it just ran and ran…. for decades.
    4) My idea of a perfect system…. Leave the data on the big iron, put a nice web front end over it, and call it a day.
    5) Prediction … Someday not too long from now, all the headaches that accompany modern technology stacks, will be integrated into one box and be as easy to manage as my laptop. It will be the new “ModFrame”.

    1. It’s not the time it takes to fix the problem. It’s the many weeks it can take to thoroughly understand the language and the code so that your “fix” doesn’t break the system in other ways plus the time it takes to test the fix under all conditions to uncover such bugs.

      I’m self taught, starting with BASIC in 1981. I managed to avoid COBOL till the late 90s when I was hired as an Oracle DBA by a state DMV agency to help completely replace their systems. Lives literally depended on these systems so “downtime was not an option.” The experience of working in a mature IT shop changed my whole approach to coding. It was the greatest experience I’ve ever had.

      While IT shops like this aren’t as obsessive as NASA, the organization and development practices are similar. There were the classic 3 systems… development, test(QA), and production for everything. Most importantly, there were policies and practices in place (and enforced) that insured that nothing got into production without being carefully scrutinized by multiple people. Humans are not as fast as computers so this takes time.

      The inexperienced and management will always grossly underestimate the time required for any software development task. When they got their shiny new system all developed and “ready to go live,” they gathered my “database maintenance” team in a room and told us we had 2 weeks to extract and transform all the data from the old system (COBOL-I and a handful of ISAM files on MVS) to the new (Oracle-7 with over 400 tables on Unix). It was all we could do to keep from rolling on the floor laughing right in front of them. Six months later we finished our task of converting all DMV data for over 14 million people.

      Yes. It will always take far longer than you can imagine. In my own experience, Scotty’s strategy of multiplying all time estimates by 4 has gotten me in the ballpark. I’ve found that I can only imagine about 25% of the task at the outset. I’ve also found other people’s projects to take 3-4 times as long as the original estimate. In projects where management has forced adherence to the original timetable, the results are always either a disaster or a lie (purposely putting the buggy system into production and not counting the additional weeks or months of debugging as development).

  36. This is a great article on why COBOL isn’t the problem. My main concern with articles like this is that they feel aimed at programmers and engineers, rather than a more general educated audience. I saw the same things occurring back when this article was written, and decided that it was a good time to try to educate as many people as I could, especially non-programmers, on why COBOL in particular (and programming languages in general) aren’t the root cause of large system failures like this. It basically ended up being a high-level description of technical debt.

    I didn’t write it alone, but got a lot of input from other people at my company. I published it on my company’s tech blog, but I insisted it only be for public outreach; the only solutions we offer are free advice. You can see it at

    https://www.lucidchart.com/techblog/2020/11/13/why-cobol-isnt-the-problem/

  37. Thank you for covering this; I recall ‘cobol’ from my research for article on Y2K titled ‘Will Y2K Cause Nuclear Catastrophe’ a title the Oregon Peaceworker chose (bit alarmist, imo, but also a bit accurate, based on my research.) I simply wanted to push the Oregonian in my state to cover this issue as after 2 months of digging I went from ‘this is nothing’ to ‘this is a really serious problem.’ My article was the last article I’ve ever written as it nearly gave me nervous breakdown to learn we Have No Emergency Plan really to speak of (so the recent fear of FEMA is quite a Turn for me after seeing how little they’re actually funded. I think Covid changed all that.) I know the world’s activists were successful in getting most of the factories to shut down–we called it the cleanest day on planet earth since the Industrial Revolution. One huge problem was getting to some of the devices that might malfunction, such as in cable lines that cross the sea. I know we traded teams w/ Russia & had their people onsite at nuclear stations so they could see we weren’t attacking them if the code malfunctioned (I believe Norad ran into problems.) The saddest part for me was the only institutions that protected themselves were the Banks, which frankly wouldn’t cause a toxic nuclear mess if they went amiss, but they got the money–I’m wondering now if this wasn’t all a Disaster Capitalism scheme, which I knew nothing about back then. I just knew MSM wasn’t reporting on this, so I had to and it worked as Oregonian ran cover story on Y2K a couple weeks after my article–probably cuz the headline scared the crap out of everyone, so I guess it worked and my editor was right.

  38. I was (am?) a COBOL analyst-programmer for many years. Currently I program in NetCOBOL (OO) for nonsense of mine. In the vicinity of Y2K I no longer worked in the area of systems (I was a professor of mathematics and physics), but the solution to Y2K was dumber and simpler than all the madness it generated.

Leave a Reply to RenCancel reply

Please be kind and respectful to help make the comments section excellent. (Comment Policy)

This site uses Akismet to reduce spam. Learn how your comment data is processed.