• Justus (unregistered)

    +1

  • MT (unregistered)

    Thanks for this.

  • Shut the fuck up (unregistered)

    This article makes me want to change my handle on the site.

  • (nodebb)

    technical solutions are almost never enough to address technical debt.

    The rest of the TIP-specific part of the article is (rightly) an expansion of this piece. The key thing is that although technical solutions are required to reduce the debt, its existence is due to non-technical problems, and those problems represent the primary obstacle to getting the technical solutions built.

  • Melnorme (unregistered)

    Subtle.

  • Bjørn (unregistered)

    The end part – away from TIP towards the current political situation – was unexpected, yet fully justified. Thank you for that.

  • Cornelius (unregistered)

    I absolutely love this site and reading all the WTFs always makes me smile and cringe at the same time. In my personal opinion, this article was the best technical/non-technical piece I have ever read. The comparison of technical debt to real life is completely true and does not only hold true for the US but also for Germany and all other countries across the globe.

    Keep up the great work and may your technical debt be as little as possible.

  • Steve (unregistered)

    +1 from a heavily technically-indebted UK

  • Just a guy (unregistered)

    Like Cornelius, I’m a long time reader and I love this site and the perspective it gives me on my profession, and I, too, think this is one of the best things that Remy has ever written. Thanks, Remy, for this.

  • Alex (unregistered)

    Excellent post, re-framing social issues for a technical audience.

    I do think, though, that there needs to be a distinction between technical debt and what I've heard called "revealed preference". Technical debt is horrible, and we can all agree that it's horrible; the only disagreement is over how much effort we should put into fixing it. Conversely, when a stakeholder's obstructionism reveals their underlying preference for a more opaque system that empowers them personally, both the problem and its remedies are completely different. You can't treat the triumvirate like you would the rendering/templating engine.

    In the case of the current societal clusterfuck, whilst there is certainly an element of technical debt that enables individual bad actors to run riot*, this feels more like it is revealing the preferences of some key demographics:

    • The preference of well-off whites to pull the ladder up behind them;
    • The preference of police unions to protect their members at all costs;
    • The preference of the average non-black voter to not have to think about this crap.

    If the protests have achieved nothing else, they have at least put that last group on notice that the current system is not a sustainable one, that the underlying fault is generally not with the activists, and that the real way to get protests off our TVs is to start paying off the damn capital.

    • E.g. lack of bodycams and/or lack of enforcement of rules around enabling/disabling them.
  • my name is missing (unregistered)

    The hardest thing about fixing technical debt is the decision to change; this is as true in software as in society. If you can fully commit to change then you can fix anything; if you do it half-ass it rounds down.

  • Scott (unregistered)

    Staying away from the political side, TIP sounds a lot like our main application. Something written in old technology, that wasn't even done right by those standards.

    So, I rewrote most of it. Being an ASP.NET web forms app, I created new pages, to allow for side-by-side testing. Instead of "hey, let's set a bunch of different variables, each of which have side effects," I took a more standard approach--gather all the data, set a DataSource, and have DataBind perform any necessary side effects (for display things).

    Of course, sometimes the old spaghetti code was pretty much unreadable, and we'd find that new produced a different result than old. Also of course, nobody could explain what the calculation should be; just that it was different.

    Anyway, that new stuff has been waiting for over a year to be fully tested...

  • TMMITW (unregistered)

    Well said!

  • Sole Purpose Of Visit (unregistered)

    I can imagine some commenters shying away from the "political" comparison, but I'm proud of you, Remy. Well played.

    Eventually, all "big systems" become so large and entrenched that it doesn't matter what the software, the politics, the legal apparatus, the accounting rules say. They just gum up with technical debt.

    I wish there was a comprehensive way to solve this, other than "the first thing we do, let's kill all the lawyers."

  • Jaloopa (unregistered)

    I was expecting more swivel eyed loons in the comments. Very nice article

  • tbo (unregistered) in reply to Jaloopa

    Same here. I looked at the fact this had 15 comments and thought, "Surely some idiot doesn't want political commentary on his technical site!" But nope. 100% praise so far. How rare is that.

    This belongs in the article list on the left of the main page already.

  • (nodebb)

    It was a good article until you started commenting on things you hadn't thought through very well. The social problems we're encountering recently are iatrogenic illness. Centuries ago, there were actual injustices. In recent decades, however, people have been trained to think that their personal failures are caused by crooked umpires, when in reality, the crooked umpires in the old direction have all been replaced. Now they knew the umpires are still crooked, but in the opposite direction. If the wrong team loses, the league gets pressured to replace the umpires.

    This is the primary cause of the recent social problems we've had. When people fail, they've been trained to blame the system, instead of looking in the mirror.

  • ok (unregistered)

    ok so that was pretty boring and pointless, apparently there was some current politics at the end (I was just skimming at this point)

    sigh can we have back the good old days where there wasn't politics LITERALLY EVERYWHERE

  • (nodebb)

    Thank you.

    There are multiple components to this systemic debt. One is similar to the technical debt in computer-based systems...that is the programming that we all have received by the society in which we were raised. Unfortunately, Max Planck's observation is relevant here: "A new ... truth does not generally triumph by persuading its opponents and getting them to admit their errors, but rather by its opponents gradually dying out and giving way to a new generation that is raised on it." The best the rest of us can do is learn to recognize these programming mistakes in ourselves and in others & actively work to resist this programming. And work to raise the next generation to be better than us.

    Another is the gross inequity in initial conditions for members of society. It is readily apparent that the children of slaves and the children of plantation owners would not have equal outcomes regardless of individual effort. And those inequalities have been compounded by decades of policies and programs which have had unequal effects...social security, GI bill, mortgage insurance, policing, etc. One can debate whether the inequality was intentional or inadvertent, but the impact has been clear.

  • Sole Purpose Of Visit (unregistered) in reply to jinpa

    Iatrogenic?

    I do not think that word means what you think it means...

  • Nathan Spitzer (unregistered)

    +1 - Maybe in the end TIP had one more lesson to give from the grave! If this article can make some of us techies realize we have to get out of our comfort zone and start engaging in <gasp>politics</gasp> and start raising our voice on the issues of the day it will have served a valuable function even after the last server was shut down and last disk degaussed.

    Great article and spot on.

  • Matt B (unregistered)

    My current place has something similar. The application isn't TOO bad, but its still running on old versions; for example using Elasticsarch 2 (current version is 7) & Redis 3 (current version is 6). But the infrastructure??? SO MUCH technical debt. For example its running on VMWare 5.5 (out of support) and Ubuntu 14.04 AND 12.04. We have two datacenter, and things are half assed highly available: HAHA.

    And there is no desire to change. Its so frustrating and so much added toil.

  • I don't usually leave comments (unregistered) in reply to Sole Purpose Of Visit

    I think they were using the word in a "cure is worse than the disease" context, like the damage inflicted by starting political change is not worth it.

    I would have perhaps phrased it like "The social problems we're encountering recently are largely caused by attempts to force change where change isn't needed." Of course it's up to Jinpa if that's an accurate reinterpretation or not. For the record, I had to google "iatrogenic".

  • Tipping Point (unregistered)

    A good article, though I hope it's an exception and we're not going to get US politics making its way into articles here too often. And I wish US politics would stay in the US - over here in the UK we've had people violently confronting the police, damaging monuments and flouting coronavirus distancing guidelines, all based on an event in the US which almost none of those people have any more connection to than I do.

    On the more technical part of it - you can handle tech debt within engineering practices if you keep on top of it. But as soon as the cost of the 'interest' gets high enough that you can't hide it, you need to get management to approve it, and it's very hard to get non-technical people to understand why it's important (even though these people generally are quite happy to get their car serviced and their dentist to do a checkup regularly). So what ends up happening is you don't get that approval, you get told to do more feature work, the time spent fixing bugs goes up because you never paid off the debt, and the debt inevitably increases. This is a positive feedback loop that results in complete project failure fairly quickly if no managers get on board with fixing it.

  • (nodebb) in reply to Alex

    If the protests have achieved nothing else, they have at least put that last group on notice that the current system is not a sustainable one

    They've always known that the public is not happy about the balance of power. That's why there are many layers of defense to ensure that no accountability ever happens. There are so many laws, so many union rules, so much court precedent, so many relationships, that none of the current generation of police or their leadership will feel any real blow back.

    No District Attorney office will pursue police with any great vigor, because they can't win cases without the help of the police in the court room. No financial penalty will have a deterrent effect because the money doesn't come out of the police budget. No real evidence will be unearthed any any but the most blatant cases because of who does the investigating.

    On the other hand, the police have very real power to determine who the leaders of the opposition are, and to make their lives miserable. Even if the microscope is turned up to eleven, we have so many seldom used laws that an angry police department can financially and socially ruin anyone they want to without ever being on the wrong side of the law.

    If anything comes out of this, it's going to be the weakening of some of the above and maybe some real change thirty years from now. The only quick fix is to stand up a new law enforcement infrastructure and decommission the old one.

  • Meir (unregistered) in reply to sibtrag

    And if there were still children of slaves in the USA, that might be true.

    You know, you could, by the same token, say that it "is readily apparent that the children of Eastern European ghetto inhabitants and pogrom victims" aren't going to have equal outcomes with the descendants of those who put them there and inflicted those on them - but you would be wrong, since that hasn't been the case. You could similarly claim that "it is readily apparent that the children of Irish tenants forced out by the Great Famine" aren't going to have equal outcomes with the descendants of the absentee landlords who ruled them - but again, you'd be wrong.

  • Hal (unregistered)

    Remy has either given us a great analog for the present social situation or left us with a page and a half of pointless blather by someone simply framing every problem through the lens of their own expertise. As much respect as I have for Remy based on his other writings and opinions offered, I am tempted to come down on the side this being a insufficiently complex model to describe the social, economic, tribal conditions and politics embedded in the United States after 2 2/3 centuries.

    Everyone wants a quick fix for all this, make the protest over, be better. However I don't think there is simple answer. What is the solution when a project accrues to much technical dept? -> Its rip and replace inst it? The idea of doing "something" to address the vestigial race/tribalism in out society is popular the idea. If your answer is really fundamentally alter the American system well that is going to negative consequences for a lot of folks not just presently in power, but also just folks who are comfortable. Its one thing get people to accept changing the IT system at work and having to surrender a little fiefdom; I suspect there will be a lot less appetite when for example its your own child's future and a tax policy change really could alter for example if you can afford to pay their way thru college or not.

  • Prime Mover (unregistered)

    "The application had a complex sorting/paging system designed to prevent fetching more than one page of data at a time for performance reasons. While it did that, it had no ability to sort in descending order, and adding that functionality would have entailed a full rewrite."

    This is exactly one of the fuckups that I've just walked away from in my last billet. Having spent 3 years trying and finally succeeding in getting the interface to the back end fixed so that we could finally properly implement the same sorting and filtering techniques that we had when running the system manually, some bright f%%%ing spark decided to change the interface once again for trendy techno-fashionable reason (why anybody listened to that a%%%hole I haven't got a clue) and in doing so we lost all the fine control over the sort and filter capabilities.

    But I think I would have been pushed if I hadn't jumped, I'd made so many enemies in the offshore team by (in my position as technical design authority) telling the truth about how sh%t their code was that they stopped passing their sh%t code by me, and eventually restructured it so as to make the entire s/w dev process offshore. Then it got worse. So I jumped.

  • Prime Mover (unregistered)

    "damaging monuments" is possibly the most trivial of all social crimes. Who cares if an ugly statue of a despicable scoundrel ends up in the bottom of a canal? (Oh, apart from blasphemy, flag-burning and public intoxication, those are more trivial.)

  • I took some notes (unregistered)

    Focus on core business.

    List everything not core business. At least you are aware.

    Fix bugs.

    Add features.

    Limit hot fixes.

    Use IDEs and version control instead of inner platforms.

    Build clear rules outside the tech.

    If simple features like sort do not work, remove the complex code, go back to the tutorial.

    Either don't do it, or do it full effort.

    Encourage others. Favor the launch, not who launches.

    Track goals, like reduced costs. Money matters. Code and operations don't.

  • Concerning politics (unregistered) in reply to I took some notes

    Toyota makes cars like US supermarkets stock shelves and checkout. Problem? No changes. Turn on your blinking cashier light. Wait. Problem is fixed whether you are on board or not. No one is fired. People go to work. People shop. Same thing.

    Launch off a bug fix instead of a feature please. Same thing, problem fixed, no one can tell. Much easier to do.

    If people won't change themselves, why even try to change them? If people cannot control themselves, what threat are they in controlling others?

  • Milo (unregistered)

    Keep politics out of my daily tech reading. Thank you.

  • why (unregistered) in reply to Milo

    Two paragraphs of date string concat would be so much better.

    One thing I'm learning as I get older, the "not fight" part of that old adage. Just run away. Better yet don't show up. Don't contend, don't even be seen or capable of comprehension. Before attacking paradise, try to name it let alone find it.

  • Hasseman (unregistered) in reply to Jaime

    Do not talk with ethe police ever : https://www.youtube.com/watch?v=d-7o9xYp7eE

  • Officer Johnny Holzkopf (unregistered)

    As it has been pointed out, it all starts with a decision. But that isn't sufficient: On all levels, you must be willing to accept the consequences of the decision. This will eventually lead to loss of control, loss of privilege, loss of money, and so on. People will not give those up, be it application stakeholders who want to be "the master controller", or police officers who have their own imaginations and concepts of how law should be enforced. The only thing "lower level participants" can do is: Do what "those with power" don't like: drag their activities into the public focus. It's quite typical for criminals that they don't want the public to know what they do, how they do it, how they profit from it, and how all their things work, and especially they don't like to see their names in the news. Personal responsibility (or to be precise: the absence of it) leads to "those with power" becoming what and they are, and staying where they are. For example, in a corporate setting, a paper trail invariantly accumulating their bad decisions could help avoid creating significant technical debt. "Who is responsible for this mess?" could finally be answered: Manager Bob's explicit instruction and his signature. In society, the same concept could be applied by sousveillance: Police misconduct gets documented and published, with a name tag. No "officer 12345" and "We didn't issue such a number, Sir, you must have beaten up yourself." problems. Don't give evildoers a chance to hide. Decision - consequence. The idea is simple. However a working implementation sadly is not. (We have similar problems here in Germany where "cover up" is a game played on all levels of judiciary, executive, and legislature, so all good intentions end up on paper, but in reality it simply doesn't work.)

  • (nodebb)

    (Did not see this in the comments, apologies if I missed it)...

    Technical Debt is caused by deliberate decisions that provide short term value but incur long term cost.

    Bad code is not technical debt. I figure about 80% of what is called technical debt is not going to match the above definition.

  • Tom (unregistered)

    I normally don't comment, but wow. What a great comparison. Thank you!

  • (nodebb) in reply to TheCPUWizard

    Bad code is not technical debt.

    Bad code might be the direct result of short term decisions. Saving money by not training people and by paying far less than the market rate are both decisions that are likely to lead to bad code. Not doing code reviews (well) and/or not investing in tooling that can improve code, too.

    Not getting rid of an under performer because it's hard to have these types of conversations is also an act that creates bad code. Unfortunately, the only beneficiary here is the manager, so it's closer to company theft than company debt.

  • Loko8765 (unregistered) in reply to Prime Mover

    Burning the Stars and Stripes is not illegal (in fact the US Flag Code specifies burning as the preferred method to end its useful life -- OK, it should be done in a dignified way) There have been several attempts to make it a crime, but it's still protected legal speech.

    That's not saying it won't cost you your life, because that, as always, depends on the professionalism of the police at the scene.

  • Loko8765 (unregistered)

    Now Remy... did you reimplement date calculations? Sorry, stupid question. HOW MANY different reimplementations did you have? If it was just one, because the team was tight, then it's just par for the course.

  • (nodebb)

    Gotta say, I didn't see that twist coming.

    Bravo.

  • Tipping Point (unregistered) in reply to TheCPUWizard

    Bad code is not technical debt

    I disagree, anything which should have been done better and is going to require time to improve in future is technical debt. That's true whether it's a design level shortcut, a lack of proper testing or just bad code.

    Bad code is 'bad' because it is more likely to have bugs or other problems (performance issues, maintainability of the codebase, etc) - if it isn't causing you any problems then almost by definition it's not actually bad.

  • Sean Anderson (unregistered)

    Companies build things that look like or mimic thier internal communication patterns.

  • Anon E Mouse (unregistered)

    Technical debt, like social debt, is a cultural problem. A succession of day-to-day decisions that give preference to leaving problems as they are because the known status quo feels safer than the unknown impact of a change. If it isn't obviously broken, it's easier to assume there is nothing in need of fixing. When it becomes obvious that something is really wrong and in need of fixing, it is typically after seriously bad things have happened. The better culture is to be pro-active about addressing those things that will become problems before they do. However, if problems never surface, it's easy to say the remedy was never needed and the cost to implement it was wasted.

  • AnonDBA (unregistered)

    I live this right now. Tech debt is nearly always the result of failures in leadership. Non-technical salespeople in charge of technical decisions, bureaucracy for its own sake, and the good old ‘cost center’ view of IT in general.

    Sadly, these issues are rarely resolved without a near failure of the business. And many don’t even learn at that point...

Leave a comment on “On Systemic Debt”

Log In or post as a guest

Replying to comment #:

Log Out

« Return to Article