• Debra (unregistered)

    Loved the video. Strong Parks & Recreation vibes.

    Honest question, though : why would you use an external service for processing your application's coverage reports?

  • (author) in reply to Debra

    Honest answer? It was like that when I joined the team. But it's some pre-built automation for annotating the PRs, and for whatever reason, it's wired up to post to slack with every new PR as well. "Visibility" I guess?

  • Simon (unregistered)

    Perfect timing of this today as Google's Europe-west2 goes down and cripples many businesses in the UK

  • 516052 (unregistered)

    He who sacrifices reliability just to save some money deserves neither and shall loose both.

  • (nodebb)

    @Debra: "why would you use an external service for processing your application's coverage reports?"

    Storage with other service functionality (webhooks, bus messaging, etc.)? When some of these types of cloud services "just work", it's easy to assume they will always be available, until they are not. Looking at you Azure Service Bus.

  • randy (unregistered)

    Reminds me of CircleCI's outage last week. It was four hours before I could finally deliver a critical patch... After the first hour, I should have been allowed to just build it on my machine and fake the rest of the pipe.

  • (nodebb)

    I would bet most of the waiting was trying to find a human at Amazon to answer the phone. Because their account status probably said they were fully paid up and everything is running fine. But on the backend, Amazon disabled their account for non-payment because somehow it failed to acknowledge the fact the bill was paid. Of course, the AI customer service tools and other things have you running around in circles and confusion because you can't get a person on the line who can see what's actually happening.

  • A Human (unregistered)

    TRWTF is aws.

  • The culprit (unregistered)

    Coveralls had gone down, and our CI step that talked to them was the culprit.

    I hope you mean "was the culprit for the CI Jobs failing" and not "was the culprit for Coveralls going down"....

  • (nodebb) in reply to Worf

    From reading the post-mortem I don't think that's likely. They basically always had an outstanding balance. There was a billing dispute in addition to some problem with the transactions tab. It didn't show the payments that they had been making for the past 6 months.

  • Dave P (unregistered)

    The bigger WTF: Why does the build have any external dependencies at all. It is vulnerable to not just an external service being down, but also an external service being hacked and silently inserting malware into your product. The build needs to succeed with no network access beyond the build cluster.

    I'll accept an exception for "sign the build with a corporate signing key" or something similar, but that should only add a simple firewall exception, not unfettered outbound access.

  • (nodebb) in reply to Dave P

    Why does the build have any external dependencies at all.

    Because this isn't a build product, but rather reporting (complex!) metadata about your test coverage. If the service got hacked, it doesn't imply that your software gained any malware. It can't send information that way (and typically runs in a different job to the one producing a signed build, because it's downstream of your tests, and you don't ship those).

    As for why have an external dependency? Because their coverage reporting UI is pretty good, and they're easy to integrate with. The default reports you get from other tools tend to be not very informative at all. It's not just that you see that you've got 70% coverage, but that you see exactly what the 30% of uncovered lines are and can figure out if you care about fiddling around to improve the coverage. So yes, you can live without it, but it's nice to have, and a PITA to reimplement yourself.

    A failure to upload the coverage report shouldn't fail builds. That step should be marked as non-critical (and requiring particular coverage levels to accept a PR is open to gaming with all sorts of stupid hackery that reduces real reliability).

  • Strahd Ivarius (unregistered) in reply to Simon

    Did the Telegraph managed to blame the current government?

Leave a comment on “The Outage”

Log In or post as a guest

Replying to comment #:

« Return to Article