How to Use Engineering Metrics, Uber overcomes Flaky Tests with this strategy
Newsletter - Issue #08
Let’s dive into the best handpicked stories, starting with ⤵️
CTO Diaries | Business Needs & Technical Roadmap - A hide-and-seek game
One of the finest CTO coaches & ex-CTO at eBay, Stephan Schmidt in this guest post, highlights his experience, detailing his approach to maintaining growth and innovation in a dynamic tech organization in our CTO Diaries:
Deeply understand the Business vision. Engage with key stakeholders and actively participate in strategic meetings to gain valuable insights.
Translate the vision. This requires strategic thinking and technical acumen to ensure that technical initiatives directly support business objectives.
Alignment. Foster a culture of cross-functional teamwork and keep stakeholders regularly updated on the progress of technical initiatives.
Working with limited resources. Use frameworks like JTBD to focus on initiatives with the most significant impact. Stay flexible as business needs are dynamic and can change over time.
Staying Agile. Regularly reassess technical priorities to ensure alignment with business needs and invest in scalable solutions
Metrics. “We use a range of metrics to track the performance of our technical initiatives, from system uptime and response times to production incidents and retention rates.”
Loving our blogs? Share it with your fellow tech leader ❤️
Article of the Week ⭐
📊 How to Use Engineering Metrics
Yes, you’re seeing correctly. That’s
tuning in via Hybrid Hacker as they joined forces this week with Refactoring. Exciting times! We could call 2024 the year of engineering intelligence. With a new wave of KPIs coming in to confuse everyone with acronym’s like DORA, SPACE, DevEx, TT and Flow.Here’s a practical approach to engineering metrics from Luca:
🎯 Goal of metrics — Set clear expectations to avoid chasing your tail.
🚦 Types of metrics — The most important part, we prepared a summary below
🏁 How to get started — Put it into practice gently with your team
👑 Advice from the best — Luca connected with CEOs and Founders of software engineering and intelligence tools, including Typo’s Kovid 👋
It’s genuinely hard to point at many things that all software teams do, which in turn makes it hard to build products to improve them.
That said, I believe we are at a tipping point now, where there is finally more consolidation, many workflows have been figured out, and we have a better idea of what an elite engineering team looks like.
Like all models, metrics are imperfect. They only capture a part of the real thing, Luca emphasises, “They gives you clues, but never the whole picture.“ Many managers get frustrated by this as the temptation to silo the department and measure it with local KPIs is a lucrative illusion for anyone seeking certainty in the business.
Luca helps us clue together the various metrics frameworks by focusing on what he calls the WIP model, that stands for:
❤️ Wellbeing—How the team is doing
⚖️ Investment—What the team is doing
📈 Productivity—How the team is doing it
The full article has more detail on this, but here’s the general overview:
Things moving too slow → Lead time or Cycle time
Struggling with how to allocate your time → Investment balance
Slow/unreliable release process → DORA metrics
Building the wrong things → Feature adoption metrics
Welcome to groCTO. Every week we publish a curated selection of articles, along with market updates and podcasts on engineering culture. As a curious CTO your inbox is easily cluttered with posts that are irrelevant to you.
By subscribing, you empower us to deliver to you matters that have already shown to be valuable to fellow CTOs and technology readers.
It’s a lot of work and the amount of potential data is enormous. Do not take it lightly. Here’s what Luca (and Nicola) recommend as a starting point:
Start with ❤️ wellbeing conversations
Pick one metric and figure out how to measure
Rotate stick with the metric or pick a new one after a while
The conversations with engineering intelligence founders covers a few common topics: connect measurements with business outcomes, combine measurements, and look for trends rather than targets.
Other highlights 👇
Uber overcomes Flaky Tests with this strategy
Flaky and slow tests make my skin crawl.
Uber has a particular problem of a successful company: scale. Namely, their engineering practices have to keep up with their scale. They report running over 10000+ tests on average for each diff. On a typical day they produce 2.5k+ diffs.
Put those two numbers and the word flaky into the calculator and it makes a 🤯 face. Battling flakiness is important because it undermines the confidence the organization has into the test suite even when it’s green.
Here’s the overview of the strategy Uber used to get flakiness down to a minimum in their enormous test suites across multiple monorepos.
Establish Testopedia
Key qualities
FSM: A state machine to grade the quality of a test
FQN: Each test has a global identifier to separate it across all repos
Notifications: When tests are reported as flaky notify the right team
Testopedia sits in our CI infrastructure b/w reporting and consumers, Credits - Uber
Language-agnostic data ingestion
Uber hosts repos of many primary languages so the automation has to handle all the platforms with minimal oversight on the language details. At its core it uses gRPC to pass data from test runs and measures statistical data such as speed, reliability and tags.
Cures for flaky tests
Tests that are specifically marked “critical” will be run on CI jobs regardless of flakiness
Engineers can specifically added tags or keywords in diffs to opt out of that behavior
Other flaky tests, such as integration tests, are run in non-blocking mode as FYI only
In addition, their strategy involves reducing flakiness impact by empowering each team to configure their local CI pipelines on how to handle flaky tests, when to run them, when to skip them and how it impacts deployment builds.
And that’s not all! They are planning to incorporate GenAI to clamp down on flaky tests and categorize failure patterns.
Testopedia Architecture Diagram, Credits - Uber
Sunday Spotlight -Dad Joke🥱
Q: Why did the robot go on a diet?
A: Because it had too many bytes!
We always appreciate a good tech-related Dad joke, so please share any, we would definitely add to our newsletter!
Also, let us all take a moment, to wish all the fathers and father figures among us a very Happy Father’s Day. Your dedication both at work and at home is truly admirable. Enjoy this special day with your loved ones.
The Role of Engineering in Product Model Transformation
from helps you reassert your mental model on how Product Engineering, DevOps and Lean principles combined help you shape an organization that consistenly serves its customers at a high standard.Software engineers don’t work for the Product team; they are part of the Product team.
He stipulates that in order to build products, traditional organizations need to go from a mindset of projects and work optimizations to a Product Operating Model. The Product Operating Model is the brain child of the Silicon Valley Product Group which you may recognize by its Founder, Product Leader and Author—Marty Cagan.
Mirek highlights three phases of such a transformation:
Changing the way things are built,
Changing the way problems are solved,
Changing the way decisions are made about which problems to solve.
The first article in a series of three covers the engineering culture aspects.
Changing the Way Things Are Built
Frequent, small releases
Product instrumentation
Monitoring
Staged rollout & early validation
No surprise—this is a great overlap with pure Agile principles, Continuous Delivery supported by DORA metrics. Staged roll-out is a popular approach when developing through a trunk-based, PR-less workflow using feature flags. This separates deployments from releases.
P.S.: Did you know there is a CNCF project for standardising feature flags—OpenFeature?
That’s for Today!
Whether you're hustling with your side projects, catching up with the latest technologies, or simply relaxing & recharging, we hope you have a lovely day.
See you next week, Ciao 👋
Credits 🙏
Curators- Diligently curated by our community members Denis & Kovid
Sponsors- This newsletter is sponsored by Typo AI - Ship reliable software faster.
1) Subscribe to the newsletter — If you aren’t already, consider becoming a groCTO subscriber.
2) Share amongst fellow Engineering Leaders and CTO’s! Your referral empowers & builds our groCTO community.
I loved reading the Uber overcomes Flaky Tests strategy! Keep the newsletter ongoing. Thank you!