Hi HN! I'm Antoine, co-founder and CEO at Okay (YC W20). We are building analytics for engineering teams and we have released our new application on Github Marketplace.
Getting good metrics on engineering work is hard. On the one hand, you can build your own pipeline by ingesting API data from Github, JIRA, your CI tool, etc. Then you have to ETL that data, link it to teams and services, and finally select a dashboarding tool to display metrics. That gives you ultimate flexibility at a high cost of maintenance. On the other hand, you can go with out-of-the-box products but a lot of them focus on very specific metrics, such as DORA Metrics, PR throughput, etc. - not necessarily what you'll be interested in tracking, and you might even disagree with the metrics definitions or ethos of the product.
We've gone through several iterations of Okay and we think we now have the right balance between flexibility and ease-of-use. We solve ingestion + ETL and we give dashboarding features, but we leave users free to define metrics using our no-code query builder (with plain SQL coming soon). We specifically avoid forming opinions on what users should actually be building in terms of dashboards - in that sense, we are closer to an observability tool than to "development analytics" products. We are seeing a range of use-cases, such as incident tracking (with JIRA and PagerDuty data), sprint reviews (JIRA / Github), velocity & DORA (mainly Github + deployment data). We are building on BigQuery, DBT for the ETL and our custom ingestion code for API syncs. Our frontend is using Vue with a mix of D3.js and common charting libraries.
We'd love to get the community feedback on this current product approach! This Github app makes it easy to start on our free plan.
The job market for top engineers is supply-constrained, as demonstrated by the rapid increase in SWE wages in the past 15 years. If you agree that top engineers are the primary input to these companies' success, then it makes sense for them to adopt a "hoarding" behavior.
In other words, these companies have over-hired as a way to prevent competitors from hiring these same engineers. Although this has created a situation where the company has hired past the theoretical "productive" point, it was still a rational behavior.
Now that the tide is turning, the productivity goal becomes relatively more important than the competitive goal. In the long run though, I don't think the job market will fundamentally change - there is still a shortage of top engineering talent in the US.
80% of the value of OKRs is in the discussions they create, not in the actual list of OKRs that is produced in the process. The more an organization focuses on the technical aspects of producing a perfect OKR list or on "grading" the OKRs, the less value they will have.
In particular, the "KR" part - quantifiable outcomes for the goals - usually helps in clarifying vague projects or ideas that may otherwise harm the team's focus.
It's similar to the famous Warren Buffett's advice on identifying your priorities: pick 20 ideas you have and identify the top 5 goals that you absolutely want to get done; then, throw away the other 15 goals and make sure that you never get tempted to work on them.
Yeah, I agree with this. It's like writing a summary study sheet before an exam...the actual sheet is not that useful, but the work that went into it is. I've found OKRs to be pretty good for clarifying focus areas, both in my org and for personal development. Both the Os and the KRs will frequently change, but good managers/orgs don't really care about granular tracking
It's great to see deeper pieces about developer productivity - in this case starting from first principles and the actual experience of developers. The notion of feedback loops has also been explored in e.g. https://martinfowler.com/articles/developer-effectiveness.ht...
The bad reputation of developer productivity metrics comes from the misguided assumption that developers should be measured. The better approach is to treat developers as customers of the management team / engineering enablement team /etc. In that sense, developer productivity is actually a measurement of management effectiveness / organizational health.
Once you build your developer productivity approach on this basis, what needs to be measured becomes much clearer - for example: interruptions caused by meetings, performance of local tooling like local builds, latency of CI, number of on-call pages triggering outside business hours, etc.
The right set of metrics depends on your team and can be sourced from surveys and just talking to people about what's painful on a daily basis. As a quick and dirty solution, I'd even recommend piping Github webhooks and other events to a product analytics tool like Amplitude or Mixpanel. You'd be surprised how fast you can understand things like build or CI latency by using a classic funnel visualization.
A lot of great engineering teams are migrating to this approach, especially when they have a dedicated platform / enablement / productivity engineering team.
I think it's a little bit more complicated than that.
What you are describing is "treat developer productivity as a supply side problem." developers continually demand resources to improve their productivity.
However, there's two issues:
#1: Developers don't necessarily know how to be productive developers.
#2: Developers might not be motivated to improve their productivity.
Hence, it's not necessarily an efficient market.
I find that you need to control for market inefficiencies by:
- Control for #2 by having a "tech lead" or senior engineer be directly responsible for their developer's performance. Whether that's a 2 parent leadership team (people manager + tech leader) or otherwise, developers must have direct oversight of their personal productivity.
- Have an appropriate incentives in place for developers to improve. A couple of places (IBM) actually have an excellent infrastructure for providing productivity. But no incentive.
> Control for #2 by having a "tech lead" or senior engineer be directly responsible for their developer's performance. Whether that's a 2 parent leadership team (people manager + tech leader) or otherwise, developers must have direct oversight of their personal productivity.
Tech leads in most engineering firms don't have that kind of control. That's why OP mentioned "manager". You can't make someone responsible for stuff they don't control. Even if a manager "tasks" a lead with this they usually don't in fact own it. The paradigm you're asking for requires a people manager to be leveled the same as their tech lead so that they share responsibility for outcomes. Most places don't work that way.
> Have an appropriate incentives in place for developers to improve. A couple of places (IBM) actually have an excellent infrastructure for providing productivity. But no incentive.
OPs point is that developers are a reflection of their environment more than they are of their own knowledge. There's a DevOps study from DORA that covers this I think, as well.
> Tech leads in most engineering firms don't have that kind of control
That's a mistake by most engineering firms.
Direct technical leadership is extremely important. Weekly direct feedback and performance evaluation on technical tasks. By a technical leader who is directly responsible for getting this person improved.
I think, the vast majority of the status quo re: "managers" in the software industry is absurd rubbish.
I use the army/medicine analogy.
In the military/medicine, a junior doctor/2nd lieutenant has every aspect of their work overseen and audited by an expert who can do their job better than they can.
In the software industry we often skip this step.
Most software is cut price rubbish. Engineers who enjoy leadership and mentoring other engineers cost $$$. It's not cost effective, so bad management ensues.
> OPs point is that developers are a reflection of their environment
Definitely agree. I'd say my points are additive, they're not an alternative to OP's.
Thank you! I've read through every comment here, and
"#1: Developers don't necessarily know how to be productive developers.
#2: Developers might not be motivated to improve their productivity.
Hence, it's not necessarily an efficient market."
Has been my experience lately in thinking about this problem. Those who are productive generally know how to be productive and _want_ to improve their productivity. Many have the motivation but don't know how, and many others are hampered by external factors that limit motivation, or don't care.
>Once you build your developer productivity approach on this basis, what needs to be measured becomes much clearer - for example: interruptions caused by meetings, performance of local tooling like local builds, latency of CI, number of on-call pages triggering outside business hours, etc.
But those metrics don't really help management decide how many hours developers must be required to sit at their desks per day, how much unpaid overtime to demand from them, or how many years to wait before giving them a tiny raise! /s
Yes, engagement has a direct impact on the performance of individuals, and managers are responsible for creating an engaging environment - which shows why individual, team and manager performance are hard to disentangle.
Measuring development performance at the individual level is futile given the collaborative nature of the work and its complexity. Team and manager performance are more tractable.
Team performance is highly influenced by external disruptions, which is why the culture of engineering management since the 1980s has been centered around "removing blockers". Focusing on building a smooth-running operation (good hardware and software, fast code reviews, fast deployments, minimum amount of meetings, etc.) is probably what this article means by DecisionOps.
Re: managers, one formula I've used in the past (inspired by Andy Grove I believe) for managers' performance is a combination of the following: how is their team delivering ? How are they helping neighboring teams deliver as well? Do people want to work for this person (internal transfers + hiring) ? Or do people want to leave this team ?
Currently we only support these ecosystems because most of our early customers are on these platforms. We do have a tracing-like API that you can use to send custom events, but that would require more integration work of course.
Is there somewhere where we can suggest/vote for the next integrations? I can only seem to find a message that you are adding more integrations. We'd love to see Microsoft/Office 365 integration, and GitLab.
The tool looks awesome, exactly the right kind of metrics, without the invasive accounting for every second that others have tried to enforce in the past! Excellent work!
For time spent on local builds, we expose a tracing-like API where you can tag the build id, start and end events, as well as connect it back to the right team. This custom event gets joined with everything else. I'll reach out to you on your email!
