Five Years In, Backstage Is Just Getting Started
LONDON — While Backstage has only just turned five, it really is the grown-up of the platform engineering world, having set the standard for internal developer portals. Not only in popularity but in open source community impact as well.
Backstage started as an open ecosystem within the audio streaming company Spotify, where developers inner-sourced more than a hundred plugins to extend its capabilities. Over the last five years, it became a large open source project with more than 3,400 adopters, including Airbnb, Booking.com, H&M, HCA Healthcare, LEGO, OVO Energy, Philips, Toyota North America and SeatGeek. More than two million developers outside the founding company now use Backstage.
Last year, the Backstage open framework for building developer portals was one of the Cloud Native Computing Foundation’s (CNCF) top five projects in terms of velocity. In 2023, it had more commits than any other CNCF project. The open source project is already undergoing the security audit needed to join the CNCF’s graduated projects.
The Backstage open source ecosystem now has about 1,600 contributors, with 13,000 developers having earned the Backstage certification. There are more than 230 Backstage plugins, including those integrating with Argo CD, Datadog, GitHub Actions, Grafana, Snyk and Terraform. Spotify Portal is the pre-build internal developer portal Software as a Service (SaaS) offering released last year.
At KubeCon + CloudNativeCon Europe, The New Stack sat down with Tyson Singer, head of platform and technology at Spotify, and Chris Aniszczyk, CTO of the CNCF, for an exclusive reflection on how Backstage grew from an internal dev portal into the open source framework with 89% market share. And, of course, they talked about some exciting announcements about Backstage plugins and the future of platform engineering in the age of AI.
Open Source Paves the Way
Spotify engineering aims to be the first to open source its innovations — like its internal developer portal framework — because otherwise, its developers will eventually be forced to move to other tooling.
The company learned that lesson the hard way with Kubernetes, when Spotify first chose a different vendor for its container orchestration. Later, when Spotify had to replatform to K8s when that orchestrator triumphed in the market, Singer said, the move cost the streaming company $10 million. Open source, he said, is just good economic sense.
Once that shift to Kubernetes was complete, Spotify became an early investor in its own developer experience and developer productivity.
“One of the things that Spotify learned early on, quite a long time ago — before the [term] ‘platform engineering’ was actually coined — was that there are a set of pressures that you end up having that really focuses you on improving the developer experience,” Singer said.
“We were competing with the biggest companies in the world. We were growing very fast, and we needed to really focus on fast feature development,” he continued. “As we shifted away from spending as much time on [those] lower-level infrastructure layers, we shifted up the stack to make those feature developers move much faster.”
As the tech industry rides another economic downturn, he only predicts more emphasis on developer experience. Companies have to do more with less.
“A lot of our tech companies [customers] are looking at: How do I get more efficient with my workforce?” Singer said. “Backstage is — and IDPs in general are — a very, very effective way to do that, to make things simpler [and] more similar.”
One focus of his platform and engineering organization is to figure out how to foster more T-shaped developers: those who dive deep into an area of expertise, but also have a good understanding of other domains.
At Spotify, this translates to backend developers being able to do fast data engineering. But to achieve that, everything has to look the same, across the whole data ecosystem, pipelines and endpoints. That’s done via a software catalog that encourages cross-organizational discoverability and reusability.
“I think as we make more types of capabilities available, we see more and more folks be like, ‘That’s our problem as well,’” Singer said. And then, once you solve for the main developer experience problems, like for backend developers, you can replicate those solutions for other parts of an R&D organization, like the data ecosystem.
Currently in the alpha testing stage for existing Portal customers, this data experience plugin for Backstage will soon be released into the premium Portal offering. This product will:
- Enable organizations to discover, understand and manage business data within Portal.
- Ingest metadata from different data sources, then model it in the Backstage software catalog, providing visibility into the data ecosystem on the same plane as all other software entities.
- Integrate with data warehouse tools BigQuery, Redshift, Snowflake and dbt.
- Integrate with core features of Portal to enable teams to search, catalog and manage datasets in the exact same manner as software.
“Now I can solve all the problems from the chaotic data ecosystem also in there, using the same paradigms,” Singer said, “using the same things like Soundcheck that allow you to drive quality into your ecosystem, defragment the ecosystem and get to the point that every company needs [for] faster development and features.”
Soundcheck is one of the most popular Backstage plugins that focuses on team health and alignment around DevOps best practices.
Maintainer, Contributor and User, Too
Backstage makes Spotify an open source business, in addition to its core mission of audio streaming.
“One of the challenges of being a maintainer, contributor and user of a product is that you started with something that maybe was a little bit different than what you released,” said Singer.
This “diamond problem,” as he calls it, emerges when in-house and open source projects diverge. As the open source project starts to dominate, the in-house team works to bring them back together. To be more cost-effective, the Spotify team has been migrating and refactoring its own version of Backstage to be more like the open source version. The two versions are very near convergence, Singer said.
This business/open source game of Jenga comes down to tool and language choices, too. He described Spotify as “primarily a Google shop,” which means his platform team would normally build for data products like BigQuery and Dataflow. However, because Snowflake is also very popular, even though Spotify doesn’t use it, the data experience plugin had to be compatible with it.
“We developed the plugin so that you could integrate with enough data sources that it wasn’t too hard for us to just make a few tweaks,” Singer said.
“That’s a fairly common pattern, as we take the plugins that we have internally, which we think are valuable, and do sort of 70% of the work that everybody needs to do. And then there’s like a 30% where we have to customize to cases that we don’t use.”
In the end, he added, these seemingly selfless choices are actually a “positive force for quality,” as these choices help Spotify abstract the problem while reinforcing domain boundaries.
In another curb benefit, PagerDuty is currently a design partner for Portal. Because PagerDuty “has put a lot of work into making sure that their plugin is one of the most valuable plugins out there,” Singer said, this plugin “instruments the Backstage ecosystem, and provides high-quality signal back to Spotify developers.”
Metrics Always Matter
Since an internal developer platform or portal is distinguished by being optional to use, adoption rate is one of the key measures of success. Before looking to external usage, it’s important to look within Spotify.
All of the open source community growth has been achieved while still engaging the Spotify developer teams with Backstage. Spotify’s internal Backstage adoption rate remains at an impressive 96% across all of its research and development workers, not just developers.
“We follow a tremendous number of metrics for our development ecosystem,” Singer said, including the instrumentation of the entire development ecosystem, the integrated development environments and human resources systems within Backstage.
At a time when #RTO — return to office — is trending, Spotify has publicly doubled down on its remote work policy. That decision wasn’t just made on a whim. As Singer said, “We look at: Are our remote developers more or less productive than people who are in our offices? And we really don’t see any difference.”
Being a long-term partner of the DX developer insights platform, Spotify and DX had already partnered and released the open source DX Backstage plugin. In March, it was announced that all Spotify Portal for Backstage customers will now have access to DX’s suite of measurement capabilities natively within Portal.
“They’re really good,” Singer said. “I think this is going to be a really big value add for folks as they try to figure out how to get value out of the Backstage and get [return on investment] and understand where they should focus.
“DX has one thing that others don’t have, which is they have a lot of information around the industry. So I look at their industry data so that you can compare and contrast how you’re doing, as well.”
Part of the mission of platform engineering teams is to prove the business value of developers and, by extension, why it’s important to dedicate funds to developer productivity. Singer said the Portal enterprise team has been talking with large enterprises, including banks, where finance and procurement teams are definitely decision-makers.
“At Spotify, we got signals that a developer portal can 2x your improvement. But when you’re a bank and you have 30,000 developers, like 5%, 2% improvement, those are massive numbers, and that gets them really excited because they have so many developers,” he said. “We see lots of those signals coming in that large development organizations that are orders of magnitude larger than Spotify see the value proposition of moving forward with this.”
Riding the Wave of Platform Engineering
Platform engineering and internal developer portals are the next wave Singer and Aniszczyk witness crashing into the tech industry, making a pivotal change the way the cloud, Kubernetes and now AI have done.
Indeed, while it has a dominant market share, Backstage only being adopted by about 3,400 organizations didn’t sound like that much.
“What ends up happening is, if they do a homegrown version, it’s like a half-assed IDP — maybe they wrote their own service catalog,” Aniszczyk said, but it’s only a matter of time for these companies to adopt Backstage, too.
“In the early days of Kubernetes, people wrote their own little schedulers on top,” he said. “People will do homegrown things for a while, until there’s a better solution out there, which there is now.”
While Backstage was originally conceived as being for mid-size and large companies addressing complexity in the cloud, Singer said he is seeing a lot of adoption in startups as small as 50 engineers.
“You already see this pressure and the problem sets at smaller companies,” he said. “I think that is a reflection of the way that modern software is built — so componentized, so broken down, moving so fast — that small companies realize the value proposition as well. So I think that bodes well to the wave continuing to build.”
Add to this, in the age of AI, more code is being created than ever, faster.
“Using Cursor AI, all these amazing things, you can create software super fast,” Singer said. “And that’s been one of the challenges at Spotify: we’re really good at creating software. We’ve got 90 million lines of code, or something like that, and so that creates complexity quickly.
“You see these companies. They’re going fast. They’re creating software. And then reuse already becomes a problem. And that becomes the boat anchor that weighs you down eventually.”
In the Face of AI
The emergence of AI agents doesn’t rattle Singer. “The more you create code with these agentic IDEs,” he said, “the more likely it is to, one, get created fast, and two, be low quality.”
This makes platform engineering, IDP guardrails and visibility even more essential. Backstage works to reel the AI rush back in with features like Soundcheck and templating capabilities that should help standardize and maintain quality. With dashboard, observability and monitoring plugins, users can find and flag security violations when things don’t follow the chosen data ecosystem’s policies.
Perhaps Spotify’s biggest announcement at KubeCon Europe is the AiKA plugin for Portal. This “AI knowledge assistant” is an AI chatbot that was first integrated into Spotify’s internal Backstage instance, its IDE, Slack and more. A retrieval-augmented generation (RAG)-based chatbot, AiKA understands the organizational and data ecosystem, including referencing its source material.
Ever the dogfooding organization, Spotify is finding colleagues are eager to use AiKA even outside technical roles, with more than 1,000 employees using the chatbot daily.
AiKA should be released into alpha for existing Portal customers by the end of April. Check back at The New Stack to learn more about AiKA later this month.
