Putting Our Product Analytics on Autopilot With a Persistent Goal

A few weeks into the Telos Cloud alpha, we wanted to understand how our early users were using the product.

We saw signups in our database, but nothing more. Did users actually run services? Did they get stuck? Where was activity concentrated? Were people converting from free to paid?

These are all relatively ordinary questions - but the answers depended on product analytics, focused on our specific workflows. We wanted something custom, but had no desire to maintain more software.

We thought this was a natural fit for our own platform, so we wrote that into a Goal and let Telos handle it for us. (If you're new here: Telos is a platform for declarative, goal-oriented software engineering. Read Introducing Telos Cloud for an overview.)

In our last post, we showed that a Goal could build and evolve a multiplayer FPS. That was a demonstration of what the platform could do (we are not, sadly, a game studio). This is something we built on Telos to solve one of our real business problems.

Could we go bespoke?

Of the SaaS offerings, the obvious answer was PostHog. It's excellent for what it does, and it runs on ClickHouse too. But we'd mostly read the data through a coding agent anyway, and we'd still be behind the screen - prompting away.

As a lean team, we didn't have the time to configure an off-the-shelf solution - build funnels, create dashboards, and tune thresholds. We kept coming back to the same question - could we get something simpler - tailored just for us, but without any of the maintenance overhead?

That's exactly the problem we built Telos to solve. You write a Goal and Telos builds the service, runs it, and keeps checking it against the Goal for as long as it lives. So, halfway through setting up an account on PostHog, we decided to dogfood our own platform.

We boiled it down to four questions:

1. What are people doing?
2. What's breaking?
3. How performant is the API?
4. Who is performing actions?

There was no reason to stop at just the software - Telos could own the entire analytics function. A software service, as well as the intelligence layer on top.

Turning a Goal into a live production service

We worked with our coding agents to write out the Goal spec against our high-level questions. We made sure we:

• Described higher-level outcomes. "Meaningful change surfaces before anyone asks."
• Set performance constraints. "Every default view responds within 3 seconds."
• Attached our internal standards as rubrics. We used our internal design system, so it would feel Telos-native.
• Defined a reconciliation interval. We set interval: 6h in the spec, so Telos re-checks the running service four times a day.

A tailored fit

Half a day after that first apply, we had a running service built for us: event ingestion, identity resolution, ClickHouse storage, a generated interface, and a production deployment with copy-paste client snippets.

All of that from one telos apply.

We wired the schemaless API into our frontend and backend and started shipping the events we cared about.

Here's what the result looked like:

The service composes its own pages from the evidence, and adds, removes, and reorganizes them as the data changes. package_pulled is an event only our product emits, and it got its own panel because the evidence warranted one.

It's not a one-size-fits-all SaaS dashboard with our events poured in. The pages exist because our events do. This was one of the outcomes we were most excited about - we didn't have to learn a new product. Instead, it learned ours.

Within a week of ingestion, the service had already surfaced two things that were immediately impactful:

• API p95 had climbed to 1,033 ms, 691% above its seven-day daily mean, with the query and where to click.
• 48 database is locked exceptions against a baseline of 35. This was a real bug in our control plane code we hadn't noticed (...and probably wouldn't have).

The system healed itself

The real test of our system was whether it could keep the software alive continuously.

For four weeks, the service just ran. Then, the day after our open beta, the weather changed. The database crossed a million live production events and the three-second view SLA we had written into the Goal began to fail.

Over the course of this period - nothing had suddenly broken. The store had grown organically with real usage, and queries that were fine in week one weren't fine after week four.

We didn't find out until the next morning. By then Telos had spent the whole night on it, finding and fixing real regressions.

All without human intervention.

Here are the highlights:

Hands-free mode

Software is a living thing. While you might ship code that is true to requirements on day 0, the world moves on: data grows, usage patterns shift, and requirements evolve. Continuously reconciling against this drift is in fact where most of the cost of software lies!

Robert Glass put maintenance at 60 percent of the lifetime cost of software two decades ago, and nothing since has made it cheaper.

Usually, this wear and tear accumulates until a real incident occurs, where we would have had to jump in - open an agent session, point it at the repository, and ask it to investigate. With Telos, we were able to run the system on autopilot - like a self-driving car correcting drift before the rider even feels it.

We've all been on the human side of nights like this one: get paged in the middle of the night, painstakingly comb through query logs, and deliriously push up a fix. This time we slept through all of it, and only noticed it during a routine morning viewing of the logs. We'd like a lot more mornings like that.

Try it out

With this new way of working, writing the Goal was still the hard part, and we think deciding what we actually want should stay with human teams. For everything else, we were happy to take it off our plate.

Telos Cloud is in open beta. Try it in the cloud, or run the harness locally (source code).

If you have an internal service you've been putting off building, or an off-the-shelf SaaS product you want more out of, give this to your coding agent to get started:

Questions, bug reports, and strange Goals are welcome at [email protected].