Tracing Business Operations, Not HTTP Calls — Part 2

Part 2 of 2. Part 1: Everything Is a Source covers the small pipeline framework this tracing is built on.

When a user at Tilt says “my bid didn’t go through”, what actually happened? The app read the current user, sent a GraphQL mutation, maybe retried it, maybe refreshed a token, and the backend called two more services. Before this summer I could see pieces of that in different tools, but never the whole thing as one story.

This article is about how I designed and shipped mobile distributed tracing for Tilt’s Android and iOS apps, using OpenTelemetry in a Kotlin Multiplatform module and Grafana on the other end. The short version: the trace root is the business operation, not the HTTP call, and because of the pipeline from Part 1, opting a function into tracing costs one argument:

.buildOnce(ShopperTracedOperation.MakeBid(roomId, auctionId, bidValue))

The longer version includes an alpha SDK, a coroutine context that will not move, and a few bugs that nobody would have found without traces.

What I could not see

Tilt is a live shopping app. Sellers run live auctions, and an auction often lasts only a few seconds. If placing a bid hangs, the bid does not count, and the user does not care whether it was their network, the app or the backend.

When complaints like “the auction froze” came in, diagnosing them was slow and largely guesswork. The app had a small per-request performance tracker that emitted start, end and error events. It was a flat list of timings: no parent and child, no trace ID, no link to anything the backend saw. And by then nothing was consuming it anyway.

The deeper problem was the unit of measurement. In the shared module, business operations live in repositories, and a single repository function often combines several queries and subscriptions. Tracing each GraphQL operation on its own would give me accurate numbers for the wrong thing. Tracing a whole app session would give me one giant trace that nobody can investigate.

So the goal became: one trace per business operation. Placing a bid, loading the home feed, sending a phone verification code. Each one should be a single waterfall, from the moment the app starts the work to the last backend service that touched it. And it should carry the IDs people actually ask about: the user, the room, the auction.

I also set myself one constraint early on. Other developers would add tracing to their own features, and if they had to think about where to start a span or how to nest it, the result would be a mess. They needed an abstraction that was more intuitive than spans.

Why OpenTelemetry and Grafana

I did not run a vendor bake-off, and I will not pretend I did. Two facts made the choice for me.

First, the backend already used Grafana, and backend teams were introducing tracing at the same time. A mobile trace is only half as useful if it stops at the app’s edge. With OpenTelemetry and the W3C traceparent header, a span started on the phone and the spans the backend creates for the same request land in one trace. No other option gave me that end-to-end view with the stack we already ran.

Second, OpenTelemetry now has a Kotlin Multiplatform SDK, opentelemetry-kotlin, which came to the project as a donation earlier this year. One implementation in commonMain covers both platforms. It is alpha, and I accepted that deliberately. As you will see, alpha has a price.

On the Grafana side, traces go to Tempo and are explored with TraceQL. A self-hosted stack is also far cheaper than sending every span to a SaaS vendor, which mattered because I wanted to store everything at first, without sampling.

For local work, one container is enough: grafana/otel-lgtm bundles a collector, Tempo, Loki, Prometheus-compatible storage and Grafana. I ran it on my laptop and pointed the Android emulator and the iOS simulator at it. Being able to tap a button and see the span in Grafana seconds later made the design work much faster.

Architecture at a glance

The root span opens where the pipeline ends (highlighted). Each HTTP attempt carries traceparent, so the backend’s spans join the same trace, and both sides reach Tempo through the same collector.

The trace root is the business operation

When I started, spans and traces were new to me. I had a hunch that the pipeline framework could carry tracing, but I almost talked myself out of it. Combining the two turned out to be the best decision in the whole project.

Look at what the pipeline already gives you. Every business operation ends in build() or buildOnce(), which is exactly where the operation begins and ends. Every network call goes through source(apolloCall), which is exactly where a GraphQL operation begins and ends. The seams I needed were already there. So the API became: name the operation at the terminal, and the framework does the rest.

Step one is a typed operation in a catalogue:

class MakeBid(roomId: String, auctionId: String, bidValue: Int) : ShopperTracedOperation(
    name = "makeBid",
    attributes = mapOf(
        TraceAttributes.ROOM_ID to roomId,
        TraceAttributes.AUCTION_ID to auctionId,
        TraceAttributes.BID_VALUE to bidValue,
    ),
)

Step two is passing it to the terminal:

suspend fun makeABid(...): TiltResult<AuctionBidTiming> = source(currentUserStore.currentUser)
    .transformToSource { user -> source(apolloClient.mutation(AuctionMakeABidMutation(...))) }
    .transform { data -> /* map timing */ }
    .buildOnce(ShopperTracedOperation.MakeBid(roomId, auctionId, bidValue))

That is the whole developer-facing API. I rejected the obvious alternative, buildOnce(“makeBid”, mapOf(“room_id” to …)), because sooner or later someone writes room.id in one place and room_id in another, and every query in Grafana breaks quietly. Attribute keys live in one object of constants, and their types are kept, so TraceQL can compare numbers as numbers.

Under that one argument, a trace looks like this:

biz op: makeBid                    root, opened by buildOnce
  operation: AuctionMakeABid       one span per pipeline source
    attempt: AuctionMakeABid       one span per HTTP request, carries traceparent
      hasura /v1/graphql           backend spans, same trace

The attempt layer came from a mid-project pivot. A span above the retry logic cannot see individual retries, so I added an Apollo interceptor below retry and auth that opens one span per physical request and injects traceparent into each one. A retried bid now shows up as two attempts under one operation, and the backend sees two distinct requests.

The pipeline gave me one more thing for free: error capture. Every operator turns exceptions into a typed TiltResult.failure, so the terminal sees every failure, whether it came from the network, a converter or a checkNotNull in a transform. The root span records it once, with the error type and code.

Later I added opt-in spans for steps that are not network calls but still matter, like a local data source or a heavy transform. My first version added a separate traced operator. I threw it away in review, because it made the existing operators second-class the moment tracing was on. The rule I settled on: tracing must be invisible in the operators. An existing transform takes an optional name. Unnamed, it behaves exactly as before. Named, it emits a span. Nothing else in the pipeline changes.

Streams are different

One-shot operations map neatly onto a trace: it starts, it ends. A live room is not one-shot. It is a set of subscriptions that can run for an hour.

The tempting answer is one long span per subscription. That is the session-trace anti-pattern again, just one level down: a root that never closes, a waterfall nobody can read. I chose bounded roots instead. build(operation) opens a root when the stream is collected and closes it at the first real combined state, the moment the screen has data to show. What I trace for streams is setup: how long it takes from opening the room to seeing the auctions.

Why not also trace each emission? I tried to design it and kept hitting the same wall. A Flow’s coroutine context is fixed when collection starts, and the conversion runs upstream of the terminal. Starting a fresh trace per emission would need machinery that knows which phase the stream is in, everywhere in the chain. And the signal would be weak. Many subscriptions emit once and then stay quiet for minutes, which is perfectly normal. I did not want to flood Tempo with traces that mostly say “nothing happened”. So per-emission tracing is deferred on purpose, and I wrote down why.

The pipeline’s shape forced a few other decisions:

Context crosses into Apollo explicitly. My first version passed the parent span through the coroutine context with withContext inside a flow {} builder. The unit tests passed. On a device it crashed with “Flow invariant is violated”, because a flow may not change its context from inside. The parent span now travels in Apollo’s ExecutionContext, which is designed for exactly that.
Placeholders do not count. startWith(emptyList()) unblocks the UI before data arrives. If it closed the setup root, every stream would report a perfect setup time. startWith always takes a value, never a source, so I mark it as synthetic and the root waits for real data.
WebSocket connections are their own traces. A socket connection has no business operation that owns it, so ws: connect is a standalone root, named after the client it belongs to. To join backend spans, traceparent rides in the connection_init payload, because the subscription protocol has no header slot per message.

Living with an alpha SDK

The Kotlin Multiplatform OpenTelemetry SDK let me write tracing once for both platforms. It also made me a part-time SDK maintainer.

I kept a thin wrapper of my own (Tracer, Span, SpanContext) between the module and the SDK. Not to hide the platform, but to isolate churn: when an alpha API changes, one file changes, not every repository.

That wrapper earned its keep. Three SDK defects reached me in practice, and each one was worked around in the module’s own code:

Exporting while offline crashed the app. A transport failure escaped the exporter. A small wrapper around the HTTP engine now turns it into an ordinary failed export.
A non-protobuf error body crashed the exporter. When an error response came back with a body that was not protobuf, decoding it threw. The same wrapper now empties non-2xx bodies before the SDK reads them.
The batch span processor was not thread-safe. On iOS it caused a native crash under load that became one of the app’s most frequent crashes. A teammate diagnosed it and replaced the processor with one built on a coroutine Channel.

Two more gaps shaped the design:

No metrics. The SDK has scaffolding for a Meter but no instruments and no metric exporter. Instead of faking metrics with extra spans, I left that to the backend: rate, errors and duration can be derived from spans in the OpenTelemetry Collector with the spanmetrics connector. Until that is in place, dashboards use TraceQL metrics in Tempo, which only cover a short time window.
Phones have bad clocks. The first end-to-end traces showed backend spans starting visibly after the client span that caused them. The SDK takes timestamps from the device clock, which can drift. The app already had a server-synced clock, so I passed that to the SDK instead.

Would I choose the alpha SDK again? Yes. Every one of these problems had a contained workaround, and the alternative was two separate native implementations that would drift apart. But I would plan for it from the start, wrapper first.

Running it in production

Shipping the code was half the work. The other half was deciding how tracing behaves once real users generate it.

Gate it from the app, read it once. Tracing is switched on per user by a feature flag that lives in the apps, not in the shared module. The module only receives a tracingEnabled lambda and does not know which flag service is behind it. The flag is read lazily on the first span and cached for the process, so it never flips halfway through a trace. Without a configured endpoint, the tracer is a no-op and the SDK is never even created.

Store everything, roll out by cohort. There is no sampling on the device. Sampling on the phone would make it impossible to later capture specific users on the collector side. Volume is controlled by how many users have the flag on, and I raised that gradually, Android first, then iOS.

One mobile service. Both apps report as a single service.name, with the platform as an attribute. I rarely care about Android or iOS alone, because almost everything worth tracing involves the backend too. And service.name is the one attribute Grafana treats as first-class, in the service graph and the default views.

Recovered is not failed. A failed attempt followed by a successful retry produces a child span with an error under a root that succeeded. If alerting looked at any error span, every recovered retry and every token refresh would fire a false alarm. Error rates key on the root span’s status only.

Cancelled is not success either. If the user leaves the screen mid-bid, the operation is cancelled, not failed. Early on, those looked like successful bids in the trace list. Now the root carries an explicit cancelled marker.

A digest beats a pager. My first plan was alerts on slow or hung operations. In practice a single user on a dead network can trigger them, and that is noise. What I actually wanted was to know, every day, how bids are doing. So the alerting idea turned into a daily digest.

Make it self-serve. The project was done when another developer could add tracing to their query or mutation and see the trace in Tempo without asking me. I wrote a tracing guide and an interactive agent skill: it picks the right catalogue, proposes attributes, generates the TracedOperation, compiles once, and ends with ready-to-paste TraceQL queries plus a check that the backend joins the trace. When I announced it, I added one warning: do not add tracing everywhere just because you can. Add it deliberately, where you need to diagnose an operation.

What the traces revealed

Tracing paid for itself before it reached production.

During the spike I put the app through airplane-mode stress tests with a local Grafana, and the waterfalls showed problems I had no way to see before:

A bid could hang for a long time when the network dropped mid-request, and longer still with a retry, because the retry reused a dead pooled connection.
A zombie WebSocket. After a network change, the socket could stay half-open: no error, no data, frozen until the next network change. That became its own fix project.
Phantom bids. A cancelled bid looked like a success in the trace list, which is how the cancelled marker was born.

I made a deliberate call not to fix these before shipping tracing. Only production traces could tell me whether real users actually hit them, and how often. The bid traces later kicked off a separate project to improve the bidding experience and client-side timeouts.

In production, traces changed how investigations go:

Complaints that were not what they seemed. When users reported that bids “did not go up”, the traces for that room showed bids completing quickly and without errors. The real problem was how fast the other clients saw the new price, not the bid itself. Without traces, I would have been tuning the wrong path. One caveat surfaced too: older app versions do not emit traces, so a missing trace is not a missing bid. Check the app version first.
A backend bug, seen from the phone. Setup traces for the live room had a long tail. It came from one subscription that, in a specific case, completed without sending any data. The client saw no error, so nothing retried, and the room waited. I could hand the backend team trace IDs and the exact query to reproduce it.
Stale state behind “late” bids. Correlating bid traces with analytics showed that some bids were rejected because the bidder’s app was one price step behind, not because they were slow. That led to a design where the bid response itself returns the updated auction, so the bidder’s screen is never behind their own bid.

None of these were exotic bugs. They were simply invisible until each step of an operation had a start, an end and a parent.

What I would tell myself in June

Pick the unit before the tool. The most important decision was not OpenTelemetry or Grafana. It was that a trace is a business operation, bounded, with the IDs people ask about. Everything else followed from that.
Hide spans behind your own seams. Developers should name an operation, not place spans. If your codebase has a natural boundary for business operations, hook into it. If it does not, that is worth fixing first.
Tracing must be invisible where it is not asked for. Unnamed steps cost nothing and look exactly like before. Named steps emit a span. No parallel API.
Design for retries and cancellation on day one. Recovered is not failed, cancelled is not success. Getting those wrong turns a dashboard into a noise generator.
Budget for the alpha. A young SDK is still the right call if it removes a whole second implementation. Wrap it, and expect to patch around it.
Observe first, then fix. Shipping tracing before fixing what the spike found told me which problems real users have. Some mattered a lot, and I would not have guessed which ones.

And one lesson that spans both parts of this series. The pipeline framework was built to make data code safer for people who do not live in coroutines. Months later, the same structure turned tracing into one argument per function. Small, boring infrastructure that you own compounds.

If you skipped it, Part 1: Everything Is a Source tells the story of that framework.

Tracing Business Operations, Not HTTP Calls — Part 2 was originally published in ProAndroidDev on Medium, where people are continuing the conversation by highlighting and responding to this story.