IntermediateLLMOps & AI Observability project · about 8 hours

Tracing and cost tracking for an LLM app

Instrument an LLM app with tracing, token counts and cost per request.

The real-world brief

A SaaS company has no idea why its AI bill doubled.

What you will build

  • Traces for every request
  • Token and cost per user
  • Latency breakdown
  • A cost dashboard

Skills you will practise

TracingCost trackingLatency analysisDashboards

Tools

Langfuse or Arize PhoenixOpenTelemetryPythonAn LLM API

Project tasks

The detailed steps, checklists and trainer review open in the SOTT student portal for enrolled students.

1Instrument Steps in portal
2Cost Steps in portal
3Latency Steps in portal
4Dashboard Steps in portal

How companies use LLMOps & AI Observability

Insurance: Faster answers from policy documents. Agents searched long PDFs to answer customer questions. A retrieval-augmented assistant answers with citations to the exact clause. Handling time fell and answers became consistent.

Retail: Product descriptions at scale. Writing descriptions for thousands of new products took weeks. A generative AI pipeline drafts descriptions from product data for a human to approve. New products go live in days, with editors reviewing every text.

IT services: Helpdesk triage. Engineers spent hours sorting and routing tickets. An AI model classifies, summarises and routes tickets, with humans handling exceptions. Tickets reach the right team faster.

Questions

Can I add the Tracing and cost tracking for an LLM app project to my resume?

Yes. Once your trainer approves it, list it under Projects with the tools you used, one line on the problem it solves and a link to your repository and demo video.

How long does this project take?

About 8 hours of hands-on work, and it fits best after week 2 of the LLMOps & AI Observability course.

Where do I get the step-by-step tasks?

Enrolled students get every task with steps, "done when" checks, the submission form and a trainer review in the SOTT student portal.