Moving a language model workload from paid APIs to self-hosting

Context

The client ran a language model workload through a paid third-party AI API, so every request was billed at the provider's per-use pricing.

Client details are withheld for confidentiality. The facts below are limited to what we can verify.

The challenge

API pricing grows with every request, so the cost of the workload rose with use and sat outside the client's control.

The client wanted to run the model on infrastructure it controls while keeping the output good enough for the job it already did.

What we did

1. 1

   Measure the current workload

   We looked at how the workload used the API and what it cost, so there was a clear baseline to compare against.

2. 2

   Choose an open-weight model

   We tested open-weight models against the client's own tasks and picked one that did the job well enough to replace the API.

3. 3

   Set up model serving

   We deployed the model on infrastructure the client controls, with serving, scaling and monitoring in place.

4. 4

   Move the workload across

   The workload moved from the API to the self-hosted model, and running cost was compared with the earlier API pricing.

Results

• About 60% lower model running cost than API pricing
• The language model now runs on infrastructure the client controls
• Requests no longer go to a third-party AI API