In this article
Running a model yourself can be a sensible choice. So can using a hosted API. The business decision depends on the workload, rights, operating capability and customer requirements. A downloadable model does not remove operating work. A hosted service does not remove product responsibility.
The phrase “we own the model” often hides several different claims. The company may possess a copy of weights, hold a licence to use them, have trained an adapter or control the deployment infrastructure. Those arrangements are not identical, and none should be inferred from a convenient label.
Check the rights attached to the actual artefact
Hugging Face’s licence documentation illustrates that repositories can identify different applicable licences. Read the actual licence for the specific model and version, including any additional conditions. Access to weights is not proof of ownership of the underlying training data or unrestricted permission for every commercial use.
Ask an appropriate legal owner to resolve unclear terms before the product makes commitments. Keep a record of the artefact, version, source and approved use. This is an operating discipline, not a claim that every open-weight model has the same legal position.
Write down what self-hosting adds to the team’s work
Someone must prepare the runtime, manage capacity, monitor failures, control access, patch software and test changes. Someone must also decide what happens when the system cannot serve a request within the promised time. These duties exist even if the first demo ran successfully on a spare machine.
The NVIDIA NeMo Framework documentation offers one route into model training and customisation tooling. It is useful technical material to study. Adopting a framework does not by itself create the staffing, service ownership or operational readiness your particular product needs.
For a small team, the opportunity cost matters. An engineer maintaining inference infrastructure may be doing valuable work, but that time cannot also be spent resolving a customer’s integration problem. Make the tradeoff explicit rather than counting the model download as the whole implementation.
Compare the whole operating arrangement
| Decision | Self-managed deployment | Hosted API |
|---|---|---|
| Capacity | Plan hardware, queues and headroom | Plan quotas, rate limits and supplier availability |
| Changes | Own runtime and model upgrade work | Track provider changes and retest the application |
| Data handling | Control the deployment while securing it yourself | Verify the service’s terms, settings and processing path |
| Cost | Include infrastructure, utilisation and engineering time | Include calls, retries, support and integration work |
| Failures | Provide an operating owner and recovery plan | Provide an application owner and supplier fallback plan |
This is my comparison framework, not a claim that every supplier or hosting setup behaves identically. Some organisations combine managed infrastructure, open weights and external APIs. The important boundary is who is responsible for each part.
Calculate cost at realistic load
A busy production service and an occasional internal assistant can have very different economics. Hardware reserved for a peak may sit idle at other times. Sharing capacity can improve utilisation while introducing queueing and operational constraints. An API bill changes with actual use, but repeated calls and long requests can also make it grow.
Build a scenario from your own workload. Count requests, input and output size, retries, review effort and the expected busy period. Keep assumptions visible and test them. Do not compare a theoretical fully utilised server with an API workload that includes every real-world inefficiency.
The comparable unit is an accepted outcome at the required quality and service level. A cheaper generation that causes more correction may be more expensive overall. Use the business evaluation scorecard to keep quality in the calculation.
Test independence rather than announcing it
Self-hosting may change which suppliers you depend on rather than eliminate dependencies. Hardware, cloud capacity, libraries, model licences and specialist knowledge still matter. Ask what happens if one of those changes or the person who understands the setup leaves.
A hosted API has its own dependency map. Model versions and features can be retired; limits can affect a rollout. The model-deprecation guide explains why a tested replacement path matters. Treat both arrangements as systems to operate, not identities to defend.
Use a bounded proof before a permanent commitment
Choose a representative workflow and compare feasible setups against the same acceptance criteria. Measure response time under realistic concurrency, failure recovery, operator effort and total cost. Include the awkward requests and the day when demand exceeds the comfortable average.
Record what would make you change course. A workload may grow enough to justify a different deployment, or a customer requirement may narrow the options. An explicit review trigger is more useful than a promise that the architecture will never change.
My role is to connect that evidence with product priorities, customer needs and the people maintaining the service. If your team is debating “open versus closed” without a shared workload, start there. A clear task and an accountable owner will make the next conversation much more useful. The server in the executive chair will still need someone on call.
Questions people ask
Are open-weight models free to operate?
Access to weights does not remove infrastructure, engineering, security, evaluation and support costs. The licence and permitted uses also depend on the specific model.
Is self-hosting always cheaper than an AI API?
No. Compare realistic workload, utilisation, staffing, service requirements and cost per accepted outcome. Either route can be appropriate under different conditions.
Does downloading model weights mean owning the training data?
No. Possession of weights, rights under a model licence, control of deployment and rights to underlying training data are separate questions.
Something here sounds like your workplace?
Define the use case, connect approved systems and test a bounded workflow before expanding it.