The first question in self-hosted AI should not be whether a local GPU is cheaper.
It should be: Which controls does this workload require, and which obligations am I prepared to own?
That question can lead to self-hosted infrastructure. It can also lead to managed inference or a hybrid design. The answer depends on the workload, not the ideology around the hardware.
Self-hosting changes dependencies; it does not eliminate them. Removing an external inference API can give the operator more control over a data path and capacity. It also transfers responsibility for patching, uptime, power, cooling, observability, replacement capacity, and recovery.
Control does not disappear when the API call disappears. It moves.
What one local GPU build actually changes
A useful concrete example comes from an a16z GPU-server build. The authors give three reasons for running their system locally: no calls to an external inference service, no inference data sent through that external service, and no provider usage throttles.
Those are real changes to the workload boundary. They are not proof that the resulting system is cheaper, more secure, more private, more reliable, or easier to operate.
The distinction matters because infrastructure discussions often jump from a narrow architectural fact to a universal outcome:
- Keeping an inference path off an external service is not a complete privacy guarantee.
- Removing a provider quota does not create unlimited capacity.
- Owning the hardware does not prove that identity, access, patching, logging, or recovery controls work.
- Avoiding one provider dependency does not make the system independent of facilities, components, software, or operators.
Even the hardware detail needs a boundary. The a16z authors say they built and tested the servers with RTX 4090 GPUs. Their discussion of RTX 5090 compatibility was not a tested result.
The build is useful evidence for what self-hosting can remove from one architecture. It is not a verdict on where every AI workload should run.
You are choosing who owns the failure modes
Managed infrastructure and self-hosted infrastructure both contain dependencies. They assign them differently.
In a managed service, the provider may own physical facilities, underlying hardware, parts of the network, platform availability, and some maintenance. The customer still owns obligations that vary by service: identity, configuration, data handling, application behavior, monitoring, and incident decisions. AWS describes this explicitly in its shared-responsibility model, although the exact division is provider- and service-specific.
In a self-hosted system, the operator can take direct authority over more of the stack. That authority arrives with work. Capacity planning, component failure, software compatibility, vulnerability response, thermal management, power, observability, backups, and recovery no longer sit behind one inference endpoint.
So the useful comparison is not:
Do I want control or dependency?
It is:
Which controls must I hold directly, which obligations can I carry continuously, and which responsibilities should I assign to a provider?
That is a workload decision.
The Control Boundary Test
Use five checks to make that decision explicit: Data Path, Capacity, Security Authority, Reliability, and Operations.
This is an operator framework, not an industry standard. Its purpose is to prevent a team from choosing infrastructure before it has named the required controls and accepted the resulting duties.
1. Data Path
Decision question: Must prompts, retrieved context, outputs, or regulated data avoid an external inference-service path? What architecture and controls demonstrate that requirement?
Start with the actual data classes and routes. “Sensitive” is too broad to select an architecture by itself.
Name what enters the model, where retrieval happens, what is logged, what is retained, which systems can inspect the traffic, and whether any fallback can send the request somewhere else. If policy prohibits an external model-service path, self-hosted or tightly controlled private infrastructure may deserve evaluation.
But physical placement is only one part of the evidence. A local endpoint can still expose data through weak access controls, logs, backups, dependencies, administrative access, or a badly designed network.
For each candidate boundary, record:
- the data classes allowed to cross it;
- the controls that enforce the route;
- evidence that those controls work;
- the fallback when the route is unavailable; and
- the condition that would force reconsideration.
2. Capacity
Decision question: Does the workload require operator-controlled scheduling or insulation from shared-service quotas? Who provisions peak capacity, and what happens when it is exhausted?
Removing a provider throttle does not remove a capacity limit. It changes the owner of that limit.
With self-hosted inference, the limit may be GPU memory, queue depth, thermal capacity, power, hardware availability, or the time required to add another machine. The operator decides how work is prioritized and also absorbs the peak.
Managed inference can involve shared quotas, but that is not the only managed option. Microsoft, for example, documents provisioned deployments with dedicated model-processing throughput. That does not make the option right for a particular workload; it shows why “cloud means shared capacity” is too simple.
The evidence should be workload-specific: arrival rate, concurrency, context size, latency target, model choice, batch behavior, peak duration, and failure behavior. Without that evidence, neither local nor managed capacity has won.
3. Security Authority
Decision question: Who owns identity, authorization, segmentation, logging, patching, vulnerability response, and incident handling? What evidence verifies those controls?
Local does not mean trusted. NIST’s Zero Trust Architecture explicitly rejects granting implicit trust solely because an asset is on a local network or owned by the enterprise.
That principle cuts both ways. A managed endpoint is not automatically unsafe, and an owned server is not automatically secure. The decision depends on who can implement and verify the required controls.
A self-hosted boundary may be appropriate when the organization must directly control keys, identities, network segmentation, update timing, logging, or incident response. It is a poor trade if those controls exist only as intentions and no one owns their operation.
Managed inference is also not one uniform data policy. Microsoft states that data covered by its Azure data, privacy, and security documentation is not made available to OpenAI and is not used to train foundation models without customer permission. That is a provider-specific example, not a claim about every service. The team still has to inspect the exact contract, configuration, retention behavior, geography, and controls for the workload it intends to run.
The boundary should follow demonstrable security authority, not assumptions about location.
4. Reliability
Decision question: What availability and recovery objectives apply? Who supplies redundancy, failover, replacement capacity, backups, and tested recovery?
“Local continuity” can be a valid requirement. It still needs a failure model.
What happens when a GPU fails? When an update breaks model loading? When power is interrupted? When storage corrupts? When the only operator is unavailable? If the system fails over, does the alternate route preserve the same data and authorization rules?
A managed service has different failure modes: regional availability, provider incidents, quotas, account or billing problems, network reachability, model retirement, and service changes. Moving the boundary does not eliminate reliability work. It changes the mechanisms and owners.
Do not choose the winner from a generic uptime claim. Define the workload’s objective, then require evidence: redundancy design, recovery time, recovery point, spare capacity, backup validation, incident ownership, and a tested restore or failover path.
5. Operations
Decision question: Who owns deployment, upgrades, compatibility, observability, power, cooling, maintenance, and lifecycle replacement? Can that owner carry the obligation continuously?
This is where nominal control becomes operating reality.
A team may be able to assemble a capable server and still be unable to support it as production infrastructure. The relevant test is not whether someone can make the model run once. It is whether an accountable owner can keep the workload within its control and reliability requirements over time.
List the recurring duties. Name an owner for each one. Estimate operator coverage and replacement paths. Define how the team will detect model-serving degradation, dependency drift, thermal problems, storage pressure, failed backups, and unpatched vulnerabilities.
If those obligations exceed the team’s operating capacity, a managed boundary may produce more useful control by assigning infrastructure work to a party equipped to carry it. If provider constraints violate a hard workload requirement, direct operation may be justified despite the additional work.
The right boundary is the one the organization can operate, not the one that looks most independent in a diagram.
Three workloads, three possible boundaries
The test should permit different answers.
Sensitive internal inference
Suppose an approved architecture prohibits prompts and retrieved context from crossing an external model-service boundary.
Self-hosted or tightly controlled private inference should be evaluated because it can satisfy that specific data-path requirement. It has not passed the test yet. The team must still demonstrate identity, authorization, segmentation, encryption, logging, patching, incident response, capacity, and recovery.
The workload is not self-hosted because it is “sensitive.” It is self-hosted only if that boundary satisfies the required controls and the operator can carry the obligations.
A bursty product feature
Suppose demand varies sharply and a small team has no appetite for hardware operations.
Managed inference should be evaluated because it can shift infrastructure duties and may provide standard or provisioned capacity. The choice still depends on service limits, data controls, economics, model availability, and the feature’s reliability objective.
The workload is not managed because managed services are universally cheaper or more reliable. It is managed if the provider boundary satisfies the controls better than the team can satisfy them directly.
Local continuity with elastic overflow
Suppose a core workflow must continue locally during a network or provider outage, but it also benefits from elastic overflow or access to other models.
A hybrid design may fit. “Hybrid” is not permission to route anything anywhere. The team must define which data classes can leave the local boundary, which models are eligible, who authorizes external routing, what happens during degradation, and which requests fail closed instead of failing over.
Hybrid can combine capabilities. It can also combine failure modes. The routing rules are part of the control boundary.
Write the decision down
Before buying hardware or selecting an inference provider, create one record for the workload:
Workload:
Required control:
Chosen boundary: self-hosted | managed | hybrid
Accepted obligation:
Control owner:
Evidence:
Fallback:
Reconsideration trigger:
Complete the record once for each of the five checks. If the evidence is missing, write “unknown.” Do not let an unmeasured assumption become an infrastructure conclusion.
Cost, latency, throughput, energy use, reliability, maintenance effort, and hardware lifespan all matter. They should be measured against the proposed workload and boundary. Without workload-specific evidence, they remain questions—not reasons to declare a universal winner.
The strongest infrastructure choice is not the one with the most hardware under your desk or the fewest services on an architecture diagram.
It is the boundary whose controls you can demonstrate and whose obligations you can actually carry.