Local LLM inference means running a language model's computation on hardware you own, inside your own network, instead of sending text to a provider's servers. The model weights sit on local storage, the processing happens on a local chip, and no part of the query leaves the building. For an Australian law firm, this is the difference between asserting that privileged material stayed within a controlled environment and demonstrating it — a distinction the NSW Supreme Court's Practice Note SC Gen 23 makes practitioners responsible for.
How does local inference actually work?
Less mysteriously than most people assume.
A language model is a large file of numerical weights — many gigabytes. Running it means loading those weights into memory and performing arithmetic against them for each token generated. That computation can happen on any sufficiently capable processor. There is nothing about it that inherently requires a data centre; frontier providers use them because training models and serving millions of concurrent users demands enormous scale. Serving one law firm does not.
Three things make this practical now that were not a few years ago. Open-weight models — released with downloadable weights rather than API-only access — have become genuinely capable. Quantisation techniques compress weights so a model that once needed specialised accelerators now runs on unified-memory hardware with modest loss in quality. And consumer and small-business hardware has caught up: chips with large pools of fast unified memory can hold a capable model entirely in memory.
The practical result is that a single appliance in a comms cabinet can serve a small-to-mid-sized firm at usable speed.
What hardware does a firm actually need?
The binding constraint is memory, not raw processing power. The model's weights must fit in fast memory, and memory bandwidth largely determines how quickly it generates text.
The demonstration device we run is a GMKtec EVO X2 with an AMD Ryzen AI Max+ 395 and 128 GB of unified memory. That is enough to hold capable models entirely in memory and generate at conversational speed, in a unit small enough for a comms cabinet and quiet enough for an office. Larger firms can be sized up — more memory, more concurrent users — and the architecture does not change.
What that hardware does not need is a rack, specialised cooling, or a data-centre contract. It needs power, a network port, and somewhere physically secure.
What can local models actually do — and not do?
Worth being precise here, because both overselling and underselling are common.
What they do well is the work built around your own documents. Summarising a long matter file. Extracting structured data from contracts and correspondence. Comparing two versions of an agreement clause by clause and reporting what changed. Answering questions about a document set with citations back to the source passages. Drafting against firm precedents and playbooks. Classifying and routing inbound correspondence.
The common thread is that the intelligence required is comprehension and careful handling of text you supply, rather than novel reasoning from the model's own knowledge. That is the bulk of what legal AI is genuinely useful for, and it is precisely the work that requires feeding in confidential material.
Where they fall short is the hardest end of open-ended reasoning. On a genuinely novel legal question requiring synthesis across many areas, the largest frontier models remain ahead of what fits on a single appliance. They are also weaker at very long context windows, and — like all language models — they can state incorrect things fluently.
That last point deserves emphasis rather than a footnote. A local model is not a more truthful model. SC Gen 23 requires practitioners to verify that every citation, authority and legislative reference exists and is accurate, and expressly states that verification cannot be performed using a generative AI program. Running the model locally changes where the data goes. It does not change your obligation to check the output.
How do you verify nothing leaves?
This is the actual advantage, and it is worth understanding how the verification works rather than accepting it as a claim.
Network monitoring. Put the appliance on a monitored network segment and watch outbound traffic while the system is in use. Local inference produces none related to the queries.
Disconnect it. The most convincing test available: unplug the network cable and confirm the system still answers questions. A cloud-backed tool cannot. In the LegalBox configuration, zero network egress is the default state rather than a configuration option, which is why this test works.
Firewall it. Egress rules at the network boundary that deny the appliance outbound access, verified by your own IT team or an external auditor.
Physical control. The device is in your office, on your network, in a cabinet you control. Nothing is rented from a data centre you have never visited.
The reason this matters is evidentiary rather than technical. When a client asks where their matter files are processed, or a tender asks you to describe your AI data handling, "we can show you" is a categorically different answer from "our vendor's terms say". Under APP 8 and s 16C of the Privacy Act, a firm remains accountable for personal information it discloses to an overseas recipient — an exposure that does not arise if there is no disclosure.
What are the real trade-offs?
Four, stated plainly.
Capability ceiling. Local models trail the frontier on the hardest reasoning. For document-centred work this rarely binds; for novel legal analysis it might.
Capital cost instead of subscription. You buy hardware outright. There is no per-seat fee and no metered token bill — because no tokens are being sent anywhere — but the money goes out at the start. The economics favour this only if usage is genuine and sustained.
Maintenance. Patching, monitoring, backups, eventual model upgrades. Either your IT support does it or your vendor does, but somebody must.
Security becomes yours. Data on your premises is data you are responsible for. An appliance holding your entire document set is a high-value target sitting inside your network, and it needs the same physical and network security discipline as your document management system. On-premises is not automatically safer — it is more controllable, which only helps if you exercise the control. Given that the OAIC recorded 1,205 breach notifications in 2025 with legal, accounting and management services among the top five sectors, this is not a hypothetical obligation.
Is this right for your firm?
Local inference makes sense when confidential and privileged material is the input to the work — which for most practising firms is where AI's value actually sits. It makes sense when clients or tenders impose data-handling requirements you must evidence. It makes sense when usage is heavy enough that per-token pricing becomes a running cost you would rather convert to a one-off. And it makes sense when you want the behaviour of your tools to change only when you decide, rather than when a provider ships an update.
It makes less sense for a firm still working out whether AI is useful at all. That question is cheaper to answer with a subscription and non-confidential work first. Buying hardware to find out is the wrong order.
We take the comparison further in on-premises AI vs cloud AI for Australian law firms.
Want to see it run with the cable pulled?
If you want to see local inference working on real documents — including the part where we disconnect the network and it keeps going — it takes about twenty minutes. Book a discovery call and we will walk through LegalBox on your kind of matter files.
Sources
- Supreme Court of New South Wales, Practice Note SC Gen 23 (in effect 3 February 2025) — verification of citations, controlled-environment requirement
- Federal Court of Australia, Practice Note GPN-AI
- OAIC, Guidance on privacy and the use of commercially available AI products
- OAIC, APP 8: Cross-border disclosure of personal information
- OAIC, Data breach notifications increase to all-time high in 2025
This article is general information about technology choices, not legal advice. Hardware specifications describe our current demonstration configuration and are subject to change. Model capability claims reflect open-weight models available at the time of writing.
