Local AI
Local AI means running AI on computers you control. The guide on this page explains the components, trade-offs, applications, and security considerations involved in building and using a local AI system.
Overview
Local AI means running an AI model and its supporting software on hardware you control, rather than relying on remote cloud infrastructure. It provides you with greater control over private data, model selection, system configuration, and long-term access to your AI tools.
This guide is intended for computer-savvy readers who have used AI but are new to running it locally. It is a conceptual introduction, not a step-by-step installation manual. It grew from notes collected while exploring local AI and discovering that building a useful system involves more than choosing and downloading a model.
Components of Local AI
Computer Hardware. AI models can require substantial memory. The amount of memory determines which models can run and how much context they can process, while memory bandwidth strongly affects performance.
AI Models. The model provides the learned capabilities of an AI system. Its size depends on the number of parameters and the precision used to represent them. Quantization reduces storage and memory requirements, making larger models feasible on smaller computers. Mixture-of-Experts (MoE) models and context length also impact memory size and performance.
Inference Engines. An AI model cannot operate by itself. An inference engine loads it into memory, processes input prompts, and generates output tokens. Two principal phases of inference are input processing (prefill) and output generation (decode). They both matter for performance.
AI Workspaces and Agent Harnesses. A workspace lets people interact with models through chat and other features. Agent harnesses extend this approach by supporting agents that can plan work, invoke tools, inspect their results, and continue toward a goal with limited human intervention.
Applications
The simplest application of local AI is conversational chat, but local AI can also search the web, read and summarize documents, write code, and execute specialized workflows. Tools provide the ability to perform actions; skills describe reusable task procedures; and Model Context Protocol (MCP) servers provide a standard interface for accessing tools and resources. In some cases, a purpose-built application may be more predictable than a general-purpose agent.
Security
Running a model locally does not automatically prevent data from reaching the Internet: web search, cloud services, and connected tools may expose information. Autonomous agents can also make mistakes, overwrite files, or delete valuable data. Appropriate protections include limiting access to files and credentials, restricting network connections, requiring approval for sensitive actions, using isolated environments, and keeping reliable backups that allow recovery when something goes wrong.
Download
Download a complete guide with further details: A Guide to Local AI (PDF)
Resources
YouTube Channels with Reviews of Hardware and AI Models
Inference Engine
AI Workspace and Quantized Models
Open Weight Models
Unified API for Cloud Models
Agent Harness
Benchmarks and Other Resources
- Artificial Analysis — AI model benchmarks
- Artificial Acceleration — AI status dashboard