TensorFold v1.2.9

TensorFold is an open-source local AI inference engine designed to run large language models directly on personal computers and GPU servers. Developed primarily in Zig, it provides native GPU execution through Metal on Apple Silicon and CUDA on NVIDIA hardware, without requiring a Python or MLX installation to serve models.

Unlike tools that simply wrap existing inference frameworks, TensorFold implements its own model execution and GPU kernels. It also provides an API server compatible with popular AI clients, making it possible to use locally hosted models with applications designed for OpenAI or Anthropic services.

The project goes beyond basic model inference by including a desktop workspace, web interface, model management tools, and experimental capabilities for modifying model weights while running.

Features

Native Local LLM Inference

TensorFold runs language models directly on supported hardware using native GPU backends.

Its main advantages include:

• Metal acceleration for Apple Silicon.

• CUDA acceleration for NVIDIA GPUs.

• Native HIP support for selected AMD hardware.

• Direct model checkpoint loading and tokenization.

• No Python or MLX installation required for native serving.

• Configurable context length and generation parameters.

• Support for concurrent inference requests.

The engine is designed to make local inference more accessible without requiring users to maintain a complicated Python-based machine learning environment.

Hardware support varies by backend and model architecture. Not every model is compatible with every GPU, so users should consult the project's model compatibility information before downloading large checkpoints.

Model Management

TensorFold includes commands for downloading and managing models from Hugging Face and other supported sources.

For example, users can download a model and start an inference server with:

tensorfold pull mlx-community/Qwen3.5-2B-MLX-4bit
tensorfold serve mlx-community/Qwen3.5-2B-MLX-4bit --name local-model

The application also supports existing local checkpoint directories and models already present in the Hugging Face cache.

Model configuration options include context length, maximum output tokens, temperature, sampling parameters, and reasoning settings. This provides flexibility when adapting a model for coding, chat, document processing, or other workloads.

An important consideration is that model weights have their own licenses. TensorFold's Apache 2.0 license does not automatically grant permission to use every model distributed through external repositories.

OpenAI and Anthropic-Compatible API

One of TensorFold's most useful features is its local API server.

It supports several popular API formats, including:

• OpenAI Chat Completions.

• OpenAI Responses API.

• OpenAI Completions.

• Anthropic Messages API.

This allows compatible applications to communicate with locally hosted models instead of relying on cloud AI providers.

For example, after starting the server, users can test it with:

curl http://127.0.0.1:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "local-model",
    "messages": [
      {
        "role": "user",
        "content": "Explain how local LLM inference works."
      }
    ],
    "max_tokens": 256
  }'

The default server address is 127.0.0.1:8080. Authentication options are available for deployments that need API access control.

This compatibility makes TensorFold useful for integrating local models with coding assistants, chat interfaces, and other applications that already support standard AI APIs.

Concurrent Requests and Performance Controls

TensorFold includes controls for managing inference workloads and available GPU memory.

Its serving configuration supports:

• Parallel inference requests.

• Automatic concurrency selection.

• Request queuing.

• Context-length limits.

• Prompt caching.

• Maximum output token limits.

• Sampling and reasoning configuration.

• Memory diagnostics.

The automatic parallelism option attempts to determine how many requests can run concurrently based on available resources. Explicit concurrency limits are also supported.

Prompt caching can reduce repeated processing when conversations share common prefixes. Additional options allow users to control memory allocation and retained prompt data.

These features are particularly useful when operating a local inference server for multiple applications or users.

TensorFold Desktop Workspace

TensorFold also provides a desktop application for managing local AI workloads.

The workspace combines several functions:

• Local model discovery and downloads.

• Model execution and chat.

• Hardware recommendations.

• Live generation metrics.

• Coding agent management.

• Document editing.

• Local inference management.

This provides a more approachable interface for users who prefer graphical applications over terminal-based workflows.

The macOS desktop application is distributed as an Apple Silicon beta. Its mobile clients remain preview software, so users should not expect the same maturity as the native inference engine.

TensorFold Web

For users who prefer a browser interface, TensorFold includes a local web application.

It provides a graphical environment for selecting models, running inference, and chatting with locally hosted models.

The web interface complements the command-line server and desktop application, giving users another way to interact with their local AI environment.

AI Coding Workers

TensorFold's desktop workspace also includes coding workers designed to assist with development tasks.

This extends the application beyond conventional chat-based inference. Users can manage coding-oriented workflows alongside their local models and documents.

The desktop workspace is still evolving, so its agent functionality should be evaluated separately from the more established inference engine.

Living Weights

One of TensorFold's more experimental capabilities is Living Weights.

Instead of keeping newly learned information exclusively in an external database or conversation history, this feature attempts to write selected facts directly into a model's checkpoint files.

Users can provide information through a dedicated API endpoint, and TensorFold processes the material as lessons that modify the model's weights.

The project also provides a 3D visualization interface showing learned facts and their associated model components.

However, Living Weights is explicitly experimental. The project documents limitations involving interference between related facts, declining recall after learning multiple facts, and reduced decoding performance on some hardware.

It is an interesting research feature, but it should not be considered a replacement for established retrieval-augmented generation systems or a reliable persistent memory solution.

Performance and Compatibility

TensorFold is built with Zig and uses native GPU execution rather than depending on a Python-based inference stack for its serving engine.

This architecture reduces the number of runtime dependencies and gives the project direct control over model execution, memory management, and GPU kernels.

Performance depends on the model architecture, quantization format, GPU memory, context length, and available hardware acceleration.

Apple Silicon: TensorFold uses Metal for GPU execution on supported Apple Silicon Macs. Unified memory can be useful when running larger models, although the available memory must accommodate model weights, context data, and concurrent requests.

NVIDIA GPUs: The Linux release supports CUDA-based inference. Its documented Linux release requirements include an NVIDIA driver version R580 or newer and CUDA 13.0 compatibility. Supported GPU architectures and model combinations vary.

AMD GPUs: Native HIP support is available for selected hardware and models. The project documents specific qualification results rather than guaranteeing compatibility with every AMD graphics card.

The project also advertises Vulkan support, but users should verify the actual backend and model compatibility for their target hardware rather than assuming every advertised backend supports every model.

TensorFold supports Linux and macOS for its documented native workflows, with additional Windows support through its native Windows build documentation.

System Requirements

macOS

• Apple Silicon Mac.

• Compatible Metal GPU support.

• Sufficient unified memory for the selected model.

• Homebrew for the recommended installation method.

Linux

• 64-bit Linux system.

• glibc 2.28 or newer for the documented NVIDIA release archives.

• NVIDIA GPU and R580-or-newer driver for the documented CUDA release.

• CUDA 13.0-compatible runtime environment.

• Sufficient GPU or unified memory for the selected model.

Windows

• Supported Windows environment.

• Compatible NVIDIA GPU and driver for the CUDA workflow.

• Follow the project's native Windows build and installation documentation.

There is no universal minimum RAM requirement because memory consumption depends heavily on the model, quantization, context size, and concurrency settings.

Pros and Cons

Pros

• Free and open-source under Apache 2.0.

• Native inference engine written in Zig.

• Metal acceleration for Apple Silicon.

• CUDA support for NVIDIA GPUs.

• Native HIP support for selected AMD hardware.

• Does not require Python or MLX for native serving.

• Supports OpenAI-compatible and Anthropic-compatible APIs.

• Includes model downloading and management.

• Supports concurrent requests and prompt caching.

• Provides desktop and browser-based interfaces.

• Includes performance diagnostics and memory controls.

• Experimental Living Weights functionality.

Cons

• Hardware and model compatibility vary considerably.

• NVIDIA Linux releases require a relatively recent driver.

• Large models require substantial memory.

• Some desktop and mobile functionality remains in beta or preview.

• Living Weights is experimental and has documented limitations.

• The project has a learning curve for users unfamiliar with local LLM deployment.

• Not every model available through Hugging Face is compatible with its native engine.

How to Install

macOS

The recommended installation method for Apple Silicon is Homebrew:

brew install ashhart/tensorfold/tensorfold

Download a model and start the server:

tensorfold pull mlx-community/Qwen3.5-2B-MLX-4bit

tensorfold serve mlx-community/Qwen3.5-2B-MLX-4bit \
  --name local-model \
  --context 8192 \
  --no-thinking

Linux with NVIDIA GPU

Download the appropriate release archive for your CPU architecture. The project provides separate archives for x86_64 and ARM64 systems.

For example:

V=1.2.8

curl -fLO "https://github.com/ashhart/TensorFold/releases/download/v$V/tensorfold-$V-linux-x86_64.tar.gz"

tar -xzf "tensorfold-$V-linux-x86_64.tar.gz"

export PATH="$PWD/tensorfold-$V-linux-x86_64/bin:$PATH"

Download a compatible model and start inference:

tensorfold pull mlx-community/Qwen3.5-2B-MLX-4bit

tensorfold serve mlx-community/Qwen3.5-2B-MLX-4bit \
  --name local-model \
  --context 8192 \
  --no-thinking

Check the latest release instructions before installation because package names, supported architectures, and hardware requirements can change.

Verify the Installation

After starting the server, open another terminal and test the API:

curl -fsS http://127.0.0.1:8080/v1/models

The command should return the models currently available through the running server.

For additional options, use:

tensorfold --help
tensorfold capabilities --json
tensorfold serve --help

Final Verdict

TensorFold is an ambitious local AI inference project that combines a native Zig engine, GPU acceleration, model management, and API compatibility in one ecosystem. Its ability to serve local language models without requiring Python or MLX is a major advantage for users who want a more self-contained inference environment.

The OpenAI and Anthropic-compatible APIs make it practical for integrating local models with existing applications, while its desktop and web interfaces provide alternatives to command-line operation. Advanced features such as prompt caching, concurrency controls, and memory diagnostics also make it interesting for more demanding inference workloads.

Its main limitation is hardware and model compatibility. Users must verify that their GPU, driver, and chosen checkpoint are supported before expecting good results. The desktop application and Living Weights feature also have different maturity levels from the core inference engine.

Overall, TensorFold is a promising choice for developers and enthusiasts who want to run local LLMs with native GPU acceleration and a flexible API server.

Sources

• TensorFold GitHub Repository and Documentation

• TensorFold Releases

• Desktop Application Documentation

• License and Third-Party Notices