Local LLM Server on Android: Turning Old Phones into Private AI

TL;DR: I wanted private AI features for my apps without sending data to the cloud, so I built VicinoLLM an open-source local LLM server on Android. It turns any device, from a 7-year-old Galaxy S10 to a modern S24 Ultra, into a secure, OpenAI-compatible API endpoint on your LAN.

Writing this in April 2026, the current stable release of FairEmail,a privacy-focused email client I use daily, includes a feature I initially overlooked: “AI (OpenAI-compatible)”. If you point it at an OpenAI-compatible endpoint, you get all the modern conveniences: summarize this long thread, draft a polite reply, or translate a message from German to English.

The problem hit me immediately. I am not about to pipe my personal inbox to OpenAI, Anthropic, or any other cloud provider. The entire reason I use privacy-centric apps is to retain control over my data. What I needed was a local endpoint,AI running on hardware I physically own,and I simply didn’t have one set up.

This isn’t just an email problem. Half the apps on my laptop and phone now offer a “bring your own LLM” field. Note-taking apps, RSS readers, and code editors all assume you will happily paste an API key and pay a cloud provider. Almost none of them assume you might want to run the model yourself.

Having run this setup daily for the past few months, routing all my email summarization and note-taking through my own pocket-sized server, I can confidently say the era of mandatory cloud AI is over. I realized I already had the hardware. I just needed the software. That is how VicinoLLM was born. Vicino means “near” in Italian, reflecting the core philosophy: your AI should live near you, on your own hardware, not in a data center across the ocean.

When most people think of running large language models, they picture massive GPU rigs. But the landscape has shifted dramatically. I wanted to see exactly what it takes to run a local LLM server on Android, so I tested the extremes of my hardware drawer.

Tested environments:

• High-end: Samsung Galaxy S24 Ultra (12GB RAM, Snapdragon 8 Gen 3)
• Legacy: Samsung Galaxy S10 (8GB RAM, Snapdragon 855 – released in 2019)
• Framework: MediaPipe LLM Inference API v0.10.14
• Models: Gemma 2B (INT4) and Gemma 4B (INT8)

You might assume a 7-year-old phone is useless for AI. I certainly did. But I relied on quantization—a technique that reduces the precision of the numbers used in an AI model (like converting 32-bit floats to 4-bit integers) to make it significantly smaller and faster—to fit these models onto older hardware.

On the Galaxy S10, using a 4-bit quantized version of Gemma 2B, the model loads into about 1.8GB of RAM. It generates text at roughly 8-10 tokens per second. That isn’t lightning fast, but for summarizing an email in the background while I read another, it is perfectly adequate. Millions of these capable devices are sitting in drawers right now, completely underutilized.

On the flip side, my daily driver is an S24 Ultra. Running Gemma 4B on this device is a completely different experience. The neural processing unit (NPU) chews through prompts, spitting out 35-40 tokens per second. It is genuinely difficult to distinguish this local performance from hitting ChatGPT over a mobile data connection.

Why I Didn’t Just Use Termux

I spent a weekend evaluating existing options before writing any code. My first thought was to install Termux, compile llama.cpp, and run a Python server. If you have ever tried this, you know the pain.

While you can get Linux binaries running on Android, the emulation layers and lack of direct hardware acceleration slow the model down five to eight times compared to native execution. A prompt that takes 2 seconds natively takes 15 seconds in Termux. That is the difference between a usable daily tool and a frustrating novelty.

I also looked at existing open-source Android AI apps. They are almost universally client apps. You open them, type a message, and get a reply. They do not expose an API to your local network. You cannot point your email client or your laptop’s code editor at them.

Here is how the options stack up:

Connecting the Pieces: How to Use It

Once the server is running on your phone, connecting it is incredibly satisfying. Here is exactly how I have it set up with FairEmail:

1. I open VicinoLLM on my old S10 (which sits plugged in on my desk) and tap “Start Server”.
2. http://192.168.1.45:8080.
3. In FairEmail on my main phone, I go to Settings > AI.
4. I select “Custom OpenAI Endpoint” and enter http://192.168.1.45:8080/v1.
5. I enter a dummy API key (VicinoLLM ignores it, but FairEmail requires the field to be filled).

That is it. Instantly, all the AI features in the email client come to life, powered entirely by the phone sitting three feet away. No subscriptions, no data harvesting, no cloud latency.

If you want to use this while you are away from your house, you cannot just expose the port to the public internet—that is a massive security risk. Instead, I use Tailscale. By putting both my main phone and the old S10 on the same Tailscale mesh network, I can securely route AI requests back to my house from anywhere in the world.

FAQ

Does VicinoLLM work with FairEmail?

Yes, right out of the box. Point FairEmail’s custom AI settings at your phone’s LAN address (e.g., http://192.168.1.X:8080/v1). Both devices need to be on the same Wi-Fi, or connected via a VPN like Tailscale. I use it daily for inbox summaries and reply drafts.

What Android phone do I need to run a local LLM?

You need a reasonably modern Android device with at least 6 GB of RAM, running Android 10 or higher. The smaller 2B models run fine on a 2019 Samsung S10 (8GB RAM). Larger 4B/7B models prefer 12 GB of RAM, like you would find in a Pixel 8 Pro or Samsung S24 Ultra.

Does my data ever leave the phone?

No. The AI model, your conversation history, and any text you process stay entirely on the device. The only exception is if you explicitly enable the optional web search feature, which must contact an external search engine to retrieve live information.

Can I use it away from home?

Not by default, as the server only listens on your local network for security reasons. If you want remote access, install a mesh VPN like Tailscale on both your client device and the Android server. This provides secure, private remote access.

How much does the app cost?

Nothing. VicinoLLM is free and open source under the Apache 2.0 license. There is no paid tier, no subscription, and no ads. I included a small crypto tip jar in the settings, but it is entirely optional.

Why build this yourself instead of waiting for Google or Apple?

Incentives are misaligned. Big tech companies have every financial reason to keep your AI usage on their cloud servers—that is where they collect telemetry and subscription revenue. A free, private, local-only alternative isn’t in their business model. We have to build it ourselves.

Conclusion

VicinoLLM exists because a single settings menu in an email client reminded me that my hardware is far more capable than I was letting it be. The gap between “I want AI that respects my privacy” and “I can actually have it” was just small enough to close with a weekend project that quietly turned into a month of deep-diving into Android background services and Ktor routing.

We have been conditioned to think of smartphones as thin clients—glass screens that just fetch data from a server somewhere else. But a modern flagship, or even a flagship from five years ago, is a remarkably powerful computer. Running a local LLM server on Android flips the dynamic. It takes the power of generative AI and puts it entirely under your physical control.

If you have an old Android phone sitting in a drawer, or a current one doing nothing special most of the day, give it a try. The hardware is already in your house. You just need to give it a new job. The source code is available on my repository, completely free. Reclaim your privacy, and put those idle NPUs to work.