LocalLLM (Offline USB Chat)

This project runs a fully offline local LLM chat app from a USB drive on Windows using portable WinPython.

It is designed for restricted environments (no internet, no local installs).

Quick Start (Windows)

1. Put at least one .gguf model file in models\.
2. Ensure portable Python exists under WinPython\ (for example WinPython\python-3.10.x.amd64\python.exe).
3. Install dependencies into that portable Python:
   • WinPython\python-3.10.x.amd64\python.exe -m pip install -r requirements.txt
4. Double-click Start_AI.bat.
5. In chat, type your question (clear resets, exit quits).

Project Structure

LocalLLM/
├─ Start_AI.bat
├─ requirements.txt
├─ src/
│  └─ chat.py
├─ models/
│  └─ your-model.gguf
└─ WinPython/
   └─ python-3.10.x.amd64/ (or similar folder containing python.exe)

What This App Does

• Loads the first .gguf model from models/ automatically.
• Runs a terminal chat assistant with streaming output.
• Uses a system prompt that makes the AI behave like a university Machine Learning tutor.
• Manages context safely so large pasted text is less likely to crash the session.

1) Prepare USB Folder

Copy this project folder to your USB root (or any USB path), then ensure:

1. models/ contains at least one .gguf file.
2. WinPython/ contains a portable Python with python.exe inside.
   • Preferred path pattern:
     • WinPython\python-3.10.x.amd64\python.exe
   • The launcher also falls back to searching recursively under WinPython\.

2) Install Dependencies (Offline-Friendly)

You must install llama-cpp-python into the portable WinPython environment.

Option A: Online once, then copy USB

If you can run online once on a staging machine:

WinPython\python-3.10.x.amd64\python.exe -m pip install -r requirements.txt

Then move the fully prepared folder to USB.

Option B: Fully offline target machine (recommended for restricted PCs)

1. On a machine with internet, download wheels for your exact Python version/architecture:

python -m pip download -r requirements.txt -d wheels

2. Copy the wheels/ folder onto USB (inside project root).
3. On the restricted Windows machine, install from local wheels only:

WinPython\python-3.10.x.amd64\python.exe -m pip install --no-index --find-links wheels -r requirements.txt

Notes:

• Wheel compatibility must match your WinPython version (for example cp310/cp311/cp313).
• llama-cpp-python may require selecting a compatible prebuilt wheel.

3) Run the Chat App on Windows

Double-click:

Start_AI.bat

The script:

• changes terminal color to green-on-black (color 0A)
• finds portable Python under WinPython\
• runs src\chat.py
• pauses on error so the window does not disappear immediately

Chat Commands

In the chat:

• exit or quit: close app
• clear: reset conversation memory

Troubleshooting

"Portable Python executable not found"

• Ensure python.exe exists under WinPython\...
• Keep Start_AI.bat in project root

"No .gguf model found"

• Place at least one .gguf file in models\

Import error for llama_cpp

• Install dependencies into the same portable Python used by Start_AI.bat
• Re-check wheel compatibility with your Python version

App is slow or runs out of memory

• Use a smaller quantized GGUF model
• Close other programs
• Optionally tune env vars before launch:
  • LOCAL_LLM_CTX (default 4096)
  • LOCAL_LLM_MAX_NEW_TOKENS (default 512)
  • LOCAL_LLM_CONTEXT_RESERVE (default 256)

Security/Operations Notes

• This app is local/offline by design; no cloud API keys are required.
• Keep model/license files and Python runtime compliant with your organization policies.