[Proposal] Run an LLM Locally with Java: Connect to Ollama

[thesurenk]

Request Issue

No response

Website Section

Home --> Tutorials --> Java with AI

Proposal Details

Summary

A beginner friendly tutorial showing how to connect Java to a local LLM via Ollama, using only HttpClient. No cloud keys, no SDKs, no frameworks.

What the tutorial covers

• Why local: privacy, offline development, zero API cost, fast iteration.
• Prereqs: install Ollama, start the local server, pull a model.
• First call: Java HttpClient POST to Ollama’s /api/generate with:
  • model
  • prompt
  • stream: false (clean single JSON response)
• Parsing JSON: print raw response first; optionally parse response field with Jackson.
• Sanity checks: timeout, server-not-running errors, model-not-available guidance.

Example Ollama models to use (and pull commands)

Recommended general-purpose starters

• llama3.2:3b
ollama pull llama3.2:3b
• llama3.2:1b (smaller/faster)
  ollama pull llama3.2:1b

Coding-focused

• qwen2.5-coder:7b
ollama pull qwen2.5-coder:7b

Small + efficient

• phi3:mini
ollama pull phi3:mini

Optional multimodal (vision)

• llava
ollama pull llava

Tip for the tutorial

• “If you’re unsure, start with llama3.2:3b.”
• “If you want coding help, try qwen2.5-coder:7b.”
• “If you hit memory limits, use llama3.2:1b or phi3:mini.”

Author References

1. Linkedin https://www.linkedin.com/in/ksurendra/
2. Personal site/portfolio: https://surenk.com/
3. Writing (Substack): https://www.techinpieces.com/ and https://surenk.medium.com/
4. Helidon blog: https://medium.com/helidon/anatomy-of-helidon-mcp-ollama-designing-ai-enhanced-java-microservices-a9ddaba1325d
5. Oracle blog: https://blogs.oracle.com/emeapartnerweblogic/post/helidon-with-swagger-openapi-by-suren-konathala
6. Open source work: https://github.com/thesurenk