A while back I bought myself a little box based on a AMD Ryzen AI Max+ 395 with 128GB of shared memory. The reason, simple, running a few local LLMs to process data I don’t want to share with anybody, and as a PoC to take into my professional services.

My Setup

I started by discussing a few use cases with a good friend, who plays a lot with image editing and generation and has done some audio and video things. We, I guess obviously, discussed going for an NVIDIA GFX card with 24GB of RAM vs. a system with an embedded CPU with a larger amount of shared RAM. My perspective was 30..40..50 output tokens per second on a smaller model vs. maybe 10..15 output tokens per second on a properly large model. His perspective was more on the payload, image generation, especially when based on other images and especially when these images are combined to a video simply require a notable amount of RAM. As a result I decided to go for the shared memory, to give him possibilities and for myself, I value the quality of the output more than the speed.
Looking at prices, and just paying it out of my own pockets, I ended up going for a GMKTec EVO-X2 running an AMD Ryzen AI Max+ 395 with 128GB of shared memory. I took the 1TB version, as I didn’t really care about the HDD and, well, saved a few Euros.
Having planned to share it with friends, and thus also refinancing it a little, I had Claude Vibe-Code me a Django based LLM router with metering. It allows the creation of API keys, which can then be utilised via an OpenAI compatible API. (It is important to note that “OpenAI compatible” doesn’t have anything to do with chatGPT. They were simply smart enough to give a now widely used API their own name.) The application shows the currently running model(s), contains a small voting system, allows changing the model and simply counts how many tokens a user has used. While the overall setup was initially all running on my Ailer0, the GMKTec box, I migrated it to a different machine. Thus, the setup is now multi-node compatible, and supports multi-tenant usage. As said, I was aiming at potentially using the setup in my professional services. Not having mentioned it yet, the setup runs in Docker.
The initial inference side was based on Ollama , as I had used it before and it supported the models I initially wanted to try. After some research, and further discussions with “Kumpel Claude”, I decided to add OpenWebUI as a quick and simple interface. Having a running Authentik for SSO, I also integrated everything with that.

The First Tests

Quick, easy, simple. I have a load-out in the mornings, one in the evenings, one during the night, thus the model automatically changes as expected. Some of the load-outs actually run more than one model at a time, simply to optimise a few tasks. For me, I was happy, it’s not Claude, its not chatGPT, but it delivers exactly what I need: Evaluating data, enriching information utilising MCP and RAG.
For my friend’s side, well, things didn’t quite go as well. I initially setup comfy , which I’m running on my normal box with my GFX card on 12GB of RAM. While it ran and it enjoyed using the extra RAM, most things are developed for NVIDIA’s architecture, thus performance, well, lets simply not call it performance. Using it wasn’t fun. I then tried a few other image and video generation models and setups. The first thing I learned here, was that while Ollama was compatible with all of the typical inference models, a few of the graphics options only run on vLLM . Thus, added to the stack. Sadly with no positive results as, well, I wasn’t able to find anything I actually enjoyed using that ran on my AMD CPU.

Continuing

As potentially expected, I added a few abliterated models, which as expected are able to use first semester physics and chemistry and will answer next to all questions. In return, I sometimes have the feeling, that the abliterated models sometimes lack a little logic and consistency. That said, this is a known issue.
Going further I added a searXNG to allow the inclusion of some fresh data and created a few use cases. It took a few tries, but then ran smoothly. I also added Text to Speech based on Qwen3 TTS. I have a full flow creating a few hundred samples based on a static text, allow me to listen to them, and then clone the voices I like. Why cloning? Well, I hadn’t realized, that when using generic TTS, the text itself is part of the context from which the voice is generated. Practically this means, new text new voice :-( Coming from here, I also added a little STT.
The system is now fully based on “loadouts”, where a loadout conists of one or more models in addition to the layer on which the model runs, vLLM or Ollama. As such, when switching from an Ollama based model to a vLLM based model, the ollama container is fully stopped, the vLLM container launched and then the model loaded.

First, Simple Use Cases

My first larger use case was my little ISO, Information Security Officer. Following my principle, of actually using AI as a tool not as solution, I’ve created a database of various Security standards, papers and other publications and am using the LLM as an interface. As such, every question I ask is paired with a reference from the database, if there is one, otherwise the model gets to properly infer and give me its opinion. Following this approach, while the response is still far from human, its clean and good, so practically, it saves me a fair amount of time on certain topics. (I’m still far from having fed it everything I want it to know.)
Daily business? Well, my board game collection now has a chat interface answering basic rule questions, based on RAG. My inventory offers the same. Identifying names, anonymizing data, summarizing. All that said, there multiple models I’ve been wanting to try, which simply won’t run on the AMD architecture, but, hey…

Random Notes

• Allocation the RAM to the GPU via BIOS seems like the correct approach, but my current setup, to make it work, actually has the full RAM allocated to the CPU and then utilized from the GPU via the Kernel. There is a small issue, that otherwise the memory cannot be fully used. That said, its the same physical memory on the same BUS on the same CPU, thus next to no speed impact.
• Loading models can take a fair amount of time, especially the larger one, so ad-hoc changes might fail
• There are smaller models and simpler workloads , that are faster and just as good on my old GTX 1650, as such i.e. TTS and STT run there
  • It’s one of my nodes

Numbers

Having played with a few models, here the current benchmark:

The numbers are based on the following questions

1. What is the capital of Japan?

2. Convert 98.6 degrees Fahrenheit to Celsius, showing your work.

3. (context: widget-code, turn 1) Write a Python function called total_even that takes a list of integers and returns the sum of the even numbers in it.

4. (context: widget-code, turn 2) Now modify total_even so it returns a tuple (sum, count) — the sum of even numbers and how many were found.

5. (context: widget-code, turn 3) If I called this updated function on [1, 3, 5, 7], what would it return? Explain briefly why.

6. A train leaves City A at 60 mph heading toward City B, 300 miles away. At the same time, a second train leaves City B heading toward City A at 90 mph. How long until they meet, and how far from City A?

7. Explain the difference between TCP and UDP, and give one real-world scenario where each would be the better choice.

8. (context: inventory, turn 1) Let’s build a simple inventory. I have three items: Widget A costs $5, Widget B costs $10, Widget C costs $20. Represent this as a Python dictionary mapping name to price.

9. (context: inventory, turn 2) Add a fourth item, Widget D, costing $15, to that dictionary.

10. (context: inventory, turn 3) Write a function that takes this dictionary and returns the name of the single most expensive widget.

11. (context: inventory, turn 4) Now modify the function to return the two most expensive widgets, ordered from most to least expensive.

12. (context: inventory, turn 5) If I also added a Widget E costing $20 — the same price as Widget C — what would your function return for the top two, and why might that result be ambiguous or order-dependent?

13. Long-context recall (needle in a haystack). Read the following passage carefully, then answer the question at the end.

    The Meridian Shipping Company was founded in 1884 by a group of merchants in the port city of Calloway. Over the following decades, it expanded its fleet from two small schooners to more than forty steel-hulled freighters, eventually becoming one of the largest independent shipping operators on the eastern seaboard. Its early years were marked by fierce competition with the rival Hartwell Line, which controlled most of the lucrative grain trade routes.

    By the 1920s, Meridian had diversified into passenger transport, operating a small fleet of ocean liners between Calloway and the northern ports. This expansion required significant investment in infrastructure, including a new dry dock and a dedicated repair facility on the outskirts of the city. The company’s headquarters moved three times during this period before finally settling into the Beaumont Tower, a twelve-story building completed in 1931.

    The maintenance access code for Generator Room 3 at the Beaumont Tower facility is 84217. This code was changed only once in the building’s history, after a security review in 1958 recommended stricter controls on the basement utility areas.

    The Great Depression hit Meridian hard, forcing the company to sell off half its fleet and lay off nearly a third of its workforce. It survived largely due to a series of government shipping contracts during the Second World War, which kept its remaining vessels in continuous operation ferrying supplies across the Atlantic.

    In the postwar decades, Meridian gradually shifted away from passenger service entirely, focusing instead on containerized cargo as the shipping industry modernized. The company was eventually acquired by a larger conglomerate in 1987, though the Meridian name was retained for its original shipping division until as recently as 2003.

    Question: What is the maintenance access code for Generator Room 3, and in what year was it last changed?

14. Respond with ONLY valid JSON — no prose, no markdown code fences, no explanation before or after. The JSON must describe a fictional person with exactly these keys: "name" (string), "age" (integer), "skills" (array of strings, at least 2 items).

15. Write a short story of approximately 400 words about a lighthouse keeper who discovers something unusual during a storm.

16. The following Python function is supposed to check whether a string is a palindrome (reads the same forwards and backwards, ignoring spaces and case), but it has a bug. Find the bug, explain what’s wrong, and provide a corrected version.

    def is_palindrome(s):
        s = s.lower().replace(" ", "")
        for i in range(len(s) // 2):
            if s[i] != s[len(s) - i]:
                return False
        return True
    

17. Erkläre in zwei bis drei Sätzen, warum der Himmel tagsüber blau erscheint.

18. In what year did Albert Einstein win the Nobel Prize in Physics for his theory of general relativity? (false premise: he won the 1921 prize for the photoelectric effect, not for relativity)

And Now?

Well, I have a few use cases, I’m running on my local setup and will post an update when I manage.
Should you need access to a AI Max+ 395 with 128GB or require a securely run small model, feel free to drop me a message and we’ll work something out!