ROCm OS?

[Hueristic]

I picked up 8 GPU's to run a ROCm accelerated LLM but have no clue what OS dependencies are required, The cpu on the 8 slot board is an old skylake celery (soldered) so I want to keep it as lightweight as possible so can I get away with something like PCLinuxOS or will I need a tiny kernel?
I'm obviously avoiding Bloated SystemD distros.
I haven't found any good sources for info I straight up ROCm setups.

Also does anyone have a clue what the impact of running lower pci-e lanes will have, I may have to move to a board with more lanes but will see if the 1 GB/s restriction is manageable or not, I remember on things like FAH back in the day it didn't matter.

 

[Hueristic]

Bump

 

[Solaris17]

Tinygrad utilizes AMD 7900XTXs maybe it can be a source of enlightenment. They use ubuntu 22.04 but they do have GIT, instructions and an FAQ. Might be a good resource if you are targeting ROCm.

tinygrad: A simple and powerful neural network framework

 

[Denver]

I honestly think you'd get more help posting on Reddit's "Local Llama." Also, Vulkan is now faster than ROCm, dangerously close to CUDA, at least for LLMs;

 

[Hueristic]

Cool, thx for the input guys.

After some more research I've come up with this that looks helpful so I will drop a link if anyone stumbles across this in the futre with the same question.

GitHub - rocm-arch/rocm-arch: A collection of Arch Linux PKGBUILDS for the ROCm platform

A collection of Arch Linux PKGBUILDS for the ROCm platform - rocm-arch/rocm-arch

github.com

 

[igormp]

I picked up 8 GPU's to run a ROCm accelerated LLM but have no clue what OS dependencies are required, The cpu on the 8 slot board is an old skylake celery (soldered) so I want to keep it as lightweight as possible so can I get away with something like PCLinuxOS or will I need a tiny kernel?
I'm obviously avoiding Bloated SystemD distros.
I haven't found any good sources for info I straight up ROCm setups.

Also does anyone have a clue what the impact of running lower pci-e lanes will have, I may have to move to a board with more lanes but will see if the 1 GB/s restriction is manageable or not, I remember on things like FAH back in the day it didn't matter.

Your OS of choice shouldn't make much of a difference, specially CPU-wise.
Avoiding systemd might give you more headaches with almost no benefit. No reason to go with a tiny kernel either.

As said above, localllama on reddit is a good place to start.

 

[Rover4444]

I picked up 8 GPU's to run a ROCm accelerated LLM but have no clue what OS dependencies are required, The cpu on the 8 slot board is an old skylake celery (soldered) so I want to keep it as lightweight as possible so can I get away with something like PCLinuxOS or will I need a tiny kernel?
I'm obviously avoiding Bloated SystemD distros.
I haven't found any good sources for info I straight up ROCm setups.

Also does anyone have a clue what the impact of running lower pci-e lanes will have, I may have to move to a board with more lanes but will see if the 1 GB/s restriction is manageable or not, I remember on things like FAH back in the day it didn't matter.

Headless Arch Linux, use archinstall and choose your init system there. Afterwards check the GPGPU page on the wiki and install the ROCm packages.

Running fewer or slower lanes doesn't impact inference speed, but loading and unloading the model will be slow.

I honestly think you'd get more help posting on Reddit's "Local Llama." Also, Vulkan is now faster than ROCm, dangerously close to CUDA, at least for LLMs;
View attachment 387519

It's awful. Maybe there's a --lowvram option that kobold doesn't expose in their GUI, but my processing speeds take a nosedive with Vulkan even with the extra VRAM and you can't isolate the model itself onto the GPUs. ROCm is much better.

 

[Hueristic]

Headless Arch Linux, use archinstall and choose your init system there. Afterwards check the GPGPU page on the wiki and install the ROCm packages.

Running fewer or slower lanes doesn't impact inference speed, but loading and unloading the model will be slow.

It's awful. Maybe there's a --lowvram option that kobold doesn't expose in their GUI, but my processing speeds take a nosedive with Vulkan even with the extra VRAM and you can't isolate the model itself onto the GPUs. ROCm is much better.

Thanks for the detailed first hand info!

 

[Rover4444]

I honestly think you'd get more help posting on Reddit's "Local Llama." Also, Vulkan is now faster than ROCm, dangerously close to CUDA, at least for LLMs;
View attachment 387519

By the way, NV_coopmat2 is NVIDIA exclusive. Mesa supports KHR_coopmat.

Thanks for the detailed first hand info!

You're very welcome! If you have any extra questions or problems, feel free to ask!

No sane man should be forced to talk to plebbitors.