AMD Radeon Pro v540 Research Thread

[mczkrt]

Even now there's only been success with a single of the two GPUs on board so whether that second one ever becomes usable is a gamble too.

To set the record straight, I have a setup running on AWS drivers with both GPUs operational individually, and confirmed working in Vulkan mGPU in Gravitymark. I don't know of any other app or benchmark that use mGPU under Linux to test it further.

 

[PixelButts]

To set the record straight, I have a setup running on AWS drivers with both GPUs operational individually, and confirmed working in Vulkan mGPU in Gravitymark. I don't know of any other app or benchmark that use mGPU under Linux to test it further.

Really it was both working? That's news to me.
I'll set out to make mine work (maybe) this week but its going to be a rough setup. I dont have the Linux drivers on hand unfortunately, I deleted my AWS account before I snagged em like an idiot. Best I can try as a result is the Windows version.

 

[mczkrt]

That is true.

mGPU works. But it looks like card is in some kind of debugging mode, because power rarely crosses 60W per core. I inspected the BIOS with MorePowerTools, and limit is set at 130W per core, with 105C temp target. And it's not a global power target, as single GPU also works at about 60W.

View attachment 297683

 

[PixelButts]

Update on my end. While Windows accepted the LTT drivers and the 2019 drivers it outright would not accept the 2016/Archive drivers, so guess those are for sure server-only. All I could get was OpenGL. I'm now trying just a normal linux install and seeing if I can get any life out of it there. I dug up my b550 board, the thinkstation PSU I got off the side of the road, and my trusty 750ti to give me some video output while I mess with it.
Shoutouts to my rubber bands securing the fans to keep it cool

Update, no success on my end unfortunately with Linux either, so I'm calling it quits for the mean time

 

[LabRat 891]

However, as mczkrt said, no inf for it is present in the R.ID driver packages

Correct.
The way their drivers work, it changes the name and has some kind of 'in-between' deviceID-emulation in-software.


Example:
A "WX 9100" (Former MI25) Installed using the 'gaming' settings, is ID'd as an RX Vega 64 Series

Conversely, My "WX 9100" and RX 6500 XT both are renamed to Radeon Instinct MI25 and Radeon Pro W6500 respectively, when using the 'enterprise' settings.

So, any Navi 12 GPU 'should' get re-named once the R.ID installer detects the card.
The .INF used by Windows and installed by the R.ID installer, will not 'list' the actual device, more often than not.

The issue w/ Navi 12, is how unreleased it was; I'm not sure what deviceIDs it might use in emulation.

Regardless, under normal circumstances using DevMgr,
you cannot 'force install' R.ID drivers; they will not work as advertised/expected.
^I have tried to do exactly this, as part of getting Vega+Navi AFMF working.

Speaking of AFMFing non-Navi2x/3x cards...
Has anyone tried to 'pair' a Navi 12 w/ another more-supported card, and had the driver(s) pass frames over PCIe to the better-supported card?
Windows 11 can globally set a 'rendering GPU' while retaining whatever display outputs (on whatever other card) is/are in use.

 

[mczkrt]

Yes, I figured that the final .inf must be dynamically generated based on the settings in the installer. Still, if no corresponding device id is present in base .inf then no driver will be built.

 

[PixelButts]

Fair enough, I'm no expert on this as it is so at least I'm learning. Personally I'm more curious what's preventing the Windows drivers from working as expected. Installation itself is fine but still needing manually adding the drivers seems weird, the device itself reports as working fine but that's clearly not true, especially given the result being only OpenGL being supported, so surely something else is needed or some setup is wrong? Maybe some special driver is on the AWS instances that isn't in the driver package as well?

 

[mczkrt]

I figured out it is relatively straightforward to have working 3D rendering when output is set to Virtual Display, which you can then view via Xpra/XRDP on a different machine. And that is with Ubuntu 22.04 and 23.20.00.48 drivers from AMD website. But you can't use any other graphics card at the same time.

What I can't get past, is getting DRI_PRIME offloading to work. As soon as I connect another card with a display (Does not matter if it is Nvidia or AMD), its OpenGL renderer is the only one available in the system, and no amount of manipulation of Xorg or DRI allows me to offload rendering to V540. But this might be more of a Linux quirk than this card fault. If anyone has experience with this, I would welcome any input.

If we manage to clear this, I will have a short guide ready.

 

[PixelButts]

I figured out it is relatively straightforward to have working 3D rendering when output is set to Virtual Display, which you can then view via Xpra/XRDP on a different machine. And that is with Ubuntu 22.04 and 23.20.00.48 drivers from AMD website. But you can't use any other graphics card at the same time.

What I can't get past, is getting DRI_PRIME offloading to work. As soon as I connect another card with a display (Does not matter if it is Nvidia or AMD), its OpenGL renderer is the only one available in the system, and no amount of manipulation of Xorg or DRI allows me to offload rendering to V540. But this might be more of a Linux quirk than this card fault. If anyone has experience with this, I would welcome any input.

If we manage to clear this, I will have a short guide ready.

So essentially if you can make a full remote-only system with no other video output, it'll potentially just work? I might have a solution then; I'll give this a try.

Trying on my end once more. Windows 11 this time, and the drivers seem to take no problem. Trying to install Parsec (because it uses the hardware if it works) seems to completely ruin the drive creating an inaccessible boot device which I find a little funny honestly. Will continue trying now that I have a more reasonable plan to do this. I just wish I had more than a USB HDD to do it because wow is it slow

Doublepost, bad form (and like 8 edits), I know, but I did get some interesting results.

Setup is as follows

• A TUF B550 board
• Windows 11 Pro (nothing special just got it from MS
• Server 2019 drivers
• Installed via the pnputils method as AWS describes, but manually applied the drivers to the devices
• Used one of my (12) v320 (Stadia) cards as a placeholder while I configure a remote desktop setup, I then used some basic firepro due to convenience

How this played out

• Setup went as expected on a USB HDD with Windows To Go (it's bad)
• Installing the drivers manually was correctly handled, no issues reported when they were added
• Rebooted, checked and they reported error 43, whatever no biggie
• Removed v320/Firepro and rebooted
• White light being on made me wonder if it successfully POSTed, it did so alright
• Now both v520 entries report error code 12, which I have not personally seen on this yet

Interesting result to say the least. I then tried to install the PCI Memory controller because it was just bugging me to no end and it seems to be fine I guess? Nothing seems to change there.
Any suggestions on going from here? I'm open to some ideas. I would test parsec, but seeing as code 12 kinda prevents that (and I keep getting inaccessible boot device after install, likely due to this being Windows To Go on a garbage USB HDD), so RDP is the best option I have here.

 

[mczkrt]

I have seen error 12 straight away after installing drivers on X58 platform. I don't think it indicates any kind of progress.

Ok, I managed to get render offload to work, but how... no clue. It did not work in any way with display on Nvidia card, not with proprietary drivers or open source. So I swapped it for RX480 and it started to work straight away without touching anything. But it did not work on previous OS install, when both RX480 and V540 were connected during setup.

I will try to write a guide and maybe with all steps documented, it will be more clear how to arrive at working setup.

 

[Rdiablo]

I find myself ruminating on whether our concerted endeavours on Windows may be somewhat misdirected. The AWS driver, it appears, invariably necessitates compulsory installation. For the V520, I have successfully identified an appropriate driver, sourced from Bootcampdriver - the W11-Radeon-Software-Adrenalin-Oct23-BootCamp-Red-Edition. The initial installation might require manual intervention, given that it identifies the V520 MxGPU, which deviates from our intended aim. During the manual installation process, one should opt for the AMD Radeon PRO Graphics. Upon installation completion, the graphics card is expected to function seamlessly.

However, the Adrenalin Edition might still display irregularities. In such instances, we should proceed with rerunning the BootCampDriver installation program, subsequent to which the Adrenalin Edition should regain its normal functionality.

However, the mini-DP still remains non-operational.

From the data presented by GPU-Z, it is discernible that this driver is more suitably tailored for this card. Of course, the aforementioned driver compatibility pertains to the V520; I am without a V540 graphics card. Intriguingly, the V540 displays 'made in China', and yet, as a Chinese national, I find myself incapable of procuring this graphics card within China's borders.

If the above information proves beneficial to you, I wonder if it would be possible to assist me in purchasing this card and having it shipped to me? Rest assured, I am fully prepared to bear all associated costs.

 

[mczkrt]

I would say that the key lies in vBIOS, because once you flash image from a more mainstream card, windows drivers will install just fine - albeit with limited clocks (5600M Pro) or only for a single boot (BC160). There is a mention in this thread about working vBIOS built in 2023, but conversation was strictly in mining context, so we have no information how it behaved in Windows.

I did try modifying .inf from Bootcamp drivers and running as an unsigned driver, this made no difference on stock vBIOS.

 

[PixelButts]

I did try modifying .inf from Bootcamp drivers and running as an unsigned driver, this made no difference on stock vBIOS.

There also the fact a number of these cards have different vBIOS on them, making it harder.

 

[mczkrt]

Is that true? Those posted in this thread look all the same.

 

[PixelButts]

Seems I'm misremembering then. I saw this post here https://www.techpowerup.com/forums/threads/amd-radeon-pro-v540-research-thread.308894/post-5097015 and Thought it came from this card, but seems it came from a normal v520

There were some interesting posts on page 3 and 4 though

AMD Radeon Pro v540 Research Thread

the blue connectors have the same amount as my firepro w8100 sink connector i also bought the v540 and verified it, its even in same order the blue connectors have the same amount as my firepro w8100 sink connector i also bought the v540 and verified it, its even in same order also i have a...

www.techpowerup.com

AMD Radeon Pro v540 Research Thread

Despite no forthcoming opinions, I went on flashing adventure anyway. It turns out that card can be made to work in Windows, however daily use is out of question (at present). Have CH341A with add-on clip on hand. How to get there: - Have some way to flash a vbios from software. I used my...

www.techpowerup.com

 

[mczkrt]

Here is a promised guide, I have personally repeated it 2 times with positive results, but I give no guarantees that it will work in every case.

Purpose of this guide is to showcase 3D rendering capabilities of V540 under Linux OS, in a most straightforward way, using freely available software.

It is possible that hardware compatibility is very limited, and this guide might not work on a notably different system.

Hardware prerequisites:​

-AM4 platform, preferably with Zen2 CPU or newer.

-AM5 is confirmed as working.​

-No info about running on modern Intel platform.​

-2nd GPU with a display connected, preferably with AMD Polaris chip or newer.

-Intel integrated or ARC might work, not tested.​

-Nvidia probably won't work.​

-Cooling solution for V540 that will keep it from overheating for at least 30 mins.

BIOS setup:​

-Above 4G decoding - on
-Resizable BAR - off
-IOMMU - off
-SR-IOV - off (thanks PixelButts)

Software:​

Kubuntu 22.04 LTS Desktop (other desktop managers might work, not tested)

https://cdimage.ubuntu.com/kubuntu/releases/22.04.3/release/kubuntu-22.04.3-desktop-amd64.iso

AMD Linux drivers installer

https://repo.radeon.com/amdgpu-install/23.30.3/ubuntu/jammy/amdgpu-install_5.7.50703-1_all.deb

1. OS installation​

Choose following options during setup:

-Minimal​

-Download updates​

-3rd party drivers (optional)​

-log in automatically​

2. Basic OS setup​

Open Konsole and run:

sudo -i
apt update
apt upgrade
reboot

3. Downgrade kernel to GA​

Open Konsole and run:

sudo -i
apt install linux-image-generic linux-headers-5.15.0-92-generic
apt remove --purge linux-generic-hwe-22.04 linux-hwe-6* linux-modules-6* linux-modules-extra-6*
update-grub
reboot

Make sure packages image and headers are of the same version.
Select No at the dialog screen.

5.15 kernel without drivers will throw errors after reboot and look frozen, be patient.

4. Install drivers​

Open Konsole and run: (or use Firefox to download AMD installer)

cd ~/Downloads
wget https://repo.radeon.com/amdgpu-install/23.30.3/ubuntu/jammy/amdgpu-install_5.7.50703-1_all.deb
sudo apt install ./amdgpu-install_5.7.50703-1_all.deb
sudo amdgpu-install --usecase=dkms,graphics,multimedia
sudo reboot

OS will probably hang during shutdown, hard reset here.

5. Attaching virtual display​

Open Konsole and run:

lspci

Look for Navi12 Display controller and note the bus id of 1st GPU (lower number)
Like:

0a:00.0 Display controller: Advanced Micro Devices, Inc. [AMD/ATI] Navi 12 [Radeon Pro 5600M / V520] (rev c1)
0d:00.0 Display controller: Advanced Micro Devices, Inc. [AMD/ATI] Navi 12 [Radeon Pro 5600M / V520] (rev c1)

0a:00.0 is our <bus id>

sudo -i
nano /etc/default/grub

Look for GRUB_CMDLINE_LINUX_DEFAULT="quiet splash" and add after splash amdgpu.virtual_display=0000:<bus id>,1
It should look like this:

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash amdgpu.virtual_display=0000:0a:00.0,1"

Ctrl+X then y

update-grub
reboot

6. Install tools​

Open Konsole and run:

sudo apt install switcheroo-control lm-sensors
switcherooctl

Your display card should be listed as Device:0 and two Navi12 devices should be listed as 1 and 2.

switcherooctl -g 1 glxgears -info

Scroll up in the console window, GL_RENDERER should say 7360:c1 (navi12 etc.)
Close the window with gears or Ctrl+C in the console.

7. Benchmarks​

Download via Firefox Unigine Heaven and Superposition.
Open Downloads folder, right click on each installer, open Properties and in Permissions tab click checkbox: Is Executable
Double click installer and Execute
In Konsole run:

cd ~/Downloads/Unigine_Heaven-4.0/
switcherooctl -g 1 ./heaven

Open new Konsole window and run

watch -n 1 sensors

Now run Heaven benchmark, preferably not fullscreen and with lower resolution. In console with running sensors You should see that one of navi12 GPUs and your display are under load. You can also compare FPS when running Heaven directly, without switcherooctl.

Same procedure with running Superposition.

I hope this guide will help someone and I'm curious if it's even valid on Intel platform.

 

[PixelButts]

Oh hell yeah. I will try this tomorrow after work, as I have all the required components for this and would be able to do all of this.
I also have an ARC card, will also solve that ARC question for you along the way, though considering that needs kernel 6.x I have my doubts.

 

[mczkrt]

6.5 is not supported by AMD installer yet, but 6.2 is and is installed by default.

In step 3 you can try:

sudo -i
apt remove --purge linux-hwe-6.5* linux-image-6.5* linux-headers-6.5* linux-modules-6.5* linux-modules-extra-6.5*
update-grub
reboot

 

[PixelButts]

Following your instructions and I find myself getting this error thrown at me. Ignore the scsi stuff. Looking into it I checked my bios settings.

I turned IOMMU to auto, previously disabled. I also turned SR-IOV to disabled, previously enabled. Error went away. Went back and turned off IOMMU again as a sanity check, kept SR-IOV disabled this time, error still gone. Guess SR-IOV was causing it to fail init which yeah ok I guess.

For sanity check my BIOS is currently

• SR-IOV - off
• Above 4G decoding - on
• Resizable BAR - off
• IOMMU - off

I also made a few typos so I triple checked my work this time, turns out I made more than one typo in the grub config. Be sure not to mess up those : and .
Anyway, result worked it does in fact render the gears through it.

Can confirm it does work so far in heaven as well.

Decided to also try RPCS3, since I'm more familiar with this, and sure enough it lets you select one of them.

Ran lshw just for good measure to see if any are unclaimed and looks like no

Also tried out DMCV, but it defaulted to the RX580, so I needed Proton and the following command: DRI_PRIME=1 DXVK_FILTER_DEVICE_NAME=NAVI12 %command%
I also noticed some artifacting. Cant tell whether this was due to refresh rate or anything, but I did turn on vsync and cap to 60hz and the artifacting went away. It's possible the RX580 cant keep up with the video output? it's currently in a 4x slot which probably isn't helping.

Not sure what's goin on with the displays though, kinda weird but given the lshw output I guess that's fair?

Honestly I genuinely appreciate your effort mczkrt, hopefully this gives a little more validation. Any suggestions for checking raw compute? Only one I have is giving me funky results that I dont trust for a second.

 

[mczkrt]

Great find re SR-IOV, I will try to edit the guide.

Virtual display shouldn't be needed for render offload, but I couldn't get OpenGL to work otherwise - only one renderer was detected. Probably more of a Linux quirk. But if you were to connect remotely via Xpra, it would open separate window for each display.

For compute you have to modify --usecase, like:

sudo amdgpu-install --usecase=dkms,graphics,multimedia,opencl,rocm,hip

(not tested)

More info:

Installation via AMDGPU installer — ROCm installation (Linux)

Installation via AMDGPU installer

rocm.docs.amd.com

 

[Solaris17]

Great find re SR-IOV, I will try to edit the guide.

time limit has been unlocked.

 

[PixelButts]

Great find re SR-IOV, I will try to edit the guide.

Virtual display shouldn't be needed for render offload, but I couldn't get OpenGL to work otherwise - only one renderer was detected. Probably more of a Linux quirk. But if you were to connect remotely via Xpra, it would open separate window for each display.

For compute you have to modify --usecase, like:

sudo amdgpu-install --usecase=dkms,graphics,multimedia,opencl,rocm,hip

(not tested)

More info:

Installation via AMDGPU installer — ROCm installation (Linux)

Installation via AMDGPU installer

rocm.docs.amd.com

Much appreciated. For context my buddy was interested in SR-IOV and vGPU stuff and insisted I keep trying until I can verify some cards of mine were capable of it. Left the setting on ever since as I moved motherboards after that point. Honestly that's probably why I had trouble getting those Stadia v320 cards booting in that motherboard...
For now I think I'll leave it alone for now and buy a proper case, print the fan mount and get that blower fan so I can properly use it. I dont exactly like having that much electricity and wires spread out along my floor.

 

[Catch2223]

Looks like some progress has been made since I last checked in!

On a hardware front, I gave the cooler re-design some more thought so I could make something that respected the airflow direction and could be powered directly by a motherboard header.

I went back to a tube axial fan so that we can still mount the fan on the rear of the GPU but get the right airflow direction. It is spec'ed to work with motherboard headers to simplify running the fans. This fan is LOUD though. If you want a quiet solution the backwards blower is going to be better. If you want to use everything the "right way" then this is the better option. The temps are okay with this cooler. Nothing is going to overheat but you will be running than fan at 100% if you get it under 100% load. If you are not heavily loading the GPU you can get away with much lower percentages. This is the largest fan I can fit if the GPUs are to be able to fit side by side on a standard motherboard.

Sadly, this design also requires some additional hardware meaning you will need 2 M3x35mm screws. These can be sourced from McMaster Carr, Amazon, or your local hardware store most likely. As for interfacing with a motherboard you will need to crimp the contacts onto the leads! I took my time and crimped the contacts with my Leatherman and it really wasn't bad. This is a PWM fan so it is 4 wire making speed control simple if you go this route.

Here is the shopping list:
1x 3D printed manifold - Files attached! Figure it out
1x Fan - https://www.digikey.com/en/products/detail/sanyo-denki-america-inc/9GA0412P3K01/6192260
4x Contacts - https://www.digikey.com/en/products...1500?s=N4IgTCBcDaIOoFkCMA2ArEg7AWgHIBEQBdAXyA
1x Connector Housing - https://www.digikey.com/en/products...405319?s=N4IgTCBcDaIOoFkDMYAcBWAtAOQCIgF0BfIA
2x M3x35MM Socket Head Cap Screw - https://www.mcmaster.com/91290A135/

NOTE: When printed depending on the printer the holes for the screws may need to be drilled out to final size. Tapping is not required if you get it right. 7/64 bit seemed to work best for me.

Final look:

All in all I personally like this solution much better than my first blower design. It works very well and is not hard to assemble. However, it is loud...

Hope you guys enjoy and as I said before if someone wants this cooler as a kit I am happy to ship you one at cost! Happy hacking!

 

[Catch2223]

Alright guys interesting development here.

One, these are back on eBay for a normal price from a seller here:
Buy it now
Auction

Two, go look at the photos. These do not dawn the "Engineering Sample" sticker all of the other cards seem to have. This raises the question, were these cards released??? Anyone have more info? I wonder if the vbios is the same on these cards??? They even have a back plate?! What!

 

[PixelButts]

Alright guys interesting development here.

One, these are back on eBay for a normal price from a seller here:
Buy it now
Auction

Two, go look at the photos. These do not dawn the "Engineering Sample" sticker all of the other cards seem to have. This raises the question, were these cards released??? Anyone have more info? I wonder if the vbios is the same on these cards??? They even have a back plate?! What!

Seems the buy it now one is gone already