HELP! Quad RTX 3090, x2 EVGA 1600 G2: Burnt out +12v on MoBo power connector

[OneMoar]

Cables a bit on the thin side there looks like it was the 12v that went up ?
edit so yea didin't see he was running 4 3090's
there isn't many motherboard/psus combos that are going to put up with a 300W load on the PEG slots combined with powering everything else
the molex connector is only going to provide anouther 50-65W extra to all 4 slots

 

[quanash]

The issue is simple. Motherboard choice for 4 x RTX 3090 usage is wrong.

When you want to connect 4X RTX 3090 for heavy tasks like Deep Learning, you have to choose a motherboard that has a PCI-ex supplementary power instead of MOLEX.

Max. power support of MOLEX is way more less than PCI-ex Max. power support. If the GPUs power draw from PCI-ex slots exceed what MOLEX power cable can support, the needed power will be drawn from the motherboard 24 pin cable's 12V cable. The load will burn the cable in the end.

Similar issues have been around since the start of crypto mining frenzy. Saw several ( hundreds if not thousands ) burnt 24 pin PSU cables / PSU sockets because of insufficient 12V support to PCI-ex slots. Same is still valid for PCI-ex risers for crypto mining. PCI-ex power usage is recommended instead of MOLEX.

If you are going to continue to use those motherboard + GPU configuration, my suggestion will be reducing the number of the GPUs used on the motherboard. If you are going to look for another motherboard option, choose a model with PCI-ex power socket.

I have built hundreds of Deep Learning systems for companies, research centers and universities. None of them had such a problem since I'm choosing the components knowing what will be the real loads on each of them.

 

[xtreemchaos]

If theres ever a reason for 2 PSUs you have one. and why thay have put a molex connector on a board like that i dont know what thay are thinking. great rigg by the way its the stuff dreams are made of well done.

 

[quanash]

If theres ever a reason for 2 PSUs you have one. and why thay have put a molex connector on a board like that i dont know what thay are thinking. great rigg by the way its the stuff dreams are made of well done.

The reasoning is simple. While they were designing the TRX40 motherboards, they didn't foresee that next gen high end GPUs ( RTX 3090 ) were going to be this much power hungry. Those boards released before launch of RTX3000 series. Also, those motherboards are designed for gaming / creators. This means they are not specifically designed for multi GPU setups ( more than 2 GPUs, since SLI/crossfire has died ) to be used as workstation / GPU server.


WRX80, workstation variant of TRX40 is designed specifically for this kind usage and released after Ampere GPUs, so the board ASUS designed has 2x PCI-ex power socket for supplementary power to 7 PCI-ex slots. With this kind of power delivery, you can use 7 x RTX 3090s with the motherboard, if you water cool them.

You have to consider what was designed for what and what was launched before what, when you decide which components to use in your system and that system will be used for what purpose.

 

[thesmokingman]

very likely that has been the issue with your nice setup. going for aesthetics, you have to be very careful with all technical aspects too

just FYG, my FD flex is rated to 40w 3.33A.
Flex VRC-25 _Product sheet (fractal-design.com)
cable run length would be an issue for you though.

not to be alarmist, but they may have already burned out without showing through the insulation. just to be sure, do a simple continuity check on any common multimeter. they look like passive cables without additional electronic circuitry running on them.


ah, reminds me of the days i used to do quad CFX systems before going to solitary Titans.

I'll second the sentiment on aesthetics. Leave those third party cables for a show queen not for a production machine. I've run into issues with cable extensions in the past and avoid them like the plague. They often lead to random issues and it's just not worth it for the show on a production machine. That said, there is one option which are good to use and those are the custom sleeved cables made by the manufacturer like the EVGA full colored cable sets which are validated. I've used the EVGA custom cables with fine results. Those would be the only ones I'd use if I had to use one.

Build: 3970x, dual 2080ti, 8TB m.2 RAID = Render Monster

I'm building a render machine for work on a variety of applications. It will be an investment in reducing completion times in rendering and content creation workloads. This is a build for an old college buddy who works in the industry. On to the component list. Also, going to start with two...

hardforum.com

 

[Ominence]

6pin pcie is only rated to 75w. i was under the impression AMP connectors are good for 130w, 18AWG. Anyway, they can handle a ridiculous amount of power when correctly configured.
1-480424-0 : MATE-N-LOK Rectangular Power Connectors | TE Connectivity
794236-1 : MATE-N-LOK Rectangular Power Connectors | TE Connectivity
1-480426-0 : MATE-N-LOK Rectangular Power Connectors | TE Connectivity
even the Molex version is good for 75-80w, not to mention both rated for 250vac!
Molex Connector Part Number - 15-24-4742

WRX80, workstation variant of TRX40 is designed specifically for this kind usage and released after Ampere GPUs, so the board ASUS designed has 2x PCI-ex power socket for supplementary power to 7 PCI-ex slots. With this kind of power delivery, you can use 7 x RTX 3090s with the motherboard, if you water cool them.

just one of the reasons i was really looking for dual 8pin CPU power when going for my paltry 10c part! gives me a future upgrade path being the bigger reason.

as an aside, i remember some of my cheaper seasonic gold versions recommended not piggybacking the 2nd 6/8pin gpu combo power connector over a certain wattage. might have been 250w i think. in that case they wanted a 2nd 8pin pcie power to be run directly from the psu.

 

[OneMoar]

the Molex connector is not what failed replacing the wire/connector is not likely todo anything because thats not what got hot ffs
replacing the Molex connection is NOT going to magically make the motherboard take more power from the Molex and not from the 24p
as a note the POWER SUPPLY had the issue not the board the connector on the PSU its self was/is the weakest link I doubt EVGA has correctly allocated enough power to the 24P for this use-case

 

[looniam]

6pin pcie is only rated to 75w. i was under the impression AMP connectors are good for 130w, 18AWG. Anyway, they can handle a ridiculous amount of power when correctly configured.
1-480424-0 : MATE-N-LOK Rectangular Power Connectors | TE Connectivity
794236-1 : MATE-N-LOK Rectangular Power Connectors | TE Connectivity
1-480426-0 : MATE-N-LOK Rectangular Power Connectors | TE Connectivity
even the Molex version is good for 75-80w, not to mention both rated for 250vac!
Molex Connector Part Number - 15-24-4742


just one of the reasons i was really looking for dual 8pin CPU power when going for my paltry 10c part! gives me a future upgrade part being the bigger reason.

as an aside, i remember some of my cheaper seasonic gold versions recommended not piggybacking the 2nd 6/8pin gpu combo power connector over a certain wattage. might have been 250w i think. in that case they wanted a 2nd 8pin pcie power to be run directly from the psu.

intel's psu guide tells which exact parts to get:

 

[Ominence]

intel's psu guide tells which exact parts to get:

Molex ver. good for 8amps apparently 18AWG.
Microsoft Word - PS-8981-4M4P Rev D1.doc (molex.com)
it's interesting that 8981 references back to 8980 series.
the poor molex is planned for obsolescence!

@GerKNG

i had an AX1200i fail on me just over 12months after use. i hope corsair quality has improved since then.

 

[looniam]

the poor molex is planned for obsolescence!

lol naw, just go look at their catalogue (or more like get lost) and it will refer to the most recent part(s). those peripheral connector are referred as molex because they are the major supplier for anything w/connector mobo, gpu cards, psus.

they got more terminals rated from 9 to 13 amps than you can shake a stick at.

 

[delshay]

Would it not be better to run this under a single PSU. Not sure if this got released & I don't know how good it is. 2200W PSU Holy Smokes, EVGA Made A 2200 Watt Power Supply! CES 2018 - YouTube

 

[DeathtoGnomes]

I'll just drop this here for anyone, it doesnt chart down to 18 gauge. The 2nd chart might be more useful.

Watts = Volts X Amps OR Amps = Watts / Volts

 

[quanash]

6pin pcie is only rated to 75w. i was under the impression AMP connectors are good for 130w, 18AWG. Anyway, they can handle a ridiculous amount of power when correctly configured.
1-480424-0 : MATE-N-LOK Rectangular Power Connectors | TE Connectivity
794236-1 : MATE-N-LOK Rectangular Power Connectors | TE Connectivity
1-480426-0 : MATE-N-LOK Rectangular Power Connectors | TE Connectivity
even the Molex version is good for 75-80w, not to mention both rated for 250vac!
Molex Connector Part Number - 15-24-4742


just one of the reasons i was really looking for dual 8pin CPU power when going for my paltry 10c part! gives me a future upgrade part being the bigger reason.

as an aside, i remember some of my cheaper seasonic gold versions recommended not piggybacking the 2nd 6/8pin gpu combo power connector over a certain wattage. might have been 250w i think. in that case they wanted a 2nd 8pin pcie power to be run directly from the psu.

12V EPS power ( 8pin CPU ) NEVER shares power with PCI-ex slots. Those are ONLY for CPUs. If you have a high core count CPU and CPU heavy workload dual 12V EPS power connection just divides the power into 2 different channel ( cable and socket vise ), thus reduce the heat creation and melting down possibility of the 12V EPS sockets and give much more stabile power to the CPU, for a better run.

For RTX 3090 GPUs you have to use single cable for each Power socket, using piggy back / daisy chain cables are not recommended if you are going to use those GPUs for Deep Learning and Scientific calculations. For light/medium rendering or modeling purposes, you can use piggy back / daisy chain, there GPUs don't hit hard as other applications.

Also on LINUX environment, Nvidia drivers are not as strict as Windows drivers on power draw/efficiency. I saw much more power draw on LINUX than windows. This also should be considered while configuring the system.

the Molex connector is not what failed replacing the wire/connector is not likely todo anything because thats not what got hot ffs
replacing the Molex connection is NOT going to magically make the motherboard take more power from the Molex and not from the 24p
as a note the POWER SUPPLY had the issue not the board the connector on the PSU its self was/is the weakest link I doubt EVGA has correctly allocated enough power to the 24P for this use-case

Motherboard asked more 12V from 24-pin power cable once the supplied power from MOLEX wasn't enough. This request resulted in melt down on 12V cable and connection on the PSU. This is not PSU's fault. When you draw more power than initially designed for those melt downs are normal. This motherboard might have 4 PCI-ex slots, but this doesn't mean that it is electrically designed for 4 RTX 3090s which were launched after the launch motherboard.

If I was using this board or had to use, I wouldn't use more than 2 x RTX 3090 GPUs on it.

When this board was designed RTX 2080Ti TDP was around 250/260 WATT, but RTX 3090 FE TDP is 350 WATT ( The GPUs in use is around 425-450 WATT Sorry mixed it with STRIX, TUF's rated TDP is 350 WATT ).

TDP referance:

https://images.nvidia.com/aem-dam/en-zz/Solutions/design-visualization/technologies/turing-architecture/NVIDIA-Turing-Architecture-Whitepaper.pdf

https://images.nvidia.com/aem-dam/en-zz/Solutions/geforce/ampere/pdf/NVIDIA-ampere-GA102-GPU-Architecture-Whitepaper-V1.pdf

 

[Ominence]

12V EPS power ( 8pin CPU ) NEVER shares power with PCI-ex slots. Those are ONLY for CPUs.

it's funny when these connectors are not always keyed on some modular psus and to make matters worse, have different wiring on the psu end for the different terminals! really bad for unsuspecting people who may end up erroneously connecting the pcie to the cpu and vice versa- psu side. if you don't read the markings, could be in for a rude surprise.

 

[quanash]

it's funny when these connectors are not always keyed on some modular psus and to make matters worse, have different wiring on the psu end for the different terminals! really bad for unsuspecting people who may end up erroneously connecting the pcie to the cpu and vice versa- psu side. if you don't read the markings, could be in for a rude surprise.

Motherboard manufacturers are no different. Check this board layout; There are 2 12V EPS ( ATX_12V_1 & ATX_12V_2 ) for CPU power, but they are not the only one that powers the CPU. There is also a PCI-ex power that is being used as supplementary CPU power on the board ( PCIE_8PIN_PWR ). An untrained eye can mix this with a PCI-ex slot power and might not power it, or just power it and miss the other 2 PCI-ex power socket that originally gives the supplementary power to the PCI-ex slots ( PCIE_PWR_1 & PCIE_PWR_1 ). And ALL those shown with the same number ( number 5 ) to add extra confusion misunderstanding alongside with the 24-pin power.

I wish there was a standard on both PSUs, and motherboards about the power delivery.

P.S. : For whom want to know the model of the motherboard, it is ASUS Pro WS WRX80E-SAGE SE WIFI.

 

[OneMoar]

12V EPS power ( 8pin CPU ) NEVER shares power with PCI-ex slots. Those are ONLY for CPUs. If you have a high core count CPU and CPU heavy workload dual 12V EPS power connection just divides the power into 2 different channel ( cable and socket vise ), thus reduce the heat creation and melting down possibility of the 12V EPS sockets and give much more stabile power to the CPU, for a better run.

For RTX 3090 GPUs you have to use single cable for each Power socket, using piggy back / daisy chain cables are not recommended if you are going to use those GPUs for Deep Learning and Scientific calculations. For light/medium rendering or modeling purposes, you can use piggy back / daisy chain, there GPUs don't hit hard as other applications.

Also on LINUX environment, Nvidia drivers are not as strict as Windows drivers on power draw/efficiency. I saw much more power draw on LINUX than windows. This also should be considered while configuring the system.



Motherboard asked more 12V from 24-pin power cable once the supplied power from MOLEX wasn't enough. This request resulted in melt down on 12V cable and connection on the PSU. This is not PSU's fault. When you draw more power than initially designed for those melt downs are normal. This motherboard might have 4 PCI-ex slots, but this doesn't mean that it is electrically designed for 4 RTX 3090s which were launched after the launch motherboard.

If I was using this board or had to use, I wouldn't use more than 2 x RTX 3090 GPUs on it.

When this board was designed RTX 2080Ti TDP was around 250/260 WATT, but RTX 3090 FE TDP is 350 WATT ( The GPUs in use is around 425-450 WATT Sorry mixed it with STRIX, TUF's rated TDP is 350 WATT ).

TDP referance:

https://images.nvidia.com/aem-dam/en-zz/Solutions/design-visualization/technologies/turing-architecture/NVIDIA-Turing-Architecture-Whitepaper.pdf

https://images.nvidia.com/aem-dam/en-zz/Solutions/geforce/ampere/pdf/NVIDIA-ampere-GA102-GPU-Architecture-Whitepaper-V1.pdf

you mis understand it melted on the PSU end of the cable which SHOULD BE the point of least resistance in the circuit, (shorter distance from the LOWSIDE rectifiers to the failure point)

normally you would see this type of failure on the motherboard side where the resistance is higher (further away from the lowside of the rectifiers where the resistance is higher)
the further away from the source you get the higher the resistance the more the voltage droops > the more amperage you draw

I am wondering if the solder joints on the custom cables or bad fitment on the connectors output once it started getting hot the resistance skyrocketed leading to a run-away condition

in a overdraw condition like this that type of failure tends to indicate where the problem was (weakest link) had it been a true overdraw scenario where there was simply just to much power required it should have failed on the motherboard side so that being said it failed at the point where the connection was the weakest which by all rights should not have been directly off the psu


The motherboard probably has some current limiting resistors between the Molex and the PCIE power to prevent thing kind of failure from happening there and taking out thousands of dollars worth of gpu its better to burn the board or psu up then to kill relatively expensive hardware

my money is on the cables being incorrectly crimped/soldered or shotty connectors with loose tolerances

now as to weather or not the board can handle 300W of load on the PCIE slots thats something ASUS should be-able to answer

 

[quanash]

you mis understand it melted on the PSU end of the cable which SHOULD BE the point of least resistance in the circuit, (shorter distance from the LOWSIDE rectifiers to the failure point)

normally you would see this type of failure on the motherboard side where the resistance is higher (further away from the lowside of the rectifiers where the resistance is higher)
the further away from the source you get the higher the resistance the more the voltage droops > the more amperage you draw

I am wondering if the solder joints on the custom cables or bad fitment on the connectors output once it started getting hot the resistance skyrocketed leading to a run-away condition

in a overdraw condition like this that type of failure tends to indicate where the problem was (weakest link) had it been a true overdraw scenario where there was simply just to much power required it should have failed on the motherboard side so that being said it failed at the point where the connection was the weakest which by all rights should not have been directly off the psu


The motherboard probably has some current limiting resistors between the Molex and the PCIE power to prevent thing kind of failure from happening there and taking out thousands of dollars worth of gpu its better to burn the board or psu up then to kill relatively expensive hardware

my money is on the cables being incorrectly crimped/soldered

I didn't misunderstand it. When you use extension cables with your PSUs, you can see the damage on PSU side instead of motherboard 24 pin.

If there was any issue on the extension cables, he would have seen system instability and shutdowns before the melting happened, when he loaded the system. Both CPU and GPU workloads were fine until the melting happened, thus I don't think there was any connection issue.

 

[Ominence]

P.S. : For whom want to know the model of the motherboard, it is ASUS Pro WS WRX80E-SAGE SE WIFI.

i like the commentary in the manual there- for stability connect moar connectors!

 

[OneMoar]

I didn't misunderstand it. When you use extension cables with your PSUs, you can see the damage on PSU side instead of motherboard 24 pin.

If there was any issue on the extension cables, he would have seen system instability and shutdowns before the melting happened, when he loaded the system. Both CPU and GPU workloads were fine until the melting happened, thus I don't think there was any connection issue.

No not necessarily it depends on what on the GPU's is being powered by the PCIE bus on anything high end normally the PCIE Bus power is isolated and only powers auxiliary circuits(basic stuff required to boot the card to a point where the External power kicks in and the core is turned on) sometimes the vram or a part of a AUXILIARY Phase but they are separate circuits because obviously there isn't enough juice there to run anything substantial tho on the 3090 I don't even think any of the vram gets its power from there because of the power requirements

remember he is not gaming he is crunching numbers and that workload is far more fault tolerant then running a game that workload is also famous for massive swings in power consumption and that was likely a factor here
hes also got riser cables too so by all rights the connection to the motherboard from the gpus is a worst case it should have failed there
@buildzoid do you know what the power split is on the 3090 whats powered by slotpower whats powered by PCIE-8pin I know a 3090 draws any ware from 60-70w from PEG_12v

 

[Ominence]

If there was any issue on the extension cables, he would have seen system instability and shutdowns before the melting happened, when he loaded the system. Both CPU and GPU workloads were fine until the melting happened, thus I don't think there was any connection issue.

not that OP was using a benchmark PSU but hopefully reverting to at least the stock 24pin cable may give better unburnt results due to firmer connector fit and correct AWG usage.

 

[newtekie1]

ou mis understand it melted on the PSU end of the cable which SHOULD BE the point of least resistance in the circuit, (shorter distance from the LOWSIDE rectifiers to the failure point)

It actually melted at both sides, the motherboard side and the PSU side.

When the 24-pin spec was created, it was designed for one high power PCI-E card pulling 75w. The extra 4-pins added were just for the PCI-E bus, but 3 are ground and only 1 is an extra 12v. That means there are only 2 12v pins powering the PCI-E bus. If you put 4 GPUs on the PCI-E bus, all pulling from just those 2 pins, they are bound to melt under high load. The extra 4-pin connector at the bottom of OPs board is supposed to help provide 50% more power capability to the PCI-E bus, but that still probably isn't going to be enough for 4 extreme draw GPUs. Each card is allowed to draw 75w from the bus, that's 300w total from just 3 12v pins. This is a fault of the motherboard and it's design, not the PSU.

 

[quanash]

No not necessarily it depends on what on the GPU's is being powered by the PCIE bus on anything high end normally the PCIE Bus power is isolated and only powers auxiliary circuits(basic stuff required to boot the card to a point where the External power kicks in and the core is turned on) sometimes the vram or a part of a AUXLITARY Phase but they are separate circuits

remember he is not gaming he is crunching numbers and that workload is far more fault tolerant then running a game that workload is also famous for massive swings in power consumption and that was likely a factor here

This is true depending on the operating system, workload and timeline.

But you should consider OP's comment below;

"
I just built a custom rig for Deep Learning and after 2-3 months of constant use, I noticed the 2 +12v lines from PSU to MoBo have burned up.
"

2-3 months of constant use for Deep Learning means hammering down both the GPU and CPU way more worse while gaming. Also as I said before, you are not in Windows environment while using a system for Deep Learning. In LINUX, Nvidia GPUs are not tightly power regulated as Windows in terms of driver control on power states/usage.

 

[OneMoar]

It actually melted at both sides, the motherboard side and the PSU side.

When the 24-pin spec was created, it was designed for one high power PCI-E card pulling 75w. The extra 4-pins added were just for the PCI-E bus, but 3 are ground and only 1 is an extra 12v. That means there are only 2 12v pins powering the PCI-E bus. If you put 4 GPUs on the PCI-E bus, all pulling from just those 2 pins, they are bound to melt under high load. The extra 4-pin connector at the bottom of OPs board is supposed to help provide 50% more power capability to the PCI-E bus, but that still probably isn't going to be enough for 4 extreme draw GPUs. Each card is allowed to draw 75w from the bus, that's 300w total from just 3 12v pins. This is a fault of the motherboard and it's design, not the PSU.

I don't think ASUS ever expected anybody to use this board in this manner(With SLI now being limited to 2GPU's)
its a consumer grade board he needs to talk to supermicro and get a enterprise grade board he didn't state that it burned at both ends in the op

I don't know if i would trust a molex connector with 150W in any scenario on paper its a 156W but thats on paper and likely no modern psu is going to be cabled for the full brunt of that because molex is generally relegated to low power devices like optical drives or usb cards

as for Linux's power management sucking ... go figure
but what we have here is a bunch of factors leading to spectacular failure and and I think the proverbial straw in this case was the cabling

 

[quanash]

I don't think ASUS ever expected anybody to use this board in this manner(With SLI now being limited to 2GPU's)
its a consumer grade board he needs to talk to supermicro and get a enterprise grade board he didn't state that it burned at both ends in the op

I don't know if i would trust a molex connector with 150W in any scenario on paper its a 156W but thats on paper and likely no modern psu is going to be cabled for the full brunt of that because molex is generally relegated to low power devices like optical drives or usb cards

as for Linux's power management sucking ... go figure
but what we have here is a bunch of factors leading to spectacular failure and and I think the proverbial straw in this case was the cabling

Not only the motherboards, but also the GPUs.

I prefer to use a GPU specifically designed for Workstation / GPU server usage. Such as ASUS Turbo / MSI Aero / Gigabyte Turbo Series RTX 3090 GPUs.

Those might be slower than a gaming GPU, but they are much more better choice when it comes to heavy loads like rendering and Deep Learning. Their BIOS' are configured for this purpose and they are not as aggressive as gaming versions, since they are needed for stability instead of OC'ing. Their tamed nature also prevents any kind of spikes and also is good to be used in Linux.

 

[Ominence]

I prefer to use a GPU specifically designed for Workstation / GPU server usage. Such as ASUS Turbo / MSI Aero / Gigabyte Turbo Series RTX 3090 GPUs.

ah, blower fans, love them, automatic case heat ejectors!! i would love to have a 3 slot version, vapo chambered off course.