# New Evidence of Hacked Supermicro Hardware



## Jetster (Oct 9, 2018)

The more recent manipulation is different from the one described in the Bloomberg Businessweek report last week, but it shares key characteristics: They’re both designed to give attackers invisible access to data on a computer network in which the server is installed; and the alterations were found to have been made at the factory as the motherboard was being produced by a Supermicro subcontractor in China.

https://www.bloomberg.com/news/arti...ro-hardware-found-in-u-s-telecom?srnd=premium


----------



## Flanker (Oct 10, 2018)

Interested to know which OEM manufactures their boards. The largest electronics OEM in China are not really Chinese companies


----------



## Salty_sandwich (Oct 10, 2018)

The plot thickens …. wonder what will come of this?


----------



## FreedomEclipse (Oct 10, 2018)

Flanker said:


> Interested to know which OEM manufactures their boards. The largest electronics OEM in China are not really Chinese companies



Foxconn possibly. But at the same time i dont think it would be them because they are a huge company and they have many many big million dorrah contracts from Apple, Samsung, Sony, Dell... the list goes on.


----------



## Solaris17 (Oct 10, 2018)

This doesn't make alot of sense. Still 0 proof. No white papers or disassembly.

And most of all. You cant NOT see the network traffic.

You are telling me these companies went 3 years with super micro devices connected to business critical infra and did not see a peep in wireshark, monitoring software, edge firewalls, transport logs?

please.

meanwhile all of my supermicro servers are quite as ghosts when put on private lans and analyzed.


----------



## Xzibit (Oct 10, 2018)

Solaris17 said:


> This doesn't make alot of sense. Still 0 proof. No white papers or disassembly.
> 
> And most of all. You cant NOT see the network traffic.
> 
> ...



Doubt Sepio Systems would be so public about it if it wasnt the case.



> *Three security experts who have analyzed foreign hardware implants for the U.S. Department of Defense confirmed that the way Sepio's software detected the implant is sound.* One of the few ways to identify suspicious hardware is by looking at the lowest levels of network traffic. Those include not only normal network transmissions, but also analog signals -- such as power consumption -- that can indicate the presence of a covert piece of hardware.


----------



## Frick (Oct 10, 2018)

FreedomEclipse said:


> Foxconn possibly. But at the same time i dont think it would be them because they are a huge company and they have many many big million dorrah contracts from Apple, Samsung, Sony, Dell... the list goes on.



I read (can't remember if it was in the original piece or someones opinion about it) that they subcontract to smaller players when they have increased demands and not enough capacity to go around. So not Foxconn.



Solaris17 said:


> This doesn't make alot of sense. Still 0 proof. No white papers or disassembly.
> 
> And most of all. You cant NOT see the network traffic.
> 
> ...



Honest question: how would they be able to tell? I don't know how enterprise network security works, but given that a lot of them are hacked to begin with, or host stuff, how wold they know? I assume they have automatic systems in place; how do they tell nefarious connections from normal activity?


----------



## Xzibit (Oct 10, 2018)

Frick said:


> Honest question: how would they be able to tell? I don't know how enterprise network security works, but given that a lot of them are hacked to begin with, or host stuff, how wold they know? I assume they have automatic systems in place; how do they tell nefarious connections from normal activity?



From the article



> In the case of the telecommunications company, *Sepio's technology detected that the tampered Supermicro server actually appeared on the network as two devices in one. The legitimate server was communicating one way*, and the implant another, *but all the traffic appeared to be coming from the same trusted server*, which allowed it to pass through security filters.


----------



## Solaris17 (Oct 10, 2018)

Frick said:


> I read (can't remember if it was in the original piece or someones opinion about it) that they subcontract to smaller players when they have increased demands and not enough capacity to go around. So not Foxconn.
> 
> 
> 
> Honest question: how would they be able to tell? I don't know how enterprise network security works, but given that a lot of them are hacked to begin with, or host stuff, how wold they know? I assume they have automatic systems in place; how do they tell nefarious connections from normal activity?



It really depends. and for that I will water it down not because I dont think anyone will get it but because its easier for me to do with all the thoughts swimming in my head.

The traffic is probably encrypted.

This is fair and means that your right they wouldnt be able to "see" that it went to www.china.com

However.

Its the practice that makes me so sceptical and there are many sides and moving parts to that.

1: These are high profile companies (makes sense for a supply chain attack right?)  like *google* alot of carrier grade companies customize the BIOS of these servers.

2: These companies arent just "ISPs" they are TRANSPORT providers they peer (move) traffic between sub sea cables and route between carrier (ISP) networks.

3: Systems that get dropped into this type of environment are stringently tested they do not just buy servers and switches and throw up a new data center equipment buildouts just to handle a small area in say your city cost millions in planning and architecting.

4: Suppose even if it were true and even if the data were encrypted, the server itself does not do routing. supermicro does not make Cisco 9508 core network racks. These servers probably cover a multitude of pourposes and might even be just a small server part of a much larger node or cluster that actually holds data. This is important because security is on everyones mind in the network/admin field right now. These servers are behind managmeent VLANs and are only permitted access to specific things.

With that said. The job of any carrier is to transport packets. It is a common misconception that a "good ISP" will protect me from bad guys are bad things. (maybe not at TPU but you would be surprised) that is not the case. A carrier network moves traffic lots of it. Whats important about this though is that rightly so, they would not "block" this traffic from going to any country its that if it originates on the "servers" it probably wont get to see the light of day.

Like I said while carriers do not generally employ any kind of blocking on the carrier level these servers are protected assets. They are only allowed to communicate with this or that network, they are also only accessible via specified VLANs and OOB (out of band) management systems.

The calls home would never connect. They wouldnt be allowed to get a public route.

This is where it starts to tie together. You see wireshark network monitoring edge firewalls and controlled routes deal with too much traffic to see this kind of thing. The operators are human. Thats exactly why I dont trust it though. Because its the SERVERs that are "compromised" and its the SERVERs that WOULD get caught. The internal core servers will trigger alerts and logs before any core router tells the night, switch operator that you are going to a porn site.


I am not saying I am smarter then these people. I am just saying the way this story sounds does not add up to best practice. My concentration is in security and thats not how this works. The amended article mentions people that worked for the CIA checked it and stated the way they discovered the bug is sound.

Ok but who was it?

Why isn't Sepio releasing the documents?

Why was supermicro only given 24 hours to respond when the industry (security and bug) generally mandated 90 days before public release?

How come the most guarded global network carriers did not see illegitimate traffic trying to transverse there network?

In situations like this you have to be on guard. There is no story to be had in the security industry, only facts. Without a picture and documentation it is NOT real.



Xzibit said:


> From the article



That still makes no sense. That data has to want to GO somewhere. even encrypted it is attempting to transport to some IP address or polling DNS for a domain that isnt supermicro. There is an infinitesimally small chance this wouldnt be seen. Show me the logs.

Anyway thanks for asking. It's always good to want to know a bit more. Would love to see how it pans out. If true the tech behind it is amazing, or gross negligence of some of the biggest tech companies on the planet. Should be a hell of a ride or lastly its all BS. Should be a fun ride.


----------



## xkm1948 (Oct 10, 2018)

I will just quote from HardOCP comments:



> I have a theory -
> 
> 
> 
> ...






> Discovered by another Israeli security company... Just like CTS Labs and the AMD CPU "vulnerability"... Strikes me as just a little bit odd.
> 
> Something about this whole thing stinks.
> ​




Not the first time someone behind is trying to stir the water a bit. I am not buying this shit.


----------



## hat (Oct 10, 2018)

We could be reading this story for a lot of reasons. Maybe it's true, or maybe it's a smear campaign targeted at Supermicro, or maybe China. Who knows?


----------



## Flanker (Oct 10, 2018)

FreedomEclipse said:


> Foxconn possibly. But at the same time i dont think it would be them because they are a huge company and they have many many big million dorrah contracts from Apple, Samsung, Sony, Dell... the list goes on.


I guessed foxconn as well, but like you said, it has way too many ties with US businesses. It would also make people think twice about using Taiwanese OEM's.


----------



## StrayKAT (Oct 10, 2018)

hat said:


> We could be reading this story for a lot of reasons. Maybe it's true, or maybe it's a smear campaign targeted at Supermicro, or maybe China. Who knows?



I'm almost inclined to believe it. I don't want to however.. since I'm a fan of SM.

edit: I should point out that California has too many ties to China in general. So it wouldn't surprise me. Google and Apple are already kowtowing to them as it is (newly leaked Google docs show just how much - link). Even a Senator - Dianne Feinstein - had a chauffer for 20 years who turned out to be a spy for China. And somehow she didn't know. Nor is our government investigating it, as of yet. And somehow even our media doesn't want to talk about it enough (.. although there's the occassional editorial - link). I find this is even more bizarre than the Super Micro business. And it's much bigger than Super Micro too.


----------



## dorsetknob (Oct 10, 2018)

SuperMicro Should Reverse engineer several of their own boards ( random sampling) after all they own the blueprints/specs  and know exactly what Semiconductor components should be installed.
They can then confirm that the Boards are made as they Designed and spec'ed them with the Correct components as per those original Blueprints.


----------



## DeathtoGnomes (Oct 10, 2018)

I would assume that large orders of products would be negotiated to follow certain specifications and price. So it wouldnt surprise me that custom orders deem it necessary to make new motherboards from scratch, at which time they can be modified without customer knowledge. 

tinhat wearers agree.


----------



## StrayKAT (Oct 10, 2018)

DeathtoGnomes said:


> I would assume that large orders of products would be negotiated to follow certain specifications and price. So it wouldnt surprise me that custom orders deem it necessary to make new motherboards from scratch, at which time they can be modified without customer knowledge.
> 
> tinhat wearers agree.



It's _tinfoil _hats. I don't think regular tin hats have the correct magical calibration. 

I imagine the custom orders follow a standard template.. but yeah, it's more open to abuse.


----------



## Liquid Cool (Oct 10, 2018)

I buy my straw hats in January.  So, please shoot me a message if the price of SM motherboards crash.  I had one a few years ago and I'd like another.  

It was an extremely solid board once I got past my own ignorance.....

Best,

Liquid Cool


----------



## Salty_sandwich (Oct 10, 2018)

It was a few years back now so I can't remember if it was on the news or I read it or both, but anyway it was about those credit card reader they have in petrol stations, they found out that they where stealing credit card info and sending info to some server (something along those lines) then they found that these where being installed at the factory where they where made, it turned out someone or group had infiltrated the work force and where installing these devices at the factory, the company was not aware this was going on until it was reported.

so I guess its always poss that rouge people/organisation could of infiltrated the factory and the owners probs would never know, not saying that's what happened but that stuff can and does happen.


----------



## silentbogo (Oct 10, 2018)

The story is kind of shady at best. 

First, it was a chip embedded into motherboard, then it was a software hack, then it was both of the above, only now embedded into Ethernet port... 
In all instances it started w/ Bloomberg publishing this stuff on behalf of Sepio, and in all instances there is nothing to go by except "an anonymous source from a high-profile tech company" and "[insert your super-serious agency here] has confirmed". Even went as far as making these scary infographics with components being stripped away off the motherboard to reveal a tiny spec, or making photos of some random UDFN-6 component near the pencil tip to make it look even scarier...


----------



## enxo218 (Oct 10, 2018)

focus has shifted from methodology and implications of hack to verification of validity of claims...I have lost interest


----------



## Frick (Oct 10, 2018)

xkm1948 said:


> I will just quote from HardOCP comments:
> 
> 
> 
> ...





silentbogo said:


> The story is kind of shady at best.
> 
> First, it was a chip embedded into motherboard, then it was a software hack, then it was both of the above, only now embedded into Ethernet port...
> In all instances it started w/ Bloomberg publishing this stuff on behalf of Sepio, and in all instances there is nothing to go by except "an anonymous source from a high-profile tech company" and "[insert your super-serious agency here] has confirmed". Even went as far as making these scary infographics with components being stripped away off the motherboard to reveal a tiny spec, or making photos of some random UDFN-6 component near the pencil tip to make it look even scarier...



But Sepio wasn't involved in the first article afaik, the Ethernet thing was much later.



Solaris17 said:


> This is important because security is on everyones mind in the network/admin field right now.



Thanks for the answer! I highlighted this bit as the attacks were supposedly done some years ago, and I know (or at least assume, from randomly following Krebs and various tech sites articles ) the security field evolves pretty fast... Is it possible it was easier to do this in 2014/15 than today?


----------



## newtekie1 (Oct 10, 2018)

https://www.servethehome.com/yossi-...-positioning-his-research-against-supermicro/

IMO, this is a better article on the issue.  The security firm that allegedly found the issues didn't just find them in Supermicro products, and they can't be sure it was put there during manufacturing in China.

The thing that I find interesting is we have yet even see this supposed hardware that they found.  It hasn't been analyzed by any other source to figure out exactly what it is and what it does.


----------



## Prima.Vera (Oct 11, 2018)

I think everybody should relax for a bit and chill the hypocrisy.
You are all acting like China spying on US it's the worst thing happened since the invention of Politics or Java. 
The US had and has the most advances spying System in the world with the tentacles spread all over the world. Heck, the Internet itself it's the biggest and most complex tool ever developed by Humanity, with all it's 7 Layers possible to hacking, spying, etc, etc.
This news it's just a grain in the sand, more bashing on China, just because...


----------



## R-T-B (Oct 11, 2018)

Xzibit said:


> Doubt Sepio Systems would be so public about it if it wasnt the case.



So it either is a recent occurance, or this stinks to high heavens precisely because it's not real.  I'm not sure which one is more plausible honestly...  But there is no way this has been going on long term and no one noticed the net traffic.

Either way, I remain a skeptic without documentation (which if this is real, should benefit everyone).  The fact that none has been provided stinks to high heavens and has me in @Solaris17's camp



newtekie1 said:


> The thing that I find interesting is we have yet even see this supposed hardware that they found. It hasn't been analyzed by any other source to figure out exactly what it is and what it does.



This.  So much this.


----------



## Solaris17 (Oct 11, 2018)

Frick said:


> But Sepio wasn't involved in the first article afaik, the Ethernet thing was much later.
> 
> 
> 
> Thanks for the answer! I highlighted this bit as the attacks were supposedly done some years ago, and I know (or at least assume, from randomly following Krebs and various tech sites articles ) the security field evolves pretty fast... Is it possible it was easier to do this in 2014/15 than today?



hm, I'm not sure, I cant imagine it would be though. remember these servers are only in production for 3-5 years before they are swapped out if the company makes enough (which these companies do) so that would only raise my doubt higher. As for actual detection, I dont think the capacity would have changed that much. The protocols monitored came out in the 80s. The evolution of the modern data center has certainly changed alot and the security of such is certainly more complex. At the end of the day though traffic cant go from X to X remains the same. We just have fancy graphs and more VMs now.


----------



## OneMoar (Oct 11, 2018)

nonsense supermicro have been around for how long ?


----------



## StrayKAT (Oct 11, 2018)

Prima.Vera said:


> I think everybody should relax for a bit and chill the hypocrisy.
> You are all acting like China spying on US it's the worst thing happened since the invention of Politics or Java.
> The US had and has the most advances spying System in the world with the tentacles spread all over the world. Heck, the Internet itself it's the biggest and most complex tool ever developed by Humanity, with all it's 7 Layers possible to hacking, spying, etc, etc.
> This news it's just a grain in the sand, more bashing on China, just because...



I don't like US' spying either, but it's "tentacles" were encouraged in the Cold War. They didn't create the beast on their own. It's also technically the "5 eyes" (US, UK, Aus, Can, NZ) along with some play with other nations that share intel with each other. Third, at least in the US, as much data as the government collects, it can't officially admit to any of it. The 4th Amendment requires a warrant to invade people's property, etc.. So the spies have to find legal tricks to first obtain a warrant on legitimate grounds, and only then can they inject the crap they've already been collecting under the guise of a "new investigation". It's not always easy. Until politicians convince their idiotic population to get rid of this Amendment, it'll serve us well.

China plays by no such rules... and has no such allies.


----------



## Prima.Vera (Oct 11, 2018)

Well you know, potato - potatoe. 
In U.S. at least there are some "laws" that, in theory, protect the citizens from those kind of ... maneuvers. In China, they even openly admit spying on their own people, and not only that, it's a state law also. Naturally, what expectation one has over a country who doesn't care anything about their citizens private lives and IPs? The moment an international company is opening a fab in China, they can say bye-bye to any proprietary IP or even patent they might have. It's game over.
Just look at their phone industry for example. Callous perfect design clones of Apple's designs and functionalities.


----------



## Athlonite (Oct 11, 2018)

Until we see conclusive evidence of these "chips" and what they do as far as I'm concerned it's all just conjecture and hearsay no pics no logs no backward engineering


----------



## Jetster (Oct 11, 2018)

They did not release the evidence yet because the security company that found it is bound by a contract of non disclosure.


----------



## dorsetknob (Oct 23, 2018)

Story update
https://www.theregister.co.uk/2018/10/22/super_micro_chinese_spy_chip_sec/
excerpt culled from story

They also inspect hardware before it is put into production: as well as visual inspections, it is possible to scan a motherboard for electromagnetic emissions and identify anything unexpected, such as a tiny chip smuggled onto or inside a PCB – there's even a patent on this kind of technology. Finally, the chip shown in the Bloomberg piece is too small to realistically contain the necessary logic and all the data to insert a viable backdoor into a software stack. It is likely just an illustration – meaning, the journalists had no evidence of a chip to show.


----------



## R-T-B (Oct 23, 2018)

dorsetknob said:


> Finally, the chip shown in the Bloomberg piece is too small to realistically contain the necessary logic and all the data to insert a viable backdoor into a software stack.



I called that on day 1.

Bloomberg is usually good but this whole story honestly stinks to high heavens.  Show us evidence or I'm going to keep saying that.


----------



## StrayKAT (Oct 23, 2018)

R-T-B said:


> I called that on day 1.
> 
> Bloomberg is usually good but this whole story honestly stinks to high heavens.  Show us evidence or I'm going to keep saying that.



I remember you saying that. I'd still like a lot more to be out in the open though.

As for Bloomberg, I don't trust Bloomberg himself at all.. but not sure what to think of his site. I mean, I can trust raw business news, but it's not always that.


----------



## silentbogo (Oct 23, 2018)

dorsetknob said:


> Story update
> https://www.theregister.co.uk/2018/10/22/super_micro_chinese_spy_chip_sec/
> excerpt culled from story
> 
> They also inspect hardware before it is put into production: as well as visual inspections, it is possible to scan a motherboard for electromagnetic emissions and identify anything unexpected, such as a tiny chip smuggled onto or inside a PCB – there's even a patent on this kind of technology. Finally, the chip shown in the Bloomberg piece is too small to realistically contain the necessary logic and all the data to insert a viable backdoor into a software stack. It is likely just an illustration – meaning, the journalists had no evidence of a chip to show.



The most important part of that article also reveals the purpose of Bloomberg "newspiece":


> The Bloomberg article – published on October 3 – wiped more than 40 per cent off Super Micro's share price within a matter of hours. But, despite all the three main companies included in the report – Apple, Amazon and Super Micro – all strenuously denying the story was true, Super Micro's share price has not recovered.
> 
> ...
> 
> "We trust you appreciate the difficulty of proving that something did not happen, even though the reporters have produced no affected motherboard or any such malicious hardware chip. As we have said firmly, no one has shown us a motherboard containing any unauthorized hardware chip, we are not aware of any such unauthorized chip, and no government agency has alerted us to the existence of any unauthorized chip."


----------



## StrayKAT (Oct 23, 2018)

silentbogo said:


> The most important part of that article also reveals the purpose of Bloomberg "newspiece":



Now that just pisses me off :\

edit: I mean SM taking all the damage here.


----------



## cdawall (Oct 23, 2018)

Still seems like a crock of it. Just like the first story.


----------



## Athlonite (Oct 23, 2018)

sounds abit like the AMD smear to gain shares cheap SM should sue Bloomberg for any such losses I'd say they'd win over Bloomberg


----------



## DeathtoGnomes (Oct 23, 2018)

So CTS scandal all over again. Man are those guys trying hard to make a profit. Assuming that is the case here.


----------



## Liquid Cool (Oct 23, 2018)

*New No Evidence of Hacked Supermicro Hardware*

,

Liquid Cool


----------

