AMD today announced the expansion of the AMD Versal adaptive system on chip (SoC) portfolio with the new Versal AI Edge Series Gen 2 and Versal Prime Series Gen 2 adaptive SoCs, which bring preprocessing, AI inference, and postprocessing together in a single device for end-to-end acceleration of AI-driven embedded systems.

These initial devices in the Versal Series Gen 2 portfolio build on the first generation with powerful new AI Engines expected to deliver up to 3x higher TOPs-per-watt than first generation Versal AI Edge Series devicesi, while new high-performance integrated Arm CPUs are expected to offer up to 10x more scalar compute than first gen Versal AI Edge and Prime series devicesii.

Advantech (2395.TW), a global leader in intelligent IoT systems and embedded platforms, is excited to announce the EAI-3101, a brand-new industrial PCIe GPU card powered by the Intel Arc A380E, built for 5-year longevity. Featuring 128 Intel Xe matrix AI engines, this GPU card delivers outstanding AI computing power of 5.018 TFLOPS, surpassing the capabilities of the NVIDIA T1000, 2 times over. With ray tracing technology and Intel XeSS AI-upscaling, the EAI-3101 supports up to 8K UHD resolution and achieves a 50% enhancement in graphics performance over the NVIDIA T1000.

To aid in quickly realizing Vision AI, Advantech provides the Edge AI SDK, a rapid AI development toolkit compatible with Intel OpenVINO, which can process the same workload in 40% less time. This groundbreaking graphics solution, with optimized thermal design and an auto smart fan, is specially engineered for image processing and AI acceleration across gaming, medical analysis, and video surveillance. Advantech will demonstrate the EAI-3101 GPU card from April 9th to 11th at the Embedded World 2024 (Hall 3, booth no. 339) in Nuremberg, Germany.

All-in-one computers are often traditionally seen as lower-powered alternatives to traditional desktop workstations. However, a new offering from Alafia AI, a startup focused on medical imaging appliances, aims to shatter that perception. The company's upcoming Alafia Aivas SuperWorkstation packs serious hardware muscle, demonstrating that all-in-one systems can match the performance of their more modular counterparts. At the heart of the Aivas SuperWorkstation lies a 128-core Ampere Altra processor, running at 3.0 GHz clock speed. This CPU is complemented by not one but three NVIDIA L4 GPUs for compute, and a single NVIDIA RTX 4000 Ada GPU for video output, delivering a combined 28,416 CUDA cores for accelerated parallel computing tasks. The system doesn't skimp on other components, either. It features a 4K touch display with up to 360 nits of brightness, an extensive 2 TB of DDR4 RAM, and storage options up to an 8 TB solid-state drive. This combination of cutting-edge CPU, GPU, memory, and storage is squarely aimed at the demands of medical imaging and AI development workloads.

The all-in-one form factor packs this incredible hardware into a sleek, purposefully designed clinical research appliance. While initially targeting software developers, Alafia AI hopes that institutions that can optimize their applications for the Arm architecture can eventually deploy the Aivas SuperWorkstation for production medical imaging workloads. The company is aiming for application integration in Q3 2024 and full ecosystem device integration by Q4 2024. With this powerful new offering, Alafia AI is challenging long-held assumptions about the performance limitations of all-in-one systems. The Aivas SuperWorkstation demonstrates that the right hardware choices can transform these compact form factors into true powerhouse workstations. Especially with a combined total output of three NVIDIA L4 compute units, alongside RTX 4000 Ada graphics card, the AIO is more powerful than some of the high-end desktop workstations.

While we are all used to having a system with a CPU, GPU, and, recently, NPU—X-Silicon Inc. (XSi), a startup founded by former Silicon Valley veterans—has unveiled an interesting RISC-V processor that can simultaneously handle CPU, GPU, and NPU workloads in a chip. This innovative chip architecture, which will be open-source, aims to provide a flexible and efficient solution for a wide range of applications, including artificial intelligence, virtual reality, automotive systems, and IoT devices. The new microprocessor combines a RISC-V CPU core with vector capabilities and GPU acceleration into a single chip, creating a versatile all-in-one processor. By integrating the functionality of a CPU and GPU into a single core, X-Silicon's design offers several advantages over traditional architectures. The chip utilizes the open-source RISC-V instruction set architecture (ISA) for both CPU and GPU operations, running a single instruction stream. This approach promises lower memory footprint execution and improved efficiency, as there is no need to copy data between separate CPU and GPU memory spaces.

Called the C-GPU architecture, X-Silicon uses RISC-V Vector Core, which has 16 32-bit FPUs and a Scaler ALU for processing regular integers as well as floating point instructions. A unified instruction decoder feeds the cores, which are connected to a thread scheduler, texture unit, rasterizer, clipping engine, neural engine, and pixel processors. All is fed into a frame buffer, which feeds the video engine for video output. The setup of the cores allows the users to program each core individually for HPC, AI, video, or graphics workloads. Without software, there is no usable chip, which prompts X-Silicon to work on OpenGL ES, Vulkan, Mesa, and OpenCL APIs. Additionally, the company plans to release a hardware abstraction layer (HAL) for direct chip programming. According to Jon Peddie Research (JPR), the industry has been seeking an open-standard GPU that is flexible and scalable enough to support various markets. X-Silicon's CPU/GPU hybrid chip aims to address this need by providing manufacturers with a single, open-chip design that can handle any desired workload. The XSi gave no timeline, but it has plans to distribute the IP to OEMs and hyperscalers, so the first silicon is still away.

AMD "Zen 5" CPU microarchitecture will introduce a significant performance increase for AVX-512 workloads, with some sources reported as high as 40% performance increases over "Zen 4" in benchmarks that use AVX-512. A Moore's Law is Dead report detailing the execution engine of "Zen 5" holds the answer to how the company managed this—using a true 512-bit FPU. Currently, AMD uses a dual-pumped 256-bit FPU to execute AVX-512 workloads on "Zen 4." The updated FPU should significantly improve the core's performance in workloads that take advantage of 512-bit AVX or VNNI instructions, such as AI.

Giving "Zen 5" a 512-bit FPU meant that AMD also had to scale up the ancillaries—all the components that keep the FPU fed with data and instructions. The company therefore increased the capacity of the L1 DTLB. The load-store queues have been widened to meet the needs of the new FPU. The L1 Data cache has been doubled in bandwidth, and increased in size by 50%. The L1D is now 48 KB in size, up from 32 KB in "Zen 4." FPU MADD latency has been reduced by 1 cycle. Besides the FPU, AMD also increased the number of Integer execution pipes to 10, from 8 on "Zen 4." The exclusive L2 cache per core remains 1 MB in size.Update 07:02 UTC: Moore's Law is Dead reached out to us and said that the slide previously posted by them, which we had used in an earlier version of this article, is fake, but said that the information contained in that slide is correct, and that they stand by the information.

SK hynix Inc., the world's leading producer of High-Bandwidth Memory (HBM) chips, announced today that it will invest an estimated $3.87 billion in West Lafayette, Indiana to build an advanced packaging fabrication and R&D facility for AI products. The project, the first of its kind in the United States, is expected to drive innovation in the nation's AI supply chain, while bringing more than a thousand new jobs to the region.

The company held an investment agreement ceremony with officials from Indiana State, Purdue University, and the U.S. government at Purdue University in West Lafayette on the 3rd and officially announced the plan. At the event, officials from each party including Governor of Indiana Eric Holcomb, Senator Todd Young, Director of the White House Office of Science and Technology Policy Arati Prabhakar, Assistant Secretary of Commerce Arun Venkataraman, Secretary of Commerce State of Indiana David Rosenberg, Purdue University President Mung Chiang, Chairman of Purdue Research Foundation Mitch Daniels, Mayor of city of West Lafayette Erin Easter, Ambassador of the Republic of Korea to the United States Hyundong Cho, Consul General of the Republic of Korea in Chicago Junghan Kim, SK vice chairman Jeong Joon Yu, SK hynix CEO Kwak Noh-Jung and SK hynix Head of Package & Test Choi Woojin, participated.

According to the latest report from The Verge, Microsoft is currently testing a new AI-driven chatbot designed to automate support tasks for its Xbox gaming platform. As the report notes, Microsoft is experimenting with an animated AI character that will assist in answering Xbox support inquiries. The Xbox AI chatbot is connected to Microsoft's Xbox network and ecosystem support documentation. It can answer questions and process game refunds from the Microsoft support website, all aiming to provide users with simple and quick assistance on support topics using natural language, drawing information from existing Xbox support pages. Training on Microsoft's enterprise data will help Microsoft reduce the AI model's hallucinations and instruct it to do only as intended.

As a result, the chatbot's responses closely resemble the information Microsoft provides to its customers to automate support tasks. Recently, Microsoft has expanded the test pool for its new Xbox chatbot, suggesting that the "Xbox Support Virtual Agent" may soon handle support inquiries for all Xbox customers. The development of the Xbox chatbot prototype is part of a broader initiative within Microsoft Gaming to introduce AI-powered features and tools for the Xbox platform and developer tools. The company is also reportedly working on providing AI capabilities for game content creation, gameplay, and the Xbox platform and devices. However, Xbox employees have yet to publicly confirm these more extensive AI efforts for Microsoft Gaming, likely due to the company's cautious approach to presenting AI in gaming. Nevertheless, AI will soon become an integral part of gaming consoles.

Intel Corporation today outlined a new financial reporting structure that is aligned with the company's previously announced foundry operating model for 2024 and beyond. This new structure is designed to drive increased cost discipline and higher returns by providing greater transparency, accountability and incentives across the business. To support the new structure, Intel provided recast operating segment financial results for the years 2023, 2022 and 2021. The company also shared a targeted path toward long-term growth and profitability of Intel Foundry, as well as clear goals for driving financial performance improvement and shareholder value creation.

"Intel's differentiated position as both a world-class semiconductor manufacturer and a fabless technology leader creates significant opportunities to drive long-term sustainable growth across these two complementary businesses," said Pat Gelsinger, Intel CEO. "Implementing this new model marks a key achievement in our IDM 2.0 transformation as we hone our execution engine, stand up the industry's first and only systems foundry with geographically diverse leading-edge manufacturing capacity, and advance our mission to bring AI Everywhere."

AMD has introduced the Ryzen Embedded 8000 Series processors, the first AMD embedded devices to combine NPUs based on the AMD XDNA architecture with traditional CPU and GPU elements, optimized for workload versatility and adaptability targeting industrial AI applications. Embedded solution engineers and developers can harness the processing power and leadership features for a variety of industrial AI applications including machine vision, robotics, and industrial automation. AI is widely used in machine vision applications today to enhance quality control and inspection processes.

AI can also help robots make real-time, route-planning decisions and adapt to dynamic environments. In industrial automation, AI processing helps intelligent edge devices perform complex analysis and decision-making without relying on cloud connectivity. This allows for real-time monitoring, predictive maintenance, and autonomous control of industrial processes, enhancing operational efficiency and reducing downtime.

The expansion of AI technology affects not only the production and demand for graphics cards but also the electricity grid that powers them. Data centers hosting thousands of GPUs are becoming more common, and the industry has been building new facilities for GPU-enhanced servers to serve the need for more AI. However, these powerful GPUs often consume over 500 Watts per single card, and NVIDIA's latest Blackwell B200 GPU has a TGP of 1000 Watts or a single kilowatt. These kilowatt GPUs will be present in data centers with 10s of thousands of cards, resulting in multi-megawatt facilities. To combat the load on the national electricity grid, US President Joe Biden's administration has been discussing with big tech to re-evaluate their power sources, possibly using smaller nuclear plants. According to an Axios interview with Energy Secretary Jennifer Granholm, she has noted that "AI itself isn't a problem because AI could help to solve the problem." However, the problem is the load-bearing of the national electricity grid, which can't sustain the rapid expansion of the AI data centers.

The Department of Energy (DOE) has been reportedly talking with firms, most notably hyperscalers like Microsoft, Google, and Amazon, to start considering nuclear fusion and fission power plants to satisfy the need for AI expansion. We have already discussed the plan by Microsoft to embed a nuclear reactor near its data center facility and help manage the load of thousands of GPUs running AI training/inference. However, this time, it is not just Microsoft. Other tech giants are reportedly thinking about nuclear as well. They all need to offload their AI expansion from the US national power grid and develop a nuclear solution. Nuclear power is a mere 20% of the US power sourcing, and DOE is currently financing a Holtec Palisades 800-MW electric nuclear generating station with $1.52 billion in funds for restoration and resumption of service. Microsoft is investing in a Small Modular Reactors (SMRs) microreactor energy strategy, which could be an example for other big tech companies to follow.

AMD's Instinct family has welcomed a new addition—the MI388X AI accelerator—as discovered in a lengthy regulatory 10K filing (submitted to the SEC). The document reveals that the unannounced SKU—along with the MI250, MI300X and MI300A integrated circuits—cannot be sold to Chinese customers due to updated US trade regulations (new requirements were issued around October 2023). Versal VC2802 and VE2802 FPGA products are also mentioned in the same section. Earlier this month, AMD's Chinese market-specific Instinct MI309 package was deemed to be too powerful for purpose by the US Department of Commerce.

AMD has not published anything about the Instinct MI388X's official specification, and technical details have not emerged via leaks. The "X" tag likely implies that it has been designed for AI and HPC applications, akin to the recently launched MI300X accelerator. The designation of a higher model number could (naturally) point to a potentially more potent spec sheet, although Tom's Hardware posits that MI388X is a semi-custom spinoff of an existing model.

Arm China is making strides in the AI accelerator market with its new neural processing unit (NPU) called Zhouyi. The company aims to integrate the NPU into low-cost domestic CPUs, potentially giving it an edge over competitors like AMD and Intel. Initially a part of Arm Holdings, which licensed IP in China, Arm China took on a new strategy of developing its own IP specifically for Chinese customers a few years ago. While the company does not develop high-performance general-purpose cores, its Zhouyi NPU could become a fundamental building block for affordable processors. A significant step forward is the upcoming addition of an open-source driver for Zhouyi to the Linux kernel. This will make the IP easy to program for software developers, increasing its appeal to chip designers.

Being an open-source driver, the integration in the Linux kernel brings assurance to developers that Zhouyi NPU could be the first in many generations from Arm China. While Zhouyi may not directly compete with offerings from AMD or Intel, its potential for widespread adoption in millions of devices could help Arm China acquire local customers with their IP. The project, which began three years ago with a kernel-only driver, has since evolved into a full driver stack. There is even a development kit board called EAIDK310, powered by Rockwell SoC and Zhouyi NPU, which is available on Aliexpress and Amazon. The integration of AI accelerator technology into the Linux ecosystem is a significant development, though there is still work to be done. Nonetheless, Arm China's Zhouyi NPU and open-source driver are essential to making AI capabilities more accessible and widely available in the domestic Chinese market.

Leading global motherboard manufacturer, ASRock, has successively released software based on Microsoft Windows 10/11 and Canonical Ubuntu Linux platforms since the end of last year, which can help users quickly download, install and configure artificial intelligence software. After receiving great response from the market, ASRock has revealed the 2024 Q1 update of AI QuickSet today, adding two new artificial intelligence (AI) tools, Whisper Desktop and AudioCraft, allowing users of ASRock AMD Radeon RX 7000 series graphics cards to experience more diverse artificial intelligence (AI) applications!

ASRock AI QuickSet software tool 1.2.4 Windows version supports Microsoft Windows 10/11 64-bit operating system, while Linux version 1.1.6 supports Canonical Ubuntu 22.04.4 Desktop (64-bit) operating system, through ASRock AMD Radeon RX 7000 series graphics cards and AMD ROCm software platform provide powerful computing capabilities to support a variety of well-known artificial intelligence (AI) applications. The 1.2.4 Windows version supports image generation tools such as DirectML Shark and Stable Diffusion web UI, as well as the newly added Whisper Desktop speech recognition tool; and the 1.1.6 Linux version supports Image/Manga Translator, Stable Diffusion CLI & web UI image generation tool, and Text generation web UI Llama 2 text generation tool using Meta Llama 2 language model, Ultralytics YOLOv8 object recognition tool, and the newly added AudioCraft audio generation tool.

Ryan McCurdy, President of Lenovo North America, revealed ambitious forward-thinking product roadmap during an interview with CRN magazine. A hybrid strategic approach will create an anticipated AI fast lane on future hardware—McCurdy, a former Intel veteran, stated: "there will be a steady stream of product development to add (AI PC) hardware capabilities in a chicken-and-egg scenario for the OS and for the (independent software vendor) community to develop their latest AI capabilities on top of that hardware...So we are really paving the AI autobahn from a hardware perspective so that we can get the AI software cars to go faster on them." Lenovo—as expected—is jumping on the AI-on-device train, but it will be diversifying its range of AI server systems with new AMD and Intel-powered options. The company has reacted to recent Team Green AI GPU supply issues—alternative units are now in the picture: "with NVIDIA, I think there's obviously lead times associated with it, and there's some end customer identification, to make sure that the products are going to certain identified end customers. As we showcased at Tech World with NVIDIA on stage, AMD on stage, Intel on stage and Microsoft on stage, those industry partnerships are critical to not only how we operate on a tactical supply chain question but also on a strategic what's our value proposition."

McCurdy did not go into detail about upcoming Intel-based server equipment, but seemed excited about AMD's Instinct MI300X accelerator—Lenovo was (previously) announced as one of the early OEM takers of Team Red's latest CDNA 3.0 tech. CRN asked about the firm's outlook for upcoming MI300X-based inventory—McCurdy responded with: "I won't comment on an unreleased product, but the partnership I think illustrates the larger point, which is the industry is looking for a broad array of options. Obviously, when you have any sort of lead times, especially six-month, nine-month and 12-month lead times, there is interest in this incredible technology to be more broadly available. I think you could say in a very generic sense, demand is as high as we've ever seen for the product. And then it comes down to getting the infrastructure launched, getting testing done, and getting workloads validated, and all that work is underway. So I think there is a very hungry end customer-partner user base when it comes to alternatives and a more broad, diverse set of solutions."

The Taiwanese Ministry of Economic Affairs (MOEA) managed to herd government representatives and leading Information and Communication Technology (ICT) industry figures together for an important meeting, according to DigiTimes Asia. The report suggests that the main topic of discussion focused on an anticipated growth of Taiwan's ICT industry—current market trends were analyzed, revealing that the nation absolutely dominates in the AI server segment. The MOEA has (allegedly) determined that Taiwan has shipped 90% of global AI server equipment—DigiTimes claims (based on insider info) that: "American brand vendors are expected to source their AI servers from Taiwanese partners." North American customers could be (presently) 100% reliant on supplies of Taiwanese-produced equipment—a scenario that potentially complicates ongoing international tensions.

The report posits that involved parties have formed plans to seize opportunities within an evergrowing global demand for AI hardware—a 90% market dominance is clearly not enough for some very ambitious industry bosses—although manufacturers will need to jump over several (rising) cost hurdles. Key components for AI servers are reported to be much higher than vanilla server parts—DigiTimes believes that AI processor/accelerator chips are priced close to ten times higher than general purpose server CPUs. Similar price hikes have reportedly affected AI adjacent component supply chains—notably cooling, power supplies and passive parts. Taiwanese manufacturers have spread operations around the world, but industry watchdogs (largely) believe that the best stuff gets produced on home ground—global expansions are underway, perhaps inching closer to better balanced supply conditions.

Synology today announced that Surveillance Station will support automatic license plate recognition (ALPR) in collaboration with Plate Recognizer. This partnership provides added capabilities that Synology NAS and AI-enhanced NVRs bring to a variety of deployment needs, such as those in commercial property management, transport infrastructure monitoring, and more.

"ALPR capabilities are game-changers for video surveillance," said Nini Hsieh, Synology Product Manager for video surveillance. "Synology's Surveillance Station plus Plate Recognizer combines two powerful systems into a comprehensive but easy-to-use solution." Customers of Synology and Plate Recognizer gain automated footage labeling capabilities, allowing efficient post-event reviews. The advanced detection capabilities offered by Plate Recognizer is able to tackle a wide variety of license plates, even under challenging lighting or extreme camera angles. Combined with Surveillance Station's scalable recording, archiving, and multi-site management capabilities, customers gain robust real-time vehicle identification and assurance of footage for post-incident analysis.

Just a month after releasing the 0.1 beta preview of Chat with RTX, now called ChatRTX, NVIDIA has swiftly addressed critical security vulnerabilities discovered in its cutting-edge AI chatbot. The chatbot was found to be susceptible to cross-site scripting attacks (CWE-79) and improper privilege management attacks (CWE-269) in version 0.2 and all prior releases. The identified vulnerabilities posed significant risks to users' personal data and system security. Cross-site scripting attacks could allow malicious actors to inject scripts into the chatbot's interface, potentially compromising sensitive information. The improper privilege management flaw could also enable attackers to escalate their privileges and gain administrative control over users' systems and files.

Upon becoming aware of these vulnerabilities, NVIDIA promptly released an updated version of ChatRTX 0.2, available for download from its official website. The latest iteration of the software addresses these security issues, providing users with a more secure experience. As ChatRTX utilizes retrieval augmented generation (RAG) and NVIDIA Tensor-RT LLM software to allow users to train the chatbot on their personal data, the presence of such vulnerabilities is particularly concerning. Users are strongly advised to update their ChatRTX software to the latest version to mitigate potential risks and protect their personal information. ChatRTX remains in beta version, with no official release candidate timeline announced. As NVIDIA continues to develop and refine this innovative AI chatbot, the company must prioritize security and promptly address any vulnerabilities that may arise, ensuring a safe and reliable user experience.

Microsoft, Intel, and AMD are attempting to jumpstart demand in the PC industry again, under the aegis of the AI PC—devices with native acceleration for AI workloads. Both Intel and AMD have mobile processors with on-silicon NPUs (neural processing units), which are designed to accelerate the first wave of AI-enhanced client experiences on Windows 11 23H2. Microsoft's bulwark with democratizing AI has been Copilot, as a licensee of Open AI GPT-4, GPT-4 Turbo, Dali, and other generative AI tools from the Open AI stable. Copilot is currently Microsoft's most heavily invested application, with its most capital and best minds mobilized to making it the most popular AI assistant. Microsoft even pushed for the AI PC designator to PC OEMs, which requires them to have a dedicated Copilot key akin to the Start key (we'll see how anti-competition regulators deal with that).

The problem with Microsoft's tango with Intel and AMD to push AI PCs, is that Copilot doesn't really use an NPU, not even at the edge—you input a query or a prompt, and Copilot hands it over to a cloud-based AI service. This is about to change, with Microsoft announcing that Copilot will be able to run locally on AI PCs. Microsoft identified several kinds of Copilot use-cases that an NPU can handle on-device, which should speed up response times to Copilot queries, but this requires the NPU to have at least 40 TOPS of performance. This is a problem for the current crop of processors with NPUs. Intel's Core Ultra "Meteor Lake" has an AI Boost NPU with 10 TOPS on tap, while the Ryzen 8040 "Hawk Point" is only slightly faster, with a 16 TOPS Ryzen AI NPU. AMD has already revealed that the XDNA 2-based 2nd Generation Ryzen AI NPU in its upcoming "Strix Point" processors will come with over 40 TOPS of performance, and it stands to reason that the NPUs in Intel's "Arrow Lake" or "Lunar Lake" processors are comparable in performance; which should enable on-device Copilot.

Samsung debuted its Mach-1 generation of AI processors during a recent shareholder meeting—the South Korean megacorp anticipates an early 2025 launch window. Their application-specific integrated circuit (ASIC) design is expected to "excel in edge computing applications," with a focus on low power and efficiency-oriented operating environments. Naver Corporation was a key NVIDIA high-end AI customer in South Korea (and Japan), but the leading search platform firm and creator of HyperCLOVA X LLM (reportedly) deliberated on an adoption alternative hardware last October. The Korea Economic Daily believes that Naver's relationship with Samsung is set to grow, courtesy of a proposed $752 million investment: "the world's top memory chipmaker, will supply its next-generation Mach-1 artificial intelligence chips to Naver Corp. by the end of this year."

Reports from last December indicated that the two companies were deep into the process of co-designing power-efficient AI accelerators—Naver's main goal is to finalize a product that will offer eight times more energy efficiency than NVIDIA's H100 AI accelerator. Naver's alleged bulk order—of roughly 150,000 to 200,000 Samsung Mach-1 AI chips—appears to be a stopgap. Industry insiders reckon that Samsung's first-gen AI accelerator is much cheaper when compared to NVIDIA H100 GPU price points—a per-unit figure of $3756 is mentioned in the KED Global article. Samsung is speculated to be shopping its fledgling AI tech to Microsoft and Meta.

It's official: NVIDIA delivered the world's fastest platform in industry-standard tests for inference on generative AI. In the latest MLPerf benchmarks, NVIDIA TensorRT-LLM—software that speeds and simplifies the complex job of inference on large language models—boosted the performance of NVIDIA Hopper architecture GPUs on the GPT-J LLM nearly 3x over their results just six months ago. The dramatic speedup demonstrates the power of NVIDIA's full-stack platform of chips, systems and software to handle the demanding requirements of running generative AI. Leading companies are using TensorRT-LLM to optimize their models. And NVIDIA NIM—a set of inference microservices that includes inferencing engines like TensorRT-LLM—makes it easier than ever for businesses to deploy NVIDIA's inference platform.

Raising the Bar in Generative AI
TensorRT-LLM running on NVIDIA H200 Tensor Core GPUs—the latest, memory-enhanced Hopper GPUs—delivered the fastest performance running inference in MLPerf's biggest test of generative AI to date. The new benchmark uses the largest version of Llama 2, a state-of-the-art large language model packing 70 billion parameters. The model is more than 10x larger than the GPT-J LLM first used in the September benchmarks. The memory-enhanced H200 GPUs, in their MLPerf debut, used TensorRT-LLM to produce up to 31,000 tokens/second, a record on MLPerf's Llama 2 benchmark. The H200 GPU results include up to 14% gains from a custom thermal solution. It's one example of innovations beyond standard air cooling that systems builders are applying to their NVIDIA MGX designs to take the performance of Hopper GPUs to new heights.

Today, MLCommons published results of the industry-standard MLPerf v4.0 benchmark for inference. Intel's results for Intel Gaudi 2 accelerators and 5th Gen Intel Xeon Scalable processors with Intel Advanced Matrix Extensions (Intel AMX) reinforce the company's commitment to bring "AI Everywhere" with a broad portfolio of competitive solutions. The Intel Gaudi 2 AI accelerator remains the only benchmarked alternative to Nvidia H100 for generative AI (GenAI) performance and provides strong performance-per-dollar. Further, Intel remains the only server CPU vendor to submit MLPerf results. Intel's 5th Gen Xeon results improved by an average of 1.42x compared with 4th Gen Intel Xeon processors' results in MLPerf Inference v3.1.

"We continue to improve AI performance on industry-standard benchmarks across our portfolio of accelerators and CPUs. Today's results demonstrate that we are delivering AI solutions that deliver to our customers' dynamic and wide-ranging AI requirements. Both Intel Gaudi and Xeon products provide our customers with options that are ready to deploy and offer strong price-to-performance advantages," said Zane Ball, Intel corporate vice president and general manager, DCAI Product Management.

SK Hynix is planning a large $4 billion chip-packaging and testing facility in Indiana, USA. The company is still in the planning stage of the decision to invest in the US. "[the company] is reviewing its advanced chip packaging investment in the US, but hasn't made a final decision yet," a company spokesperson told the Wall Street Journal. The primary product focus for this plant will be stacked HBM memory meant to be consumed by the AI GPU and self-driving automobile industries. The plant could also focus on other exotic memory types, such as high-density server memory; and perhaps even compute-in-memory. The plant is expected to start operations in 2028, and will create up to 1,000 skilled jobs. SK Hynix is counting for state- and federal tax incentives to propel this investment; under government initiatives such as the CHIPS Act. SK Hynix is a significant supplier of HBM to NVIDIA for its AI GPUs. Its HBM3E features in the latest NVIDIA "Blackwell" GPUs.

Intel Corporation today announced the creation of two new artificial intelligence (AI) initiatives as part of the AI PC Acceleration Program: the AI PC Developer Program and the addition of independent hardware vendors to the program. These are critical milestones in Intel's pursuit of enabling the software and hardware ecosystem to optimize and maximize AI on more than 100 million Intel-based AI PCs through 2025.

"We have made great strides with our AI PC Acceleration Program by working with the ecosystem. Today, with the addition of the AI PC Developer Program, we are expanding our reach to go beyond large ISVs and engage with small- and medium-sized players and aspiring developers. Our goal is to drive a frictionless experience by offering a broad set of tools including the new AI-ready Developer Kit," said Carla Rodriguez, Intel vice president and general manager of Client Software Ecosystem Enabling.

Canalys anticipates that China's PC (excluding tablets) market will rebound to 3% growth in 2024 and 10% growth in 2025, primarily fueled by refresh demand from the commercial sector. The tablet market is expected to grow by 4% in both 2024 and 2025, benefiting from increasing penetration as digitalization deepens.

"2024 is expected to bring modest relief to a struggling PC market in China, but a challenging environment will remain," said Canalys Analyst Emma Xu. "Ongoing economic structural adjustments are a key priority as the government seeks new avenues for economic growth, with a core focus on technology-driven innovation. AI emerged as a central theme during the latest 'Two Sessions' in China, with enthusiasm for AI spanning commercial entities and government initiatives aimed at establishing a domestic AI ecosystem across industries. Significant opportunities for the PC industry are set to arise from this commercial push, especially as it coincides with the upcoming device refresh and the emergence of AI-capable PCs."

A manufacturing plant near Hsinchu, Taiwan's Silicon Valley, is among facilities worldwide boosting energy efficiency with AI-enabled digital twins. A virtual model can help streamline operations, maximizing throughput for its physical counterpart, say engineers at Wistron, a global designer and manufacturer of computers and electronics systems. In the first of several use cases, the company built a digital copy of a room where NVIDIA DGX systems undergo thermal stress tests (pictured above). Early results were impressive.

Making Smart Simulations
Using NVIDIA Modulus, a framework for building AI models that understand the laws of physics, Wistron created digital twins that let them accurately predict the airflow and temperature in test facilities that must remain between 27 and 32 degrees C. A simulation that would've taken nearly 15 hours with traditional methods on a CPU took just 3.3 seconds on an NVIDIA GPU running inference with an AI model developed using Modulus, a whopping 15,000x speedup. The results were fed into tools and applications built by Wistron developers with NVIDIA Omniverse, a platform for creating 3D workflows and applications based on OpenUSD.