Neural Rendering

John Spitzer, a key figure at NVIDIA since 1999, reminisced about the early days of fixed-function shaders. Back then, game developers faced significant limitations in expressing their creative visions. The introduction of programmable shaders with GeForce 3 marked a pivotal moment, allowing for vertex and pixel shading customization. Subsequent developments, such as high-level shading languages (HLSL) and DirectX advancements—from geometry shaders (DX10) to ray tracing (DX12)—have continuously expanded the creative toolkit for developers.


But today, NVIDIA's GeForce 50 "Blackwell," promises to redefine the landscape once more. Blackwell introduces the concept of developer-created neural shading, allowing for a host of neural-driven effects, including neural materials, neural volumes, and even neural radiance fields.


Until now, accessing Tensor Cores within a graphics API was not possible. This changed through NVIDIA's collaboration with Microsoft, resulting in the Cooperative Vectors API for DirectX. Combined with a new shading language, Slang, this breakthrough enables developers to integrate neural techniques directly into their workflows, potentially replacing parts of the traditional graphics pipeline. Slang splits large, complex functions into smaller pieces that are easier to handle.

Given that this is a DirectX standard API feature, there is nothing that stops AMD and Intel from integrating Neural Rendering (Cooperative Vectors) into their graphics drivers.


One of the standout innovations is neural materials. Traditional materials in real-time graphics involve complex shader code, often requiring significant computational resources to replicate effects seen in high-budget CGI films. Neural materials address this by converting shader code and texture layers into a compressed neural representation. This results in up to a 7:1 compression ratio and enables small neural networks to generate stunning, film-like materials in real-time.

For example, silk rendered with traditional shaders might lack the multicolored sheen seen in real life. Neural materials, however, capture intricate details like color variation and reflections, bringing such surfaces to life with unparalleled realism—and at a fraction of the memory cost.


Another significant advancement is the Neural Radiance Cache, which dynamically trains a neural network during gameplay using the gamer's GPU. This innovation allows light transport to be cached spatially, enabling near-infinite light bounces in a scene. The result? More realistic indirect lighting and shadows with minimal performance impact. NRC partially traces 1 or 2 rays before storing them in a radiance cache, and infers an infinite amount of rays and bounces for a more accurate representation of indirect lighting in the game scene. In their demo, the technology even managed to increase frame rates by reducing the computational load of ray tracing, while maintaining visual fidelity.

NRC is now available through the RTX Global Illumination SDK, and will be available in Portal with RTX, and, in the coming months, RTX Remix.


Rendering translucent materials, like skin, has always posed a challenge in real-time graphics. NVIDIA's RTX Skin leverages subsurface scattering—a technique borrowed from film rendering—to simulate light penetrating and diffusing through materials. For instance, thin areas of skin now emit a subtle glow, adding depth and realism to characters in motion. Artists can finely tune these effects to achieve their desired visual outcomes.


NVIDIA also showcased advancements in rendering realistic faces and hair, two notoriously difficult challenges in gaming. The extremely high triangle counts also make it difficult for ray tracing to maintain and update the BVH structure. By using diffusion models, NVIDIA's tools can generate photorealistic facial animations and expressions in real-time, overcoming the uncanny valley effect. For hair, Blackwell introduces "Linear Swept Spheres"—a more efficient geometric representation that drastically reduces memory overhead without compromising visual quality. On Blackwell these are GPU-accelerated.


Games have grown exponentially in geometric complexity, with scenes now comprising billions of polygons. Blackwell addresses this with RTX Mega Geometry, enabling developers to use high-resolution meshes like those from Unreal Engine 5's Nanite directly within ray-traced scenes. This eliminates the need for low-resolution proxy meshes, preserving visual detail while optimizing performance through efficient compression and clustering.


As NVIDIA celebrates the one-year anniversary of RTX Remix, the modding community continues to embrace these tools, expanding the boundaries of what's possible in gaming and beyond. With technologies like neural shading, mega geometry, and RTX Skin, the future of graphics promises to be more immersive, accessible, and awe-inspiring than ever.