rendering

Features

Photorealistic rendering

A large proportion of computer graphics research has worked towards producing images that resemble photographs. Fundamental techniques that make this possible were invented in the 1980s, but at the end of the decade, photorealism for complex scenes was still considered a distant goal. Today, photorealism is routinely achievable for offline rendering, but remains difficult for real-time rendering.

In order to produce realistic images, rendering must simulate how light travels from light sources, is reflected, refracted, and scattered (often many times) by objects in the scene, passes through a camera lens, and finally reaches the film or sensor of the camera. The physics used in these simulations is primarily geometrical optics, in which particles of light follow (usually straight) lines called rays, but in some situations (such as when rendering thin films, like the surface of soap bubbles) the wave nature of light must be taken into account.

Effects that may need to be simulated include:

Shadows, including both shadows with sharp edges and soft shadows with umbra and penumbra

Reflections in mirrors and smooth surfaces, as well as rough or rippled reflective surfaces

Refraction – the bending of light when it crosses a boundary between two transparent materials such as air and glass. The amount of bending varies with the wavelength of the light, which may cause colored fringes or "rainbows" to appear.

Volumetric effects – absorption and scattering when light travels through partially transparent or translucent substances (called participating media because they modify the light rather than simply allow rays to pass through)

Caustics – bright patches, sometimes with distinct filaments and a folded or twisted appearance, resulting when light is reflected or refracted before illuminating an object.

Other styles of 3D rendering

For some applications (including early stages of 3D modeling), simplified rendering styles such as wireframe rendering may be appropriate, particularly when the material and surface details have not been defined and only the shape of an object is known. Games and other real-time applications may use simpler and less realistic rendering techniques as an artistic or design choice, or to allow higher frame rates on lower-end hardware.

Orthographic and isometric projections can be used for a stylized effect or to ensure that parallel lines are depicted as parallel in CAD rendering.

Non-photorealistic rendering (NPR) uses techniques like edge detection and posterization to produce 3D images that resemble technical illustrations, cartoons, or other styles of drawing or painting.

2D rendering

In 2D computer graphics the positions and sizes of shapes are specified using 2D coordinates (x and y) instead of 3D coordinates (x, y, and z). 2D rendering APIs often use a resolution-independent coordinate system, with a viewport determining how to convert coordinates to pixel indexes called device coordinates. Transformations such as scaling, translation, and rotation may be applied before rendering the shapes. These affine transformations are often represented by 3 × 3 matrices, allowing easier composition of transformations.

Higher-quality 2D rendering engines such as SVG renderers usually implement anti-aliasing to reduce the jagged appearance of rasterized lines and shape edges. When rendering overlapping shapes, renderers commonly use a "painter's model" in which the shapes are drawn in some determined order, or their contributions to each pixel are composited using blending operations that may depend on the order of the inputs. Renderers may allow giving shapes a "z index" or "stacking order" to specify the rendering or blending order (unlike the z coordinate used in 3D rendering, this third coordinate only indicates an order, not a distance, and cannot be meaningfully rotated together with the x and y coordinates).

2D rendering typically does not simulate light propagation (which would likely require specifying 3D positions or thickness for the shapes). Effects such as drop shadows and transparency are defined by mathematical functions with no physical basis.

2D rendering for print output may need to support very high resolutions, e.g. 600 or 1200 DPI for a typical laser printer, or 2400 DPI or higher for an imagesetter or platesetter. Grayscale and color images require halftones (or some form of dithering or stochastic screening) and color separations. A rendering engine called a raster image processor (RIP) converts input data such as PDF files into the high-resolution bitmap images used by the printer.

Inputs

Before a 3D scene or 2D image can be rendered, it must be described in a way that the rendering software can understand. Historically, inputs for both 2D and 3D rendering were usually text files, which are easier than binary files for humans to edit and debug. For 3D graphics, text formats have largely been supplanted by more efficient binary formats, and by APIs which allow interactive applications to communicate directly with a rendering component without generating a file on disk (although a scene description is usually still created in memory prior to rendering).

Traditional rendering algorithms use geometric descriptions of 3D scenes or 2D images. Applications and algorithms that render visualizations of data scanned from the real world, or scientific simulations, may require different types of input data.

The PostScript format (which is often credited with the rise of desktop publishing) provides a standardized, interoperable way to describe 2D graphics and page layout. The Scalable Vector Graphics (SVG) format is also text-based, and the PDF format uses the PostScript language internally. In contrast, although many 3D graphics file formats have been standardized (including text-based formats such as VRML and X3D), different rendering applications typically use formats tailored to their needs, and this has led to a proliferation of proprietary and open formats, with binary files being more common.

2D vector graphics

A vector graphics image description may include:

Coordinates and curvature information for line segments, arcs, and Bézier curves (which may be used as boundaries of filled shapes)

Center coordinates, width, and height (or bounding rectangle coordinates) of basic shapes such as rectangles, circles and ellipses

Color, width and pattern (such as dashed or dotted) for rendering lines

Colors, patterns, and gradients for filling shapes

Bitmap image data (either embedded or in an external file) along with scale and position information

Text to be rendered (along with size, position, orientation, color, and font)

Clipping information, if only part of a shape or bitmap image should be rendered

Transparency and compositing information for rendering overlapping shapes

Color space information, allowing the image to be rendered consistently on different displays and printers

3D geometry

A geometric scene description may include:

Size, position, and orientation of geometric primitives such as spheres and cones (which may be combined in various ways to create more complex objects)

Vertex coordinates and surface normal vectors for meshes of triangles or polygons (often rendered as smooth surfaces by subdividing the mesh)

Transformations for positioning, rotating, and scaling objects within a scene (allowing parts of the scene to use different local coordinate systems).

"Camera" information describing how the scene is being viewed (position, direction, focal length, and field of view)

Light information (location, type, brightness, and color)

Optical properties of surfaces, such as albedo, roughness, and refractive index,

Optical properties of media through which light passes (transparent solids, liquids, clouds, smoke), e.g. absorption and scattering cross sections

Bitmap image data used as texture maps for surfaces

Small scripts or programs for generating complex 3D shapes or scenes procedurally

Description of how object and camera locations and other information change over time, for rendering an animation

Many file formats exist for storing individual 3D objects or "models". These can be imported into a larger scene, or loaded on-demand by rendering software or games. A realistic scene may require hundreds of items like household objects, vehicles, and trees, and 3D artists often utilize large libraries of models. In game production, these models (along with other data such as textures, audio files, and animations) are referred to as "assets".

Volumetric data

Scientific and engineering visualization often requires rendering volumetric data generated by 3D scans or simulations. Perhaps the most common source of such data is medical CT and MRI scans, which need to be rendered for diagnosis. Volumetric data can be extremely large, and requires specialized data formats to store it efficiently, particularly if the volume is sparse (with empty regions that do not contain data).

Before rendering, level sets for volumetric data can be extracted and converted into a mesh of triangles, e.g. by using the marching cubes algorithm. Algorithms have also been developed that work directly with volumetric data, for example to render realistic depictions of the way light is scattered and absorbed by clouds and smoke, and this type of volumetric rendering is used extensively in visual effects for movies. When rendering lower-resolution volumetric data without interpolation, the individual cubes or "voxels" may be visible, an effect sometimes used deliberately for game graphics.

Photogrammetry and scanning

Photographs of real world objects can be incorporated into a rendered scene by using them as textures for 3D objects. Photos of a scene can also be stitched together to create panoramic images or environment maps, which allow the scene to be rendered very efficiently but only from a single viewpoint. Scanning of real objects and scenes using structured light or lidar produces point clouds consisting of the coordinates of millions of individual points in space, sometimes along with color information. These point clouds may either be rendered directly or converted into meshes before rendering. (Note: "point cloud" sometimes also refers to a minimalist rendering style that can be used for any 3D geometry, similar to wireframe rendering.)

Neural approximations and light fields

A more recent, experimental approach is description of scenes using radiance fields which define the color, intensity, and direction of incoming light at each point in space. (This is conceptually similar to, but not identical to, the light field recorded by a hologram.) For any useful resolution, the amount of data in a radiance field is so large that it is impractical to represent it directly as volumetric data, and an approximation function must be found. Neural networks are typically used to generate and evaluate these approximations, sometimes using video frames, or a collection of photographs of a scene taken at different angles, as "training data".

Algorithms related to neural networks have recently been used to find approximations of a scene as 3D Gaussians. The resulting representation is similar to a point cloud, except that it uses fuzzy, partially-transparent blobs of varying dimensions and orientations instead of points. As with neural radiance fields, these approximations are often generated from photographs or video frames.

Outputs

The output of rendering may be displayed immediately on the screen (many times a second, in the case of real-time rendering such as games) or saved in a raster graphics file format such as JPEG or PNG. High-end rendering applications commonly use the OpenEXR file format, which can represent finer gradations of colors and high dynamic range lighting, allowing tone mapping or other adjustments to be applied afterwards without loss of quality.

Quickly rendered animations can be saved directly as video files, but for high-quality rendering, individual frames (which may be rendered by different computers in a cluster or render farm and may take hours or even days to render) are output as separate files and combined later into a video clip.

The output of a renderer sometimes includes more than just RGB color values. For example, the spectrum can be sampled using multiple wavelengths of light, or additional information such as depth (distance from camera) or the material of each point in the image can be included (this data can be used during compositing or when generating texture maps for real-time rendering, or used to assist in removing noise from a path-traced image). Transparency information can be included, allowing rendered foreground objects to be composited with photographs or video. It is also sometimes useful to store the contributions of different lights, or of specular and diffuse lighting, as separate channels, so lighting can be adjusted after rendering. The OpenEXR format allows storing many channels of data in a single file. Renderers such as Blender and Pixar RenderMan support a large variety of configurable values called Arbitrary Output Variables (AOVs).

Techniques

Choosing how to render a 3D scene usually involves trade-offs between speed, memory usage, and realism (although realism is not always desired). The algorithms developed over the years follow a loose progression, with more advanced methods becoming practical as computing power and memory capacity increased. Multiple techniques may be used for a single final image.

An important distinction is between image order algorithms, which iterate over pixels in the image, and object order algorithms, which iterate over objects in the scene. For simple scenes, object order is usually more efficient, as there are fewer objects than pixels.

2D vector graphics

The vector displays of the 1960s-1970s used deflection of an electron beam to draw line segments directly on the screen. Nowadays, vector graphics are rendered by rasterization algorithms that also support filled shapes. In principle, any 2D vector graphics renderer can be used to render 3D objects by first projecting them onto a 2D image plane.

3D rasterization

Adapts 2D rasterization algorithms so they can be used more efficiently for 3D rendering, handling hidden surface removal via scanline or z-buffer techniques. Different realistic or stylized effects can be obtained by coloring the pixels covered by the objects in different ways. Surfaces are typically divided into meshes of triangles before being rasterized. Rasterization is usually synonymous with "object order" rendering (as described above).

Ray casting

Uses geometric formulas to compute the first object that a ray intersects. It can be used to implement "image order" rendering by casting a ray for each pixel, and finding a corresponding point in the scene. Ray casting is a fundamental operation used for both graphical and non-graphical purposes, e.g. determining whether a point is in shadow, or checking what an enemy can see in a game.

Ray tracing

Simulates the bouncing paths of light caused by specular reflection and refraction, requiring a varying number of ray casting operations for each path. Advanced forms use Monte Carlo techniques to render effects such as area lights, depth of field, blurry reflections, and soft shadows, but computing global illumination is usually in the domain of path tracing.

Rasterization

The term rasterization (in a broad sense) encompasses many techniques used for 2D rendering and real-time 3D rendering. 3D animated films were rendered by rasterization before ray tracing and path tracing became practical.

A renderer combines rasterization with geometry processing (which is not specific to rasterization) and pixel processing which computes the RGB color values to be placed in the framebuffer for display.

The main tasks of rasterization (including pixel processing) are:

Determining which pixels are covered by each geometric shape in the 3D scene or 2D image (this is the actual rasterization step, in the strictest sense)

Blending between colors and depths defined at the vertices of shapes, e.g. using barycentric coordinates (interpolation)

Determining if parts of shapes are hidden by other shapes, due to 2D layering or 3D depth (hidden surface removal)

Evaluating a function for each pixel covered by a shape (shading)

Smoothing edges of shapes so pixels are less visible (anti-aliasing)

Blending overlapping transparent shapes (compositing)

3D rasterization is typically part of a graphics pipeline in which an application provides lists of triangles to be rendered, and the rendering system transforms and projects their coordinates, determines which triangles are potentially visible in the viewport, and performs the above rasterization and pixel processing tasks before displaying the final result on the screen.

Historically, 3D rasterization used algorithms like the Warnock algorithm and scanline rendering (also called "scan-conversion"), which can handle arbitrary polygons and can rasterize many shapes simultaneously. Although such algorithms are still important for 2D rendering, 3D rendering now usually divides shapes into triangles and rasterizes them individually using simpler methods.

Ray casting

One of the simplest ways to render a 3D scene is to test if a ray starting at the viewpoint (the "eye" or "camera") intersects any of the geometric shapes in the scene, repeating this test using a different ray direction for each pixel. This method, called ray casting, was important in early computer graphics, and is a fundamental building block for more advanced algorithms. Ray casting can be used to render shapes defined by constructive solid geometry (CSG) operations.

Early ray casting experiments include the work of Arthur Appel in the 1960s. Appel rendered shadows by casting an additional ray from each visible surface point towards a light source. He also tried rendering the density of illumination by casting random rays from the light source towards the object and plotting the intersection points (similar to the later technique called photon mapping).

When rendering scenes containing many objects, testing the intersection of a ray with every object becomes very expensive. Special data structures are used to speed up this process by allowing large numbers of objects to be excluded quickly (such as objects behind the camera). These structures are analogous to database indexes for finding the relevant objects. The most common are the bounding volume hierarchy (BVH), which stores a pre-computed bounding box or sphere for each branch of a tree of objects, and the k-d tree which recursively divides space into two parts. Recent GPUs include hardware acceleration for BVH intersection tests. K-d trees are a special case of binary space partitioning, which was frequently used in early computer graphics (it can also generate a rasterization order for the painter's algorithm). Octrees, another historically popular technique, are still often used for volumetric data.

Geometric formulas are sufficient for finding the intersection of a ray with shapes like spheres, polygons, and polyhedra, but for most curved surfaces there is no analytic solution, or the intersection is difficult to compute accurately using limited precision floating point numbers. Root-finding algorithms such as Newton's method can sometimes be used. To avoid these complications, curved surfaces are often approximated as meshes of triangles. Volume rendering (e.g. rendering clouds and smoke), and some surfaces such as fractals, may require ray marching instead of basic ray casting.

Ray tracing

Ray casting can be used to render an image by tracing light rays backwards from a simulated camera. After finding a point on a surface where a ray originated, another ray is traced towards the light source to determine if anything is casting a shadow on that point. If not, a reflectance model (such as Lambertian reflectance for matte surfaces, or the Phong reflection model for glossy surfaces) is used to compute the probability that a photon arriving from the light would be reflected towards the camera, and this is multiplied by the brightness of the light to determine the pixel brightness. If there are multiple light sources, brightness contributions of the lights are added together. For color images, calculations are repeated for multiple wavelengths of light (e.g. red, green, and blue).

Classical ray tracing (also called Whitted-style or recursive ray tracing) extends this method so it can render mirrors and transparent objects. If a ray traced backwards from the camera originates at a point on a mirror, the reflection formula from geometric optics is used to calculate the direction the reflected ray came from, and another ray is cast backwards in that direction. If a ray originates at a transparent surface, rays are cast backwards for both reflected and refracted rays (using Snell's law to compute the refracted direction), and so ray tracing needs to support a branching "tree" of rays. In simple implementations, a recursive function is called to trace each ray.

Ray tracing usually performs anti-aliasing by taking the average of multiple samples for each pixel. It may also use multiple samples for effects like depth of field and motion blur. If evenly spaced ray directions or times are used for each of these features, many rays are required, and some aliasing will remain. Cook-style, stochastic, or Monte Carlo ray tracing avoids this problem by using random sampling instead of evenly spaced samples. This type of ray tracing is commonly called distributed ray tracing, or distribution ray tracing because it samples rays from probability distributions. Distribution ray tracing can also render realistic "soft" shadows from large lights by using a random sample of points on the light when testing for shadowing, and it can simulate chromatic aberration by sampling multiple wavelengths from the spectrum of light.

Real surface materials reflect small amounts of light in almost every direction because they have small (or microscopic) bumps and grooves. A distribution ray tracer can simulate this by sampling possible ray directions, which allows rendering blurry reflections from glossy and metallic surfaces. However, if this procedure is repeated recursively to simulate realistic indirect lighting, and if more than one sample is taken at each surface point, the tree of rays quickly becomes huge. Another kind of ray tracing, called path tracing, handles indirect light more efficiently, avoiding branching, and ensures that the distribution of all possible paths from a light source to the camera is sampled in an unbiased way.

Radiosity

Radiosity (named after the radiometric quantity of the same name) is a method for rendering objects illuminated by light bouncing off rough or matte surfaces. This type of illumination is called indirect light, environment lighting, diffuse lighting, or diffuse interreflection, and the problem of rendering it realistically is called global illumination. Rasterization and basic forms of ray tracing (other than distribution ray tracing and path tracing) can only roughly approximate indirect light, e.g. by adding a uniform "ambient" lighting amount chosen by the artist. Radiosity techniques are also suited to rendering scenes with area lights such as rectangular fluorescent lighting panels, which are difficult for rasterization and traditional ray tracing. Radiosity is considered a physically-based method, meaning that it aims to simulate the flow of light in an environment using equations and experimental data from physics, however it often assumes that all surfaces are opaque and perfectly Lambertian, which reduces realism and limits its applicability.

In the original radiosity method (first proposed in 1984) now called classical radiosity, surfaces and lights in the scene are split into pieces called patches, a process called meshing (this step makes it a finite element method). The rendering code must then determine what fraction of the light being emitted or diffusely reflected (scattered) by each patch is received by each other patch. These fractions are called form factors or view factors (first used in engineering to model radiative heat transfer). The form factors are multiplied by the albedo of the receiving surface and put in a matrix. The lighting in the scene can then be expressed as a matrix equation (or equivalently a system of linear equations) that can be solved by methods from linear algebra.

Solving the radiosity equation gives the total amount of light emitted and reflected by each patch, which is divided by area to get a value called radiosity that can be used when rasterizing or ray tracing to determine the color of pixels corresponding to visible parts of the patch. For real-time rendering, this value (or more commonly the irradiance, which does not depend on local surface albedo) can be pre-computed and stored in a texture (called an irradiance map) or stored as vertex data for 3D models. This feature was used in architectural visualization software to allow real-time walk-throughs of a building interior after computing the lighting.

The large size of the matrices used in classical radiosity (the square of the number of patches) causes problems for realistic scenes. Practical implementations may use Jacobi or Gauss-Seidel iterations, which is equivalent (at least in the Jacobi case) to simulating the propagation of light one bounce at a time until the amount of light remaining (not yet absorbed by surfaces) is insignificant. The number of iterations (bounces) required is dependent on the scene, not the number of patches, so the total work is proportional to the square of the number of patches (in contrast, solving the matrix equation using Gaussian elimination requires work proportional to the cube of the number of patches). Form factors may be recomputed when they are needed, to avoid storing a complete matrix in memory.

Path tracing

As part of the approach known as physically based rendering, path tracing has become the dominant technique for rendering realistic scenes, including effects for movies. For example, the popular open source 3D software Blender uses path tracing in its Cycles renderer. Images produced using path tracing for global illumination are generally noisier than when using radiosity (the main competing algorithm for realistic lighting), but radiosity can be difficult to apply to complex scenes and is prone to artifacts that arise from using a tessellated representation of irradiance.

Like distributed ray tracing, path tracing is a kind of stochastic or randomized ray tracing that uses Monte Carlo or Quasi-Monte Carlo integration. It was proposed and named in 1986 by Jim Kajiya in the same paper as the rendering equation. Kajiya observed that much of the complexity of distributed ray tracing could be avoided by only tracing a single path from the camera at a time (in Kajiya's implementation, this "no branching" rule was broken by tracing additional rays from each surface intersection point to randomly chosen points on each light source). Kajiya suggested reducing the noise present in the output images by using stratified sampling and importance sampling for making random decisions such as choosing which ray to follow at each step of a path. Even with these techniques, path tracing would not have been practical for film rendering, using computers available at the time, because the computational cost of generating enough samples to reduce variance to an acceptable level was too high. Monster House, the first feature film rendered entirely using path tracing, was not released until 20 years later.

In its basic form, path tracing is inefficient (requiring too many samples) for rendering caustics and scenes where light enters indirectly through narrow spaces. Attempts were made to address these weaknesses in the 1990s. Bidirectional path tracing has similarities to photon mapping, tracing rays from the light source and the camera separately, and then finding ways to connect these paths (but unlike photon mapping it usually samples new light paths for each pixel rather than using the same cached data for all pixels). Metropolis light transport samples paths by modifying paths that were previously traced, spending more time exploring paths that are similar to other "bright" paths, which increases the chance of discovering even brighter paths. Multiple importance sampling provides a way to reduce variance when combining samples from more than one sampling method, particularly when some samples are much noisier than the others.

Machine learning

As of 2003, machine learning (ML) techniques were being used in other areas of computer graphics, such as texture synthesis, fitting surfaces to point clouds, and fitting curves to motion capture data. The use of ML for rendering expanded with the availability of GPUs that can evaluate neural networks (especially convolutional neural networks) quickly.

Neural networks and Gaussian mixture models have been used in conjunction with rendering techniques such as path tracing to encode bidirectional reflectance distribution functions (BRDFs), or to encode cached radiance at points in space (for biased acceleration of rendering or for path guiding). Neural radiance fields (NeRFs) take the latter approach to an extreme, encoding radiance at all points in a volume, and using evaluation of this approximation as the primary rendering method.

One of the most widely used applications of machine learning in rendering is denoising of path traced images (in practice, most path tracing now likely uses such a denoiser). Neural networks trained on pairs of noisy and low-noise images (or sometimes pairs of images with uncorrelated noise) can use data such as surface normal and albedo in addition to the rendered image, to reduce blurring or artifacts and preserve textures, and are effective at removing the scattered bright pixels called "fireflies" that frequently occur in path tracing. Denoisers specialized for real-time rendering can work with very low sample counts and improve temporal coherence to reduce flickering.

Hardware acceleration

Rendering is usually limited by available computing power and memory bandwidth, and so specialized hardware has been developed to speed it up ("accelerate" it), particularly for real-time rendering. Hardware features such as a framebuffer for raster graphics are required to display the output of rendering smoothly in real time.

Hardware acceleration does not replace the use of software for rendering, rather it speeds up selected operations or calculations using dedicated circuits, or runs portions of the software's code on a different type of processor.

History

In the era of vector monitors (also called calligraphic displays), a display processing unit (DPU) was a dedicated CPU or coprocessor that maintained a list of visual elements and redrew them continuously on the screen by controlling an electron beam. Advanced DPUs such as Evans & Sutherland's Line Drawing System-1 (and later models produced into the 1980s) incorporated 3D coordinate transformation features to accelerate rendering of wire-frame images. Evans & Sutherland also made the Digistar planetarium projection system, which was a vector display that could render both stars and wire-frame graphics (the vector-based Digistar and Digistar II were used in many planetariums, and a few may still be in operation). A Digistar prototype was used for rendering 3D star fields for the film Star Trek II: The Wrath of Khan – some of the first 3D computer graphics sequences ever seen in a feature film.

Shaded 3D graphics rendering in the 1970s and early 1980s was usually implemented on general-purpose computers, such as the PDP-10 used by researchers at the University of Utah. It was difficult to speed up using specialized hardware because it involves a pipeline of complex steps, requiring data addressing, decision-making, and computation capabilities typically only provided by CPUs (although dedicated circuits for speeding up particular operations were proposed). Supercomputers or specially designed multi-CPU computers or clusters were sometimes used for ray tracing. In 1981, James H. Clark and Marc Hannah designed the Geometry Engine, a VLSI chip for performing some of the steps of the 3D rasterization pipeline, and started the company Silicon Graphics (SGI) to commercialize this technology.

Home computers and game consoles in the 1980s contained graphics coprocessors that were capable of scrolling and filling areas of the display, and drawing sprites and lines, though they were not useful for rendering realistic images. Towards the end of the 1980s PC graphics cards and arcade games with 3D rendering acceleration began to appear, and in the 1990s such technology became commonplace. Today, even low-power mobile processors typically incorporate 3D graphics acceleration features.

GPUs

The 3D graphics accelerators of the 1990s evolved into modern GPUs. GPUs are general-purpose processors, like CPUs, but they are designed for tasks that can be broken into many small, similar, mostly independent sub-tasks (such as rendering individual pixels) and performed in parallel. This means that a GPU can speed up any rendering algorithm that can be split into subtasks in this way, in contrast to 1990s 3D accelerators which were only designed to speed up specific rasterization algorithms and simple shading and lighting effects (although tricks could be used to perform more general computations).

Due to their origins, GPUs typically still provide specialized hardware acceleration for some steps of a traditional 3D rasterization pipeline, including hidden surface removal using a z-buffer, and texture mapping with mipmaps, but these features are no longer always used. Recent GPUs have features to accelerate finding the intersections of rays with a bounding volume hierarchy, to help speed up all variants of ray tracing and path tracing, as well as neural network acceleration features sometimes useful for rendering.

GPUs are usually integrated with high-bandwidth memory systems to support the read and write bandwidth requirements of high-resolution, real-time rendering, particularly when multiple passes are required to render a frame, however memory latency may be higher than on a CPU, which can be a problem if the critical path in an algorithm involves many memory accesses. GPU design accepts high latency as inevitable (in part because a large number of threads are sharing the memory bus) and attempts to "hide" it by efficiently switching between threads, so a different thread can be performing computations while the first thread is waiting for a read or write to complete.

Rendering algorithms will run efficiently on a GPU only if they can be implemented using small groups of threads that perform mostly the same operations. As an example of code that meets this requirement: when rendering a small square of pixels in a simple ray-traced image, all threads will likely be intersecting rays with the same object and performing the same lighting computations. For performance and architectural reasons, GPUs run groups of around 16–64 threads called warps or wavefronts in lock-step (all threads in the group are executing the same instructions at the same time). If not all threads in the group need to run particular blocks of code (due to conditions) then some threads will be idle, or the results of their computations will be discarded, causing degraded performance.

Hardware and software rendering

Historically, the term hardware rendering (possibly an abbreviation of "hardware accelerated rendering" or "hardware-assisted rendering") was sometimes used to mean rendering using a hardware-accelerated rasterization pipeline (typically for real-time rendering). In contrast, software rendering meant offline rendering using software that was not limited by the capabilities of graphics hardware, and could use more realistic and higher quality techniques. Both hardware and software have evolved, and although these terms are still used, their meaning may now be context-dependent.

When OpenGL and Direct3D were introduced in the 1990s, there was a need to use these APIs on computers that did not have hardware acceleration for 3D graphics. "Fallback" rendering implementations were provided that did not require special hardware, and these are sometimes called "software renderers" (today hardware acceleration is almost always available, and these CPU-only implementations are used primarily for testing). More recently, "software rendering" may also mean rendering that does not use graphics APIs such as OpenGL, Metal, Direct3D, or Vulkan.

Types of renderers that in the past might have been called "software renderers" (e.g. path tracing renderers used for offline rendering for movies) now commonly use GPU acceleration, often via APIs such as CUDA or OpenCL, which are not graphics-specific. Since these latter APIs allow running C++ code on a GPU, it is now possible to run the same rendering code on either a CPU or GPU.

Chronology of algorithms and techniques

The following is a rough timeline of frequently mentioned rendering techniques, including areas of current research. Note that even in cases where an idea was named in a specific paper, there were almost always multiple researchers or teams working in the same area (including earlier related work). When a method is first proposed it is often very inefficient, and it takes additional research and practical efforts to turn it into a useful technique.

The list focuses on academic research and does not include hardware. (For more history see #External links, as well as Computer graphics#History and Golden age of arcade video games#Technology.)