diff --git a/research/README.md b/research/README.md new file mode 100644 index 0000000..65dfb4f --- /dev/null +++ b/research/README.md @@ -0,0 +1,22 @@ +# Research Archive + +This directory preserves implementation research and hard-won calibration +lessons. It is allowed to mention external tools, papers, vendor docs, and +failed paths because forgetting those days would be an expensive little act of +self-sabotage. + +Research is not runtime authority. + +Use it to shape native implementation decisions: + +- direct camera and audio drivers; +- pinned/native buffers; +- GPU feature extraction, flow, matching, and fusion; +- Aquarium D3D12 compute/render paths; +- Faust/native DSP; +- OBS-facing program outputs. + +Do not promote mirrored examples, old probes, or external library demos into the +Mimir hot path just because they exist here. If research points at a concept, +rebuild the live version in the Mimir/Aquarium/native stack with a clear owner +and invariant. diff --git a/research/ambisonic-aquasynth/bibliography.md b/research/ambisonic-aquasynth/bibliography.md new file mode 100644 index 0000000..0fa879a --- /dev/null +++ b/research/ambisonic-aquasynth/bibliography.md @@ -0,0 +1,43 @@ +# Bibliography: Ambisonic AquaSynth Audio Field + +Mirrored copies live under `research/ambisonic-aquasynth/mirrors/`. + +## Core Ambisonics + +- [ICST Ambisonics, "Ambisonics Formats Explained: A-Format, B-Format, FuMa, AmbiX, FOA and HOA"](https://ambisonics.ch/learn/ambisonics-formats/) + Mirror: `mirrors/icst-ambisonics-formats.html` + +- [RingBuffer, "Understanding Ambisonics"](https://ringbuffer.org/spatial_audio/ambisonics/understanding-ambisonics/) + Mirror: `mirrors/ringbuffer-understanding-ambisonics.html` + +- Leo McCormack, Archontis Politis, Raimundo Gonzalez, Tapio Lokki, Ville Pulkki, ["Parametric Ambisonic Encoding of Arbitrary Microphone Arrays"](https://doi.org/10.1109/TASLP.2022.3182857), IEEE/ACM TASLP, 2022. + Open access landing page: [Aaltodoc](https://aaltodoc.aalto.fi/items/cb74eb6f-dd96-4bde-826c-f77f837a42d8) + Mirrored PDF: `mirrors/mccormack-2022-parametric-ambisonic-encoding.pdf` + +## Faust / Production Tooling + +- [Faust Libraries, `hoa.lib`](https://faustlibraries.grame.fr/libs/hoa/) + Mirror: `mirrors/faust-hoa-lib.html` + +- [IEM Plug-in Suite](https://plugins.iem.at/) + Mirror: `mirrors/iem-plugin-suite.html` + +- [IEM Plug-in Suite, "Compatibility with other Suites"](https://plugins.iem.at/docs/compatibility/) + Mirror: `mirrors/iem-compatibility.html` + +- [SPARTA, "Overview"](https://leomccormack.github.io/sparta-site/docs/plugins/overview/) + Mirror: `mirrors/sparta-overview.html` + +## Capture, Clocking, Transport + +- [FFmpeg Devices Documentation](https://www.ffmpeg.org/ffmpeg-devices.html) + Mirror: `mirrors/ffmpeg-devices.html` + +- [JACK Audio Connection Kit API Overview](https://jackaudio.org/api/index.html) + Mirror: `mirrors/jack-api-overview.html` + +- [JACK Audio Connection Kit, "Using JACK on Windows"](https://jackaudio.org/faq/jack_on_windows.html) + Mirror: `mirrors/jack-on-windows.html` + +- [RME Manuals, "Using multiple HDSPe AoX cards"](https://docs.rme-audio.com/aoxd/705-1c_multiple_cards_win/) + Mirror: `mirrors/rme-multiple-cards-sync.html` diff --git a/research/ambisonic-aquasynth/mirrors/faust-hoa-lib.html b/research/ambisonic-aquasynth/mirrors/faust-hoa-lib.html new file mode 100644 index 0000000..3e3cbf6 --- /dev/null +++ b/research/ambisonic-aquasynth/mirrors/faust-hoa-lib.html @@ -0,0 +1,1130 @@ + + + + + + + + + + + hoa - Faust Libraries + + + + + + + + + + + + + + + + + +
+
+
+ +

hoa.lib

+

Higher-Order Ambisonics (HOA) library. Its official prefix is ho.

+

The HOA library provides functions and components for spatial audio rendering +and analysis using Higher-Order Ambisonics. It includes encoders, decoders, +rotators, and utilities for spherical harmonics and spatial transformations. +The library supports both 2D and 3D HOA processing workflows for immersive audio.

+

The HOA library is organized into 4 sections:

+ +

References

+ +

Encoding/decoding Functions

+
+

(ho.)encoder

+

Ambisonic encoder. Encodes a signal in the circular harmonics domain +depending on an order of decomposition and an angle.

+

Usage

+
encoder(N, x, a) : _
+
+

Where:

+
    +
  • N: the ambisonic order (constant numerical expression)
  • +
  • x: the signal
  • +
  • a: the angle
  • +
+

Test

+
ho = library("hoa.lib");
+os = library("oscillators.lib");
+monoSignal(freq) = os.osc(freq);
+encoder_test = ho.encoder(1, monoSignal(440), 0.0);
+
+
+

(ho.)rEncoder

+

Ambisonic encoder in 2D including source rotation. A mono signal is encoded at a certain ambisonic order +with two possible modes: either rotation with an angular speed, or static with a fixed angle (when speed is zero).

+

Usage

+
_ : rEncoder(N, sp, a, it) : _,_, ...
+
+

Where:

+
    +
  • N: the ambisonic order (constant numerical expression)
  • +
  • sp: the azimuth speed expressed as angular speed (2PI/sec), positive or negative
  • +
  • a: the fixed azimuth when the rotation stops (sp = 0) in radians
  • +
  • it : interpolation time (in milliseconds) between the rotation and the fixed modes
  • +
+

Test

+
ho = library("hoa.lib");
+os = library("oscillators.lib");
+monoSignal(freq) = os.osc(freq);
+rEncoder_test = monoSignal(440) : ho.rEncoder(1, 0.5, 0.0, 0.05);
+
+
+

(ho.)stereoEncoder

+

Encoding of a stereo pair of channels with symetric angles (a/2, -a/2).

+

Usage

+
_,_ : stereoEncoder(N, a) : _,_, ...
+
+

Where:

+
    +
  • N: the ambisonic order (constant numerical expression)
  • +
  • a : opening angle in radians, left channel at a/2 angle, right channel at -a/2 angle
  • +
+

Test

+
ho = library("hoa.lib");
+os = library("oscillators.lib");
+stereoSignal(f1, f2) = monoSignal(f1), monoSignal(f2);
+monoSignal(freq) = os.osc(freq);
+stereoEncoder_test = stereoSignal(440, 660) : ho.stereoEncoder(1, 1.0);
+
+
+

(ho.)multiEncoder

+

Encoding of a set of P signals distributed on the unit circle according to a list of P speeds and P angles.

+

Usage

+
_,_, ... : multiEncoder(N, lspeed, langle, it) : _,_, ...
+
+

Where:

+
    +
  • N: the ambisonic order (constant numerical expression)
  • +
  • lspeed : a list of P speeds in turns by second (one speed per input signal, positive or negative)
  • +
  • langle : a list of P angles in radians on the unit circle to localize the sources (one angle per input signal)
  • +
  • it : interpolation time (in milliseconds) between the rotation and the fixed modes.
  • +
+

Test

+
ho = library("hoa.lib");
+os = library("oscillators.lib");
+stereoSignal(f1, f2) = monoSignal(f1), monoSignal(f2);
+monoSignal(freq) = os.osc(freq);
+multiEncoder_test = stereoSignal(440, 660) : ho.multiEncoder(1, (0.0, 0.0), (0.0, 1.57), 0.05);
+
+
+

(ho.)decoder

+

Decodes an ambisonics sound field for a circular array of loudspeakers.

+

Usage

+
_ : decoder(N, P) : _
+
+

Where:

+
    +
  • N: the ambisonic order (constant numerical expression)
  • +
  • P: the number of speakers (constant numerical expression)
  • +
+

Test

+
ho = library("hoa.lib");
+os = library("oscillators.lib");
+ambi = ho.encoder(1, os.osc(440), 0.0);
+encoder_bus = ho.encoder(1, monoSignal(440), 0.0);
+monoSignal(freq) = os.osc(freq);
+decoder_test = encoder_bus : ho.decoder(1, 4);
+
+

Note

+

The number of loudspeakers must be greater or equal to 2n+1. +It's preferable to use 2n+2 loudspeakers.

+
+

(ho.)decoderStereo

+

Decodes an ambisonic sound field for stereophonic configuration. +An "home made" ambisonic decoder for stereophonic restitution +(30° - 330°): Sound field lose energy around 180°. You should +use inPhase optimization with ponctual sources.

+

Usage

+
_ : decoderStereo(N) : _
+
+

Where:

+
    +
  • N: the ambisonic order (constant numerical expression)
  • +
+

Test

+
ho = library("hoa.lib");
+os = library("oscillators.lib");
+ambi = ho.encoder(1, os.osc(440), 0.0);
+encoder_bus = ho.encoder(1, monoSignal(440), 0.0);
+monoSignal(freq) = os.osc(freq);
+decoderStereo_test = encoder_bus : ho.decoderStereo(1);
+
+
+

(ho.)iBasicDecoder

+

The irregular basic decoder is a simple decoder that projects the incoming ambisonic situation +to the loudspeaker situation (P loudspeakers) whatever it is, without compensation. +When there is a strong irregularity, there can be some discontinuity in the sound field.

+

Usage

+
_,_, ... : iBasicDecoder(N,la, direct, shift) : _,_, ...
+
+

Where:

+
    +
  • N: the ambisonic order (there are 2*N+1 inputs to this function)
  • +
  • la : the list of P angles in degrees, for instance (0, 85, 182, 263) for four loudspeakers
  • +
  • direct: 1 for direct mode, -1 for the indirect mode (changes the rotation direction)
  • +
  • shift : angular shift in degrees to easily adjust angles
  • +
+

Test

+
ho = library("hoa.lib");
+os = library("oscillators.lib");
+ambi = ho.encoder(1, os.osc(440), 0.0);
+encoder_bus = ho.encoder(1, monoSignal(440), 0.0);
+monoSignal(freq) = os.osc(freq);
+iBasicDecoder_test = encoder_bus : ho.iBasicDecoder(1, (0, 120, 240), 1, 0);
+
+
+

(ho.)circularScaledVBAP

+

The function provides a circular scaled VBAP with all loudspeakers and the virtual source on the unit-circle.

+

Usage

+
_ : circularScaledVBAP(l, t) : _,_, ...
+
+

Where:

+
    +
  • l : the list of angles of the loudspeakers in degrees, for instance (0, 85, 182, 263) for four loudspeakers
  • +
  • t : the current angle of the virtual source in degrees
  • +
+

Test

+
ho = library("hoa.lib");
+os = library("oscillators.lib");
+monoSignal(freq) = os.osc(freq);
+circularScaledVBAP_test = monoSignal(440) : ho.circularScaledVBAP((0, 120, 240), 60);
+
+
+

(ho.)imlsDecoder

+

Irregular decoder in 2D for an irregular configuration of P loudspeakers +using 2D VBAP for compensation.

+

Usage

+
_,_, ... : imlsDecoder(N,la, direct, shift) : _,_, ...
+
+

Where:

+
    +
  • N: the ambisonic order (constant numerical expression)
  • +
  • la : the list of P angles in degrees, for instance (0, 85, 182, 263) for four loudspeakers
  • +
  • direct: 1 for direct mode, -1 for the indirect mode (changes the rotation direction)
  • +
  • shift : angular shift in degrees to easily adjust angles
  • +
+

Test

+
ho = library("hoa.lib");
+os = library("oscillators.lib");
+ambi = ho.encoder(1, os.osc(440), 0.0);
+encoder_bus = ho.encoder(1, monoSignal(440), 0.0);
+monoSignal(freq) = os.osc(freq);
+imlsDecoder_test = encoder_bus : ho.imlsDecoder(1, (0, 90, 180, 270), 1, 0);
+
+
+

(ho.)iDecoder

+

General decoder in 2D enabling an irregular multi-loudspeaker configuration +and to switch between multi-channel and stereo.

+

Usage

+
_,_, ... : iDecoder(N, la, direct, st, g) : _,_, ...
+
+

Where:

+
    +
  • N: the ambisonic order (constant numerical expression)
  • +
  • la: the list of angles in degrees
  • +
  • direct: 1 for direct mode, -1 for the indirect mode (changes the rotation direction)
  • +
  • shift : angular shift in degrees to easily adjust angles
  • +
  • st: 1 for stereo, 0 for multi-loudspeaker configuration. When 1, stereo sounds goes through the first two channels
  • +
  • g : gain between 0 and 1
  • +
+

Test

+
ho = library("hoa.lib");
+os = library("oscillators.lib");
+ambi = ho.encoder(1, os.osc(440), 0.0);
+encoder_bus = ho.encoder(1, monoSignal(440), 0.0);
+monoSignal(freq) = os.osc(freq);
+iDecoder_test = (encoder_bus, 0.0) : ho.iDecoder(1, (0, 120, 240), 1, 0, 0.8);
+
+

Optimization Functions

+

Functions to weight the circular harmonics signals depending to the +ambisonics optimization. +It can be basic for no optimization, maxRe or inPhase.

+
+

(ho.)optimBasic

+

The basic optimization has no effect and should be used for a perfect +circle of loudspeakers with one listener at the perfect center loudspeakers +array.

+

Usage

+
_ : optimBasic(N) : _
+
+

Where:

+
    +
  • N: the ambisonic order (constant numerical expression)
  • +
+

Test

+
ho = library("hoa.lib");
+os = library("oscillators.lib");
+ambi = ho.encoder(1, os.osc(440), 0.0);
+encoder_bus = ho.encoder(1, monoSignal(440), 0.0);
+monoSignal(freq) = os.osc(freq);
+optimBasic_test = encoder_bus : ho.optimBasic(1);
+
+
+

(ho.)optimMaxRe

+

The maxRe optimization optimizes energy vector. It should be used for an +auditory confined in the center of the loudspeakers array.

+

Usage

+
_ : optimMaxRe(N) : _
+
+

Where:

+
    +
  • N: the ambisonic order (constant numerical expression)
  • +
+

Test

+
ho = library("hoa.lib");
+os = library("oscillators.lib");
+ambi = ho.encoder(1, os.osc(440), 0.0);
+encoder_bus = ho.encoder(1, monoSignal(440), 0.0);
+monoSignal(freq) = os.osc(freq);
+optimMaxRe_test = encoder_bus : ho.optimMaxRe(1);
+
+
+

(ho.)optimInPhase

+

The inPhase optimization optimizes energy vector and put all loudspeakers signals +in phase. It should be used for an auditory.

+

Usage

+
_ : optimInPhase(N) : _
+
+

Where:

+
    +
  • N: the ambisonic order (constant numerical expression)
  • +
+

Test

+
ho = library("hoa.lib");
+os = library("oscillators.lib");
+ambi = ho.encoder(1, os.osc(440), 0.0);
+encoder_bus = ho.encoder(1, monoSignal(440), 0.0);
+monoSignal(freq) = os.osc(freq);
+optimInPhase_test = encoder_bus : ho.optimInPhase(1);
+
+
+

(ho.)optim

+

Ambisonic optimizer including the three elementary optimizers: +(ho).optimBasic, (ho).optimMaxRe and (ho.)optimInPhase.

+

Usage

+
_,_, ... : optim(N, ot) : _,_, ...
+
+

Where:

+
    +
  • N: the ambisonic order (constant numerical expression)
  • +
  • ot : optimization type (0 for optimBasic, 1 for optimMaxRe, 2 for optimInPhase)
  • +
+

Test

+
ho = library("hoa.lib");
+os = library("oscillators.lib");
+ambi = ho.encoder(1, os.osc(440), 0.0);
+encoder_bus = ho.encoder(1, monoSignal(440), 0.0);
+monoSignal(freq) = os.osc(freq);
+optim_test = encoder_bus : ho.optim(1, 1);
+
+
+

(ho.)wider

+

Can be used to wide the diffusion of a localized sound. The order +depending signals are weighted and appear in a logarithmic way to +have linear changes.

+

Usage

+
_ : wider(N,w) : _
+
+

Where:

+
    +
  • N: the ambisonic order (constant numerical expression)
  • +
  • w: the width value between 0 - 1
  • +
+

Test

+
ho = library("hoa.lib");
+os = library("oscillators.lib");
+ambi = ho.encoder(1, os.osc(440), 0.0);
+encoder_bus = ho.encoder(1, monoSignal(440), 0.0);
+monoSignal(freq) = os.osc(freq);
+wider_test = encoder_bus : ho.wider(1, 0.5);
+
+
+

(ho.)mirror

+

Mirroring effect on the sound field.

+

Usage

+
_,_, ... : mirror(N, fa) : _,_, ...
+
+

Where:

+
    +
  • N: the ambisonic order (constant numerical expression)
  • +
  • fa : mirroring type (1 = original sound field, 0 = original+mirrored sound field, -1 = mirrored sound field)
  • +
+

Test

+
ho = library("hoa.lib");
+os = library("oscillators.lib");
+ambi = ho.encoder(1, os.osc(440), 0.0);
+encoder_bus = ho.encoder(1, monoSignal(440), 0.0);
+monoSignal(freq) = os.osc(freq);
+mirror_test = encoder_bus : ho.mirror(1, -1);
+
+
+

(ho.)map

+

It simulates the distance of the source by applying a gain +on the signal and a wider processing on the soundfield.

+

Usage

+
map(N, x, r, a)
+
+

Where:

+
    +
  • N: the ambisonic order (constant numerical expression)
  • +
  • x: the signal
  • +
  • r: the radius
  • +
  • a: the angle in radian
  • +
+

Test

+
ho = library("hoa.lib");
+os = library("oscillators.lib");
+monoSignal(freq) = os.osc(freq);
+map_test = ho.map(1, monoSignal(440), 0.5, 0.0);
+
+
+

(ho.)rotate

+

Rotates the sound field.

+

Usage

+
_ : rotate(N, a) : _
+
+

Where:

+
    +
  • N: the ambisonic order (constant numerical expression)
  • +
  • a: the angle in radian
  • +
+

Test

+
ho = library("hoa.lib");
+os = library("oscillators.lib");
+encoder_bus = ho.encoder(1, monoSignal(440), 0.0);
+monoSignal(freq) = os.osc(freq);
+rotate_test = encoder_bus : ho.rotate(1, 0.78);
+
+
+

(ho.)scope

+

Produces an XY pair of signals representing the ambisonic sound field.

+

Usage

+
_,_, ... : scope(N, rt) : _,_
+
+

Where:

+
    +
  • N: the ambisonic order (constant numerical expression)
  • +
  • rt : refreshment time in milliseconds
  • +
+

Test

+
ho = library("hoa.lib");
+os = library("oscillators.lib");
+encoder_bus = ho.encoder(1, monoSignal(440), 0.0);
+monoSignal(freq) = os.osc(freq);
+scope_test = encoder_bus : ho.scope(1, 0.1);
+
+

Spatial Sound Processes

+

We propose implementations of processes intricated to the ambisonic model. +The process is implemented using as many instances as the number of harmonics at at certain order. +The key control parameters of these instances are computed thanks to distribution functions +(th functions below) and to a global driving factor.

+
+

(ho.)fxDecorrelation

+

Spatial ambisonic decorrelation in fx mode.

+

fxDecorrelation applies decorrelations to spatial components already created. +The decorrelation is defined for each #i spatial component among P=2*N+1 at the ambisonic order N +as a delay of 0 if factor fa is under a certain value 1-(i+1)/P and d*F((i+1)/p) in the contrary case, +where d is the maximum delay applied (in samples) and F is a distribution function for durations. +The user can choose this delay time distribution among 22 different ones. +The delay increases according to the index of ambisonic components. +But it increases at each step and it is modulated by a threshold. +Therefore, delays are progressively revealed when the factor increases:

+
    +
  • when the factor is close to 0, only upper components are delayed;
  • +
  • when the factor increases, more and more components are delayed.
  • +
+

Usage

+
_,_, ... : fxDecorrelation(N, d, wf, fa, fd, tf) : _,_, ...
+
+

Where:

+
    +
  • N: the ambisonic order (constant numerical expression)
  • +
  • d: the maximum delay applied (in samples)
  • +
  • wf: window frequency (in Hz) for the overlapped delay
  • +
  • fa: decorrelation factor (between 0 and 1)
  • +
  • fd: feedback / level of reinjection (between 0 and 1)
  • +
  • tf: type of function of delay distribution (integer, between 0 and 21)
  • +
+

Test

+
ho = library("hoa.lib");
+os = library("oscillators.lib");
+encoder_bus = ho.encoder(1, monoSignal(440), 0.0);
+monoSignal(freq) = os.osc(freq);
+fxDecorrelation_test = encoder_bus : ho.fxDecorrelation(1, 64, 5, 0.5, 0.2, 0);
+
+
+

(ho.)synDecorrelation

+

Spatial ambisonic decorrelation in syn mode.

+

synDecorrelation generates spatial decorrelated components in ambisonics from one mono signal. +The decorrelation is defined for each #i spatial component among P=2*N+1 at the ambisonic order N +as a delay of 0 if factor fa is under a certain value 1-(i+1)/P and d*F((i+1)/p) in the contrary case, +where d is the maximum delay applied (in samples) and F is a distribution function for durations. +The user can choose this delay time distribution among 22 different ones. +The delay increases according to the index of ambisonic components. +But it increases at each step and it is modulated by a threshold. +Therefore, delays are progressively revealed when the factor increases:

+
    +
  • when the factor is close to 0, only upper components are delayed;
  • +
  • when the factor increases, more and more components are delayed.
  • +
+

When the factor is between [0; 1/P], upper harmonics are progressively faded and the level of the H0 component is compensated +to avoid source localization and to produce a large mono.

+

Usage

+
_,_, ... : synDecorrelation(N, d, wf, fa, fd, tf) : _,_, ...
+
+

Where:

+
    +
  • N: the ambisonic order (constant numerical expression)
  • +
  • d: the maximum delay applied (in samples)
  • +
  • wf: window frequency (in Hz) for the overlapped delay
  • +
  • fa: decorrelation factor (between 0 and 1)
  • +
  • fd: feedback / level of reinjection (between 0 and 1)
  • +
  • tf: type of function of delay distribution (integer, between 0 and 21)
  • +
+

Test

+
ho = library("hoa.lib");
+os = library("oscillators.lib");
+monoSignal(freq) = os.osc(freq);
+synDecorrelation_test = monoSignal(440) : ho.synDecorrelation(1, 64, 5, 0.5, 0.2, 0);
+
+
+

(ho.)fxRingMod

+

Spatial ring modulation in syn mode.

+

fxRingMod applies ring modulation to spatial components already created. +The ring modulation is defined for each spatial component among P=2*n+1 at the ambisonic order N. +For each spatial component #i, the result is either the original signal or a ring modulated signal +according to a threshold that is i/P.

+

The general process is drive by a factor fa between 0 and 1 and a modulation frequency f0. +If fa is greater than theshold (P-i-1)/P, the ith ring modulator is on with carrier frequency of f0*(i+1)/P. +On the contrary, it provides the original signal.

+

Therefore ring modulators are progressively revealed when fa increases.

+

Usage

+
_,_, ... : fxRingMod(N, f0, fa, tf) : _,_, ...
+
+

Where:

+
    +
  • N: the ambisonic order (constant numerical expression)
  • +
  • f0: the maximum delay applied (in samples)
  • +
  • fa: decorrelation factor (between 0 and 1)
  • +
  • tf: type of function of delay distribution (integer, between 0 and 21)
  • +
+

Test

+
ho = library("hoa.lib");
+os = library("oscillators.lib");
+encoder_bus = ho.encoder(1, monoSignal(440), 0.0);
+monoSignal(freq) = os.osc(freq);
+fxRingMod_test = encoder_bus : ho.fxRingMod(1, 200, 0.5, 0);
+
+
+

(ho.)synRingMod

+

Spatial ring modulation in syn mode.

+

synRingMod generates spatial components in ambisonics from one mono signal thanks to ring modulation. +The ring modulation is defined for each spatial component among P=2*n+1 at the ambisonic order N. +For each spatial component #i, the result is either the original signal or a ring modulated signal +according to a threshold that is i/P.

+

The general process is drive by a factor fa between 0 and 1 and a modulation frequency f0. +If fa is greater than theshold (P-i-1)/P, the ith ring modulator is on with carrier frequency of f0*(i+1)/P. +On the contrary, it provides the original signal.

+

Therefore ring modulators are progressively revealed when fa increases. +When the factor is between [0; 1/P], upper harmonics are progressively faded and the level of the H0 component is compensated +to avoid source localization and to produce a large mono.

+

Usage

+
_,_, ... : synRingMod(N, f0, fa, tf) : _,_, ...
+
+

Where:

+
    +
  • N: the ambisonic order (constant numerical expression)
  • +
  • f0: the maximum delay applied (in samples)
  • +
  • fa: decorrelation factor (between 0 and 1)
  • +
  • tf: type of function of delay distribution (integer, between 0 and 21)
  • +
+

Test

+
ho = library("hoa.lib");
+os = library("oscillators.lib");
+monoSignal(freq) = os.osc(freq);
+synRingMod_test = monoSignal(440) : ho.synRingMod(1, 200, 0.5, 0);
+
+

3D Functions

+
+

(ho.)encoder3D

+

Ambisonic encoder. Encodes a signal in the circular harmonics domain +depending on an order of decomposition, an angle and an elevation.

+

Usage

+
encoder3D(N, x, a, e) : _
+
+

Where:

+
    +
  • N: the ambisonic order (constant numerical expression)
  • +
  • x: the signal
  • +
  • a: the angle
  • +
  • e: the elevation
  • +
+

Test

+
ho = library("hoa.lib");
+os = library("oscillators.lib");
+encoder3D_base = ho.encoder3D(1, monoSignal(440), 0.0, 0.0);
+monoSignal(freq) = os.osc(freq);
+encoder3D_test = encoder3D_base;
+
+
+

(ho.)rEncoder3D

+

Ambisonic encoder in 3D including source rotation. A mono signal is encoded at at certain ambisonic order +with two possible modes: either rotation with 2 angular speeds (azimuth and elevation), or static with a fixed pair of angles.

+

rEncoder3D is a standard Faust function.

+

Usage

+
_ : rEncoder3D(N, azsp, elsp, az, el, it) : _,_, ...
+
+

Where:

+
    +
  • N: the ambisonic order (constant numerical expression)
  • +
  • azsp: the azimuth speed expressed as angular speed (2PI/sec), positive or negative
  • +
  • elsp: the elevation speed expressed as angular speed (2PI/sec), positive or negative
  • +
  • az: the fixed azimuth when the azimuth rotation stops (azsp = 0) in radians
  • +
  • el: the fixed elevation when the elevation rotation stops (elsp = 0) in radians
  • +
  • it : interpolation time (in milliseconds) between the rotation and the fixed modes
  • +
+

Test

+
ho = library("hoa.lib");
+os = library("oscillators.lib");
+monoSignal(freq) = os.osc(freq);
+rEncoder3D_test = monoSignal(440) : ho.rEncoder3D(1, 0.5, 0.3, 0.0, 0.0, 0.05);
+
+
+

(ho.)optimBasic3D

+

The basic optimization has no effect and should be used for a perfect +sphere of loudspeakers with one listener at the perfect center loudspeakers +array.

+

Usage

+
_ : optimBasic3D(N) : _
+
+

Where:

+
    +
  • N: the ambisonic order (constant numerical expression)
  • +
+

Test

+
ho = library("hoa.lib");
+os = library("oscillators.lib");
+ambi3D = ho.encoder3D(1, os.osc(440), 0.0, 0.0);
+encoder3D_base = ho.encoder3D(1, monoSignal(440), 0.0, 0.0);
+monoSignal(freq) = os.osc(freq);
+optimBasic3D_test = encoder3D_base : ho.optimBasic3D(1);
+
+
+

(ho.)optimMaxRe3D

+

The maxRe optimization optimize energy vector. It should be used for an +auditory confined in the center of the loudspeakers array.

+

Usage

+
_ : optimMaxRe3D(N) : _
+
+

Where:

+
    +
  • N: the ambisonic order (constant numerical expression)
  • +
+

Test

+
ho = library("hoa.lib");
+os = library("oscillators.lib");
+ambi3D = ho.encoder3D(1, os.osc(440), 0.0, 0.0);
+encoder3D_base = ho.encoder3D(1, monoSignal(440), 0.0, 0.0);
+monoSignal(freq) = os.osc(freq);
+optimMaxRe3D_test = encoder3D_base : ho.optimMaxRe3D(1);
+
+
+

(ho.)optimInPhase3D

+

The inPhase Optimization optimizes energy vector and put all loudspeakers signals +in phase. It should be used for an auditory.

+

Usage

+
_ : optimInPhase3D(N) : _
+
+

Where:

+
    +
  • N: the ambisonic order (constant numerical expression)
  • +
+

Test

+
ho = library("hoa.lib");
+os = library("oscillators.lib");
+ambi3D = ho.encoder3D(1, os.osc(440), 0.0, 0.0);
+encoder3D_base = ho.encoder3D(1, monoSignal(440), 0.0, 0.0);
+monoSignal(freq) = os.osc(freq);
+optimInPhase3D_test = encoder3D_base : ho.optimInPhase3D(1);
+
+
+

(ho.)optim3D

+

Ambisonic optimizer including the three elementary optimizers: +(ho).optimBasic3D, (ho).optimMaxRe3D and (ho.)optimInPhase3D.

+

Usage

+
_,_, ... : optim3D(N, ot) : _,_, ...
+
+

Where:

+
    +
  • N: the ambisonic order (constant numerical expression)
  • +
  • ot : optimization type (0 for optimBasic, 1 for optimMaxRe, 2 for optimInPhase)
  • +
+

Test

+
ho = library("hoa.lib");
+os = library("oscillators.lib");
+ambi3D = ho.encoder3D(1, os.osc(440), 0.0, 0.0);
+encoder3D_base = ho.encoder3D(1, monoSignal(440), 0.0, 0.0);
+monoSignal(freq) = os.osc(freq);
+optim3D_test = encoder3D_base : ho.optim3D(1, 2);
+
+
+
+ + + + + + + + + + + diff --git a/research/ambisonic-aquasynth/mirrors/ffmpeg-devices.html b/research/ambisonic-aquasynth/mirrors/ffmpeg-devices.html new file mode 100644 index 0000000..24f01e7 --- /dev/null +++ b/research/ambisonic-aquasynth/mirrors/ffmpeg-devices.html @@ -0,0 +1,2434 @@ + + + + + FFmpeg Devices Documentation + + + + + + + + + + + + + +
+ + + +
+
+

+ FFmpeg Devices Documentation +

+
+
+
+ + + + +

Table of Contents

+ + + + +

1 Description

+ +

This document describes the input and output devices provided by the +libavdevice library. +

+ +

2 Device Options

+ +

The libavdevice library provides the same interface as +libavformat. Namely, an input device is considered like a demuxer, and +an output device like a muxer, and the interface and generic device +options are the same provided by libavformat (see the ffmpeg-formats +manual). +

+

In addition each input or output device may support so-called private +options, which are specific for that component. +

+

Options may be set by specifying -option value in the +FFmpeg tools, or by setting the value explicitly in the device +AVFormatContext options or using the libavutil/opt.h API +for programmatic use. +

+ +

3 Input Devices

+ +

Input devices are configured elements in FFmpeg which enable accessing +the data coming from a multimedia device attached to your system. +

+

When you configure your FFmpeg build, all the supported input devices +are enabled by default. You can list all available ones using the +configure option "–list-indevs". +

+

You can disable all the input devices using the configure option +"–disable-indevs", and selectively enable an input device using the +option "–enable-indev=INDEV", or you can disable a particular +input device using the option "–disable-indev=INDEV". +

+

The option "-devices" of the ff* tools will display the list of +supported input devices. +

+

A description of the currently available input devices follows. +

+

3.1 alsa

+ +

ALSA (Advanced Linux Sound Architecture) input device. +

+

To enable this input device during configuration you need libasound +installed on your system. +

+

This device allows capturing from an ALSA device. The name of the +device to capture has to be an ALSA card identifier. +

+

An ALSA identifier has the syntax: +

+
hw:CARD[,DEV[,SUBDEV]]
+
+ +

where the DEV and SUBDEV components are optional. +

+

The three arguments (in order: CARD,DEV,SUBDEV) +specify card number or identifier, device number and subdevice number +(-1 means any). +

+

To see the list of cards currently recognized by your system check the +files /proc/asound/cards and /proc/asound/devices. +

+

For example to capture with ffmpeg from an ALSA device with +card id 0, you may run the command: +

+
ffmpeg -f alsa -i hw:0 alsaout.wav
+
+ +

For more information see: +http://www.alsa-project.org/alsa-doc/alsa-lib/pcm.html +

+

3.1.1 Options

+ +
+
sample_rate
+

Set the sample rate in Hz. Default is 48000. +

+
+
channels
+

Set the number of channels. Default is 2. +

+
+
+ +

3.2 android_camera

+ +

Android camera input device. +

+

This input devices uses the Android Camera2 NDK API which is +available on devices with API level 24+. The availability of +android_camera is autodetected during configuration. +

+

This device allows capturing from all cameras on an Android device, +which are integrated into the Camera2 NDK API. +

+

The available cameras are enumerated internally and can be selected +with the camera_index parameter. The input file string is +discarded. +

+

Generally the back facing camera has index 0 while the front facing +camera has index 1. +

+

3.2.1 Options

+ +
+
video_size
+

Set the video size given as a string such as 640x480 or hd720. +Falls back to the first available configuration reported by +Android if requested video size is not available or by default. +

+
+
framerate
+

Set the video framerate. +Falls back to the first available configuration reported by +Android if requested framerate is not available or by default (-1). +

+
+
camera_index
+

Set the index of the camera to use. Default is 0. +

+
+
input_queue_size
+

Set the maximum number of frames to buffer. Default is 5. +

+
+
+ +

3.3 avfoundation

+ +

AVFoundation input device. +

+

AVFoundation is the currently recommended framework by Apple for streamgrabbing on OSX >= 10.7 as well as on iOS. +

+

The input filename has to be given in the following syntax: +

+
-i "[[VIDEO]:[AUDIO]]"
+
+

The first entry selects the video input while the latter selects the audio input. +The stream has to be specified by the device name or the device index as shown by the device list. +Alternatively, the video and/or audio input device can be chosen by index using the + + -video_device_index <INDEX> + +and/or + + -audio_device_index <INDEX> + +, overriding any +device name or index given in the input filename. +

+

All available devices can be enumerated by using -list_devices true, listing +all device names and corresponding indices. +

+

There are two device name aliases: +

+
default
+

Select the AVFoundation default device of the corresponding type. +

+
+
none
+

Do not record the corresponding media type. +This is equivalent to specifying an empty device name or index. +

+
+
+ +

3.3.1 Options

+ +

AVFoundation supports the following options: +

+
+
-list_devices <TRUE|FALSE>
+

If set to true, a list of all available input devices is given showing all +device names and indices. +

+
+
-video_device_index <INDEX>
+

Specify the video device by its index. Overrides anything given in the input filename. +

+
+
-audio_device_index <INDEX>
+

Specify the audio device by its index. Overrides anything given in the input filename. +

+
+
-pixel_format <FORMAT>
+

Request the video device to use a specific pixel format. +If the specified format is not supported, a list of available formats is given +and the first one in this list is used instead. Available pixel formats are: +monob, rgb555be, rgb555le, rgb565be, rgb565le, rgb24, bgr24, 0rgb, bgr0, 0bgr, rgb0, + bgr48be, uyvy422, yuva444p, yuva444p16le, yuv444p, yuv422p16, yuv422p10, yuv444p10, + yuv420p, nv12, yuyv422, gray +

+
+
-framerate
+

Set the grabbing frame rate. Default is ntsc, corresponding to a +frame rate of 30000/1001. +

+
+
-video_size
+

Set the video frame size. +

+
+
-capture_cursor
+

Capture the mouse pointer. Default is 0. +

+
+
-capture_mouse_clicks
+

Capture the screen mouse clicks. Default is 0. +

+
+
-capture_raw_data
+

Capture the raw device data. Default is 0. +Using this option may result in receiving the underlying data delivered to the AVFoundation framework. E.g. for muxed devices that sends raw DV data to the framework (like tape-based camcorders), setting this option to false results in extracted video frames captured in the designated pixel format only. Setting this option to true results in receiving the raw DV stream untouched. +

+
+ +

3.3.2 Examples

+ +
    +
  • Print the list of AVFoundation supported devices and exit: +
    +
    $ ffmpeg -f avfoundation -list_devices true -i ""
    +
    + +
  • Record video from video device 0 and audio from audio device 0 into out.avi: +
    +
    $ ffmpeg -f avfoundation -i "0:0" out.avi
    +
    + +
  • Record video from video device 2 and audio from audio device 1 into out.avi: +
    +
    $ ffmpeg -f avfoundation -video_device_index 2 -i ":1" out.avi
    +
    + +
  • Record video from the system default video device using the pixel format bgr0 and do not record any audio into out.avi: +
    +
    $ ffmpeg -f avfoundation -pixel_format bgr0 -i "default:none" out.avi
    +
    + +
  • Record raw DV data from a suitable input device and write the output into out.dv: +
    +
    $ ffmpeg -f avfoundation -capture_raw_data true -i "zr100:none" out.dv
    +
    + + +
+ +

3.4 decklink

+ +

The decklink input device provides capture capabilities for Blackmagic +DeckLink devices. +

+

To enable this input device, you need the Blackmagic DeckLink SDK and you +need to configure with the appropriate --extra-cflags +and --extra-ldflags. +On Windows, you need to run the IDL files through widl. +

+

DeckLink is very picky about the formats it supports. Pixel format of the +input can be set with raw_format. +Framerate and video size must be determined for your device with +-list_formats 1. Audio sample rate is always 48 kHz and the number +of channels can be 2, 8 or 16. Note that all audio channels are bundled in one single +audio track. +

+

3.4.1 Options

+ +
+
list_devices
+

If set to true, print a list of devices and exit. +Defaults to false. This option is deprecated, please use the +-sources option of ffmpeg to list the available input devices. +

+
+
list_formats
+

If set to true, print a list of supported formats and exit. +Defaults to false. +

+
+
format_code <FourCC>
+

This sets the input video format to the format given by the FourCC. To see +the supported values of your device(s) use list_formats. +Note that there is a FourCC 'pal ' that can also be used +as pal (3 letters). +Default behavior is autodetection of the input video format, if the hardware +supports it. +

+
+
raw_format
+

Set the pixel format of the captured video. +Available values are: +

+
‘auto’
+
+

This is the default which means 8-bit YUV 422 or 8-bit ARGB if format +autodetection is used, 8-bit YUV 422 otherwise. +

+
+
‘uyvy422’
+
+

8-bit YUV 422. +

+
+
‘yuv422p10’
+
+

10-bit YUV 422. +

+
+
‘argb’
+
+

8-bit RGB. +

+
+
‘bgra’
+
+

8-bit RGB. +

+
+
‘rgb10’
+
+

10-bit RGB. +

+
+
+ +
+
teletext_lines
+

If set to nonzero, an additional teletext stream will be captured from the +vertical ancillary data. Both SD PAL (576i) and HD (1080i or 1080p) +sources are supported. In case of HD sources, OP47 packets are decoded. +

+

This option is a bitmask of the SD PAL VBI lines captured, specifically lines 6 +to 22, and lines 318 to 335. Line 6 is the LSB in the mask. Selected lines +which do not contain teletext information will be ignored. You can use the +special all constant to select all possible lines, or +standard to skip lines 6, 318 and 319, which are not compatible with +all receivers. +

+

For SD sources, ffmpeg needs to be compiled with --enable-libzvbi. For +HD sources, on older (pre-4K) DeckLink card models you have to capture in 10 +bit mode. +

+
+
channels
+

Defines number of audio channels to capture. Must be ‘2’, ‘8’ or ‘16’. +Defaults to ‘2’. +

+
+
duplex_mode
+

Sets the decklink device duplex/profile mode. Must be ‘unset’, ‘half’, ‘full’, +‘one_sub_device_full’, ‘one_sub_device_half’, ‘two_sub_device_full’, +‘four_sub_device_half’ +Defaults to ‘unset’. +

+

Note: DeckLink SDK 11.0 have replaced the duplex property by a profile property. +For the DeckLink Duo 2 and DeckLink Quad 2, a profile is shared between any 2 +sub-devices that utilize the same connectors. For the DeckLink 8K Pro, a profile +is shared between all 4 sub-devices. So DeckLink 8K Pro support four profiles. +

+

Valid profile modes for DeckLink 8K Pro(with DeckLink SDK >= 11.0): +‘one_sub_device_full’, ‘one_sub_device_half’, ‘two_sub_device_full’, +‘four_sub_device_half’ +

+

Valid profile modes for DeckLink Quad 2 and DeckLink Duo 2: +‘half’, ‘full’ +

+
+
timecode_format
+

Timecode type to include in the frame and video stream metadata. Must be +‘none’, ‘rp188vitc’, ‘rp188vitc2’, ‘rp188ltc’, +‘rp188hfr’, ‘rp188any’, ‘vitc’, ‘vitc2’, or ‘serial’. +Defaults to ‘none’ (not included). +

+

In order to properly support 50/60 fps timecodes, the ordering of the queried +timecode types for ‘rp188any’ is HFR, VITC1, VITC2 and LTC for >30 fps +content. Note that this is slightly different to the ordering used by the +DeckLink API, which is HFR, VITC1, LTC, VITC2. +

+
+
video_input
+

Sets the video input source. Must be ‘unset’, ‘sdi’, ‘hdmi’, +‘optical_sdi’, ‘component’, ‘composite’ or ‘s_video’. +Defaults to ‘unset’. +

+
+
audio_input
+

Sets the audio input source. Must be ‘unset’, ‘embedded’, +‘aes_ebu’, ‘analog’, ‘analog_xlr’, ‘analog_rca’ or +‘microphone’. Defaults to ‘unset’. +

+
+
video_pts
+

Sets the video packet timestamp source. Must be ‘video’, ‘audio’, +‘reference’, ‘wallclock’ or ‘abs_wallclock’. +Defaults to ‘video’. +

+
+
audio_pts
+

Sets the audio packet timestamp source. Must be ‘video’, ‘audio’, +‘reference’, ‘wallclock’ or ‘abs_wallclock’. +Defaults to ‘audio’. +

+
+
draw_bars
+

If set to ‘true’, color bars are drawn in the event of a signal loss. +Defaults to ‘true’. +This option is deprecated, please use the signal_loss_action option. +

+
+
signal_loss_action
+

Sets the action to take in the event of a signal loss. Accepts one of the +following values: +

+
+
1, none
+

Do nothing on signal loss. This usually results in black frames. +

+
2, bars
+

Draw color bars on signal loss. Only supported for 8-bit input signals. +

+
3, repeat
+

Repeat the last video frame on signal loss. +

+
+ +

Defaults to ‘bars’. +

+
+
queue_size
+

Sets maximum input buffer size in bytes. If the buffering reaches this value, +incoming frames will be dropped. +Defaults to ‘1073741824’. +

+
+
audio_depth
+

Sets the audio sample bit depth. Must be ‘16’ or ‘32’. +Defaults to ‘16’. +

+
+
decklink_copyts
+

If set to true, timestamps are forwarded as they are without removing +the initial offset. +Defaults to false. +

+
+
timestamp_align
+

Capture start time alignment in seconds. If set to nonzero, input frames are +dropped till the system timestamp aligns with configured value. +Alignment difference of up to one frame duration is tolerated. +This is useful for maintaining input synchronization across N different +hardware devices deployed for ’N-way’ redundancy. The system time of different +hardware devices should be synchronized with protocols such as NTP or PTP, +before using this option. +Note that this method is not foolproof. In some border cases input +synchronization may not happen due to thread scheduling jitters in the OS. +Either sync could go wrong by 1 frame or in a rarer case +timestamp_align seconds. +Defaults to ‘0’. +

+
+
wait_for_tc (bool)
+

Drop frames till a frame with timecode is received. Sometimes serial timecode +isn’t received with the first input frame. If that happens, the stored stream +timecode will be inaccurate. If this option is set to true, input frames +are dropped till a frame with timecode is received. +Option timecode_format must be specified. +Defaults to false. +

+
+
enable_klv(bool)
+

If set to true, extracts KLV data from VANC and outputs KLV packets. +KLV VANC packets are joined based on MID and PSC fields and aggregated into +one KLV packet. +Defaults to false. +

+
+
+ +

3.4.2 Examples

+ +
    +
  • List input devices: +
    +
    ffmpeg -sources decklink
    +
    + +
  • List supported formats: +
    +
    ffmpeg -f decklink -list_formats 1 -i 'Intensity Pro'
    +
    + +
  • Capture video clip at 1080i50: +
    +
    ffmpeg -format_code Hi50 -f decklink -i 'Intensity Pro' -c:a copy -c:v copy output.avi
    +
    + +
  • Capture video clip at 1080i50 10 bit: +
    +
    ffmpeg -raw_format yuv422p10 -format_code Hi50 -f decklink -i 'UltraStudio Mini Recorder' -c:a copy -c:v copy output.avi
    +
    + +
  • Capture video clip at 1080i50 with 16 audio channels: +
    +
    ffmpeg -channels 16 -format_code Hi50 -f decklink -i 'UltraStudio Mini Recorder' -c:a copy -c:v copy output.avi
    +
    + +
+ +

3.5 dshow

+ +

Windows DirectShow input device. +

+

DirectShow support is enabled when FFmpeg is built with the mingw-w64 project. +Currently only audio and video devices are supported. +

+

Multiple devices may be opened as separate inputs, but they may also be +opened on the same input, which should improve synchronism between them. +

+

The input name should be in the format: +

+
+
TYPE=NAME[:TYPE=NAME]
+
+ +

where TYPE can be either audio or video, +and NAME is the device’s name or alternative name.. +

+

3.5.1 Options

+ +

If no options are specified, the device’s defaults are used. +If the device does not support the requested options, it will +fail to open. +

+
+
video_size
+

Set the video size in the captured video. +

+
+
framerate
+

Set the frame rate in the captured video. +

+
+
sample_rate
+

Set the sample rate (in Hz) of the captured audio. +

+
+
sample_size
+

Set the sample size (in bits) of the captured audio. +

+
+
channels
+

Set the number of channels in the captured audio. +

+
+
list_devices
+

If set to true, print a list of devices and exit. +

+
+
list_options
+

If set to true, print a list of selected device’s options +and exit. +

+
+
video_device_number
+

Set video device number for devices with the same name (starts at 0, +defaults to 0). +

+
+
audio_device_number
+

Set audio device number for devices with the same name (starts at 0, +defaults to 0). +

+
+
pixel_format
+

Select pixel format to be used by DirectShow. This may only be set when +the video codec is not set or set to rawvideo. +

+
+
audio_buffer_size
+

Set audio device buffer size in milliseconds (which can directly +impact latency, depending on the device). +Defaults to using the audio device’s +default buffer size (typically some multiple of 500ms). +Setting this value too low can degrade performance. +See also +http://msdn.microsoft.com/en-us/library/windows/desktop/dd377582(v=vs.85).aspx +

+
+
video_pin_name
+

Select video capture pin to use by name or alternative name. +

+
+
audio_pin_name
+

Select audio capture pin to use by name or alternative name. +

+
+
crossbar_video_input_pin_number
+

Select video input pin number for crossbar device. This will be +routed to the crossbar device’s Video Decoder output pin. +Note that changing this value can affect future invocations +(sets a new default) until system reboot occurs. +

+
+
crossbar_audio_input_pin_number
+

Select audio input pin number for crossbar device. This will be +routed to the crossbar device’s Audio Decoder output pin. +Note that changing this value can affect future invocations +(sets a new default) until system reboot occurs. +

+
+
show_video_device_dialog
+

If set to true, before capture starts, popup a display dialog +to the end user, allowing them to change video filter properties +and configurations manually. +Note that for crossbar devices, adjusting values in this dialog +may be needed at times to toggle between PAL (25 fps) and NTSC (29.97) +input frame rates, sizes, interlacing, etc. Changing these values can +enable different scan rates/frame rates and avoiding green bars at +the bottom, flickering scan lines, etc. +Note that with some devices, changing these properties can also affect future +invocations (sets new defaults) until system reboot occurs. +

+
+
show_audio_device_dialog
+

If set to true, before capture starts, popup a display dialog +to the end user, allowing them to change audio filter properties +and configurations manually. +

+
+
show_video_crossbar_connection_dialog
+

If set to true, before capture starts, popup a display +dialog to the end user, allowing them to manually +modify crossbar pin routings, when it opens a video device. +

+
+
show_audio_crossbar_connection_dialog
+

If set to true, before capture starts, popup a display +dialog to the end user, allowing them to manually +modify crossbar pin routings, when it opens an audio device. +

+
+
show_analog_tv_tuner_dialog
+

If set to true, before capture starts, popup a display +dialog to the end user, allowing them to manually +modify TV channels and frequencies. +

+
+
show_analog_tv_tuner_audio_dialog
+

If set to true, before capture starts, popup a display +dialog to the end user, allowing them to manually +modify TV audio (like mono vs. stereo, Language A,B or C). +

+
+
audio_device_load
+

Load an audio capture filter device from file instead of searching +it by name. It may load additional parameters too, if the filter +supports the serialization of its properties to. +To use this an audio capture source has to be specified, but it can +be anything even fake one. +

+
+
audio_device_save
+

Save the currently used audio capture filter device and its +parameters (if the filter supports it) to a file. +If a file with the same name exists it will be overwritten. +

+
+
video_device_load
+

Load a video capture filter device from file instead of searching +it by name. It may load additional parameters too, if the filter +supports the serialization of its properties to. +To use this a video capture source has to be specified, but it can +be anything even fake one. +

+
+
video_device_save
+

Save the currently used video capture filter device and its +parameters (if the filter supports it) to a file. +If a file with the same name exists it will be overwritten. +

+
+
use_video_device_timestamps
+

If set to false, the timestamp for video frames will be +derived from the wallclock instead of the timestamp provided by +the capture device. This allows working around devices that +provide unreliable timestamps. +

+
+
+ +

3.5.2 Examples

+ +
    +
  • Print the list of DirectShow supported devices and exit: +
    +
    $ ffmpeg -list_devices true -f dshow -i dummy
    +
    + +
  • Open video device Camera: +
    +
    $ ffmpeg -f dshow -i video="Camera"
    +
    + +
  • Open second video device with name Camera: +
    +
    $ ffmpeg -f dshow -video_device_number 1 -i video="Camera"
    +
    + +
  • Open video device Camera and audio device Microphone: +
    +
    $ ffmpeg -f dshow -i video="Camera":audio="Microphone"
    +
    + +
  • Print the list of supported options in selected device and exit: +
    +
    $ ffmpeg -list_options true -f dshow -i video="Camera"
    +
    + +
  • Specify pin names to capture by name or alternative name, specify alternative device name: +
    +
    $ ffmpeg -f dshow -audio_pin_name "Audio Out" -video_pin_name 2 -i video=video="@device_pnp_\\?\pci#ven_1a0a&dev_6200&subsys_62021461&rev_01#4&e2c7dd6&0&00e1#{65e8773d-8f56-11d0-a3b9-00a0c9223196}\{ca465100-deb0-4d59-818f-8c477184adf6}":audio="Microphone"
    +
    + +
  • Configure a crossbar device, specifying crossbar pins, allow user to adjust video capture properties at startup: +
    +
    $ ffmpeg -f dshow -show_video_device_dialog true -crossbar_video_input_pin_number 0
    +     -crossbar_audio_input_pin_number 3 -i video="AVerMedia BDA Analog Capture":audio="AVerMedia BDA Analog Capture"
    +
    + +
+ +

3.6 fbdev

+ +

Linux framebuffer input device. +

+

The Linux framebuffer is a graphic hardware-independent abstraction +layer to show graphics on a computer monitor, typically on the +console. It is accessed through a file device node, usually +/dev/fb0. +

+

For more detailed information read the file +Documentation/fb/framebuffer.txt included in the Linux source tree. +

+

See also http://linux-fbdev.sourceforge.net/, and fbset(1). +

+

To record from the framebuffer device /dev/fb0 with +ffmpeg: +

+
ffmpeg -f fbdev -framerate 10 -i /dev/fb0 out.avi
+
+ +

You can take a single screenshot image with the command: +

+
ffmpeg -f fbdev -framerate 1 -i /dev/fb0 -frames:v 1 screenshot.jpeg
+
+ +

3.6.1 Options

+ +
+
framerate
+

Set the frame rate. Default is 25. +

+
+
+ +

3.7 gdigrab

+ +

Win32 GDI-based screen capture device. +

+

This device allows you to capture a region of the display on Windows. +

+

Amongst options for the input filenames are such elements as: +

+
desktop
+
+

or +

+
title=window_title
+
+

or +

+
hwnd=window_hwnd
+
+ +

The first option will capture the entire desktop, or a fixed region of the +desktop. The second and third options will instead capture the contents of a single +window, regardless of its position on the screen. +

+

For example, to grab the entire desktop using ffmpeg: +

+
ffmpeg -f gdigrab -framerate 6 -i desktop out.mpg
+
+ +

Grab a 640x480 region at position 10,20: +

+
ffmpeg -f gdigrab -framerate 6 -offset_x 10 -offset_y 20 -video_size vga -i desktop out.mpg
+
+ +

Grab the contents of the window named "Calculator" +

+
ffmpeg -f gdigrab -framerate 6 -i title=Calculator out.mpg
+
+ +

3.7.1 Options

+ +
+
draw_mouse
+

Specify whether to draw the mouse pointer. Use the value 0 to +not draw the pointer. Default value is 1. +

+
+
framerate
+

Set the grabbing frame rate. Default value is ntsc, +corresponding to a frame rate of 30000/1001. +

+
+
show_region
+

Show grabbed region on screen. +

+

If show_region is specified with 1, then the grabbing +region will be indicated on screen. With this option, it is easy to +know what is being grabbed if only a portion of the screen is grabbed. +

+

Note that show_region is incompatible with grabbing the contents +of a single window. +

+

For example: +

+
ffmpeg -f gdigrab -show_region 1 -framerate 6 -video_size cif -offset_x 10 -offset_y 20 -i desktop out.mpg
+
+ +
+
video_size
+

Set the video frame size. The default is to capture the full screen if desktop is selected, or the full window size if title=window_title is selected. +

+
+
offset_x
+

When capturing a region with video_size, set the distance from the left edge of the screen or desktop. +

+

Note that the offset calculation is from the top left corner of the primary monitor on Windows. If you have a monitor positioned to the left of your primary monitor, you will need to use a negative offset_x value to move the region to that monitor. +

+
+
offset_y
+

When capturing a region with video_size, set the distance from the top edge of the screen or desktop. +

+

Note that the offset calculation is from the top left corner of the primary monitor on Windows. If you have a monitor positioned above your primary monitor, you will need to use a negative offset_y value to move the region to that monitor. +

+
+
+ +

3.8 iec61883

+ +

FireWire DV/HDV input device using libiec61883. +

+

To enable this input device, you need libiec61883, libraw1394 and +libavc1394 installed on your system. Use the configure option +--enable-libiec61883 to compile with the device enabled. +

+

The iec61883 capture device supports capturing from a video device +connected via IEEE1394 (FireWire), using libiec61883 and the new Linux +FireWire stack (juju). This is the default DV/HDV input method in Linux +Kernel 2.6.37 and later, since the old FireWire stack was removed. +

+

Specify the FireWire port to be used as input file, or "auto" +to choose the first port connected. +

+

3.8.1 Options

+ +
+
dvtype
+

Override autodetection of DV/HDV. This should only be used if auto +detection does not work, or if usage of a different device type +should be prohibited. Treating a DV device as HDV (or vice versa) will +not work and result in undefined behavior. +The values auto, dv and hdv are supported. +

+
+
dvbuffer
+

Set maximum size of buffer for incoming data, in frames. For DV, this +is an exact value. For HDV, it is not frame exact, since HDV does +not have a fixed frame size. +

+
+
dvguid
+

Select the capture device by specifying its GUID. Capturing will only +be performed from the specified device and fails if no device with the +given GUID is found. This is useful to select the input if multiple +devices are connected at the same time. +Look at /sys/bus/firewire/devices to find out the GUIDs. +

+
+
+ +

3.8.2 Examples

+ +
    +
  • Grab and show the input of a FireWire DV/HDV device. +
    +
    ffplay -f iec61883 -i auto
    +
    + +
  • Grab and record the input of a FireWire DV/HDV device, +using a packet buffer of 100000 packets if the source is HDV. +
    +
    ffmpeg -f iec61883 -i auto -dvbuffer 100000 out.mpg
    +
    + +
+ +

3.9 jack

+ +

JACK input device. +

+

To enable this input device during configuration you need libjack +installed on your system. +

+

A JACK input device creates one or more JACK writable clients, one for +each audio channel, with name client_name:input_N, where +client_name is the name provided by the application, and N +is a number which identifies the channel. +Each writable client will send the acquired data to the FFmpeg input +device. +

+

Once you have created one or more JACK readable clients, you need to +connect them to one or more JACK writable clients. +

+

To connect or disconnect JACK clients you can use the jack_connect +and jack_disconnect programs, or do it through a graphical interface, +for example with qjackctl. +

+

To list the JACK clients and their properties you can invoke the command +jack_lsp. +

+

Follows an example which shows how to capture a JACK readable client +with ffmpeg. +

+
# Create a JACK writable client with name "ffmpeg".
+$ ffmpeg -f jack -i ffmpeg -y out.wav
+
+# Start the sample jack_metro readable client.
+$ jack_metro -b 120 -d 0.2 -f 4000
+
+# List the current JACK clients.
+$ jack_lsp -c
+system:capture_1
+system:capture_2
+system:playback_1
+system:playback_2
+ffmpeg:input_1
+metro:120_bpm
+
+# Connect metro to the ffmpeg writable client.
+$ jack_connect metro:120_bpm ffmpeg:input_1
+
+ +

For more information read: +http://jackaudio.org/ +

+

3.9.1 Options

+ +
+
channels
+

Set the number of channels. Default is 2. +

+
+
+ +

3.10 kmsgrab

+ +

KMS video input device. +

+

Captures the KMS scanout framebuffer associated with a specified CRTC or plane as a +DRM object that can be passed to other hardware functions. +

+

Requires either DRM master or CAP_SYS_ADMIN to run. +

+

If you don’t understand what all of that means, you probably don’t want this. Look at +x11grab instead. +

+

3.10.1 Options

+ +
+
device
+

DRM device to capture on. Defaults to /dev/dri/card0. +

+
+
format
+

Pixel format of the framebuffer. This can be autodetected if you are running Linux 5.7 +or later, but needs to be provided for earlier versions. Defaults to bgr0, +which is the most common format used by the Linux console and Xorg X server. +

+
+
format_modifier
+

Format modifier to signal on output frames. This is necessary to import correctly into +some APIs. It can be autodetected if you are running Linux 5.7 or later, but will need +to be provided explicitly when needed in earlier versions. See the libdrm documentation +for possible values. +

+
+
crtc_id
+

KMS CRTC ID to define the capture source. The first active plane on the given CRTC +will be used. +

+
+
plane_id
+

KMS plane ID to define the capture source. Defaults to the first active plane found if +neither crtc_id nor plane_id are specified. +

+
+
framerate
+

Framerate to capture at. This is not synchronised to any page flipping or framebuffer +changes - it just defines the interval at which the framebuffer is sampled. Sampling +faster than the framebuffer update rate will generate independent frames with the same +content. Defaults to 30. +

+
+
+ +

3.10.2 Examples

+ +
    +
  • Capture from the first active plane, download the result to normal frames and encode. +This will only work if the framebuffer is both linear and mappable - if not, the result +may be scrambled or fail to download. +
    +
    ffmpeg -f kmsgrab -i - -vf 'hwdownload,format=bgr0' output.mp4
    +
    + +
  • Capture from CRTC ID 42 at 60fps, map the result to VAAPI, convert to NV12 and encode as H.264. +
    +
    ffmpeg -crtc_id 42 -framerate 60 -f kmsgrab -i - -vf 'hwmap=derive_device=vaapi,scale_vaapi=w=1920:h=1080:format=nv12' -c:v h264_vaapi output.mp4
    +
    + +
  • To capture only part of a plane the output can be cropped - this can be used to capture +a single window, as long as it has a known absolute position and size. For example, to +capture and encode the middle quarter of a 1920x1080 plane: +
    +
    ffmpeg -f kmsgrab -i - -vf 'hwmap=derive_device=vaapi,crop=960:540:480:270,scale_vaapi=960:540:nv12' -c:v h264_vaapi output.mp4
    +
    + +
+ +

3.11 lavfi

+ +

Libavfilter input virtual device. +

+

This input device reads data from the open output pads of a libavfilter +filtergraph. +

+

For each filtergraph open output, the input device will create a +corresponding stream which is mapped to the generated output. +The filtergraph is specified through the option graph. +

+

3.11.1 Options

+ +
+
graph
+

Specify the filtergraph to use as input. Each video open output must be +labelled by a unique string of the form "outN", where N is a +number starting from 0 corresponding to the mapped input stream +generated by the device. +The first unlabelled output is automatically assigned to the "out0" +label, but all the others need to be specified explicitly. +

+

The suffix "+subcc" can be appended to the output label to create an extra +stream with the closed captions packets attached to that output +(experimental; only for EIA-608 / CEA-708 for now). +The subcc streams are created after all the normal streams, in the order of +the corresponding stream. +For example, if there is "out19+subcc", "out7+subcc" and up to "out42", the +stream #43 is subcc for stream #7 and stream #44 is subcc for stream #19. +

+

If not specified defaults to the filename specified for the input +device. +

+
+
graph_file
+

Set the filename of the filtergraph to be read and sent to the other +filters. Syntax of the filtergraph is the same as the one specified by +the option graph. +

+
+
dumpgraph
+

Dump graph to stderr. +

+
+
+ +

3.11.2 Examples

+ +
    +
  • Create a color video stream and play it back with ffplay: +
    +
    ffplay -f lavfi -graph "color=c=pink [out0]" dummy
    +
    + +
  • As the previous example, but use filename for specifying the graph +description, and omit the "out0" label: +
    +
    ffplay -f lavfi color=c=pink
    +
    + +
  • Create three different video test filtered sources and play them: +
    +
    ffplay -f lavfi -graph "testsrc [out0]; testsrc,hflip [out1]; testsrc,negate [out2]" test3
    +
    + +
  • Read an audio stream from a file using the amovie source and play it +back with ffplay: +
    +
    ffplay -f lavfi "amovie=test.wav"
    +
    + +
  • Read an audio stream and a video stream and play it back with +ffplay: +
    +
    ffplay -f lavfi "movie=test.avi[out0];amovie=test.wav[out1]"
    +
    + +
  • Dump decoded frames to images and Closed Captions to an RCWT backup: +
    +
    ffmpeg -f lavfi -i "movie=test.ts[out0+subcc]" -map v frame%08d.png -map s -c copy -f rcwt subcc.bin
    +
    + +
+ +

3.12 libcdio

+ +

Audio-CD input device based on libcdio. +

+

To enable this input device during configuration you need libcdio +installed on your system. It requires the configure option +--enable-libcdio. +

+

This device allows playing and grabbing from an Audio-CD. +

+

For example to copy with ffmpeg the entire Audio-CD in /dev/sr0, +you may run the command: +

+
ffmpeg -f libcdio -i /dev/sr0 cd.wav
+
+ +

3.12.1 Options

+
+
speed
+

Set drive reading speed. Default value is 0. +

+

The speed is specified CD-ROM speed units. The speed is set through +the libcdio cdio_cddap_speed_set function. On many CD-ROM +drives, specifying a value too large will result in using the fastest +speed. +

+
+
paranoia_mode
+

Set paranoia recovery mode flags. It accepts one of the following values: +

+
+
‘disable’
+
‘verify’
+
‘overlap’
+
‘neverskip’
+
‘full’
+
+ +

Default value is ‘disable’. +

+

For more information about the available recovery modes, consult the +paranoia project documentation. +

+
+ +

3.13 libdc1394

+ +

IIDC1394 input device, based on libdc1394 and libraw1394. +

+

Requires the configure option --enable-libdc1394. +

+

3.13.1 Options

+
+
framerate
+

Set the frame rate. Default is ntsc, corresponding to a frame +rate of 30000/1001. +

+
+
pixel_format
+

Select the pixel format. Default is uyvy422. +

+
+
video_size
+

Set the video size given as a string such as 640x480 or hd720. +Default is qvga. +

+
+ +

3.14 openal

+ +

The OpenAL input device provides audio capture on all systems with a +working OpenAL 1.1 implementation. +

+

To enable this input device during configuration, you need OpenAL +headers and libraries installed on your system, and need to configure +FFmpeg with --enable-openal. +

+

OpenAL headers and libraries should be provided as part of your OpenAL +implementation, or as an additional download (an SDK). Depending on your +installation you may need to specify additional flags via the +--extra-cflags and --extra-ldflags for allowing the build +system to locate the OpenAL headers and libraries. +

+

An incomplete list of OpenAL implementations follows: +

+
+
Creative
+

The official Windows implementation, providing hardware acceleration +with supported devices and software fallback. +See http://openal.org/. +

+
OpenAL Soft
+

Portable, open source (LGPL) software implementation. Includes +backends for the most common sound APIs on the Windows, Linux, +Solaris, and BSD operating systems. +See http://kcat.strangesoft.net/openal.html. +

+
Apple
+

OpenAL is part of Core Audio, the official Mac OS X Audio interface. +See http://developer.apple.com/technologies/mac/audio-and-video.html +

+
+ +

This device allows one to capture from an audio input device handled +through OpenAL. +

+

You need to specify the name of the device to capture in the provided +filename. If the empty string is provided, the device will +automatically select the default device. You can get the list of the +supported devices by using the option list_devices. +

+

3.14.1 Options

+ +
+
channels
+

Set the number of channels in the captured audio. Only the values +1 (monaural) and 2 (stereo) are currently supported. +Defaults to 2. +

+
+
sample_size
+

Set the sample size (in bits) of the captured audio. Only the values +8 and 16 are currently supported. Defaults to +16. +

+
+
sample_rate
+

Set the sample rate (in Hz) of the captured audio. +Defaults to 44.1k. +

+
+
list_devices
+

If set to true, print a list of devices and exit. +Defaults to false. +

+
+
+ +

3.14.2 Examples

+ +

Print the list of OpenAL supported devices and exit: +

+
$ ffmpeg -list_devices true -f openal -i dummy out.ogg
+
+ +

Capture from the OpenAL device DR-BT101 via PulseAudio: +

+
$ ffmpeg -f openal -i 'DR-BT101 via PulseAudio' out.ogg
+
+ +

Capture from the default device (note the empty string ” as filename): +

+
$ ffmpeg -f openal -i '' out.ogg
+
+ +

Capture from two devices simultaneously, writing to two different files, +within the same ffmpeg command: +

+
$ ffmpeg -f openal -i 'DR-BT101 via PulseAudio' out1.ogg -f openal -i 'ALSA Default' out2.ogg
+
+

Note: not all OpenAL implementations support multiple simultaneous capture - +try the latest OpenAL Soft if the above does not work. +

+

3.15 oss

+ +

Open Sound System input device. +

+

The filename to provide to the input device is the device node +representing the OSS input device, and is usually set to +/dev/dsp. +

+

For example to grab from /dev/dsp using ffmpeg use the +command: +

+
ffmpeg -f oss -i /dev/dsp /tmp/oss.wav
+
+ +

For more information about OSS see: +http://manuals.opensound.com/usersguide/dsp.html +

+

3.15.1 Options

+ +
+
sample_rate
+

Set the sample rate in Hz. Default is 48000. +

+
+
channels
+

Set the number of channels. Default is 2. +

+
+
+ +

3.16 pulse

+ +

PulseAudio input device. +

+

To enable this output device you need to configure FFmpeg with --enable-libpulse. +

+

The filename to provide to the input device is a source device or the +string "default" +

+

To list the PulseAudio source devices and their properties you can invoke +the command pactl list sources. +

+

More information about PulseAudio can be found on http://www.pulseaudio.org. +

+

3.16.1 Options

+
+
server
+

Connect to a specific PulseAudio server, specified by an IP address. +Default server is used when not provided. +

+
+
name
+

Specify the application name PulseAudio will use when showing active clients, +by default it is the LIBAVFORMAT_IDENT string. +

+
+
stream_name
+

Specify the stream name PulseAudio will use when showing active streams, +by default it is "record". +

+
+
sample_rate
+

Specify the samplerate in Hz, by default 48kHz is used. +

+
+
channels
+

Specify the channels in use, by default 2 (stereo) is set. +

+
+
frame_size
+

This option does nothing and is deprecated. +

+
+
fragment_size
+

Specify the size in bytes of the minimal buffering fragment in PulseAudio, it +will affect the audio latency. By default it is set to 50 ms amount of data. +

+
+
wallclock
+

Set the initial PTS using the current time. Default is 1. +

+
+
+ +

3.16.2 Examples

+

Record a stream from default device: +

+
ffmpeg -f pulse -i default /tmp/pulse.wav
+
+ +

3.17 sndio

+ +

sndio input device. +

+

To enable this input device during configuration you need libsndio +installed on your system. +

+

The filename to provide to the input device is the device node +representing the sndio input device, and is usually set to +/dev/audio0. +

+

For example to grab from /dev/audio0 using ffmpeg use the +command: +

+
ffmpeg -f sndio -i /dev/audio0 /tmp/oss.wav
+
+ +

3.17.1 Options

+ +
+
sample_rate
+

Set the sample rate in Hz. Default is 48000. +

+
+
channels
+

Set the number of channels. Default is 2. +

+
+
+ +

3.18 video4linux2, v4l2

+ +

Video4Linux2 input video device. +

+

"v4l2" can be used as alias for "video4linux2". +

+

If FFmpeg is built with v4l-utils support (by using the +--enable-libv4l2 configure option), it is possible to use it with the +-use_libv4l2 input device option. +

+

The name of the device to grab is a file device node, usually Linux +systems tend to automatically create such nodes when the device +(e.g. an USB webcam) is plugged into the system, and has a name of the +kind /dev/videoN, where N is a number associated to +the device. +

+

Video4Linux2 devices usually support a limited set of +widthxheight sizes and frame rates. You can check which are +supported using -list_formats all for Video4Linux2 devices. +Some devices, like TV cards, support one or more standards. It is possible +to list all the supported standards using -list_standards all. +

+

The time base for the timestamps is 1 microsecond. Depending on the kernel +version and configuration, the timestamps may be derived from the real time +clock (origin at the Unix Epoch) or the monotonic clock (origin usually at +boot time, unaffected by NTP or manual changes to the clock). The +-timestamps abs or -ts abs option can be used to force +conversion into the real time clock. +

+

Some usage examples of the video4linux2 device with ffmpeg +and ffplay: +

    +
  • List supported formats for a video4linux2 device: +
    +
    ffplay -f video4linux2 -list_formats all /dev/video0
    +
    + +
  • Grab and show the input of a video4linux2 device: +
    +
    ffplay -f video4linux2 -framerate 30 -video_size hd720 /dev/video0
    +
    + +
  • Grab and record the input of a video4linux2 device, leave the +frame rate and size as previously set: +
    +
    ffmpeg -f video4linux2 -input_format mjpeg -i /dev/video0 out.mpeg
    +
    +
+ +

For more information about Video4Linux, check http://linuxtv.org/. +

+

3.18.1 Options

+ +
+
standard
+

Set the standard. Must be the name of a supported standard. To get a +list of the supported standards, use the list_standards +option. +

+
+
channel
+

Set the input channel number. Default to -1, which means using the +previously selected channel. +

+
+
video_size
+

Set the video frame size. The argument must be a string in the form +WIDTHxHEIGHT or a valid size abbreviation. +

+
+
pixel_format
+

Select the pixel format (only valid for raw video input). +

+
+
input_format
+

Set the preferred pixel format (for raw video) or a codec name. +This option allows one to select the input format, when several are +available. +

+
+
framerate
+

Set the preferred video frame rate. +

+
+
list_formats
+

List available formats (supported pixel formats, codecs, and frame +sizes) and exit. +

+

Available values are: +

+
‘all’
+

Show all available (compressed and non-compressed) formats. +

+
+
‘raw’
+

Show only raw video (non-compressed) formats. +

+
+
‘compressed’
+

Show only compressed formats. +

+
+ +
+
list_standards
+

List supported standards and exit. +

+

Available values are: +

+
‘all’
+

Show all supported standards. +

+
+ +
+
timestamps, ts
+

Set type of timestamps for grabbed frames. +

+

Available values are: +

+
‘default’
+

Use timestamps from the kernel. +

+
+
‘abs’
+

Use absolute timestamps (wall clock). +

+
+
‘mono2abs’
+

Force conversion from monotonic to absolute timestamps. +

+
+ +

Default value is default. +

+
+
use_libv4l2
+

Use libv4l2 (v4l-utils) conversion functions. Default is 0. +

+
+
+ +

3.19 vfwcap

+ +

VfW (Video for Windows) capture input device. +

+

The filename passed as input is the capture driver number, ranging from +0 to 9. You may use "list" as filename to print a list of drivers. Any +other filename will be interpreted as device number 0. +

+

3.19.1 Options

+ +
+
video_size
+

Set the video frame size. +

+
+
framerate
+

Set the grabbing frame rate. Default value is ntsc, +corresponding to a frame rate of 30000/1001. +

+
+
+ +

3.20 x11grab

+ +

X11 video input device. +

+

To enable this input device during configuration you need libxcb +installed on your system. It will be automatically detected during +configuration. +

+

This device allows one to capture a region of an X11 display. +

+

The filename passed as input has the syntax: +

+
[hostname]:display_number.screen_number[+x_offset,y_offset]
+
+ +

hostname:display_number.screen_number specifies the +X11 display name of the screen to grab from. hostname can be +omitted, and defaults to "localhost". The environment variable +DISPLAY contains the default display name. +

+

x_offset and y_offset specify the offsets of the grabbed +area with respect to the top-left border of the X11 screen. They +default to 0. +

+

Check the X11 documentation (e.g. man X) for more detailed +information. +

+

Use the xdpyinfo program for getting basic information about +the properties of your X11 display (e.g. grep for "name" or +"dimensions"). +

+

For example to grab from :0.0 using ffmpeg: +

+
ffmpeg -f x11grab -framerate 25 -video_size cif -i :0.0 out.mpg
+
+ +

Grab at position 10,20: +

+
ffmpeg -f x11grab -framerate 25 -video_size cif -i :0.0+10,20 out.mpg
+
+ +

3.20.1 Options

+ +
+
select_region
+

Specify whether to select the grabbing area graphically using the pointer. +A value of 1 prompts the user to select the grabbing area graphically +by clicking and dragging. A single click with no dragging will select the +whole screen. A region with zero width or height will also select the whole +screen. This option overwrites the video_size, grab_x, and +grab_y options. Default value is 0. +

+
+
draw_mouse
+

Specify whether to draw the mouse pointer. A value of 0 specifies +not to draw the pointer. Default value is 1. +

+
+
follow_mouse
+

Make the grabbed area follow the mouse. The argument can be +centered or a number of pixels PIXELS. +

+

When it is specified with "centered", the grabbing region follows the mouse +pointer and keeps the pointer at the center of region; otherwise, the region +follows only when the mouse pointer reaches within PIXELS (greater than +zero) to the edge of region. +

+

For example: +

+
ffmpeg -f x11grab -follow_mouse centered -framerate 25 -video_size cif -i :0.0 out.mpg
+
+ +

To follow only when the mouse pointer reaches within 100 pixels to edge: +

+
ffmpeg -f x11grab -follow_mouse 100 -framerate 25 -video_size cif -i :0.0 out.mpg
+
+ +
+
framerate
+

Set the grabbing frame rate. Default value is ntsc, +corresponding to a frame rate of 30000/1001. +

+
+
show_region
+

Show grabbed region on screen. +

+

If show_region is specified with 1, then the grabbing +region will be indicated on screen. With this option, it is easy to +know what is being grabbed if only a portion of the screen is grabbed. +

+
+
region_border
+

Set the region border thickness if -show_region 1 is used. +Range is 1 to 128 and default is 3 (XCB-based x11grab only). +

+

For example: +

+
ffmpeg -f x11grab -show_region 1 -framerate 25 -video_size cif -i :0.0+10,20 out.mpg
+
+ +

With follow_mouse: +

+
ffmpeg -f x11grab -follow_mouse centered -show_region 1 -framerate 25 -video_size cif -i :0.0 out.mpg
+
+ +
+
window_id
+

Grab this window, instead of the whole screen. Default value is 0, which maps to +the whole screen (root window). +

+

The id of a window can be found using the xwininfo program, possibly with options -tree and +-root. +

+

If the window is later enlarged, the new area is not recorded. Video ends when +the window is closed, unmapped (i.e., iconified) or shrunk beyond the video +size (which defaults to the initial window size). +

+

This option disables options follow_mouse and select_region. +

+
+
video_size
+

Set the video frame size. Default is the full desktop or window. +

+
+
grab_x
+
grab_y
+

Set the grabbing region coordinates. They are expressed as offset from +the top left corner of the X11 window and correspond to the +x_offset and y_offset parameters in the device name. The +default value for both options is 0. +

+
+ +

4 Output Devices

+ +

Output devices are configured elements in FFmpeg that can write +multimedia data to an output device attached to your system. +

+

When you configure your FFmpeg build, all the supported output devices +are enabled by default. You can list all available ones using the +configure option "–list-outdevs". +

+

You can disable all the output devices using the configure option +"–disable-outdevs", and selectively enable an output device using the +option "–enable-outdev=OUTDEV", or you can disable a particular +input device using the option "–disable-outdev=OUTDEV". +

+

The option "-devices" of the ff* tools will display the list of +enabled output devices. +

+

A description of the currently available output devices follows. +

+

4.1 alsa

+ +

ALSA (Advanced Linux Sound Architecture) output device. +

+

4.1.1 Examples

+ +
    +
  • Play a file on default ALSA device: +
    +
    ffmpeg -i INPUT -f alsa default
    +
    + +
  • Play a file on soundcard 1, audio device 7: +
    +
    ffmpeg -i INPUT -f alsa hw:1,7
    +
    +
+ +

4.2 AudioToolbox

+ +

AudioToolbox output device. +

+

Allows native output to CoreAudio devices on OSX. +

+

The output filename can be empty (or -) to refer to the default system output device or a number that refers to the device index as shown using: -list_devices true. +

+

Alternatively, the audio input device can be chosen by index using the + + -audio_device_index <INDEX> + +, overriding any device name or index given in the input filename. +

+

All available devices can be enumerated by using -list_devices true, listing +all device names, UIDs and corresponding indices. +

+

4.2.1 Options

+ +

AudioToolbox supports the following options: +

+
+
-audio_device_index <INDEX>
+

Specify the audio device by its index. Overrides anything given in the output filename. +

+
+
+ +

4.2.2 Examples

+ +
    +
  • Print the list of supported devices and output a sine wave to the default device: +
    +
    $ ffmpeg -f lavfi -i sine=r=44100 -f audiotoolbox -list_devices true -
    +
    + +
  • Output a sine wave to the device with the index 2, overriding any output filename: +
    +
    $ ffmpeg -f lavfi -i sine=r=44100 -f audiotoolbox -audio_device_index 2 -
    +
    + +
+ +

4.3 caca

+ +

CACA output device. +

+

This output device allows one to show a video stream in CACA window. +Only one CACA window is allowed per application, so you can +have only one instance of this output device in an application. +

+

To enable this output device you need to configure FFmpeg with +--enable-libcaca. +libcaca is a graphics library that outputs text instead of pixels. +

+

For more information about libcaca, check: +http://caca.zoy.org/wiki/libcaca +

+

4.3.1 Options

+ +
+
window_title
+

Set the CACA window title, if not specified default to the filename +specified for the output device. +

+
+
window_size
+

Set the CACA window size, can be a string of the form +widthxheight or a video size abbreviation. +If not specified it defaults to the size of the input video. +

+
+
driver
+

Set display driver. +

+
+
algorithm
+

Set dithering algorithm. Dithering is necessary +because the picture being rendered has usually far more colours than +the available palette. +The accepted values are listed with -list_dither algorithms. +

+
+
antialias
+

Set antialias method. Antialiasing smoothens the rendered +image and avoids the commonly seen staircase effect. +The accepted values are listed with -list_dither antialiases. +

+
+
charset
+

Set which characters are going to be used when rendering text. +The accepted values are listed with -list_dither charsets. +

+
+
color
+

Set color to be used when rendering text. +The accepted values are listed with -list_dither colors. +

+
+
list_drivers
+

If set to true, print a list of available drivers and exit. +

+
+
list_dither
+

List available dither options related to the argument. +The argument must be one of algorithms, antialiases, +charsets, colors. +

+
+ +

4.3.2 Examples

+ +
    +
  • The following command shows the ffmpeg output is an +CACA window, forcing its size to 80x25: +
    +
    ffmpeg -i INPUT -c:v rawvideo -pix_fmt rgb24 -window_size 80x25 -f caca -
    +
    + +
  • Show the list of available drivers and exit: +
    +
    ffmpeg -i INPUT -pix_fmt rgb24 -f caca -list_drivers true -
    +
    + +
  • Show the list of available dither colors and exit: +
    +
    ffmpeg -i INPUT -pix_fmt rgb24 -f caca -list_dither colors -
    +
    +
+ +

4.4 decklink

+ +

The decklink output device provides playback capabilities for Blackmagic +DeckLink devices. +

+

To enable this output device, you need the Blackmagic DeckLink SDK and you +need to configure with the appropriate --extra-cflags +and --extra-ldflags. +On Windows, you need to run the IDL files through widl. +

+

DeckLink is very picky about the formats it supports. Pixel format is always +uyvy422, framerate, field order and video size must be determined for your +device with -list_formats 1. Audio sample rate is always 48 kHz. +

+

4.4.1 Options

+ +
+
list_devices
+

If set to true, print a list of devices and exit. +Defaults to false. This option is deprecated, please use the +-sinks option of ffmpeg to list the available output devices. +

+
+
list_formats
+

If set to true, print a list of supported formats and exit. +Defaults to false. +

+
+
preroll
+

Amount of time to preroll video in seconds. +Defaults to 0.5. +

+
+
duplex_mode
+

Sets the decklink device duplex/profile mode. Must be ‘unset’, ‘half’, ‘full’, +‘one_sub_device_full’, ‘one_sub_device_half’, ‘two_sub_device_full’, +‘four_sub_device_half’ +Defaults to ‘unset’. +

+

Note: DeckLink SDK 11.0 have replaced the duplex property by a profile property. +For the DeckLink Duo 2 and DeckLink Quad 2, a profile is shared between any 2 +sub-devices that utilize the same connectors. For the DeckLink 8K Pro, a profile +is shared between all 4 sub-devices. So DeckLink 8K Pro support four profiles. +

+

Valid profile modes for DeckLink 8K Pro(with DeckLink SDK >= 11.0): +‘one_sub_device_full’, ‘one_sub_device_half’, ‘two_sub_device_full’, +‘four_sub_device_half’ +

+

Valid profile modes for DeckLink Quad 2 and DeckLink Duo 2: +‘half’, ‘full’ +

+
+
timing_offset
+

Sets the genlock timing pixel offset on the used output. +Defaults to ‘unset’. +

+
+
link
+

Sets the SDI video link configuration on the used output. Must be +‘unset’, ‘single’ link SDI, ‘dual’ link SDI or ‘quad’ link +SDI. +Defaults to ‘unset’. +

+
+
sqd
+

Enable Square Division Quad Split mode for Quad-link SDI output. +Must be ‘unset’, ‘true’ or ‘false’. +Defaults to unset. +

+
+
level_a
+

Enable SMPTE Level A mode on the used output. +Must be ‘unset’, ‘true’ or ‘false’. +Defaults to unset. +

+
+
vanc_queue_size
+

Sets maximum output buffer size in bytes for VANC data. If the buffering reaches this value, +outgoing VANC data will be dropped. +Defaults to ‘1048576’. +

+
+
+ +

4.4.2 Examples

+ +
    +
  • List output devices: +
    +
    ffmpeg -sinks decklink
    +
    + +
  • List supported formats: +
    +
    ffmpeg -i test.avi -f decklink -list_formats 1 'DeckLink Mini Monitor'
    +
    + +
  • Play video clip: +
    +
    ffmpeg -i test.avi -f decklink -pix_fmt uyvy422 'DeckLink Mini Monitor'
    +
    + +
  • Play video clip with non-standard framerate or video size: +
    +
    ffmpeg -i test.avi -f decklink -pix_fmt uyvy422 -s 720x486 -r 24000/1001 'DeckLink Mini Monitor'
    +
    + +
+ +

4.5 fbdev

+ +

Linux framebuffer output device. +

+

The Linux framebuffer is a graphic hardware-independent abstraction +layer to show graphics on a computer monitor, typically on the +console. It is accessed through a file device node, usually +/dev/fb0. +

+

For more detailed information read the file +Documentation/fb/framebuffer.txt included in the Linux source tree. +

+

4.5.1 Options

+
+
xoffset
+
yoffset
+

Set x/y coordinate of top left corner. Default is 0. +

+
+ +

4.5.2 Examples

+

Play a file on framebuffer device /dev/fb0. +Required pixel format depends on current framebuffer settings. +

+
ffmpeg -re -i INPUT -c:v rawvideo -pix_fmt bgra -f fbdev /dev/fb0
+
+ +

See also http://linux-fbdev.sourceforge.net/, and fbset(1). +

+

4.6 oss

+ +

OSS (Open Sound System) output device. +

+

4.7 pulse

+ +

PulseAudio output device. +

+

To enable this output device you need to configure FFmpeg with --enable-libpulse. +

+

More information about PulseAudio can be found on http://www.pulseaudio.org +

+

4.7.1 Options

+
+
server
+

Connect to a specific PulseAudio server, specified by an IP address. +Default server is used when not provided. +

+
+
name
+

Specify the application name PulseAudio will use when showing active clients, +by default it is the LIBAVFORMAT_IDENT string. +

+
+
stream_name
+

Specify the stream name PulseAudio will use when showing active streams, +by default it is set to the specified output name. +

+
+
device
+

Specify the device to use. Default device is used when not provided. +List of output devices can be obtained with command pactl list sinks. +

+
+
buffer_size
+
buffer_duration
+

Control the size and duration of the PulseAudio buffer. A small buffer +gives more control, but requires more frequent updates. +

+

buffer_size specifies size in bytes while +buffer_duration specifies duration in milliseconds. +

+

When both options are provided then the highest value is used +(duration is recalculated to bytes using stream parameters). If they +are set to 0 (which is default), the device will use the default +PulseAudio duration value. By default PulseAudio set buffer duration +to around 2 seconds. +

+
+
prebuf
+

Specify pre-buffering size in bytes. The server does not start with +playback before at least prebuf bytes are available in the +buffer. By default this option is initialized to the same value as +buffer_size or buffer_duration (whichever is bigger). +

+
+
minreq
+

Specify minimum request size in bytes. The server does not request less +than minreq bytes from the client, instead waits until the buffer +is free enough to request more bytes at once. It is recommended to not set +this option, which will initialize this to a value that is deemed sensible +by the server. +

+
+
+ +

4.7.2 Examples

+

Play a file on default device on default server: +

+
ffmpeg  -i INPUT -f pulse "stream name"
+
+ +

4.8 sndio

+ +

sndio audio output device. +

+

4.9 v4l2

+ +

Video4Linux2 output device. +

+

4.10 xv

+ +

XV (XVideo) output device. +

+

This output device allows one to show a video stream in a X Window System +window. +

+

4.10.1 Options

+ +
+
display_name
+

Specify the hardware display name, which determines the display and +communications domain to be used. +

+

The display name or DISPLAY environment variable can be a string in +the format hostname[:number[.screen_number]]. +

+

hostname specifies the name of the host machine on which the +display is physically attached. number specifies the number of +the display server on that host machine. screen_number specifies +the screen to be used on that server. +

+

If unspecified, it defaults to the value of the DISPLAY environment +variable. +

+

For example, dual-headed:0.1 would specify screen 1 of display +0 on the machine named “dual-headed”. +

+

Check the X11 specification for more detailed information about the +display name format. +

+
+
window_id
+

When set to non-zero value then device doesn’t create new window, +but uses existing one with provided window_id. By default +this options is set to zero and device creates its own window. +

+
+
window_size
+

Set the created window size, can be a string of the form +widthxheight or a video size abbreviation. If not +specified it defaults to the size of the input video. +Ignored when window_id is set. +

+
+
window_x
+
window_y
+

Set the X and Y window offsets for the created window. They are both +set to 0 by default. The values may be ignored by the window manager. +Ignored when window_id is set. +

+
+
window_title
+

Set the window title, if not specified default to the filename +specified for the output device. Ignored when window_id is set. +

+
+ +

For more information about XVideo see http://www.x.org/. +

+

4.10.2 Examples

+ +
    +
  • Decode, display and encode video input with ffmpeg at the +same time: +
    +
    ffmpeg -i INPUT OUTPUT -f xv display
    +
    + +
  • Decode and display the input video to multiple X11 windows: +
    +
    ffmpeg -i INPUT -f xv normal -vf negate -f xv negated
    +
    +
+ + +

5 See Also

+ +

ffmpeg, ffplay, ffprobe, +libavdevice +

+ +

6 Authors

+ +

The FFmpeg developers. +

+

For details about the authorship, see the Git history of the project +(https://git.ffmpeg.org/ffmpeg), e.g. by typing the command +git log in the FFmpeg source directory, or browsing the +online repository at https://git.ffmpeg.org/ffmpeg. +

+

Maintainers for the specific components are listed in the file +MAINTAINERS in the source code tree. +

+ + +

+ This document was generated on May 17, 2026 using makeinfo. +

+ +

Hosting provided by telepoint.bg

+
+
+
+ + + + + + + + + \ No newline at end of file diff --git a/research/ambisonic-aquasynth/mirrors/icst-ambisonics-formats.html b/research/ambisonic-aquasynth/mirrors/icst-ambisonics-formats.html new file mode 100644 index 0000000..52f335c --- /dev/null +++ b/research/ambisonic-aquasynth/mirrors/icst-ambisonics-formats.html @@ -0,0 +1,17 @@ +Ambisonics Formats Explained: A-Format, B-Format, FuMa, ambiX, FOA and HOA | ICST Ambisonics

Ambisonics Formats Explained: A-Format, B-Format, FuMa, AmbiX, FOA and HOA

Level: Beginner to Intermediate | Audience: Composers, students, producers, technicians, studio residents.

Use this page when you see terms such as A-Format, B-Format, FuMa, ambiX, ACN/SN3D, FOA, or HOA and need to know what they mean in practice.

1. Why “format” is confusing in Ambisonics

Ambisonics uses the word format for several different things:

  • the raw output of a microphone
  • the internal spatial signal in a DAW
  • the channel order inside a file
  • the normalization convention used by the spherical harmonics
  • the Ambisonics order, which determines the channel count

That is why “Is this B-Format?” is often not specific enough. A better practical question is:

Is this an ambiX B-Format file, with ACN channel ordering and SN3D normalization, and which Ambisonics order does it use?

2. A-Format vs B-Format

A-Format is usually the raw signal from an Ambisonic microphone. In a first-order tetrahedral microphone, this means four capsule signals before spatial decoding or correction. A-Format is microphone-specific and normally should not be treated as a finished Ambisonics master.

B-Format is the Ambisonics signal representation used for production, exchange, decoding, and archiving. It describes a sound field around a listening point. First-order B-Format has four channels; higher-order B-Format has more.

Practical rule:

  • A-Format comes from the microphone and must be converted.
  • B-Format is the spatial scene you can edit, route, decode, archive, or render.

3. FuMa vs ambiX

FuMa is the older Ambisonics convention. It uses named first-order channels such as W, X, Y, Z and MaxN-style normalization. It is still encountered in older tools, historical material, and some first-order workflows.

ambiX is the modern exchange convention for Ambisonics. It uses:

  • ACN channel ordering
  • SN3D normalization
  • ordinary multichannel audio files, usually WAV or RF64

In current ICST workflows, ambiX is the default. If a tool asks for channel ordering and normalization, choose ACN/SN3D unless you have a clear reason to do otherwise.

4. ACN, SN3D and N3D

ACN means Ambisonic Channel Number. It defines the channel order. Instead of using historical names such as W, X, Y, Z, each spherical harmonic component gets a numbered channel.

SN3D and N3D are normalization conventions. They define the scaling of Ambisonics components. The spatial scene may be conceptually the same, but a mismatch between SN3D and N3D changes levels between channels and can make decoding wrong.

For most plugin and DAW work today:

  • ACN/SN3D = ambiX, recommended for exchange and production
  • ACN/N3D = common in some research, analysis, and mathematical contexts
  • FuMa = legacy convention, mainly first-order

5. FOA vs HOA

FOA means First-Order Ambisonics. It uses 4 channels and is useful for basic recording, simple spatial work, compatibility, and many entry-level microphone workflows.

HOA means Higher-Order Ambisonics. It starts at second order and increases the spatial resolution by adding more spherical harmonic components.

Higher order can give sharper localization, more stable rendering, and a larger useful listening area, but it also needs more channels and more careful routing.

6. Channel counts by order

For full 3D Ambisonics, the channel count is:

channels = (order + 1)^2
+
OrderNameChannels
1FOA4
2HOA-29
3HOA-316
4HOA-425
5HOA-536
6HOA-649
7HOA-764

In REAPER, this is why ICST sessions often use a 64-channel B-Format bus: it can carry HOA-7 without silently cutting off higher-order channels.

7. Scene-based vs channel-based audio

Ambisonics is scene-based. The channels are not speaker feeds. They are components of a spatial sound-field representation.

This is different from channel-based audio, where channel 1 might mean left speaker, channel 2 right speaker, channel 3 center speaker, and so on.

In Ambisonics, the decoder decides how the B-Format scene becomes loudspeaker signals or binaural headphone output. That is the reason the same B-Format master can be rendered for a dome, an octagon, a studio array, headphones, or stereo.

8. Practical ICST recommendation

ICST Recommendation

For new Ambisonics work, use ambiX: +ACN channel ordering + SN3D normalization.

For higher-order work in REAPER, use a 64-channel B-Format bus for HOA-7 when needed. Decode only at the monitoring or rendering stage.

This keeps the session open: the same B-Format master can feed loudspeaker decoding, binaural monitoring, archive export, and later reinterpretation on another playback system.

9. Quick checklist: Which format should I use?

Use ambiX B-Format, ACN/SN3D when:

  • you are starting a new Ambisonics production
  • you are working with ICST, IEM, SPARTA, or other modern HOA tools
  • you are exporting a B-Format archive master
  • you need reliable exchange between DAWs, plugins, and rendering tools

Use A-Format only when:

  • you are handling raw Ambisonic microphone recordings before conversion
  • you still need the manufacturer-specific A-to-B conversion step

Use FuMa only when:

  • an older tool or archive specifically requires it
  • you are converting historical first-order material

Always document:

  • Ambisonics order
  • channel count
  • channel ordering
  • normalization
  • sample rate and bit depth
  • decoder or renderer used for monitoring
\ No newline at end of file diff --git a/research/ambisonic-aquasynth/mirrors/iem-compatibility.html b/research/ambisonic-aquasynth/mirrors/iem-compatibility.html new file mode 100644 index 0000000..128e642 --- /dev/null +++ b/research/ambisonic-aquasynth/mirrors/iem-compatibility.html @@ -0,0 +1,188 @@ + + + + + + + + + Compatibility with other Suites + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ +
+
+ + + + +
+
+
+
+
+

Compatibility with other Suites

+

+ This guide tells you information about the used Ambisonic format and the compatibility with other plug-ins suites. +

+ +
+
+ + + + + + +

The short facts

+

The IEM Plug-in Suite uses the AmbiX convention, that is ACN channel ordering and SN3D normalization. That is the current standard, which most plug-in suites use. Some older suites use the Furse-Malham (FuMa) format, in order to use them with AmbiX, you’ll need to convert their signals to AmbiX and vice versa.

+

Here’s a short and non-complete list of plug-in suites which also use the AmbiX convetion and can be used directly with the IEM Plug-in Suite:

+
    +
  • Matthias Kronlachner’s ambix plugin suite
  • +
  • SPARTA / Compass plug-in suites
  • +
  • SSA-Plugins
  • +
  • BlueRippleSound plug-ins
  • +
  • …
  • +
+

AmbiX format

+

The AmbiX format is the current standard for Ambisonics. It handles two important things: the channel ordering and the normalization. Both are related to the spherical harmonics, the math behind Ambisonics. Let’s take a look!

+
Channel ordering: ACN
+

The channel ordering used in the Ambix Format is called ACN which stands for Ambisonic Channel Order. What a greate name! The difference to the FuMa odering is that the channels aren’t ordered alphabetically any more. Here the first oder components for both FuMa and ACN: +FuMa: WXYZ +ACN: WYZX

+

(W is the omni-directional signal; X, Y, and Z the figure-of-eights in x, y, and z direction)

+

So why that non-sense? Well, let’s say it’s math… However, in combination with the SN3D normalization it has a neat property, which will we come to in a minute (or second for the fast readers)!

+
SN3D Normalization
+

Semi-normalization is used in the AmbiX format. This one makes sure that when you encode a source, the levels of all channels won’t exceed the one in the first (omni, W) channel. In general, this is quite handy, so that clipping doesn’t occur in your DAW.

+

Note: in a mix, the levels can exceed the one of the omni-channel!

+

N3D normalization on the other side will boost the signals in each order, which will lead to very high levels let’s say in the 7th order components in channels 50-64. So why would N3D be any good? Well, the math only works in N3D. For a matter of fact, things like beamforming or decoding (which is the same actually), will happen in N3D. But you as a user don’t have to take care of that, the plug-ins will internally apply a conversion if necessary, no worries :-)

+
Encoding a signal using the IEM Plug-in Suite
+

Per default, signals will be encoded with SN3D and ACN. You can set the normalization to N3D in most plug-ins, but you shouldn’t, as then the signals won’t be AmbiX compatible any more. Please do that only, if you know what you are doing :-)

+

When using spherical harmonics, the coefficient (factor/weight) for the omni-channel is usualy 1/sqrt(4*pi) which is something like 0.282094792. In all encoders, the encoding coefficients will be adjusted so that the omni channel will have unity gain.

+

On the other side, when decoding, e.g. with the **Probe**Decoder the weights are adjusted, so that encoding into one direction and decoding with the same direction will lead to unity again.

+
The neat property!
+

Using SN3D and ACN, in other words AmbiX, the first two channels of your Ambisonic signals will hold mid/side signals, which you can simply decode with a mid/side decoder, like the one which comes with Reaper. So a very very simple StereoDecoder would be exactlyt that. Neat, right?

+ + + +
+
+
+ + + + + + + + + diff --git a/research/ambisonic-aquasynth/mirrors/iem-plugin-suite.html b/research/ambisonic-aquasynth/mirrors/iem-plugin-suite.html new file mode 100644 index 0000000..1a2a7c1 --- /dev/null +++ b/research/ambisonic-aquasynth/mirrors/iem-plugin-suite.html @@ -0,0 +1,258 @@ + + + + + + + + + IEM Plug-in Suite + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ +
+
+ + + + +
+
+
+ +
+
+ IEM Plug-in Suite +

IEM Plug-in Suite

+

The IEM Plug-in Suite is a free and Open-Source audio plugin suite including Ambisonic plug-ins up to 7th order created by staff and students of the
Institute of Electronic Music and Acoustics.

+
+ +
+ Download + Release Page + Source Code +

Latest Version v1.15.0, 10th April 2025

+
+ +
+
NOTE: Breaking changes!
Version v1.15.0 and v1.3.0 have introduced some breaking changes! Projects using one of the previous versions might not work as expected. For more information see the Breaking changes article and/or the ChangeLog. +
+ +
+ + +
+
+
+
+
DualDelay
+

The DualDelay has been one of the first plug-ins in the IEM Suite. Therefore it was time for a big makeover! It has been optimized to suite needs of composers and musicians by adding a sync-to-DAW function, a tap delay button as well as moving from milliseconds to BPM with multiplies for setting times. As cherry on top, it doesn't only support a rotation of the signal, but also warping!

+
+
+
+
+
+
+
+
GranularEncoder
+

The GranularEncoder is an Ambisonic granular synthesis plug-in. That's right, it is the first Ambisonic synthesizer plug-in released in the suite! The best thing about it: you can use any mono or stereo audio signal, and turn it into an enveloping sound scene, or crush the signal into microscopic sonic particles!

+
+
+
+
+
+
+
+
MultiBandCompressor
+

The MultiBandCompressor is a very handy tool in your project's mastering stage! It works like a stereo multi-band compressor, however, it will make sure the spatial image is not altered! The Ambisonic input signal is split up into four bands, which can be compressed individually.

+
+
+
+
+
+
+
+
StereoEncoder
+

Use the StereoEncoder plug-in to encode mono or stereo audio signals into the Ambisonic domain - up to 7th order. The interface lets you easily set the orientation, width and rotation of the stereo pair. With a simple double-click onto the SpherePanner you can change the display to a linear elevation, which gives you a higher accuracy towards the horizon.

+

With help of the Quaternion input, you can use use data from orientation trackers (e.g. headtrackers) to steer your source into the right direction.

+
+
+
+
+
+
+
+
RoomEncoder
+

The RoomEncoder is so far the most complex and computation-intensive plug-in within the IEM plug-in suite. It allows you to put a source and a listener into a virtual shoebox-shaped room and render over 200 wall reflections. Perfect to enhance the externalization of binaural playback.

+

You can move the source and the listener freely within the room, which will automatically generate Doppler-shift effects. Use the overall reflection coefficient and the low-shelve and high-shelve filters to change the character of the reflections.

+

But thats not all: The RoomEncoder can make use of DirectivitySignals as an input signal. This means that your source can have a frequency-dependent and time-variable directivity. The DirectivityShaper plugin is a good tool to create various directivities.

+
+
+
+
+
+
+
+
AllRADecoder
+

With the AllRADecoder you can design an Ambisonic decoder for an any desired loudspeaker layout using the AllRAD approach.

+

The resulting decoder is automatically applied to the incoming Ambisonic audio stream, so it can be directly inserted into a decoder track of your audio project. You might also want to use it as a standalone application to design a decoder and write it into a JSON configuration file for the use in other plug-ins (e.g. SimpleDecoder).

+
+
+
+
+
+
+
+
EnergyVisualizer
+

This plug-in is perfect for metering the spatial image. The EnergyVisualizer visualizes the energy distribution on the sphere of the Ambisonic input signal. The energy levels are color-coded according to the color map. The Ambisonic Signal is decoded using 480 virtual loudspeakers with a subsequent RMS calculation and OpenGL visualization.

+
+
+
+
+ + + +
+
+
+
+ Higher Order Ambisonics +
+

One Plug-In - Seven Orders

+

Support for up to 7th order Ambisonics, selectable within each plug-in.

+ +
+
+
+ Open-Source +
+

Open-Source

+

Free to use, free to change. Contribute!

+
+
+
+ Cross-platform +
+

Cross-platform

+

Compiled Plug-ins available for macOS, Windows and Linux.

+
+
+
+
+

Made with JUCE

+

Created with the JUCE framework.

+
+
+

Created by staff and students

+

The plug-ins result from the institute's current research and student projects.

+
+
+

License

+

The IEM Plug-in Suite is licensed under the GPLv3 License.

+
+
+
+
+
+ + + + + + + + + diff --git a/research/ambisonic-aquasynth/mirrors/jack-api-overview.html b/research/ambisonic-aquasynth/mirrors/jack-api-overview.html new file mode 100644 index 0000000..d7a08c9 --- /dev/null +++ b/research/ambisonic-aquasynth/mirrors/jack-api-overview.html @@ -0,0 +1,176 @@ + + + + + + + +JACK-AUDIO-CONNECTION-KIT: JACK Audio Connection Kit + + + + + + + + + + +
+
+ + + + + + +
+
JACK-AUDIO-CONNECTION-KIT +
+
+
+ + + + +
+ +
+
+ + +
+ +
+ +
+
+
JACK Audio Connection Kit
+
+
+

+Introduction

+

JACK is a low-latency audio server, written for any operating system that is reasonably POSIX compliant. It currently exists for Linux, OS X, Solaris, FreeBSD and Windows. It can connect several client applications to an audio device, and allow them to share audio with each other. Clients can run as separate processes like normal applications, or within the JACK server as "plugins".

+

JACK was designed from the ground up for professional audio work, and its design focuses on two key areas: synchronous execution of all clients, and low latency operation.

+
See also
http://jackaudio.org
+

+JACK Overview

+

Traditionally it has been hard if not impossible to write audio applications that can share data with each other. In addition, configuring and managing audio interface hardware has often been one of the most complex aspect of writing audio software.

+

JACK changes all this by providing an API that does several things:

1. provides a high level abstraction for programmers that
+   removes the audio interface hardware from the picture and
+   allows them to concentrate on the core functionality of
+   their software.
+
+2. allows applications to send and receive audio data to/from
+   each other as well as the audio interface. There is no
+   difference in how an application sends or receives
+   data regardless of whether it comes from/goes to another application 
+   or an audio interface.
+

For programmers with experience of several other audio APIs such as PortAudio, Apple's CoreAudio, Steinberg's VST and ASIO as well as many others, JACK presents a familiar model: your program provides a "callback" function that will be executed at the right time. Your callback can send and receive data as well as do other signal processing tasks. You are not responsible for managing audio interfaces or threading, and there is no "format negotiation": all audio data within JACK is represented as 32 bit floating point values.

+

For those with experiences rooted in the Unix world, JACK presents a somewhat unfamiliar API. Most Unix APIs are based on the read/write model spawned by the "everything is a file" abstraction that Unix is rightly famous for. The problem with this design is that it fails to take the realtime nature of audio interfaces into account, or more precisely, it fails to force application developers to pay sufficient attention to this aspect of their task. In addition, it becomes rather difficult to facilitate inter-application audio routing when different programs are not all running synchronously.

+

Using JACK within your program is very simple, and typically consists of just:

+
    +
  • calling jack_client_open() to connect to the JACK server.
  • +
  • registering "ports" to enable data to be moved to and from your application.
  • +
  • registering a "process callback" which will be called at the right time by the JACK server.
  • +
  • telling JACK that your application is ready to start processing data.
  • +
+

There is a lot more that you can do with JACK's interfaces, but for many applications, this is all that is needed. The simple_client.c example demonstrates a complete (simple!) JACK application that just copies the signal arriving at its input port to its output port. Similarly, inprocess.c shows how to write an internal client "plugin" that runs within the JACK server process.

+

+Reference

+

The JACK programming interfaces are described in several header files. We present them here broken into useful categories to make the API a little clearer:

+ +

The full API is described in:

+
    +
  • <jack/jack.h> is the main JACK interface.
  • +
  • <jack/statistics.h> provides interfaces for monitoring the performance of a running JACK server.
  • +
  • <jack/intclient.h> allows loading and unloading JACK internal clients.
  • +
  • <jack/ringbuffer.h> defines a simple API for using lock-free ringbuffers. These are a good way to pass data between threads, when streaming realtime data to slower media, like audio file playback or recording.
  • +
  • <jack/transport.h> defines a simple transport control mechanism for starting, stopping and repositioning clients. This is described in the JACK Transport Design document.
  • +
  • <jack/types.h> defines the main JACK data types.
  • +
  • <jack/thread.h> functions standardize thread creation for JACK and its clients.
  • +
  • <jack/midiport.h> functions to handle reading and writing of MIDI data to a port
  • +
  • <jack/session.h> functions that form the JACK session API
  • +
  • <jack/control.h> the API for starting and controlling a JACK server
  • +
  • <jack/metadata.h> the API for managing metadata about objects within JACK (clients and ports)
  • +
+

In addition, the tools directory provides numerous examples of simple JACK clients that nevertheless use the API to do something useful. It includes

+
    +
  • a metronome.
  • +
  • a recording client that can capture any number of channels from any JACK sources and store them as an audio file.
  • +
  • command line clients to control the transport mechanism, change the buffer size and more.
    +
  • +
  • commands to load and unload JACK internal clients.
  • +
  • tools for checking the status of a running JACK system.
  • +
+

and many more.

+

+Porting

+

JACK is designed to be portable to any system supporting the relevant POSIX and ANSI C standards. It currently runs under GNU/Linux, Mac OS X and Berkeley Unix on several different processor architectures. If you want to port JACK to another platform, please read the Porting JACK document.

+

+License

+

Copyright (C) 2001-2019 by Paul Davis, Stephane Letz, Jack O'Quinn, Torben Hohn, Filipe Coelho and others.

+

JACK is free software; you can redistribute it and/or modify it under the terms of the GNU GPL and LGPL licenses as published by the Free Software Foundation, http://www.gnu.org. The JACK server uses the GPL, as noted in the source file headers. However, the JACK library is licensed under the LGPL, allowing proprietary programs to link with it and use JACK services. You should have received a copy of these Licenses along with the program; if not, write to the Free Software Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.

+

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

+
+
+ + + + diff --git a/research/ambisonic-aquasynth/mirrors/jack-on-windows.html b/research/ambisonic-aquasynth/mirrors/jack-on-windows.html new file mode 100644 index 0000000..5a212f2 --- /dev/null +++ b/research/ambisonic-aquasynth/mirrors/jack-on-windows.html @@ -0,0 +1,183 @@ + + + + + + + + Using JACK on Windows | JACK Audio Connection Kit + + + + + + + + + + + + +
+
+ View on GitHub + + JACK Logo - Home + + +
+
+ +
+
+

Using JACK on Windows

+ +

Introduction

+ +

As on other platforms, JACK on Windows is a multi-purpose audio server that +connects audio and MIDI streams using the Windows system drivers. +It connects (OUTPUT) to those audio drivers using the PortAudio project +(so JACK can connect using dsound, wmme, ASIO, etc) depending on what +the soundcard’s drivers support.

+ +

JACK provides an Audio, MIDI and transport API (Application Programming Interface) +that can be used natively by audio application developers. +There are already some existing native JACK Windows applications +(that is, audio applications that use the JACK API directly). +Through the API, JACK on Windows offers native JACK connections (INPUT), +and the JACK-Router ASIO driver (INPUT) that offers an ASIO linker to the JACK Server.

+ +

So any ASIO application will be able to use the JACK-Router ASIO driver to +connect to JACK, and JACK can then be used to route that audio to other +applications connected to the JACK Server, or out to the system sound device.

+ +

Several applications have native JACK support:

+ + + +

Native JACK applications can make use of the full power of the JACK audio and +MIDI connection system (for instance, they can open or close JACK connections +from within the application, rather than having to set an ASIO driver to use +use JACK-Router). +Also, the use of natively provided JACK ports provides the ability to sync +audio streams between applications.

+ +

The JACK-Router ASIO driver is “only” there so that users are able to use ASIO +applications in the Windows context of JACK, but this connection obviously has +some limitations that native applications won’t have.

+ +

JACK uses a ‘Server <-> Client’ architecture to supply the audio connection +between your sound device and the ASIO application(s). +When the JACK Server is running, the JACK-Router driver is then able to create +connections between clients (ASIO applications) and the JACK Server. +These connections can be seen in the QjackCtl -> Graph screen.

+ +

If the JACK Server is not running, then JACK-Router will not be available to +your ASIO application(s), and an error report may be shown.

+ +

By default, JACK-Router is set to provide 4 Input and 4 Output connections. +This setting can be customized to suite your needs by editing the +JackRouter.ini file within your +C:\Program Files\JACK\jack-router\win32 (or C:\Program Files\JACK\jack-router\win64 +folder depending of the JACK 32 or 64-bit version). +In Windows Explorer, right click on the file and choose the ‘Edit’ option. +Change the settings in the [IO] portion of the file.

+ +

Installation and Configuration

+ +
    +
  1. +

    Download the JACK2 installer for Windows from the download page.

    +
  2. +
  3. +

    Use the installer. Either the 32-bit or the 64-bit one (which still allows 32-bit clients).

    +
  4. +
  5. +

    Pick “Full installation (with JACK-Router)” during installation.

    +
  6. +
  7. +

    From the Windows Start menu, locate the QjackCtl application and run it.

    +
  8. +
  9. +

    On the QjackCtl application, click “Setup” and adjust “Interface”, “Sample Rate” and “Frames/Period” as needed for your device/sound-card. + Do not change settings you do not understand.

    +
  10. +
  11. +

    Click on OK to close the Setup dialog.

    +
  12. +
  13. +

    Click on “Start” to start the JACK Server with the parameters you just set up.

    +
  14. +
+ +

When the JACK Server is running the JACK-Router ASIO interface will be +available to all ASIO capable applications.

+ +

Uninstalling JACK

+ +

Just open the “Programs and Features” in the Windows Control Panel and +remove “JACK2 version …” from there. +This will uninstall the JACK Server, JACK-Router and QjackCtl.

+ +

Connecting Audio Applications

+ +
    +
  1. +

    Start the JACK Server using QjackCtl

    +
  2. +
  3. +

    With the Jack Audio Server started, the JackRouter ASIO driver + will be available for your ASIO supported applications.

    +
  4. +
  5. +

    Within your audio application, select JACK-Router as the audio device.

    +
  6. +
  7. +

    Within QjackCtl, click the “Graph” button.

    +
  8. +
+ +

You can route the output of ASIO applications into the inputs of other ASIO +applications (like a VST host application), and then connect the audio output +of that application to your system output (or a combination of both).

+ +

Multiple channel audio is also supported, but may require more configuration +than the simple setup described here.

+ +
+
+ + + + diff --git a/research/ambisonic-aquasynth/mirrors/mccormack-2022-parametric-ambisonic-encoding.pdf b/research/ambisonic-aquasynth/mirrors/mccormack-2022-parametric-ambisonic-encoding.pdf new file mode 100644 index 0000000..9c41938 Binary files /dev/null and b/research/ambisonic-aquasynth/mirrors/mccormack-2022-parametric-ambisonic-encoding.pdf differ diff --git a/research/ambisonic-aquasynth/mirrors/ringbuffer-understanding-ambisonics.html b/research/ambisonic-aquasynth/mirrors/ringbuffer-understanding-ambisonics.html new file mode 100644 index 0000000..777d902 --- /dev/null +++ b/research/ambisonic-aquasynth/mirrors/ringbuffer-understanding-ambisonics.html @@ -0,0 +1,478 @@ + + + + + +Understanding Ambisonics | RingBuffer + + + + + + + + + + + + + + + + + + + + +Skip to main content + + +
+
+
+ +
+ +
+
+ +
+ +
+
+
+
+ +
+
+ + + +

Understanding Ambisonics

+ + + + +
+

Mid–Side Stereo: 1D Ambisonics

+

Mid–Side (MS) stereo can be understood as a one-dimensional decomposition +of a sound field along a single horizontal axis and works with the same principle as Ambisonics. +MS represents the sound field using:

+
    +
  • M (Mid) — an cardioid (or omnidirectional) component

  • +
  • S (Side) — a bidirectional (figure-of-eight) component along the left–right axis

  • +
+
/images/spatial/mid_side.png

MS configuration with omni (blue) and figure of 8 (greed/red).

+

This structure already mirrors the first-order Ambisonic idea of +decomposing the sound field into orthogonal basis functions.

+
+

MS Encoding

+

Given conventional stereo signals \(L\) and \(R\):

+
+\begin{equation*} +M = \frac{L + R}{2} +\end{equation*} +
+
+\begin{equation*} +S = \frac{L - R}{2} +\end{equation*} +
+

The inverse decoding is:

+
+\begin{equation*} +L = M + S +\end{equation*} +
+
+\begin{equation*} +R = M - S +\end{equation*} +
+

+

Interpretation

+
    +
  • \(M\) corresponds to an omnidirectional pressure term

  • +
  • \(S\) corresponds to a dipole aligned left–right

  • +
+

The phases of the dipole encode the spatial information.

+

Mathematically, this is equivalent to a 1D first-order harmonic expansion:

+
    +
  • \(M \leftrightarrow Y_0^0\) (monopole)

  • +
  • \(S \leftrightarrow Y_1^{\pm1}\) (horizontal dipole)

  • +
+

In first-order Ambisonics (ACN/SN3D), the related components are:

+
    +
  • \(W\) — omnidirectional

  • +
  • \(Y\) — left/right dipole

  • +

+

Matrix Form

+

MS encoding can also be written as a matrix transform:

+
+\begin{equation*} +\begin{bmatrix} +M \\ +S +\end{bmatrix} += +\frac{1}{2} +\begin{bmatrix} +1 & 1 \\ +1 & -1 +\end{bmatrix} +\begin{bmatrix} +L \\ +R +\end{bmatrix} +\end{equation*} +
+

and decoding:

+
+\begin{equation*} +\begin{bmatrix} +L \\ +R +\end{bmatrix} += +\begin{bmatrix} +1 & 1 \\ +1 & -1 +\end{bmatrix} +\begin{bmatrix} +M \\ +S +\end{bmatrix} +\end{equation*} +
+

+

A vs B Format

+

A-format refers to the raw microphone capsule signals (e.g., tetrahedral mic outputs) when capturing 3D sound fields. +This format is mic-specific and not interchangeable. To further process it, it usually needs to be converted to B-format. +A→B conversion matrices are microphone-model-specific, considering capsule geometry and calibration. +Microphone vendors need to supply the decoder.

+
/images/spatial/ambisonics/ambisonics_capsule.png

B-format is made up of the spherical-harmonic components of the sound field. +This is the standard Ambisonics format with defined channel order and normalization. +B-format is portable and can be reproduced on any rendering system (loudspeaker setups, binaural), following standardized decoding algorithms.

+

+

Spherical Harmonics

+

Basic Ambisonics does not define a sound filed through positions, but through angles of incedence. +Ambisonics is based on a decomposition of a sound field into spherical harmonics and dates back to Gerzon's theory of Peryphony (Gerzon, 1973). +These spherical harmonics encode a sound field into to different axes, +The number of Ambisonics channels $N$ is equal to the number of spherical harmonics. +It can be calculated for a given order $M$ with the following formula:

+
+\begin{equation*} +N = (M+1)^2 +\end{equation*} +
+

Figure 1 shows the first 16 spherical harmonics. The first row ($N=1$) is the omnidirectional sound pressure +for the order $M=0$.

+
    +
  • Rows 1-2 together represent the $N=4$ spherical harmonics of the first order Ambisonics signal.

  • +
  • Rows 1-3 correspond to $M=2$, respectively $N=9$.

  • +
  • Rows 1-4 to the third order Ambisonics signal with $N=16$ spherical harmonics.

  • +
+

First order ambisonics is sufficient to encode a threedimensional sound field. +The higher the Ambisonics order, the more precise the directional encoding and the better the localization of virtual sound sources.

+
/images/spatial/ambisonics/third-order-ambisonics.png

Fig. 1: Spherical harmonics up to order 3 [1].

+

+

Common B-format Conventions

+

B-format conventions define the relation between Ambisonic channel order and spherical harmonics. +There are different conventions for the sequence of the individual signals, +as well as for the normalization.

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Ambisonics conventions (Common B-format variants)
ConventionTypeChannel order (1st order)NormalizationNotes / Where used
+FuMa (Furse–Malham)B-format (FOA)W, X, Y, ZFuMa (“maxN” style; W is scaled by 1/√2)Legacy 1st-order B-format used in older DAWs and toolchains; awkward for higher orders (≥2).
AmbiXB-format (FOA/HOA) +ACN order → [0:W, 1:Y, 2:Z, 3:X] +SN3DDe-facto modern production standard (Reaper+AmbiX, many VR/AR SDKs, YouTube VR). Portable and HOA-friendly.
ACN/N3DB-format (FOA/HOA) +ACN order → [0:W, 1:Y, 2:Z, 3:X] + +N3D (orthonormal)Common in research and HOA libraries; convenient for math/analysis and per-order processing.
+

+

Ambisonic Formats

+

An Ambisonics B Format file or signal carries all $N$ spherical harmonics. +Figure 2 shows a first order B Format signal.

+
/images/spatial/ambisonics/first-order-signal.png

Fig. 2: Four channels of a first order Ambisonics signal.

+
/images/spatial/ambisonics/first-order-harmonics.png

Fig. 2: Spherical harmonics of a first order Ambisonics signal.

+

+

ACN, Normalizations, and 1st-Order Mappings

+

ACN (Ambisonic Channel Numbering)

+

The channel index \(n\) is

+
+\begin{equation*} +n = \ell(\ell+1) + m, \qquad \ell = 0..L,\ \ m = -\ell..\ell. +\end{equation*} +
+

For 1st order (\(\ell = 0,1\)), the ACN indices map to channels as

+
+\begin{equation*} +[\,0:\ Y_0^0 = W,\quad 1:\ Y_1^{-1} = Y,\quad 2:\ Y_1^{0} = Z,\quad 3:\ Y_1^{1} = X\,]. +\end{equation*} +
+

Normalizations

+
    +
  • SN3D (“semi-normalized”): \(Y_0^0 = 1\). Widely used in production (AmbiX).

  • +
  • +

    N3D (fully normalized): spherical harmonics are orthonormal over the unit sphere, i.e.,

    +
    +\begin{equation*} +\int_{S^2} Y_n^m(\Omega)\, Y_{n'}^{m'}(\Omega)\, d\Omega += \delta_{nn'}\,\delta_{mm'} +\end{equation*} +
    +

    This normalization yields consistent energy per order and simplifies theoretical work, analysis, and algorithm design in higher-order Ambisonics; however, the order-dependent scaling can increase peak amplitudes in higher orders, requiring additional headroom or gain compensation to avoid clipping in practical implementations.

    +
  • +
  • FuMa (legacy): distinct scaling; notably \(W_{\text{FuMa}} = \tfrac{1}{\sqrt{2}}\,W_{\text{SN3D}}\).

  • +

1st-Order Mappings (FuMa ↔ AmbiX/ACN–SN3D)

+

Channel order

+
    +
  • FuMa order: [W, X, Y, Z]

  • +
  • AmbiX (ACN/SN3D) order: [W, Y, Z, X] (i.e., ACN indices [0,1,2,3] -> [W,Y,Z,X])

  • +

+

References

+
+

2019

+
  • Franz Zotter and Matthias Frank. +Ambisonics: A Practical 3D Audio Theory for Recording, Studio Production, Sound Reinforcement, and Virtual Reality. +Springer, 2019.
    [details] + [BibTeX▼] + +
+

2015

+
  • Matthias Frank, Franz Zotter, and Alois Sontacchi. +Producing 3d audio in ambisonics. +In Audio Engineering Society Conference: 57th International Conference: The Future of Audio Entertainment Technology–Cinema, Television and the Internet. Audio Engineering Society, 2015.
    [details] + [BibTeX▼] + +
+

2009

+
  • Frank Melchior, Andreas Gräfe, and Andreas Partzsch. +Spatial audio authoring for ambisonics reproduction. +In Proc. of the Ambisonics Symposium. 2009.
    [details] + [BibTeX▼] + +
+

1973

+
  • Michael A. Gerzon. +Periphony: With-Height Sound Reproduction. +Journal of the Audio Engineering Society, 21(1):2–10, 1973.
    [details] + [BibTeX▼] + +
+
+
+
+
+
+ + + + + diff --git a/research/ambisonic-aquasynth/mirrors/rme-multiple-cards-sync.html b/research/ambisonic-aquasynth/mirrors/rme-multiple-cards-sync.html new file mode 100644 index 0000000..4989dc8 --- /dev/null +++ b/research/ambisonic-aquasynth/mirrors/rme-multiple-cards-sync.html @@ -0,0 +1,331 @@ + + + + + + Using multiple HDSPe AoX cards :: RME Manuals + + + + + + + + + + + + + + +
+ +
+
+ +
+ +
+
+

Using multiple HDSPe AoX cards

+
+

The current driver supports the operation of up to three HDSPe AoX cards. +It is possible to use both HDSPe AoX-M and HDSPe AoX-D at the same time. +To ensure proper functionality, all cards must remain synchronized, either through a valid sync source via word clock or by using Auto-Sync with synchronized signals.

+
+

Synchronization Setup

+
+

To maintain precise synchronization between multiple cards, internal connectors are available. +The cards should be synchronized to the leader card by connecting the corresponding SYNC headers on the card with the supplied cable. +Ensure that the clock modes for all cards are configured correctly in their respective Settings dialogs.

+
+
+
    +
  • +

    When all units are supplied with a synchronous clock signal (i.e., all units display Sync in their Settings dialogs), all channels across the cards can be utilized simultaneously. +This setup is particularly streamlined under ASIO, as the ASIO driver combines all cards into a single interface.

    +
  • +
+
+

TotalMix with multiple cards

+
+

TotalMix is integrated into the hardware of each HDSPe AoX-D, allowing for up to three independent mixers when using three cards. +However, these mixers are separate and do not support data exchange between cards. +As a result, a global mixer for all units is not available.

+
+
+ + + + + +
+ + +If required, submixes could be aggregated from different cards using the optional MADI ports. +
+
+ + +
+ +
+
+
+ + + + + + diff --git a/research/ambisonic-aquasynth/mirrors/sparta-overview.html b/research/ambisonic-aquasynth/mirrors/sparta-overview.html new file mode 100644 index 0000000..dd6c71f --- /dev/null +++ b/research/ambisonic-aquasynth/mirrors/sparta-overview.html @@ -0,0 +1,10 @@ +Overview - SPARTA +

Overview

SPARTA is a collection of flexible audio plug-ins for spatial audio production, editing, and playback. The plugins are developed primarily by members of the Acoustics Lab at Aalto University, Finland. However, they have also received contributions from many other researchers, developers, and enthusiasts, from around the world.

Parametric rendering

The SPARTA installer also includes the COMPASS suite, HO-DirAC suite, CroPaC Binaural Decoder, and the HO-SIRR room impulse response renderer. +These plug-ins are restricted to lower order Ambisonics input. However, they also leverage parametric processing and are adaptive, which allows them to go beyond the capabilities of conventional linear Ambisonics algorithms. In many cases, parametric rendering of first-order Ambisonics can often match, or exceed, the perceptual performance of traditional renderers operating at much higher Ambisonic orders.

All included plug-ins

SPARTA

Learn more about the SPARTA Suite →

  • sparta_ambiBIN - A binaural ambisonic decoder (up to 10th order) with a built-in SOFA loader and head-tracking support via OSC messages. Includes: Least-Squares (LS), spatial re-sampling (SPR, virtual loudspeakers), time-alignment (TA), and magnitude least-squares (Mag-LS) decoding options.
  • sparta_ambiDEC - A frequency-dependent loudspeaker ambisonic decoder (up to 10th order) with user specifiable loudspeaker directions (up to 128), which may be optionally imported via JSON configuration files. Includes: All-Round (AllRAD), Energy-Preserving (EPAD), Spatial (SAD), and Mode-Matching (MMD) ambisonic decoding options. The loudspeaker signals may be optionally binauralised for headphone playback.
  • sparta_ambiDRC - A frequency-dependent dynamic range compressor (DRC) for ambisonic signals (up to 10th order).
  • sparta_ambiENC - An ambisonic encoder/panner (up to 10th order), with support for up to 128 input channels; the directions for which may also be imported via JSON configuration files.
  • sparta_ambiRoomSim - A shoebox room simulator based on the image-source method, supporting multiple sources and ambisonic receivers.
  • sparta_array2sh - A microphone array Ambisonics encoder (up to 10th order), with presets for several commercially available A-format and higher-order microphone arrays. The plug-in can also present objective evaluation metrics for the currently selected configuration.
  • sparta_beamformer - A spherical harmonic domain beamforming plug-in with multiple beamforming strategies (up to 128 output beams).
  • sparta_binauraliser - A binaural panner (up to 128 input channels) with a built-in SOFA loader and head-tracking support via OSC messages.
  • sparta_binauraliserNF - Binauraliser, with the addition of proximity filtering for near field sound sources.
  • sparta_decorrelator - A simple multi-channel signal decorrelator (up to 128 input channels).
  • sparta_dirass - A sound-field visualiser based on re-assigning the energy of beamformers. This re-assignment is based on DoA estimates extracted from “spatially-constrained” regions, which are centred around each beamformer look-direction.
  • sparta_matrixconv - A matrix convolver with an optional partitioned convolution mode. The user need only specify the number of inputs and load the filters via a wav file.
  • sparta_multiconv - A multi-channel convolver with an optional partitioned convolution mode. Unlike “MatrixConv”, this plug-in does not perform any matrixing. Instead, each input channel is convolved with the respective filter; i.e. numInputs = numFilters = numOutputs.
  • sparta_6DoFconv - A time-varying partitioned convolution multi-channel convolver for SOFA files containing RIRs with multiple listener positions.
  • sparta_panner - A 3D loudspeaker array panner using the VBAP method (up to 128 inputs and outputs).
  • sparta_pitchshifter - A simple multi-channel pitch shifter (up to 128 channels) based on the phase vocoder approach.
  • sparta_powermap - A sound-field visualisation plug-in based on ambisonic signals as input (up to 10th order), with PWD/MVDR/MUSIC/Min-Norm options.
  • sparta_rotator - A flexible ambisonic rotator (up to 10th order) with head-tracking support via OSC messages.
  • sparta_sldoa - A frequency-dependent sound-field visualiser (up to 10th order), based on depicting the direction-of-arrival (DoA) estimates derived from spatially localised active-intensity vectors. The low frequency estimates are shown with blue icons, mid-frequencies with green, and high-frequencies with red.
  • sparta_spreader - An arbitrary array (e.g. HRIRs or microphone array IRs) panner with coherent and incoherent spreading options.

COMPASS

Learn more about the COMPASS Suite →

  • compass_binaural - An adaptive binaural ambisonic decoder (up to 3rd order input) based on the parametric COMPASS model, with a built-in SOFA loader and head-tracking support via OSC messages.
  • compass_binauralVR - Same as the compass_binaural plug-in, but also supporting listener translation around the receiver position and support for multiple simultaneous listeners.
  • compass_decoder - A parametrically enhanced loudspeaker ambisonic decoder (up to 3rd order input).
  • compass_6dof - A six degrees-of-freedom (6DoF) renderer based on multiple Ambisonic receivers as input, which supports listener translation both within and beyond the convex hull of the receiver arrangement.
  • compass_gravitator - A parametric sound-field focussing plug-in.
  • compass_sidechain - A plug-in that manipulates the spatial properties of one Ambisonic recording based on the spatial analysis of a different recording.
  • compass_spatedit - A flexible spatial editing plug-in.
  • compass_tracker - A multiple target acoustic tracker which can optionally steer a beamformer towards each target.
  • compass_upmixer - An Ambisonic upmixer (1-3rd order input, 2-7th order output).

HO-DirAC

Learn more about the HO-DirAC Suite →

  • hodirac_binaural - An adaptive binaural ambisonic decoder (up to 3rd order input) based on the parametric HO-DirAC model, with a built-in SOFA loader and head-tracking support via OSC messages.
  • hodirac_decoder - A parametrically enhanced loudspeaker ambisonic decoder (up to 3rd order input).
  • hodirac_upmixer - An Ambisonic upmixer (1-3rd order input, 2-5th order output).

Note these plug-ins are released under their own licensing terms.

Other plug-ins

  • hades_renderer - A flexible head-worn microphone array to binaural renderer.
  • cropac_binaural - An adaptive binaural first-order ambisonic decoder based on the parametric CroPaC model, with a built-in SOFA loader and head-tracking support via OSC messages.
  • HOSIRR - An Ambisonic room impulse response (RIR) renderer for arbitrary loudspeaker setups, based on the Higher-order Spatial Impulse Response Rendering (HO-SIRR) algorithm.
  • UltrasonicSuperHearing - A plug-in for the binaural reproduction of ultrasonic sound sources captured using a 6-sensor ultrasonic microphone array, which uses direction-of-arrival estimation and pitch-down-shifting prior to binauralisation in order to provide the appropriate localisation cues.

Edit this page on GitHub

+ + + \ No newline at end of file diff --git a/research/ambisonic-aquasynth/summary.md b/research/ambisonic-aquasynth/summary.md new file mode 100644 index 0000000..9598201 --- /dev/null +++ b/research/ambisonic-aquasynth/summary.md @@ -0,0 +1,83 @@ +# Ambisonic AquaSynth Field: Distilled Research + +## Objective + +Consolidate four live microphones -- shielded cardioid, Deru shotgun, and two camera mics -- into one synchronized spatial audio scene that can feed an AquaSynth Faust DSP graph, while also placing dynamic synth patch outputs in 3D. + +## Current Mechanism + +Mimir currently treats audio as independent OBS-ingestable endpoints: one FFmpeg process per audio source, one SRT port per source, and OBS owns the mix. That is coherent for manual OBS mixing, but it is the wrong ownership model for a single spatial field. An ambisonic bus needs clocked multichannel audio before spatial encoding, then effects, then decode/render. + +## Invariants + +- Capture sync must be owned before spatial encoding. Drift correction after the fact can make playback tolerable, but it does not create a physically coherent sound field. +- The ambisonic representation must declare order, channel count, channel order, normalization, sample rate, and render target. +- The Faust graph should receive one declared spatial bus, not four unsynchronized mono stories wearing a coat. +- Microphone geometry and directivity must be explicit if microphone feeds are treated as a captured field. +- Synth outputs are easier: each synth voice is a mono or stereo source with explicit azimuth/elevation/radius metadata, then encoded into the same ambisonic bus. + +## Main Finding + +Four arbitrary microphones are not automatically a first-order ambisonic microphone. ICST distinguishes A-format as microphone-specific raw capsule output and B-format as the production/exchange sound-field representation; A-format must be converted before it is a usable ambisonic master [ICST]. A tetrahedral ambisonic mic works because capsule geometry and calibration are part of the device model. Your mic set is heterogeneous: cardioid, shotgun, and camera mics have different positions, directivities, gains, noise floors, latency, and likely clocks. + +McCormack et al. show that arbitrary microphone arrays can be encoded into Ambisonics, but even conventional linear encoding is limited by geometry, sensor placement, regularization, spatial aliasing, and bandwidth; their proposed parametric method uses spatial analysis, filtering, source/ambient decomposition, and measured or modeled array behavior [McCormack 2022]. Translation: this is research-grade if we want "real recovered sound field," not a cute four-channel matrix. + +## Recommended Architecture + +```mermaid +flowchart TD + A["4 physical mic feeds"] --> B["Clocked multichannel capture"] + B --> C["Calibration: gain, delay, polarity, noise, geometry"] + C --> D["Scene encoder"] + E["Dynamic synth patch voices"] --> F["Position metadata: azimuth, elevation, radius"] + F --> G["Faust HOA/FOA source encoders"] + D --> H["Ambisonic bus"] + G --> H + H --> I["AquaSynth Faust spatial DSP"] + I --> J["Decoder / renderer: binaural, stereo, OBS, speakers"] +``` + +## Practical Path + +1. Capture all four microphones through one clock domain. + Prefer one multichannel audio interface or a driver/server that exposes one synchronized device. RME's documentation for multiple cards is blunt that cards need a shared sync source or sync headers to remain synchronized [RME]. JACK's design centers synchronous client execution and low-latency inter-application routing, and on Windows it can bridge ASIO clients through JACK-Router [JACK API; JACK Windows]. FFmpeg DirectShow can open multiple devices and says opening them on the same input "should improve synchronism," but that is not the same as proving sample-accurate shared-clock capture [FFmpeg Devices]. + +2. Pick the honest spatial model. + For v1, do source-based ambisonic scene building: treat each mic as a directional source at a measured position, with calibration delays and gains. This produces a usable spatial mix, not a physically exact reconstruction. For v2, investigate measured arbitrary-array encoding if the room-capture realism matters enough to justify calibration sweeps and signal analysis. + +3. Use ambiX unless a tool forces otherwise. + ICST and IEM both identify ambiX as ACN channel ordering with SN3D normalization, and IEM explicitly uses that convention [ICST; IEM Compatibility]. This should be the default interchange format for the bus. + +4. Start with FOA, design the graph for HOA. + First-order Ambisonics is 4 channels. Higher order uses `(order + 1)^2` channels and gives sharper spatial resolution at higher routing and CPU cost [ICST]. Faust `hoa.lib` provides encoders, decoders, rotators, optimizers, and 3D encoders; `encoder3D(N, x, a, e)` directly fits dynamic synth voices [Faust hoa.lib]. The graph should parameterize order so FOA can ship first without boxing out HOA. + +5. Keep OBS as render target, not spatial authority. + If this path becomes live, OBS should ingest the rendered output or selected post-DSP stems. OBS should not be asked to preserve the ambisonic field while separately mixing the pre-field microphones. That splits authority and invites drift. + +## Cut Line + +Cut or suspend the old one-port-per-audio-source model for any microphone that participates in the ambisonic field. It can remain for plain OBS monitor sources, but the spatial bus must be one synchronized capture path. + +Do not build a bespoke arbitrary-array encoder first. Use Faust `hoa.lib` for source encoding and established tooling like IEM/SPARTA for verification. Only consider a custom array encoder after we have measured mic geometry, clock behavior, impulse responses, and proven that source-based encoding is not enough. + +## Open Questions + +- Are the two camera mics available as embedded channels from one capture device, or as separate USB/device clocks? +- Do we have a multichannel interface that can ingest all four mics into one ASIO device? +- Is AquaSynth currently a Faust host, a generated plugin target, or a separate graph compiler/runtime? +- Is the desired output binaural headphone monitoring, stereo OBS program audio, surround speakers, or a stored ambisonic master? +- Do we need the room itself reconstructed, or do we need a convincing controllable spatial mix for performance? + +## Citations + +- [ICST] ICST Ambisonics, "Ambisonics Formats Explained: A-Format, B-Format, FuMa, AmbiX, FOA and HOA." Mirror: `mirrors/icst-ambisonics-formats.html`. +- [RingBuffer] RingBuffer, "Understanding Ambisonics." Mirror: `mirrors/ringbuffer-understanding-ambisonics.html`. +- [McCormack 2022] McCormack, Politis, Gonzalez, Lokki, and Pulkki, "Parametric Ambisonic Encoding of Arbitrary Microphone Arrays," IEEE/ACM TASLP, 2022. Mirror: `mirrors/mccormack-2022-parametric-ambisonic-encoding.pdf`. +- [Faust hoa.lib] Faust Libraries, `hoa.lib`. Mirror: `mirrors/faust-hoa-lib.html`. +- [IEM] IEM Plug-in Suite. Mirror: `mirrors/iem-plugin-suite.html`. +- [IEM Compatibility] IEM Plug-in Suite, "Compatibility with other Suites." Mirror: `mirrors/iem-compatibility.html`. +- [SPARTA] SPARTA, "Overview." Mirror: `mirrors/sparta-overview.html`. +- [FFmpeg Devices] FFmpeg Devices Documentation. Mirror: `mirrors/ffmpeg-devices.html`. +- [JACK API] JACK Audio Connection Kit API Overview. Mirror: `mirrors/jack-api-overview.html`. +- [JACK Windows] JACK Audio Connection Kit, "Using JACK on Windows." Mirror: `mirrors/jack-on-windows.html`. +- [RME] RME Manuals, "Using multiple HDSPe AoX cards." Mirror: `mirrors/rme-multiple-cards-sync.html`. diff --git a/research/feedback-calibration/bibliography.md b/research/feedback-calibration/bibliography.md new file mode 100644 index 0000000..b1074cc --- /dev/null +++ b/research/feedback-calibration/bibliography.md @@ -0,0 +1,32 @@ +# Bibliography: Feedback As Continuous Calibration + +Mirrored copies live under `research/feedback-calibration/mirrors/`. + +## Echo / Feedback System Identification + +- S. Goetze, M. Kallinger, A. Mertins, and K. Kammeyer, "System Identification for Multi-Channel Listening-Room Compensation Using an Acoustic Echo Canceller," HSCMA 2008. + Mirror: `mirrors/goetze-2008-aec-listening-room-compensation.html` + +- F. Lindstrom, C. Schueldt, and I. Claesson, "Efficient Multichannel NLMS Implementation for Acoustic Echo Cancellation," EURASIP Journal on Audio, Speech, and Music Processing, 2007. + Mirror: `mirrors/lindstrom-2007-efficient-multichannel-nlms.html` + +- J.-M. Valin, "A New Robust Frequency Domain Echo Canceller With Closed-Loop Learning Rate Adaptation," arXiv:1602.08609, 2016. + Mirror: `mirrors/valin-2016-frequency-domain-echo-canceller-learning-rate.pdf` + +- "Double-Talk Robust Acoustic Echo Cancellation," EUSIPCO 2021. + Mirror: `mirrors/eusipco-2021-double-talk-robust-aec.pdf` + +- Apple Machine Learning Research, "Double-talk Robust Multichannel Acoustic Echo Cancellation Using Least Squares MIMO Adaptive Filtering," 2020. + Mirror: `mirrors/apple-mimo-adaptive-echo-cancellation.html` + +## Online Room / Acoustic Impulse Response Estimation + +- J. Nikunen and T. Virtanen, "Estimation of Time-Varying Room Impulse Responses of Multiple Sound Sources from Observed Mixture and Isolated Source Signals," ICASSP 2018. + Mirror: `mirrors/nikunen-virtanen-2018-time-varying-rir.pdf` + +- T. Haubner, A. Brendel, and W. Kellermann, "Online Acoustic System Identification Exploiting Kalman Filtering and an Adaptive Impulse Response Subspace Model," arXiv:2105.03337, 2021. + Mirror: `mirrors/haubner-2021-online-acoustic-system-identification.pdf` + +## Notes + +MathWorks AEC and Filtered-X LMS examples were consulted via browser snippets/opened page content, but direct mirroring was blocked by access controls from PowerShell. They are not relied on as mirrored primary artifacts in the summary. diff --git a/research/feedback-calibration/mirrors/apple-mimo-adaptive-echo-cancellation.html b/research/feedback-calibration/mirrors/apple-mimo-adaptive-echo-cancellation.html new file mode 100644 index 0000000..2c4b3dd --- /dev/null +++ b/research/feedback-calibration/mirrors/apple-mimo-adaptive-echo-cancellation.html @@ -0,0 +1,7 @@ +Double-talk Robust Multichannel Acoustic Echo Cancellation Using Least Squares MIMO Adaptive Filtering: Transversal, Array, and Lattice Forms - Apple Machine Learning Research

View publication

In this paper, we address the problem of noise-robust multiple-input multiple-output (MIMO) adaptive filtering that is optimal in least-squares sense with application to multichannel acoustic echo cancellation. We formulate the problem as minimization of a multichannel least squares cost function that incorporates near-end speech and noise statistics resulting in a novel noise-robust framework for MIMO adaptive filtering. Although the issue of numerical stability has been widely explored in the context of recursive least squares (RLS) filtering, a rigorous mathematical treatment of the MIMO case in the context of numerically stable noise-robust multichannel echo cancellation remains absent. Guided by quantization-error modeling, we resolve the issue of numerical instability in our noise-robust scheme by utilizing transversal RLS filtering of Type 2. Thereafter, an explicit derivation of its inverse QR-decomposition (IQRD) counterpart based on Givens rotations is presented. We also derive computationally efficient lattice forms for our noise-robust RLS Type-2 and IQRD algorithms. It is highlighted that propagation of angle-normalized errors occurs naturally within the numerically stable least squares lattice (LSL). Thus, our approach combines the four sought after attributes in a multichannel echo cancellation scheme, i.e., computational efficiency, numerical stability, fast convergence and tracking, and robustness against noise. We analyze our formulations using simulations in terms of convergence, re-convergence, robustness in the presence of double-talk, and numerical stability.

Related readings and updates.

Speech dereverberation has been an important component of effective far-field voice interfaces in many applications. Algorithms based on multichannel linear prediction (MCLP) have been shown to be especially effective for blind speech dereverberation and numerous variants have been introduced in the literature. Most of these approaches can be derived from a common framework, where the MCLP problem for speech dereverberation is formulated as a…

Read more

The typical audio environment for HomePod has many challenges — echo, reverberation, and noise. Unlike Siri on iPhone, which operates close to the user’s mouth, Siri on HomePod must work well in a far-field setting. Users want to invoke Siri from many locations, like the couch or the kitchen, without regard to where HomePod sits. A complete online system, which addresses all of the environmental issues that HomePod can experience, requires a tight integration of various multichannel signal processing technologies. Accordingly, the Audio Software Engineering and Siri Speech teams built a system that integrates both supervised deep learning models and unsupervised online learning algorithms and that leverages multiple microphone signals. The system selects the optimal audio stream for the speech recognizer by using top-down knowledge from the “Hey Siri” trigger phrase detectors. In this article, we discuss the machine learning techniques we use for online signal processing, as well as the challenges we faced and our solutions for achieving environmental and algorithmic robustness while ensuring energy efficiency.

Read more
\ No newline at end of file diff --git a/research/feedback-calibration/mirrors/eusipco-2021-double-talk-robust-aec.pdf b/research/feedback-calibration/mirrors/eusipco-2021-double-talk-robust-aec.pdf new file mode 100644 index 0000000..0d3ad6d Binary files /dev/null and b/research/feedback-calibration/mirrors/eusipco-2021-double-talk-robust-aec.pdf differ diff --git a/research/feedback-calibration/mirrors/goetze-2008-aec-listening-room-compensation.html b/research/feedback-calibration/mirrors/goetze-2008-aec-listening-room-compensation.html new file mode 100644 index 0000000..5ad3d43 --- /dev/null +++ b/research/feedback-calibration/mirrors/goetze-2008-aec-listening-room-compensation.html @@ -0,0 +1,881 @@ + + + + + + + + + + + + + + + + System Identification for Multi-Channel Listening-Room Compensation Using an Acoustic Echo Canceller + - Universität zu Lübeck + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+
+ +Zur Hauptnavigation wechseln + + Zur Suche wechseln + +Zum Hauptinhalt wechseln + + + + + + + + + +
+ +
+ + + + + +
+ + + + + +
+
+
+

System Identification for Multi-Channel Listening-Room Compensation Using an Acoustic Echo Canceller

+

S. Goetze, M. Kallinger, A. Mertins, K. Kammeyer

+ + + +
+ + + + + + + + + + + + + + + + +
+ +
+ +
+ +
+ + + + 7 + +   Link öffnet neuen Tab + + Zitate (Scopus) +
+ + +
+ + +
+ +
+ +
+ + + +
+ +
+ +
+ +
+ +
+ +
+ + +
+ +
+ + + +
+ +
+ +
+
+
+
+ + +

Abstract

+
Modern hands-free telecommunication devices jointly apply several subsystems, e.g. for noise reduction (NR), acoustic echo cancellation (AEC) and listening-room compensation (LRC). In this contribution the combination of an equalizer for listening room compensation and an acoustic echo canceller is analyzed. Inverse filtering of room impulse responses (RIRs) is a challenging task since they are, in general, mixed phase systems having hundreds of zeros inside and outside near the unit circle in the z-domain. Furthermore, a reliable estimate of the RIR which shall be inverted is important. Since RIRs are time-variant due to possible changes of the acoustic environment, they have to be identified adaptively. If an AEC (or any other adaptive method) is used to identify the time variant room impulse responses the estimate's distance to the real RIRs may be too high for a satisfying equalization, especially in periods of initial convergence of the AEC or after RIR changes. Therefore, we propose to estimate the convergence state of the AEC and to incorporate this knowledge into the equalizer design.
+
OriginalspracheEnglisch
Titel2008 Hands-Free Speech Communication and Microphone Arrays
Seitenumfang4
Herausgeber (Verlag)IEEE
Erscheinungsdatum01.05.2008
Seiten224-227
Aufsatznummer4538727
ISBN (Print)978-1-4244-2337-8
ISBN (elektronisch)978-1-4244-2338-5
DOIs
PublikationsstatusVeröffentlicht - 01.05.2008
Veranstaltung2008 Hands-free Speech Communication and Microphone Arrays - Trento, Italien
Dauer: 06.05.2008 → 08.05.2008
Konferenznummer: 73345
+ + + + + +
+

UN SDGs

+

Dieser Output leistet einen Beitrag zu folgendem(n) Ziel(en) für nachhaltige Entwicklung

+
    + +
  1. + + SDG 9 – Industrie, Innovation und Infrastruktur +
    + SDG 9 – Industrie, Innovation und Infrastruktur + +
    +
  2. + +
+
+ + + + +
+ + +
+
+
+ +
+
+
+

+ + Fingerprint +

+ + Untersuchen Sie die Forschungsthemen von „System Identification for Multi-Channel Listening-Room Compensation Using an Acoustic Echo Canceller“. Zusammen bilden sie einen einzigartigen Fingerprint. + +
+ + + +
+
+ + + + + + + + + + + + + + + + + + + +
+
+
+

Zitieren

+
    + + + + + + + + + + + + + + + +
+
+ + + + + + + + + + + + + + + + + +
+
+ +
+
+ + + + + + + + + + + + + + + + + + + + + + + + +
+
+ +
+ +
+ + + + + + + + + + + + + + + + + + + diff --git a/research/feedback-calibration/mirrors/haubner-2021-online-acoustic-system-identification.pdf b/research/feedback-calibration/mirrors/haubner-2021-online-acoustic-system-identification.pdf new file mode 100644 index 0000000..f721b11 Binary files /dev/null and b/research/feedback-calibration/mirrors/haubner-2021-online-acoustic-system-identification.pdf differ diff --git a/research/feedback-calibration/mirrors/lindstrom-2007-efficient-multichannel-nlms.html b/research/feedback-calibration/mirrors/lindstrom-2007-efficient-multichannel-nlms.html new file mode 100644 index 0000000..f431891 --- /dev/null +++ b/research/feedback-calibration/mirrors/lindstrom-2007-efficient-multichannel-nlms.html @@ -0,0 +1,88 @@ + + + + + + + + Client Challenge + + + + + + + + diff --git a/research/feedback-calibration/mirrors/nikunen-virtanen-2018-time-varying-rir.pdf b/research/feedback-calibration/mirrors/nikunen-virtanen-2018-time-varying-rir.pdf new file mode 100644 index 0000000..349de6c Binary files /dev/null and b/research/feedback-calibration/mirrors/nikunen-virtanen-2018-time-varying-rir.pdf differ diff --git a/research/feedback-calibration/mirrors/valin-2016-frequency-domain-echo-canceller-learning-rate.pdf b/research/feedback-calibration/mirrors/valin-2016-frequency-domain-echo-canceller-learning-rate.pdf new file mode 100644 index 0000000..6b76a31 Binary files /dev/null and b/research/feedback-calibration/mirrors/valin-2016-frequency-domain-echo-canceller-learning-rate.pdf differ diff --git a/research/feedback-calibration/summary.md b/research/feedback-calibration/summary.md new file mode 100644 index 0000000..590c67d --- /dev/null +++ b/research/feedback-calibration/summary.md @@ -0,0 +1,96 @@ +# Feedback As Continuous Calibration + +## Objective + +Treat speaker-to-microphone feedback as a useful measurement stream, not merely noise to suppress, so the system can continuously learn the playback-room-microphone transfer paths while running. + +## Current Mechanism + +The playback calibration note treats speaker measurements as an explicit calibration pass. The live adaptive sync note warns that speaker bleed can corrupt mic sync/localization. Both are true, but incomplete: because the emitted speaker signal is known, the bleed is also an online system-identification signal. + +## Main Finding + +This is the territory of acoustic echo cancellation, adaptive feedback cancellation, and online acoustic system identification. The useful model is: + +```text +mic_signal = live_sources + speaker_output convolved with room/speaker/mic paths + noise +``` + +If the system knows `speaker_output`, an adaptive filter can estimate the transfer path from each speaker output channel to each microphone. Goetze et al. explicitly connect acoustic echo cancellation to listening-room compensation and note that room impulse responses are time-varying and must be identified adaptively, while also warning that equalizer design should account for the echo canceller's convergence state [Goetze 2008]. That warning matters: a half-converged estimator is not truth; it is a drunk ruler. + +Nikunen and Virtanen describe online estimation of time-varying room impulse responses when isolated source signals are available, including playback material output through loudspeakers in a live performance recorded for 3D spatial audio [Nikunen/Virtanen 2018]. That is nearly our shape: known synth/playback streams, observed far-field mic mixtures, and spatial reconstruction. + +Haubner et al. frame online acoustic impulse response estimation as a Kalman/system-identification problem and target faster convergence under correlated excitation and interfering noise [Haubner 2021]. Multichannel AEC work, including Apple ML's MIMO least-squares framing, addresses the harder version where multiple speakers and multiple microphones are coupled and "double-talk" or near-end sound is present [Apple 2020]. + +## Required Ownership + +Feedback mining should be its own subsystem: + +```mermaid +flowchart TD + A["Known speaker/render output"] --> B["Echo path estimator"] + C["Microphone returns"] --> B + B --> D["Speaker->mic transfer matrix H_sm"] + D --> E["Echo prediction"] + C --> F["Live source estimator"] + E --> F + D --> G["Playback calibration updates"] + D --> H["Sync/localization confidence gates"] + F --> I["Aligned capture/spatial scene"] +``` + +It should not be hidden inside the drift loop or the ambisonic encoder. The transfer matrix `H_sm` owns "what our speakers currently sound like at our microphones." The clock loop owns sampling-rate mismatch. The localization loop owns external source position. The decoder owns speaker rendering. Four authorities. No soup. + +## Design Implications + +- Continuous feedback is a calibration signal when the outgoing audio is known and time-aligned. +- It is also an interference signal for voice capture, so the system should predict and optionally subtract or downweight it. +- Adaptation must be gated during strong unknown near-field speech, heavy clipping, nonlinear speaker behavior, or low coherence. +- Each speaker-to-mic path is a filter, not just one delay. Early taps are useful for direct-path timing and geometry; later taps describe room reflections. +- In multichannel playback, the estimator is MIMO: every speaker can leak into every mic. +- The estimator should publish confidence, convergence, residual error, and last-change time. Do not let downstream systems treat a stale or diverging filter as a fact. + +## Candidate V1 + +1. Start offline/controlled: + - play one speaker at a time with chirp/noise + - estimate fixed speaker-to-reference-mic impulse responses + - store speaker delay, gain, polarity, and early reflection sketch + +2. Add online passive tracking: + - feed known speaker output and mic input into an adaptive filter + - start with two paths: left speaker -> reference mic, right speaker -> reference mic + - estimate residual echo and convergence + +3. Add live gating: + - freeze or slow adaptation during strong unknown speech + - adapt faster during known probe/noise windows or synth-only output + - reject updates when residual/error spikes indicate nonlinearity or movement + +4. Scale to MIMO: + - speaker channels x microphone channels + - use the resulting transfer matrix to improve playback correction, echo prediction, and confidence in mic localization + +## How It Helps AquaSynth + +- The system can tell whether a synth voice rendered to "front left" actually arrives front-left-ish at the mic array. +- Speaker leakage can be subtracted or modeled before estimating external voice position. +- The room response can update slowly as people move, doors open, or speaker volume changes. +- Calibration pings become optional accelerators rather than the only source of truth. + +## Risks + +- Adaptive echo filters can diverge under double-talk unless gated or made robust. +- Loudspeakers are nonlinear at high levels; a linear impulse response will not explain distortion. +- Correlated stereo/ambisonic speaker feeds make MIMO echo identification harder because the system has trouble deciding which speaker caused which mic energy. +- If we use the estimator to aggressively cancel feedback, we may remove useful spatial ambience. The goal is not sterile cancellation; it is intelligible ownership of the signal. + +## Citations + +- [Goetze 2008] S. Goetze, M. Kallinger, A. Mertins, and K. Kammeyer, "System Identification for Multi-Channel Listening-Room Compensation Using an Acoustic Echo Canceller." Mirror: `mirrors/goetze-2008-aec-listening-room-compensation.html`. +- [Lindstrom 2007] F. Lindstrom, C. Schueldt, and I. Claesson, "Efficient Multichannel NLMS Implementation for Acoustic Echo Cancellation." Mirror: `mirrors/lindstrom-2007-efficient-multichannel-nlms.html`. +- [Valin 2016] J.-M. Valin, "A New Robust Frequency Domain Echo Canceller With Closed-Loop Learning Rate Adaptation." Mirror: `mirrors/valin-2016-frequency-domain-echo-canceller-learning-rate.pdf`. +- [EUSIPCO 2021] "Double-Talk Robust Acoustic Echo Cancellation." Mirror: `mirrors/eusipco-2021-double-talk-robust-aec.pdf`. +- [Apple 2020] Apple Machine Learning Research, "Double-talk Robust Multichannel Acoustic Echo Cancellation Using Least Squares MIMO Adaptive Filtering." Mirror: `mirrors/apple-mimo-adaptive-echo-cancellation.html`. +- [Nikunen/Virtanen 2018] J. Nikunen and T. Virtanen, "Estimation of Time-Varying Room Impulse Responses of Multiple Sound Sources from Observed Mixture and Isolated Source Signals." Mirror: `mirrors/nikunen-virtanen-2018-time-varying-rir.pdf`. +- [Haubner 2021] T. Haubner, A. Brendel, and W. Kellermann, "Online Acoustic System Identification Exploiting Kalman Filtering and an Adaptive Impulse Response Subspace Model." Mirror: `mirrors/haubner-2021-online-acoustic-system-identification.pdf`. diff --git a/research/gpu-fusion-splatting/bibliography.md b/research/gpu-fusion-splatting/bibliography.md new file mode 100644 index 0000000..ec74bd8 --- /dev/null +++ b/research/gpu-fusion-splatting/bibliography.md @@ -0,0 +1,51 @@ +# Bibliography: GPU Fusion And Live Splat Rendering + +Mirrored copies live under `research/gpu-fusion-splatting/mirrors/`. + +## CUDA / GPU Runtime + +- [NVIDIA CUDA C++ Programming Guide](https://docs.nvidia.com/cuda/cuda-programming-guide/pdf/cuda-programming-guide.pdf) + Mirror: `mirrors/nvidia-cuda-programming-guide.pdf` + +- [NVIDIA CUDA C++ Programming Guide, asynchronous/concurrent execution](https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#asynchronous-concurrent-execution) + Mirror: `mirrors/nvidia-cuda-async-concurrent-execution.html` + +- [NVIDIA Vision Programming Interface](https://docs.nvidia.com/vpi/) + Mirror: `mirrors/nvidia-vpi-main.html` + +- [NVIDIA Optical Flow SDK Programming Guide](https://docs.nvidia.com/video-technologies/optical-flow-sdk/pdf/NVOFA_Programming_Guide.pdf) + Mirror: `mirrors/nvidia-optical-flow-programming-guide.pdf` + +- [OpenCV CUDA-accelerated Computer Vision module](https://docs.opencv.org/3.4/d1/d1e/group__cuda.html) + Mirror: `mirrors/opencv-cuda-module.html` + +## Gaussian Splatting Implementations + +- [GraphDeco/Inria, `gaussian-splatting`](https://github.com/graphdeco-inria/gaussian-splatting) + Mirrors: `mirrors/graphdeco-gaussian-splatting-github.html`, `mirrors/graphdeco-gaussian-splatting-readme.md` + +- [NVIDIA Developer Blog, "Real-Time GPU-Accelerated Gaussian Splatting with NVIDIA DesignWorks Sample vk_gaussian_splatting"](https://developer.nvidia.com/blog/real-time-gpu-accelerated-gaussian-splatting-with-nvidia-designworks-sample-vk_gaussian_splatting/) + Mirror: `mirrors/nvidia-vk-gaussian-splatting-blog.html` + +- [gsplat documentation](https://docs.gsplat.studio/main/) + Mirror: `mirrors/gsplat-docs.html` + +- [Nerfstudio Project, `gsplat`](https://github.com/nerfstudio-project/gsplat) + Mirror: `mirrors/gsplat-github.html` + +- M. Ye et al., ["gsplat: An Open-Source Library for Gaussian Splatting"](https://jmlr.org/papers/volume26/24-1476/24-1476.pdf), JMLR, 2025. + Mirror: `mirrors/ye-2025-gsplat.pdf` + +- [Nerfstudio, "Splatfacto"](https://docs.nerf.studio/nerfology/methods/splat.html) + Mirror: `mirrors/nerfstudio-splatfacto.html` + +## Gaussian Splatting Papers + +- Bernhard Kerbl, Georgios Kopanas, Thomas Leimkuehler, and George Drettakis, ["3D Gaussian Splatting for Real-Time Radiance Field Rendering"](https://arxiv.org/abs/2308.04079), SIGGRAPH 2023. + Mirror: `mirrors/kerbl-2023-3d-gaussian-splatting.pdf` + +- Guanjun Wu et al., ["4D Gaussian Splatting for Real-Time Dynamic Scene Rendering"](https://openaccess.thecvf.com/content/CVPR2024/papers/Wu_4D_Gaussian_Splatting_for_Real-Time_Dynamic_Scene_Rendering_CVPR_2024_paper.pdf), CVPR 2024. + Mirror: `mirrors/wu-2024-4d-gaussian-splatting.pdf` + +- Zhexi Peng et al., ["RTG-SLAM: Real-time 3D Reconstruction at Scale using Gaussian Splatting"](https://arxiv.org/abs/2404.19706), SIGGRAPH 2024. + Mirror: `mirrors/peng-2024-rtg-slam.pdf` diff --git a/research/gpu-fusion-splatting/mirrors/graphdeco-gaussian-splatting-github.html b/research/gpu-fusion-splatting/mirrors/graphdeco-gaussian-splatting-github.html new file mode 100644 index 0000000..355ff27 --- /dev/null +++ b/research/gpu-fusion-splatting/mirrors/graphdeco-gaussian-splatting-github.html @@ -0,0 +1,2316 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + GitHub - graphdeco-inria/gaussian-splatting: Original reference implementation of "3D Gaussian Splatting for Real-Time Radiance Field Rendering" · GitHub + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+
+ + +
+ Skip to content + + + + + + + + + + + +
+
+ + + + + + + + + + + + + + + + + + + +
+ +
+ + + + + + + + +
+ + + + + +
+ + + + + + + + + +
+
+
+ + + + + + + + + + + + + +
+ +
+ +
+ +
+ + + + / + + gaussian-splatting + + + Public +
+ + +
+ +
+ + +
+
+ +
+
+ + + + +
+ + + + + +
+ + + + + + + + + + + + + + + + + + +
+ + + + +

graphdeco-inria/gaussian-splatting

+ + + +

Repository files navigation

3D Gaussian Splatting for Real-Time Radiance Field Rendering

+

Bernhard Kerbl*, Georgios Kopanas*, Thomas Leimkühler, George Drettakis (* indicates equal contribution)
+| Webpage | Full Paper | Video | Other GRAPHDECO Publications | FUNGRAPH project page |
+| T&T+DB COLMAP (650MB) | Pre-trained Models (14 GB) | Viewers for Windows (60MB) | Evaluation Images (7 GB) |
+Teaser image

+

This repository contains the official authors implementation associated with the paper "3D Gaussian Splatting for Real-Time Radiance Field Rendering", which can be found here. We further provide the reference images used to create the error metrics reported in the paper, as well as recently created, pre-trained models.

+

+ + +

+

Abstract: Radiance Field methods have recently revolutionized novel-view synthesis of scenes captured with multiple photos or videos. However, achieving high visual quality still requires neural networks that are costly to train and render, while recent faster methods inevitably trade off speed for quality. For unbounded and complete scenes (rather than isolated objects) and 1080p resolution rendering, no current method can achieve real-time display rates. We introduce three key elements that allow us to achieve state-of-the-art visual quality while maintaining competitive training times and importantly allow high-quality real-time (≥ 30 fps) novel-view synthesis at 1080p resolution. First, starting from sparse points produced during camera calibration, we represent the scene with 3D Gaussians that preserve desirable properties of continuous volumetric radiance fields for scene optimization while avoiding unnecessary computation in empty space; Second, we perform interleaved optimization/density control of the 3D Gaussians, notably optimizing anisotropic covariance to achieve an accurate representation of the scene; Third, we develop a fast visibility-aware rendering algorithm that supports anisotropic splatting and both accelerates training and allows realtime rendering. We demonstrate state-of-the-art visual quality and real-time rendering on several established datasets.

+
+
+

BibTeX

+
@Article{kerbl3Dgaussians,
+      author       = {Kerbl, Bernhard and Kopanas, Georgios and Leimk{\"u}hler, Thomas and Drettakis, George},
+      title        = {3D Gaussian Splatting for Real-Time Radiance Field Rendering},
+      journal      = {ACM Transactions on Graphics},
+      number       = {4},
+      volume       = {42},
+      month        = {July},
+      year         = {2023},
+      url          = {https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/}
+}
+
+
+

Funding and Acknowledgments

+

This research was funded by the ERC Advanced grant FUNGRAPH No 788065. The authors are grateful to Adobe for generous donations, the OPAL infrastructure from Université Côte d’Azur and for the HPC resources from GENCI–IDRIS (Grant 2022-AD011013409). The authors thank the anonymous reviewers for their valuable feedback, P. Hedman and A. Tewari for proofreading earlier drafts also T. Müller, A. Yu and S. Fridovich-Keil for helping with the comparisons.

+

NEW FEATURES !

+

We have limited resources for maintaining and updating the code. However, we have added a few new features since the original release that are inspired by some of the excellent work many other researchers have been doing on 3DGS. We will be adding other features within the ability of our resources.

+

Update of October 2024: We integrated training speed acceleration and made it compatible with depth regularization, anti-aliasing and exposure compensation. We have enhanced the SIBR real time viewer by correcting bugs and adding features in the Top View that allows visualization of input and user cameras.

+

Update of Spring 2024: +Orange Labs has kindly added OpenXR support for VR viewing.

+

Step-by-step Tutorial

+

Jonathan Stephens made a fantastic step-by-step tutorial for setting up Gaussian Splatting on your machine, along with instructions for creating usable datasets from videos. If the instructions below are too dry for you, go ahead and check it out here.

+

Colab

+

User camenduru was kind enough to provide a Colab template that uses this repo's source (status: August 2023!) for quick and easy access to the method. Please check it out here.

+

Cloning the Repository

+

The repository contains submodules, thus please check it out with

+
# SSH
+git clone git@github.com:graphdeco-inria/gaussian-splatting.git --recursive
+

or

+
# HTTPS
+git clone https://github.com/graphdeco-inria/gaussian-splatting --recursive
+

Overview

+

The codebase has 4 main components:

+
    +
  • A PyTorch-based optimizer to produce a 3D Gaussian model from SfM inputs
  • +
  • A network viewer that allows to connect to and visualize the optimization process
  • +
  • An OpenGL-based real-time viewer to render trained models in real-time.
  • +
  • A script to help you turn your own images into optimization-ready SfM data sets
  • +
+

The components have different requirements w.r.t. both hardware and software. They have been tested on Windows 10 and Ubuntu Linux 22.04. Instructions for setting up and running each of them are found in the sections below.

+

Optimizer

+

The optimizer uses PyTorch and CUDA extensions in a Python environment to produce trained models.

+

Hardware Requirements

+
    +
  • CUDA-ready GPU with Compute Capability 7.0+
  • +
  • 24 GB VRAM (to train to paper evaluation quality)
  • +
  • Please see FAQ for smaller VRAM configurations
  • +
+

Software Requirements

+
    +
  • Conda (recommended for easy setup)
  • +
  • C++ Compiler for PyTorch extensions (we used Visual Studio 2019 for Windows)
  • +
  • CUDA SDK 11 for PyTorch extensions, install after Visual Studio (we used 11.8, known issues with 11.6)
  • +
  • C++ Compiler and CUDA SDK must be compatible
  • +
+

Setup

+

Local Setup

+

Our default, provided install method is based on Conda package and environment management:

+
SET DISTUTILS_USE_SDK=1 # Windows only
+conda env create --file environment.yml
+conda activate gaussian_splatting
+

Please note that this process assumes that you have CUDA SDK 11 installed, not 12. For modifications, see below.

+

Tip: Downloading packages and creating a new environment with Conda can require a significant amount of disk space. By default, Conda will use the main system hard drive. You can avoid this by specifying a different package download location and an environment on a different drive:

+
conda config --add pkgs_dirs <Drive>/<pkg_path>
+conda env create --file environment.yml --prefix <Drive>/<env_path>/gaussian_splatting
+conda activate <Drive>/<env_path>/gaussian_splatting
+

Modifications

+

If you can afford the disk space, we recommend using our environment files for setting up a training environment identical to ours. If you want to make modifications, please note that major version changes might affect the results of our method. However, our (limited) experiments suggest that the codebase works just fine inside a more up-to-date environment (Python 3.8, PyTorch 2.0.0, CUDA 12). Make sure to create an environment where PyTorch and its CUDA runtime version match and the installed CUDA SDK has no major version difference with PyTorch's CUDA version.

+

Known Issues

+

Some users experience problems building the submodules on Windows (cl.exe: File not found or similar). Please consider the workaround for this problem from the FAQ.

+

Running

+

To run the optimizer, simply use

+
python train.py -s <path to COLMAP or NeRF Synthetic dataset>
+
+Command Line Arguments for train.py +

--source_path / -s

+

Path to the source directory containing a COLMAP or Synthetic NeRF data set.

+

--model_path / -m

+

Path where the trained model should be stored (output/<random> by default).

+

--images / -i

+

Alternative subdirectory for COLMAP images (images by default).

+

--eval

+

Add this flag to use a MipNeRF360-style training/test split for evaluation.

+

--resolution / -r

+

Specifies resolution of the loaded images before training. If provided 1, 2, 4 or 8, uses original, 1/2, 1/4 or 1/8 resolution, respectively. For all other values, rescales the width to the given number while maintaining image aspect. If not set and input image width exceeds 1.6K pixels, inputs are automatically rescaled to this target.

+

--data_device

+

Specifies where to put the source image data, cuda by default, recommended to use cpu if training on large/high-resolution dataset, will reduce VRAM consumption, but slightly slow down training. Thanks to HrsPythonix.

+

--white_background / -w

+

Add this flag to use white background instead of black (default), e.g., for evaluation of NeRF Synthetic dataset.

+

--sh_degree

+

Order of spherical harmonics to be used (no larger than 3). 3 by default.

+

--convert_SHs_python

+

Flag to make pipeline compute forward and backward of SHs with PyTorch instead of ours.

+

--convert_cov3D_python

+

Flag to make pipeline compute forward and backward of the 3D covariance with PyTorch instead of ours.

+

--debug

+

Enables debug mode if you experience erros. If the rasterizer fails, a dump file is created that you may forward to us in an issue so we can take a look.

+

--debug_from

+

Debugging is slow. You may specify an iteration (starting from 0) after which the above debugging becomes active.

+

--iterations

+

Number of total iterations to train for, 30_000 by default.

+

--ip

+

IP to start GUI server on, 127.0.0.1 by default.

+

--port

+

Port to use for GUI server, 6009 by default.

+

--test_iterations

+

Space-separated iterations at which the training script computes L1 and PSNR over test set, 7000 30000 by default.

+

--save_iterations

+

Space-separated iterations at which the training script saves the Gaussian model, 7000 30000 <iterations> by default.

+

--checkpoint_iterations

+

Space-separated iterations at which to store a checkpoint for continuing later, saved in the model directory.

+

--start_checkpoint

+

Path to a saved checkpoint to continue training from.

+

--quiet

+

Flag to omit any text written to standard out pipe.

+

--feature_lr

+

Spherical harmonics features learning rate, 0.0025 by default.

+

--opacity_lr

+

Opacity learning rate, 0.05 by default.

+

--scaling_lr

+

Scaling learning rate, 0.005 by default.

+

--rotation_lr

+

Rotation learning rate, 0.001 by default.

+

--position_lr_max_steps

+

Number of steps (from 0) where position learning rate goes from initial to final. 30_000 by default.

+

--position_lr_init

+

Initial 3D position learning rate, 0.00016 by default.

+

--position_lr_final

+

Final 3D position learning rate, 0.0000016 by default.

+

--position_lr_delay_mult

+

Position learning rate multiplier (cf. Plenoxels), 0.01 by default.

+

--densify_from_iter

+

Iteration where densification starts, 500 by default.

+

--densify_until_iter

+

Iteration where densification stops, 15_000 by default.

+

--densify_grad_threshold

+

Limit that decides if points should be densified based on 2D position gradient, 0.0002 by default.

+

--densification_interval

+

How frequently to densify, 100 (every 100 iterations) by default.

+

--opacity_reset_interval

+

How frequently to reset opacity, 3_000 by default.

+

--lambda_dssim

+

Influence of SSIM on total loss from 0 to 1, 0.2 by default.

+

--percent_dense

+

Percentage of scene extent (0--1) a point must exceed to be forcibly densified, 0.01 by default.

+
+
+

Note that similar to MipNeRF360, we target images at resolutions in the 1-1.6K pixel range. For convenience, arbitrary-size inputs can be passed and will be automatically resized if their width exceeds 1600 pixels. We recommend to keep this behavior, but you may force training to use your higher-resolution images by setting -r 1.

+

The MipNeRF360 scenes are hosted by the paper authors here. You can find our SfM data sets for Tanks&Temples and Deep Blending here. If you do not provide an output model directory (-m), trained models are written to folders with randomized unique names inside the output directory. At this point, the trained models may be viewed with the real-time viewer (see further below).

+

Evaluation

+

By default, the trained models use all available images in the dataset. To train them while withholding a test set for evaluation, use the --eval flag. This way, you can render training/test sets and produce error metrics as follows:

+
python train.py -s <path to COLMAP or NeRF Synthetic dataset> --eval # Train with train/test split
+python render.py -m <path to trained model> # Generate renderings
+python metrics.py -m <path to trained model> # Compute error metrics on renderings
+

If you want to evaluate our pre-trained models, you will have to download the corresponding source data sets and indicate their location to render.py with an additional --source_path/-s flag. Note: The pre-trained models were created with the release codebase. This code base has been cleaned up and includes bugfixes, hence the metrics you get from evaluating them will differ from those in the paper.

+
python render.py -m <path to pre-trained model> -s <path to COLMAP dataset>
+python metrics.py -m <path to pre-trained model>
+
+Command Line Arguments for render.py +

--model_path / -m

+

Path to the trained model directory you want to create renderings for.

+

--skip_train

+

Flag to skip rendering the training set.

+

--skip_test

+

Flag to skip rendering the test set.

+

--quiet

+

Flag to omit any text written to standard out pipe.

+

The below parameters will be read automatically from the model path, based on what was used for training. However, you may override them by providing them explicitly on the command line.

+

--source_path / -s

+

Path to the source directory containing a COLMAP or Synthetic NeRF data set.

+

--images / -i

+

Alternative subdirectory for COLMAP images (images by default).

+

--eval

+

Add this flag to use a MipNeRF360-style training/test split for evaluation.

+

--resolution / -r

+

Changes the resolution of the loaded images before training. If provided 1, 2, 4 or 8, uses original, 1/2, 1/4 or 1/8 resolution, respectively. For all other values, rescales the width to the given number while maintaining image aspect. 1 by default.

+

--white_background / -w

+

Add this flag to use white background instead of black (default), e.g., for evaluation of NeRF Synthetic dataset.

+

--convert_SHs_python

+

Flag to make pipeline render with computed SHs from PyTorch instead of ours.

+

--convert_cov3D_python

+

Flag to make pipeline render with computed 3D covariance from PyTorch instead of ours.

+
+
+Command Line Arguments for metrics.py +

--model_paths / -m

+

Space-separated list of model paths for which metrics should be computed.

+
+
+

We further provide the full_eval.py script. This script specifies the routine used in our evaluation and demonstrates the use of some additional parameters, e.g., --images (-i) to define alternative image directories within COLMAP data sets. If you have downloaded and extracted all the training data, you can run it like this:

+
python full_eval.py -m360 <mipnerf360 folder> -tat <tanks and temples folder> -db <deep blending folder>
+

In the current version, this process takes about 7h on our reference machine containing an A6000. If you want to do the full evaluation on our pre-trained models, you can specify their download location and skip training.

+
python full_eval.py -o <directory with pretrained models> --skip_training -m360 <mipnerf360 folder> -tat <tanks and temples folder> -db <deep blending folder>
+

If you want to compute the metrics on our paper's evaluation images, you can also skip rendering. In this case it is not necessary to provide the source datasets. You can compute metrics for multiple image sets at a time.

+
python full_eval.py -m <directory with evaluation images>/garden ... --skip_training --skip_rendering
+
+Command Line Arguments for full_eval.py +

--skip_training

+

Flag to skip training stage.

+

--skip_rendering

+

Flag to skip rendering stage.

+

--skip_metrics

+

Flag to skip metrics calculation stage.

+

--output_path

+

Directory to put renderings and results in, ./eval by default, set to pre-trained model location if evaluating them.

+

--mipnerf360 / -m360

+

Path to MipNeRF360 source datasets, required if training or rendering.

+

--tanksandtemples / -tat

+

Path to Tanks&Temples source datasets, required if training or rendering.

+

--deepblending / -db

+

Path to Deep Blending source datasets, required if training or rendering.

+
+
+

Interactive Viewers

+

We provide two interactive viewers for our method: remote and real-time. Our viewing solutions are based on the SIBR framework, developed by the GRAPHDECO group for several novel-view synthesis projects.

+

Hardware Requirements

+
    +
  • OpenGL 4.5-ready GPU and drivers (or latest MESA software)
  • +
  • 4 GB VRAM recommended
  • +
  • CUDA-ready GPU with Compute Capability 7.0+ (only for Real-Time Viewer)
  • +
+

Software Requirements

+
    +
  • Visual Studio or g++, not Clang (we used Visual Studio 2019 for Windows)
  • +
  • CUDA SDK 11, install after Visual Studio (we used 11.8)
  • +
  • CMake (recent version, we used 3.24)
  • +
  • 7zip (only on Windows)
  • +
+

Pre-built Windows Binaries

+

We provide pre-built binaries for Windows here. We recommend using them on Windows for an efficient setup, since the building of SIBR involves several external dependencies that must be downloaded and compiled on-the-fly.

+

Installation from Source

+

If you cloned with submodules (e.g., using --recursive), the source code for the viewers is found in SIBR_viewers. The network viewer runs within the SIBR framework for Image-based Rendering applications.

+

Windows

+

CMake should take care of your dependencies.

+
cd SIBR_viewers
+cmake -Bbuild .
+cmake --build build --target install --config RelWithDebInfo
+

You may specify a different configuration, e.g. Debug if you need more control during development.

+

Ubuntu 22.04

+

You will need to install a few dependencies before running the project setup.

+
# Dependencies
+sudo apt install -y libglew-dev libassimp-dev libboost-all-dev libgtk-3-dev libopencv-dev libglfw3-dev libavdevice-dev libavcodec-dev libeigen3-dev libxxf86vm-dev libembree-dev
+# Project setup
+cd SIBR_viewers
+cmake -Bbuild . -DCMAKE_BUILD_TYPE=Release # add -G Ninja to build faster
+cmake --build build -j24 --target install
+

Ubuntu 20.04

+

Backwards compatibility with Focal Fossa is not fully tested, but building SIBR with CMake should still work after invoking

+
git checkout fossa_compatibility
+

Navigation in SIBR Viewers

+

The SIBR interface provides several methods of navigating the scene. By default, you will be started with an FPS navigator, which you can control with W, A, S, D, Q, E for camera translation and I, K, J, L, U, O for rotation. Alternatively, you may want to use a Trackball-style navigator (select from the floating menu). You can also snap to a camera from the data set with the Snap to button or find the closest camera with Snap to closest. The floating menues also allow you to change the navigation speed. You can use the Scaling Modifier to control the size of the displayed Gaussians, or show the initial point cloud.

+

Running the Network Viewer

+
+ + + remoteviewer.mp4 + + + + +
+ +

After extracting or installing the viewers, you may run the compiled SIBR_remoteGaussian_app[_config] app in <SIBR install dir>/bin, e.g.:

+
./<SIBR install dir>/bin/SIBR_remoteGaussian_app
+

The network viewer allows you to connect to a running training process on the same or a different machine. If you are training on the same machine and OS, no command line parameters should be required: the optimizer communicates the location of the training data to the network viewer. By default, optimizer and network viewer will try to establish a connection on localhost on port 6009. You can change this behavior by providing matching --ip and --port parameters to both the optimizer and the network viewer. If for some reason the path used by the optimizer to find the training data is not reachable by the network viewer (e.g., due to them running on different (virtual) machines), you may specify an override location to the viewer by using -s <source path>.

+
+Primary Command Line Arguments for Network Viewer +

--path / -s

+

Argument to override model's path to source dataset.

+

--ip

+

IP to use for connection to a running training script.

+

--port

+

Port to use for connection to a running training script.

+

--rendering-size

+

Takes two space separated numbers to define the resolution at which network rendering occurs, 1200 width by default. +Note that to enforce an aspect that differs from the input images, you need --force-aspect-ratio too.

+

--load_images

+

Flag to load source dataset images to be displayed in the top view for each camera.

+
+
+

Running the Real-Time Viewer

+
+ + + realtimeviewer.mp4 + + + + +
+ +

After extracting or installing the viewers, you may run the compiled SIBR_gaussianViewer_app[_config] app in <SIBR install dir>/bin, e.g.:

+
./<SIBR install dir>/bin/SIBR_gaussianViewer_app -m <path to trained model>
+

It should suffice to provide the -m parameter pointing to a trained model directory. Alternatively, you can specify an override location for training input data using -s. To use a specific resolution other than the auto-chosen one, specify --rendering-size <width> <height>. Combine it with --force-aspect-ratio if you want the exact resolution and don't mind image distortion.

+

To unlock the full frame rate, please disable V-Sync on your machine and also in the application (Menu → Display). In a multi-GPU system (e.g., laptop) your OpenGL/Display GPU should be the same as your CUDA GPU (e.g., by setting the application's GPU preference on Windows, see below) for maximum performance.

+

Teaser image

+

In addition to the initial point cloud and the splats, you also have the option to visualize the Gaussians by rendering them as ellipsoids from the floating menu. +SIBR has many other functionalities, please see the documentation for more details on the viewer, navigation options etc. There is also a Top View (available from the menu) that shows the placement of the input cameras and the original SfM point cloud; please note that Top View slows rendering when enabled. The real-time viewer also uses slightly more aggressive, fast culling, which can be toggled in the floating menu. If you ever encounter an issue that can be solved by turning fast culling off, please let us know.

+
+Primary Command Line Arguments for Real-Time Viewer +

--model-path / -m

+

Path to trained model.

+

--iteration

+

Specifies which of state to load if multiple are available. Defaults to latest available iteration.

+

--path / -s

+

Argument to override model's path to source dataset.

+

--rendering-size

+

Takes two space separated numbers to define the resolution at which real-time rendering occurs, 1200 width by default. Note that to enforce an aspect that differs from the input images, you need --force-aspect-ratio too.

+

--load_images

+

Flag to load source dataset images to be displayed in the top view for each camera.

+

--device

+

Index of CUDA device to use for rasterization if multiple are available, 0 by default.

+

--no_interop

+

Disables CUDA/GL interop forcibly. Use on systems that may not behave according to spec (e.g., WSL2 with MESA GL 4.5 software rendering).

+
+
+

Processing your own Scenes

+

Our COLMAP loaders expect the following dataset structure in the source path location:

+
<location>
+|---images
+|   |---<image 0>
+|   |---<image 1>
+|   |---...
+|---sparse
+    |---0
+        |---cameras.bin
+        |---images.bin
+        |---points3D.bin
+
+

For rasterization, the camera models must be either a SIMPLE_PINHOLE or PINHOLE camera. We provide a converter script convert.py, to extract undistorted images and SfM information from input images. Optionally, you can use ImageMagick to resize the undistorted images. This rescaling is similar to MipNeRF360, i.e., it creates images with 1/2, 1/4 and 1/8 the original resolution in corresponding folders. To use them, please first install a recent version of COLMAP (ideally CUDA-powered) and ImageMagick. Put the images you want to use in a directory <location>/input.

+
<location>
+|---input
+    |---<image 0>
+    |---<image 1>
+    |---...
+
+

If you have COLMAP and ImageMagick on your system path, you can simply run

+
python convert.py -s <location> [--resize] #If not resizing, ImageMagick is not needed
+

Alternatively, you can use the optional parameters --colmap_executable and --magick_executable to point to the respective paths. Please note that on Windows, the executable should point to the COLMAP .bat file that takes care of setting the execution environment. Once done, <location> will contain the expected COLMAP data set structure with undistorted, resized input images, in addition to your original images and some temporary (distorted) data in the directory distorted.

+

If you have your own COLMAP dataset without undistortion (e.g., using OPENCV camera), you can try to just run the last part of the script: Put the images in input and the COLMAP info in a subdirectory distorted:

+
<location>
+|---input
+|   |---<image 0>
+|   |---<image 1>
+|   |---...
+|---distorted
+    |---database.db
+    |---sparse
+        |---0
+            |---...
+
+

Then run

+
python convert.py -s <location> --skip_matching [--resize] #If not resizing, ImageMagick is not needed
+
+Command Line Arguments for convert.py +

--no_gpu

+

Flag to avoid using GPU in COLMAP.

+

--skip_matching

+

Flag to indicate that COLMAP info is available for images.

+

--source_path / -s

+

Location of the inputs.

+

--camera

+

Which camera model to use for the early matching steps, OPENCV by default.

+

--resize

+

Flag for creating resized versions of input images.

+

--colmap_executable

+

Path to the COLMAP executable (.bat on Windows).

+

--magick_executable

+

Path to the ImageMagick executable.

+
+
+

Training speed acceleration

+

We integrated the drop-in replacements from Taming-3dgs1 with fused ssim into the original codebase to speed up training times. Once installed, the accelerated rasterizer delivers a $\times$ 1.6 training time speedup using --optimizer_type default and a $\times$ 2.7 training time speedup using --optimizer_type sparse_adam.

+

To get faster training times you must first install the accelerated rasterizer to your environment:

+
pip uninstall diff-gaussian-rasterization -y
+cd submodules/diff-gaussian-rasterization
+rm -r build
+git checkout 3dgs_accel
+pip install .
+

Then you can add the following parameter to use the sparse adam optimizer when running train.py:

+
--optimizer_type sparse_adam
+

Note that this custom rasterizer has a different behaviour than the original version, for more details on training times please see stats for training times.

+

1. Mallick and Goel, et al. ‘Taming 3DGS: High-Quality Radiance Fields with Limited Resources’. SIGGRAPH Asia 2024 Conference Papers, 2024, https://doi.org/10.1145/3680528.3687694, github

+

Depth regularization

+

To have better reconstructed scenes we use depth maps as priors during optimization with each input images. It works best on untextured parts ex: roads and can remove floaters. Several papers have used similar ideas to improve various aspects of 3DGS; (e.g. DepthRegularizedGS, SparseGS, DNGaussian). The depth regularization we integrated is that used in our Hierarchical 3DGS paper, but applied to the original 3DGS; for some scenes (e.g., the DeepBlending scenes) it improves quality significantly; for others it either makes a small difference or can even be worse. For example results showing the potential benefit and statistics on quality please see here: Stats for depth regularization.

+

When training on a synthetic dataset, depth maps can be produced and they do not require further processing to be used in our method.

+

For real world datasets depth maps should be generated for each input images, to generate them please do the following:

+
    +
  1. Clone Depth Anything v2: +
    git clone https://github.com/DepthAnything/Depth-Anything-V2.git
    +
    +
  2. +
  3. Download weights from Depth-Anything-V2-Large and place it under Depth-Anything-V2/checkpoints/
  4. +
  5. Generate depth maps: +
    python Depth-Anything-V2/run.py --encoder vitl --pred-only --grayscale --img-path <path to input images> --outdir <output path>
    +
    +
  6. +
  7. Generate a depth_params.json file using: +
    python utils/make_depth_scale.py --base_dir <path to colmap> --depths_dir <path to generated depths>
    +
    +
  8. +
+

A new parameter should be set when training if you want to use depth regularization -d <path to depth maps>.

+

Exposure compensation

+

To compensate for exposure changes in the different input images we optimize an affine transformation for each image just as in Hierarchical 3dgs.

+

This can greatly improve reconstruction results for "in the wild" captures, e.g., with a smartphone when the exposure setting of the camera is not fixed. For example results showing the potential benefit and statistics on quality please see here: Stats for exposure compensation.

+

Add the following parameters to enable it:

+
--exposure_lr_init 0.001 --exposure_lr_final 0.0001 --exposure_lr_delay_steps 5000 --exposure_lr_delay_mult 0.001 --train_test_exp
+
+

Again, other excellent papers have used similar ideas e.g. NeRF-W, URF.

+

Anti-aliasing

+

We added the EWA Filter from Mip Splatting in our codebase to remove aliasing. It is disabled by default but you can enable it by adding --antialiasing when training on a scene using train.py or rendering using render.py. Antialiasing can be toggled in the SIBR viewer, it is disabled by default but you should enable it when viewing a scene trained using --antialiasing. +aa +this scene was trained using --antialiasing.

+

SIBR: Top view

+
+

Views > Top view

+
+

The Top view renders the SfM point cloud in another view with the corresponding input cameras and the Point view user camera. This allows visualization of how far the viewer is from the input cameras for example.

+

It is a 3D view so the user can navigate through it just as in the Point view (modes available: FPS, trackball, orbit).

+ + +

top view open

+

Options are available to customize this view, meshes can be disabled/enabled and their scales can be modified.

+ + +

top view options +A useful additional functionality is to move to the position of an input image, and progressively fade out to the SfM point view in that position (e.g., to verify camera alignment). Views from input cameras can be displayed in the Top view (note that --images-path must be set in the command line). One can snap the Top view camera to the closest input camera from the user camera in the Point view by clicking Top view settings > Cameras > Snap to closest.

+ + +

top view image alpha

+

OpenXR support

+

OpenXR is supported in the branch gaussian_code_release_openxr +Within that branch, you can find documentation for VR support here.

+

FAQ

+
    +
  • +

    Where do I get data sets, e.g., those referenced in full_eval.py? The MipNeRF360 data set is provided by the authors of the original paper on the project site. Note that two of the data sets cannot be openly shared and require you to consult the authors directly. For Tanks&Temples and Deep Blending, please use the download links provided at the top of the page. Alternatively, you may access the cloned data (status: August 2023!) from HuggingFace

    +
  • +
  • +

    How can I use this for a much larger dataset, like a city district? The current method was not designed for these, but given enough memory, it should work out. However, the approach can struggle in multi-scale detail scenes (extreme close-ups, mixed with far-away shots). This is usually the case in, e.g., driving data sets (cars close up, buildings far away). For such scenes, you can lower the --position_lr_init, --position_lr_final and --scaling_lr (x0.3, x0.1, ...). The more extensive the scene, the lower these values should be. Below, we use default learning rates (left) and --position_lr_init 0.000016 --scaling_lr 0.001" (right).

    +
  • +
+ + + + + + + +
Default learning rate result Reduced learning rate result
+
    +
  • +

    I'm on Windows and I can't manage to build the submodules, what do I do? Consider following the steps in the excellent video tutorial here, hopefully they should help. The order in which the steps are done is important! Alternatively, consider using the linked Colab template.

    +
  • +
  • +

    It still doesn't work. It says something about cl.exe. What do I do? User Henry Pearce found a workaround. You can you try adding the visual studio path to your environment variables (your version number might differ); +C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30133\bin\Hostx64\x64 +Then make sure you start a new conda prompt and cd to your repo location and try this;

    +
  • +
+
conda activate gaussian_splatting
+cd <dir_to_repo>/gaussian-splatting
+pip install submodules\diff-gaussian-rasterization
+pip install submodules\simple-knn
+
+
    +
  • +

    I'm on macOS/Puppy Linux/Greenhat and I can't manage to build, what do I do? Sorry, we can't provide support for platforms outside of the ones we list in this README. Consider using the linked Colab template.

    +
  • +
  • +

    I don't have 24 GB of VRAM for training, what do I do? The VRAM consumption is determined by the number of points that are being optimized, which increases over time. If you only want to train to 7k iterations, you will need significantly less. To do the full training routine and avoid running out of memory, you can increase the --densify_grad_threshold, --densification_interval or reduce the value of --densify_until_iter. Note however that this will affect the quality of the result. Also try setting --test_iterations to -1 to avoid memory spikes during testing. If --densify_grad_threshold is very high, no densification should occur and training should complete if the scene itself loads successfully.

    +
  • +
  • +

    24 GB of VRAM for reference quality training is still a lot! Can't we do it with less? Yes, most likely. By our calculations it should be possible with way less memory (~8GB). If we can find the time we will try to achieve this. If some PyTorch veteran out there wants to tackle this, we look forward to your pull request!

    +
  • +
  • +

    How can I use the differentiable Gaussian rasterizer for my own project? Easy, it is included in this repo as a submodule diff-gaussian-rasterization. Feel free to check out and install the package. It's not really documented, but using it from the Python side is very straightforward (cf. gaussian_renderer/__init__.py).

    +
  • +
  • +

    Wait, but <insert feature> isn't optimized and could be much better? There are several parts we didn't even have time to think about improving (yet). The performance you get with this prototype is probably a rather slow baseline for what is physically possible.

    +
  • +
  • +

    Something is broken, how did this happen? We tried hard to provide a solid and comprehensible basis to make use of the paper's method. We have refactored the code quite a bit, but we have limited capacity to test all possible usage scenarios. Thus, if part of the website, the code or the performance is lacking, please create an issue. If we find the time, we will do our best to address it.

    +
  • +
+
+ +
+
+
+
+

About

+ +

+ Original reference implementation of "3D Gaussian Splatting for Real-Time Radiance Field Rendering" +

+ + +

Topics

+ + +

Resources

+ + + +

License

+ + + + + + + + + + + + + + + + +

Stars

+ + +

Watchers

+ + +

Forks

+ + + + +
+ +
+
+ + +
+
+

+ Releases

+ +
No releases published
+ +
+
+ + + + +
+
+ + + +

+ Packages +

+ + +
+
+
 
+
+
+
+
 
+
+
+
+
 
+
+ + + + +
+
+
+ + + + + +
+
+ + +

+ Contributors

+ +
    +
  • +
    +
  • +
  • +
    +
  • +
  • +
    +
  • +
+ + +
+
+
+ + +
+
+

Languages

+
+ + +
+ + +
+
+ + +
+
+
+ + + + +
+ +
+ +
+
+ +
+ + + + + + + + + + + + + + + + + + + + + + +
+
+
+ + + diff --git a/research/gpu-fusion-splatting/mirrors/graphdeco-gaussian-splatting-readme.md b/research/gpu-fusion-splatting/mirrors/graphdeco-gaussian-splatting-readme.md new file mode 100644 index 0000000..eb2cf6c --- /dev/null +++ b/research/gpu-fusion-splatting/mirrors/graphdeco-gaussian-splatting-readme.md @@ -0,0 +1,615 @@ +# 3D Gaussian Splatting for Real-Time Radiance Field Rendering +Bernhard Kerbl*, Georgios Kopanas*, Thomas Leimkühler, George Drettakis (* indicates equal contribution)
+| [Webpage](https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/) | [Full Paper](https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/3d_gaussian_splatting_high.pdf) | [Video](https://youtu.be/T_kXY43VZnk) | [Other GRAPHDECO Publications](http://www-sop.inria.fr/reves/publis/gdindex.php) | [FUNGRAPH project page](https://fungraph.inria.fr) |
+| [T&T+DB COLMAP (650MB)](https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/datasets/input/tandt_db.zip) | [Pre-trained Models (14 GB)](https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/datasets/pretrained/models.zip) | [Viewers for Windows (60MB)](https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/binaries/viewers.zip) | [Evaluation Images (7 GB)](https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/evaluation/images.zip) |
+![Teaser image](assets/teaser.png) + +This repository contains the official authors implementation associated with the paper "3D Gaussian Splatting for Real-Time Radiance Field Rendering", which can be found [here](https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/). We further provide the reference images used to create the error metrics reported in the paper, as well as recently created, pre-trained models. + + + + + + +Abstract: *Radiance Field methods have recently revolutionized novel-view synthesis of scenes captured with multiple photos or videos. However, achieving high visual quality still requires neural networks that are costly to train and render, while recent faster methods inevitably trade off speed for quality. For unbounded and complete scenes (rather than isolated objects) and 1080p resolution rendering, no current method can achieve real-time display rates. We introduce three key elements that allow us to achieve state-of-the-art visual quality while maintaining competitive training times and importantly allow high-quality real-time (≥ 30 fps) novel-view synthesis at 1080p resolution. First, starting from sparse points produced during camera calibration, we represent the scene with 3D Gaussians that preserve desirable properties of continuous volumetric radiance fields for scene optimization while avoiding unnecessary computation in empty space; Second, we perform interleaved optimization/density control of the 3D Gaussians, notably optimizing anisotropic covariance to achieve an accurate representation of the scene; Third, we develop a fast visibility-aware rendering algorithm that supports anisotropic splatting and both accelerates training and allows realtime rendering. We demonstrate state-of-the-art visual quality and real-time rendering on several established datasets.* + +
+
+

BibTeX

+
@Article{kerbl3Dgaussians,
+      author       = {Kerbl, Bernhard and Kopanas, Georgios and Leimk{\"u}hler, Thomas and Drettakis, George},
+      title        = {3D Gaussian Splatting for Real-Time Radiance Field Rendering},
+      journal      = {ACM Transactions on Graphics},
+      number       = {4},
+      volume       = {42},
+      month        = {July},
+      year         = {2023},
+      url          = {https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/}
+}
+
+
+ + + +## Funding and Acknowledgments + +This research was funded by the ERC Advanced grant FUNGRAPH No 788065. The authors are grateful to Adobe for generous donations, the OPAL infrastructure from Université Côte d’Azur and for the HPC resources from GENCI–IDRIS (Grant 2022-AD011013409). The authors thank the anonymous reviewers for their valuable feedback, P. Hedman and A. Tewari for proofreading earlier drafts also T. Müller, A. Yu and S. Fridovich-Keil for helping with the comparisons. + +## NEW FEATURES ! + +We have limited resources for maintaining and updating the code. However, we have added a few new features since the original release that are inspired by some of the excellent work many other researchers have been doing on 3DGS. We will be adding other features within the ability of our resources. + +**Update of October 2024**: We integrated [training speed acceleration](#training-speed-acceleration) and made it compatible with [depth regularization](#depth-regularization), [anti-aliasing](#anti-aliasing) and [exposure compensation](#exposure-compensation). We have enhanced the SIBR real time viewer by correcting bugs and adding features in the [Top View](#sibr-top-view) that allows visualization of input and user cameras. + +**Update of Spring 2024**: +Orange Labs has kindly added [OpenXR support](#openxr-support) for VR viewing. + +## Step-by-step Tutorial + +Jonathan Stephens made a fantastic step-by-step tutorial for setting up Gaussian Splatting on your machine, along with instructions for creating usable datasets from videos. If the instructions below are too dry for you, go ahead and check it out [here](https://www.youtube.com/watch?v=UXtuigy_wYc). + +## Colab + +User [camenduru](https://github.com/camenduru) was kind enough to provide a Colab template that uses this repo's source (status: August 2023!) for quick and easy access to the method. Please check it out [here](https://github.com/camenduru/gaussian-splatting-colab). + +## Cloning the Repository + +The repository contains submodules, thus please check it out with +```shell +# SSH +git clone git@github.com:graphdeco-inria/gaussian-splatting.git --recursive +``` +or +```shell +# HTTPS +git clone https://github.com/graphdeco-inria/gaussian-splatting --recursive +``` + +## Overview + +The codebase has 4 main components: +- A PyTorch-based optimizer to produce a 3D Gaussian model from SfM inputs +- A network viewer that allows to connect to and visualize the optimization process +- An OpenGL-based real-time viewer to render trained models in real-time. +- A script to help you turn your own images into optimization-ready SfM data sets + +The components have different requirements w.r.t. both hardware and software. They have been tested on Windows 10 and Ubuntu Linux 22.04. Instructions for setting up and running each of them are found in the sections below. + + + + +## Optimizer + +The optimizer uses PyTorch and CUDA extensions in a Python environment to produce trained models. + +### Hardware Requirements + +- CUDA-ready GPU with Compute Capability 7.0+ +- 24 GB VRAM (to train to paper evaluation quality) +- Please see FAQ for smaller VRAM configurations + +### Software Requirements +- Conda (recommended for easy setup) +- C++ Compiler for PyTorch extensions (we used Visual Studio 2019 for Windows) +- CUDA SDK 11 for PyTorch extensions, install *after* Visual Studio (we used 11.8, **known issues with 11.6**) +- C++ Compiler and CUDA SDK must be compatible + +### Setup + +#### Local Setup + +Our default, provided install method is based on Conda package and environment management: +```shell +SET DISTUTILS_USE_SDK=1 # Windows only +conda env create --file environment.yml +conda activate gaussian_splatting +``` +Please note that this process assumes that you have CUDA SDK **11** installed, not **12**. For modifications, see below. + +Tip: Downloading packages and creating a new environment with Conda can require a significant amount of disk space. By default, Conda will use the main system hard drive. You can avoid this by specifying a different package download location and an environment on a different drive: + +```shell +conda config --add pkgs_dirs / +conda env create --file environment.yml --prefix //gaussian_splatting +conda activate //gaussian_splatting +``` + +#### Modifications + +If you can afford the disk space, we recommend using our environment files for setting up a training environment identical to ours. If you want to make modifications, please note that major version changes might affect the results of our method. However, our (limited) experiments suggest that the codebase works just fine inside a more up-to-date environment (Python 3.8, PyTorch 2.0.0, CUDA 12). Make sure to create an environment where PyTorch and its CUDA runtime version match and the installed CUDA SDK has no major version difference with PyTorch's CUDA version. + +#### Known Issues + +Some users experience problems building the submodules on Windows (```cl.exe: File not found``` or similar). Please consider the workaround for this problem from the FAQ. + +### Running + +To run the optimizer, simply use + +```shell +python train.py -s +``` + +
+Command Line Arguments for train.py + + #### --source_path / -s + Path to the source directory containing a COLMAP or Synthetic NeRF data set. + #### --model_path / -m + Path where the trained model should be stored (```output/``` by default). + #### --images / -i + Alternative subdirectory for COLMAP images (```images``` by default). + #### --eval + Add this flag to use a MipNeRF360-style training/test split for evaluation. + #### --resolution / -r + Specifies resolution of the loaded images before training. If provided ```1, 2, 4``` or ```8```, uses original, 1/2, 1/4 or 1/8 resolution, respectively. For all other values, rescales the width to the given number while maintaining image aspect. **If not set and input image width exceeds 1.6K pixels, inputs are automatically rescaled to this target.** + #### --data_device + Specifies where to put the source image data, ```cuda``` by default, recommended to use ```cpu``` if training on large/high-resolution dataset, will reduce VRAM consumption, but slightly slow down training. Thanks to [HrsPythonix](https://github.com/HrsPythonix). + #### --white_background / -w + Add this flag to use white background instead of black (default), e.g., for evaluation of NeRF Synthetic dataset. + #### --sh_degree + Order of spherical harmonics to be used (no larger than 3). ```3``` by default. + #### --convert_SHs_python + Flag to make pipeline compute forward and backward of SHs with PyTorch instead of ours. + #### --convert_cov3D_python + Flag to make pipeline compute forward and backward of the 3D covariance with PyTorch instead of ours. + #### --debug + Enables debug mode if you experience erros. If the rasterizer fails, a ```dump``` file is created that you may forward to us in an issue so we can take a look. + #### --debug_from + Debugging is **slow**. You may specify an iteration (starting from 0) after which the above debugging becomes active. + #### --iterations + Number of total iterations to train for, ```30_000``` by default. + #### --ip + IP to start GUI server on, ```127.0.0.1``` by default. + #### --port + Port to use for GUI server, ```6009``` by default. + #### --test_iterations + Space-separated iterations at which the training script computes L1 and PSNR over test set, ```7000 30000``` by default. + #### --save_iterations + Space-separated iterations at which the training script saves the Gaussian model, ```7000 30000 ``` by default. + #### --checkpoint_iterations + Space-separated iterations at which to store a checkpoint for continuing later, saved in the model directory. + #### --start_checkpoint + Path to a saved checkpoint to continue training from. + #### --quiet + Flag to omit any text written to standard out pipe. + #### --feature_lr + Spherical harmonics features learning rate, ```0.0025``` by default. + #### --opacity_lr + Opacity learning rate, ```0.05``` by default. + #### --scaling_lr + Scaling learning rate, ```0.005``` by default. + #### --rotation_lr + Rotation learning rate, ```0.001``` by default. + #### --position_lr_max_steps + Number of steps (from 0) where position learning rate goes from ```initial``` to ```final```. ```30_000``` by default. + #### --position_lr_init + Initial 3D position learning rate, ```0.00016``` by default. + #### --position_lr_final + Final 3D position learning rate, ```0.0000016``` by default. + #### --position_lr_delay_mult + Position learning rate multiplier (cf. Plenoxels), ```0.01``` by default. + #### --densify_from_iter + Iteration where densification starts, ```500``` by default. + #### --densify_until_iter + Iteration where densification stops, ```15_000``` by default. + #### --densify_grad_threshold + Limit that decides if points should be densified based on 2D position gradient, ```0.0002``` by default. + #### --densification_interval + How frequently to densify, ```100``` (every 100 iterations) by default. + #### --opacity_reset_interval + How frequently to reset opacity, ```3_000``` by default. + #### --lambda_dssim + Influence of SSIM on total loss from 0 to 1, ```0.2``` by default. + #### --percent_dense + Percentage of scene extent (0--1) a point must exceed to be forcibly densified, ```0.01``` by default. + +
+
+ +Note that similar to MipNeRF360, we target images at resolutions in the 1-1.6K pixel range. For convenience, arbitrary-size inputs can be passed and will be automatically resized if their width exceeds 1600 pixels. We recommend to keep this behavior, but you may force training to use your higher-resolution images by setting ```-r 1```. + +The MipNeRF360 scenes are hosted by the paper authors [here](https://jonbarron.info/mipnerf360/). You can find our SfM data sets for Tanks&Temples and Deep Blending [here](https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/datasets/input/tandt_db.zip). If you do not provide an output model directory (```-m```), trained models are written to folders with randomized unique names inside the ```output``` directory. At this point, the trained models may be viewed with the real-time viewer (see further below). + +### Evaluation +By default, the trained models use all available images in the dataset. To train them while withholding a test set for evaluation, use the ```--eval``` flag. This way, you can render training/test sets and produce error metrics as follows: +```shell +python train.py -s --eval # Train with train/test split +python render.py -m # Generate renderings +python metrics.py -m # Compute error metrics on renderings +``` + +If you want to evaluate our [pre-trained models](https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/datasets/pretrained/models.zip), you will have to download the corresponding source data sets and indicate their location to ```render.py``` with an additional ```--source_path/-s``` flag. Note: The pre-trained models were created with the release codebase. This code base has been cleaned up and includes bugfixes, hence the metrics you get from evaluating them will differ from those in the paper. +```shell +python render.py -m -s +python metrics.py -m +``` + +
+Command Line Arguments for render.py + + #### --model_path / -m + Path to the trained model directory you want to create renderings for. + #### --skip_train + Flag to skip rendering the training set. + #### --skip_test + Flag to skip rendering the test set. + #### --quiet + Flag to omit any text written to standard out pipe. + + **The below parameters will be read automatically from the model path, based on what was used for training. However, you may override them by providing them explicitly on the command line.** + + #### --source_path / -s + Path to the source directory containing a COLMAP or Synthetic NeRF data set. + #### --images / -i + Alternative subdirectory for COLMAP images (```images``` by default). + #### --eval + Add this flag to use a MipNeRF360-style training/test split for evaluation. + #### --resolution / -r + Changes the resolution of the loaded images before training. If provided ```1, 2, 4``` or ```8```, uses original, 1/2, 1/4 or 1/8 resolution, respectively. For all other values, rescales the width to the given number while maintaining image aspect. ```1``` by default. + #### --white_background / -w + Add this flag to use white background instead of black (default), e.g., for evaluation of NeRF Synthetic dataset. + #### --convert_SHs_python + Flag to make pipeline render with computed SHs from PyTorch instead of ours. + #### --convert_cov3D_python + Flag to make pipeline render with computed 3D covariance from PyTorch instead of ours. + +
+ +
+Command Line Arguments for metrics.py + + #### --model_paths / -m + Space-separated list of model paths for which metrics should be computed. +
+
+ +We further provide the ```full_eval.py``` script. This script specifies the routine used in our evaluation and demonstrates the use of some additional parameters, e.g., ```--images (-i)``` to define alternative image directories within COLMAP data sets. If you have downloaded and extracted all the training data, you can run it like this: +```shell +python full_eval.py -m360 -tat -db +``` +In the current version, this process takes about 7h on our reference machine containing an A6000. If you want to do the full evaluation on our pre-trained models, you can specify their download location and skip training. +```shell +python full_eval.py -o --skip_training -m360 -tat -db +``` + +If you want to compute the metrics on our paper's [evaluation images](https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/evaluation/images.zip), you can also skip rendering. In this case it is not necessary to provide the source datasets. You can compute metrics for multiple image sets at a time. +```shell +python full_eval.py -m /garden ... --skip_training --skip_rendering +``` + +
+Command Line Arguments for full_eval.py + + #### --skip_training + Flag to skip training stage. + #### --skip_rendering + Flag to skip rendering stage. + #### --skip_metrics + Flag to skip metrics calculation stage. + #### --output_path + Directory to put renderings and results in, ```./eval``` by default, set to pre-trained model location if evaluating them. + #### --mipnerf360 / -m360 + Path to MipNeRF360 source datasets, required if training or rendering. + #### --tanksandtemples / -tat + Path to Tanks&Temples source datasets, required if training or rendering. + #### --deepblending / -db + Path to Deep Blending source datasets, required if training or rendering. +
+
+ +## Interactive Viewers +We provide two interactive viewers for our method: remote and real-time. Our viewing solutions are based on the [SIBR](https://sibr.gitlabpages.inria.fr/) framework, developed by the GRAPHDECO group for several novel-view synthesis projects. + +### Hardware Requirements +- OpenGL 4.5-ready GPU and drivers (or latest MESA software) +- 4 GB VRAM recommended +- CUDA-ready GPU with Compute Capability 7.0+ (only for Real-Time Viewer) + +### Software Requirements +- Visual Studio or g++, **not Clang** (we used Visual Studio 2019 for Windows) +- CUDA SDK 11, install *after* Visual Studio (we used 11.8) +- CMake (recent version, we used 3.24) +- 7zip (only on Windows) + +### Pre-built Windows Binaries +We provide pre-built binaries for Windows [here](https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/binaries/viewers.zip). We recommend using them on Windows for an efficient setup, since the building of SIBR involves several external dependencies that must be downloaded and compiled on-the-fly. + +### Installation from Source +If you cloned with submodules (e.g., using ```--recursive```), the source code for the viewers is found in ```SIBR_viewers```. The network viewer runs within the SIBR framework for Image-based Rendering applications. + +#### Windows +CMake should take care of your dependencies. +```shell +cd SIBR_viewers +cmake -Bbuild . +cmake --build build --target install --config RelWithDebInfo +``` +You may specify a different configuration, e.g. ```Debug``` if you need more control during development. + +#### Ubuntu 22.04 +You will need to install a few dependencies before running the project setup. +```shell +# Dependencies +sudo apt install -y libglew-dev libassimp-dev libboost-all-dev libgtk-3-dev libopencv-dev libglfw3-dev libavdevice-dev libavcodec-dev libeigen3-dev libxxf86vm-dev libembree-dev +# Project setup +cd SIBR_viewers +cmake -Bbuild . -DCMAKE_BUILD_TYPE=Release # add -G Ninja to build faster +cmake --build build -j24 --target install +``` + +#### Ubuntu 20.04 +Backwards compatibility with Focal Fossa is not fully tested, but building SIBR with CMake should still work after invoking +```shell +git checkout fossa_compatibility +``` + +### Navigation in SIBR Viewers +The SIBR interface provides several methods of navigating the scene. By default, you will be started with an FPS navigator, which you can control with ```W, A, S, D, Q, E``` for camera translation and ```I, K, J, L, U, O``` for rotation. Alternatively, you may want to use a Trackball-style navigator (select from the floating menu). You can also snap to a camera from the data set with the ```Snap to``` button or find the closest camera with ```Snap to closest```. The floating menues also allow you to change the navigation speed. You can use the ```Scaling Modifier``` to control the size of the displayed Gaussians, or show the initial point cloud. + +### Running the Network Viewer + + + +https://github.com/graphdeco-inria/gaussian-splatting/assets/40643808/90a2e4d3-cf2e-4633-b35f-bfe284e28ff7 + + + +After extracting or installing the viewers, you may run the compiled ```SIBR_remoteGaussian_app[_config]``` app in ```/bin```, e.g.: +```shell +.//bin/SIBR_remoteGaussian_app +``` +The network viewer allows you to connect to a running training process on the same or a different machine. If you are training on the same machine and OS, no command line parameters should be required: the optimizer communicates the location of the training data to the network viewer. By default, optimizer and network viewer will try to establish a connection on **localhost** on port **6009**. You can change this behavior by providing matching ```--ip``` and ```--port``` parameters to both the optimizer and the network viewer. If for some reason the path used by the optimizer to find the training data is not reachable by the network viewer (e.g., due to them running on different (virtual) machines), you may specify an override location to the viewer by using ```-s ```. + +
+Primary Command Line Arguments for Network Viewer + + #### --path / -s + Argument to override model's path to source dataset. + #### --ip + IP to use for connection to a running training script. + #### --port + Port to use for connection to a running training script. + #### --rendering-size + Takes two space separated numbers to define the resolution at which network rendering occurs, ```1200``` width by default. + Note that to enforce an aspect that differs from the input images, you need ```--force-aspect-ratio``` too. + #### --load_images + Flag to load source dataset images to be displayed in the top view for each camera. +
+
+ +### Running the Real-Time Viewer + + + + +https://github.com/graphdeco-inria/gaussian-splatting/assets/40643808/0940547f-1d82-4c2f-a616-44eabbf0f816 + + + + +After extracting or installing the viewers, you may run the compiled ```SIBR_gaussianViewer_app[_config]``` app in ```/bin```, e.g.: +```shell +.//bin/SIBR_gaussianViewer_app -m +``` + +It should suffice to provide the ```-m``` parameter pointing to a trained model directory. Alternatively, you can specify an override location for training input data using ```-s```. To use a specific resolution other than the auto-chosen one, specify ```--rendering-size ```. Combine it with ```--force-aspect-ratio``` if you want the exact resolution and don't mind image distortion. + +**To unlock the full frame rate, please disable V-Sync on your machine and also in the application (Menu → Display). In a multi-GPU system (e.g., laptop) your OpenGL/Display GPU should be the same as your CUDA GPU (e.g., by setting the application's GPU preference on Windows, see below) for maximum performance.** + +![Teaser image](assets/select.png) + +In addition to the initial point cloud and the splats, you also have the option to visualize the Gaussians by rendering them as ellipsoids from the floating menu. +SIBR has many other functionalities, please see the [documentation](https://sibr.gitlabpages.inria.fr/) for more details on the viewer, navigation options etc. There is also a Top View (available from the menu) that shows the placement of the input cameras and the original SfM point cloud; please note that Top View slows rendering when enabled. The real-time viewer also uses slightly more aggressive, fast culling, which can be toggled in the floating menu. If you ever encounter an issue that can be solved by turning fast culling off, please let us know. + +
+Primary Command Line Arguments for Real-Time Viewer + + #### --model-path / -m + Path to trained model. + #### --iteration + Specifies which of state to load if multiple are available. Defaults to latest available iteration. + #### --path / -s + Argument to override model's path to source dataset. + #### --rendering-size + Takes two space separated numbers to define the resolution at which real-time rendering occurs, ```1200``` width by default. Note that to enforce an aspect that differs from the input images, you need ```--force-aspect-ratio``` too. + #### --load_images + Flag to load source dataset images to be displayed in the top view for each camera. + #### --device + Index of CUDA device to use for rasterization if multiple are available, ```0``` by default. + #### --no_interop + Disables CUDA/GL interop forcibly. Use on systems that may not behave according to spec (e.g., WSL2 with MESA GL 4.5 software rendering). +
+
+ +## Processing your own Scenes + +Our COLMAP loaders expect the following dataset structure in the source path location: + +``` + +|---images +| |--- +| |--- +| |---... +|---sparse + |---0 + |---cameras.bin + |---images.bin + |---points3D.bin +``` + +For rasterization, the camera models must be either a SIMPLE_PINHOLE or PINHOLE camera. We provide a converter script ```convert.py```, to extract undistorted images and SfM information from input images. Optionally, you can use ImageMagick to resize the undistorted images. This rescaling is similar to MipNeRF360, i.e., it creates images with 1/2, 1/4 and 1/8 the original resolution in corresponding folders. To use them, please first install a recent version of COLMAP (ideally CUDA-powered) and ImageMagick. Put the images you want to use in a directory ```/input```. +``` + +|---input + |--- + |--- + |---... +``` + If you have COLMAP and ImageMagick on your system path, you can simply run +```shell +python convert.py -s [--resize] #If not resizing, ImageMagick is not needed +``` +Alternatively, you can use the optional parameters ```--colmap_executable``` and ```--magick_executable``` to point to the respective paths. Please note that on Windows, the executable should point to the COLMAP ```.bat``` file that takes care of setting the execution environment. Once done, `````` will contain the expected COLMAP data set structure with undistorted, resized input images, in addition to your original images and some temporary (distorted) data in the directory ```distorted```. + +If you have your own COLMAP dataset without undistortion (e.g., using ```OPENCV``` camera), you can try to just run the last part of the script: Put the images in ```input``` and the COLMAP info in a subdirectory ```distorted```: +``` + +|---input +| |--- +| |--- +| |---... +|---distorted + |---database.db + |---sparse + |---0 + |---... +``` +Then run +```shell +python convert.py -s --skip_matching [--resize] #If not resizing, ImageMagick is not needed +``` + +
+Command Line Arguments for convert.py + + #### --no_gpu + Flag to avoid using GPU in COLMAP. + #### --skip_matching + Flag to indicate that COLMAP info is available for images. + #### --source_path / -s + Location of the inputs. + #### --camera + Which camera model to use for the early matching steps, ```OPENCV``` by default. + #### --resize + Flag for creating resized versions of input images. + #### --colmap_executable + Path to the COLMAP executable (```.bat``` on Windows). + #### --magick_executable + Path to the ImageMagick executable. +
+
+ +### Training speed acceleration + +We integrated the drop-in replacements from [Taming-3dgs](https://humansensinglab.github.io/taming-3dgs/)1 with [fused ssim](https://github.com/rahul-goel/fused-ssim/tree/main) into the original codebase to speed up training times. Once installed, the accelerated rasterizer delivers a **$\times$ 1.6 training time speedup** using `--optimizer_type default` and a **$\times$ 2.7 training time speedup** using `--optimizer_type sparse_adam`. + +To get faster training times you must first install the accelerated rasterizer to your environment: + +```bash +pip uninstall diff-gaussian-rasterization -y +cd submodules/diff-gaussian-rasterization +rm -r build +git checkout 3dgs_accel +pip install . +``` + +Then you can add the following parameter to use the sparse adam optimizer when running `train.py`: + +```bash +--optimizer_type sparse_adam +``` + +*Note that this custom rasterizer has a different behaviour than the original version, for more details on training times please see [stats for training times](results.md/#training-times-comparisons)*. + +*1. Mallick and Goel, et al. ‘Taming 3DGS: High-Quality Radiance Fields with Limited Resources’. SIGGRAPH Asia 2024 Conference Papers, 2024, https://doi.org/10.1145/3680528.3687694, [github](https://github.com/humansensinglab/taming-3dgs)* + + +### Depth regularization + +To have better reconstructed scenes we use depth maps as priors during optimization with each input images. It works best on untextured parts ex: roads and can remove floaters. Several papers have used similar ideas to improve various aspects of 3DGS; (e.g. [DepthRegularizedGS](https://robot0321.github.io/DepthRegGS/index.html), [SparseGS](https://formycat.github.io/SparseGS-Real-Time-360-Sparse-View-Synthesis-using-Gaussian-Splatting/), [DNGaussian](https://fictionarry.github.io/DNGaussian/)). The depth regularization we integrated is that used in our [Hierarchical 3DGS](https://repo-sam.inria.fr/fungraph/hierarchical-3d-gaussians/) paper, but applied to the original 3DGS; for some scenes (e.g., the DeepBlending scenes) it improves quality significantly; for others it either makes a small difference or can even be worse. For example results showing the potential benefit and statistics on quality please see here: [Stats for depth regularization](results.md). + +When training on a synthetic dataset, depth maps can be produced and they do not require further processing to be used in our method. + +For real world datasets depth maps should be generated for each input images, to generate them please do the following: +1. Clone [Depth Anything v2](https://github.com/DepthAnything/Depth-Anything-V2?tab=readme-ov-file#usage): + ``` + git clone https://github.com/DepthAnything/Depth-Anything-V2.git + ``` +2. Download weights from [Depth-Anything-V2-Large](https://huggingface.co/depth-anything/Depth-Anything-V2-Large/resolve/main/depth_anything_v2_vitl.pth?download=true) and place it under `Depth-Anything-V2/checkpoints/` +3. Generate depth maps: + ``` + python Depth-Anything-V2/run.py --encoder vitl --pred-only --grayscale --img-path --outdir + ``` +5. Generate a `depth_params.json` file using: + ``` + python utils/make_depth_scale.py --base_dir --depths_dir + ``` + +A new parameter should be set when training if you want to use depth regularization `-d `. + +### Exposure compensation +To compensate for exposure changes in the different input images we optimize an affine transformation for each image just as in [Hierarchical 3dgs](https://repo-sam.inria.fr/fungraph/hierarchical-3d-gaussians/). + +This can greatly improve reconstruction results for "in the wild" captures, e.g., with a smartphone when the exposure setting of the camera is not fixed. For example results showing the potential benefit and statistics on quality please see here: [Stats for exposure compensation](results.md). + +Add the following parameters to enable it: +``` +--exposure_lr_init 0.001 --exposure_lr_final 0.0001 --exposure_lr_delay_steps 5000 --exposure_lr_delay_mult 0.001 --train_test_exp +``` +Again, other excellent papers have used similar ideas e.g. [NeRF-W](https://nerf-w.github.io/), [URF](https://urban-radiance-fields.github.io/). + +### Anti-aliasing +We added the EWA Filter from [Mip Splatting](https://niujinshuchong.github.io/mip-splatting/) in our codebase to remove aliasing. It is disabled by default but you can enable it by adding `--antialiasing` when training on a scene using `train.py` or rendering using `render.py`. Antialiasing can be toggled in the SIBR viewer, it is disabled by default but you should enable it when viewing a scene trained using `--antialiasing`. +![aa](/assets/aa_onoff.gif) +*this scene was trained using `--antialiasing`*. + +### SIBR: Top view +> `Views > Top view` + +The `Top view` renders the SfM point cloud in another view with the corresponding input cameras and the `Point view` user camera. This allows visualization of how far the viewer is from the input cameras for example. + +It is a 3D view so the user can navigate through it just as in the `Point view` (modes available: FPS, trackball, orbit). + + +![top view open](assets/top_view_open.gif) + +Options are available to customize this view, meshes can be disabled/enabled and their scales can be modified. + + +![top view options](assets/top_view_options.gif) +A useful additional functionality is to move to the position of an input image, and progressively fade out to the SfM point view in that position (e.g., to verify camera alignment). Views from input cameras can be displayed in the `Top view` (*note that `--images-path` must be set in the command line*). One can snap the `Top view` camera to the closest input camera from the user camera in the `Point view` by clicking `Top view settings > Cameras > Snap to closest`. + + +![top view image alpha](assets/top_view_image_alpha.gif) + +### OpenXR support + +OpenXR is supported in the branch `gaussian_code_release_openxr` +Within that branch, you can find documentation for VR support [here](https://gitlab.inria.fr/sibr/sibr_core/-/tree/gaussian_code_release_openxr?ref_type=heads). + + +## FAQ +- *Where do I get data sets, e.g., those referenced in ```full_eval.py```?* The MipNeRF360 data set is provided by the authors of the original paper on the project site. Note that two of the data sets cannot be openly shared and require you to consult the authors directly. For Tanks&Temples and Deep Blending, please use the download links provided at the top of the page. Alternatively, you may access the cloned data (status: August 2023!) from [HuggingFace](https://huggingface.co/camenduru/gaussian-splatting) + + +- *How can I use this for a much larger dataset, like a city district?* The current method was not designed for these, but given enough memory, it should work out. However, the approach can struggle in multi-scale detail scenes (extreme close-ups, mixed with far-away shots). This is usually the case in, e.g., driving data sets (cars close up, buildings far away). For such scenes, you can lower the ```--position_lr_init```, ```--position_lr_final``` and ```--scaling_lr``` (x0.3, x0.1, ...). The more extensive the scene, the lower these values should be. Below, we use default learning rates (left) and ```--position_lr_init 0.000016 --scaling_lr 0.001"``` (right). + +| ![Default learning rate result](assets/worse.png "title-1") | ![Reduced learning rate result](assets/better.png "title-2") | +| --- | --- | + +- *I'm on Windows and I can't manage to build the submodules, what do I do?* Consider following the steps in the excellent video tutorial [here](https://www.youtube.com/watch?v=UXtuigy_wYc), hopefully they should help. The order in which the steps are done is important! Alternatively, consider using the linked Colab template. + +- *It still doesn't work. It says something about ```cl.exe```. What do I do?* User Henry Pearce found a workaround. You can you try adding the visual studio path to your environment variables (your version number might differ); +```C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30133\bin\Hostx64\x64``` +Then make sure you start a new conda prompt and cd to your repo location and try this; +``` +conda activate gaussian_splatting +cd /gaussian-splatting +pip install submodules\diff-gaussian-rasterization +pip install submodules\simple-knn +``` + +- *I'm on macOS/Puppy Linux/Greenhat and I can't manage to build, what do I do?* Sorry, we can't provide support for platforms outside of the ones we list in this README. Consider using the linked Colab template. + +- *I don't have 24 GB of VRAM for training, what do I do?* The VRAM consumption is determined by the number of points that are being optimized, which increases over time. If you only want to train to 7k iterations, you will need significantly less. To do the full training routine and avoid running out of memory, you can increase the ```--densify_grad_threshold```, ```--densification_interval``` or reduce the value of ```--densify_until_iter```. Note however that this will affect the quality of the result. Also try setting ```--test_iterations``` to ```-1``` to avoid memory spikes during testing. If ```--densify_grad_threshold``` is very high, no densification should occur and training should complete if the scene itself loads successfully. + +- *24 GB of VRAM for reference quality training is still a lot! Can't we do it with less?* Yes, most likely. By our calculations it should be possible with **way** less memory (~8GB). If we can find the time we will try to achieve this. If some PyTorch veteran out there wants to tackle this, we look forward to your pull request! + + +- *How can I use the differentiable Gaussian rasterizer for my own project?* Easy, it is included in this repo as a submodule ```diff-gaussian-rasterization```. Feel free to check out and install the package. It's not really documented, but using it from the Python side is very straightforward (cf. ```gaussian_renderer/__init__.py```). + +- *Wait, but `````` isn't optimized and could be much better?* There are several parts we didn't even have time to think about improving (yet). The performance you get with this prototype is probably a rather slow baseline for what is physically possible. + +- *Something is broken, how did this happen?* We tried hard to provide a solid and comprehensible basis to make use of the paper's method. We have refactored the code quite a bit, but we have limited capacity to test all possible usage scenarios. Thus, if part of the website, the code or the performance is lacking, please create an issue. If we find the time, we will do our best to address it. diff --git a/research/gpu-fusion-splatting/mirrors/gsplat-docs.html b/research/gpu-fusion-splatting/mirrors/gsplat-docs.html new file mode 100644 index 0000000..464f50b --- /dev/null +++ b/research/gpu-fusion-splatting/mirrors/gsplat-docs.html @@ -0,0 +1,493 @@ + + + + + + + + + gsplat documentation + + + + + + + + + + + + + + + + Contents + + + + + + Menu + + + + + + + + Expand + + + + + + Light mode + + + + + + + + + + + + + + Dark mode + + + + + + + Auto light/dark, in light mode + + + + + + + + + + + + + + + Auto light/dark, in dark mode + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +Skip to content + + + +
+
+
+ +
+ +
+
+ +
+ +
+
+ +
+
+
+ + + + + Back to top + +
+ +
+ +
+ +
+
+
+

gsplat

+Example training image + +
+

Overview

+

gsplat is an open-source library for CUDA-accelerated differentiable rasterization of +3D gaussians with Python bindings. It is inspired by the SIGGRAPH paper “3D Gaussian Splatting for +Real-Time Rendering of Radiance Fields” [1], but we’ve made gsplat even +faster, more memory efficient, and with a growing list of new features!

+
    +
  • gsplat is developed with efficiency in mind. Comparing to the official implementation, +gsplat enables up to 4x less training memory footprint, and up to 15% less training time on Mip-NeRF 360 captures, and potential more on larger scenes. See Evaluation for details.

  • +
  • gsplat is designed to support extremely large scene rendering, which is magnitudes +faster than the official CUDA backend diff-gaussian-rasterization. See Render a Large Scene for an example.

  • +
  • gsplat offers many extra features, including batch rasterization, +N-D feature rendering, depth rendering, sparse gradient, +multi-GPU distributed rasterization +etc. See Rasterization for details.

  • +
  • gsplat is equipped with the latest and greatest 3D Gaussian Splatting techniques, +including absgrad, +anti-aliasing, +3DGS-MCMC etc. And more to come!

  • +
+
+ +
+
+

Installation

+

gsplat is available on PyPI and can be installed with pip:

+
pip install gsplat
+
+
+

To get the latest features, it can also be installed from source:

+
pip install git+https://github.com/nerfstudio-project/gsplat
+
+
+
+
+

Contributing

+

This repository was born from the curiosity of people on the Nerfstudio team trying to +understand a new rendering technique. We welcome contributions of any kind and are open +to feedback, bug-reports, and improvements to help expand the capabilities of this software.

+

This project is developed by the contributors coming from following institutes (unordered):

+
    +
  • UC Berkeley

  • +
  • NVIDIA

  • +
  • ShanghaiTech University

  • +
  • Amazon

  • +
  • Meta

  • +
  • IIIT

  • +
  • LumaAI

  • +
  • SpectacularAI

  • +
  • Aalto University

  • +
  • CMU

  • +
+
+ +
+

Citations

+
+
+
+[1] +

Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics, July 2023. URL: https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/.

+
+
+
+
+
+ +
+
+ +
+ +
+
+ + + + + + \ No newline at end of file diff --git a/research/gpu-fusion-splatting/mirrors/gsplat-github.html b/research/gpu-fusion-splatting/mirrors/gsplat-github.html new file mode 100644 index 0000000..ecc80a2 --- /dev/null +++ b/research/gpu-fusion-splatting/mirrors/gsplat-github.html @@ -0,0 +1,1886 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + GitHub - nerfstudio-project/gsplat: CUDA accelerated rasterization of gaussian splatting · GitHub + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+
+ + +
+ Skip to content + + + + + + + + + + + +
+
+ + + + + + + + + + + + + + + + + + + +
+ +
+ + + + + + + + +
+ + + + + +
+ + + + + + + + + +
+
+
+ + + + + + + + + + + + + +
+ +
+ +
+ +
+ + + + / + + gsplat + + + Public +
+ + +
+ +
+ + +
+
+ +
+
+ + + + +
+ + + + + +
+ + + + + + + + + + + + + + + + + + +
+ + + + +

nerfstudio-project/gsplat

+ + + +

Repository files navigation

gsplat

+

Core Tests. +Docs

+

http://www.gsplat.studio/

+

gsplat is an open-source library for CUDA accelerated rasterization of gaussians with python bindings. It is inspired by the SIGGRAPH paper 3D Gaussian Splatting for Real-Time Rendering of Radiance Fields, but we’ve made gsplat even faster, more memory efficient, and with a growing list of new features!

+
+
+ + + gsplat-quick-intro.mp4 + + + + +
+
+

News

+

[Jan 2026] PPISP is integrated as an alternative way of bilateral grid to compensate the training views.

+

[May 2025] Arbitrary batching (over multiple scenes and multiple viewpoints) is supported now!! Checkout here for more details! Kudos to Junchen Liu.

+

[May 2025] Jonathan Stephens makes a great tutorial video for Windows users on how to install gsplat and get start with 3DGUT.

+

[April 2025] NVIDIA 3DGUT is now integrated in gsplat! Checkout here for more details. [NVIDIA Tech Blog] [NVIDIA Sweepstakes]

+

Installation

+

Dependence: Please install Pytorch first.

+

The easiest way is to install from PyPI. In this way it will build the CUDA code on the first run (JIT).

+
pip install gsplat
+

Alternatively you can install gsplat from source. In this way it will build the CUDA code during installation.

+
pip install git+https://github.com/nerfstudio-project/gsplat.git
+

We also provide pre-compiled wheels for both linux and windows on certain python-torch-CUDA combinations (please check first which versions are supported). Note this way you would have to manually install gsplat's dependencies. For example, to install gsplat for pytorch 2.0 and cuda 11.8 you can run

+
pip install ninja numpy jaxtyping rich
+pip install gsplat --index-url https://docs.gsplat.studio/whl/pt20cu118
+
+

To build gsplat from source on Windows, please check this instruction.

+

Evaluation

+

This repo comes with a standalone script that reproduces the official Gaussian Splatting with exactly the same performance on PSNR, SSIM, LPIPS, and converged number of Gaussians. Powered by gsplat’s efficient CUDA implementation, the training takes up to 4x less GPU memory with up to 15% less time to finish than the official implementation. Full report can be found here.

+
cd examples
+pip install -r requirements.txt
+# download mipnerf_360 benchmark data
+python datasets/download_dataset.py
+# run batch evaluation
+bash benchmarks/basic.sh
+

Examples

+

We provide a set of examples to get you started! Below you can find the details about +the examples (requires installing some extrra dependencies via pip install -r examples/requirements.txt --no-build-isolation)

+ +

Development and Contribution

+

This repository was born from the curiosity of people on the Nerfstudio team trying to understand a new rendering technique. We welcome contributions of any kind and are open to feedback, bug-reports, and improvements to help expand the capabilities of this software.

+

This project is developed by the contributors coming from following institutes (unordered):

+
    +
  • UC Berkeley
  • +
  • NVIDIA
  • +
  • ShanghaiTech University
  • +
  • Amazon
  • +
  • Meta
  • +
  • IIIT
  • +
  • LumaAI
  • +
  • SpectacularAI
  • +
  • Aalto University
  • +
  • CMU
  • +
+

We also have a white paper with about the project with benchmarking and mathematical supplement with conventions and derivations, available here. If you find this library useful in your projects or papers, please consider citing:

+
@article{ye2025gsplat,
+  title={gsplat: An open-source library for Gaussian splatting},
+  author={Ye, Vickie and Li, Ruilong and Kerr, Justin and Turkulainen, Matias and Yi, Brent and Pan, Zhuoyang and Seiskari, Otto and Ye, Jianbo and Hu, Jeffrey and Tancik, Matthew and Angjoo Kanazawa},
+  journal={Journal of Machine Learning Research},
+  volume={26},
+  number={34},
+  pages={1--17},
+  year={2025}
+}
+
+

We welcome contributions of any kind and are open to feedback, bug-reports, and improvements to help expand the capabilities of this software. Please check docs/DEV.md for more info about development.

+
+ +
+
+
+
+

About

+ +

+ CUDA accelerated rasterization of gaussian splatting +

+
+ + + docs.gsplat.studio/ + +
+ +

Topics

+ + +

Resources

+ + + +

License

+ + + + + + + + + + + + + + + + +

Stars

+ + +

Watchers

+ + +

Forks

+ + + + +
+ +
+
+ + + + + + + +
+
+ + + +

+ Packages +

+ + +
+
+
 
+
+
+
+
 
+
+
+
+
 
+
+ + + + +
+
+
+ + + + + +
+
+ + +

+ Contributors

+ +
    +
  • +
    +
  • +
  • +
    +
  • +
  • +
    +
  • +
+ + +
+
+
+ + +
+
+

Languages

+
+ + + + + +
+ + +
+
+ + +
+
+
+ + + + +
+ +
+ +
+
+ +
+ +
+

Footer

+ + + + +
+
+ + + + + © 2026 GitHub, Inc. + +
+ + +
+
+ + + + + + + + + + + + + + + + + + + + +
+
+
+ + + diff --git a/research/gpu-fusion-splatting/mirrors/kerbl-2023-3d-gaussian-splatting.pdf b/research/gpu-fusion-splatting/mirrors/kerbl-2023-3d-gaussian-splatting.pdf new file mode 100644 index 0000000..50f6ef5 Binary files /dev/null and b/research/gpu-fusion-splatting/mirrors/kerbl-2023-3d-gaussian-splatting.pdf differ diff --git a/research/gpu-fusion-splatting/mirrors/nerfstudio-splatfacto.html b/research/gpu-fusion-splatting/mirrors/nerfstudio-splatfacto.html new file mode 100644 index 0000000..5f88de0 --- /dev/null +++ b/research/gpu-fusion-splatting/mirrors/nerfstudio-splatfacto.html @@ -0,0 +1,610 @@ + + + + + + + + + + + + + + + + + + + + + + Splatfacto - nerfstudio + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Contents + + + + + + Menu + + + + + + + + Expand + + + + + + Light mode + + + + + + + + + + + + + + Dark mode + + + + + + + Auto light/dark mode + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+ +
+
+ +
+ +
+
+ +
+
+
+ + + + + Back to top + +
+ +
+ +
+ +
+
+
+

Splatfacto#

+

Nerfstudio's Gaussian Splatting Implementation

+ +

Paper Website

+

3D Gaussian Splatting was proposed in SIGGRAPH 2023 from INRIA, and is a completely different method of representing radiance fields by explicitly storing a collection of 3D volumetric gaussians. These can be “splatted”, or projected, onto a 2D image provided a camera pose, and rasterized to obtain per-pixel colors. Because rasterization is very fast on GPUs, this method can render much faster than neural representations of radiance fields.

+

To avoid confusion with the original paper, we refer to nerfstudio’s implementation as “Splatfacto”, which will drift away from the original as more features are added. Just as Nerfacto is a blend of various different methods, Splatfacto will be a blend of different gaussian splatting methodologies.

+
+

Installation#

+

GSplat

+

Nerfstudio uses gsplat as its gaussian rasterization backend, an in-house re-implementation which is designed to be more developer friendly. This can be installed with pip install gsplat. The associated CUDA code will be compiled the first time gsplat is executed. Some users with PyTorch 2.0 have experienced issues with this, which can be resolved by either installing gsplat from source, or upgrading torch to 2.1.

+
+
+

Data#

+

Gaussian splatting works much better if you initialize it from pre-existing geometry, such as SfM points from COLMAP. COLMAP datasets or datasets from ns-process-data will automatically save these points and initialize gaussians on them. Other datasets currently do not support initialization, and will initialize gaussians randomly. Initializing from other data inputs (i.e. depth from phone app scanners) may be supported in the future.

+

Because the method trains on full images instead of bundles of rays, there is a new datamanager in full_images_datamanager.py which undistorts input images, caches them, and provides single images at each train step.

+
+
+

Running the Method#

+

To run splatfacto, run ns-train splatfacto --data <data>. Just like NeRF methods, the splat can be interactively viewed in the web-viewer, loaded from a checkpoint, rendered, and exported.

+

We provide a few additional variants:

+
+ + + + + + + + + + + + + + + + + + + + +

Method

Description

Memory

Speed

splatfacto

Default Model

~6GB

Fast

splatfacto-big

More Gaussians, Higher Quality

~12GB

Slower

+
+

A full evalaution of Nerfstudio’s implementation of Gaussian Splatting against the original Inria method can be found here.

+
+

Quality and Regularization#

+

The default settings provided maintain a balance between speed, quality, and splat file size, but if you care more about quality than training speed or size, you can decrease the alpha cull threshold +(threshold to delete translucent gaussians) and disable culling after 15k steps like so: ns-train splatfacto --pipeline.model.cull_alpha_thresh=0.005 --pipeline.model.continue_cull_post_densification=False --data <data>

+

A common artifact in splatting is long, spikey gaussians. PhysGaussian proposes a scale regularizer that encourages gaussians to be more evenly shaped. To enable this, set the pipeline.model.use_scale_regularization flag to True.

+
+
+
+

Details#

+

For more details on the method, see the original paper. Additionally, for a detailed derivation of the gradients used in the gsplat library, see here.

+
+
+

Exporting splats#

+

Gaussian splats can be exported as a .ply file which are ingestable by a variety of online web viewers. You can do this via the viewer, or ns-export gaussian-splat --load-config <config> --output-dir exports/splat. Currently splats can only be exported from trained splats, not from nerfacto.

+

Nerfstudio’s splat export currently supports multiple third-party splat viewers:

+ +
+
+

FAQ#

+
    +
  • Can I export a mesh or pointcloud?

  • +
+

Currently these export options are not supported, but may be in the future. Contributions are always welcome!

+
    +
  • Can I render fisheye, equirectangular, orthographic images?

  • +
+

Currently, no. Gaussian rasterization assumes a perspective camera for its rasterization pipeline. Implementing other camera models is of interest but not currently planned.

+
+
+ +
+
+ +
+ +
+
+ + \ No newline at end of file diff --git a/research/gpu-fusion-splatting/mirrors/nvidia-cuda-async-concurrent-execution.html b/research/gpu-fusion-splatting/mirrors/nvidia-cuda-async-concurrent-execution.html new file mode 100644 index 0000000..f4cdcf3 --- /dev/null +++ b/research/gpu-fusion-splatting/mirrors/nvidia-cuda-async-concurrent-execution.html @@ -0,0 +1,27458 @@ + + + + + + + + + + CUDA C++ Programming Guide (Legacy) — CUDA C++ Programming Guide + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + +
+ +
+
+
+
    + + + + +
  • + »
  • + +
  • CUDA C++ Programming Guide (Legacy)
  • + +
  • +
  • +
  • + + + v13.2 | + + + + PDF + + + + | + + + + Archive + + +   +
  • + +
+
+
+
+
+ +

CUDA C++ Programming Guide (Legacy)

+
+

Warning

+

This document has been replaced by a new CUDA Programming Guide. The information in this document should be considered legacy, and this document is no longer being updated as of CUDA 13.0. Please refer to the CUDA Programming Guide for up-to-date information on CUDA.

+
+
+

+1. Overview +

+

CUDA is a parallel computing platform and programming model developed by NVIDIA that enables dramatic increases in computing performance by harnessing the power of the GPU. It allows developers to accelerate compute-intensive applications using C, C++, and Fortran, and is widely adopted in fields such as deep learning, scientific computing, and high-performance computing (HPC).

+
+
+

+2. What Is the CUDA C Programming Guide? +

+

The CUDA C Programming Guide is the official, comprehensive resource that explains how to write programs using the CUDA platform. It provides detailed documentation of the CUDA architecture, programming model, language extensions, and performance guidelines. Whether you’re just getting started or optimizing complex GPU kernels, this guide is an essential reference for effectively leveraging CUDA’s full capabilities.

+
+
+

+3. Introduction +

+
+

Warning

+

This document has been replaced by a new CUDA Programming Guide. The information in this document should be considered legacy, and this document is no longer being updated as of CUDA 13.0. Please refer to the CUDA Programming Guide for up-to-date information on CUDA.

+
+
+

+3.1. The Benefits of Using GPUs +

+

The Graphics Processing Unit (GPU)1 provides much higher instruction throughput and memory bandwidth than the CPU within a similar price and power envelope. Many applications leverage these higher capabilities to run faster on the GPU than on the CPU (see GPU Applications). Other computing devices, like FPGAs, are also very energy efficient, but offer much less programming flexibility than GPUs.

+

This difference in capabilities between the GPU and the CPU exists because they are designed with different goals in mind. While the CPU is designed to excel at executing a sequence of operations, called a thread, as fast as possible and can execute a few tens of these threads in parallel, the GPU is designed to excel at executing thousands of them in parallel (amortizing the slower single-thread performance to achieve greater throughput).

+

The GPU is specialized for highly parallel computations and therefore designed such that more transistors are devoted to data processing rather than data caching and flow control. The schematic Figure 1 shows an example distribution of chip resources for a CPU versus a GPU.

+
+The GPU Devotes More Transistors to Data Processing +
+

Figure 1 The GPU Devotes More Transistors to Data Processing

+
+
+

Devoting more transistors to data processing, for example, floating-point computations, is beneficial for highly parallel computations; the GPU can hide memory access latencies with computation, instead of relying on large data caches and complex flow control to avoid long memory access latencies, both of which are expensive in terms of transistors.

+

In general, an application has a mix of parallel parts and sequential parts, so systems are designed with a mix of GPUs and CPUs in order to maximize overall performance. Applications with a high degree of parallelism can exploit this massively parallel nature of the GPU to achieve higher performance than on the CPU.

+
+
1
+
+

The graphics qualifier comes from the fact that when the GPU was originally created, two decades ago, it was designed as a specialized processor to accelerate graphics rendering. Driven by the insatiable market demand for real-time, high-definition, 3D graphics, it has evolved into a general processor used for many more workloads than just graphics rendering.

+
+
+
+
+

+3.2. CUDA®: A General-Purpose Parallel Computing Platform and Programming Model +

+

In November 2006, NVIDIA® introduced CUDA®, a general purpose parallel computing platform and programming model that leverages the parallel compute engine in NVIDIA GPUs to solve many complex computational problems in a more efficient way than on a CPU.

+

CUDA comes with a software environment that allows developers to use C++ as a high-level programming language. As illustrated by Figure 2, other languages, application programming interfaces, or directives-based approaches are supported, such as FORTRAN, DirectCompute, OpenACC.

+
+GPU Computing Applications. CUDA is designed to support various languages and application programming interfaces. +
+

Figure 2 GPU Computing Applications. CUDA is designed to support various languages and application programming interfaces.

+
+
+
+
+

+3.3. A Scalable Programming Model +

+

The advent of multicore CPUs and manycore GPUs means that mainstream processor chips are now parallel systems. The challenge is to develop application software that transparently scales its parallelism to leverage the increasing number of processor cores, much as 3D graphics applications transparently scale their parallelism to manycore GPUs with widely varying numbers of cores.

+

The CUDA parallel programming model is designed to overcome this challenge while maintaining a low learning curve for programmers familiar with standard programming languages such as C.

+

At its core are three key abstractions — a hierarchy of thread groups, shared memories, and barrier synchronization — that are simply exposed to the programmer as a minimal set of language extensions.

+

These abstractions provide fine-grained data parallelism and thread parallelism, nested within coarse-grained data parallelism and task parallelism. They guide the programmer to partition the problem into coarse sub-problems that can be solved independently in parallel by blocks of threads, and each sub-problem into finer pieces that can be solved cooperatively in parallel by all threads within the block.

+

This decomposition preserves language expressivity by allowing threads to cooperate when solving each sub-problem, and at the same time enables automatic scalability. Indeed, each block of threads can be scheduled on any of the available multiprocessors within a GPU, in any order, concurrently or sequentially, so that a compiled CUDA program can execute on any number of multiprocessors as illustrated by Figure 3, and only the runtime system needs to know the physical multiprocessor count.

+

This scalable programming model allows the GPU architecture to span a wide market range by simply scaling the number of multiprocessors and memory partitions: from the high-performance enthusiast GeForce GPUs and professional Quadro and Tesla computing products to a variety of inexpensive, mainstream GeForce GPUs (see CUDA-Enabled GPUs for a list of all CUDA-enabled GPUs).

+
+Automatic Scalability +
+

Figure 3 Automatic Scalability

+
+
+
+

Note

+

A GPU is built around an array of Streaming Multiprocessors (SMs) (see Hardware Implementation for more details). A multithreaded program is partitioned into blocks of threads that execute independently from each other, so that a GPU with more multiprocessors will automatically execute the program in less time than a GPU with fewer multiprocessors.

+
+
+
+
+

+4. Changelog +

+ + ++++ + + + + + + + + + + + + + + + + + + + + +
+Table 1 Change Log +

Version

Changes

13.0

Moved the instruction throughput table from the Performance Guidelines section of the CUDA C++ Programming Guide to the Instruction-optimization section of the CUDA C++ Best Practices Guide. Removed unsupported architectures and corrected entries for integer arithmetic and type conversion.

12.9

Added section Error Log Management and CUDA_LOG_FILE to CUDA Environment Variables

12.8

Added section TMA Swizzle

+
+
+

+5. Programming Model +

+
+

Warning

+

This document has been replaced by a new CUDA Programming Guide. The information in this document should be considered legacy, and this document is no longer being updated as of CUDA 13.0. Please refer to the CUDA Programming Guide for up-to-date information on CUDA.

+
+

This chapter introduces the main concepts behind the CUDA programming model by outlining how they are exposed in C++.

+

An extensive description of CUDA C++ is given in Programming Interface.

+

Full code for the vector addition example used in this chapter and the next can be found in the vectorAdd CUDA sample.

+
+

+5.1. Kernels +

+

CUDA C++ extends C++ by allowing the programmer to define C++ functions, called kernels, that, when called, are executed N times in parallel by N different CUDA threads, as opposed to only once like regular C++ functions.

+

A kernel is defined using the __global__ declaration specifier and the number of CUDA threads that execute that kernel for a given kernel call is specified using a new <<<...>>>execution configuration syntax (see Execution Configuration). Each thread that executes the kernel is given a unique thread ID that is accessible within the kernel through built-in variables.

+

As an illustration, the following sample code, using the built-in variable threadIdx, adds two vectors A and B of size N and stores the result into vector C.

+
+
// Kernel definition
+__global__ void VecAdd(float* A, float* B, float* C)
+{
+    int i = threadIdx.x;
+    C[i] = A[i] + B[i];
+}
+
+int main()
+{
+    ...
+    // Kernel invocation with N threads
+    VecAdd<<<1, N>>>(A, B, C);
+    ...
+}
+
+
+

Here, each of the N threads that execute VecAdd() performs one pair-wise addition.

+
+
+

+5.2. Thread Hierarchy +

+

For convenience, threadIdx is a 3-component vector, so that threads can be identified using a one-dimensional, two-dimensional, or three-dimensional thread index, forming a one-dimensional, two-dimensional, or three-dimensional block of threads, called a thread block. This provides a natural way to invoke computation across the elements in a domain such as a vector, matrix, or volume.

+

The index of a thread and its thread ID relate to each other in a straightforward way: For a one-dimensional block, they are the same; for a two-dimensional block of size (Dx, Dy), the thread ID of a thread of index (x, y) is (x + y Dx); for a three-dimensional block of size (Dx, Dy, Dz), the thread ID of a thread of index (x, y, z) is (x + y Dx + z Dx Dy).

+

As an example, the following code adds two matrices A and B of size NxN and stores the result into matrix C.

+
+
// Kernel definition
+__global__ void MatAdd(float A[N][N], float B[N][N],
+                       float C[N][N])
+{
+    int i = threadIdx.x;
+    int j = threadIdx.y;
+    C[i][j] = A[i][j] + B[i][j];
+}
+
+int main()
+{
+    ...
+    // Kernel invocation with one block of N * N * 1 threads
+    int numBlocks = 1;
+    dim3 threadsPerBlock(N, N);
+    MatAdd<<<numBlocks, threadsPerBlock>>>(A, B, C);
+    ...
+}
+
+
+

There is a limit to the number of threads per block, since all threads of a block are expected to reside on the same streaming multiprocessor core and must share the limited memory resources of that core. On current GPUs, a thread block may contain up to 1024 threads.

+

However, a kernel can be executed by multiple equally-shaped thread blocks, so that the total number of threads is equal to the number of threads per block times the number of blocks.

+

Blocks are organized into a one-dimensional, two-dimensional, or three-dimensional grid of thread blocks as illustrated by Figure 4. The number of thread blocks in a grid is usually dictated by the size of the data being processed, which typically exceeds the number of processors in the system.

+
+Grid of Thread Blocks +
+

Figure 4 Grid of Thread Blocks

+
+
+

The number of threads per block and the number of blocks per grid specified in the <<<...>>> syntax can be of type int or dim3. Two-dimensional blocks or grids can be specified as in the example above.

+

Each block within the grid can be identified by a one-dimensional, two-dimensional, or three-dimensional unique index accessible within the kernel through the built-in blockIdx variable. The dimension of the thread block is accessible within the kernel through the built-in blockDim variable.

+

Extending the previous MatAdd() example to handle multiple blocks, the code becomes as follows.

+
+
// Kernel definition
+__global__ void MatAdd(float A[N][N], float B[N][N],
+float C[N][N])
+{
+    int i = blockIdx.x * blockDim.x + threadIdx.x;
+    int j = blockIdx.y * blockDim.y + threadIdx.y;
+    if (i < N && j < N)
+        C[i][j] = A[i][j] + B[i][j];
+}
+
+int main()
+{
+    ...
+    // Kernel invocation
+    dim3 threadsPerBlock(16, 16);
+    dim3 numBlocks(N / threadsPerBlock.x, N / threadsPerBlock.y);
+    MatAdd<<<numBlocks, threadsPerBlock>>>(A, B, C);
+    ...
+}
+
+
+

A thread block size of 16x16 (256 threads), although arbitrary in this case, is a common choice. The grid is created with enough blocks to have one thread per matrix element as before. For simplicity, this example assumes that the number of threads per grid in each dimension is evenly divisible by the number of threads per block in that dimension, although that need not be the case.

+

Thread blocks are required to execute independently. It must be possible to execute blocks in any order, in parallel or in series. This independence requirement allows thread blocks to be scheduled in any order and across any number of cores as illustrated by Figure 3, enabling programmers to write code that scales with the number of cores.

+

Threads within a block can cooperate by sharing data through some shared memory and by synchronizing their execution to coordinate memory accesses. More precisely, one can specify synchronization points in the kernel by calling the __syncthreads() intrinsic function; __syncthreads() acts as a barrier at which all threads in the block must wait before any is allowed to proceed. Shared Memory gives an example of using shared memory. In addition to __syncthreads(), the Cooperative Groups API provides a rich set of thread-synchronization primitives.

+

For efficient cooperation, shared memory is expected to be a low-latency memory near each processor core (much like an L1 cache) and __syncthreads() is expected to be lightweight.

+
+

+5.2.1. Thread Block Clusters +

+

With the introduction of NVIDIA Compute Capability 9.0, the CUDA programming model introduces an optional level of hierarchy called Thread Block Clusters that are made up of thread blocks. Similar to how threads in a thread block are guaranteed to be co-scheduled on a streaming multiprocessor, thread blocks in a cluster are also guaranteed to be co-scheduled on a GPU Processing Cluster (GPC) in the GPU.

+

Similar to thread blocks, clusters are also organized into a one-dimension, two-dimension, or three-dimension grid of thread block clusters as illustrated by Figure 5. The number of thread blocks in a cluster can be user-defined, and a maximum of 8 thread blocks in a cluster is supported as a portable cluster size in CUDA. +Note that on GPU hardware or MIG configurations which are too small to support 8 multiprocessors the maximum cluster size will be reduced accordingly. Identification of these smaller configurations, as well as of larger configurations supporting a thread block cluster size beyond 8, is architecture-specific and can be queried using the cudaOccupancyMaxPotentialClusterSize API.

+
+Grid of Thread Block Clusters +
+

Figure 5 Grid of Thread Block Clusters

+
+
+
+

Note

+

In a kernel launched using cluster support, the gridDim variable still denotes the size in terms of number of thread blocks, for compatibility purposes. The rank of a block in a cluster can be found using the Cluster Group API.

+
+

A thread block cluster can be enabled in a kernel either using a compile-time kernel attribute using __cluster_dims__(X,Y,Z) or using the CUDA kernel launch API cudaLaunchKernelEx. The example below shows how to launch a cluster using a compile-time kernel attribute. The cluster size using kernel attribute is fixed at compile time and then the kernel can be launched using the classical <<< , >>>. If a kernel uses compile-time cluster size, the cluster size cannot be modified when launching the kernel.

+
+
// Kernel definition
+// Compile time cluster size 2 in X-dimension and 1 in Y and Z dimension
+__global__ void __cluster_dims__(2, 1, 1) cluster_kernel(float *input, float* output)
+{
+
+}
+
+int main()
+{
+    float *input, *output;
+    // Kernel invocation with compile time cluster size
+    dim3 threadsPerBlock(16, 16);
+    dim3 numBlocks(N / threadsPerBlock.x, N / threadsPerBlock.y);
+
+    // The grid dimension is not affected by cluster launch, and is still enumerated
+    // using number of blocks.
+    // The grid dimension must be a multiple of cluster size.
+    cluster_kernel<<<numBlocks, threadsPerBlock>>>(input, output);
+}
+
+
+

A thread block cluster size can also be set at runtime and the kernel can be launched using the CUDA kernel launch API cudaLaunchKernelEx. The code example below shows how to launch a cluster kernel using the extensible API.

+
+
// Kernel definition
+// No compile time attribute attached to the kernel
+__global__ void cluster_kernel(float *input, float* output)
+{
+
+}
+
+int main()
+{
+    float *input, *output;
+    dim3 threadsPerBlock(16, 16);
+    dim3 numBlocks(N / threadsPerBlock.x, N / threadsPerBlock.y);
+
+    // Kernel invocation with runtime cluster size
+    {
+        cudaLaunchConfig_t config = {0};
+        // The grid dimension is not affected by cluster launch, and is still enumerated
+        // using number of blocks.
+        // The grid dimension should be a multiple of cluster size.
+        config.gridDim = numBlocks;
+        config.blockDim = threadsPerBlock;
+
+        cudaLaunchAttribute attribute[1];
+        attribute[0].id = cudaLaunchAttributeClusterDimension;
+        attribute[0].val.clusterDim.x = 2; // Cluster size in X-dimension
+        attribute[0].val.clusterDim.y = 1;
+        attribute[0].val.clusterDim.z = 1;
+        config.attrs = attribute;
+        config.numAttrs = 1;
+
+        cudaLaunchKernelEx(&config, cluster_kernel, input, output);
+    }
+}
+
+
+

In GPUs with compute capability 9.0, all the thread blocks in the cluster are guaranteed to be co-scheduled on a single GPU Processing Cluster (GPC) and allow thread blocks in the cluster to perform hardware-supported synchronization using the Cluster Group API cluster.sync(). Cluster group also provides member functions to query cluster group size in terms of number of threads or number of blocks using num_threads() and num_blocks() API respectively. The rank of a thread or block in the cluster group can be queried using dim_threads() and dim_blocks() API respectively.

+

Thread blocks that belong to a cluster have access to the Distributed Shared Memory. Thread blocks in a cluster have the ability to read, write, and perform atomics to any address in the distributed shared memory. Distributed Shared Memory gives an example of performing histograms in distributed shared memory.

+
+
+

+5.2.2. Blocks as Clusters +

+

With __cluster_dims__, the number of launched clusters is kept implicit and can only be calculated manually.

+
+
__cluster_dims__((2, 2, 2)) __global__ void foo();
+
+// 8x8x8 clusters each with 2x2x2 thread blocks.
+foo<<<dim3(16, 16, 16), dim3(1024, 1, 1)>>>();
+
+
+

In the above example, the kernel is launched as a grid of 16x16x16 thread blocks, or in fact a grid of 8x8x8 clusters. Alternatively, with another compile-time kernel attribute __block_size__, one is allowed to launch a grid explicitly configured with the number of thread block clusters.

+
+
// Implementation detail of how many threads per block and blocks per cluster
+// is handled as an attribute of the kernel.
+__block_size__((1024, 1, 1), (2, 2, 2)) __global__ void foo();
+
+// 8x8x8 clusters.
+foo<<<dim3(8, 8, 8)>>>();
+
+
+

__block_size__ requires two fields each being a tuple of 3 elements. The first tuple denotes block dimension and second cluster size. The second tuple is assumed to be (1,1,1) if it’s not passed. To specify the stream, one must pass 1 and 0 as the second and third arguments within <<<>>> and lastly the stream. Passing other values would lead to undefined behavior.

+

Note that it is illegal for the second tuple of __block_size__ and __cluster_dims__ to be specified at the same time. It’s also illegal to use __block_size__ with an empty __cluster_dims__. When the second tuple of __block_size__ is specified, it implies the “Blocks as Clusters” being enabled and the compiler would recognize the first argument inside <<<>>> as the number of clusters instead of thread blocks.

+
+
+
+

+5.3. Memory Hierarchy +

+

CUDA threads may access data from multiple memory spaces during their execution as illustrated by Figure 6. Each thread has private local memory. Each thread block has shared memory visible to all threads of the block and with the same lifetime as the block. Thread blocks in a thread block cluster can perform read, write, and atomics operations on each other’s shared memory. All threads have access to the same global memory.

+

There are also two additional read-only memory spaces accessible by all threads: the constant and texture memory spaces. The global, constant, and texture memory spaces are optimized for different memory usages (see Device Memory Accesses). Texture memory also offers different addressing modes, as well as data filtering, for some specific data formats (see Texture and Surface Memory).

+

The global, constant, and texture memory spaces are persistent across kernel launches by the same application.

+
+Memory Hierarchy +
+

Figure 6 Memory Hierarchy

+
+
+
+
+

+5.4. Heterogeneous Programming +

+

As illustrated by Figure 7, the CUDA programming model assumes that the CUDA threads execute on a physically separate device that operates as a coprocessor to the host running the C++ program. This is the case, for example, when the kernels execute on a GPU and the rest of the C++ program executes on a CPU.

+

The CUDA programming model also assumes that both the host and the device maintain their own separate memory spaces in DRAM, referred to as host memory and device memory, respectively. Therefore, a program manages the global, constant, and texture memory spaces visible to kernels through calls to the CUDA runtime (described in Programming Interface). This includes device memory allocation and deallocation as well as data transfer between host and device memory.

+

Unified Memory provides managed memory to bridge the host and device memory spaces. Managed memory is accessible from all CPUs and GPUs in the system as a single, coherent memory image with a common address space. This capability enables oversubscription of device memory and can greatly simplify the task of porting applications by eliminating the need to explicitly mirror data on host and device. See Unified Memory Programming for an introduction to Unified Memory.

+
+Heterogeneous Programming +
+

Figure 7 Heterogeneous Programming

+
+
+

Note

+

Serial code executes on the host while parallel code executes on the device.

+
+
+
+
+
+
+

+5.5. Asynchronous SIMT Programming Model +

+

In the CUDA programming model a thread is the lowest level of abstraction for doing a computation or a memory operation. Starting with devices based on the NVIDIA Ampere GPU Architecture, the CUDA programming model provides acceleration to memory operations via the asynchronous programming model. The asynchronous programming model defines the behavior of asynchronous operations with respect to CUDA threads.

+

The asynchronous programming model defines the behavior of Asynchronous Barrier for synchronization between CUDA threads. The model also explains and defines how cuda::memcpy_async can be used to move data asynchronously from global memory while computing in the GPU.

+
+

+5.5.1. Asynchronous Operations +

+

An asynchronous operation is defined as an operation that is initiated by a CUDA thread and is executed asynchronously as-if by another thread. In a well formed program one or more CUDA threads synchronize with the asynchronous operation. The CUDA thread that initiated the asynchronous operation is not required to be among the synchronizing threads.

+

Such an asynchronous thread (an as-if thread) is always associated with the CUDA thread that initiated the asynchronous operation. An asynchronous operation uses a synchronization object to synchronize the completion of the operation. Such a synchronization object can be explicitly managed by a user (e.g., cuda::memcpy_async) or implicitly managed within a library (e.g., cooperative_groups::memcpy_async).

+

A synchronization object could be a cuda::barrier or a cuda::pipeline. These objects are explained in detail in Asynchronous Barrier and Asynchronous Data Copies using cuda::pipeline. These synchronization objects can be used at different thread scopes. A scope defines the set of threads that may use the synchronization object to synchronize with the asynchronous operation. The following table defines the thread scopes available in CUDA C++ and the threads that can be synchronized with each.

+ ++++ + + + + + + + + + + + + + + + + + + + + + + + + +

Thread Scope

Description

cuda::thread_scope::thread_scope_thread

Only the CUDA thread which initiated asynchronous operations synchronizes.

cuda::thread_scope::thread_scope_block

All or any CUDA threads within the same thread block as the initiating thread synchronizes.

cuda::thread_scope::thread_scope_device

All or any CUDA threads in the same GPU device as the initiating thread synchronizes.

cuda::thread_scope::thread_scope_system

All or any CUDA or CPU threads in the same system as the initiating thread synchronizes.

+

These thread scopes are implemented as extensions to standard C++ in the CUDA Standard C++ library.

+
+
+
+

+5.6. Compute Capability +

+

The compute capability of a device is represented by a version number, also sometimes called its “SM version”. This version number identifies the features supported by the GPU hardware and is used by applications at runtime to determine which hardware features and/or instructions are available on the present GPU.

+

The compute capability comprises a major revision number X and a minor revision number Y and is denoted by X.Y.

+

The major revision number indicates the core GPU architecture of a device. Devices with the same major revision number share the same fundamental architecture. The table below lists the major revision numbers corresponding to each NVIDIA GPU architecture.

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+Table 2 GPU Architecture and Major Revision Numbers +

Major Revision Number

NVIDIA GPU Architecture

9

NVIDIA Hopper GPU Architecture

8

NVIDIA Ampere GPU Architecture

7

NVIDIA Volta GPU Architecture

6

NVIDIA Pascal GPU Architecture

5

NVIDIA Maxwell GPU Architecture

3

NVIDIA Kepler GPU Architecture

+

The minor revision number corresponds to an incremental improvement to the core architecture, possibly including new features.

+ + + + + + + + + + + + + + + + +
+Table 3 Incremental Updates in GPU Architectures +

Compute Capability

NVIDIA GPU Architecture

Based On

7.5

NVIDIA Turing GPU Architecture

NVIDIA Volta GPU Architecture

+

CUDA-Enabled GPUs lists of all CUDA-enabled devices along with their compute capability. Compute Capabilities gives the technical specifications of each compute capability.

+
+

Note

+

The compute capability version of a particular GPU should not be confused with the CUDA version (for example, CUDA 7.5, CUDA 8, CUDA 9), which is the version of the CUDA software platform. The CUDA platform is used by application developers to create applications that run on many generations of GPU architectures, including future GPU architectures yet to be invented. While new versions of the CUDA platform often add native support for a new GPU architecture by supporting the compute capability version of that architecture, new versions of the CUDA platform typically also include software features that are independent of hardware generation.

+
+

The Tesla and Fermi architectures are no longer supported starting with CUDA 7.0 and CUDA 9.0, respectively.

+
+
+
+

+6. Programming Interface +

+
+

Warning

+

This document has been replaced by a new CUDA Programming Guide. The information in this document should be considered legacy, and this document is no longer being updated as of CUDA 13.0. Please refer to the CUDA Programming Guide for up-to-date information on CUDA.

+
+

CUDA C++ provides a simple path for users familiar with the C++ programming language to easily write programs for execution by the device.

+

It consists of a minimal set of extensions to the C++ language and a runtime library.

+

The core language extensions have been introduced in Programming Model. They allow programmers to define a kernel as a C++ function and use some new syntax to specify the grid and block dimension each time the function is called. A complete description of all extensions can be found in C++ Language Extensions. Any source file that contains some of these extensions must be compiled with nvcc as outlined in Compilation with NVCC.

+

The runtime is introduced in CUDA Runtime. It provides C and C++ functions that execute on the host to allocate and deallocate device memory, transfer data between host memory and device memory, manage systems with multiple devices, etc. A complete description of the runtime can be found in the CUDA reference manual.

+

The runtime is built on top of a lower-level C API, the CUDA driver API, which is also accessible by the application. The driver API provides an additional level of control by exposing lower-level concepts such as CUDA contexts - the analogue of host processes for the device - and CUDA modules - the analogue of dynamically loaded libraries for the device. Most applications do not use the driver API as they do not need this additional level of control and when using the runtime, context and module management are implicit, resulting in more concise code. As the runtime is interoperable with the driver API, most applications that need some driver API features can default to use the runtime API and only use the driver API where needed. The driver API is introduced in Driver API and fully described in the reference manual.

+
+

+6.1. Compilation with NVCC +

+

Kernels can be written using the CUDA instruction set architecture, called PTX, which is described in the PTX reference manual. It is however usually more effective to use a high-level programming language such as C++. In both cases, kernels must be compiled into binary code by nvcc to execute on the device.

+

nvcc is a compiler driver that simplifies the process of compiling C++ or PTX code: It provides simple and familiar command line options and executes them by invoking the collection of tools that implement the different compilation stages. This section gives an overview of nvcc workflow and command options. A complete description can be found in the nvcc user manual.

+
+

+6.1.1. Compilation Workflow +

+
+

+6.1.1.1. Offline Compilation +

+

Source files compiled with nvcc can include a mix of host code (i.e., code that executes on the host) and device code (i.e., code that executes on the device). nvcc’s basic workflow consists in separating device code from host code and then:

+
    +
  • compiling the device code into an assembly form (PTX code) and/or binary form (cubin object),

  • +
  • and modifying the host code by replacing the <<<...>>> syntax introduced in Kernels (and described in more details in Execution Configuration) by the necessary CUDA runtime function calls to load and launch each compiled kernel from the PTX code and/or cubin object.

  • +
+

The modified host code is output either as C++ code that is left to be compiled using another tool or as object code directly by letting nvcc invoke the host compiler during the last compilation stage.

+

Applications can then:

+
    +
  • Either link to the compiled host code (this is the most common case),

  • +
  • Or ignore the modified host code (if any) and use the CUDA driver API (see Driver API) to load and execute the PTX code or cubin object.

  • +
+
+
+

+6.1.1.2. Just-in-Time Compilation +

+

Any PTX code loaded by an application at runtime is compiled further to binary code by the device driver. This is called just-in-time compilation. Just-in-time compilation increases application load time, but allows the application to benefit from any new compiler improvements coming with each new device driver. It is also the only way for applications to run on devices that did not exist at the time the application was compiled, as detailed in Application Compatibility.

+

When the device driver just-in-time compiles some PTX code for some application, it automatically caches a copy of the generated binary code in order to avoid repeating the compilation in subsequent invocations of the application. The cache - referred to as compute cache - is automatically invalidated when the device driver is upgraded, so that applications can benefit from the improvements in the new just-in-time compiler built into the device driver.

+

Environment variables are available to control just-in-time compilation as described in CUDA Environment Variables

+

As an alternative to using nvcc to compile CUDA C++ device code, NVRTC can be used to compile CUDA C++ device code to PTX at runtime. NVRTC is a runtime compilation library for CUDA C++; more information can be found in the NVRTC User guide.

+
+
+
+

+6.1.2. Binary Compatibility +

+

Binary code is architecture-specific. A cubin object is generated using the compiler option -code that specifies the targeted architecture: For example, compiling with -code=sm_80 produces binary code for devices of compute capability 8.0. Binary compatibility is guaranteed from one minor revision to the next one, but not from one minor revision to the previous one or across major revisions. In other words, a cubin object generated for compute capability X.y will only execute on devices of compute capability X.z where z≥y.

+
+

Note

+

Binary compatibility is supported only for the desktop. It is not supported for Tegra. Also, the binary compatibility between desktop and Tegra is not supported.

+
+
+
+

+6.1.3. PTX Compatibility +

+

Some PTX instructions are only supported on devices of higher compute capabilities. For example, Warp Shuffle Functions are only supported on devices of compute capability 5.0 and above. The -arch compiler option specifies the compute capability that is assumed when compiling C++ to PTX code. So, code that contains warp shuffle, for example, must be compiled with -arch=compute_50 (or higher).

+

PTX code produced for some specific compute capability can always be compiled to binary code of greater or equal compute capability. Note that a binary compiled from an earlier PTX version may not make use of some hardware features. For example, a binary targeting devices of compute capability 7.0 (Volta) compiled from PTX generated for compute capability 6.0 (Pascal) will not make use of Tensor Core instructions, since these were not available on Pascal. As a result, the final binary may perform worse than would be possible if the binary were generated using the latest version of PTX.

+

PTX code compiled to target Architecture-Specific Features only runs on the exact same physical architecture and nowhere else. Architecture-specific PTX code is not forward and backward compatible. +Example code compiled with sm_90a or compute_90a only runs on devices with compute capability 9.0 and is not backward or forward compatible.

+

PTX code compiled to target Family-Specific Features only runs on the exact same physical architecture and other architectures in the same family. Family-specific PTX code is forward compatible with other devices in the same family, and is not backward compatible. +Example code compiled with sm_100f or compute_100f only runs on devices with compute capability 10.0 and 10.3. Table 25 shows the compatibility of family-specific targets with compute capability.

+
+
+

+6.1.4. Application Compatibility +

+

To execute code on devices of specific compute capability, an application must load binary or PTX code that is compatible with this compute capability as described in Binary Compatibility and PTX Compatibility. In particular, to be able to execute code on future architectures with higher compute capability (for which no binary code can be generated yet), an application must load PTX code that will be just-in-time compiled for these devices (see Just-in-Time Compilation).

+

Which PTX and binary code gets embedded in a CUDA C++ application is controlled by the -arch and -code compiler options or the -gencode compiler option as detailed in the nvcc user manual. For example,

+
+
nvcc x.cu
+        -gencode arch=compute_50,code=sm_50
+        -gencode arch=compute_60,code=sm_60
+        -gencode arch=compute_70,code=\"compute_70,sm_70\"
+
+
+

embeds binary code compatible with compute capability 5.0 and 6.0 (first and second -gencode options) and PTX and binary code compatible with compute capability 7.0 (third -gencode option).

+

Host code is generated to automatically select at runtime the most appropriate code to load and execute, which, in the above example, will be:

+
    +
  • 5.0 binary code for devices with compute capability 5.0 and 5.2,

  • +
  • 6.0 binary code for devices with compute capability 6.0 and 6.1,

  • +
  • 7.0 binary code for devices with compute capability 7.0 and 7.5,

  • +
  • PTX code which is compiled to binary code at runtime for devices with compute capability later than 7.5

  • +
+

x.cu can have an optimized code path that uses warp reduction operations, for example, which are only supported in devices of compute capability 8.0 and higher. The __CUDA_ARCH__ macro can be used to differentiate various code paths based on compute capability. It is only defined for device code. When compiling with -arch=compute_80 for example, __CUDA_ARCH__ is equal to 800.

+

If x.cu is compiled for Family-Specific Features with sm_100f or compute_100f, the code can only run on devices in that specific family, which are devices with compute capability 10.0 and 10.3. For family-specific code targets an additional macro __CUDA_ARCH_FAMILY_SPECIFIC__ is defined. In this example, __CUDA_ARCH_FAMILY_SPECIFIC__ is equal to 1000.

+

If x.cu is compiled for Architecture-Specific Features with sm_100a or compute_100a, the code can only run on devices with compute capability 10.0. For architecture-specific code targets an additional macro __CUDA_ARCH_SPECIFIC__ is defined. In this example, __CUDA_ARCH_SPECIFIC__ is equal to 1000. Because architecture-specific features are a superset of family-specific features, the family-specific macro __CUDA_ARCH_FAMILY_SPECIFIC__ is also defined and is equal to 1000.

+

Applications using the driver API must compile code to separate files and explicitly load and execute the most appropriate file at runtime.

+

The Volta architecture introduces Independent Thread Scheduling which changes the way threads are scheduled on the GPU. For code relying on specific behavior of SIMT scheduling in previous architectures, Independent Thread Scheduling may alter the set of participating threads, leading to incorrect results. To aid migration while implementing the corrective actions detailed in Independent Thread Scheduling, Volta developers can opt-in to Pascal’s thread scheduling with the compiler option combination -arch=compute_60 -code=sm_70.

+

The nvcc user manual lists various shorthands for the -arch, -code, and -gencode compiler options. For example, -arch=sm_70 is a shorthand for -arch=compute_70 -code=compute_70,sm_70 (which is the same as -gencode arch=compute_70,code=\"compute_70,sm_70\").

+
+
+

+6.1.5. C++ Compatibility +

+

The front end of the compiler processes CUDA source files according to C++ syntax rules. Full C++ is supported for the host code. However, only a subset of C++ is fully supported for the device code as described in C++ Language Support.

+
+
+

+6.1.6. 64-Bit Compatibility +

+

The 64-bit version of nvcc compiles device code in 64-bit mode (i.e., pointers are 64-bit). Device code compiled in 64-bit mode is only supported with host code compiled in 64-bit mode.

+
+
+
+

+6.2. CUDA Runtime +

+

The runtime is implemented in the cudart library, which is linked to the application, either statically via cudart.lib or libcudart.a, or dynamically via cudart.dll or libcudart.so. Applications that require cudart.dll and/or cudart.so for dynamic linking typically include them as part of the application installation package. It is only safe to pass the address of CUDA runtime symbols between components that link to the same instance of the CUDA runtime.

+

All its entry points are prefixed with cuda.

+

As mentioned in Heterogeneous Programming, the CUDA programming model assumes a system composed of a host and a device, each with their own separate memory. Device Memory gives an overview of the runtime functions used to manage device memory.

+

Shared Memory illustrates the use of shared memory, introduced in Thread Hierarchy, to maximize performance.

+

Page-Locked Host Memory introduces page-locked host memory that is required to overlap kernel execution with data transfers between host and device memory.

+

Asynchronous Concurrent Execution describes the concepts and API used to enable asynchronous concurrent execution at various levels in the system.

+

Multi-Device System shows how the programming model extends to a system with multiple devices attached to the same host.

+

Error Checking describes how to properly check the errors generated by the runtime.

+

Call Stack mentions the runtime functions used to manage the CUDA C++ call stack.

+

Texture and Surface Memory presents the texture and surface memory spaces that provide another way to access device memory; they also expose a subset of the GPU texturing hardware.

+

Graphics Interoperability introduces the various functions the runtime provides to interoperate with the two main graphics APIs, OpenGL and Direct3D.

+
+

+6.2.1. Initialization +

+

As of CUDA 12.0, the cudaInitDevice() and cudaSetDevice() calls initialize the runtime and the primary context +associated with the specified device. Absent these calls, the runtime will implicitly use device 0 and self-initialize +as needed to process other runtime API requests. One needs to keep this in mind when timing runtime function calls and +when interpreting the error code from the first call into the runtime. Before 12.0, cudaSetDevice() would not +initialize the runtime and applications would often use the no-op runtime call cudaFree(0) to isolate the runtime +initialization from other api activity (both for the sake of timing and error handling).

+

The runtime creates a CUDA context for each device in the system (see Context for more details on CUDA contexts). This +context is the primary context for this device and is initialized at the first runtime function which requires an active +context on this device. It is shared among all the host threads of the application. As part of this context creation, the +device code is just-in-time compiled if necessary (see Just-in-Time Compilation) and loaded into device memory. This +all happens transparently. If needed, for example, for driver API interoperability, the primary context of a device can be +accessed from the driver API as described in Interoperability between Runtime and Driver APIs.

+

When a host thread calls cudaDeviceReset(), this destroys the primary context of the device the host thread currently +operates on (that is, the current device as defined in Device Selection). The next runtime function call made by any +host thread that has this device as current will create a new primary context for this device.

+
+

Note

+

The CUDA interfaces use global state that is initialized during host program initiation and destroyed during host program termination. The CUDA runtime and driver cannot detect if this state is invalid, so using any of these interfaces (implicitly or explicitly) during program initiation or termination after main) will result in undefined behavior.

+

As of CUDA 12.0, cudaSetDevice() will now explicitly initialize the runtime after changing the current device for the host thread. Previous versions of CUDA delayed runtime initialization on the new device until the first runtime call was made after cudaSetDevice(). This change means that it is now very important to check the return value of cudaSetDevice() for initialization errors.

+

The runtime functions from the error handling and version management sections of the reference manual do not initialize the runtime.

+
+
+
+

+6.2.2. Device Memory +

+

As mentioned in Heterogeneous Programming, the CUDA programming model assumes a system composed of a host and a device, each with their own separate memory. Kernels operate out of device memory, so the runtime provides functions to allocate, deallocate, and copy device memory, as well as transfer data between host memory and device memory.

+

Device memory can be allocated either as linear memory or as CUDA arrays.

+

CUDA arrays are opaque memory layouts optimized for texture fetching. They are described in Texture and Surface Memory.

+

Linear memory is allocated in a single unified address space, which means that separately allocated entities can reference one another via pointers, for example, in a binary tree or linked list. The size of the address space depends on the host system (CPU) and the compute capability of the used GPU:

+ + ++++++ + + + + + + + + + + + + + + + + + + + + + + +
+Table 4 Linear Memory Address Space +

x86_64 (AMD64)

POWER (ppc64le)

ARM64

up to compute capability 5.3 (Maxwell)

40bit

40bit

40bit

compute capability 6.0 (Pascal) or newer

up to 47bit

up to 49bit

up to 48bit

+
+

Note

+

On devices of compute capability 5.3 (Maxwell) and earlier, the CUDA driver creates an uncommitted 40bit virtual address reservation to ensure that memory allocations (pointers) fall into the supported range. This reservation appears as reserved virtual memory, but does not occupy any physical memory until the program actually allocates memory.

+
+

Linear memory is typically allocated using cudaMalloc() and freed using cudaFree() and data transfer between host memory and device memory are typically done using cudaMemcpy(). In the vector addition code sample of Kernels, the vectors need to be copied from host memory to device memory:

+
+
// Device code
+__global__ void VecAdd(float* A, float* B, float* C, int N)
+{
+    int i = blockDim.x * blockIdx.x + threadIdx.x;
+    if (i < N)
+        C[i] = A[i] + B[i];
+}
+
+// Host code
+int main()
+{
+    int N = ...;
+    size_t size = N * sizeof(float);
+
+    // Allocate input vectors h_A and h_B in host memory
+    float* h_A = (float*)malloc(size);
+    float* h_B = (float*)malloc(size);
+    float* h_C = (float*)malloc(size);
+
+    // Initialize input vectors
+    ...
+
+    // Allocate vectors in device memory
+    float* d_A;
+    cudaMalloc(&d_A, size);
+    float* d_B;
+    cudaMalloc(&d_B, size);
+    float* d_C;
+    cudaMalloc(&d_C, size);
+
+    // Copy vectors from host memory to device memory
+    cudaMemcpy(d_A, h_A, size, cudaMemcpyHostToDevice);
+    cudaMemcpy(d_B, h_B, size, cudaMemcpyHostToDevice);
+
+    // Invoke kernel
+    int threadsPerBlock = 256;
+    int blocksPerGrid =
+            (N + threadsPerBlock - 1) / threadsPerBlock;
+    VecAdd<<<blocksPerGrid, threadsPerBlock>>>(d_A, d_B, d_C, N);
+
+    // Copy result from device memory to host memory
+    // h_C contains the result in host memory
+    cudaMemcpy(h_C, d_C, size, cudaMemcpyDeviceToHost);
+
+    // Free device memory
+    cudaFree(d_A);
+    cudaFree(d_B);
+    cudaFree(d_C);
+
+    // Free host memory
+    ...
+}
+
+
+

Linear memory can also be allocated through cudaMallocPitch() and cudaMalloc3D(). These functions are recommended for allocations of 2D or 3D arrays as it makes sure that the allocation is appropriately padded to meet the alignment requirements described in Device Memory Accesses, therefore ensuring best performance when accessing the row addresses or performing copies between 2D arrays and other regions of device memory (using the cudaMemcpy2D() and cudaMemcpy3D() functions). The returned pitch (or stride) must be used to access array elements. The following code sample allocates a width x height 2D array of floating-point values and shows how to loop over the array elements in device code:

+
+
// Host code
+int width = 64, height = 64;
+float* devPtr;
+size_t pitch;
+cudaMallocPitch(&devPtr, &pitch,
+                width * sizeof(float), height);
+MyKernel<<<100, 512>>>(devPtr, pitch, width, height);
+
+// Device code
+__global__ void MyKernel(float* devPtr,
+                         size_t pitch, int width, int height)
+{
+    for (int r = 0; r < height; ++r) {
+        float* row = (float*)((char*)devPtr + r * pitch);
+        for (int c = 0; c < width; ++c) {
+            float element = row[c];
+        }
+    }
+}
+
+
+

The following code sample allocates a width x height x depth 3D array of floating-point values and shows how to loop over the array elements in device code:

+
+
// Host code
+int width = 64, height = 64, depth = 64;
+cudaExtent extent = make_cudaExtent(width * sizeof(float),
+                                    height, depth);
+cudaPitchedPtr devPitchedPtr;
+cudaMalloc3D(&devPitchedPtr, extent);
+MyKernel<<<100, 512>>>(devPitchedPtr, width, height, depth);
+
+// Device code
+__global__ void MyKernel(cudaPitchedPtr devPitchedPtr,
+                         int width, int height, int depth)
+{
+    char* devPtr = devPitchedPtr.ptr;
+    size_t pitch = devPitchedPtr.pitch;
+    size_t slicePitch = pitch * height;
+    for (int z = 0; z < depth; ++z) {
+        char* slice = devPtr + z * slicePitch;
+        for (int y = 0; y < height; ++y) {
+            float* row = (float*)(slice + y * pitch);
+            for (int x = 0; x < width; ++x) {
+                float element = row[x];
+            }
+        }
+    }
+}
+
+
+
+

Note

+

To avoid allocating too much memory and thus impacting system-wide performance, request the allocation parameters from the user based on the problem size. If the allocation fails, you can fallback to other slower memory types (cudaMallocHost(), cudaHostRegister(), etc.), or return an error telling the user how much memory was needed that was denied. If your application cannot request the allocation parameters for some reason, we recommend using cudaMallocManaged() for platforms that support it.

+
+

The reference manual lists all the various functions used to copy memory between linear memory allocated with cudaMalloc(), linear memory allocated with cudaMallocPitch() or cudaMalloc3D(), CUDA arrays, and memory allocated for variables declared in global or constant memory space.

+

The following code sample illustrates various ways of accessing global variables via the runtime API:

+
+
__constant__ float constData[256];
+float data[256];
+cudaMemcpyToSymbol(constData, data, sizeof(data));
+cudaMemcpyFromSymbol(data, constData, sizeof(data));
+
+__device__ float devData;
+float value = 3.14f;
+cudaMemcpyToSymbol(devData, &value, sizeof(float));
+
+__device__ float* devPointer;
+float* ptr;
+cudaMalloc(&ptr, 256 * sizeof(float));
+cudaMemcpyToSymbol(devPointer, &ptr, sizeof(ptr));
+
+
+

cudaGetSymbolAddress() is used to retrieve the address pointing to the memory allocated for a variable declared in global memory space. The size of the allocated memory is obtained through cudaGetSymbolSize().

+
+
+

+6.2.3. Device Memory L2 Access Management +

+

When a CUDA kernel accesses a data region in the global memory repeatedly, such data accesses can be considered to be persisting. On the other hand, if the data is only accessed once, such data accesses can be considered to be streaming.

+

Starting with CUDA 11.0, devices of compute capability 8.0 and above have the capability to influence persistence of data in the L2 cache, potentially providing higher bandwidth and lower latency accesses to global memory.

+
+

+6.2.3.1. L2 Cache Set-Aside for Persisting Accesses +

+

A portion of the L2 cache can be set aside to be used for persisting data accesses to global memory. Persisting accesses have prioritized use of this set-aside portion of L2 cache, whereas normal or streaming, accesses to global memory can only utilize this portion of L2 when it is unused by persisting accesses.

+

The L2 cache set-aside size for persisting accesses may be adjusted, within limits:

+
+
cudaGetDeviceProperties(&prop, device_id);
+size_t size = min(int(prop.l2CacheSize * 0.75), prop.persistingL2CacheMaxSize);
+cudaDeviceSetLimit(cudaLimitPersistingL2CacheSize, size); /* set-aside 3/4 of L2 cache for persisting accesses or the max allowed*/
+
+
+

When the GPU is configured in Multi-Instance GPU (MIG) mode, the L2 cache set-aside functionality is disabled.

+

When using the Multi-Process Service (MPS), the L2 cache set-aside size cannot be changed by cudaDeviceSetLimit. Instead, the set-aside size can only be specified at start up of MPS server through the environment variable CUDA_DEVICE_DEFAULT_PERSISTING_L2_CACHE_PERCENTAGE_LIMIT.

+
+
+

+6.2.3.2. L2 Policy for Persisting Accesses +

+

An access policy window specifies a contiguous region of global memory and a persistence property in the L2 cache for accesses within that region.

+

The code example below shows how to set an L2 persisting access window using a CUDA Stream.

+

CUDA Stream Example

+
+
cudaStreamAttrValue stream_attribute;                                         // Stream level attributes data structure
+stream_attribute.accessPolicyWindow.base_ptr  = reinterpret_cast<void*>(ptr); // Global Memory data pointer
+stream_attribute.accessPolicyWindow.num_bytes = num_bytes;                    // Number of bytes for persistence access.
+                                                                              // (Must be less than cudaDeviceProp::accessPolicyMaxWindowSize)
+stream_attribute.accessPolicyWindow.hitRatio  = 0.6;                          // Hint for cache hit ratio
+stream_attribute.accessPolicyWindow.hitProp   = cudaAccessPropertyPersisting; // Type of access property on cache hit
+stream_attribute.accessPolicyWindow.missProp  = cudaAccessPropertyStreaming;  // Type of access property on cache miss.
+
+//Set the attributes to a CUDA stream of type cudaStream_t
+cudaStreamSetAttribute(stream, cudaStreamAttributeAccessPolicyWindow, &stream_attribute);
+
+
+

When a kernel subsequently executes in CUDA stream, memory accesses within the global memory extent [ptr..ptr+num_bytes) are more likely to persist in the L2 cache than accesses to other global memory locations.

+

L2 persistence can also be set for a CUDA Graph Kernel Node as shown in the example below:

+

CUDA GraphKernelNode Example

+
+
cudaKernelNodeAttrValue node_attribute;                                     // Kernel level attributes data structure
+node_attribute.accessPolicyWindow.base_ptr  = reinterpret_cast<void*>(ptr); // Global Memory data pointer
+node_attribute.accessPolicyWindow.num_bytes = num_bytes;                    // Number of bytes for persistence access.
+                                                                            // (Must be less than cudaDeviceProp::accessPolicyMaxWindowSize)
+node_attribute.accessPolicyWindow.hitRatio  = 0.6;                          // Hint for cache hit ratio
+node_attribute.accessPolicyWindow.hitProp   = cudaAccessPropertyPersisting; // Type of access property on cache hit
+node_attribute.accessPolicyWindow.missProp  = cudaAccessPropertyStreaming;  // Type of access property on cache miss.
+
+//Set the attributes to a CUDA Graph Kernel node of type cudaGraphNode_t
+cudaGraphKernelNodeSetAttribute(node, cudaKernelNodeAttributeAccessPolicyWindow, &node_attribute);
+
+
+

The hitRatio parameter can be used to specify the fraction of accesses that receive the hitProp property. In both of the examples above, 60% of the memory accesses in the global memory region [ptr..ptr+num_bytes) have the persisting property and 40% of the memory accesses have the streaming property. Which specific memory accesses are classified as persisting (the hitProp) is random with a probability of approximately hitRatio; the probability distribution depends upon the hardware architecture and the memory extent.

+

For example, if the L2 set-aside cache size is 16KB and the num_bytes in the accessPolicyWindow is 32KB:

+
    +
  • With a hitRatio of 0.5, the hardware will select, at random, 16KB of the 32KB window to be designated as persisting and cached in the set-aside L2 cache area.

  • +
  • With a hitRatio of 1.0, the hardware will attempt to cache the whole 32KB window in the set-aside L2 cache area. Since the set-aside area is smaller than the window, cache lines will be evicted to keep the most recently used 16KB of the 32KB data in the set-aside portion of the L2 cache.

  • +
+

The hitRatio can therefore be used to avoid thrashing of cache lines and overall reduce the amount of data moved into and out of the L2 cache.

+

A hitRatio value below 1.0 can be used to manually control the amount of data different accessPolicyWindows from concurrent CUDA streams can cache in L2. For example, let the L2 set-aside cache size be 16KB; two concurrent kernels in two different CUDA streams, each with a 16KB accessPolicyWindow, and both with hitRatio value 1.0, might evict each others’ cache lines when competing for the shared L2 resource. However, if both accessPolicyWindows have a hitRatio value of 0.5, they will be less likely to evict their own or each others’ persisting cache lines.

+
+
+

+6.2.3.3. L2 Access Properties +

+

Three types of access properties are defined for different global memory data accesses:

+
    +
  1. cudaAccessPropertyStreaming: Memory accesses that occur with the streaming property are less likely to persist in the L2 cache because these accesses are preferentially evicted.

  2. +
  3. cudaAccessPropertyPersisting: Memory accesses that occur with the persisting property are more likely to persist in the L2 cache because these accesses are preferentially retained in the set-aside portion of L2 cache.

  4. +
  5. cudaAccessPropertyNormal: This access property forcibly resets previously applied persisting access property to a normal status. Memory accesses with the persisting property from previous CUDA kernels may be retained in L2 cache long after their intended use. This persistence-after-use reduces the amount of L2 cache available to subsequent kernels that do not use the persisting property. Resetting an access property window with the cudaAccessPropertyNormal property removes the persisting (preferential retention) status of the prior access, as if the prior access had been without an access property.

  6. +
+
+
+

+6.2.3.4. L2 Persistence Example +

+

The following example shows how to set-aside L2 cache for persistent accesses, use the set-aside L2 cache in CUDA kernels via CUDA Stream and then reset the L2 cache.

+
+
cudaStream_t stream;
+cudaStreamCreate(&stream);                                                                  // Create CUDA stream
+
+cudaDeviceProp prop;                                                                        // CUDA device properties variable
+cudaGetDeviceProperties( &prop, device_id);                                                 // Query GPU properties
+size_t size = min( int(prop.l2CacheSize * 0.75) , prop.persistingL2CacheMaxSize );
+cudaDeviceSetLimit( cudaLimitPersistingL2CacheSize, size);                                  // set-aside 3/4 of L2 cache for persisting accesses or the max allowed
+
+size_t window_size = min(prop.accessPolicyMaxWindowSize, num_bytes);                        // Select minimum of user defined num_bytes and max window size.
+
+cudaStreamAttrValue stream_attribute;                                                       // Stream level attributes data structure
+stream_attribute.accessPolicyWindow.base_ptr  = reinterpret_cast<void*>(data1);               // Global Memory data pointer
+stream_attribute.accessPolicyWindow.num_bytes = window_size;                                // Number of bytes for persistence access
+stream_attribute.accessPolicyWindow.hitRatio  = 0.6;                                        // Hint for cache hit ratio
+stream_attribute.accessPolicyWindow.hitProp   = cudaAccessPropertyPersisting;               // Persistence Property
+stream_attribute.accessPolicyWindow.missProp  = cudaAccessPropertyStreaming;                // Type of access property on cache miss
+
+cudaStreamSetAttribute(stream, cudaStreamAttributeAccessPolicyWindow, &stream_attribute);   // Set the attributes to a CUDA Stream
+
+for(int i = 0; i < 10; i++) {
+    cuda_kernelA<<<grid_size,block_size,0,stream>>>(data1);                                 // This data1 is used by a kernel multiple times
+}                                                                                           // [data1 + num_bytes) benefits from L2 persistence
+cuda_kernelB<<<grid_size,block_size,0,stream>>>(data1);                                     // A different kernel in the same stream can also benefit
+                                                                                            // from the persistence of data1
+
+stream_attribute.accessPolicyWindow.num_bytes = 0;                                          // Setting the window size to 0 disable it
+cudaStreamSetAttribute(stream, cudaStreamAttributeAccessPolicyWindow, &stream_attribute);   // Overwrite the access policy attribute to a CUDA Stream
+cudaCtxResetPersistingL2Cache();                                                            // Remove any persistent lines in L2
+
+cuda_kernelC<<<grid_size,block_size,0,stream>>>(data2);                                     // data2 can now benefit from full L2 in normal mode
+
+
+
+
+

+6.2.3.5. Reset L2 Access to Normal +

+

A persisting L2 cache line from a previous CUDA kernel may persist in L2 long after it has been used. Hence, a reset to normal for L2 cache is important for streaming or normal memory accesses to utilize the L2 cache with normal priority. There are three ways a persisting access can be reset to normal status.

+
    +
  1. Reset a previous persisting memory region with the access property, cudaAccessPropertyNormal.

  2. +
  3. Reset all persisting L2 cache lines to normal by calling cudaCtxResetPersistingL2Cache().

  4. +
  5. Eventually untouched lines are automatically reset to normal. Reliance on automatic reset is strongly discouraged because of the undetermined length of time required for automatic reset to occur.

  6. +
+
+
+

+6.2.3.6. Manage Utilization of L2 set-aside cache +

+

Multiple CUDA kernels executing concurrently in different CUDA streams may have a different access policy window assigned to their streams. However, the L2 set-aside cache portion is shared among all these concurrent CUDA kernels. As a result, the net utilization of this set-aside cache portion is the sum of all the concurrent kernels’ individual use. The benefits of designating memory accesses as persisting diminish as the volume of persisting accesses exceeds the set-aside L2 cache capacity.

+

To manage utilization of the set-aside L2 cache portion, an application must consider the following:

+
    +
  • Size of L2 set-aside cache.

  • +
  • CUDA kernels that may concurrently execute.

  • +
  • The access policy window for all the CUDA kernels that may concurrently execute.

  • +
  • When and how L2 reset is required to allow normal or streaming accesses to utilize the previously set-aside L2 cache with equal priority.

  • +
+
+
+

+6.2.3.7. Query L2 cache Properties +

+

Properties related to L2 cache are a part of cudaDeviceProp struct and can be queried using CUDA runtime API cudaGetDeviceProperties

+

CUDA Device Properties include:

+
    +
  • l2CacheSize: The amount of available L2 cache on the GPU.

  • +
  • persistingL2CacheMaxSize: The maximum amount of L2 cache that can be set-aside for persisting memory accesses.

  • +
  • accessPolicyMaxWindowSize: The maximum size of the access policy window.

  • +
+
+
+

+6.2.3.8. Control L2 Cache Set-Aside Size for Persisting Memory Access +

+

The L2 set-aside cache size for persisting memory accesses is queried using CUDA runtime API cudaDeviceGetLimit and set using CUDA runtime API cudaDeviceSetLimit as a cudaLimit. The maximum value for setting this limit is cudaDeviceProp::persistingL2CacheMaxSize.

+
+
enum cudaLimit {
+    /* other fields not shown */
+    cudaLimitPersistingL2CacheSize
+};
+
+
+
+
+
+

+6.2.4. Shared Memory +

+

As detailed in Variable Memory Space Specifiers shared memory is allocated using the __shared__ memory space specifier.

+

Shared memory is expected to be much faster than global memory as mentioned in Thread Hierarchy and detailed in Shared Memory. It can be used as scratchpad memory (or software managed cache) to minimize global memory accesses from a CUDA block as illustrated by the following matrix multiplication example.

+

The following code sample is a straightforward implementation of matrix multiplication that does not take advantage of shared memory. Each thread reads one row of A and one column of B and computes the corresponding element of C as illustrated in Figure 8. A is therefore read B.width times from global memory and B is read A.height times.

+
+
// Matrices are stored in row-major order:
+// M(row, col) = *(M.elements + row * M.width + col)
+typedef struct {
+    int width;
+    int height;
+    float* elements;
+} Matrix;
+
+// Thread block size
+#define BLOCK_SIZE 16
+
+// Forward declaration of the matrix multiplication kernel
+__global__ void MatMulKernel(const Matrix, const Matrix, Matrix);
+
+// Matrix multiplication - Host code
+// Matrix dimensions are assumed to be multiples of BLOCK_SIZE
+void MatMul(const Matrix A, const Matrix B, Matrix C)
+{
+    // Load A and B to device memory
+    Matrix d_A;
+    d_A.width = A.width; d_A.height = A.height;
+    size_t size = A.width * A.height * sizeof(float);
+    cudaMalloc(&d_A.elements, size);
+    cudaMemcpy(d_A.elements, A.elements, size,
+               cudaMemcpyHostToDevice);
+    Matrix d_B;
+    d_B.width = B.width; d_B.height = B.height;
+    size = B.width * B.height * sizeof(float);
+    cudaMalloc(&d_B.elements, size);
+    cudaMemcpy(d_B.elements, B.elements, size,
+               cudaMemcpyHostToDevice);
+
+    // Allocate C in device memory
+    Matrix d_C;
+    d_C.width = C.width; d_C.height = C.height;
+    size = C.width * C.height * sizeof(float);
+    cudaMalloc(&d_C.elements, size);
+
+    // Invoke kernel
+    dim3 dimBlock(BLOCK_SIZE, BLOCK_SIZE);
+    dim3 dimGrid(B.width / dimBlock.x, A.height / dimBlock.y);
+    MatMulKernel<<<dimGrid, dimBlock>>>(d_A, d_B, d_C);
+
+    // Read C from device memory
+    cudaMemcpy(C.elements, d_C.elements, size,
+               cudaMemcpyDeviceToHost);
+
+    // Free device memory
+    cudaFree(d_A.elements);
+    cudaFree(d_B.elements);
+    cudaFree(d_C.elements);
+}
+
+// Matrix multiplication kernel called by MatMul()
+__global__ void MatMulKernel(Matrix A, Matrix B, Matrix C)
+{
+    // Each thread computes one element of C
+    // by accumulating results into Cvalue
+    float Cvalue = 0;
+    int row = blockIdx.y * blockDim.y + threadIdx.y;
+    int col = blockIdx.x * blockDim.x + threadIdx.x;
+    for (int e = 0; e < A.width; ++e)
+        Cvalue += A.elements[row * A.width + e]
+                * B.elements[e * B.width + col];
+    C.elements[row * C.width + col] = Cvalue;
+}
+
+
+
+_images/matrix-multiplication-without-shared-memory.png +
+

Figure 8 Matrix Multiplication without Shared Memory

+
+
+

The following code sample is an implementation of matrix multiplication that does take advantage of shared memory. In this implementation, each thread block is responsible for computing one square sub-matrix Csub of C and each thread within the block is responsible for computing one element of Csub. As illustrated in Figure 9, Csub is equal to the product of two rectangular matrices: the sub-matrix of A of dimension (A.width, block_size) that has the same row indices as Csub, and the sub-matrix of B of dimension (block_size, A.width )that has the same column indices as Csub. In order to fit into the device’s resources, these two rectangular matrices are divided into as many square matrices of dimension block_size as necessary and Csub is computed as the sum of the products of these square matrices. Each of these products is performed by first loading the two corresponding square matrices from global memory to shared memory with one thread loading one element of each matrix, and then by having each thread compute one element of the product. Each thread accumulates the result of each of these products into a register and once done writes the result to global memory.

+

By blocking the computation this way, we take advantage of fast shared memory and save a lot of global memory bandwidth since A is only read (B.width / block_size) times from global memory and B is read (A.height / block_size) times.

+

The Matrix type from the previous code sample is augmented with a stride field, so that sub-matrices can be efficiently represented with the same type. __device__ functions are used to get and set elements and build any sub-matrix from a matrix.

+
+
// Matrices are stored in row-major order:
+// M(row, col) = *(M.elements + row * M.stride + col)
+typedef struct {
+    int width;
+    int height;
+    int stride;
+    float* elements;
+} Matrix;
+// Get a matrix element
+__device__ float GetElement(const Matrix A, int row, int col)
+{
+    return A.elements[row * A.stride + col];
+}
+// Set a matrix element
+__device__ void SetElement(Matrix A, int row, int col,
+                           float value)
+{
+    A.elements[row * A.stride + col] = value;
+}
+// Get the BLOCK_SIZExBLOCK_SIZE sub-matrix Asub of A that is
+// located col sub-matrices to the right and row sub-matrices down
+// from the upper-left corner of A
+ __device__ Matrix GetSubMatrix(Matrix A, int row, int col)
+{
+    Matrix Asub;
+    Asub.width    = BLOCK_SIZE;
+    Asub.height   = BLOCK_SIZE;
+    Asub.stride   = A.stride;
+    Asub.elements = &A.elements[A.stride * BLOCK_SIZE * row
+                                         + BLOCK_SIZE * col];
+    return Asub;
+}
+// Thread block size
+#define BLOCK_SIZE 16
+// Forward declaration of the matrix multiplication kernel
+__global__ void MatMulKernel(const Matrix, const Matrix, Matrix);
+// Matrix multiplication - Host code
+// Matrix dimensions are assumed to be multiples of BLOCK_SIZE
+void MatMul(const Matrix A, const Matrix B, Matrix C)
+{
+    // Load A and B to device memory
+    Matrix d_A;
+    d_A.width = d_A.stride = A.width; d_A.height = A.height;
+    size_t size = A.width * A.height * sizeof(float);
+    cudaMalloc(&d_A.elements, size);
+    cudaMemcpy(d_A.elements, A.elements, size,
+               cudaMemcpyHostToDevice);
+    Matrix d_B;
+    d_B.width = d_B.stride = B.width; d_B.height = B.height;
+    size = B.width * B.height * sizeof(float);
+    cudaMalloc(&d_B.elements, size);
+    cudaMemcpy(d_B.elements, B.elements, size,
+    cudaMemcpyHostToDevice);
+    // Allocate C in device memory
+    Matrix d_C;
+    d_C.width = d_C.stride = C.width; d_C.height = C.height;
+    size = C.width * C.height * sizeof(float);
+    cudaMalloc(&d_C.elements, size);
+    // Invoke kernel
+    dim3 dimBlock(BLOCK_SIZE, BLOCK_SIZE);
+    dim3 dimGrid(B.width / dimBlock.x, A.height / dimBlock.y);
+    MatMulKernel<<<dimGrid, dimBlock>>>(d_A, d_B, d_C);
+    // Read C from device memory
+    cudaMemcpy(C.elements, d_C.elements, size,
+               cudaMemcpyDeviceToHost);
+    // Free device memory
+    cudaFree(d_A.elements);
+    cudaFree(d_B.elements);
+    cudaFree(d_C.elements);
+}
+// Matrix multiplication kernel called by MatMul()
+ __global__ void MatMulKernel(Matrix A, Matrix B, Matrix C)
+{
+    // Block row and column
+    int blockRow = blockIdx.y;
+    int blockCol = blockIdx.x;
+    // Each thread block computes one sub-matrix Csub of C
+    Matrix Csub = GetSubMatrix(C, blockRow, blockCol);
+    // Each thread computes one element of Csub
+    // by accumulating results into Cvalue
+    float Cvalue = 0;
+    // Thread row and column within Csub
+    int row = threadIdx.y;
+    int col = threadIdx.x;
+    // Loop over all the sub-matrices of A and B that are
+    // required to compute Csub
+    // Multiply each pair of sub-matrices together
+    // and accumulate the results
+    for (int m = 0; m < (A.width / BLOCK_SIZE); ++m) {
+        // Get sub-matrix Asub of A
+        Matrix Asub = GetSubMatrix(A, blockRow, m);
+        // Get sub-matrix Bsub of B
+        Matrix Bsub = GetSubMatrix(B, m, blockCol);
+        // Shared memory used to store Asub and Bsub respectively
+        __shared__ float As[BLOCK_SIZE][BLOCK_SIZE];
+        __shared__ float Bs[BLOCK_SIZE][BLOCK_SIZE];
+        // Load Asub and Bsub from device memory to shared memory
+        // Each thread loads one element of each sub-matrix
+        As[row][col] = GetElement(Asub, row, col);
+        Bs[row][col] = GetElement(Bsub, row, col);
+        // Synchronize to make sure the sub-matrices are loaded
+        // before starting the computation
+        __syncthreads();
+        // Multiply Asub and Bsub together
+        for (int e = 0; e < BLOCK_SIZE; ++e)
+            Cvalue += As[row][e] * Bs[e][col];
+        // Synchronize to make sure that the preceding
+        // computation is done before loading two new
+        // sub-matrices of A and B in the next iteration
+        __syncthreads();
+    }
+    // Write Csub to device memory
+    // Each thread writes one element
+    SetElement(Csub, row, col, Cvalue);
+}
+
+
+
+_images/matrix-multiplication-with-shared-memory.png +
+

Figure 9 Matrix Multiplication with Shared Memory

+
+
+
+
+

+6.2.5. Distributed Shared Memory +

+

Thread block clusters introduced in compute capability 9.0 provide the ability for threads in a thread block cluster to access shared memory of all the participating thread blocks in a cluster. This partitioned shared memory is called Distributed Shared Memory, and the corresponding address space is called Distributed shared memory address space. Threads that belong to a thread block cluster, can read, write or perform atomics in the distributed address space, regardless whether the address belongs to the local thread block or a remote thread block. Whether a kernel uses distributed shared memory or not, the shared memory size specifications, static or dynamic is still per thread block. The size of distributed shared memory is just the number of thread blocks per cluster multiplied by the size of shared memory per thread block.

+

Accessing data in distributed shared memory requires all the thread blocks to exist. A user can guarantee that all thread blocks have started executing using cluster.sync() from Cluster Group API. +The user also needs to ensure that all distributed shared memory operations happen before the exit of a thread block, e.g., if a remote thread block is trying to read a given thread block’s shared memory, user needs to ensure that the shared memory read by remote thread block is completed before it can exit.

+

CUDA provides a mechanism to access to distributed shared memory, and applications can benefit from leveraging its capabilities. Lets look at a simple histogram computation and how to optimize it on the GPU using thread block cluster. A standard way of computing histograms is do the computation in the shared memory of each thread block and then perform global memory atomics. A limitation of this approach is the shared memory capacity. Once the histogram bins no longer fit in the shared memory, a user needs to directly compute histograms and hence the atomics in the global memory. With distributed shared memory, CUDA provides an intermediate step, where a depending on the histogram bins size, histogram can be computed in shared memory, distributed shared memory or global memory directly.

+

The CUDA kernel example below shows how to compute histograms in shared memory or distributed shared memory, depending on the number of histogram bins.

+
+
#include <cooperative_groups.h>
+
+// Distributed Shared memory histogram kernel
+__global__ void clusterHist_kernel(int *bins, const int nbins, const int bins_per_block, const int *__restrict__ input,
+                                   size_t array_size)
+{
+  extern __shared__ int smem[];
+  namespace cg = cooperative_groups;
+  int tid = cg::this_grid().thread_rank();
+
+  // Cluster initialization, size and calculating local bin offsets.
+  cg::cluster_group cluster = cg::this_cluster();
+  unsigned int clusterBlockRank = cluster.block_rank();
+  int cluster_size = cluster.dim_blocks().x;
+
+  for (int i = threadIdx.x; i < bins_per_block; i += blockDim.x)
+  {
+    smem[i] = 0; //Initialize shared memory histogram to zeros
+  }
+
+  // cluster synchronization ensures that shared memory is initialized to zero in
+  // all thread blocks in the cluster. It also ensures that all thread blocks
+  // have started executing and they exist concurrently.
+  cluster.sync();
+
+  for (int i = tid; i < array_size; i += blockDim.x * gridDim.x)
+  {
+    int ldata = input[i];
+
+    //Find the right histogram bin.
+    int binid = ldata;
+    if (ldata < 0)
+      binid = 0;
+    else if (ldata >= nbins)
+      binid = nbins - 1;
+
+    //Find destination block rank and offset for computing
+    //distributed shared memory histogram
+    int dst_block_rank = (int)(binid / bins_per_block);
+    int dst_offset = binid % bins_per_block;
+
+    //Pointer to target block shared memory
+    int *dst_smem = cluster.map_shared_rank(smem, dst_block_rank);
+
+    //Perform atomic update of the histogram bin
+    atomicAdd(dst_smem + dst_offset, 1);
+  }
+
+  // cluster synchronization is required to ensure all distributed shared
+  // memory operations are completed and no thread block exits while
+  // other thread blocks are still accessing distributed shared memory
+  cluster.sync();
+
+  // Perform global memory histogram, using the local distributed memory histogram
+  int *lbins = bins + cluster.block_rank() * bins_per_block;
+  for (int i = threadIdx.x; i < bins_per_block; i += blockDim.x)
+  {
+    atomicAdd(&lbins[i], smem[i]);
+  }
+}
+
+
+

The above kernel can be launched at runtime with a cluster size depending on the amount of distributed shared memory required. If histogram is small enough to fit in shared memory of just one block, user can launch kernel with cluster size 1. The code snippet below shows how to launch a cluster kernel dynamically based depending on shared memory requirements.

+
+
// Launch via extensible launch
+{
+  cudaLaunchConfig_t config = {0};
+  config.gridDim = array_size / threads_per_block;
+  config.blockDim = threads_per_block;
+
+  // cluster_size depends on the histogram size.
+  // ( cluster_size == 1 ) implies no distributed shared memory, just thread block local shared memory
+  int cluster_size = 2; // size 2 is an example here
+  int nbins_per_block = nbins / cluster_size;
+
+  //dynamic shared memory size is per block.
+  //Distributed shared memory size =  cluster_size * nbins_per_block * sizeof(int)
+  config.dynamicSmemBytes = nbins_per_block * sizeof(int);
+
+  CUDA_CHECK(::cudaFuncSetAttribute((void *)clusterHist_kernel, cudaFuncAttributeMaxDynamicSharedMemorySize, config.dynamicSmemBytes));
+
+  cudaLaunchAttribute attribute[1];
+  attribute[0].id = cudaLaunchAttributeClusterDimension;
+  attribute[0].val.clusterDim.x = cluster_size;
+  attribute[0].val.clusterDim.y = 1;
+  attribute[0].val.clusterDim.z = 1;
+
+  config.numAttrs = 1;
+  config.attrs = attribute;
+
+  cudaLaunchKernelEx(&config, clusterHist_kernel, bins, nbins, nbins_per_block, input, array_size);
+}
+
+
+
+
+

+6.2.6. Page-Locked Host Memory +

+

The runtime provides functions to allow the use of page-locked (also known as pinned) host memory (as opposed to regular pageable host memory allocated by malloc()):

+
    +
  • cudaHostAlloc() and cudaFreeHost() allocate and free page-locked host memory;

  • +
  • cudaHostRegister() page-locks a range of memory allocated by malloc() (see reference manual for limitations).

  • +
+

Using page-locked host memory has several benefits:

+
    +
  • Copies between page-locked host memory and device memory can be performed concurrently with kernel execution for some devices as mentioned in Asynchronous Concurrent Execution.

  • +
  • On some devices, page-locked host memory can be mapped into the address space of the device, eliminating the need to copy it to or from device memory as detailed in Mapped Memory.

  • +
  • On systems with a front-side bus, bandwidth between host memory and device memory is higher if host memory is allocated as page-locked and even higher if in addition it is allocated as write-combining as described in Write-Combining Memory.

  • +
+
+

Note

+

Page-locked host memory is not cached on non I/O coherent Tegra devices. Also, cudaHostRegister() is not supported on non I/O coherent Tegra devices.

+
+

The simple zero-copy CUDA sample comes with a detailed document on the page-locked memory APIs.

+
+

+6.2.6.1. Portable Memory +

+

A block of page-locked memory can be used in conjunction with any device in the system (see Multi-Device System for more details on multi-device systems), but by default, the benefits of using page-locked memory described above are only available in conjunction with the device that was current when the block was allocated (and with all devices sharing the same unified address space, if any, as described in Unified Virtual Address Space). To make these advantages available to all devices, the block needs to be allocated by passing the flag cudaHostAllocPortable to cudaHostAlloc() or page-locked by passing the flag cudaHostRegisterPortable to cudaHostRegister().

+
+
+

+6.2.6.2. Write-Combining Memory +

+

By default page-locked host memory is allocated as cacheable. It can optionally be allocated as write-combining instead by passing flag cudaHostAllocWriteCombined to cudaHostAlloc(). Write-combining memory frees up the host’s L1 and L2 cache resources, making more cache available to the rest of the application. In addition, write-combining memory is not snooped during transfers across the PCI Express bus, which can improve transfer performance by up to 40%.

+

Reading from write-combining memory from the host is prohibitively slow, so write-combining memory should in general be used for memory that the host only writes to.

+

Using CPU atomic instructions on WC memory should be avoided because not all CPU implementations guarantee that functionality.

+
+
+

+6.2.6.3. Mapped Memory +

+

A block of page-locked host memory can also be mapped into the address space of the device by passing flag cudaHostAllocMapped to cudaHostAlloc() or by passing flag cudaHostRegisterMapped to cudaHostRegister(). Such a block has therefore in general two addresses: one in host memory that is returned by cudaHostAlloc() or malloc(), and one in device memory that can be retrieved using cudaHostGetDevicePointer() and then used to access the block from within a kernel. The only exception is for pointers allocated with cudaHostAlloc() and when a unified address space is used for the host and the device as mentioned in Unified Virtual Address Space.

+

Accessing host memory directly from within a kernel does not provide the same bandwidth as device memory, but does have some advantages:

+
    +
  • There is no need to allocate a block in device memory and copy data between this block and the block in host memory; data transfers are implicitly performed as needed by the kernel;

  • +
  • There is no need to use streams (see Concurrent Data Transfers) to overlap data transfers with kernel execution; the kernel-originated data transfers automatically overlap with kernel execution.

  • +
+

Since mapped page-locked memory is shared between host and device however, the application must synchronize memory accesses using streams or events (see Asynchronous Concurrent Execution) to avoid any potential read-after-write, write-after-read, or write-after-write hazards.

+

To be able to retrieve the device pointer to any mapped page-locked memory, page-locked memory mapping must be enabled by calling cudaSetDeviceFlags() with the cudaDeviceMapHost flag before any other CUDA call is performed. Otherwise, cudaHostGetDevicePointer() will return an error.

+

cudaHostGetDevicePointer() also returns an error if the device does not support mapped page-locked host memory. Applications may query this capability by checking the canMapHostMemory device property (see Device Enumeration), which is equal to 1 for devices that support mapped page-locked host memory.

+

Note that atomic functions (see Atomic Functions) operating on mapped page-locked memory are not atomic from the point of view of the host or other devices.

+

Also note that CUDA runtime requires that 1-byte, 2-byte, 4-byte, 8-byte, and 16-byte naturally aligned +loads and stores to host memory initiated from the device are preserved as single accesses +from the point of view of the host and other devices. +On some platforms, atomics to memory may be broken by the hardware into separate +load and store operations. +These component load and store operations have the same requirements on preservation of naturally aligned accesses. +The CUDA runtime does not support a PCI Express bus topology where a PCI Express bridge splits +8-byte naturally aligned operations and NVIDIA is not aware of any topology that splits +16-byte naturally aligned operations.

+
+
+
+

+6.2.7. Memory Synchronization Domains +

+
+

+6.2.7.1. Memory Fence Interference +

+

Some CUDA applications may see degraded performance due to memory fence/flush operations waiting on more transactions than those necessitated by the CUDA memory consistency model.

+ +++++ + + + + + + + + + + + + +
+
+
__managed__ int x = 0;
+__device__  cuda::atomic<int, cuda::thread_scope_device> a(0);
+__managed__ cuda::atomic<int, cuda::thread_scope_system> b(0);
+
+
+
+

Thread 1 (SM)

+
+
x = 1;
+a = 1;
+
+
+
+

Thread 2 (SM)

+
+
while (a != 1) ;
+assert(x == 1);
+b = 1;
+
+
+
+

Thread 3 (CPU)

+
+
while (b != 1) ;
+assert(x == 1);
+
+
+
+

Consider the example above. The CUDA memory consistency model guarantees that the asserted condition will be true, so the write to x from thread 1 must be visible to thread 3, before the write to b from thread 2.

+

The memory ordering provided by the release and acquire of a is only sufficient to make x visible to thread 2, not thread 3, as it is a device-scope operation. The system-scope ordering provided by release and acquire of b, therefore, needs to ensure not only writes issued from thread 2 itself are visible to thread 3, but also writes from other threads that are visible to thread 2. This is known as cumulativity. As the GPU cannot know at the time of execution which writes have been guaranteed at the source level to be visible and which are visible only by chance timing, it must cast a conservatively wide net for in-flight memory operations.

+

This sometimes leads to interference: because the GPU is waiting on memory operations it is not required to at the source level, the fence/flush may take longer than necessary.

+

Note that fences may occur explicitly as intrinsics or atomics in code, like in the example, or implicitly to implement synchronizes-with relationships at task boundaries.

+

A common example is when a kernel is performing computation in local GPU memory, and a parallel kernel (e.g. from NCCL) is performing communications with a peer. Upon completion, the local kernel will implicitly flush its writes to satisfy any synchronizes-with relationships to downstream work. This may unnecessarily wait, fully or partially, on slower nvlink or PCIe writes from the communication kernel.

+
+
+

+6.2.7.2. Isolating Traffic with Domains +

+

Beginning with Hopper architecture GPUs and CUDA 12.0, the memory synchronization domains feature provides a way to alleviate such interference. In exchange for explicit assistance from code, the GPU can reduce the net cast by a fence operation. Each kernel launch is given a domain ID. Writes and fences are tagged with the ID, and a fence will only order writes matching the fence’s domain. In the concurrent compute vs communication example, the communication kernels can be placed in a different domain.

+

When using domains, code must abide by the rule that ordering or synchronization between distinct domains on the same GPU requires system-scope fencing. Within a domain, device-scope fencing remains sufficient. This is necessary for cumulativity as one kernel’s writes will not be encompassed by a fence issued from a kernel in another domain. In essence, cumulativity is satisfied by ensuring that cross-domain traffic is flushed to the system scope ahead of time.

+

Note that this modifies the definition of thread_scope_device. However, because kernels will default to domain 0 as described below, backward compatibility is maintained.

+
+
+

+6.2.7.3. Using Domains in CUDA +

+

Domains are accessible via the new launch attributes cudaLaunchAttributeMemSyncDomain and cudaLaunchAttributeMemSyncDomainMap. The former selects between logical domains cudaLaunchMemSyncDomainDefault and cudaLaunchMemSyncDomainRemote, and the latter provides a mapping from logical to physical domains. The remote domain is intended for kernels performing remote memory access in order to isolate their memory traffic from local kernels. Note, however, the selection of a particular domain does not affect what memory access a kernel may legally perform.

+

The domain count can be queried via device attribute cudaDevAttrMemSyncDomainCount. Hopper has 4 domains. To facilitate portable code, domains functionality can be used on all devices and CUDA will report a count of 1 prior to Hopper.

+

Having logical domains eases application composition. An individual kernel launch at a low level in the stack, such as from NCCL, can select a semantic logical domain without concern for the surrounding application architecture. Higher levels can steer logical domains using the mapping. The default value for the logical domain if it is not set is the default domain, and the default mapping is to map the default domain to 0 and the remote domain to 1 (on GPUs with more than 1 domain). Specific libraries may tag launches with the remote domain in CUDA 12.0 and later; for example, NCCL 2.16 will do so. Together, this provides a beneficial use pattern for common applications out of the box, with no code changes needed in other components, frameworks, or at application level. An alternative use pattern, for example in an application using nvshmem or with no clear separation of kernel types, could be to partition parallel streams. Stream A may map both logical domains to physical domain 0, stream B to 1, and so on.

+
+
// Example of launching a kernel with the remote logical domain
+cudaLaunchAttribute domainAttr;
+domainAttr.id = cudaLaunchAttrMemSyncDomain;
+domainAttr.val = cudaLaunchMemSyncDomainRemote;
+cudaLaunchConfig_t config;
+// Fill out other config fields
+config.attrs = &domainAttr;
+config.numAttrs = 1;
+cudaLaunchKernelEx(&config, myKernel, kernelArg1, kernelArg2...);
+
+
+
+
// Example of setting a mapping for a stream
+// (This mapping is the default for streams starting on Hopper if not
+// explicitly set, and provided for illustration)
+cudaLaunchAttributeValue mapAttr;
+mapAttr.memSyncDomainMap.default_ = 0;
+mapAttr.memSyncDomainMap.remote = 1;
+cudaStreamSetAttribute(stream, cudaLaunchAttributeMemSyncDomainMap, &mapAttr);
+
+
+
+
// Example of mapping different streams to different physical domains, ignoring
+// logical domain settings
+cudaLaunchAttributeValue mapAttr;
+mapAttr.memSyncDomainMap.default_ = 0;
+mapAttr.memSyncDomainMap.remote = 0;
+cudaStreamSetAttribute(streamA, cudaLaunchAttributeMemSyncDomainMap, &mapAttr);
+mapAttr.memSyncDomainMap.default_ = 1;
+mapAttr.memSyncDomainMap.remote = 1;
+cudaStreamSetAttribute(streamB, cudaLaunchAttributeMemSyncDomainMap, &mapAttr);
+
+
+

As with other launch attributes, these are exposed uniformly on CUDA streams, individual launches using cudaLaunchKernelEx, and kernel nodes in CUDA graphs. A typical use would set the mapping at stream level and the logical domain at launch level (or bracketing a section of stream use) as described above.

+

Both attributes are copied to graph nodes during stream capture. Graphs take both attributes from the node itself, essentially an indirect way of specifying a physical domain. Domain-related attributes set on the stream a graph is launched into are not used in execution of the graph.

+
+
+
+

+6.2.8. Asynchronous Concurrent Execution +

+

CUDA exposes the following operations as independent tasks that can operate concurrently with one another:

+
    +
  • Computation on the host;

  • +
  • Computation on the device;

  • +
  • Memory transfers from the host to the device;

  • +
  • Memory transfers from the device to the host;

  • +
  • Memory transfers within the memory of a given device;

  • +
  • Memory transfers among devices.

  • +
+

The level of concurrency achieved between these operations will depend on the feature set and compute capability of the device as described below.

+
+

+6.2.8.1. Concurrent Execution between Host and Device +

+

Concurrent host execution is facilitated through asynchronous library functions that return control to the host thread before the device completes the requested task. Using asynchronous calls, many device operations can be queued up together to be executed by the CUDA driver when appropriate device resources are available. This relieves the host thread of much of the responsibility to manage the device, leaving it free for other tasks. The following device operations are asynchronous with respect to the host:

+
    +
  • Kernel launches;

  • +
  • Memory copies within a single device’s memory;

  • +
  • Memory copies from host to device of a memory block of 64 KB or less;

  • +
  • Memory copies performed by functions that are suffixed with Async;

  • +
  • Memory set function calls.

  • +
+

Programmers can globally disable asynchronicity of kernel launches for all CUDA applications running on a system by setting the CUDA_LAUNCH_BLOCKING environment variable to 1. This feature is provided for debugging purposes only and should not be used as a way to make production software run reliably.

+

Kernel launches are synchronous if hardware counters are collected via a profiler (Nsight Compute) unless concurrent kernel profiling is enabled. Async memory copies might also be synchronous if they involve host memory that is not page-locked.

+
+
+

+6.2.8.2. Concurrent Kernel Execution +

+

Some devices of compute capability 2.x and higher can execute multiple kernels concurrently. Applications may query this capability by checking the concurrentKernels device property (see Device Enumeration), which is equal to 1 for devices that support it.

+

The maximum number of kernel launches that a device can execute concurrently depends on its compute capability and is listed in Table 27.

+

A kernel from one CUDA context cannot execute concurrently with a kernel from another CUDA context. The GPU may time slice to provide forward progress to each context. If a user wants to run kernels from multiple process simultaneously on the SM, one must enable MPS.

+

Kernels that use many textures or a large amount of local memory are less likely to execute concurrently with other kernels.

+
+
+

+6.2.8.3. Overlap of Data Transfer and Kernel Execution +

+

Some devices can perform an asynchronous memory copy to or from the GPU concurrently with kernel execution. Applications may query this capability by checking the asyncEngineCount device property (see Device Enumeration), which is greater than zero for devices that support it. If host memory is involved in the copy, it must be page-locked.

+

It is also possible to perform an intra-device copy simultaneously with kernel execution (on devices that support the concurrentKernels device property) and/or with copies to or from the device (for devices that support the asyncEngineCount property). Intra-device copies are initiated using the standard memory copy functions with destination and source addresses residing on the same device.

+
+
+

+6.2.8.4. Concurrent Data Transfers +

+

Some devices of compute capability 2.x and higher can overlap copies to and from the device. Applications may query this capability by checking the asyncEngineCount device property (see Device Enumeration), which is equal to 2 for devices that support it. In order to be overlapped, any host memory involved in the transfers must be page-locked.

+
+
+

+6.2.8.5. Streams +

+

Applications manage the concurrent operations described above through streams. A stream is a sequence of commands (possibly issued by different host threads) that execute in order. Different streams, on the other hand, may execute their commands out of order with respect to one another or concurrently; this behavior is not guaranteed and should therefore not be relied upon for correctness (for example, inter-kernel communication is undefined). The commands issued on a stream may execute when all the dependencies of the command are met. The dependencies could be previously launched commands on same stream or dependencies from other streams. The successful completion of synchronize call guarantees that all the commands launched are completed.

+
+
+6.2.8.5.1. Creation and Destruction of Streams +
+

A stream is defined by creating a stream object and specifying it as the stream parameter to a sequence of kernel launches and host <-> device memory copies. The following code sample creates two streams and allocates an array hostPtr of float in page-locked memory.

+
+
cudaStream_t stream[2];
+for (int i = 0; i < 2; ++i)
+    cudaStreamCreate(&stream[i]);
+float* hostPtr;
+cudaMallocHost(&hostPtr, 2 * size);
+
+
+

Each of these streams is defined by the following code sample as a sequence of one memory copy from host to device, one kernel launch, and one memory copy from device to host:

+
+
for (int i = 0; i < 2; ++i) {
+    cudaMemcpyAsync(inputDevPtr + i * size, hostPtr + i * size,
+                    size, cudaMemcpyHostToDevice, stream[i]);
+    MyKernel <<<100, 512, 0, stream[i]>>>
+          (outputDevPtr + i * size, inputDevPtr + i * size, size);
+    cudaMemcpyAsync(hostPtr + i * size, outputDevPtr + i * size,
+                    size, cudaMemcpyDeviceToHost, stream[i]);
+}
+
+
+

Each stream copies its portion of input array hostPtr to array inputDevPtr in device memory, processes inputDevPtr on the device by calling MyKernel(), and copies the result outputDevPtr back to the same portion of hostPtr. Overlapping Behavior describes how the streams overlap in this example depending on the capability of the device. Note that hostPtr must point to page-locked host memory for any overlap to occur.

+

Streams are released by calling cudaStreamDestroy().

+
+
for (int i = 0; i < 2; ++i)
+    cudaStreamDestroy(stream[i]);
+
+
+

In case the device is still doing work in the stream when cudaStreamDestroy() is called, the function will return immediately and the resources associated with the stream will be released automatically once the device has completed all work in the stream.

+
+
+
+6.2.8.5.2. Default Stream +
+

Kernel launches and host <-> device memory copies that do not specify any stream parameter, or equivalently that set the stream parameter to zero, are issued to the default stream. They are therefore executed in order.

+

For code that is compiled using the --default-stream per-thread compilation flag (or that defines the CUDA_API_PER_THREAD_DEFAULT_STREAM macro before including CUDA headers (cuda.h and cuda_runtime.h)), the default stream is a regular stream and each host thread has its own default stream.

+
+

Note

+

#define CUDA_API_PER_THREAD_DEFAULT_STREAM 1 cannot be used to enable this behavior when the code is compiled by nvcc as nvcc implicitly includes cuda_runtime.h at the top of the translation unit. In this case the --default-stream per-thread compilation flag needs to be used or the CUDA_API_PER_THREAD_DEFAULT_STREAM macro needs to be defined with the -DCUDA_API_PER_THREAD_DEFAULT_STREAM=1 compiler flag.

+
+

For code that is compiled using the --default-stream legacy compilation flag, the default stream is a special stream called the NULL stream and each device has a single NULL stream used for all host threads. The NULL stream is special as it causes implicit synchronization as described in Implicit Synchronization.

+

For code that is compiled without specifying a --default-stream compilation flag, --default-stream legacy is assumed as the default.

+
+
+
+6.2.8.5.3. Explicit Synchronization +
+

There are various ways to explicitly synchronize streams with each other.

+

cudaDeviceSynchronize() waits until all preceding commands in all streams of all host threads have completed.

+

cudaStreamSynchronize()takes a stream as a parameter and waits until all preceding commands in the given stream have completed. It can be used to synchronize the host with a specific stream, allowing other streams to continue executing on the device.

+

cudaStreamWaitEvent()takes a stream and an event as parameters (see Events for a description of events)and makes all the commands added to the given stream after the call to cudaStreamWaitEvent()delay their execution until the given event has completed.

+

cudaStreamQuery()provides applications with a way to know if all preceding commands in a stream have completed.

+
+
+
+6.2.8.5.4. Implicit Synchronization +
+

Two operations from different streams cannot run concurrently if any CUDA operation on the NULL stream is submitted +in-between them, unless the streams are non-blocking streams (created with the cudaStreamNonBlocking flag).

+

Applications should follow these guidelines to improve +their potential for concurrent kernel execution:

+
    +
  • All independent operations should be issued before dependent operations,

  • +
  • Synchronization of any kind should be delayed as long as possible.

  • +
+
+
+
+6.2.8.5.5. Overlapping Behavior +
+

The amount of execution overlap between two streams depends on the order in which the commands are issued to each stream +and whether or not the device supports overlap of data transfer and kernel execution (see Overlap of Data Transfer and Kernel Execution), +concurrent kernel execution (see Concurrent Kernel Execution), and/or concurrent data transfers (see Concurrent Data Transfers).

+

For example, on devices that do not support concurrent data transfers, the two streams of the code sample of Creation and Destruction of Streams +do not overlap at all because the memory copy from host to device is issued to stream[1] after the memory copy from device to host +is issued to stream[0], so it can only start once the memory copy from device to host issued to stream[0] has completed. If the +code is rewritten the following way (and assuming the device supports overlap of data transfer and kernel execution)

+
+
for (int i = 0; i < 2; ++i)
+    cudaMemcpyAsync(inputDevPtr + i * size, hostPtr + i * size,
+                    size, cudaMemcpyHostToDevice, stream[i]);
+for (int i = 0; i < 2; ++i)
+    MyKernel<<<100, 512, 0, stream[i]>>>
+          (outputDevPtr + i * size, inputDevPtr + i * size, size);
+for (int i = 0; i < 2; ++i)
+    cudaMemcpyAsync(hostPtr + i * size, outputDevPtr + i * size,
+                    size, cudaMemcpyDeviceToHost, stream[i]);
+
+
+

then the memory copy from host to device issued to stream[1] overlaps with the kernel launch issued to stream[0].

+

On devices that do support concurrent data transfers, the two streams of the code sample of Creation and Destruction of Streams +do overlap: The memory copy from host to device issued to stream[1] overlaps with the memory copy from device to host issued to +stream[0] and even with the kernel launch issued to stream[0] (assuming the device supports overlap of data transfer and kernel execution).

+
+
+
+6.2.8.5.6. Host Functions (Callbacks) +
+

The runtime provides a way to insert a CPU function call at any point into a stream via cudaLaunchHostFunc(). The provided function is executed on the host once all commands issued to the stream before the callback have completed.

+

The following code sample adds the host function MyCallback to each of two streams after issuing a host-to-device memory copy, a kernel launch and a device-to-host memory copy into each stream. The function will begin execution on the host after each of the device-to-host memory copies completes.

+
+
void CUDART_CB MyCallback(void *data){
+    printf("Inside callback %d\n", (size_t)data);
+}
+...
+for (size_t i = 0; i < 2; ++i) {
+    cudaMemcpyAsync(devPtrIn[i], hostPtr[i], size, cudaMemcpyHostToDevice, stream[i]);
+    MyKernel<<<100, 512, 0, stream[i]>>>(devPtrOut[i], devPtrIn[i], size);
+    cudaMemcpyAsync(hostPtr[i], devPtrOut[i], size, cudaMemcpyDeviceToHost, stream[i]);
+    cudaLaunchHostFunc(stream[i], MyCallback, (void*)i);
+}
+
+
+

The commands that are issued in a stream after a host function do not start executing before the function has completed.

+

A host function enqueued into a stream must not make CUDA API calls (directly or indirectly), as it might end up waiting on itself if it makes such a call leading to a deadlock.

+
+
+
+6.2.8.5.7. Stream Priorities +
+

The relative priorities of streams can be specified at creation using cudaStreamCreateWithPriority(). The range of allowable priorities, ordered as [ greatest priority, least priority ] can be obtained using the cudaDeviceGetStreamPriorityRange() function. At runtime, the GPU scheduler utilizes stream priorities to determine task execution order, but these priorities serve as hints rather than guarantees. When selecting work to launch, pending tasks in higher-priority streams take precedence over those in lower-priority streams. Higher-priority tasks do not preempt already running lower-priority tasks. The GPU does not reassess work queues during task execution, and increasing a stream’s priority will not interrupt ongoing work. Stream priorities influence task execution without enforcing strict ordering, so users can leverage stream priorities to influence task execution without relying on strict ordering guarantees.

+

The following code sample obtains the allowable range of priorities for the current device, and creates streams with the highest and lowest available priorities.

+
+
// get the range of stream priorities for this device
+int leastPriority, greatestPriority;
+cudaDeviceGetStreamPriorityRange(&leastPriority, &greatestPriority);
+// create streams with highest and lowest available priorities
+cudaStream_t st_high, st_low;
+cudaStreamCreateWithPriority(&st_high, cudaStreamNonBlocking, greatestPriority));
+cudaStreamCreateWithPriority(&st_low, cudaStreamNonBlocking, leastPriority);
+
+
+
+
+
+

+6.2.8.6. Programmatic Dependent Launch and Synchronization +

+
+

Warning

+

This document has been replaced by a new CUDA Programming Guide. The information in this document should be considered legacy, and this document is no longer being updated as of CUDA 13.0. Please refer to the CUDA Programming Guide for up-to-date information on CUDA.

+
+

The Programmatic Dependent Launch mechanism allows for a dependent secondary kernel +to launch before the primary kernel it depends on in the same CUDA stream has finished executing. +Available starting with devices of compute capability 9.0, this technique can provide performance +benefits when the secondary kernel can complete significant work that does not depend on the results of the primary kernel.

+
+
+6.2.8.6.1. Background +
+

A CUDA application utilizes the GPU by launching and executing multiple kernels on it. +A typical GPU activity timeline is shown in Figure 10.

+
+GPU activity timeline +
+

Figure 10 GPU activity timeline

+
+
+

Here, secondary_kernel is launched after primary_kernel finishes its execution. +Serialized execution is usually necessary because secondary_kernel depends on result data +produced by primary_kernel. If secondary_kernel has no dependency on primary_kernel, +both of them can be launched concurrently by using Streams. +Even if secondary_kernel is dependent on primary_kernel, there is some potential for +concurrent execution. For example, almost all the kernels have +some sort of preamble section during which tasks such as zeroing buffers or loading +constant values are performed.

+
+Preamble section of ``secondary_kernel`` +
+

Figure 11 Preamble section of secondary_kernel

+
+
+

Figure 11 demonstrates the portion of secondary_kernel that could +be executed concurrently without impacting the application. +Note that concurrent launch also allows us to hide the launch latency of secondary_kernel behind +the execution of primary_kernel.

+
+Concurrent execution of ``primary_kernel`` and ``secondary_kernel`` +
+

Figure 12 Concurrent execution of primary_kernel and secondary_kernel

+
+
+

The concurrent launch and execution of secondary_kernel shown in Figure 12 is +achievable using Programmatic Dependent Launch.

+

Programmatic Dependent Launch introduces changes to the CUDA kernel launch APIs as explained in following section. +These APIs require at least compute capability 9.0 to provide overlapping execution.

+
+
+
+6.2.8.6.2. API Description +
+

In Programmatic Dependent Launch, a primary and a secondary kernel are launched in the same CUDA stream. +The primary kernel should execute cudaTriggerProgrammaticLaunchCompletion with all thread blocks when +it’s ready for the secondary kernel to launch. The secondary kernel must be launched using the extensible launch API as shown.

+
+
__global__ void primary_kernel() {
+   // Initial work that should finish before starting secondary kernel
+
+   // Trigger the secondary kernel
+   cudaTriggerProgrammaticLaunchCompletion();
+
+   // Work that can coincide with the secondary kernel
+}
+
+__global__ void secondary_kernel()
+{
+   // Independent work
+
+   // Will block until all primary kernels the secondary kernel is dependent on have completed and flushed results to global memory
+   cudaGridDependencySynchronize();
+
+   // Dependent work
+}
+
+cudaLaunchAttribute attribute[1];
+attribute[0].id = cudaLaunchAttributeProgrammaticStreamSerialization;
+attribute[0].val.programmaticStreamSerializationAllowed = 1;
+configSecondary.attrs = attribute;
+configSecondary.numAttrs = 1;
+
+primary_kernel<<<grid_dim, block_dim, 0, stream>>>();
+cudaLaunchKernelEx(&configSecondary, secondary_kernel);
+
+
+

When the secondary kernel is launched using the cudaLaunchAttributeProgrammaticStreamSerialization attribute, +the CUDA driver is safe to launch the secondary kernel early and not wait on the +completion and memory flush of the primary before launching the secondary.

+

The CUDA driver can launch the secondary kernel when all primary thread blocks have launched and executed +cudaTriggerProgrammaticLaunchCompletion. +If the primary kernel doesn’t execute the trigger, it implicitly occurs after +all thread blocks in the primary kernel exit.

+

In either case, the secondary thread blocks might launch +before data written by the primary kernel is visible. As such, when the secondary kernel is configured with Programmatic Dependent Launch, +it must always use cudaGridDependencySynchronize +or other means to verify that the result data from the primary is available.

+

Please note that these methods provide the opportunity for the primary and secondary kernels to execute concurrently, however +this behavior is opportunistic and not guaranteed to lead to concurrent kernel execution. +Reliance on concurrent execution in this manner is unsafe and can lead to deadlock.

+
+
+
+6.2.8.6.3. Use in CUDA Graphs +
+

Programmatic Dependent Launch can be used in CUDA Graphs via stream capture or +directly via edge data. To program this feature in a CUDA Graph with edge data, use a cudaGraphDependencyType +value of cudaGraphDependencyTypeProgrammatic on an edge connecting two kernel nodes. This edge type makes the upstream kernel +visible to a cudaGridDependencySynchronize() in the downstream kernel. This type must be used with an outgoing port of +either cudaGraphKernelNodePortLaunchCompletion or cudaGraphKernelNodePortProgrammatic.

+

The resulting graph equivalents for stream capture are as follows:

+ ++++ + + + + + + + + + + + + + + + + + + + + +

Stream code (abbreviated)

Resulting graph edge

+
+
cudaLaunchAttribute attribute;
+attribute.id = cudaLaunchAttributeProgrammaticStreamSerialization;
+attribute.val.programmaticStreamSerializationAllowed = 1;
+
+
+
+
+
cudaGraphEdgeData edgeData;
+edgeData.type = cudaGraphDependencyTypeProgrammatic;
+edgeData.from_port = cudaGraphKernelNodePortProgrammatic;
+
+
+
+
+
cudaLaunchAttribute attribute;
+attribute.id = cudaLaunchAttributeProgrammaticEvent;
+attribute.val.programmaticEvent.triggerAtBlockStart = 0;
+
+
+
+
+
cudaGraphEdgeData edgeData;
+edgeData.type = cudaGraphDependencyTypeProgrammatic;
+edgeData.from_port = cudaGraphKernelNodePortProgrammatic;
+
+
+
+
+
cudaLaunchAttribute attribute;
+attribute.id = cudaLaunchAttributeProgrammaticEvent;
+attribute.val.programmaticEvent.triggerAtBlockStart = 1;
+
+
+
+
+
cudaGraphEdgeData edgeData;
+edgeData.type = cudaGraphDependencyTypeProgrammatic;
+edgeData.from_port = cudaGraphKernelNodePortLaunchCompletion;
+
+
+
+
+
+
+

+6.2.8.7. CUDA Graphs +

+

CUDA Graphs present a new model for work submission in CUDA. A graph is a series of operations, such as kernel launches, connected by dependencies, which is defined separately from its execution. This allows a graph to be defined once and then launched repeatedly. Separating out the definition of a graph from its execution enables a number of optimizations: first, CPU launch costs are reduced compared to streams, because much of the setup is done in advance; second, presenting the whole workflow to CUDA enables optimizations which might not be possible with the piecewise work submission mechanism of streams.

+

To see the optimizations possible with graphs, consider what happens in a stream: when you place a kernel into a stream, the host driver performs a sequence of operations in preparation for the execution of the kernel on the GPU. These operations, necessary for setting up and launching the kernel, are an overhead cost which must be paid for each kernel that is issued. For a GPU kernel with a short execution time, this overhead cost can be a significant fraction of the overall end-to-end execution time.

+

Work submission using graphs is separated into three distinct stages: definition, instantiation, and execution.

+
    +
  • During the definition phase, a program creates a description of the operations in the graph along with the dependencies between them.

  • +
  • Instantiation takes a snapshot of the graph template, validates it, and performs much of the setup and initialization of work with the aim of minimizing what needs to be done at launch. The resulting instance is known as an executable graph.

  • +
  • An executable graph may be launched into a stream, similar to any other CUDA work. It may be launched any number of times without repeating the instantiation.

  • +
+
+
+6.2.8.7.1. Graph Structure +
+

An operation forms a node in a graph. The dependencies between the operations are the edges. These dependencies constrain the execution sequence of the operations.

+

An operation may be scheduled at any time once the nodes on which it depends are complete. Scheduling is left up to the CUDA system.

+
+
+6.2.8.7.1.1. Node Types +
+

A graph node can be one of:

+ +
+Child Graph Example +
+

Figure 13 Child Graph Example

+
+
+
+
+
+6.2.8.7.1.2. Edge Data +
+

CUDA 12.3 introduced edge data on CUDA Graphs. Edge data modifies a dependency specified by an edge and consists of three parts: +an outgoing port, an incoming port, and a type. An outgoing port specifies when an associated edge is triggered. An incoming port +specifies what portion of a node is dependent on an associated edge. A type modifies the relation between the endpoints.

+

Port values are specific to node type and direction, and edge types may be restricted to specific node types. In all cases, +zero-initialized edge data represents default behavior. Outgoing port 0 waits on an entire task, incoming port 0 blocks an +entire task, and edge type 0 is associated with a full dependency with memory synchronizing behavior.

+

Edge data is optionally specified in various graph APIs via a parallel array to the associated nodes. If it is omitted as an +input parameter, zero-initialized data is used. If it is omitted as an output (query) parameter, the API accepts this if the +edge data being ignored is all zero-initialized, and returns cudaErrorLossyQuery if the call would discard information.

+

Edge data is also available in some stream capture APIs: cudaStreamBeginCaptureToGraph(), cudaStreamGetCaptureInfo(), +and cudaStreamUpdateCaptureDependencies(). In these cases, there is not yet a downstream node. The data is associated with +a dangling edge (half edge) which will either be connected to a future captured node or discarded at termination of stream capture. +Note that some edge types do not wait on full completion of the upstream node. These edges are ignored when considering if a +stream capture has been fully rejoined to the origin stream, and cannot be discarded at the end of capture. See Creating a Graph Using Stream Capture.

+

Currently, no node types define additional incoming ports, and only kernel nodes define additional outgoing ports. There is +one non-default dependency type, cudaGraphDependencyTypeProgrammatic, which enables Programmatic Dependent Launch between two kernel nodes.

+
+
+
+
+6.2.8.7.2. Creating a Graph Using Graph APIs +
+

Graphs can be created via two mechanisms: explicit API and stream capture. The following is an example of creating and executing the below graph.

+
+Creating a Graph Using Graph APIs Example +
+

Figure 14 Creating a Graph Using Graph APIs Example

+
+
+
+
// Create the graph - it starts out empty
+cudaGraphCreate(&graph, 0);
+
+// For the purpose of this example, we'll create
+// the nodes separately from the dependencies to
+// demonstrate that it can be done in two stages.
+// Note that dependencies can also be specified
+// at node creation.
+cudaGraphAddKernelNode(&a, graph, NULL, 0, &nodeParams);
+cudaGraphAddKernelNode(&b, graph, NULL, 0, &nodeParams);
+cudaGraphAddKernelNode(&c, graph, NULL, 0, &nodeParams);
+cudaGraphAddKernelNode(&d, graph, NULL, 0, &nodeParams);
+
+// Now set up dependencies on each node
+cudaGraphAddDependencies(graph, &a, &b, NULL, 1);     // A->B
+cudaGraphAddDependencies(graph, &a, &c, NULL, 1);     // A->C
+cudaGraphAddDependencies(graph, &b, &d, NULL, 1);     // B->D
+cudaGraphAddDependencies(graph, &c, &d, NULL, 1);     // C->D
+
+
+
+
+
+6.2.8.7.3. Creating a Graph Using Stream Capture +
+

Stream capture provides a mechanism to create a graph from existing stream-based APIs. A section of code which launches work into streams, including existing code, can be bracketed with calls to cudaStreamBeginCapture() and cudaStreamEndCapture(). See below.

+
+
cudaGraph_t graph;
+
+cudaStreamBeginCapture(stream);
+
+kernel_A<<< ..., stream >>>(...);
+kernel_B<<< ..., stream >>>(...);
+libraryCall(stream);
+kernel_C<<< ..., stream >>>(...);
+
+cudaStreamEndCapture(stream, &graph);
+
+
+

A call to cudaStreamBeginCapture() places a stream in capture mode. When a stream is being captured, work launched into the stream is not enqueued for execution. It is instead appended to an internal graph that is progressively being built up. This graph is then returned by calling cudaStreamEndCapture(), which also ends capture mode for the stream. A graph which is actively being constructed by stream capture is referred to as a capture graph.

+

Stream capture can be used on any CUDA stream except cudaStreamLegacy (the “NULL stream”). Note that it can be used on cudaStreamPerThread. If a program is using the legacy stream, it may be possible to redefine stream 0 to be the per-thread stream with no functional change. See Default Stream.

+

Whether a stream is being captured can be queried with cudaStreamIsCapturing().

+

Work can be captured to an existing graph using cudaStreamBeginCaptureToGraph(). Instead of capturing to an internal graph, work is captured to a graph provided by the user.

+
+
+6.2.8.7.3.1. Cross-stream Dependencies and Events +
+

Stream capture can handle cross-stream dependencies expressed with cudaEventRecord() and cudaStreamWaitEvent(), provided the event being waited upon was recorded into the same capture graph.

+

When an event is recorded in a stream that is in capture mode, it results in a captured event. A captured event represents a set of nodes in a capture graph.

+

When a captured event is waited on by a stream, it places the stream in capture mode if it is not already, and the next item in the stream will have additional dependencies on the nodes in the captured event. The two streams are then being captured to the same capture graph.

+

When cross-stream dependencies are present in stream capture, cudaStreamEndCapture() must still be called in the same stream where cudaStreamBeginCapture() was called; this is the origin stream. Any other streams which are being captured to the same capture graph, due to event-based dependencies, must also be joined back to the origin stream. This is illustrated below. All streams being captured to the same capture graph are taken out of capture mode upon cudaStreamEndCapture(). Failure to rejoin to the origin stream will result in failure of the overall capture operation.

+
+
// stream1 is the origin stream
+cudaStreamBeginCapture(stream1);
+
+kernel_A<<< ..., stream1 >>>(...);
+
+// Fork into stream2
+cudaEventRecord(event1, stream1);
+cudaStreamWaitEvent(stream2, event1);
+
+kernel_B<<< ..., stream1 >>>(...);
+kernel_C<<< ..., stream2 >>>(...);
+
+// Join stream2 back to origin stream (stream1)
+cudaEventRecord(event2, stream2);
+cudaStreamWaitEvent(stream1, event2);
+
+kernel_D<<< ..., stream1 >>>(...);
+
+// End capture in the origin stream
+cudaStreamEndCapture(stream1, &graph);
+
+// stream1 and stream2 no longer in capture mode
+
+
+

Graph returned by the above code is shown in Figure 14.

+
+

Note

+

When a stream is taken out of capture mode, the next non-captured item in the stream (if any) will still have a dependency on the most recent prior non-captured item, despite intermediate items having been removed.

+
+
+
+
+6.2.8.7.3.2. Prohibited and Unhandled Operations +
+

It is invalid to synchronize or query the execution status of a stream which is being captured or a captured event, because they do not represent items scheduled for execution. It is also invalid to query the execution status of or synchronize a broader handle which encompasses an active stream capture, such as a device or context handle when any associated stream is in capture mode.

+

When any stream in the same context is being captured, and it was not created with cudaStreamNonBlocking, any attempted use of the legacy stream is invalid. This is because the legacy stream handle at all times encompasses these other streams; enqueueing to the legacy stream would create a dependency on the streams being captured, and querying it or synchronizing it would query or synchronize the streams being captured.

+

It is therefore also invalid to call synchronous APIs in this case. Synchronous APIs, such as cudaMemcpy(), enqueue work to the legacy stream and synchronize it before returning.

+
+

Note

+

As a general rule, when a dependency relation would connect something that is captured with something that was not captured and instead enqueued for execution, CUDA prefers to return an error rather than ignore the dependency. An exception is made for placing a stream into or out of capture mode; this severs a dependency relation between items added to the stream immediately before and after the mode transition.

+
+

It is invalid to merge two separate capture graphs by waiting on a captured event from a stream which is being captured and is associated with a different capture graph than the event. It is invalid to wait on a non-captured event from a stream which is being captured without specifying the cudaEventWaitExternal flag.

+

A small number of APIs that enqueue asynchronous operations into streams are not currently supported in graphs and will return an error if called with a stream which is being captured, such as cudaStreamAttachMemAsync().

+
+
+
+6.2.8.7.3.3. Invalidation +
+

When an invalid operation is attempted during stream capture, any associated capture graphs are invalidated. When a capture graph is invalidated, further use of any streams which are being captured or captured events associated with the graph is invalid and will return an error, until stream capture is ended with cudaStreamEndCapture(). This call will take the associated streams out of capture mode, but will also return an error value and a NULL graph.

+
+
+
+
+6.2.8.7.4. CUDA User Objects +
+

CUDA User Objects can be used to help manage the lifetime of resources used by asynchronous work in CUDA. In particular, this feature is useful for CUDA Graphs and stream capture.

+

Various resource management schemes are not compatible with CUDA graphs. Consider for example an event-based pool or a synchronous-create, asynchronous-destroy scheme.

+
+
// Library API with pool allocation
+void libraryWork(cudaStream_t stream) {
+    auto &resource = pool.claimTemporaryResource();
+    resource.waitOnReadyEventInStream(stream);
+    launchWork(stream, resource);
+    resource.recordReadyEvent(stream);
+}
+
+
+
+
// Library API with asynchronous resource deletion
+void libraryWork(cudaStream_t stream) {
+    Resource *resource = new Resource(...);
+    launchWork(stream, resource);
+    cudaLaunchHostFunc(
+        stream,
+        [](void *resource) {
+            delete static_cast<Resource *>(resource);
+        },
+        resource,
+        0);
+    // Error handling considerations not shown
+}
+
+
+

These schemes are difficult with CUDA graphs because of the non-fixed pointer or handle for the resource which requires indirection or graph update, and the synchronous CPU code needed each time the work is submitted. They also do not work with stream capture if these considerations are hidden from the caller of the library, and because of use of disallowed APIs during capture. Various solutions exist such as exposing the resource to the caller. CUDA user objects present another approach.

+

A CUDA user object associates a user-specified destructor callback with an internal refcount, similar to C++ shared_ptr. References may be owned by user code on the CPU and by CUDA graphs. Note that for user-owned references, unlike C++ smart pointers, there is no object representing the reference; users must track user-owned references manually. A typical use case would be to immediately move the sole user-owned reference to a CUDA graph after the user object is created.

+

When a reference is associated to a CUDA graph, CUDA will manage the graph operations automatically. A cloned cudaGraph_t retains a copy of every reference owned by the source cudaGraph_t, with the same multiplicity. An instantiated cudaGraphExec_t retains a copy of every reference in the source cudaGraph_t. When a cudaGraphExec_t is destroyed without being synchronized, the references are retained until the execution is completed.

+

Here is an example use.

+
+
cudaGraph_t graph;  // Preexisting graph
+
+Object *object = new Object;  // C++ object with possibly nontrivial destructor
+cudaUserObject_t cuObject;
+cudaUserObjectCreate(
+    &cuObject,
+    object,  // Here we use a CUDA-provided template wrapper for this API,
+             // which supplies a callback to delete the C++ object pointer
+    1,  // Initial refcount
+    cudaUserObjectNoDestructorSync  // Acknowledge that the callback cannot be
+                                    // waited on via CUDA
+);
+cudaGraphRetainUserObject(
+    graph,
+    cuObject,
+    1,  // Number of references
+    cudaGraphUserObjectMove  // Transfer a reference owned by the caller (do
+                             // not modify the total reference count)
+);
+// No more references owned by this thread; no need to call release API
+cudaGraphExec_t graphExec;
+cudaGraphInstantiate(&graphExec, graph, nullptr, nullptr, 0);  // Will retain a
+                                                               // new reference
+cudaGraphDestroy(graph);  // graphExec still owns a reference
+cudaGraphLaunch(graphExec, 0);  // Async launch has access to the user objects
+cudaGraphExecDestroy(graphExec);  // Launch is not synchronized; the release
+                                  // will be deferred if needed
+cudaStreamSynchronize(0);  // After the launch is synchronized, the remaining
+                           // reference is released and the destructor will
+                           // execute. Note this happens asynchronously.
+// If the destructor callback had signaled a synchronization object, it would
+// be safe to wait on it at this point.
+
+
+

References owned by graphs in child graph nodes are associated to the child graphs, not the parents. If a child graph is updated or deleted, the references change accordingly. If an executable graph or child graph is updated with cudaGraphExecUpdate or cudaGraphExecChildGraphNodeSetParams, the references in the new source graph are cloned and replace the references in the target graph. In either case, if previous launches are not synchronized, any references which would be released are held until the launches have finished executing.

+

There is not currently a mechanism to wait on user object destructors via a CUDA API. Users may signal a synchronization object manually from the destructor code. In addition, it is not legal to call CUDA APIs from the destructor, similar to the restriction on cudaLaunchHostFunc. This is to avoid blocking a CUDA internal shared thread and preventing forward progress. It is legal to signal another thread to perform an API call, if the dependency is one way and the thread doing the call cannot block forward progress of CUDA work.

+

User objects are created with cudaUserObjectCreate, which is a good starting point to browse related APIs.

+
+
+
+6.2.8.7.5. Updating Instantiated Graphs +
+

Work submission using graphs is separated into three distinct stages: definition, instantiation, and execution. In situations where the workflow is not changing, the overhead of definition and instantiation can be amortized over many executions, and graphs provide a clear advantage over streams.

+

A graph is a snapshot of a workflow, including kernels, parameters, and dependencies, in order to replay it as rapidly and efficiently as possible. In situations where the workflow changes the graph becomes out of date and must be modified. Major changes to graph structure such as topology or types of nodes will require re-instantiation of the source graph because various topology-related optimization techniques must be re-applied.

+

The cost of repeated instantiation can reduce the overall performance benefit from graph execution, but it is common for only node parameters, such as kernel parameters and cudaMemcpy addresses, to change while graph topology remains the same. For this case, CUDA provides a lightweight mechanism known as “Graph Update,” which allows certain node parameters to be modified in-place without having to rebuild the entire graph. This is much more efficient than re-instantiation.

+

Updates will take effect the next time the graph is launched, so they will not impact previous graph launches, even if they are running at the time of the update. A graph may be updated and relaunched repeatedly, so multiple updates/launches can be queued on a stream.

+

CUDA provides two mechanisms for updating instantiated graph parameters, whole graph update and individual node update. Whole graph update allows the user to supply a topologically identical cudaGraph_t object whose nodes contain updated parameters. Individual node update allows the user to explicitly update the parameters of individual nodes. Using an updated cudaGraph_t is more convenient when a large number of nodes are being updated, or when the graph topology is unknown to the caller (i.e., The graph resulted from stream capture of a library call). Using individual node update is preferred when the number of changes is small and the user has the handles to the nodes requiring updates. Individual node update skips the topology checks and comparisons for unchanged nodes, so it can be more efficient in many cases.

+

CUDA also provides a mechanism for enabling and disabling individual nodes without affecting their current parameters.

+

The following sections explain each approach in more detail.

+
+
+6.2.8.7.5.1. Graph Update Limitations +
+

Kernel nodes:

+
    +
  • The owning context of the function cannot change.

  • +
  • A node whose function originally did not use CUDA dynamic parallelism cannot be updated to a function which uses CUDA dynamic parallelism.

  • +
+

cudaMemset and cudaMemcpy nodes:

+
    +
  • The CUDA device(s) to which the operand(s) was allocated/mapped cannot change.

  • +
  • The source/destination memory must be allocated from the same context as the original source/destination memory.

  • +
  • Only 1D cudaMemset/cudaMemcpy nodes can be changed.

  • +
+

Additional memcpy node restrictions:

+
    +
  • Changing either the source or destination memory type (i.e., cudaPitchedPtr, cudaArray_t, etc.), or the type of transfer (i.e., cudaMemcpyKind) is not supported.

  • +
+

External semaphore wait nodes and record nodes:

+
    +
  • Changing the number of semaphores is not supported.

  • +
+

Conditional nodes:

+
    +
  • The order of handle creation and assignment must match between the graphs.

  • +
  • Changing node parameters is not supported (i.e. number of graphs in the conditional, node context, etc).

  • +
  • Changing parameters of nodes within the conditional body graph is subject to the rules above.

  • +
+

Memory nodes:

+
    +
  • It is not possible to update a cudaGraphExec_t with a cudaGraph_t if the cudaGraph_t is currently instantiated as a different cudaGraphExec_t.

  • +
+

There are no restrictions on updates to host nodes, event record nodes, or event wait nodes.

+
+
+
+6.2.8.7.5.2. Whole Graph Update +
+

cudaGraphExecUpdate() allows an instantiated graph (the “original graph”) to be updated with the parameters from a topologically identical graph (the “updating” graph). The topology of the updating graph must be identical to the original graph used to instantiate the cudaGraphExec_t. In addition, the order in which the dependencies are specified must match. Finally, CUDA needs to consistently order the sink nodes (nodes with no dependencies). CUDA relies on the order of specific api calls to achieve consistent sink node ordering.

+

More explicitly, following the following rules will cause cudaGraphExecUpdate() to pair the nodes in the original graph and the updating graph deterministically:

+
    +
  1. For any capturing stream, the API calls operating on that stream must be made in the same order, including event wait and other api calls not directly corresponding to node creation.

  2. +
  3. The API calls which directly manipulate a given graph node’s incoming edges (including captured stream APIs, node add APIs, and edge addition / removal APIs) must be made in the same order. Moreover, when dependencies are specified in arrays to these APIs, the order in which the dependencies are specified inside those arrays must match.

  4. +
  5. +

    Sink nodes must be consistently ordered. Sink nodes are nodes without dependent nodes / outgoing edges in the final graph at the time of the cudaGraphExecUpdate() invocation. The following operations affect sink node ordering (if present) and must (as a combined set) be made in the same order:

    +
      +
    • Node add APIs resulting in a sink node.

    • +
    • Edge removal resulting in a node becoming a sink node.

    • +
    • cudaStreamUpdateCaptureDependencies(), if it removes a sink node from a capturing stream’s dependency set.

    • +
    • cudaStreamEndCapture().

    • +
    +
  6. +
+

The following example shows how the API could be used to update an instantiated graph:

+
+
cudaGraphExec_t graphExec = NULL;
+
+for (int i = 0; i < 10; i++) {
+    cudaGraph_t graph;
+    cudaGraphExecUpdateResult updateResult;
+    cudaGraphNode_t errorNode;
+
+    // In this example we use stream capture to create the graph.
+    // You can also use the Graph API to produce a graph.
+    cudaStreamBeginCapture(stream, cudaStreamCaptureModeGlobal);
+
+    // Call a user-defined, stream based workload, for example
+    do_cuda_work(stream);
+
+    cudaStreamEndCapture(stream, &graph);
+
+    // If we've already instantiated the graph, try to update it directly
+    // and avoid the instantiation overhead
+    if (graphExec != NULL) {
+        // If the graph fails to update, errorNode will be set to the
+        // node causing the failure and updateResult will be set to a
+        // reason code.
+        cudaGraphExecUpdate(graphExec, graph, &errorNode, &updateResult);
+    }
+
+    // Instantiate during the first iteration or whenever the update
+    // fails for any reason
+    if (graphExec == NULL || updateResult != cudaGraphExecUpdateSuccess) {
+
+        // If a previous update failed, destroy the cudaGraphExec_t
+        // before re-instantiating it
+        if (graphExec != NULL) {
+            cudaGraphExecDestroy(graphExec);
+        }
+        // Instantiate graphExec from graph. The error node and
+        // error message parameters are unused here.
+        cudaGraphInstantiate(&graphExec, graph, NULL, NULL, 0);
+    }
+
+    cudaGraphDestroy(graph);
+    cudaGraphLaunch(graphExec, stream);
+    cudaStreamSynchronize(stream);
+}
+
+
+

A typical workflow is to create the initial cudaGraph_t using either the stream capture or graph API. The cudaGraph_t is then instantiated and launched as normal. After the initial launch, a new cudaGraph_t is created using the same method as the initial graph and cudaGraphExecUpdate() is called. If the graph update is successful, indicated by the updateResult parameter in the above example, the updated cudaGraphExec_t is launched. If the update fails for any reason, the cudaGraphExecDestroy() and cudaGraphInstantiate() are called to destroy the original cudaGraphExec_t and instantiate a new one.

+

It is also possible to update the cudaGraph_t nodes directly (i.e., Using cudaGraphKernelNodeSetParams()) and subsequently update the cudaGraphExec_t, however it is more efficient to use the explicit node update APIs covered in the next section.

+

Conditional handle flags and default values are updated as part of the graph update.

+

Please see the Graph API for more information on usage and current limitations.

+
+
+
+6.2.8.7.5.3. Individual Node Update +
+

Instantiated graph node parameters can be updated directly. This eliminates the overhead of instantiation as well as the overhead of creating a new cudaGraph_t. If the number of nodes requiring update is small relative to the total number of nodes in the graph, it is better to update the nodes individually. The following methods are available for updating cudaGraphExec_t nodes:

+
    +
  • cudaGraphExecKernelNodeSetParams()

  • +
  • cudaGraphExecMemcpyNodeSetParams()

  • +
  • cudaGraphExecMemsetNodeSetParams()

  • +
  • cudaGraphExecHostNodeSetParams()

  • +
  • cudaGraphExecChildGraphNodeSetParams()

  • +
  • cudaGraphExecEventRecordNodeSetEvent()

  • +
  • cudaGraphExecEventWaitNodeSetEvent()

  • +
  • cudaGraphExecExternalSemaphoresSignalNodeSetParams()

  • +
  • cudaGraphExecExternalSemaphoresWaitNodeSetParams()

  • +
+

Please see the Graph API for more information on usage and current limitations.

+
+
+
+6.2.8.7.5.4. Individual Node Enable +
+

Kernel, memset and memcpy nodes in an instantiated graph can be enabled or disabled using the cudaGraphNodeSetEnabled() API. This allows the creation of a graph which contains a superset of the desired functionality which can be customized for each launch. The enable state of a node can be queried using the cudaGraphNodeGetEnabled() API.

+

A disabled node is functionally equivalent to empty node until it is reenabled. Node parameters are not affected by enabling/disabling a node. Enable state is unaffected by individual node update or whole graph update with cudaGraphExecUpdate(). Parameter updates while the node is disabled will take effect when the node is reenabled.

+

The following methods are available for enabling/disabling cudaGraphExec_t nodes, as well as querying their status:

+
    +
  • cudaGraphNodeSetEnabled()

  • +
  • cudaGraphNodeGetEnabled()

  • +
+

Refer to the Graph API for more information on usage and current limitations.

+
+
+
+
+6.2.8.7.6. Using Graph APIs +
+

cudaGraph_t objects are not thread-safe. It is the responsibility of the user to ensure that multiple threads do not concurrently access the same cudaGraph_t.

+

A cudaGraphExec_t cannot run concurrently with itself. A launch of a cudaGraphExec_t will be ordered after previous launches of the same executable graph.

+

Graph execution is done in streams for ordering with other asynchronous work. However, the stream is for ordering only; it does not constrain the internal parallelism of the graph, nor does it affect where graph nodes execute.

+

See Graph API.

+
+
+
+6.2.8.7.7. Device Graph Launch +
+

There are many workflows which need to make data-dependent decisions during runtime and execute different operations depending on those decisions. Rather than offloading this decision-making process to the host, which may require a round-trip from the device, users may prefer to perform it on the device. To that end, CUDA provides a mechanism to launch graphs from the device.

+

Device graph launch provides a convenient way to perform dynamic control flow from the device, be it something as simple as a loop or as complex as a device-side work scheduler. This functionality is only available on systems which support unified addressing.

+

Graphs which can be launched from the device will henceforth be referred to as device graphs, and graphs which cannot be launched from the device will be referred to as host graphs.

+

Device graphs can be launched from both the host and device, whereas host graphs can only be launched from the host. Unlike host launches, launching a device graph from the device while a previous launch of the graph is running will result in an error, returning cudaErrorInvalidValue; therefore, a device graph cannot be launched twice from the device at the same time. Launching a device graph from the host and device simultaneously will result in undefined behavior.

+
+
+6.2.8.7.7.1. Device Graph Creation +
+

In order for a graph to be launched from the device, it must be instantiated explicitly for device launch. This is achieved by passing the cudaGraphInstantiateFlagDeviceLaunch flag to the cudaGraphInstantiate() call. As is the case for host graphs, device graph structure is fixed at time of instantiation and cannot be updated without re-instantiation, and instantiation can only be performed on the host. In order for a graph to be able to be instantiated for device launch, it must adhere to various requirements.

+
+6.2.8.7.7.1.1. Device Graph Requirements +

General requirements:

+
    +
  • The graph’s nodes must all reside on a single device.

  • +
  • The graph can only contain kernel nodes, memcpy nodes, memset nodes, and child graph nodes.

  • +
+

Kernel nodes:

+
    +
  • Use of CUDA Dynamic Parallelism by kernels in the graph is not permitted.

  • +
  • Cooperative launches are permitted so long as MPS is not in use.

  • +
+

Memcpy nodes:

+
    +
  • Only copies involving device memory and/or pinned device-mapped host memory are permitted.

  • +
  • Copies involving CUDA arrays are not permitted.

  • +
  • Both operands must be accessible from the current device at time of instantiation. Note that the copy operation will be performed from the device on which the graph resides, even if it is targeting memory on another device.

  • +
+
+
+6.2.8.7.7.1.2. Device Graph Upload +

In order to launch a graph on the device, it must first be uploaded to the device to populate the necessary device resources. This can be achieved in one of two ways.

+

Firstly, the graph can be uploaded explicitly, either via cudaGraphUpload() or by requesting an upload as part of instantiation via cudaGraphInstantiateWithParams().

+

Alternatively, the graph can first be launched from the host, which will perform this upload step implicitly as part of the launch.

+

Examples of all three methods can be seen below:

+
+
// Explicit upload after instantiation
+cudaGraphInstantiate(&deviceGraphExec1, deviceGraph1, cudaGraphInstantiateFlagDeviceLaunch);
+cudaGraphUpload(deviceGraphExec1, stream);
+
+// Explicit upload as part of instantiation
+cudaGraphInstantiateParams instantiateParams = {0};
+instantiateParams.flags = cudaGraphInstantiateFlagDeviceLaunch | cudaGraphInstantiateFlagUpload;
+instantiateParams.uploadStream = stream;
+cudaGraphInstantiateWithParams(&deviceGraphExec2, deviceGraph2, &instantiateParams);
+
+// Implicit upload via host launch
+cudaGraphInstantiate(&deviceGraphExec3, deviceGraph3, cudaGraphInstantiateFlagDeviceLaunch);
+cudaGraphLaunch(deviceGraphExec3, stream);
+
+
+
+
+6.2.8.7.7.1.3. Device Graph Update +

Device graphs can only be updated from the host, and must be re-uploaded to the device upon executable graph update in order for the changes to take effect. This can be achieved using the same methods outlined in the previous section. Unlike host graphs, launching a device graph from the device while an update is being applied will result in undefined behavior.

+
+
+
+
+6.2.8.7.7.2. Device Launch +
+

Device graphs can be launched from both the host and the device via cudaGraphLaunch(), which has the same signature on the device as on the host. Device graphs are launched via the same handle on the host and the device. Device graphs must be launched from another graph when launched from the device.

+

Device-side graph launch is per-thread and multiple launches may occur from different threads at the same time, so the user will need to select a single thread from which to launch a given graph.

+
+6.2.8.7.7.2.1. Device Launch Modes +

Unlike host launch, device graphs cannot be launched into regular CUDA streams, and can only be launched into distinct named streams, which each denote a specific launch mode:

+ + ++++ + + + + + + + + + + + + + + + + + + + + +
+Table 5 Device-only Graph Launch Streams +

Stream

Launch Mode

cudaStreamGraphFireAndForget

Fire and forget launch

cudaStreamGraphTailLaunch

Tail launch

cudaStreamGraphFireAndForgetAsSibling

Sibling launch

+
+6.2.8.7.7.2.1.1. Fire and Forget Launch +

As the name suggests, a fire and forget launch is submitted to the GPU immediately, and it runs independently of the launching graph. In a fire-and-forget scenario, the launching graph is the parent, and the launched graph is the child.

+
+_images/fire-and-forget-simple.png +
+

Figure 15 Fire and forget launch

+
+
+

The above diagram can be generated by the sample code below:

+
+
__global__ void launchFireAndForgetGraph(cudaGraphExec_t graph) {
+    cudaGraphLaunch(graph, cudaStreamGraphFireAndForget);
+}
+
+void graphSetup() {
+    cudaGraphExec_t gExec1, gExec2;
+    cudaGraph_t g1, g2;
+
+    // Create, instantiate, and upload the device graph.
+    create_graph(&g2);
+    cudaGraphInstantiate(&gExec2, g2, cudaGraphInstantiateFlagDeviceLaunch);
+    cudaGraphUpload(gExec2, stream);
+
+    // Create and instantiate the launching graph.
+    cudaStreamBeginCapture(stream, cudaStreamCaptureModeGlobal);
+    launchFireAndForgetGraph<<<1, 1, 0, stream>>>(gExec2);
+    cudaStreamEndCapture(stream, &g1);
+    cudaGraphInstantiate(&gExec1, g1);
+
+    // Launch the host graph, which will in turn launch the device graph.
+    cudaGraphLaunch(gExec1, stream);
+}
+
+
+

A graph can have up to 120 total fire-and-forget graphs during the course of its execution. This total resets between launches of the same parent graph.

+
+
+6.2.8.7.7.2.1.2. Graph Execution Environments +

In order to fully understand the device-side synchronization model, it is first necessary to understand the concept of an execution environment.

+

When a graph is launched from the device, it is launched into its own execution environment. The execution environment of a given graph encapsulates all work in the graph as well as all generated fire and forget work. The graph can be considered complete when it has completed execution and when all generated child work is complete.

+

The below diagram shows the environment encapsulation that would be generated by the fire-and-forget sample code in the previous section.

+
+_images/fire-and-forget-environments.png +
+

Figure 16 Fire and forget launch, with execution environments

+
+
+

These environments are also hierarchical, so a graph environment can include multiple levels of child-environments from fire and forget launches.

+
+_images/fire-and-forget-nested-environments.png +
+

Figure 17 Nested fire and forget environments

+
+
+

When a graph is launched from the host, there exists a stream environment that parents the execution environment of the launched graph. The stream environment encapsulates all work generated as part of the overall launch. The stream launch is complete (i.e. downstream dependent work may now run) when the overall stream environment is marked as complete.

+
+_images/device-graph-stream-environment.png +
+

Figure 18 The stream environment, visualized

+
+
+
+
+6.2.8.7.7.2.1.3. Tail Launch +

Unlike on the host, it is not possible to synchronize with device graphs from the GPU via traditional methods such as cudaDeviceSynchronize() or cudaStreamSynchronize(). Rather, in order to enable serial work dependencies, a different launch mode - tail launch - is offered, to provide similar functionality.

+

A tail launch executes when a graph’s environment is considered complete - ie, when the graph and all its children are complete. When a graph completes, the environment of the next graph in the tail launch list will replace the completed environment as a child of the parent environment. Like fire-and-forget launches, a graph can have multiple graphs enqueued for tail launch.

+
+_images/tail-launch-simple.png +
+

Figure 19 A simple tail launch

+
+
+

The above execution flow can be generated by the code below:

+
+
__global__ void launchTailGraph(cudaGraphExec_t graph) {
+    cudaGraphLaunch(graph, cudaStreamGraphTailLaunch);
+}
+
+void graphSetup() {
+    cudaGraphExec_t gExec1, gExec2;
+    cudaGraph_t g1, g2;
+
+    // Create, instantiate, and upload the device graph.
+    create_graph(&g2);
+    cudaGraphInstantiate(&gExec2, g2, cudaGraphInstantiateFlagDeviceLaunch);
+    cudaGraphUpload(gExec2, stream);
+
+    // Create and instantiate the launching graph.
+    cudaStreamBeginCapture(stream, cudaStreamCaptureModeGlobal);
+    launchTailGraph<<<1, 1, 0, stream>>>(gExec2);
+    cudaStreamEndCapture(stream, &g1);
+    cudaGraphInstantiate(&gExec1, g1);
+
+    // Launch the host graph, which will in turn launch the device graph.
+    cudaGraphLaunch(gExec1, stream);
+}
+
+
+

Tail launches enqueued by a given graph will execute one at a time, in order of when they were enqueued. So the first enqueued graph will run first, and then the second, and so on.

+
+_images/tail-launch-ordering-simple.png +
+

Figure 20 Tail launch ordering

+
+
+

Tail launches enqueued by a tail graph will execute before tail launches enqueued by previous graphs in the tail launch list. These new tail launches will execute in the order they are enqueued.

+
+_images/tail-launch-ordering-complex.png +
+

Figure 21 Tail launch ordering when enqueued from multiple graphs

+
+
+

A graph can have up to 255 pending tail launches.

+
+6.2.8.7.7.2.1.3.1. Tail Self-launch +

It is possible for a device graph to enqueue itself for a tail launch, although a given graph can only have one self-launch enqueued at a time. In order to query the currently running device graph so that it can be relaunched, a new device-side function is added:

+
+
cudaGraphExec_t cudaGetCurrentGraphExec();
+
+
+

This function returns the handle of the currently running graph if it is a device graph. If the currently executing kernel is not a node within a device graph, this function will return NULL.

+

Below is sample code showing usage of this function for a relaunch loop:

+
+
__device__ int relaunchCount = 0;
+
+__global__ void relaunchSelf() {
+    int relaunchMax = 100;
+
+    if (threadIdx.x == 0) {
+        if (relaunchCount < relaunchMax) {
+            cudaGraphLaunch(cudaGetCurrentGraphExec(), cudaStreamGraphTailLaunch);
+        }
+
+        relaunchCount++;
+    }
+}
+
+
+
+
+
+6.2.8.7.7.2.1.4. Sibling Launch +

Sibling launch is a variation of fire-and-forget launch in which the graph is launched not as a child of the launching graph’s execution environment, but rather as a child of the launching graph’s parent environment. Sibling launch is equivalent to a fire-and-forget launch from the launching graph’s parent environment.

+
+_images/sibling-launch-simple.png +
+

Figure 22 A simple sibling launch

+
+
+

The above diagram can be generated by the sample code below:

+
+
__global__ void launchSiblingGraph(cudaGraphExec_t graph) {
+    cudaGraphLaunch(graph, cudaStreamGraphFireAndForgetAsSibling);
+}
+
+void graphSetup() {
+    cudaGraphExec_t gExec1, gExec2;
+    cudaGraph_t g1, g2;
+
+    // Create, instantiate, and upload the device graph.
+    create_graph(&g2);
+    cudaGraphInstantiate(&gExec2, g2, cudaGraphInstantiateFlagDeviceLaunch);
+    cudaGraphUpload(gExec2, stream);
+
+    // Create and instantiate the launching graph.
+    cudaStreamBeginCapture(stream, cudaStreamCaptureModeGlobal);
+    launchSiblingGraph<<<1, 1, 0, stream>>>(gExec2);
+    cudaStreamEndCapture(stream, &g1);
+    cudaGraphInstantiate(&gExec1, g1);
+
+    // Launch the host graph, which will in turn launch the device graph.
+    cudaGraphLaunch(gExec1, stream);
+}
+
+
+

Since sibling launches are not launched into the launching graph’s execution environment, they will not gate tail launches enqueued by the launching graph.

+
+
+
+
+
+
+6.2.8.7.8. Conditional Graph Nodes +
+

Conditional nodes allow conditional execution and looping of a graph contained within the conditional node. This allows dynamic and iterative workflows to be represented completely within a graph and frees up the host CPU to perform other work in parallel.

+

Evaluation of the condition value is performed on the device when the dependencies of the conditional node have been met. Conditional nodes can be one of the following types:

+
    +
  • Conditional IF nodes execute their body graph once if the condition value is non-zero when the node is executed. An optional second body graph can be provided and this will be executed once if the condition value is zero when the node is executed.

  • +
  • Conditional WHILE nodes execute their body graph if the condition value is non-zero when the node is executed and will continue to execute their body graph until the condition value is zero.

  • +
  • Conditional SWITCH nodes execute the nth body graph once if the condition value is equal to n. If the condition value does not correspond to a body graph, no body graph is launched.

  • +
+

A condition value is accessed by a conditional handle , which must be created before the node. The condition value can be set by device code using cudaGraphSetConditional(). A default value, applied on each graph launch, can also be specified when the handle is created.

+

When the conditional node is created, an empty graph is created and the handle is returned to the user so that the graph can be populated. This conditional body graph can be populated using either the graph APIs or cudaStreamBeginCaptureToGraph().

+

Conditional nodes can be nested.

+
+
+6.2.8.7.8.1. Conditional Handles +
+

A condition value is represented by cudaGraphConditionalHandle and is created by cudaGraphConditionalHandleCreate().

+

The handle must be associated with a single conditional node. Handles cannot be destroyed.

+

If cudaGraphCondAssignDefault is specified when the handle is created, the condition value will be initialized to the specified default at the beginning of each graph execution. If this flag is not provided, the condition value is undefined at the start of each graph execution and code should not assume that the condition value persists across executions.

+

The default value and flags associated with a handle will be updated during whole graph update.

+
+
+
+6.2.8.7.8.2. Conditional Node Body Graph Requirements +
+

General requirements:

+
    +
  • The graph’s nodes must all reside on a single device.

  • +
  • The graph can only contain kernel nodes, empty nodes, memcpy nodes, memset nodes, child graph nodes, and conditional nodes.

  • +
+

Kernel nodes:

+
    +
  • Use of CUDA Dynamic Parallelism or Device Graph Launch by kernels in the graph is not permitted.

  • +
  • Cooperative launches are permitted so long as MPS is not in use.

  • +
+

Memcpy/Memset nodes:

+
    +
  • Only copies/memsets involving device memory and/or pinned device-mapped host memory are permitted.

  • +
  • Copies/memsets involving CUDA arrays are not permitted.

  • +
  • Both operands must be accessible from the current device at time of instantiation. Note that the copy operation will be performed from the device on which the graph resides, even if it is targeting memory on another device.

  • +
+
+
+
+6.2.8.7.8.3. Conditional IF Nodes +
+

The body graph of an IF node will be executed once if the condition is non-zero when the node is executed. The following diagram depicts a 3 node graph where the middle node, B, is a conditional node:

+
+_images/conditional-if-node.png +
+

Figure 23 Conditional IF Node

+
+
+

The following code illustrates the creation of a graph containing an IF conditional node. The default value of the condition is set using an upstream kernel. The body of the conditional is populated using the graph API.

+
+
__global__ void setHandle(cudaGraphConditionalHandle handle)
+{
+    ...
+    cudaGraphSetConditional(handle, value);
+    ...
+}
+
+void graphSetup() {
+    cudaGraph_t graph;
+    cudaGraphExec_t graphExec;
+    cudaGraphNode_t node;
+    void *kernelArgs[1];
+    int value = 1;
+
+    cudaGraphCreate(&graph, 0);
+
+    cudaGraphConditionalHandle handle;
+    cudaGraphConditionalHandleCreate(&handle, graph);
+
+    // Use a kernel upstream of the conditional to set the handle value
+    cudaGraphNodeParams params = { cudaGraphNodeTypeKernel };
+    params.kernel.func = (void *)setHandle;
+    params.kernel.gridDim.x = params.kernel.gridDim.y = params.kernel.gridDim.z = 1;
+    params.kernel.blockDim.x = params.kernel.blockDim.y = params.kernel.blockDim.z = 1;
+    params.kernel.kernelParams = kernelArgs;
+    kernelArgs[0] = &handle;
+    cudaGraphAddNode(&node, graph, NULL, NULL, 0, &params);
+
+    cudaGraphNodeParams cParams = { cudaGraphNodeTypeConditional };
+    cParams.conditional.handle = handle;
+    cParams.conditional.type   = cudaGraphCondTypeIf;
+    cParams.conditional.size   = 1;
+    cudaGraphAddNode(&node, graph, &node, NULL, 1, &cParams);
+
+    cudaGraph_t bodyGraph = cParams.conditional.phGraph_out[0];
+
+    // Populate the body of the conditional node
+    ...
+    cudaGraphAddNode(&node, bodyGraph, NULL, NULL, 0, &params);
+
+    cudaGraphInstantiate(&graphExec, graph, NULL, NULL, 0);
+    cudaGraphLaunch(graphExec, 0);
+    cudaDeviceSynchronize();
+
+    cudaGraphExecDestroy(graphExec);
+    cudaGraphDestroy(graph);
+}
+
+
+

Starting in CUDA 12.8, IF nodes can also have an optional second body graph which is executed once when the node is executed if the condition value is zero.

+
+
void graphSetup() {
+    cudaGraph_t graph;
+    cudaGraphExec_t graphExec;
+    cudaGraphNode_t node;
+    void *kernelArgs[1];
+    int value = 1;
+
+    cudaGraphCreate(&graph, 0);
+
+    cudaGraphConditionalHandle handle;
+    cudaGraphConditionalHandleCreate(&handle, graph);
+
+    // Use a kernel upstream of the conditional to set the handle value
+    cudaGraphNodeParams params = { cudaGraphNodeTypeKernel };
+    params.kernel.func = (void *)setHandle;
+    params.kernel.gridDim.x = params.kernel.gridDim.y = params.kernel.gridDim.z = 1;
+    params.kernel.blockDim.x = params.kernel.blockDim.y = params.kernel.blockDim.z = 1;
+    params.kernel.kernelParams = kernelArgs;
+    kernelArgs[0] = &handle;
+    cudaGraphAddNode(&node, graph, NULL, NULL, 0, &params);
+
+    cudaGraphNodeParams cParams = { cudaGraphNodeTypeConditional };
+    cParams.conditional.handle = handle;
+    cParams.conditional.type   = cudaGraphCondTypeIf;
+    cParams.conditional.size   = 2; // Note that size is now set to '2'
+    cudaGraphAddNode(&node, graph, &node, NULL, 1, &cParams);
+
+    cudaGraph_t ifBodyGraph = cParams.conditional.phGraph_out[0];
+    cudaGraph_t elseBodyGraph = cParams.conditional.phGraph_out[1];
+
+    // Populate the body graphs of the conditional node
+    ...
+    cudaGraphAddNode(&node, ifBodyGraph, NULL, NULL, 0, &params);
+    ...
+    cudaGraphAddNode(&node, elseBodyGraph, NULL, NULL, 0, &params);
+
+    cudaGraphInstantiate(&graphExec, graph, NULL, NULL, 0);
+    cudaGraphLaunch(graphExec, 0);
+    cudaDeviceSynchronize();
+
+    cudaGraphExecDestroy(graphExec);
+    cudaGraphDestroy(graph);
+}
+
+
+
+
+
+6.2.8.7.8.4. Conditional WHILE Nodes +
+

The body graph of a WHILE node will be executed as long as the condition is non-zero. The condition will be +evaluated when the node is executed and after completion of the body graph. The following diagram depicts +a 3 node graph where the middle node, B, is a conditional node:

+
+_images/conditional-while-node.png +
+

Figure 24 Conditional WHILE Node

+
+
+

The following code illustrates the creation of a graph containing a WHILE conditional node. The handle +is created using cudaGraphCondAssignDefault to avoid the need for an upstream kernel. The body of the +conditional is populated using the graph API.

+
+
__global__ void loopKernel(cudaGraphConditionalHandle handle)
+{
+    static int count = 10;
+    cudaGraphSetConditional(handle, --count ? 1 : 0);
+}
+
+void graphSetup() {
+    cudaGraph_t graph;
+    cudaGraphExec_t graphExec;
+    cudaGraphNode_t node;
+    void *kernelArgs[1];
+
+    cuGraphCreate(&graph, 0);
+
+    cudaGraphConditionalHandle handle;
+    cudaGraphConditionalHandleCreate(&handle, graph, 1, cudaGraphCondAssignDefault);
+
+    cudaGraphNodeParams cParams = { cudaGraphNodeTypeConditional };
+    cParams.conditional.handle = handle;
+    cParams.conditional.type   = cudaGraphCondTypeWhile;
+    cParams.conditional.size   = 1;
+    cudaGraphAddNode(&node, graph, NULL, NULL, 0, &cParams);
+
+    cudaGraph_t bodyGraph = cParams.conditional.phGraph_out[0];
+
+    cudaGraphNodeParams params = { cudaGraphNodeTypeKernel };
+    params.kernel.func = (void *)loopKernel;
+    params.kernel.gridDim.x = params.kernel.gridDim.y = params.kernel.gridDim.z = 1;
+    params.kernel.blockDim.x = params.kernel.blockDim.y = params.kernel.blockDim.z = 1;
+    params.kernel.kernelParams = kernelArgs;
+    kernelArgs[0] = &handle;
+    cudaGraphAddNode(&node, bodyGraph, NULL, NULL, 0, &params);
+
+    cudaGraphInstantiate(&graphExec, graph, NULL, NULL, 0);
+    cudaGraphLaunch(graphExec, 0);
+    cudaDeviceSynchronize();
+
+    cudaGraphExecDestroy(graphExec);
+    cudaGraphDestroy(graph);
+}
+
+
+
+
+
+6.2.8.7.8.5. Conditional SWITCH Nodes +
+

SWITCH nodes, added in CUDA 12.8, execute 1 of n different graphs within the conditional node. The nth graph will be executed when the SWITCH node is evaluated if the condition value is n. If the condition value is greater than or equal to n, no graph will be executed. The following diagram depicts a 3 node graph where the middle node, B, is a conditional node:

+
+_images/conditional-switch-node.png +
+

Figure 25 Conditional SWITCH Node

+
+
+

The following code illustrates the creation of a graph containing a SWITCH conditional node. The value of the condition is set using an upstream kernel. The bodies of the conditional are populated using the graph API.

+
+
__global__ void setHandle(cudaGraphConditionalHandle handle)
+{
+    ...
+    cudaGraphSetConditional(handle, value);
+    ...
+}
+
+void graphSetup() {
+    cudaGraph_t graph;
+    cudaGraphExec_t graphExec;
+    cudaGraphNode_t node;
+    void *kernelArgs[1];
+    int value = 1;
+
+    cudaGraphCreate(&graph, 0);
+
+    cudaGraphConditionalHandle handle;
+    cudaGraphConditionalHandleCreate(&handle, graph);
+
+    // Use a kernel upstream of the conditional to set the handle value
+    cudaGraphNodeParams params = { cudaGraphNodeTypeKernel };
+    params.kernel.func = (void *)setHandle;
+    params.kernel.gridDim.x = params.kernel.gridDim.y = params.kernel.gridDim.z = 1;
+    params.kernel.blockDim.x = params.kernel.blockDim.y = params.kernel.blockDim.z = 1;
+    params.kernel.kernelParams = kernelArgs;
+    kernelArgs[0] = &handle;
+    cudaGraphAddNode(&node, graph, NULL, NULL, 0, &params);
+
+    cudaGraphNodeParams cParams = { cudaGraphNodeTypeConditional };
+    cParams.conditional.handle = handle;
+    cParams.conditional.type   = cudaGraphCondTypeSwitch;
+    cParams.conditional.size   = 5;
+    cudaGraphAddNode(&node, graph, &node, NULL, 1, &cParams);
+
+    cudaGraph_t *bodyGraphs = cParams.conditional.phGraph_out;
+
+    // Populate the first body of the conditional node
+    ...
+    cudaGraphAddNode(&node, bodyGraphs[0], NULL, NULL, 0, &params);
+    ...
+    // Populate the last body of the conditional node
+    cudaGraphAddNode(&node, bodyGraphs[4], NULL, NULL, 0, &params);
+
+    cudaGraphInstantiate(&graphExec, graph, NULL, NULL, 0);
+    cudaGraphLaunch(graphExec, 0);
+    cudaDeviceSynchronize();
+
+    cudaGraphExecDestroy(graphExec);
+    cudaGraphDestroy(graph);
+}
+
+
+
+
+
+
+

+6.2.8.8. Events +

+

The runtime also provides a way to closely monitor the device’s progress, as well as perform accurate timing, by letting the application asynchronously record events at any point in the program, and query when these events are completed. An event has completed when all tasks - or optionally, all commands in a given stream - preceding the event have completed. Events in stream zero are completed after all preceding tasks and commands in all streams are completed.

+
+
+6.2.8.8.1. Creation and Destruction of Events +
+

The following code sample creates two events:

+
+
cudaEvent_t start, stop;
+cudaEventCreate(&start);
+cudaEventCreate(&stop);
+
+
+

They are destroyed this way:

+
+
cudaEventDestroy(start);
+cudaEventDestroy(stop);
+
+
+
+
+
+6.2.8.8.2. Elapsed Time +
+

The events created in Creation and Destruction of Events can be used to time the code sample of Creation and Destruction of Streams the following way:

+
+
cudaEventRecord(start, 0);
+for (int i = 0; i < 2; ++i) {
+    cudaMemcpyAsync(inputDev + i * size, inputHost + i * size,
+                    size, cudaMemcpyHostToDevice, stream[i]);
+    MyKernel<<<100, 512, 0, stream[i]>>>
+               (outputDev + i * size, inputDev + i * size, size);
+    cudaMemcpyAsync(outputHost + i * size, outputDev + i * size,
+                    size, cudaMemcpyDeviceToHost, stream[i]);
+}
+cudaEventRecord(stop, 0);
+cudaEventSynchronize(stop);
+float elapsedTime;
+cudaEventElapsedTime(&elapsedTime, start, stop);
+
+
+
+
+
+

+6.2.8.9. Synchronous Calls +

+

When a synchronous function is called, control is not returned to the host thread before the device has completed the requested task. Whether the host thread will then yield, block, or spin can be specified by calling cudaSetDeviceFlags()with some specific flags (see reference manual for details) before any other CUDA call is performed by the host thread.

+
+
+
+

+6.2.9. Multi-Device System +

+
+

+6.2.9.1. Device Enumeration +

+

A host system can have multiple devices. The following code sample shows how to enumerate these devices, query their properties, and determine the number of CUDA-enabled devices.

+
+
int deviceCount;
+cudaGetDeviceCount(&deviceCount);
+int device;
+for (device = 0; device < deviceCount; ++device) {
+    cudaDeviceProp deviceProp;
+    cudaGetDeviceProperties(&deviceProp, device);
+    printf("Device %d has compute capability %d.%d.\n",
+           device, deviceProp.major, deviceProp.minor);
+}
+
+
+
+
+

+6.2.9.2. Device Selection +

+

A host thread can set the device it operates on at any time by calling cudaSetDevice(). Device memory allocations and kernel launches are made on the currently set device; streams and events are created in association with the currently set device. If no call to cudaSetDevice() is made, the current device is device 0.

+

The following code sample illustrates how setting the current device affects memory allocation and kernel execution.

+
+
size_t size = 1024 * sizeof(float);
+cudaSetDevice(0);            // Set device 0 as current
+float* p0;
+cudaMalloc(&p0, size);       // Allocate memory on device 0
+MyKernel<<<1000, 128>>>(p0); // Launch kernel on device 0
+cudaSetDevice(1);            // Set device 1 as current
+float* p1;
+cudaMalloc(&p1, size);       // Allocate memory on device 1
+MyKernel<<<1000, 128>>>(p1); // Launch kernel on device 1
+
+
+
+
+

+6.2.9.3. Stream and Event Behavior +

+

A kernel launch will fail if it is issued to a stream that is not associated to the current device as illustrated in the following code sample.

+
+
cudaSetDevice(0);               // Set device 0 as current
+cudaStream_t s0;
+cudaStreamCreate(&s0);          // Create stream s0 on device 0
+MyKernel<<<100, 64, 0, s0>>>(); // Launch kernel on device 0 in s0
+cudaSetDevice(1);               // Set device 1 as current
+cudaStream_t s1;
+cudaStreamCreate(&s1);          // Create stream s1 on device 1
+MyKernel<<<100, 64, 0, s1>>>(); // Launch kernel on device 1 in s1
+
+// This kernel launch will fail:
+MyKernel<<<100, 64, 0, s0>>>(); // Launch kernel on device 1 in s0
+
+
+

A memory copy will succeed even if it is issued to a stream that is not associated to the current device.

+

cudaEventRecord() will fail if the input event and input stream are associated to different devices.

+

cudaEventElapsedTime() will fail if the two input events are associated to different devices.

+

cudaEventSynchronize() and cudaEventQuery() will succeed even if the input event is associated to a device that is different from the current device.

+

cudaStreamWaitEvent() will succeed even if the input stream and input event are associated to different devices. cudaStreamWaitEvent() can therefore be used to synchronize multiple devices with each other.

+

Each device has its own default stream (see Default Stream), so commands issued to the default stream of a device may execute out of order or concurrently with respect to commands issued to the default stream of any other device.

+
+
+

+6.2.9.4. Peer-to-Peer Memory Access +

+

Depending on the system properties, specifically the PCIe and/or NVLINK topology, devices are able to address each other’s memory (i.e., a kernel executing on one device can dereference a pointer to the memory of the other device). This peer-to-peer memory access feature is supported between two devices if cudaDeviceCanAccessPeer() returns true for these two devices.

+

Peer-to-peer memory access is only supported in 64-bit applications and must be enabled between two devices by calling cudaDeviceEnablePeerAccess() as illustrated in the following code sample. On non-NVSwitch enabled systems, each device can support a system-wide maximum of eight peer connections.

+

A unified address space is used for both devices (see Unified Virtual Address Space), so the same pointer can be used to address memory from both devices as shown in the code sample below.

+
+
cudaSetDevice(0);                   // Set device 0 as current
+float* p0;
+size_t size = 1024 * sizeof(float);
+cudaMalloc(&p0, size);              // Allocate memory on device 0
+MyKernel<<<1000, 128>>>(p0);        // Launch kernel on device 0
+cudaSetDevice(1);                   // Set device 1 as current
+cudaDeviceEnablePeerAccess(0, 0);   // Enable peer-to-peer access
+                                    // with device 0
+
+// Launch kernel on device 1
+// This kernel launch can access memory on device 0 at address p0
+MyKernel<<<1000, 128>>>(p0);
+
+
+
+
+6.2.9.4.1. IOMMU on Linux +
+

On Linux only, CUDA and the display driver does not support IOMMU-enabled bare-metal PCIe peer to peer memory copy. However, CUDA and the display driver does support IOMMU via VM pass through. As a consequence, users on Linux, when running on a native bare metal system, should disable the IOMMU. The IOMMU should be enabled and the VFIO driver be used as a PCIe pass through for virtual machines.

+

On Windows the above limitation does not exist.

+

See also Allocating DMA Buffers on 64-bit Platforms.

+
+
+
+

+6.2.9.5. Peer-to-Peer Memory Copy +

+

Memory copies can be performed between the memories of two different devices.

+

When a unified address space is used for both devices (see Unified Virtual Address Space), this is done using the regular memory copy functions mentioned in Device Memory.

+

Otherwise, this is done using cudaMemcpyPeer(), cudaMemcpyPeerAsync(), cudaMemcpy3DPeer(), or cudaMemcpy3DPeerAsync() as illustrated in the following code sample.

+
+
cudaSetDevice(0);                   // Set device 0 as current
+float* p0;
+size_t size = 1024 * sizeof(float);
+cudaMalloc(&p0, size);              // Allocate memory on device 0
+cudaSetDevice(1);                   // Set device 1 as current
+float* p1;
+cudaMalloc(&p1, size);              // Allocate memory on device 1
+cudaSetDevice(0);                   // Set device 0 as current
+MyKernel<<<1000, 128>>>(p0);        // Launch kernel on device 0
+cudaSetDevice(1);                   // Set device 1 as current
+cudaMemcpyPeer(p1, 1, p0, 0, size); // Copy p0 to p1
+MyKernel<<<1000, 128>>>(p1);        // Launch kernel on device 1
+
+
+

A copy (in the implicit NULL stream) between the memories of two different devices:

+
    +
  • does not start until all commands previously issued to either device have completed and

  • +
  • runs to completion before any commands (see Asynchronous Concurrent Execution) issued after the copy to either device can start.

  • +
+

Consistent with the normal behavior of streams, an asynchronous copy between the memories of two devices may overlap with copies or kernels in another stream.

+

Note that if peer-to-peer access is enabled between two devices via cudaDeviceEnablePeerAccess() as described in Peer-to-Peer Memory Access, peer-to-peer memory copy between these two devices no longer needs to be staged through the host and is therefore faster.

+
+
+
+

+6.2.10. Unified Virtual Address Space +

+

When the application is run as a 64-bit process, a single address space is used for the host and all the devices of compute capability 2.0 and higher. All host memory allocations made via CUDA API calls and all device memory allocations on supported devices are within this virtual address range. As a consequence:

+
    +
  • The location of any memory on the host allocated through CUDA, or on any of the devices which use the unified address space, can be determined from the value of the pointer using cudaPointerGetAttributes().

  • +
  • When copying to or from the memory of any device which uses the unified address space, the cudaMemcpyKind parameter of cudaMemcpy*() can be set to cudaMemcpyDefault to determine locations from the pointers. This also works for host pointers not allocated through CUDA, as long as the current device uses unified addressing.

  • +
  • Allocations via cudaHostAlloc() are automatically portable (see Portable Memory) across all the devices for which the unified address space is used, and pointers returned by cudaHostAlloc() can be used directly from within kernels running on these devices (i.e., there is no need to obtain a device pointer via cudaHostGetDevicePointer() as described in Mapped Memory.

  • +
+

Applications may query if the unified address space is used for a particular device by checking that the unifiedAddressing device property (see Device Enumeration) is equal to 1.

+
+
+

+6.2.11. Interprocess Communication +

+

Any device memory pointer or event handle created by a host thread can be directly referenced by any other thread within the same process. It is not valid outside this process however, and therefore cannot be directly referenced by threads belonging to a different process.

+

To share device memory pointers and events across processes, an application must use the Inter Process Communication API, which is described in detail in the reference manual. The IPC API is only supported for 64-bit processes on Linux and for devices of compute capability 2.0 and higher. Note that the IPC API is not supported for cudaMallocManaged allocations.

+

Using this API, an application can get the IPC handle for a given device memory pointer using cudaIpcGetMemHandle(), pass it to another process using standard IPC mechanisms (for example, interprocess shared memory or files), and use cudaIpcOpenMemHandle() to retrieve a device pointer from the IPC handle that is a valid pointer within this other process. Event handles can be shared using similar entry points.

+

Note that allocations made by cudaMalloc() may be sub-allocated from a larger block of memory for performance reasons. In such case, CUDA IPC APIs will share the entire underlying memory block which may cause other sub-allocations to be shared, which can potentially lead to information disclosure between processes. To prevent this behavior, it is recommended to only share allocations with a 2MiB aligned size.

+

An example of using the IPC API is where a single primary process generates a batch of input data, making the data available to multiple secondary processes without requiring regeneration or copying.

+

Applications using CUDA IPC to communicate with each other should be compiled, linked, and run with the same CUDA driver and runtime.

+
+

Note

+

Since CUDA 11.5, only events-sharing IPC APIs are supported on L4T and embedded Linux Tegra devices with compute capability 7.x and higher. The memory-sharing IPC APIs are still not supported on Tegra platforms.

+
+
+
+

+6.2.12. Error Checking +

+

All runtime functions return an error code, but for an asynchronous function (see Asynchronous Concurrent Execution), this error code cannot possibly report any of the asynchronous errors that could occur on the device since the function returns before the device has completed the task; the error code only reports errors that occur on the host prior to executing the task, typically related to parameter validation; if an asynchronous error occurs, it will be reported by some subsequent unrelated runtime function call.

+

The only way to check for asynchronous errors just after some asynchronous function call is therefore to synchronize just after the call by calling cudaDeviceSynchronize() (or by using any other synchronization mechanisms described in Asynchronous Concurrent Execution) and checking the error code returned by cudaDeviceSynchronize().

+

The runtime maintains an error variable for each host thread that is initialized to cudaSuccess and is overwritten by the error code every time an error occurs (be it a parameter validation error or an asynchronous error). cudaPeekAtLastError() returns this variable. cudaGetLastError() returns this variable and resets it to cudaSuccess.

+

Kernel launches do not return any error code, so cudaPeekAtLastError() or cudaGetLastError() must be called just after the kernel launch to retrieve any pre-launch errors. To ensure that any error returned by cudaPeekAtLastError() or cudaGetLastError() does not originate from calls prior to the kernel launch, one has to make sure that the runtime error variable is set to cudaSuccess just before the kernel launch, for example, by calling cudaGetLastError() just before the kernel launch. Kernel launches are asynchronous, so to check for asynchronous errors, the application must synchronize in-between the kernel launch and the call to cudaPeekAtLastError() or cudaGetLastError().

+

Note that cudaErrorNotReady that may be returned by cudaStreamQuery() and cudaEventQuery() is not considered an error and is therefore not reported by cudaPeekAtLastError() or cudaGetLastError().

+
+
+

+6.2.13. Call Stack +

+

On devices of compute capability 2.x and higher, the size of the call stack can be queried usingcudaDeviceGetLimit() and set using cudaDeviceSetLimit().

+

When the call stack overflows, the kernel call fails with a stack overflow error if the application is run via a CUDA debugger (CUDA-GDB, Nsight) or an unspecified launch error, otherwise. +When the compiler cannot determine the stack size, it issues a warning saying Stack size cannot be statically determined. This is usually the case with recursive functions. +Once this warning is issued, user will need to set stack size manually if default stack size is not sufficient.

+
+
+

+6.2.14. Texture and Surface Memory +

+

CUDA supports a subset of the texturing hardware that the GPU uses for graphics to access texture and surface memory. Reading data from texture or surface memory instead of global memory can have several performance benefits as described in Device Memory Accesses.

+
+

+6.2.14.1. Texture Memory +

+

Texture memory is read from kernels using the device functions described in Texture Functions. The process of reading a texture calling one of these functions is called a texture fetch. Each texture fetch specifies a parameter called a texture object for the texture object API.

+

The texture object specifies:

+
    +
  • The texture, which is the piece of texture memory that is fetched. Texture objects are created at runtime and the texture is specified when creating the texture object as described in Texture Object API.

  • +
  • Its dimensionality that specifies whether the texture is addressed as a one dimensional array using one texture coordinate, a two-dimensional array using two texture coordinates, or a three-dimensional array using three texture coordinates. Elements of the array are called texels, short for texture elements. The texture width, height, and depth refer to the size of the array in each dimension. Table 27 lists the maximum texture width, height, and depth depending on the compute capability of the device.

  • +
  • The type of a texel, which is restricted to the basic integer and single-precision floating-point types and any of the 1-, 2-, and 4-component vector types defined in Built-in Vector Types that are derived from the basic integer and single-precision floating-point types.

  • +
  • The read mode, which is equal to cudaReadModeNormalizedFloat or cudaReadModeElementType. If it is cudaReadModeNormalizedFloat and the type of the texel is a 16-bit or 8-bit integer type, the value returned by the texture fetch is actually returned as floating-point type and the full range of the integer type is mapped to [0.0, 1.0] for unsigned integer type and [-1.0, 1.0] for signed integer type; for example, an unsigned 8-bit texture element with the value 0xff reads as 1. If it is cudaReadModeElementType, no conversion is performed.

  • +
  • Whether texture coordinates are normalized or not. By default, textures are referenced (by the functions of Texture Functions) using floating-point coordinates in the range [0, N-1] where N is the size of the texture in the dimension corresponding to the coordinate. For example, a texture that is 64x32 in size will be referenced with coordinates in the range [0, 63] and [0, 31] for the x and y dimensions, respectively. Normalized texture coordinates cause the coordinates to be specified in the range [0.0, 1.0-1/N] instead of [0, N-1], so the same 64x32 texture would be addressed by normalized coordinates in the range [0, 1-1/N] in both the x and y dimensions. Normalized texture coordinates are a natural fit to some applications’ requirements, if it is preferable for the texture coordinates to be independent of the texture size.

  • +
  • The addressing mode. It is valid to call the device functions of Section B.8 with coordinates that are out of range. The addressing mode defines what happens in that case. The default addressing mode is to clamp the coordinates to the valid range: [0, N) for non-normalized coordinates and [0.0, 1.0) for normalized coordinates. If the border mode is specified instead, texture fetches with out-of-range texture coordinates return zero. For normalized coordinates, the wrap mode and the mirror mode are also available. When using the wrap mode, each coordinate x is converted to frac(x)=x - floor(x) where floor(x) is the largest integer not greater than x. When using the mirror mode, each coordinate x is converted to frac(x) if floor(x) is even and 1-frac(x) if floor(x) is odd. The addressing mode is specified as an array of size three whose first, second, and third elements specify the addressing mode for the first, second, and third texture coordinates, respectively; the addressing mode are cudaAddressModeBorder, cudaAddressModeClamp, cudaAddressModeWrap, and cudaAddressModeMirror; cudaAddressModeWrap and cudaAddressModeMirror are only supported for normalized texture coordinates

  • +
  • The filtering mode which specifies how the value returned when fetching the texture is computed based on the input texture coordinates. Linear texture filtering may be done only for textures that are configured to return floating-point data. It performs low-precision interpolation between neighboring texels. When enabled, the texels surrounding a texture fetch location are read and the return value of the texture fetch is interpolated based on where the texture coordinates fell between the texels. Simple linear interpolation is performed for one-dimensional textures, bilinear interpolation for two-dimensional textures, and trilinear interpolation for three-dimensional textures. Texture Fetching gives more details on texture fetching. The filtering mode is equal to cudaFilterModePoint or cudaFilterModeLinear. If it is cudaFilterModePoint, the returned value is the texel whose texture coordinates are the closest to the input texture coordinates. If it is cudaFilterModeLinear, the returned value is the linear interpolation of the two (for a one-dimensional texture), four (for a two dimensional texture), or eight (for a three dimensional texture) texels whose texture coordinates are the closest to the input texture coordinates. cudaFilterModeLinear is only valid for returned values of floating-point type.

  • +
+

Texture Object API introduces the texture object API.

+

16-Bit Floating-Point Textures explains how to deal with 16-bit floating-point textures.

+

Textures can also be layered as described in Layered Textures.

+

Cubemap Textures and Cubemap Layered Textures describe a special type of texture, the cubemap texture.

+

Texture Gather describes a special texture fetch, texture gather.

+
+
+6.2.14.1.1. Texture Object API +
+

A texture object is created using cudaCreateTextureObject() from a resource description of type struct cudaResourceDesc, which specifies the texture, and from a texture description defined as such:

+
+
struct cudaTextureDesc
+{
+    enum cudaTextureAddressMode addressMode[3];
+    enum cudaTextureFilterMode  filterMode;
+    enum cudaTextureReadMode    readMode;
+    int                         sRGB;
+    int                         normalizedCoords;
+    unsigned int                maxAnisotropy;
+    enum cudaTextureFilterMode  mipmapFilterMode;
+    float                       mipmapLevelBias;
+    float                       minMipmapLevelClamp;
+    float                       maxMipmapLevelClamp;
+};
+
+
+
    +
  • addressMode specifies the addressing mode;

  • +
  • filterMode specifies the filter mode;

  • +
  • readMode specifies the read mode;

  • +
  • normalizedCoords specifies whether texture coordinates are normalized or not;

  • +
  • See reference manual for sRGB, maxAnisotropy, mipmapFilterMode, mipmapLevelBias, minMipmapLevelClamp, and maxMipmapLevelClamp.

  • +
+

The following code sample applies some simple transformation kernel to a texture.

+
+
// Simple transformation kernel
+__global__ void transformKernel(float* output,
+                                cudaTextureObject_t texObj,
+                                int width, int height,
+                                float theta)
+{
+    // Calculate normalized texture coordinates
+    unsigned int x = blockIdx.x * blockDim.x + threadIdx.x;
+    unsigned int y = blockIdx.y * blockDim.y + threadIdx.y;
+
+    float u = x / (float)width;
+    float v = y / (float)height;
+
+    // Transform coordinates
+    u -= 0.5f;
+    v -= 0.5f;
+    float tu = u * cosf(theta) - v * sinf(theta) + 0.5f;
+    float tv = v * cosf(theta) + u * sinf(theta) + 0.5f;
+
+    // Read from texture and write to global memory
+    output[y * width + x] = tex2D<float>(texObj, tu, tv);
+}
+
+
+
+
// Host code
+int main()
+{
+    const int height = 1024;
+    const int width = 1024;
+    float angle = 0.5;
+
+    // Allocate and set some host data
+    float *h_data = (float *)std::malloc(sizeof(float) * width * height);
+    for (int i = 0; i < height * width; ++i)
+        h_data[i] = i;
+
+    // Allocate CUDA array in device memory
+    cudaChannelFormatDesc channelDesc =
+        cudaCreateChannelDesc(32, 0, 0, 0, cudaChannelFormatKindFloat);
+    cudaArray_t cuArray;
+    cudaMallocArray(&cuArray, &channelDesc, width, height);
+
+    // Set pitch of the source (the width in memory in bytes of the 2D array pointed
+    // to by src, including padding), we dont have any padding
+    const size_t spitch = width * sizeof(float);
+    // Copy data located at address h_data in host memory to device memory
+    cudaMemcpy2DToArray(cuArray, 0, 0, h_data, spitch, width * sizeof(float),
+                        height, cudaMemcpyHostToDevice);
+
+    // Specify texture
+    struct cudaResourceDesc resDesc;
+    memset(&resDesc, 0, sizeof(resDesc));
+    resDesc.resType = cudaResourceTypeArray;
+    resDesc.res.array.array = cuArray;
+
+    // Specify texture object parameters
+    struct cudaTextureDesc texDesc;
+    memset(&texDesc, 0, sizeof(texDesc));
+    texDesc.addressMode[0] = cudaAddressModeWrap;
+    texDesc.addressMode[1] = cudaAddressModeWrap;
+    texDesc.filterMode = cudaFilterModeLinear;
+    texDesc.readMode = cudaReadModeElementType;
+    texDesc.normalizedCoords = 1;
+
+    // Create texture object
+    cudaTextureObject_t texObj = 0;
+    cudaCreateTextureObject(&texObj, &resDesc, &texDesc, NULL);
+
+    // Allocate result of transformation in device memory
+    float *output;
+    cudaMalloc(&output, width * height * sizeof(float));
+
+    // Invoke kernel
+    dim3 threadsperBlock(16, 16);
+    dim3 numBlocks((width + threadsperBlock.x - 1) / threadsperBlock.x,
+                    (height + threadsperBlock.y - 1) / threadsperBlock.y);
+    transformKernel<<<numBlocks, threadsperBlock>>>(output, texObj, width, height,
+                                                    angle);
+    // Copy data from device back to host
+    cudaMemcpy(h_data, output, width * height * sizeof(float),
+                cudaMemcpyDeviceToHost);
+
+    // Destroy texture object
+    cudaDestroyTextureObject(texObj);
+
+    // Free device memory
+    cudaFreeArray(cuArray);
+    cudaFree(output);
+
+    // Free host memory
+    free(h_data);
+
+    return 0;
+}
+
+
+
+
+
+6.2.14.1.2. 16-Bit Floating-Point Textures +
+

The 16-bit floating-point or half format supported by CUDA arrays is the same as the IEEE 754-2008 binary2 format.

+

CUDA C++ does not support a matching data type, but provides intrinsic functions to convert to and from the 32-bit floating-point format via the unsigned short type: __float2half_rn(float) and __half2float(unsigned short). These functions are only supported in device code. Equivalent functions for the host code can be found in the OpenEXR library, for example.

+

16-bit floating-point components are promoted to 32 bit float during texture fetching before any filtering is performed.

+

A channel description for the 16-bit floating-point format can be created by calling one of the cudaCreateChannelDescHalf*() functions.

+
+
+
+6.2.14.1.3. Layered Textures +
+

A one-dimensional or two-dimensional layered texture (also known as texture array in Direct3D and array texture in OpenGL) is a texture made up of a sequence of layers, all of which are regular textures of same dimensionality, size, and data type.

+

A one-dimensional layered texture is addressed using an integer index and a floating-point texture coordinate; the index denotes a layer within the sequence and the coordinate addresses a texel within that layer. A two-dimensional layered texture is addressed using an integer index and two floating-point texture coordinates; the index denotes a layer within the sequence and the coordinates address a texel within that layer.

+

A layered texture can only be a CUDA array by calling cudaMalloc3DArray() with the cudaArrayLayered flag (and a height of zero for one-dimensional layered texture).

+

Layered textures are fetched using the device functions described in tex1DLayered() and tex2DLayered(). Texture filtering (see Texture Fetching) is done only within a layer, not across layers.

+

Layered textures are only supported on devices of compute capability 2.0 and higher.

+
+
+
+6.2.14.1.4. Cubemap Textures +
+

A cubemap texture is a special type of two-dimensional layered texture that has six layers representing the faces of a cube:

+
    +
  • The width of a layer is equal to its height.

  • +
  • The cubemap is addressed using three texture coordinates x, y, and z that are interpreted as a direction vector emanating from the center of the cube and pointing to one face of the cube and a texel within the layer corresponding to that face. More specifically, the face is selected by the coordinate with largest magnitude m and the corresponding layer is addressed using coordinates (s/m+1)/2 and (t/m+1)/2 where s and t are defined in Table 6.

  • +
+ + ++++++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+Table 6 Cubemap Fetch +

face

m

s

t

|x| > |y| and |x| > |z|

x ≥ 0

0

x

-z

-y

x < 0

1

-x

z

-y

|y| > |x| and |y| > |z|

y ≥ 0

2

y

x

z

y < 0

3

-y

x

-z

|z| > |x| and |z| > |y|

z ≥ 0

4

z

x

-y

z < 0

5

-z

-x

-y

+

A cubemap texture can only be a CUDA array by calling cudaMalloc3DArray() with the cudaArrayCubemap flag.

+

Cubemap textures are fetched using the device function described in texCubemap().

+

Cubemap textures are only supported on devices of compute capability 2.0 and higher.

+
+
+
+6.2.14.1.5. Cubemap Layered Textures +
+

A cubemap layered texture is a layered texture whose layers are cubemaps of same dimension.

+

A cubemap layered texture is addressed using an integer index and three floating-point texture coordinates; the index denotes a cubemap within the sequence and the coordinates address a texel within that cubemap.

+

A cubemap layered texture can only be a CUDA array by calling cudaMalloc3DArray() with the cudaArrayLayered and cudaArrayCubemap flags.

+

Cubemap layered textures are fetched using the device function described in texCubemapLayered(). Texture filtering (see Texture Fetching) is done only within a layer, not across layers.

+

Cubemap layered textures are only supported on devices of compute capability 2.0 and higher.

+
+
+
+6.2.14.1.6. Texture Gather +
+

Texture gather is a special texture fetch that is available for two-dimensional textures only. It is performed by the tex2Dgather() function, which has the same parameters as tex2D(), plus an additional comp parameter equal to 0, 1, 2, or 3 (see tex2Dgather()). It returns four 32-bit numbers that correspond to the value of the component comp of each of the four texels that would have been used for bilinear filtering during a regular texture fetch. For example, if these texels are of values (253, 20, 31, 255), (250, 25, 29, 254), (249, 16, 37, 253), (251, 22, 30, 250), and comp is 2, tex2Dgather() returns (31, 29, 37, 30).

+

Note that texture coordinates are computed with only 8 bits of fractional precision. tex2Dgather() may therefore return unexpected results for cases where tex2D() would use 1.0 for one of its weights (α or β, see Linear Filtering). For example, with an x texture coordinate of 2.49805: xB=x-0.5=1.99805, however the fractional part of xB is stored in an 8-bit fixed-point format. Since 0.99805 is closer to 256.f/256.f than it is to 255.f/256.f, xB has the value 2. A tex2Dgather() in this case would therefore return indices 2 and 3 in x, instead of indices 1 and 2.

+

Texture gather is only supported for CUDA arrays created with the cudaArrayTextureGather flag and of width and height less than the maximum specified in Table 27 for texture gather, which is smaller than for regular texture fetch.

+

Texture gather is only supported on devices of compute capability 2.0 and higher.

+
+
+
+

+6.2.14.2. Surface Memory +

+

For devices of compute capability 2.0 and higher, a CUDA array (described in Cubemap Surfaces), created with the cudaArraySurfaceLoadStore flag, can be read and written via a surface object using the functions described in Surface Functions.

+

Table 27 lists the maximum surface width, height, and depth depending on the compute capability of the device.

+
+
+6.2.14.2.1. Surface Object API +
+

A surface object is created using cudaCreateSurfaceObject() from a resource description of type struct cudaResourceDesc. Unlike +texture memory, surface memory uses byte addressing. This means that the x-coordinate used to access a texture element via texture functions +needs to be multiplied by the byte size of the element to access the same element via a surface function. For example, the element at +texture coordinate x of a one-dimensional floating-point CUDA array bound to a texture object texObj and a surface object surfObj +is read using tex1d(texObj, x) via texObj, but surf1Dread(surfObj, 4*x) via surfObj. Similarly, the element at texture +coordinate x and y of a two-dimensional floating-point CUDA array bound to a texture object texObj and a surface object surfObj +is accessed using tex2d(texObj, x, y) via texObj, but surf2Dread(surfObj, 4*x, y) via surObj (the byte offset of the +y-coordinate is internally calculated from the underlying line pitch of the CUDA array).

+

The following code sample applies some simple transformation kernel to a surface.

+
+
// Simple copy kernel
+__global__ void copyKernel(cudaSurfaceObject_t inputSurfObj,
+                           cudaSurfaceObject_t outputSurfObj,
+                           int width, int height)
+{
+    // Calculate surface coordinates
+    unsigned int x = blockIdx.x * blockDim.x + threadIdx.x;
+    unsigned int y = blockIdx.y * blockDim.y + threadIdx.y;
+    if (x < width && y < height) {
+        uchar4 data;
+        // Read from input surface
+        surf2Dread(&data,  inputSurfObj, x * 4, y);
+        // Write to output surface
+        surf2Dwrite(data, outputSurfObj, x * 4, y);
+    }
+}
+
+// Host code
+int main()
+{
+    const int height = 1024;
+    const int width = 1024;
+
+    // Allocate and set some host data
+    unsigned char *h_data =
+        (unsigned char *)std::malloc(sizeof(unsigned char) * width * height * 4);
+    for (int i = 0; i < height * width * 4; ++i)
+        h_data[i] = i;
+
+    // Allocate CUDA arrays in device memory
+    cudaChannelFormatDesc channelDesc =
+        cudaCreateChannelDesc(8, 8, 8, 8, cudaChannelFormatKindUnsigned);
+    cudaArray_t cuInputArray;
+    cudaMallocArray(&cuInputArray, &channelDesc, width, height,
+                    cudaArraySurfaceLoadStore);
+    cudaArray_t cuOutputArray;
+    cudaMallocArray(&cuOutputArray, &channelDesc, width, height,
+                    cudaArraySurfaceLoadStore);
+
+    // Set pitch of the source (the width in memory in bytes of the 2D array
+    // pointed to by src, including padding), we dont have any padding
+    const size_t spitch = 4 * width * sizeof(unsigned char);
+    // Copy data located at address h_data in host memory to device memory
+    cudaMemcpy2DToArray(cuInputArray, 0, 0, h_data, spitch,
+                        4 * width * sizeof(unsigned char), height,
+                        cudaMemcpyHostToDevice);
+
+    // Specify surface
+    struct cudaResourceDesc resDesc;
+    memset(&resDesc, 0, sizeof(resDesc));
+    resDesc.resType = cudaResourceTypeArray;
+
+    // Create the surface objects
+    resDesc.res.array.array = cuInputArray;
+    cudaSurfaceObject_t inputSurfObj = 0;
+    cudaCreateSurfaceObject(&inputSurfObj, &resDesc);
+    resDesc.res.array.array = cuOutputArray;
+    cudaSurfaceObject_t outputSurfObj = 0;
+    cudaCreateSurfaceObject(&outputSurfObj, &resDesc);
+
+    // Invoke kernel
+    dim3 threadsperBlock(16, 16);
+    dim3 numBlocks((width + threadsperBlock.x - 1) / threadsperBlock.x,
+                    (height + threadsperBlock.y - 1) / threadsperBlock.y);
+    copyKernel<<<numBlocks, threadsperBlock>>>(inputSurfObj, outputSurfObj, width,
+                                                height);
+
+    // Copy data from device back to host
+    cudaMemcpy2DFromArray(h_data, spitch, cuOutputArray, 0, 0,
+                            4 * width * sizeof(unsigned char), height,
+                            cudaMemcpyDeviceToHost);
+
+    // Destroy surface objects
+    cudaDestroySurfaceObject(inputSurfObj);
+    cudaDestroySurfaceObject(outputSurfObj);
+
+    // Free device memory
+    cudaFreeArray(cuInputArray);
+    cudaFreeArray(cuOutputArray);
+
+    // Free host memory
+    free(h_data);
+
+  return 0;
+}
+
+
+
+
+
+6.2.14.2.2. Cubemap Surfaces +
+

Cubemap surfaces are accessed usingsurfCubemapread() and surfCubemapwrite() (surfCubemapread() and surfCubemapwrite()) as a two-dimensional layered surface, i.e., using an integer index denoting a face and two floating-point texture coordinates addressing a texel within the layer corresponding to this face. Faces are ordered as indicated in Table 6.

+
+
+
+6.2.14.2.3. Cubemap Layered Surfaces +
+

Cubemap layered surfaces are accessed using surfCubemapLayeredread() and surfCubemapLayeredwrite() (surfCubemapLayeredread() and surfCubemapLayeredwrite()) as a two-dimensional layered surface, i.e., using an integer index denoting a face of one of the cubemaps and two floating-point texture coordinates addressing a texel within the layer corresponding to this face. Faces are ordered as indicated in Table 6, so index ((2 * 6) + 3), for example, accesses the fourth face of the third cubemap.

+
+
+
+

+6.2.14.3. CUDA Arrays +

+

CUDA arrays are opaque memory layouts optimized for texture fetching. They are one dimensional, two dimensional, or three-dimensional and composed of elements, each of which has 1, 2 or 4 components that may be signed or unsigned 8-, 16-, or 32-bit integers, 16-bit floats, or 32-bit floats. CUDA arrays are only accessible by kernels through texture fetching as described in Texture Memory or surface reading and writing as described in Surface Memory.

+
+
+

+6.2.14.4. Read/Write Coherency +

+

The texture and surface memory is cached (see Device Memory Accesses) and within the same kernel call, the cache is not kept coherent with respect to global memory writes and surface memory writes, so any texture fetch or surface read to an address that has been written to via a global write or a surface write in the same kernel call returns undefined data. In other words, a thread can safely read some texture or surface memory location only if this memory location has been updated by a previous kernel call or memory copy, but not if it has been previously updated by the same thread or another thread from the same kernel call.

+
+
+
+

+6.2.15. Graphics Interoperability +

+

Some resources from OpenGL and Direct3D may be mapped into the address space of CUDA, either to enable CUDA to read data written by OpenGL or Direct3D, or to enable CUDA to write data for consumption by OpenGL or Direct3D.

+

A resource must be registered to CUDA before it can be mapped using the functions mentioned in OpenGL Interoperability and Direct3D Interoperability. These functions return a pointer to a CUDA graphics resource of type struct cudaGraphicsResource. Registering a resource is potentially high-overhead and therefore typically called only once per resource. A CUDA graphics resource is unregistered using cudaGraphicsUnregisterResource(). Each CUDA context which intends to use the resource is required to register it separately.

+

Once a resource is registered to CUDA, it can be mapped and unmapped as many times as necessary using cudaGraphicsMapResources() and cudaGraphicsUnmapResources(). cudaGraphicsResourceSetMapFlags() can be called to specify usage hints (write-only, read-only) that the CUDA driver can use to optimize resource management.

+

A mapped resource can be read from or written to by kernels using the device memory address returned by cudaGraphicsResourceGetMappedPointer() for buffers andcudaGraphicsSubResourceGetMappedArray() for CUDA arrays.

+

Accessing a resource through OpenGL, Direct3D, or another CUDA context while it is mapped produces undefined results. OpenGL Interoperability and Direct3D Interoperability give specifics for each graphics API and some code samples. SLI Interoperability gives specifics for when the system is in SLI mode.

+
+

+6.2.15.1. OpenGL Interoperability +

+

The OpenGL resources that may be mapped into the address space of CUDA are OpenGL buffer, texture, and renderbuffer objects.

+

A buffer object is registered using cudaGraphicsGLRegisterBuffer(). In CUDA, it appears as a device pointer and can therefore be read and written by kernels or via cudaMemcpy() calls.

+

A texture or renderbuffer object is registered using cudaGraphicsGLRegisterImage(). In CUDA, it appears as a CUDA array. Kernels can read from the array by binding it to a texture or surface reference. They can also write to it via the surface write functions if the resource has been registered with the cudaGraphicsRegisterFlagsSurfaceLoadStore flag. The array can also be read and written via cudaMemcpy2D() calls. cudaGraphicsGLRegisterImage() supports all texture formats with 1, 2, or 4 components and an internal type of float (for example, GL_RGBA_FLOAT32), normalized integer (for example, GL_RGBA8, GL_INTENSITY16), and unnormalized integer (for example, GL_RGBA8UI) (please note that since unnormalized integer formats require OpenGL 3.0, they can only be written by shaders, not the fixed function pipeline).

+

The OpenGL context whose resources are being shared has to be current to the host thread making any OpenGL interoperability API calls.

+

Please note: When an OpenGL texture is made bindless (say for example by requesting an image or texture handle using the glGetTextureHandle*/glGetImageHandle* APIs) it cannot be registered with CUDA. The application needs to register the texture for interop before requesting an image or texture handle.

+

The following code sample uses a kernel to dynamically modify a 2D width x height grid of vertices stored in a vertex buffer object:

+
+
GLuint positionsVBO;
+struct cudaGraphicsResource* positionsVBO_CUDA;
+
+int main()
+{
+    // Initialize OpenGL and GLUT for device 0
+    // and make the OpenGL context current
+    ...
+    glutDisplayFunc(display);
+
+    // Explicitly set device 0
+    cudaSetDevice(0);
+
+    // Create buffer object and register it with CUDA
+    glGenBuffers(1, &positionsVBO);
+    glBindBuffer(GL_ARRAY_BUFFER, positionsVBO);
+    unsigned int size = width * height * 4 * sizeof(float);
+    glBufferData(GL_ARRAY_BUFFER, size, 0, GL_DYNAMIC_DRAW);
+    glBindBuffer(GL_ARRAY_BUFFER, 0);
+    cudaGraphicsGLRegisterBuffer(&positionsVBO_CUDA,
+                                 positionsVBO,
+                                 cudaGraphicsMapFlagsWriteDiscard);
+
+    // Launch rendering loop
+    glutMainLoop();
+
+    ...
+}
+
+void display()
+{
+    // Map buffer object for writing from CUDA
+    float4* positions;
+    cudaGraphicsMapResources(1, &positionsVBO_CUDA, 0);
+    size_t num_bytes;
+    cudaGraphicsResourceGetMappedPointer((void**)&positions,
+                                         &num_bytes,
+                                         positionsVBO_CUDA));
+
+    // Execute kernel
+    dim3 dimBlock(16, 16, 1);
+    dim3 dimGrid(width / dimBlock.x, height / dimBlock.y, 1);
+    createVertices<<<dimGrid, dimBlock>>>(positions, time,
+                                          width, height);
+
+    // Unmap buffer object
+    cudaGraphicsUnmapResources(1, &positionsVBO_CUDA, 0);
+
+    // Render from buffer object
+    glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT);
+    glBindBuffer(GL_ARRAY_BUFFER, positionsVBO);
+    glVertexPointer(4, GL_FLOAT, 0, 0);
+    glEnableClientState(GL_VERTEX_ARRAY);
+    glDrawArrays(GL_POINTS, 0, width * height);
+    glDisableClientState(GL_VERTEX_ARRAY);
+
+    // Swap buffers
+    glutSwapBuffers();
+    glutPostRedisplay();
+}
+
+
+
+
void deleteVBO()
+{
+    cudaGraphicsUnregisterResource(positionsVBO_CUDA);
+    glDeleteBuffers(1, &positionsVBO);
+}
+
+__global__ void createVertices(float4* positions, float time,
+                               unsigned int width, unsigned int height)
+{
+    unsigned int x = blockIdx.x * blockDim.x + threadIdx.x;
+    unsigned int y = blockIdx.y * blockDim.y + threadIdx.y;
+
+    // Calculate uv coordinates
+    float u = x / (float)width;
+    float v = y / (float)height;
+    u = u * 2.0f - 1.0f;
+    v = v * 2.0f - 1.0f;
+
+    // calculate simple sine wave pattern
+    float freq = 4.0f;
+    float w = sinf(u * freq + time)
+            * cosf(v * freq + time) * 0.5f;
+
+    // Write positions
+    positions[y * width + x] = make_float4(u, w, v, 1.0f);
+}
+
+
+

On Windows and for Quadro GPUs, cudaWGLGetDevice() can be used to retrieve the CUDA device associated to the handle returned by wglEnumGpusNV(). Quadro GPUs offer higher performance OpenGL interoperability than GeForce and Tesla GPUs in a multi-GPU configuration where OpenGL rendering is performed on the Quadro GPU and CUDA computations are performed on other GPUs in the system.

+
+
+

+6.2.15.2. Direct3D Interoperability +

+

Direct3D interoperability is supported for Direct3D 9Ex, Direct3D 10, and Direct3D 11.

+

A CUDA context may interoperate only with Direct3D devices that fulfill the following criteria: Direct3D 9Ex devices must be created with DeviceType set to D3DDEVTYPE_HAL and BehaviorFlags with the D3DCREATE_HARDWARE_VERTEXPROCESSING flag; Direct3D 10 and Direct3D 11 devices must be created with DriverType set to D3D_DRIVER_TYPE_HARDWARE.

+

The Direct3D resources that may be mapped into the address space of CUDA are Direct3D buffers, textures, and surfaces. These resources are registered using cudaGraphicsD3D9RegisterResource(), cudaGraphicsD3D10RegisterResource(), and cudaGraphicsD3D11RegisterResource().

+

The following code sample uses a kernel to dynamically modify a 2D width x height grid of vertices stored in a vertex buffer object.

+
+
+6.2.15.2.1. Direct3D 9 Version +
+
+
IDirect3D9* D3D;
+IDirect3DDevice9* device;
+struct CUSTOMVERTEX {
+    FLOAT x, y, z;
+    DWORD color;
+};
+IDirect3DVertexBuffer9* positionsVB;
+struct cudaGraphicsResource* positionsVB_CUDA;
+
+int main()
+{
+    int dev;
+    // Initialize Direct3D
+    D3D = Direct3DCreate9Ex(D3D_SDK_VERSION);
+
+    // Get a CUDA-enabled adapter
+    unsigned int adapter = 0;
+    for (; adapter < g_pD3D->GetAdapterCount(); adapter++) {
+        D3DADAPTER_IDENTIFIER9 adapterId;
+        g_pD3D->GetAdapterIdentifier(adapter, 0, &adapterId);
+        if (cudaD3D9GetDevice(&dev, adapterId.DeviceName)
+            == cudaSuccess)
+            break;
+    }
+
+     // Create device
+    ...
+    D3D->CreateDeviceEx(adapter, D3DDEVTYPE_HAL, hWnd,
+                        D3DCREATE_HARDWARE_VERTEXPROCESSING,
+                        &params, NULL, &device);
+
+    // Use the same device
+    cudaSetDevice(dev);
+
+    // Create vertex buffer and register it with CUDA
+    unsigned int size = width * height * sizeof(CUSTOMVERTEX);
+    device->CreateVertexBuffer(size, 0, D3DFVF_CUSTOMVERTEX,
+                               D3DPOOL_DEFAULT, &positionsVB, 0);
+    cudaGraphicsD3D9RegisterResource(&positionsVB_CUDA,
+                                     positionsVB,
+                                     cudaGraphicsRegisterFlagsNone);
+    cudaGraphicsResourceSetMapFlags(positionsVB_CUDA,
+                                    cudaGraphicsMapFlagsWriteDiscard);
+
+    // Launch rendering loop
+    while (...) {
+        ...
+        Render();
+        ...
+    }
+    ...
+}
+
+
+
+
void Render()
+{
+    // Map vertex buffer for writing from CUDA
+    float4* positions;
+    cudaGraphicsMapResources(1, &positionsVB_CUDA, 0);
+    size_t num_bytes;
+    cudaGraphicsResourceGetMappedPointer((void**)&positions,
+                                         &num_bytes,
+                                         positionsVB_CUDA));
+
+    // Execute kernel
+    dim3 dimBlock(16, 16, 1);
+    dim3 dimGrid(width / dimBlock.x, height / dimBlock.y, 1);
+    createVertices<<<dimGrid, dimBlock>>>(positions, time,
+                                          width, height);
+
+    // Unmap vertex buffer
+    cudaGraphicsUnmapResources(1, &positionsVB_CUDA, 0);
+
+    // Draw and present
+    ...
+}
+
+void releaseVB()
+{
+    cudaGraphicsUnregisterResource(positionsVB_CUDA);
+    positionsVB->Release();
+}
+
+__global__ void createVertices(float4* positions, float time,
+                               unsigned int width, unsigned int height)
+{
+    unsigned int x = blockIdx.x * blockDim.x + threadIdx.x;
+    unsigned int y = blockIdx.y * blockDim.y + threadIdx.y;
+
+    // Calculate uv coordinates
+    float u = x / (float)width;
+    float v = y / (float)height;
+    u = u * 2.0f - 1.0f;
+    v = v * 2.0f - 1.0f;
+
+    // Calculate simple sine wave pattern
+    float freq = 4.0f;
+    float w = sinf(u * freq + time)
+            * cosf(v * freq + time) * 0.5f;
+
+    // Write positions
+    positions[y * width + x] =
+                make_float4(u, w, v, __int_as_float(0xff00ff00));
+}
+
+
+
+
+
+6.2.15.2.2. Direct3D 10 Version +
+
+
ID3D10Device* device;
+struct CUSTOMVERTEX {
+    FLOAT x, y, z;
+    DWORD color;
+};
+ID3D10Buffer* positionsVB;
+struct cudaGraphicsResource* positionsVB_CUDA;
+
+int main()
+{
+    int dev;
+    // Get a CUDA-enabled adapter
+    IDXGIFactory* factory;
+    CreateDXGIFactory(__uuidof(IDXGIFactory), (void**)&factory);
+    IDXGIAdapter* adapter = 0;
+    for (unsigned int i = 0; !adapter; ++i) {
+        if (FAILED(factory->EnumAdapters(i, &adapter))
+            break;
+        if (cudaD3D10GetDevice(&dev, adapter) == cudaSuccess)
+            break;
+        adapter->Release();
+    }
+    factory->Release();
+
+    // Create swap chain and device
+    ...
+    D3D10CreateDeviceAndSwapChain(adapter,
+                                  D3D10_DRIVER_TYPE_HARDWARE, 0,
+                                  D3D10_CREATE_DEVICE_DEBUG,
+                                  D3D10_SDK_VERSION,
+                                  &swapChainDesc, &swapChain,
+                                  &device);
+    adapter->Release();
+
+    // Use the same device
+    cudaSetDevice(dev);
+
+    // Create vertex buffer and register it with CUDA
+    unsigned int size = width * height * sizeof(CUSTOMVERTEX);
+    D3D10_BUFFER_DESC bufferDesc;
+    bufferDesc.Usage          = D3D10_USAGE_DEFAULT;
+    bufferDesc.ByteWidth      = size;
+    bufferDesc.BindFlags      = D3D10_BIND_VERTEX_BUFFER;
+    bufferDesc.CPUAccessFlags = 0;
+    bufferDesc.MiscFlags      = 0;
+    device->CreateBuffer(&bufferDesc, 0, &positionsVB);
+    cudaGraphicsD3D10RegisterResource(&positionsVB_CUDA,
+                                      positionsVB,
+                                      cudaGraphicsRegisterFlagsNone);
+                                      cudaGraphicsResourceSetMapFlags(positionsVB_CUDA,
+                                      cudaGraphicsMapFlagsWriteDiscard);
+
+    // Launch rendering loop
+    while (...) {
+        ...
+        Render();
+        ...
+    }
+    ...
+}
+
+
+
+
void Render()
+{
+    // Map vertex buffer for writing from CUDA
+    float4* positions;
+    cudaGraphicsMapResources(1, &positionsVB_CUDA, 0);
+    size_t num_bytes;
+    cudaGraphicsResourceGetMappedPointer((void**)&positions,
+                                         &num_bytes,
+                                         positionsVB_CUDA));
+
+    // Execute kernel
+    dim3 dimBlock(16, 16, 1);
+    dim3 dimGrid(width / dimBlock.x, height / dimBlock.y, 1);
+    createVertices<<<dimGrid, dimBlock>>>(positions, time,
+                                          width, height);
+
+    // Unmap vertex buffer
+    cudaGraphicsUnmapResources(1, &positionsVB_CUDA, 0);
+
+    // Draw and present
+    ...
+}
+
+void releaseVB()
+{
+    cudaGraphicsUnregisterResource(positionsVB_CUDA);
+    positionsVB->Release();
+}
+
+__global__ void createVertices(float4* positions, float time,
+                               unsigned int width, unsigned int height)
+{
+    unsigned int x = blockIdx.x * blockDim.x + threadIdx.x;
+    unsigned int y = blockIdx.y * blockDim.y + threadIdx.y;
+
+    // Calculate uv coordinates
+    float u = x / (float)width;
+    float v = y / (float)height;
+    u = u * 2.0f - 1.0f;
+    v = v * 2.0f - 1.0f;
+
+    // Calculate simple sine wave pattern
+    float freq = 4.0f;
+    float w = sinf(u * freq + time)
+            * cosf(v * freq + time) * 0.5f;
+
+    // Write positions
+    positions[y * width + x] =
+                make_float4(u, w, v, __int_as_float(0xff00ff00));
+}
+
+
+
+
+
+6.2.15.2.3. Direct3D 11 Version +
+
+
ID3D11Device* device;
+struct CUSTOMVERTEX {
+    FLOAT x, y, z;
+    DWORD color;
+};
+ID3D11Buffer* positionsVB;
+struct cudaGraphicsResource* positionsVB_CUDA;
+
+int main()
+{
+    int dev;
+    // Get a CUDA-enabled adapter
+    IDXGIFactory* factory;
+    CreateDXGIFactory(__uuidof(IDXGIFactory), (void**)&factory);
+    IDXGIAdapter* adapter = 0;
+    for (unsigned int i = 0; !adapter; ++i) {
+        if (FAILED(factory->EnumAdapters(i, &adapter))
+            break;
+        if (cudaD3D11GetDevice(&dev, adapter) == cudaSuccess)
+            break;
+        adapter->Release();
+    }
+    factory->Release();
+
+    // Create swap chain and device
+    ...
+    sFnPtr_D3D11CreateDeviceAndSwapChain(adapter,
+                                         D3D11_DRIVER_TYPE_HARDWARE,
+                                         0,
+                                         D3D11_CREATE_DEVICE_DEBUG,
+                                         featureLevels, 3,
+                                         D3D11_SDK_VERSION,
+                                         &swapChainDesc, &swapChain,
+                                         &device,
+                                         &featureLevel,
+                                         &deviceContext);
+    adapter->Release();
+
+    // Use the same device
+    cudaSetDevice(dev);
+
+    // Create vertex buffer and register it with CUDA
+    unsigned int size = width * height * sizeof(CUSTOMVERTEX);
+    D3D11_BUFFER_DESC bufferDesc;
+    bufferDesc.Usage          = D3D11_USAGE_DEFAULT;
+    bufferDesc.ByteWidth      = size;
+    bufferDesc.BindFlags      = D3D11_BIND_VERTEX_BUFFER;
+    bufferDesc.CPUAccessFlags = 0;
+    bufferDesc.MiscFlags      = 0;
+    device->CreateBuffer(&bufferDesc, 0, &positionsVB);
+    cudaGraphicsD3D11RegisterResource(&positionsVB_CUDA,
+                                      positionsVB,
+                                      cudaGraphicsRegisterFlagsNone);
+    cudaGraphicsResourceSetMapFlags(positionsVB_CUDA,
+                                    cudaGraphicsMapFlagsWriteDiscard);
+
+    // Launch rendering loop
+    while (...) {
+        ...
+        Render();
+        ...
+    }
+    ...
+}
+
+
+
+
void Render()
+{
+    // Map vertex buffer for writing from CUDA
+    float4* positions;
+    cudaGraphicsMapResources(1, &positionsVB_CUDA, 0);
+    size_t num_bytes;
+    cudaGraphicsResourceGetMappedPointer((void**)&positions,
+                                         &num_bytes,
+                                         positionsVB_CUDA));
+
+    // Execute kernel
+    dim3 dimBlock(16, 16, 1);
+    dim3 dimGrid(width / dimBlock.x, height / dimBlock.y, 1);
+    createVertices<<<dimGrid, dimBlock>>>(positions, time,
+                                          width, height);
+
+    // Unmap vertex buffer
+    cudaGraphicsUnmapResources(1, &positionsVB_CUDA, 0);
+
+    // Draw and present
+    ...
+}
+
+void releaseVB()
+{
+    cudaGraphicsUnregisterResource(positionsVB_CUDA);
+    positionsVB->Release();
+}
+
+    __global__ void createVertices(float4* positions, float time,
+                          unsigned int width, unsigned int height)
+{
+    unsigned int x = blockIdx.x * blockDim.x + threadIdx.x;
+    unsigned int y = blockIdx.y * blockDim.y + threadIdx.y;
+
+// Calculate uv coordinates
+    float u = x / (float)width;
+    float v = y / (float)height;
+    u = u * 2.0f - 1.0f;
+    v = v * 2.0f - 1.0f;
+
+    // Calculate simple sine wave pattern
+    float freq = 4.0f;
+    float w = sinf(u * freq + time)
+            * cosf(v * freq + time) * 0.5f;
+
+    // Write positions
+    positions[y * width + x] =
+                make_float4(u, w, v, __int_as_float(0xff00ff00));
+}
+
+
+
+
+
+

+6.2.15.3. SLI Interoperability +

+

In a system with multiple GPUs, all CUDA-enabled GPUs are accessible via the CUDA driver and runtime as separate devices. There are however special considerations as described below when the system is in SLI mode.

+

First, an allocation in one CUDA device on one GPU will consume memory on other GPUs that are part of the SLI configuration of the Direct3D or OpenGL device. Because of this, allocations may fail earlier than otherwise expected.

+

Second, applications should create multiple CUDA contexts, one for each GPU in the SLI configuration. While this is not a strict requirement, it avoids unnecessary data transfers between devices. The application can use the cudaD3D[9|10|11]GetDevices() for Direct3D and cudaGLGetDevices() for OpenGL set of calls to identify the CUDA device handle(s) for the device(s) that are performing the rendering in the current and next frame. Given this information the application will typically choose the appropriate device and map Direct3D or OpenGL resources to the CUDA device returned by cudaD3D[9|10|11]GetDevices() or cudaGLGetDevices() when the deviceList parameter is set to cudaD3D[9|10|11]DeviceListCurrentFrame or cudaGLDeviceListCurrentFrame.

+

Please note that resource returned from cudaGraphicsD9D[9|10|11]RegisterResource and cudaGraphicsGLRegister[Buffer|Image] must be only used on device the registration happened. Therefore on SLI configurations when data for different frames is computed on different CUDA devices it is necessary to register the resources for each separately.

+

See Direct3D Interoperability and OpenGL Interoperability for details on how the CUDA runtime interoperate with Direct3D and OpenGL, respectively.

+
+
+
+

+6.2.16. External Resource Interoperability +

+

External resource interoperability allows CUDA to import certain resources that are explicitly exported by other APIs. These objects are typically exported by other APIs using handles native to the Operating System, like file descriptors on Linux or NT handles on Windows. They could also be exported using other unified interfaces such as the NVIDIA Software Communication Interface. There are two types of resources that can be imported: memory objects and synchronization objects.

+

Memory objects can be imported into CUDA using cudaImportExternalMemory(). An imported memory object can be accessed from within kernels using device pointers mapped onto the memory object via cudaExternalMemoryGetMappedBuffer()or CUDA mipmapped arrays mapped via cudaExternalMemoryGetMappedMipmappedArray(). Depending on the type of memory object, it may be possible for more than one mapping to be setup on a single memory object. The mappings must match the mappings setup in the exporting API. Any mismatched mappings result in undefined behavior. Imported memory objects must be freed using cudaDestroyExternalMemory(). Freeing a memory object does not free any mappings to that object. Therefore, any device pointers mapped onto that object must be explicitly freed using cudaFree() and any CUDA mipmapped arrays mapped onto that object must be explicitly freed using cudaFreeMipmappedArray(). It is illegal to access mappings to an object after it has been destroyed.

+

Synchronization objects can be imported into CUDA using cudaImportExternalSemaphore(). An imported synchronization object can then be signaled using cudaSignalExternalSemaphoresAsync() and waited on using cudaWaitExternalSemaphoresAsync(). It is illegal to issue a wait before the corresponding signal has been issued. Also, depending on the type of the imported synchronization object, there may be additional constraints imposed on how they can be signaled and waited on, as described in subsequent sections. Imported semaphore objects must be freed using cudaDestroyExternalSemaphore(). All outstanding signals and waits must have completed before the semaphore object is destroyed.

+
+

+6.2.16.1. Vulkan Interoperability +

+
+
+6.2.16.1.1. Matching device UUIDs +
+

When importing memory and synchronization objects exported by Vulkan, they must be imported and mapped on the same device as they were created on. The CUDA device that corresponds to the Vulkan physical device on which the objects were created can be determined by comparing the UUID of a CUDA device with that of the Vulkan physical device, as shown in the following code sample. Note that the Vulkan physical device should not be part of a device group that contains more than one Vulkan physical device. The device group as returned by vkEnumeratePhysicalDeviceGroups that contains the given Vulkan physical device must have a physical device count of 1.

+
+
int getCudaDeviceForVulkanPhysicalDevice(VkPhysicalDevice vkPhysicalDevice) {
+    VkPhysicalDeviceIDProperties vkPhysicalDeviceIDProperties = {};
+    vkPhysicalDeviceIDProperties.sType = VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_ID_PROPERTIES;
+    vkPhysicalDeviceIDProperties.pNext = NULL;
+
+    VkPhysicalDeviceProperties2 vkPhysicalDeviceProperties2 = {};
+    vkPhysicalDeviceProperties2.sType = VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_PROPERTIES_2;
+    vkPhysicalDeviceProperties2.pNext = &vkPhysicalDeviceIDProperties;
+
+    vkGetPhysicalDeviceProperties2(vkPhysicalDevice, &vkPhysicalDeviceProperties2);
+
+    int cudaDeviceCount;
+    cudaGetDeviceCount(&cudaDeviceCount);
+
+    for (int cudaDevice = 0; cudaDevice < cudaDeviceCount; cudaDevice++) {
+        cudaDeviceProp deviceProp;
+        cudaGetDeviceProperties(&deviceProp, cudaDevice);
+        if (!memcmp(&deviceProp.uuid, vkPhysicalDeviceIDProperties.deviceUUID, VK_UUID_SIZE)) {
+            return cudaDevice;
+        }
+    }
+    return cudaInvalidDeviceId;
+}
+
+
+
+
+
+6.2.16.1.2. Importing Memory Objects +
+

On Linux and Windows 10, both dedicated and non-dedicated memory objects exported by Vulkan can be imported into CUDA. On Windows 7, only dedicated memory objects can be imported. When importing a Vulkan dedicated memory object, the flag cudaExternalMemoryDedicated must be set.

+

A Vulkan memory object exported using VK_EXTERNAL_MEMORY_HANDLE_TYPE_OPAQUE_FD_BIT can be imported into CUDA using the file descriptor associated with that object as shown below. Note that CUDA assumes ownership of the file descriptor once it is imported. Using the file descriptor after a successful import results in undefined behavior.

+
+
cudaExternalMemory_t importVulkanMemoryObjectFromFileDescriptor(int fd, unsigned long long size, bool isDedicated) {
+    cudaExternalMemory_t extMem = NULL;
+    cudaExternalMemoryHandleDesc desc = {};
+
+    memset(&desc, 0, sizeof(desc));
+
+    desc.type = cudaExternalMemoryHandleTypeOpaqueFd;
+    desc.handle.fd = fd;
+    desc.size = size;
+    if (isDedicated) {
+        desc.flags |= cudaExternalMemoryDedicated;
+    }
+
+    cudaImportExternalMemory(&extMem, &desc);
+
+    // Input parameter 'fd' should not be used beyond this point as CUDA has assumed ownership of it
+
+    return extMem;
+}
+
+
+

A Vulkan memory object exported using VK_EXTERNAL_MEMORY_HANDLE_TYPE_OPAQUE_WIN32_BIT can be imported into CUDA using the NT handle associated with that object as shown below. Note that CUDA does not assume ownership of the NT handle and it is the application’s responsibility to close the handle when it is not required anymore. The NT handle holds a reference to the resource, so it must be explicitly freed before the underlying memory can be freed.

+
+
cudaExternalMemory_t importVulkanMemoryObjectFromNTHandle(HANDLE handle, unsigned long long size, bool isDedicated) {
+    cudaExternalMemory_t extMem = NULL;
+    cudaExternalMemoryHandleDesc desc = {};
+
+    memset(&desc, 0, sizeof(desc));
+
+    desc.type = cudaExternalMemoryHandleTypeOpaqueWin32;
+    desc.handle.win32.handle = handle;
+    desc.size = size;
+    if (isDedicated) {
+        desc.flags |= cudaExternalMemoryDedicated;
+    }
+
+    cudaImportExternalMemory(&extMem, &desc);
+
+    // Input parameter 'handle' should be closed if it's not needed anymore
+    CloseHandle(handle);
+
+    return extMem;
+}
+
+
+

A Vulkan memory object exported using VK_EXTERNAL_MEMORY_HANDLE_TYPE_OPAQUE_WIN32_BIT can also be imported using a named handle if one exists as shown below.

+
+
cudaExternalMemory_t importVulkanMemoryObjectFromNamedNTHandle(LPCWSTR name, unsigned long long size, bool isDedicated) {
+    cudaExternalMemory_t extMem = NULL;
+    cudaExternalMemoryHandleDesc desc = {};
+
+    memset(&desc, 0, sizeof(desc));
+
+    desc.type = cudaExternalMemoryHandleTypeOpaqueWin32;
+    desc.handle.win32.name = (void *)name;
+    desc.size = size;
+    if (isDedicated) {
+        desc.flags |= cudaExternalMemoryDedicated;
+    }
+
+    cudaImportExternalMemory(&extMem, &desc);
+
+    return extMem;
+}
+
+
+

A Vulkan memory object exported using VK_EXTERNAL_MEMORY_HANDLE_TYPE_OPAQUE_WIN32_KMT_BIT can be imported into CUDA using the globally shared D3DKMT handle associated with that object as shown below. Since a globally shared D3DKMT handle does not hold a reference to the underlying memory it is automatically destroyed when all other references to the resource are destroyed.

+
+
cudaExternalMemory_t importVulkanMemoryObjectFromKMTHandle(HANDLE handle, unsigned long long size, bool isDedicated) {
+    cudaExternalMemory_t extMem = NULL;
+    cudaExternalMemoryHandleDesc desc = {};
+
+    memset(&desc, 0, sizeof(desc));
+
+    desc.type = cudaExternalMemoryHandleTypeOpaqueWin32Kmt;
+    desc.handle.win32.handle = (void *)handle;
+    desc.size = size;
+    if (isDedicated) {
+        desc.flags |= cudaExternalMemoryDedicated;
+    }
+
+    cudaImportExternalMemory(&extMem, &desc);
+
+    return extMem;
+}
+
+
+
+
+
+6.2.16.1.3. Mapping Buffers onto Imported Memory Objects +
+

A device pointer can be mapped onto an imported memory object as shown below. The offset and size of the mapping must match that specified when creating the mapping using the corresponding Vulkan API. All mapped device pointers must be freed using cudaFree().

+
+
void * mapBufferOntoExternalMemory(cudaExternalMemory_t extMem, unsigned long long offset, unsigned long long size) {
+
+    void *ptr = NULL;
+
+    cudaExternalMemoryBufferDesc desc = {};
+
+
+
+    memset(&desc, 0, sizeof(desc));
+
+
+
+    desc.offset = offset;
+
+    desc.size = size;
+
+
+
+    cudaExternalMemoryGetMappedBuffer(&ptr, extMem, &desc);
+
+
+
+    // Note: ‘ptr’ must eventually be freed using cudaFree()
+
+    return ptr;
+
+}
+
+
+
+
+
+6.2.16.1.4. Mapping Mipmapped Arrays onto Imported Memory Objects +
+

A CUDA mipmapped array can be mapped onto an imported memory object as shown below. The offset, dimensions, format and number of mip levels must match that specified when creating the mapping using the corresponding Vulkan API. Additionally, if the mipmapped array is bound as a color target in Vulkan, the flagcudaArrayColorAttachment must be set. All mapped mipmapped arrays must be freed using cudaFreeMipmappedArray(). The following code sample shows how to convert Vulkan parameters into the corresponding CUDA parameters when mapping mipmapped arrays onto imported memory objects.

+
+
cudaMipmappedArray_t mapMipmappedArrayOntoExternalMemory(cudaExternalMemory_t extMem, unsigned long long offset, cudaChannelFormatDesc *formatDesc, cudaExtent *extent, unsigned int flags, unsigned int numLevels) {
+    cudaMipmappedArray_t mipmap = NULL;
+    cudaExternalMemoryMipmappedArrayDesc desc = {};
+
+    memset(&desc, 0, sizeof(desc));
+
+    desc.offset = offset;
+    desc.formatDesc = *formatDesc;
+    desc.extent = *extent;
+    desc.flags = flags;
+    desc.numLevels = numLevels;
+
+    // Note: 'mipmap' must eventually be freed using cudaFreeMipmappedArray()
+    cudaExternalMemoryGetMappedMipmappedArray(&mipmap, extMem, &desc);
+
+    return mipmap;
+}
+
+cudaChannelFormatDesc getCudaChannelFormatDescForVulkanFormat(VkFormat format)
+{
+    cudaChannelFormatDesc d;
+
+    memset(&d, 0, sizeof(d));
+
+    switch (format) {
+    case VK_FORMAT_R8_UINT:             d.x = 8;  d.y = 0;  d.z = 0;  d.w = 0;  d.f = cudaChannelFormatKindUnsigned; break;
+    case VK_FORMAT_R8_SINT:             d.x = 8;  d.y = 0;  d.z = 0;  d.w = 0;  d.f = cudaChannelFormatKindSigned;   break;
+    case VK_FORMAT_R8G8_UINT:           d.x = 8;  d.y = 8;  d.z = 0;  d.w = 0;  d.f = cudaChannelFormatKindUnsigned; break;
+    case VK_FORMAT_R8G8_SINT:           d.x = 8;  d.y = 8;  d.z = 0;  d.w = 0;  d.f = cudaChannelFormatKindSigned;   break;
+    case VK_FORMAT_R8G8B8A8_UINT:       d.x = 8;  d.y = 8;  d.z = 8;  d.w = 8;  d.f = cudaChannelFormatKindUnsigned; break;
+    case VK_FORMAT_R8G8B8A8_SINT:       d.x = 8;  d.y = 8;  d.z = 8;  d.w = 8;  d.f = cudaChannelFormatKindSigned;   break;
+    case VK_FORMAT_R16_UINT:            d.x = 16; d.y = 0;  d.z = 0;  d.w = 0;  d.f = cudaChannelFormatKindUnsigned; break;
+    case VK_FORMAT_R16_SINT:            d.x = 16; d.y = 0;  d.z = 0;  d.w = 0;  d.f = cudaChannelFormatKindSigned;   break;
+    case VK_FORMAT_R16G16_UINT:         d.x = 16; d.y = 16; d.z = 0;  d.w = 0;  d.f = cudaChannelFormatKindUnsigned; break;
+    case VK_FORMAT_R16G16_SINT:         d.x = 16; d.y = 16; d.z = 0;  d.w = 0;  d.f = cudaChannelFormatKindSigned;   break;
+    case VK_FORMAT_R16G16B16A16_UINT:   d.x = 16; d.y = 16; d.z = 16; d.w = 16; d.f = cudaChannelFormatKindUnsigned; break;
+    case VK_FORMAT_R16G16B16A16_SINT:   d.x = 16; d.y = 16; d.z = 16; d.w = 16; d.f = cudaChannelFormatKindSigned;   break;
+    case VK_FORMAT_R32_UINT:            d.x = 32; d.y = 0;  d.z = 0;  d.w = 0;  d.f = cudaChannelFormatKindUnsigned; break;
+    case VK_FORMAT_R32_SINT:            d.x = 32; d.y = 0;  d.z = 0;  d.w = 0;  d.f = cudaChannelFormatKindSigned;   break;
+    case VK_FORMAT_R32_SFLOAT:          d.x = 32; d.y = 0;  d.z = 0;  d.w = 0;  d.f = cudaChannelFormatKindFloat;    break;
+    case VK_FORMAT_R32G32_UINT:         d.x = 32; d.y = 32; d.z = 0;  d.w = 0;  d.f = cudaChannelFormatKindUnsigned; break;
+    case VK_FORMAT_R32G32_SINT:         d.x = 32; d.y = 32; d.z = 0;  d.w = 0;  d.f = cudaChannelFormatKindSigned;   break;
+    case VK_FORMAT_R32G32_SFLOAT:       d.x = 32; d.y = 32; d.z = 0;  d.w = 0;  d.f = cudaChannelFormatKindFloat;    break;
+    case VK_FORMAT_R32G32B32A32_UINT:   d.x = 32; d.y = 32; d.z = 32; d.w = 32; d.f = cudaChannelFormatKindUnsigned; break;
+    case VK_FORMAT_R32G32B32A32_SINT:   d.x = 32; d.y = 32; d.z = 32; d.w = 32; d.f = cudaChannelFormatKindSigned;   break;
+    case VK_FORMAT_R32G32B32A32_SFLOAT: d.x = 32; d.y = 32; d.z = 32; d.w = 32; d.f = cudaChannelFormatKindFloat;    break;
+    default: assert(0);
+    }
+
+
+
+
    return d;
+}
+
+cudaExtent getCudaExtentForVulkanExtent(VkExtent3D vkExt, uint32_t arrayLayers, VkImageViewType vkImageViewType) {
+    cudaExtent e = { 0, 0, 0 };
+
+    switch (vkImageViewType) {
+    case VK_IMAGE_VIEW_TYPE_1D:         e.width = vkExt.width; e.height = 0;            e.depth = 0;           break;
+    case VK_IMAGE_VIEW_TYPE_2D:         e.width = vkExt.width; e.height = vkExt.height; e.depth = 0;           break;
+    case VK_IMAGE_VIEW_TYPE_3D:         e.width = vkExt.width; e.height = vkExt.height; e.depth = vkExt.depth; break;
+    case VK_IMAGE_VIEW_TYPE_CUBE:       e.width = vkExt.width; e.height = vkExt.height; e.depth = arrayLayers; break;
+    case VK_IMAGE_VIEW_TYPE_1D_ARRAY:   e.width = vkExt.width; e.height = 0;            e.depth = arrayLayers; break;
+    case VK_IMAGE_VIEW_TYPE_2D_ARRAY:   e.width = vkExt.width; e.height = vkExt.height; e.depth = arrayLayers; break;
+    case VK_IMAGE_VIEW_TYPE_CUBE_ARRAY: e.width = vkExt.width; e.height = vkExt.height; e.depth = arrayLayers; break;
+    default: assert(0);
+    }
+
+    return e;
+}
+
+unsigned int getCudaMipmappedArrayFlagsForVulkanImage(VkImageViewType vkImageViewType, VkImageUsageFlags vkImageUsageFlags, bool allowSurfaceLoadStore) {
+    unsigned int flags = 0;
+
+    switch (vkImageViewType) {
+    case VK_IMAGE_VIEW_TYPE_CUBE:       flags |= cudaArrayCubemap;                    break;
+    case VK_IMAGE_VIEW_TYPE_CUBE_ARRAY: flags |= cudaArrayCubemap | cudaArrayLayered; break;
+    case VK_IMAGE_VIEW_TYPE_1D_ARRAY:   flags |= cudaArrayLayered;                    break;
+    case VK_IMAGE_VIEW_TYPE_2D_ARRAY:   flags |= cudaArrayLayered;                    break;
+    default: break;
+    }
+
+    if (vkImageUsageFlags & VK_IMAGE_USAGE_COLOR_ATTACHMENT_BIT) {
+        flags |= cudaArrayColorAttachment;
+    }
+
+    if (allowSurfaceLoadStore) {
+        flags |= cudaArraySurfaceLoadStore;
+    }
+    return flags;
+}
+
+
+
+
+
+6.2.16.1.5. Importing Synchronization Objects +
+

A Vulkan semaphore object exported using VK_EXTERNAL_SEMAPHORE_HANDLE_TYPE_OPAQUE_FD_BITcan be imported into CUDA using the file descriptor associated with that object as shown below. Note that CUDA assumes ownership of the file descriptor once it is imported. Using the file descriptor after a successful import results in undefined behavior.

+
+
cudaExternalSemaphore_t importVulkanSemaphoreObjectFromFileDescriptor(int fd) {
+    cudaExternalSemaphore_t extSem = NULL;
+    cudaExternalSemaphoreHandleDesc desc = {};
+
+    memset(&desc, 0, sizeof(desc));
+
+    desc.type = cudaExternalSemaphoreHandleTypeOpaqueFd;
+    desc.handle.fd = fd;
+
+    cudaImportExternalSemaphore(&extSem, &desc);
+
+    // Input parameter 'fd' should not be used beyond this point as CUDA has assumed ownership of it
+
+    return extSem;
+}
+
+
+

A Vulkan semaphore object exported using VK_EXTERNAL_SEMAPHORE_HANDLE_TYPE_OPAQUE_WIN32_BIT can be imported into CUDA using the NT handle associated with that object as shown below. Note that CUDA does not assume ownership of the NT handle and it is the application’s responsibility to close the handle when it is not required anymore. The NT handle holds a reference to the resource, so it must be explicitly freed before the underlying semaphore can be freed.

+
+
cudaExternalSemaphore_t importVulkanSemaphoreObjectFromNTHandle(HANDLE handle) {
+    cudaExternalSemaphore_t extSem = NULL;
+    cudaExternalSemaphoreHandleDesc desc = {};
+
+    memset(&desc, 0, sizeof(desc));
+
+    desc.type = cudaExternalSemaphoreHandleTypeOpaqueWin32;
+    desc.handle.win32.handle = handle;
+
+    cudaImportExternalSemaphore(&extSem, &desc);
+
+    // Input parameter 'handle' should be closed if it's not needed anymore
+    CloseHandle(handle);
+
+    return extSem;
+}
+
+
+

A Vulkan semaphore object exported using VK_EXTERNAL_SEMAPHORE_HANDLE_TYPE_OPAQUE_WIN32_BIT can also be imported using a named handle if one exists as shown below.

+
+
cudaExternalSemaphore_t importVulkanSemaphoreObjectFromNamedNTHandle(LPCWSTR name) {
+    cudaExternalSemaphore_t extSem = NULL;
+    cudaExternalSemaphoreHandleDesc desc = {};
+
+    memset(&desc, 0, sizeof(desc));
+
+    desc.type = cudaExternalSemaphoreHandleTypeOpaqueWin32;
+    desc.handle.win32.name = (void *)name;
+
+    cudaImportExternalSemaphore(&extSem, &desc);
+
+    return extSem;
+}
+
+
+

A Vulkan semaphore object exported using VK_EXTERNAL_SEMAPHORE_HANDLE_TYPE_OPAQUE_WIN32_KMT_BIT can be imported into CUDA using the globally shared D3DKMT handle associated with that object as shown below. Since a globally shared D3DKMT handle does not hold a reference to the underlying semaphore it is automatically destroyed when all other references to the resource are destroyed.

+
+
cudaExternalSemaphore_t importVulkanSemaphoreObjectFromKMTHandle(HANDLE handle) {
+    cudaExternalSemaphore_t extSem = NULL;
+    cudaExternalSemaphoreHandleDesc desc = {};
+
+    memset(&desc, 0, sizeof(desc));
+
+    desc.type = cudaExternalSemaphoreHandleTypeOpaqueWin32Kmt;
+    desc.handle.win32.handle = (void *)handle;
+
+    cudaImportExternalSemaphore(&extSem, &desc);
+
+    return extSem;
+}
+
+
+
+
+
+6.2.16.1.6. Signaling/Waiting on Imported Synchronization Objects +
+

An imported Vulkan semaphore object can be signaled as shown below. Signaling such a semaphore object sets it to the signaled state. The corresponding wait that waits on this signal must be issued in Vulkan. Additionally, the wait that waits on this signal must be issued after this signal has been issued.

+
+
void signalExternalSemaphore(cudaExternalSemaphore_t extSem, cudaStream_t stream) {
+    cudaExternalSemaphoreSignalParams params = {};
+
+    memset(&params, 0, sizeof(params));
+
+    cudaSignalExternalSemaphoresAsync(&extSem, &params, 1, stream);
+}
+
+
+

An imported Vulkan semaphore object can be waited on as shown below. Waiting on such a semaphore object waits until it reaches the signaled state and then resets it back to the unsignaled state. The corresponding signal that this wait is waiting on must be issued in Vulkan. Additionally, the signal must be issued before this wait can be issued.

+
+
void waitExternalSemaphore(cudaExternalSemaphore_t extSem, cudaStream_t stream) {
+    cudaExternalSemaphoreWaitParams params = {};
+
+    memset(&params, 0, sizeof(params));
+
+    cudaWaitExternalSemaphoresAsync(&extSem, &params, 1, stream);
+}
+
+
+
+
+
+

+6.2.16.2. OpenGL Interoperability +

+

Traditional OpenGL-CUDA interop as outlined in OpenGL Interoperability works by CUDA directly consuming handles created in OpenGL. However, since OpenGL can also consume memory and synchronization objects created in Vulkan, there exists an alternative approach to doing OpenGL-CUDA interop. Essentially, memory and synchronization objects exported by Vulkan could be imported into both, OpenGL and CUDA, and then used to coordinate memory accesses between OpenGL and CUDA. Please refer to the following OpenGL extensions for further details on how to import memory and synchronization objects exported by Vulkan:

+
    +
  • GL_EXT_memory_object

  • +
  • GL_EXT_memory_object_fd

  • +
  • GL_EXT_memory_object_win32

  • +
  • GL_EXT_semaphore

  • +
  • GL_EXT_semaphore_fd

  • +
  • GL_EXT_semaphore_win32

  • +
+
+
+

+6.2.16.3. Direct3D 12 Interoperability +

+
+
+6.2.16.3.1. Matching Device LUIDs +
+

When importing memory and synchronization objects exported by Direct3D 12, they must be imported and mapped on the same device as they were created on. The CUDA device that corresponds to the Direct3D 12 device on which the objects were created can be determined by comparing the LUID of a CUDA device with that of the Direct3D 12 device, as shown in the following code sample. Note that the Direct3D 12 device must not be created on a linked node adapter. I.e. the node count as returned by ID3D12Device::GetNodeCount must be 1.

+
+
int getCudaDeviceForD3D12Device(ID3D12Device *d3d12Device) {
+    LUID d3d12Luid = d3d12Device->GetAdapterLuid();
+
+    int cudaDeviceCount;
+    cudaGetDeviceCount(&cudaDeviceCount);
+
+    for (int cudaDevice = 0; cudaDevice < cudaDeviceCount; cudaDevice++) {
+        cudaDeviceProp deviceProp;
+        cudaGetDeviceProperties(&deviceProp, cudaDevice);
+        char *cudaLuid = deviceProp.luid;
+
+        if (!memcmp(&d3d12Luid.LowPart, cudaLuid, sizeof(d3d12Luid.LowPart)) &&
+            !memcmp(&d3d12Luid.HighPart, cudaLuid + sizeof(d3d12Luid.LowPart), sizeof(d3d12Luid.HighPart))) {
+            return cudaDevice;
+        }
+    }
+    return cudaInvalidDeviceId;
+}
+
+
+
+
+
+6.2.16.3.2. Importing Memory Objects +
+

A shareable Direct3D 12 heap memory object, created by setting the flag D3D12_HEAP_FLAG_SHARED in the call to ID3D12Device::CreateHeap, can be imported into CUDA using the NT handle associated with that object as shown below. Note that it is the application’s responsibility to close the NT handle when it is not required anymore. The NT handle holds a reference to the resource, so it must be explicitly freed before the underlying memory can be freed.

+
+
cudaExternalMemory_t importD3D12HeapFromNTHandle(HANDLE handle, unsigned long long size) {
+    cudaExternalMemory_t extMem = NULL;
+    cudaExternalMemoryHandleDesc desc = {};
+
+    memset(&desc, 0, sizeof(desc));
+
+    desc.type = cudaExternalMemoryHandleTypeD3D12Heap;
+    desc.handle.win32.handle = (void *)handle;
+    desc.size = size;
+
+    cudaImportExternalMemory(&extMem, &desc);
+
+    // Input parameter 'handle' should be closed if it's not needed anymore
+    CloseHandle(handle);
+
+    return extMem;
+}
+
+
+

A shareable Direct3D 12 heap memory object can also be imported using a named handle if one exists as shown below.

+
+
cudaExternalMemory_t importD3D12HeapFromNamedNTHandle(LPCWSTR name, unsigned long long size) {
+    cudaExternalMemory_t extMem = NULL;
+    cudaExternalMemoryHandleDesc desc = {};
+
+    memset(&desc, 0, sizeof(desc));
+
+    desc.type = cudaExternalMemoryHandleTypeD3D12Heap;
+    desc.handle.win32.name = (void *)name;
+    desc.size = size;
+
+    cudaImportExternalMemory(&extMem, &desc);
+
+    return extMem;
+}
+
+
+

A shareable Direct3D 12 committed resource, created by setting the flag D3D12_HEAP_FLAG_SHARED in the call to D3D12Device::CreateCommittedResource, can be imported into CUDA using the NT handle associated with that object as shown below. When importing a Direct3D 12 committed resource, the flag cudaExternalMemoryDedicated must be set. Note that it is the application’s responsibility to close the NT handle when it is not required anymore. The NT handle holds a reference to the resource, so it must be explicitly freed before the underlying memory can be freed.

+
+
cudaExternalMemory_t importD3D12CommittedResourceFromNTHandle(HANDLE handle, unsigned long long size) {
+    cudaExternalMemory_t extMem = NULL;
+    cudaExternalMemoryHandleDesc desc = {};
+
+    memset(&desc, 0, sizeof(desc));
+
+    desc.type = cudaExternalMemoryHandleTypeD3D12Resource;
+    desc.handle.win32.handle = (void *)handle;
+    desc.size = size;
+    desc.flags |= cudaExternalMemoryDedicated;
+
+    cudaImportExternalMemory(&extMem, &desc);
+
+    // Input parameter 'handle' should be closed if it's not needed anymore
+    CloseHandle(handle);
+
+    return extMem;
+}
+
+
+

A shareable Direct3D 12 committed resource can also be imported using a named handle if one exists as shown below.

+
+
cudaExternalMemory_t importD3D12CommittedResourceFromNamedNTHandle(LPCWSTR name, unsigned long long size) {
+    cudaExternalMemory_t extMem = NULL;
+    cudaExternalMemoryHandleDesc desc = {};
+
+    memset(&desc, 0, sizeof(desc));
+
+    desc.type = cudaExternalMemoryHandleTypeD3D12Resource;
+    desc.handle.win32.name = (void *)name;
+    desc.size = size;
+    desc.flags |= cudaExternalMemoryDedicated;
+
+    cudaImportExternalMemory(&extMem, &desc);
+
+    return extMem;
+}
+
+
+
+
+
+6.2.16.3.3. Mapping Buffers onto Imported Memory Objects +
+

A device pointer can be mapped onto an imported memory object as shown below. The offset and size of the mapping must match that specified when creating the mapping using the corresponding Direct3D 12 API. All mapped device pointers must be freed using cudaFree().

+
+
void * mapBufferOntoExternalMemory(cudaExternalMemory_t extMem, unsigned long long offset, unsigned long long size) {
+    void *ptr = NULL;
+    cudaExternalMemoryBufferDesc desc = {};
+
+    memset(&desc, 0, sizeof(desc));
+
+    desc.offset = offset;
+    desc.size = size;
+
+    cudaExternalMemoryGetMappedBuffer(&ptr, extMem, &desc);
+
+    // Note: 'ptr' must eventually be freed using cudaFree()
+    return ptr;
+}
+
+
+
+
+
+6.2.16.3.4. Mapping Mipmapped Arrays onto Imported Memory Objects +
+

A CUDA mipmapped array can be mapped onto an imported memory object as shown below. The offset, dimensions, format and number of mip levels must match that specified when creating the mapping using the corresponding Direct3D 12 API. Additionally, if the mipmapped array can be bound as a render target in Direct3D 12, the flag cudaArrayColorAttachment must be set. All mapped mipmapped arrays must be freed using cudaFreeMipmappedArray(). The following code sample shows how to convert Vulkan parameters into the corresponding CUDA parameters when mapping mipmapped arrays onto imported memory objects.

+
+
cudaMipmappedArray_t mapMipmappedArrayOntoExternalMemory(cudaExternalMemory_t extMem, unsigned long long offset, cudaChannelFormatDesc *formatDesc, cudaExtent *extent, unsigned int flags, unsigned int numLevels) {
+    cudaMipmappedArray_t mipmap = NULL;
+    cudaExternalMemoryMipmappedArrayDesc desc = {};
+
+    memset(&desc, 0, sizeof(desc));
+
+    desc.offset = offset;
+    desc.formatDesc = *formatDesc;
+    desc.extent = *extent;
+    desc.flags = flags;
+    desc.numLevels = numLevels;
+
+    // Note: 'mipmap' must eventually be freed using cudaFreeMipmappedArray()
+    cudaExternalMemoryGetMappedMipmappedArray(&mipmap, extMem, &desc);
+
+    return mipmap;
+}
+
+cudaChannelFormatDesc getCudaChannelFormatDescForDxgiFormat(DXGI_FORMAT dxgiFormat)
+{
+    cudaChannelFormatDesc d;
+
+    memset(&d, 0, sizeof(d));
+
+    switch (dxgiFormat) {
+    case DXGI_FORMAT_R8_UINT:            d.x = 8;  d.y = 0;  d.z = 0;  d.w = 0;  d.f = cudaChannelFormatKindUnsigned; break;
+    case DXGI_FORMAT_R8_SINT:            d.x = 8;  d.y = 0;  d.z = 0;  d.w = 0;  d.f = cudaChannelFormatKindSigned;   break;
+    case DXGI_FORMAT_R8G8_UINT:          d.x = 8;  d.y = 8;  d.z = 0;  d.w = 0;  d.f = cudaChannelFormatKindUnsigned; break;
+    case DXGI_FORMAT_R8G8_SINT:          d.x = 8;  d.y = 8;  d.z = 0;  d.w = 0;  d.f = cudaChannelFormatKindSigned;   break;
+    case DXGI_FORMAT_R8G8B8A8_UINT:      d.x = 8;  d.y = 8;  d.z = 8;  d.w = 8;  d.f = cudaChannelFormatKindUnsigned; break;
+    case DXGI_FORMAT_R8G8B8A8_SINT:      d.x = 8;  d.y = 8;  d.z = 8;  d.w = 8;  d.f = cudaChannelFormatKindSigned;   break;
+    case DXGI_FORMAT_R16_UINT:           d.x = 16; d.y = 0;  d.z = 0;  d.w = 0;  d.f = cudaChannelFormatKindUnsigned; break;
+    case DXGI_FORMAT_R16_SINT:           d.x = 16; d.y = 0;  d.z = 0;  d.w = 0;  d.f = cudaChannelFormatKindSigned;   break;
+    case DXGI_FORMAT_R16G16_UINT:        d.x = 16; d.y = 16; d.z = 0;  d.w = 0;  d.f = cudaChannelFormatKindUnsigned; break;
+    case DXGI_FORMAT_R16G16_SINT:        d.x = 16; d.y = 16; d.z = 0;  d.w = 0;  d.f = cudaChannelFormatKindSigned;   break;
+    case DXGI_FORMAT_R16G16B16A16_UINT:  d.x = 16; d.y = 16; d.z = 16; d.w = 16; d.f = cudaChannelFormatKindUnsigned; break;
+    case DXGI_FORMAT_R16G16B16A16_SINT:  d.x = 16; d.y = 16; d.z = 16; d.w = 16; d.f = cudaChannelFormatKindSigned;   break;
+    case DXGI_FORMAT_R32_UINT:           d.x = 32; d.y = 0;  d.z = 0;  d.w = 0;  d.f = cudaChannelFormatKindUnsigned; break;
+    case DXGI_FORMAT_R32_SINT:           d.x = 32; d.y = 0;  d.z = 0;  d.w = 0;  d.f = cudaChannelFormatKindSigned;   break;
+    case DXGI_FORMAT_R32_FLOAT:          d.x = 32; d.y = 0;  d.z = 0;  d.w = 0;  d.f = cudaChannelFormatKindFloat;    break;
+    case DXGI_FORMAT_R32G32_UINT:        d.x = 32; d.y = 32; d.z = 0;  d.w = 0;  d.f = cudaChannelFormatKindUnsigned; break;
+    case DXGI_FORMAT_R32G32_SINT:        d.x = 32; d.y = 32; d.z = 0;  d.w = 0;  d.f = cudaChannelFormatKindSigned;   break;
+    case DXGI_FORMAT_R32G32_FLOAT:       d.x = 32; d.y = 32; d.z = 0;  d.w = 0;  d.f = cudaChannelFormatKindFloat;    break;
+    case DXGI_FORMAT_R32G32B32A32_UINT:  d.x = 32; d.y = 32; d.z = 32; d.w = 32; d.f = cudaChannelFormatKindUnsigned; break;
+    case DXGI_FORMAT_R32G32B32A32_SINT:  d.x = 32; d.y = 32; d.z = 32; d.w = 32; d.f = cudaChannelFormatKindSigned;   break;
+    case DXGI_FORMAT_R32G32B32A32_FLOAT: d.x = 32; d.y = 32; d.z = 32; d.w = 32; d.f = cudaChannelFormatKindFloat;    break;
+    default: assert(0);
+
+
+
+
    }
+
+    return d;
+}
+
+cudaExtent getCudaExtentForD3D12Extent(UINT64 width, UINT height, UINT16 depthOrArraySize, D3D12_SRV_DIMENSION d3d12SRVDimension) {
+    cudaExtent e = { 0, 0, 0 };
+
+    switch (d3d12SRVDimension) {
+    case D3D12_SRV_DIMENSION_TEXTURE1D:        e.width = width; e.height = 0;      e.depth = 0;                break;
+    case D3D12_SRV_DIMENSION_TEXTURE2D:        e.width = width; e.height = height; e.depth = 0;                break;
+    case D3D12_SRV_DIMENSION_TEXTURE3D:        e.width = width; e.height = height; e.depth = depthOrArraySize; break;
+    case D3D12_SRV_DIMENSION_TEXTURECUBE:      e.width = width; e.height = height; e.depth = depthOrArraySize; break;
+    case D3D12_SRV_DIMENSION_TEXTURE1DARRAY:   e.width = width; e.height = 0;      e.depth = depthOrArraySize; break;
+    case D3D12_SRV_DIMENSION_TEXTURE2DARRAY:   e.width = width; e.height = height; e.depth = depthOrArraySize; break;
+    case D3D12_SRV_DIMENSION_TEXTURECUBEARRAY: e.width = width; e.height = height; e.depth = depthOrArraySize; break;
+    default: assert(0);
+    }
+
+    return e;
+}
+
+unsigned int getCudaMipmappedArrayFlagsForD3D12Resource(D3D12_SRV_DIMENSION d3d12SRVDimension, D3D12_RESOURCE_FLAGS d3d12ResourceFlags, bool allowSurfaceLoadStore) {
+    unsigned int flags = 0;
+
+    switch (d3d12SRVDimension) {
+    case D3D12_SRV_DIMENSION_TEXTURECUBE:      flags |= cudaArrayCubemap;                    break;
+    case D3D12_SRV_DIMENSION_TEXTURECUBEARRAY: flags |= cudaArrayCubemap | cudaArrayLayered; break;
+    case D3D12_SRV_DIMENSION_TEXTURE1DARRAY:   flags |= cudaArrayLayered;                    break;
+    case D3D12_SRV_DIMENSION_TEXTURE2DARRAY:   flags |= cudaArrayLayered;                    break;
+    default: break;
+    }
+
+    if (d3d12ResourceFlags & D3D12_RESOURCE_FLAG_ALLOW_RENDER_TARGET) {
+        flags |= cudaArrayColorAttachment;
+    }
+    if (allowSurfaceLoadStore) {
+        flags |= cudaArraySurfaceLoadStore;
+    }
+
+    return flags;
+}
+
+
+
+
+
+6.2.16.3.5. Importing Synchronization Objects +
+

A shareable Direct3D 12 fence object, created by setting the flag D3D12_FENCE_FLAG_SHARED in the call to ID3D12Device::CreateFence, can be imported into CUDA using the NT handle associated with that object as shown below. Note that it is the application’s responsibility to close the handle when it is not required anymore. The NT handle holds a reference to the resource, so it must be explicitly freed before the underlying semaphore can be freed.

+
+
cudaExternalSemaphore_t importD3D12FenceFromNTHandle(HANDLE handle) {
+    cudaExternalSemaphore_t extSem = NULL;
+    cudaExternalSemaphoreHandleDesc desc = {};
+
+    memset(&desc, 0, sizeof(desc));
+
+    desc.type = cudaExternalSemaphoreHandleTypeD3D12Fence;
+    desc.handle.win32.handle = handle;
+
+    cudaImportExternalSemaphore(&extSem, &desc);
+
+    // Input parameter 'handle' should be closed if it's not needed anymore
+    CloseHandle(handle);
+
+    return extSem;
+}
+
+
+

A shareable Direct3D 12 fence object can also be imported using a named handle if one exists as shown below.

+
+
cudaExternalSemaphore_t importD3D12FenceFromNamedNTHandle(LPCWSTR name) {
+    cudaExternalSemaphore_t extSem = NULL;
+    cudaExternalSemaphoreHandleDesc desc = {};
+
+    memset(&desc, 0, sizeof(desc));
+
+    desc.type = cudaExternalSemaphoreHandleTypeD3D12Fence;
+    desc.handle.win32.name = (void *)name;
+
+    cudaImportExternalSemaphore(&extSem, &desc);
+
+    return extSem;
+}
+
+
+
+
+
+6.2.16.3.6. Signaling/Waiting on Imported Synchronization Objects +
+

An imported Direct3D 12 fence object can be signaled as shown below. Signaling such a fence object sets its value to the one specified. The corresponding wait that waits on this signal must be issued in Direct3D 12. Additionally, the wait that waits on this signal must be issued after this signal has been issued.

+
+
void signalExternalSemaphore(cudaExternalSemaphore_t extSem, unsigned long long value, cudaStream_t stream) {
+    cudaExternalSemaphoreSignalParams params = {};
+
+    memset(&params, 0, sizeof(params));
+
+    params.params.fence.value = value;
+
+    cudaSignalExternalSemaphoresAsync(&extSem, &params, 1, stream);
+}
+
+
+

An imported Direct3D 12 fence object can be waited on as shown below. Waiting on such a fence object waits until its value becomes greater than or equal to the specified value. The corresponding signal that this wait is waiting on must be issued in Direct3D 12. Additionally, the signal must be issued before this wait can be issued.

+
+
void waitExternalSemaphore(cudaExternalSemaphore_t extSem, unsigned long long value, cudaStream_t stream) {
+    cudaExternalSemaphoreWaitParams params = {};
+
+    memset(&params, 0, sizeof(params));
+
+    params.params.fence.value = value;
+
+    cudaWaitExternalSemaphoresAsync(&extSem, &params, 1, stream);
+}
+
+
+
+
+
+

+6.2.16.4. Direct3D 11 Interoperability +

+
+
+6.2.16.4.1. Matching Device LUIDs +
+

When importing memory and synchronization objects exported by Direct3D 11, they must be imported and mapped on the same device as they were created on. The CUDA device that corresponds to the Direct3D 11 device on which the objects were created can be determined by comparing the LUID of a CUDA device with that of the Direct3D 11 device, as shown in the following code sample.

+
+
int getCudaDeviceForD3D11Device(ID3D11Device *d3d11Device) {
+    IDXGIDevice *dxgiDevice;
+    d3d11Device->QueryInterface(__uuidof(IDXGIDevice), (void **)&dxgiDevice);
+
+    IDXGIAdapter *dxgiAdapter;
+    dxgiDevice->GetAdapter(&dxgiAdapter);
+
+    DXGI_ADAPTER_DESC dxgiAdapterDesc;
+    dxgiAdapter->GetDesc(&dxgiAdapterDesc);
+
+    LUID d3d11Luid = dxgiAdapterDesc.AdapterLuid;
+
+    int cudaDeviceCount;
+    cudaGetDeviceCount(&cudaDeviceCount);
+
+    for (int cudaDevice = 0; cudaDevice < cudaDeviceCount; cudaDevice++) {
+        cudaDeviceProp deviceProp;
+        cudaGetDeviceProperties(&deviceProp, cudaDevice);
+        char *cudaLuid = deviceProp.luid;
+
+        if (!memcmp(&d3d11Luid.LowPart, cudaLuid, sizeof(d3d11Luid.LowPart)) &&
+            !memcmp(&d3d11Luid.HighPart, cudaLuid + sizeof(d3d11Luid.LowPart), sizeof(d3d11Luid.HighPart))) {
+            return cudaDevice;
+        }
+    }
+    return cudaInvalidDeviceId;
+}
+
+
+
+
+
+6.2.16.4.2. Importing Memory Objects +
+

A shareable Direct3D 11 texture resource, viz, ID3D11Texture1D, ID3D11Texture2D or ID3D11Texture3D, can be created by setting either the D3D11_RESOURCE_MISC_SHARED or D3D11_RESOURCE_MISC_SHARED_KEYEDMUTEX (on Windows 7) or D3D11_RESOURCE_MISC_SHARED_NTHANDLE (on Windows 10) when calling ID3D11Device:CreateTexture1D, ID3D11Device:CreateTexture2D or ID3D11Device:CreateTexture3D respectively. A shareable Direct3D 11 buffer resource, ID3D11Buffer, can be created by specifying either of the above flags when calling ID3D11Device::CreateBuffer. A shareable resource created by specifying the D3D11_RESOURCE_MISC_SHARED_NTHANDLE can be imported into CUDA using the NT handle associated with that object as shown below. Note that it is the application’s responsibility to close the NT handle when it is not required anymore. The NT handle holds a reference to the resource, so it must be explicitly freed before the underlying memory can be freed. When importing a Direct3D 11 resource, the flag cudaExternalMemoryDedicated must be set.

+
+
cudaExternalMemory_t importD3D11ResourceFromNTHandle(HANDLE handle, unsigned long long size) {
+    cudaExternalMemory_t extMem = NULL;
+    cudaExternalMemoryHandleDesc desc = {};
+
+    memset(&desc, 0, sizeof(desc));
+
+    desc.type = cudaExternalMemoryHandleTypeD3D11Resource;
+    desc.handle.win32.handle = (void *)handle;
+    desc.size = size;
+    desc.flags |= cudaExternalMemoryDedicated;
+
+    cudaImportExternalMemory(&extMem, &desc);
+
+    // Input parameter 'handle' should be closed if it's not needed anymore
+    CloseHandle(handle);
+
+    return extMem;
+}
+
+
+

A shareable Direct3D 11 resource can also be imported using a named handle if one exists as shown below.

+
+
cudaExternalMemory_t importD3D11ResourceFromNamedNTHandle(LPCWSTR name, unsigned long long size) {
+    cudaExternalMemory_t extMem = NULL;
+    cudaExternalMemoryHandleDesc desc = {};
+
+    memset(&desc, 0, sizeof(desc));
+
+    desc.type = cudaExternalMemoryHandleTypeD3D11Resource;
+    desc.handle.win32.name = (void *)name;
+    desc.size = size;
+    desc.flags |= cudaExternalMemoryDedicated;
+
+    cudaImportExternalMemory(&extMem, &desc);
+
+    return extMem;
+}
+
+
+

A shareable Direct3D 11 resource, created by specifying the D3D11_RESOURCE_MISC_SHARED or D3D11_RESOURCE_MISC_SHARED_KEYEDMUTEX, can be imported into CUDA using the globally shared D3DKMT handle associated with that object as shown below. Since a globally shared D3DKMT handle does not hold a reference to the underlying memory it is automatically destroyed when all other references to the resource are destroyed.

+
+
cudaExternalMemory_t importD3D11ResourceFromKMTHandle(HANDLE handle, unsigned long long size) {
+    cudaExternalMemory_t extMem = NULL;
+    cudaExternalMemoryHandleDesc desc = {};
+
+    memset(&desc, 0, sizeof(desc));
+
+    desc.type = cudaExternalMemoryHandleTypeD3D11ResourceKmt;
+    desc.handle.win32.handle = (void *)handle;
+    desc.size = size;
+    desc.flags |= cudaExternalMemoryDedicated;
+
+    cudaImportExternalMemory(&extMem, &desc);
+
+    return extMem;
+}
+
+
+
+
+
+6.2.16.4.3. Mapping Buffers onto Imported Memory Objects +
+

A device pointer can be mapped onto an imported memory object as shown below. The offset and size of the mapping must match that specified when creating the mapping using the corresponding Direct3D 11 API. All mapped device pointers must be freed using cudaFree().

+
+
void * mapBufferOntoExternalMemory(cudaExternalMemory_t extMem, unsigned long long offset, unsigned long long size) {
+    void *ptr = NULL;
+    cudaExternalMemoryBufferDesc desc = {};
+
+    memset(&desc, 0, sizeof(desc));
+
+    desc.offset = offset;
+    desc.size = size;
+
+    cudaExternalMemoryGetMappedBuffer(&ptr, extMem, &desc);
+
+    // Note: ‘ptr’ must eventually be freed using cudaFree()
+    return ptr;
+}
+
+
+
+
+
+6.2.16.4.4. Mapping Mipmapped Arrays onto Imported Memory Objects +
+

A CUDA mipmapped array can be mapped onto an imported memory object as shown below. The offset, dimensions, format and number of mip levels must match that specified when creating the mapping using the corresponding Direct3D 11 API. Additionally, if the mipmapped array can be bound as a render target in Direct3D 12, the flag cudaArrayColorAttachment must be set. All mapped mipmapped arrays must be freed using cudaFreeMipmappedArray(). The following code sample shows how to convert Direct3D 11 parameters into the corresponding CUDA parameters when mapping mipmapped arrays onto imported memory objects.

+
+
cudaMipmappedArray_t mapMipmappedArrayOntoExternalMemory(cudaExternalMemory_t extMem, unsigned long long offset, cudaChannelFormatDesc *formatDesc, cudaExtent *extent, unsigned int flags, unsigned int numLevels) {
+    cudaMipmappedArray_t mipmap = NULL;
+    cudaExternalMemoryMipmappedArrayDesc desc = {};
+
+    memset(&desc, 0, sizeof(desc));
+
+    desc.offset = offset;
+    desc.formatDesc = *formatDesc;
+    desc.extent = *extent;
+    desc.flags = flags;
+    desc.numLevels = numLevels;
+
+    // Note: 'mipmap' must eventually be freed using cudaFreeMipmappedArray()
+    cudaExternalMemoryGetMappedMipmappedArray(&mipmap, extMem, &desc);
+
+    return mipmap;
+}
+
+cudaChannelFormatDesc getCudaChannelFormatDescForDxgiFormat(DXGI_FORMAT dxgiFormat)
+{
+    cudaChannelFormatDesc d;
+    memset(&d, 0, sizeof(d));
+    switch (dxgiFormat) {
+    case DXGI_FORMAT_R8_UINT:            d.x = 8;  d.y = 0;  d.z = 0;  d.w = 0;  d.f = cudaChannelFormatKindUnsigned; break;
+    case DXGI_FORMAT_R8_SINT:            d.x = 8;  d.y = 0;  d.z = 0;  d.w = 0;  d.f = cudaChannelFormatKindSigned;   break;
+    case DXGI_FORMAT_R8G8_UINT:          d.x = 8;  d.y = 8;  d.z = 0;  d.w = 0;  d.f = cudaChannelFormatKindUnsigned; break;
+    case DXGI_FORMAT_R8G8_SINT:          d.x = 8;  d.y = 8;  d.z = 0;  d.w = 0;  d.f = cudaChannelFormatKindSigned;   break;
+    case DXGI_FORMAT_R8G8B8A8_UINT:      d.x = 8;  d.y = 8;  d.z = 8;  d.w = 8;  d.f = cudaChannelFormatKindUnsigned; break;
+    case DXGI_FORMAT_R8G8B8A8_SINT:      d.x = 8;  d.y = 8;  d.z = 8;  d.w = 8;  d.f = cudaChannelFormatKindSigned;   break;
+    case DXGI_FORMAT_R16_UINT:           d.x = 16; d.y = 0;  d.z = 0;  d.w = 0;  d.f = cudaChannelFormatKindUnsigned; break;
+    case DXGI_FORMAT_R16_SINT:           d.x = 16; d.y = 0;  d.z = 0;  d.w = 0;  d.f = cudaChannelFormatKindSigned;   break;
+    case DXGI_FORMAT_R16G16_UINT:        d.x = 16; d.y = 16; d.z = 0;  d.w = 0;  d.f = cudaChannelFormatKindUnsigned; break;
+    case DXGI_FORMAT_R16G16_SINT:        d.x = 16; d.y = 16; d.z = 0;  d.w = 0;  d.f = cudaChannelFormatKindSigned;   break;
+    case DXGI_FORMAT_R16G16B16A16_UINT:  d.x = 16; d.y = 16; d.z = 16; d.w = 16; d.f = cudaChannelFormatKindUnsigned; break;
+    case DXGI_FORMAT_R16G16B16A16_SINT:  d.x = 16; d.y = 16; d.z = 16; d.w = 16; d.f = cudaChannelFormatKindSigned;   break;
+    case DXGI_FORMAT_R32_UINT:           d.x = 32; d.y = 0;  d.z = 0;  d.w = 0;  d.f = cudaChannelFormatKindUnsigned; break;
+    case DXGI_FORMAT_R32_SINT:           d.x = 32; d.y = 0;  d.z = 0;  d.w = 0;  d.f = cudaChannelFormatKindSigned;   break;
+    case DXGI_FORMAT_R32_FLOAT:          d.x = 32; d.y = 0;  d.z = 0;  d.w = 0;  d.f = cudaChannelFormatKindFloat;    break;
+    case DXGI_FORMAT_R32G32_UINT:        d.x = 32; d.y = 32; d.z = 0;  d.w = 0;  d.f = cudaChannelFormatKindUnsigned; break;
+    case DXGI_FORMAT_R32G32_SINT:        d.x = 32; d.y = 32; d.z = 0;  d.w = 0;  d.f = cudaChannelFormatKindSigned;   break;
+    case DXGI_FORMAT_R32G32_FLOAT:       d.x = 32; d.y = 32; d.z = 0;  d.w = 0;  d.f = cudaChannelFormatKindFloat;    break;
+    case DXGI_FORMAT_R32G32B32A32_UINT:  d.x = 32; d.y = 32; d.z = 32; d.w = 32; d.f = cudaChannelFormatKindUnsigned; break;
+    case DXGI_FORMAT_R32G32B32A32_SINT:  d.x = 32; d.y = 32; d.z = 32; d.w = 32; d.f = cudaChannelFormatKindSigned;   break;
+    case DXGI_FORMAT_R32G32B32A32_FLOAT: d.x = 32; d.y = 32; d.z = 32; d.w = 32; d.f = cudaChannelFormatKindFloat;    break;
+    default: assert(0);
+    }
+
+
+
+
    return d;
+}
+
+cudaExtent getCudaExtentForD3D11Extent(UINT64 width, UINT height, UINT16 depthOrArraySize, D3D12_SRV_DIMENSION d3d11SRVDimension) {
+    cudaExtent e = { 0, 0, 0 };
+
+    switch (d3d11SRVDimension) {
+    case D3D11_SRV_DIMENSION_TEXTURE1D:        e.width = width; e.height = 0;      e.depth = 0;                break;
+    case D3D11_SRV_DIMENSION_TEXTURE2D:        e.width = width; e.height = height; e.depth = 0;                break;
+    case D3D11_SRV_DIMENSION_TEXTURE3D:        e.width = width; e.height = height; e.depth = depthOrArraySize; break;
+    case D3D11_SRV_DIMENSION_TEXTURECUBE:      e.width = width; e.height = height; e.depth = depthOrArraySize; break;
+    case D3D11_SRV_DIMENSION_TEXTURE1DARRAY:   e.width = width; e.height = 0;      e.depth = depthOrArraySize; break;
+    case D3D11_SRV_DIMENSION_TEXTURE2DARRAY:   e.width = width; e.height = height; e.depth = depthOrArraySize; break;
+    case D3D11_SRV_DIMENSION_TEXTURECUBEARRAY: e.width = width; e.height = height; e.depth = depthOrArraySize; break;
+    default: assert(0);
+    }
+    return e;
+}
+
+unsigned int getCudaMipmappedArrayFlagsForD3D12Resource(D3D11_SRV_DIMENSION d3d11SRVDimension, D3D11_BIND_FLAG d3d11BindFlags, bool allowSurfaceLoadStore) {
+    unsigned int flags = 0;
+
+    switch (d3d11SRVDimension) {
+    case D3D11_SRV_DIMENSION_TEXTURECUBE:      flags |= cudaArrayCubemap;                    break;
+    case D3D11_SRV_DIMENSION_TEXTURECUBEARRAY: flags |= cudaArrayCubemap | cudaArrayLayered; break;
+    case D3D11_SRV_DIMENSION_TEXTURE1DARRAY:   flags |= cudaArrayLayered;                    break;
+    case D3D11_SRV_DIMENSION_TEXTURE2DARRAY:   flags |= cudaArrayLayered;                    break;
+    default: break;
+    }
+
+    if (d3d11BindFlags & D3D11_BIND_RENDER_TARGET) {
+        flags |= cudaArrayColorAttachment;
+    }
+
+    if (allowSurfaceLoadStore) {
+        flags |= cudaArraySurfaceLoadStore;
+    }
+
+    return flags;
+}
+
+
+
+
+
+6.2.16.4.5. Importing Synchronization Objects +
+

A shareable Direct3D 11 fence object, created by setting the flag D3D11_FENCE_FLAG_SHARED in the call to ID3D11Device5::CreateFence, can be imported into CUDA using the NT handle associated with that object as shown below. Note that it is the application’s responsibility to close the handle when it is not required anymore. The NT handle holds a reference to the resource, so it must be explicitly freed before the underlying semaphore can be freed.

+
+
cudaExternalSemaphore_t importD3D11FenceFromNTHandle(HANDLE handle) {
+    cudaExternalSemaphore_t extSem = NULL;
+    cudaExternalSemaphoreHandleDesc desc = {};
+
+    memset(&desc, 0, sizeof(desc));
+
+    desc.type = cudaExternalSemaphoreHandleTypeD3D11Fence;
+    desc.handle.win32.handle = handle;
+
+    cudaImportExternalSemaphore(&extSem, &desc);
+
+    // Input parameter 'handle' should be closed if it's not needed anymore
+    CloseHandle(handle);
+
+    return extSem;
+}
+
+
+

A shareable Direct3D 11 fence object can also be imported using a named handle if one exists as shown below.

+
+
cudaExternalSemaphore_t importD3D11FenceFromNamedNTHandle(LPCWSTR name) {
+    cudaExternalSemaphore_t extSem = NULL;
+    cudaExternalSemaphoreHandleDesc desc = {};
+
+    memset(&desc, 0, sizeof(desc));
+
+    desc.type = cudaExternalSemaphoreHandleTypeD3D11Fence;
+    desc.handle.win32.name = (void *)name;
+
+    cudaImportExternalSemaphore(&extSem, &desc);
+
+    return extSem;
+}
+
+
+

A shareable Direct3D 11 keyed mutex object associated with a shareable Direct3D 11 resource, viz, IDXGIKeyedMutex, created by setting the flag D3D11_RESOURCE_MISC_SHARED_KEYEDMUTEX, can be imported into CUDA using the NT handle associated with that object as shown below. Note that it is the application’s responsibility to close the handle when it is not required anymore. The NT handle holds a reference to the resource, so it must be explicitly freed before the underlying semaphore can be freed.

+
+
cudaExternalSemaphore_t importD3D11KeyedMutexFromNTHandle(HANDLE handle) {
+    cudaExternalSemaphore_t extSem = NULL;
+    cudaExternalSemaphoreHandleDesc desc = {};
+
+    memset(&desc, 0, sizeof(desc));
+
+    desc.type = cudaExternalSemaphoreHandleTypeKeyedMutex;
+    desc.handle.win32.handle = handle;
+
+    cudaImportExternalSemaphore(&extSem, &desc);
+
+    // Input parameter 'handle' should be closed if it's not needed anymore
+    CloseHandle(handle);
+
+    return extSem;
+}
+
+
+

A shareable Direct3D 11 keyed mutex object can also be imported using a named handle if one exists as shown below.

+
+
cudaExternalSemaphore_t importD3D11KeyedMutexFromNamedNTHandle(LPCWSTR name) {
+    cudaExternalSemaphore_t extSem = NULL;
+    cudaExternalSemaphoreHandleDesc desc = {};
+
+    memset(&desc, 0, sizeof(desc));
+
+    desc.type = cudaExternalSemaphoreHandleTypeKeyedMutex;
+    desc.handle.win32.name = (void *)name;
+
+    cudaImportExternalSemaphore(&extSem, &desc);
+
+    return extSem;
+}
+
+
+

A shareable Direct3D 11 keyed mutex object can be imported into CUDA using the globally shared D3DKMT handle associated with that object as shown below. Since a globally shared D3DKMT handle does not hold a reference to the underlying memory it is automatically destroyed when all other references to the resource are destroyed.

+
+
cudaExternalSemaphore_t importD3D11FenceFromKMTHandle(HANDLE handle) {
+    cudaExternalSemaphore_t extSem = NULL;
+    cudaExternalSemaphoreHandleDesc desc = {};
+
+    memset(&desc, 0, sizeof(desc));
+
+    desc.type = cudaExternalSemaphoreHandleTypeKeyedMutexKmt;
+    desc.handle.win32.handle = handle;
+
+    cudaImportExternalSemaphore(&extSem, &desc);
+
+    // Input parameter 'handle' should be closed if it's not needed anymore
+    CloseHandle(handle);
+
+    return extSem;
+}
+
+
+
+
+
+6.2.16.4.6. Signaling/Waiting on Imported Synchronization Objects +
+

An imported Direct3D 11 fence object can be signaled as shown below. Signaling such a fence object sets its value to the one specified. The corresponding wait that waits on this signal must be issued in Direct3D 11. Additionally, the wait that waits on this signal must be issued after this signal has been issued.

+
+
void signalExternalSemaphore(cudaExternalSemaphore_t extSem, unsigned long long value, cudaStream_t stream) {
+    cudaExternalSemaphoreSignalParams params = {};
+
+    memset(&params, 0, sizeof(params));
+
+    params.params.fence.value = value;
+
+    cudaSignalExternalSemaphoresAsync(&extSem, &params, 1, stream);
+}
+
+
+

An imported Direct3D 11 fence object can be waited on as shown below. Waiting on such a fence object waits until its value becomes greater than or equal to the specified value. The corresponding signal that this wait is waiting on must be issued in Direct3D 11. Additionally, the signal must be issued before this wait can be issued.

+
+
void waitExternalSemaphore(cudaExternalSemaphore_t extSem, unsigned long long value, cudaStream_t stream) {
+    cudaExternalSemaphoreWaitParams params = {};
+
+    memset(&params, 0, sizeof(params));
+
+    params.params.fence.value = value;
+
+    cudaWaitExternalSemaphoresAsync(&extSem, &params, 1, stream);
+}
+
+
+

An imported Direct3D 11 keyed mutex object can be signaled as shown below. Signaling such a keyed mutex object by specifying a key value releases the keyed mutex for that value. The corresponding wait that waits on this signal must be issued in Direct3D 11 with the same key value. Additionally, the Direct3D 11 wait must be issued after this signal has been issued.

+
+
void signalExternalSemaphore(cudaExternalSemaphore_t extSem, unsigned long long key, cudaStream_t stream) {
+    cudaExternalSemaphoreSignalParams params = {};
+
+    memset(&params, 0, sizeof(params));
+
+    params.params.keyedmutex.key = key;
+
+    cudaSignalExternalSemaphoresAsync(&extSem, &params, 1, stream);
+}
+
+
+

An imported Direct3D 11 keyed mutex object can be waited on as shown below. A timeout value in milliseconds is needed when waiting on such a keyed mutex. The wait operation waits until the keyed mutex value is equal to the specified key value or until the timeout has elapsed. The timeout interval can also be an infinite value. In case an infinite value is specified the timeout never elapses. The windows INFINITE macro must be used to specify an infinite timeout. The corresponding signal that this wait is waiting on must be issued in Direct3D 11. Additionally, the Direct3D 11 signal must be issued before this wait can be issued.

+
+
void waitExternalSemaphore(cudaExternalSemaphore_t extSem, unsigned long long key, unsigned int timeoutMs, cudaStream_t stream) {
+    cudaExternalSemaphoreWaitParams params = {};
+
+    memset(&params, 0, sizeof(params));
+
+    params.params.keyedmutex.key = key;
+    params.params.keyedmutex.timeoutMs = timeoutMs;
+
+    cudaWaitExternalSemaphoresAsync(&extSem, &params, 1, stream);
+}
+
+
+
+
+
+

+6.2.16.5. NVIDIA Software Communication Interface Interoperability (NVSCI) +

+

NvSciBuf and NvSciSync are interfaces developed for serving the following purposes:

+
    +
  • NvSciBuf: Allows applications to allocate and exchange buffers in memory

  • +
  • NvSciSync: Allows applications to manage synchronization objects at operation boundaries

  • +
+

More details on these interfaces are available at: https://docs.nvidia.com/drive.

+
+
+6.2.16.5.1. Importing Memory Objects +
+

For allocating an NvSciBuf object compatible with a given CUDA device, the corresponding GPU id must be set with NvSciBufGeneralAttrKey_GpuId in the NvSciBuf attribute list as shown below. Optionally, applications can specify the following attributes -

+
    +
  • NvSciBufGeneralAttrKey_NeedCpuAccess: Specifies if CPU access is required for the buffer

  • +
  • NvSciBufRawBufferAttrKey_Align: Specifies the alignment requirement of NvSciBufType_RawBuffer

  • +
  • NvSciBufGeneralAttrKey_RequiredPerm: Different access permissions can be configured for different UMDs per NvSciBuf memory object instance. For example, to provide the GPU with read-only access permissions to the buffer, create a duplicate NvSciBuf object using NvSciBufObjDupWithReducePerm() with NvSciBufAccessPerm_Readonly as the input parameter. Then import this newly created duplicate object with reduced permission into CUDA as shown

  • +
  • NvSciBufGeneralAttrKey_EnableGpuCache: To control GPU L2 cacheability

  • +
  • NvSciBufGeneralAttrKey_EnableGpuCompression: To specify GPU compression

  • +
+
+

Note

+

For more details on these attributes and their valid input options, refer to NvSciBuf Documentation.

+
+

The following code snippet illustrates their sample usage.

+
+
NvSciBufObj createNvSciBufObject() {
+   // Raw Buffer Attributes for CUDA
+    NvSciBufType bufType = NvSciBufType_RawBuffer;
+    uint64_t rawsize = SIZE;
+    uint64_t align = 0;
+    bool cpuaccess_flag = true;
+    NvSciBufAttrValAccessPerm perm = NvSciBufAccessPerm_ReadWrite;
+
+    NvSciRmGpuId gpuid[] ={};
+    CUuuid uuid;
+    cuDeviceGetUuid(&uuid, dev));
+
+    memcpy(&gpuid[0].bytes, &uuid.bytes, sizeof(uuid.bytes));
+    // Disable cache on dev
+    NvSciBufAttrValGpuCache gpuCache[] = {{gpuid[0], false}};
+    NvSciBufAttrValGpuCompression gpuCompression[] = {{gpuid[0], NvSciBufCompressionType_GenericCompressible}};
+    // Fill in values
+    NvSciBufAttrKeyValuePair rawbuffattrs[] = {
+         { NvSciBufGeneralAttrKey_Types, &bufType, sizeof(bufType) },
+         { NvSciBufRawBufferAttrKey_Size, &rawsize, sizeof(rawsize) },
+         { NvSciBufRawBufferAttrKey_Align, &align, sizeof(align) },
+         { NvSciBufGeneralAttrKey_NeedCpuAccess, &cpuaccess_flag, sizeof(cpuaccess_flag) },
+         { NvSciBufGeneralAttrKey_RequiredPerm, &perm, sizeof(perm) },
+         { NvSciBufGeneralAttrKey_GpuId, &gpuid, sizeof(gpuid) },
+         { NvSciBufGeneralAttrKey_EnableGpuCache &gpuCache, sizeof(gpuCache) },
+         { NvSciBufGeneralAttrKey_EnableGpuCompression &gpuCompression, sizeof(gpuCompression) }
+    };
+
+    // Create list by setting attributes
+    err = NvSciBufAttrListSetAttrs(attrListBuffer, rawbuffattrs,
+            sizeof(rawbuffattrs)/sizeof(NvSciBufAttrKeyValuePair));
+
+    NvSciBufAttrListCreate(NvSciBufModule, &attrListBuffer);
+
+    // Reconcile And Allocate
+    NvSciBufAttrListReconcile(&attrListBuffer, 1, &attrListReconciledBuffer,
+                       &attrListConflictBuffer)
+    NvSciBufObjAlloc(attrListReconciledBuffer, &bufferObjRaw);
+    return bufferObjRaw;
+}
+
+
+
+
NvSciBufObj bufferObjRo; // Readonly NvSciBuf memory obj
+// Create a duplicate handle to the same memory buffer with reduced permissions
+NvSciBufObjDupWithReducePerm(bufferObjRaw, NvSciBufAccessPerm_Readonly, &bufferObjRo);
+return bufferObjRo;
+
+
+

The allocated NvSciBuf memory object can be imported in CUDA using the NvSciBufObj handle as shown below. Application should query the allocated NvSciBufObj for attributes required for filling CUDA External Memory Descriptor. Note that the attribute list and NvSciBuf objects should be maintained by the application. If the NvSciBuf object imported into CUDA is also mapped by other drivers, then based on NvSciBufGeneralAttrKey_GpuSwNeedCacheCoherency output attribute value the application must use NvSciSync objects (refer to Importing Synchronization Objects) as appropriate barriers to maintain coherence between CUDA and the other drivers.

+
+

Note

+

For more details on how to allocate and maintain NvSciBuf objects refer to NvSciBuf API Documentation.

+
+
+
cudaExternalMemory_t importNvSciBufObject (NvSciBufObj bufferObjRaw) {
+
+    /*************** Query NvSciBuf Object **************/
+    NvSciBufAttrKeyValuePair bufattrs[] = {
+                { NvSciBufRawBufferAttrKey_Size, NULL, 0 },
+                { NvSciBufGeneralAttrKey_GpuSwNeedCacheCoherency, NULL, 0 },
+                { NvSciBufGeneralAttrKey_EnableGpuCompression, NULL, 0 }
+    };
+    NvSciBufAttrListGetAttrs(retList, bufattrs,
+        sizeof(bufattrs)/sizeof(NvSciBufAttrKeyValuePair)));
+                ret_size = *(static_cast<const uint64_t*>(bufattrs[0].value));
+
+    // Note cache and compression are per GPU attributes, so read values for specific gpu by comparing UUID
+    // Read cacheability granted by NvSciBuf
+    int numGpus = bufattrs[1].len / sizeof(NvSciBufAttrValGpuCache);
+    NvSciBufAttrValGpuCache[] cacheVal = (NvSciBufAttrValGpuCache *)bufattrs[1].value;
+    bool ret_cacheVal;
+    for (int i = 0; i < numGpus; i++) {
+        if (memcmp(gpuid[0].bytes, cacheVal[i].gpuId.bytes, sizeof(CUuuid)) == 0) {
+            ret_cacheVal = cacheVal[i].cacheability);
+        }
+    }
+
+    // Read compression granted by NvSciBuf
+    numGpus = bufattrs[2].len / sizeof(NvSciBufAttrValGpuCompression);
+    NvSciBufAttrValGpuCompression[] compVal = (NvSciBufAttrValGpuCompression *)bufattrs[2].value;
+    NvSciBufCompressionType ret_compVal;
+    for (int i = 0; i < numGpus; i++) {
+        if (memcmp(gpuid[0].bytes, compVal[i].gpuId.bytes, sizeof(CUuuid)) == 0) {
+            ret_compVal = compVal[i].compressionType);
+        }
+    }
+
+    /*************** NvSciBuf Registration With CUDA **************/
+
+    // Fill up CUDA_EXTERNAL_MEMORY_HANDLE_DESC
+    cudaExternalMemoryHandleDesc memHandleDesc;
+    memset(&memHandleDesc, 0, sizeof(memHandleDesc));
+    memHandleDesc.type = cudaExternalMemoryHandleTypeNvSciBuf;
+    memHandleDesc.handle.nvSciBufObject = bufferObjRaw;
+    // Set the NvSciBuf object with required access permissions in this step
+    memHandleDesc.handle.nvSciBufObject = bufferObjRo;
+    memHandleDesc.size = ret_size;
+    cudaImportExternalMemory(&extMemBuffer, &memHandleDesc);
+    return extMemBuffer;
+ }
+
+
+
+
+
+6.2.16.5.2. Mapping Buffers onto Imported Memory Objects +
+

A device pointer can be mapped onto an imported memory object as shown below. The offset and size of the mapping can be filled as per the attributes of the allocated NvSciBufObj. All mapped device pointers must be freed using cudaFree().

+
+
void * mapBufferOntoExternalMemory(cudaExternalMemory_t extMem, unsigned long long offset, unsigned long long size) {
+    void *ptr = NULL;
+    cudaExternalMemoryBufferDesc desc = {};
+
+    memset(&desc, 0, sizeof(desc));
+
+    desc.offset = offset;
+    desc.size = size;
+
+    cudaExternalMemoryGetMappedBuffer(&ptr, extMem, &desc);
+
+    // Note: 'ptr' must eventually be freed using cudaFree()
+    return ptr;
+}
+
+
+
+
+
+6.2.16.5.3. Mapping Mipmapped Arrays onto Imported Memory Objects +
+

A CUDA mipmapped array can be mapped onto an imported memory object as shown below. The offset, dimensions and format can be filled as per the attributes of the allocated NvSciBufObj. All mapped mipmapped arrays must be freed using cudaFreeMipmappedArray(). The following code sample shows how to convert NvSciBuf attributes into the corresponding CUDA parameters when mapping mipmapped arrays onto imported memory objects.

+
+

Note

+

The number of mip levels must be 1.

+
+
+
cudaMipmappedArray_t mapMipmappedArrayOntoExternalMemory(cudaExternalMemory_t extMem, unsigned long long offset, cudaChannelFormatDesc *formatDesc, cudaExtent *extent, unsigned int flags, unsigned int numLevels) {
+    cudaMipmappedArray_t mipmap = NULL;
+    cudaExternalMemoryMipmappedArrayDesc desc = {};
+
+    memset(&desc, 0, sizeof(desc));
+
+    desc.offset = offset;
+    desc.formatDesc = *formatDesc;
+    desc.extent = *extent;
+    desc.flags = flags;
+    desc.numLevels = numLevels;
+
+    // Note: 'mipmap' must eventually be freed using cudaFreeMipmappedArray()
+    cudaExternalMemoryGetMappedMipmappedArray(&mipmap, extMem, &desc);
+
+    return mipmap;
+}
+
+
+
+
+
+6.2.16.5.4. Importing Synchronization Objects +
+

NvSciSync attributes that are compatible with a given CUDA device can be generated using cudaDeviceGetNvSciSyncAttributes(). The returned attribute list can be used to create a NvSciSyncObj that is guaranteed compatibility with a given CUDA device.

+
+
NvSciSyncObj createNvSciSyncObject() {
+    NvSciSyncObj nvSciSyncObj
+    int cudaDev0 = 0;
+    int cudaDev1 = 1;
+    NvSciSyncAttrList signalerAttrList = NULL;
+    NvSciSyncAttrList waiterAttrList = NULL;
+    NvSciSyncAttrList reconciledList = NULL;
+    NvSciSyncAttrList newConflictList = NULL;
+
+    NvSciSyncAttrListCreate(module, &signalerAttrList);
+    NvSciSyncAttrListCreate(module, &waiterAttrList);
+    NvSciSyncAttrList unreconciledList[2] = {NULL, NULL};
+    unreconciledList[0] = signalerAttrList;
+    unreconciledList[1] = waiterAttrList;
+
+    cudaDeviceGetNvSciSyncAttributes(signalerAttrList, cudaDev0, CUDA_NVSCISYNC_ATTR_SIGNAL);
+    cudaDeviceGetNvSciSyncAttributes(waiterAttrList, cudaDev1, CUDA_NVSCISYNC_ATTR_WAIT);
+
+    NvSciSyncAttrListReconcile(unreconciledList, 2, &reconciledList, &newConflictList);
+
+    NvSciSyncObjAlloc(reconciledList, &nvSciSyncObj);
+
+    return nvSciSyncObj;
+}
+
+
+

An NvSciSync object (created as above) can be imported into CUDA using the NvSciSyncObj handle as shown below. Note that ownership of the NvSciSyncObj handle continues to lie with the application even after it is imported.

+
+
cudaExternalSemaphore_t importNvSciSyncObject(void* nvSciSyncObj) {
+    cudaExternalSemaphore_t extSem = NULL;
+    cudaExternalSemaphoreHandleDesc desc = {};
+
+    memset(&desc, 0, sizeof(desc));
+
+    desc.type = cudaExternalSemaphoreHandleTypeNvSciSync;
+    desc.handle.nvSciSyncObj = nvSciSyncObj;
+
+    cudaImportExternalSemaphore(&extSem, &desc);
+
+    // Deleting/Freeing the nvSciSyncObj beyond this point will lead to undefined behavior in CUDA
+
+    return extSem;
+}
+
+
+
+
+
+6.2.16.5.5. Signaling/Waiting on Imported Synchronization Objects +
+

An imported NvSciSyncObj object can be signaled as outlined below. Signaling NvSciSync backed semaphore object initializes the fence parameter passed as input. This fence parameter is waited upon by a wait operation that corresponds to the aforementioned signal. Additionally, the wait that waits on this signal must be issued after this signal has been issued. If the flags are set to cudaExternalSemaphoreSignalSkipNvSciBufMemSync then memory synchronization operations (over all the imported NvSciBuf in this process) that are executed as a part of the signal operation by default are skipped. When NvsciBufGeneralAttrKey_GpuSwNeedCacheCoherency is FALSE, this flag should be set.

+
+
void signalExternalSemaphore(cudaExternalSemaphore_t extSem, cudaStream_t stream, void *fence) {
+    cudaExternalSemaphoreSignalParams signalParams = {};
+
+    memset(&signalParams, 0, sizeof(signalParams));
+
+    signalParams.params.nvSciSync.fence = (void*)fence;
+    signalParams.flags = 0; //OR cudaExternalSemaphoreSignalSkipNvSciBufMemSync
+
+    cudaSignalExternalSemaphoresAsync(&extSem, &signalParams, 1, stream);
+
+}
+
+
+

An imported NvSciSyncObj object can be waited upon as outlined below. Waiting on NvSciSync backed semaphore object waits until the input fence parameter is signaled by the corresponding signaler. Additionally, the signal must be issued before the wait can be issued. If the flags are set to cudaExternalSemaphoreWaitSkipNvSciBufMemSync then memory synchronization operations (over all the imported NvSciBuf in this process) that are executed as a part of the signal operation by default are skipped. When NvsciBufGeneralAttrKey_GpuSwNeedCacheCoherency is FALSE, this flag should be set.

+
+
void waitExternalSemaphore(cudaExternalSemaphore_t extSem, cudaStream_t stream, void *fence) {
+     cudaExternalSemaphoreWaitParams waitParams = {};
+
+    memset(&waitParams, 0, sizeof(waitParams));
+
+    waitParams.params.nvSciSync.fence = (void*)fence;
+    waitParams.flags = 0; //OR cudaExternalSemaphoreWaitSkipNvSciBufMemSync
+
+    cudaWaitExternalSemaphoresAsync(&extSem, &waitParams, 1, stream);
+}
+
+
+
+
+
+
+
+

+6.3. Versioning and Compatibility +

+

There are two version numbers that developers should care about when developing a CUDA application: The compute capability that describes the general specifications and features of the compute device (see Compute Capability) and the version of the CUDA driver API that describes the features supported by the driver API and runtime.

+

The version of the driver API is defined in the driver header file as CUDA_VERSION. It allows developers to check whether their application requires a newer device driver than the one currently installed. This is important, because the driver API is backward compatible, meaning that applications, plug-ins, and libraries (including the CUDA runtime) compiled against a particular version of the driver API will continue to work on subsequent device driver releases as illustrated in Figure 26. The driver API is not forward compatible, which means that applications, plug-ins, and libraries (including the CUDA runtime) compiled against a particular version of the driver API will not work on previous versions of the device driver.

+

It is important to note that there are limitations on the mixing and matching of versions that is supported:

+
    +
  • Since only one version of the CUDA Driver can be installed at a time on a system, the installed driver must be of the same or higher version than the maximum Driver API version against which any application, plug-ins, or libraries that must run on that system were built.

  • +
  • All plug-ins and libraries used by an application must use the same version of the CUDA Runtime unless they statically link to the Runtime, in which case multiple versions of the runtime can coexist in the same process space. Note that if nvcc is used to link the application, the static version of the CUDA Runtime library will be used by default, and all CUDA Toolkit libraries are statically linked against the CUDA Runtime.

  • +
  • All plug-ins and libraries used by an application must use the same version of any libraries that use the runtime (such as cuFFT, cuBLAS, …) unless statically linking to those libraries.

  • +
+
+The Driver API Is Backward but Not Forward Compatible +
+

Figure 26 The Driver API Is Backward but Not Forward Compatible

+
+
+

For Tesla GPU products, CUDA 10 introduced a new forward-compatible upgrade path for the user-mode components of the CUDA Driver. This feature is described in CUDA Compatibility. The requirements on the CUDA Driver version described here apply to the version of the user-mode components.

+
+
+

+6.4. Compute Modes +

+

On Tesla solutions running Windows Server 2008 and later or Linux, one can set any device in a system in one of the three following modes using NVIDIA’s System Management Interface (nvidia-smi), which is a tool distributed as part of the driver:

+
    +
  • Default compute mode: Multiple host threads can use the device (by calling cudaSetDevice() on this device, when using the runtime API, or by making current a context associated to the device, when using the driver API) at the same time.

  • +
  • Exclusive-process compute mode: Only one CUDA context may be created on the device across all processes in the system. The context may be current to as many threads as desired within the process that created that context.

  • +
  • Prohibited compute mode: No CUDA context can be created on the device.

  • +
+

This means, in particular, that a host thread using the runtime API without explicitly calling cudaSetDevice() might be associated with a device other than device 0 if device 0 turns out to be in prohibited mode or in exclusive-process mode and used by another process. cudaSetValidDevices() can be used to set a device from a prioritized list of devices.

+

Note also that, for devices featuring the Pascal architecture onwards (compute capability with major revision number 6 and higher), there exists support for Compute Preemption. This allows compute tasks to be preempted at instruction-level granularity, rather than thread block granularity as in prior Maxwell and Kepler GPU architecture, with the benefit that applications with long-running kernels can be prevented from either monopolizing the system or timing out. However, there will be context switch overheads associated with Compute Preemption, which is automatically enabled on those devices for which support exists. The individual attribute query function cudaDeviceGetAttribute() with the attribute cudaDevAttrComputePreemptionSupported can be used to determine if the device in use supports Compute Preemption. Users wishing to avoid context switch overheads associated with different processes can ensure that only one process is active on the GPU by selecting exclusive-process mode.

+

Applications may query the compute mode of a device by checking the attribute cudaDevAttrComputeMode.

+
+
+

+6.5. Mode Switches +

+

GPUs that have a display output dedicate some DRAM memory to the so-called primary surface, which is used to refresh the display device whose output is viewed by the user. When users initiate a mode switch of the display by changing the resolution or bit depth of the display (using NVIDIA control panel or the Display control panel on Windows), the amount of memory needed for the primary surface changes. For example, if the user changes the display resolution from 1280x1024x32-bit to 1600x1200x32-bit, the system must dedicate 7.68 MB to the primary surface rather than 5.24 MB. (Full-screen graphics applications running with anti-aliasing enabled may require much more display memory for the primary surface.) On Windows, other events that may initiate display mode switches include launching a full-screen DirectX application, hitting Alt+Tab to task switch away from a full-screen DirectX application, or hitting Ctrl+Alt+Del to lock the computer.

+

If a mode switch increases the amount of memory needed for the primary surface, the system may have to cannibalize memory allocations dedicated to CUDA applications. Therefore, a mode switch results in any call to the CUDA runtime to fail and return an invalid context error.

+
+
+

+6.6. Tesla Compute Cluster Mode for Windows +

+

Using NVIDIA’s System Management Interface (nvidia-smi), the Windows device driver can be put in TCC (Tesla Compute Cluster) mode for devices of the Tesla and Quadro Series.

+

TCC mode removes support for any graphics functionality.

+
+
+
+

+7. Hardware Implementation +

+
+

Warning

+

This document has been replaced by a new CUDA Programming Guide. The information in this document should be considered legacy, and this document is no longer being updated as of CUDA 13.0. Please refer to the CUDA Programming Guide for up-to-date information on CUDA.

+
+

The NVIDIA GPU architecture is built around a scalable array of multithreaded Streaming Multiprocessors (SMs). When a CUDA program on the host CPU invokes a kernel grid, the blocks of the grid are enumerated and distributed to multiprocessors with available execution capacity. The threads of a thread block execute concurrently on one multiprocessor, and multiple thread blocks can execute concurrently on one multiprocessor. As thread blocks terminate, new blocks are launched on the vacated multiprocessors.

+

A multiprocessor is designed to execute hundreds of threads concurrently. To manage such a large number of threads, it employs a unique architecture called SIMT (Single-Instruction, Multiple-Thread) that is described in SIMT Architecture. The instructions are pipelined, leveraging instruction-level parallelism within a single thread, as well as extensive thread-level parallelism through simultaneous hardware multithreading as detailed in Hardware Multithreading. Unlike CPU cores, they are issued in order and there is no branch prediction or speculative execution.

+

SIMT Architecture and Hardware Multithreading describe the architecture features of the streaming multiprocessor that are common to all devices. Compute Capability 5.x, Compute Capability 6.x, and Compute Capability 7.x provide the specifics for devices of compute capabilities 5.x, 6.x, and 7.x respectively.

+

The NVIDIA GPU architecture uses a little-endian representation.

+
+

+7.1. SIMT Architecture +

+

The multiprocessor creates, manages, schedules, and executes threads in groups of 32 parallel threads called warps. Individual threads composing a warp start together at the same program address, but they have their own instruction address counter and register state and are therefore free to branch and execute independently. The term warp originates from weaving, the first parallel thread technology. A half-warp is either the first or second half of a warp. A quarter-warp is either the first, second, third, or fourth quarter of a warp.

+

When a multiprocessor is given one or more thread blocks to execute, it partitions them into warps and each warp gets scheduled by a warp scheduler for execution. The way a block is partitioned into warps is always the same; each warp contains threads of consecutive, increasing thread IDs with the first warp containing thread 0. Thread Hierarchy describes how thread IDs relate to thread indices in the block.

+

A warp executes one common instruction at a time, so full efficiency is realized when all 32 threads of a warp agree on their execution path. If threads of a warp diverge via a data-dependent conditional branch, the warp executes each branch path taken, disabling threads that are not on that path. Branch divergence occurs only within a warp; different warps execute independently regardless of whether they are executing common or disjoint code paths.

+

The SIMT architecture is akin to SIMD (Single Instruction, Multiple Data) vector organizations in that a single instruction controls multiple processing elements. A key difference is that SIMD vector organizations expose the SIMD width to the software, whereas SIMT instructions specify the execution and branching behavior of a single thread. In contrast with SIMD vector machines, SIMT enables programmers to write thread-level parallel code for independent, scalar threads, as well as data-parallel code for coordinated threads. For the purposes of correctness, the programmer can essentially ignore the SIMT behavior; however, substantial performance improvements can be realized by taking care that the code seldom requires threads in a warp to diverge. In practice, this is analogous to the role of cache lines in traditional code: Cache line size can be safely ignored when designing for correctness but must be considered in the code structure when designing for peak performance. Vector architectures, on the other hand, require the software to coalesce loads into vectors and manage divergence manually.

+

Prior to NVIDIA Volta, warps used a single program counter shared amongst all 32 threads in the warp together with an active mask specifying the active threads of the warp. As a result, threads from the same warp in divergent regions or different states of execution cannot signal each other or exchange data, and algorithms requiring fine-grained sharing of data guarded by locks or mutexes can easily lead to deadlock, depending on which warp the contending threads come from.

+

Starting with the NVIDIA Volta architecture, Independent Thread Scheduling allows full concurrency between threads, regardless of warp. With Independent Thread Scheduling, the GPU maintains execution state per thread, including a program counter and call stack, and can yield execution at a per-thread granularity, either to make better use of execution resources or to allow one thread to wait for data to be produced by another. A schedule optimizer determines how to group active threads from the same warp together into SIMT units. This retains the high throughput of SIMT execution as in prior NVIDIA GPUs, but with much more flexibility: threads can now diverge and reconverge at sub-warp granularity.

+

Independent Thread Scheduling can lead to a rather different set of threads participating in the executed code than intended if the developer made assumptions about warp-synchronicity2 of previous hardware architectures. In particular, any warp-synchronous code (such as synchronization-free, intra-warp reductions) should be revisited to ensure compatibility with NVIDIA Volta and beyond. See Compute Capability 7.x for further details.

+
+

Note

+

The threads of a warp that are participating in the current instruction are called the active threads, whereas threads not on the current instruction are inactive (disabled). Threads can be inactive for a variety of reasons including having exited earlier than other threads of their warp, having taken a different branch path than the branch path currently executed by the warp, or being the last threads of a block whose number of threads is not a multiple of the warp size.

+

If a non-atomic instruction executed by a warp writes to the same location in global or shared memory for more than one of the threads of the warp, the number of serialized writes that occur to that location varies depending on the compute capability of the device (see Compute Capability 5.x, Compute Capability 6.x, and Compute Capability 7.x), and which thread performs the final write is undefined.

+

If an atomic instruction executed by a warp reads, modifies, and writes to the same location in global memory for more than one of the threads of the warp, each read/modify/write to that location occurs and they are all serialized, but the order in which they occur is undefined.

+
+
+
+

+7.2. Hardware Multithreading +

+

The execution context (program counters, registers, and so on) for each warp processed by a multiprocessor is maintained on-chip during the entire lifetime of the warp. Therefore, switching from one execution context to another has no cost, and at every instruction issue time, a warp scheduler selects a warp that has threads ready to execute its next instruction (the active threads of the warp) and issues the instruction to those threads.

+

In particular, each multiprocessor has a set of 32-bit registers that are partitioned among the warps, and a parallel data cache or shared memory that is partitioned among the thread blocks.

+

The number of blocks and warps that can reside and be processed together on the multiprocessor for a given kernel depends on the amount of registers and shared memory used by the kernel and the amount of registers and shared memory available on the multiprocessor. There are also a maximum number of resident blocks and a maximum number of resident warps per multiprocessor. These limits as well the amount of registers and shared memory available on the multiprocessor are a function of the compute capability of the device and are given in Compute Capabilities. If there are not enough registers or shared memory available per multiprocessor to process at least one block, the kernel will fail to launch.

+

The total number of warps in a block is as follows:

+

\(\text{ceil}\left( \frac{T}{W_{size}},1 \right)\)

+
    +
  • T is the number of threads per block,

  • +
  • Wsize is the warp size, which is equal to 32,

  • +
  • ceil(x, y) is equal to x rounded up to the nearest multiple of y.

  • +
+

The total number of registers and total amount of shared memory allocated for a block are documented in the CUDA Occupancy Calculator provided in the CUDA Toolkit.

+
+
2
+
+

The term warp-synchronous refers to code that implicitly assumes threads in the same warp are synchronized at every instruction.

+
+
+
+
+
+

+8. Performance Guidelines +

+
+

Warning

+

This document has been replaced by a new CUDA Programming Guide. The information in this document should be considered legacy, and this document is no longer being updated as of CUDA 13.0. Please refer to the CUDA Programming Guide for up-to-date information on CUDA.

+
+
+

+8.1. Overall Performance Optimization Strategies +

+

Performance optimization revolves around four basic strategies:

+
    +
  • Maximize parallel execution to achieve maximum utilization;

  • +
  • Optimize memory usage to achieve maximum memory throughput;

  • +
  • Optimize instruction usage to achieve maximum instruction throughput;

  • +
  • Minimize memory thrashing.

  • +
+

Which strategies will yield the best performance gain for a particular portion of an application depends on the performance limiters for that portion; optimizing instruction usage of a kernel that is mostly limited by memory accesses will not yield any significant performance gain, for example. Optimization efforts should therefore be constantly directed by measuring and monitoring the performance limiters, for example using the CUDA profiler. Also, comparing the floating-point operation throughput or memory throughput—whichever makes more sense—of a particular kernel to the corresponding peak theoretical throughput of the device indicates how much room for improvement there is for the kernel.

+
+
+

+8.2. Maximize Utilization +

+

To maximize utilization the application should be structured in a way that it exposes as much parallelism as possible and efficiently maps this parallelism to the various components of the system to keep them busy most of the time.

+
+

+8.2.1. Application Level +

+

At a high level, the application should maximize parallel execution between the host, the devices, and the bus connecting the host to the devices, by using asynchronous functions calls and streams as described in Asynchronous Concurrent Execution. It should assign to each processor the type of work it does best: serial workloads to the host; parallel workloads to the devices.

+

For the parallel workloads, at points in the algorithm where parallelism is broken because some threads need to synchronize in order to share data with each other, there are two cases: Either these threads belong to the same block, in which case they should use __syncthreads() and share data through shared memory within the same kernel invocation, or they belong to different blocks, in which case they must share data through global memory using two separate kernel invocations, one for writing to and one for reading from global memory. The second case is much less optimal since it adds the overhead of extra kernel invocations and global memory traffic. Its occurrence should therefore be minimized by mapping the algorithm to the CUDA programming model in such a way that the computations that require inter-thread communication are performed within a single thread block as much as possible.

+
+
+

+8.2.2. Device Level +

+

At a lower level, the application should maximize parallel execution between the multiprocessors of a device.

+

Multiple kernels can execute concurrently on a device, so maximum utilization can also be achieved by using streams to enable enough kernels to execute concurrently as described in Asynchronous Concurrent Execution.

+
+
+

+8.2.3. Multiprocessor Level +

+

At an even lower level, the application should maximize parallel execution between the various functional units within a multiprocessor.

+

As described in Hardware Multithreading, a GPU multiprocessor primarily relies on thread-level parallelism to maximize utilization of its functional units. Utilization is therefore directly linked to the number of resident warps. At every instruction issue time, a warp scheduler selects an instruction that is ready to execute. This instruction can be another independent instruction of the same warp, exploiting instruction-level parallelism, or more commonly an instruction of another warp, exploiting thread-level parallelism. If a ready to execute instruction is selected it is issued to the active threads of the warp. The number of clock cycles it takes for a warp to be ready to execute its next instruction is called the latency, and full utilization is achieved when all warp schedulers always have some instruction to issue for some warp at every clock cycle during that latency period, or in other words, when latency is completely “hidden”. The number of instructions required to hide a latency of L clock cycles depends on the respective throughputs of these instructions (see the CUDA C++ Best Practices Guide for the throughputs of various arithmetic instructions). If we assume instructions with maximum throughput, it is equal to:

+
    +
  • 4L for devices of compute capability 5.x, 6.1, 6.2, 7.x and 8.x since for these devices, a multiprocessor issues one instruction per warp over one clock cycle for four warps at a time, as mentioned in Compute Capabilities.

  • +
  • 2L for devices of compute capability 6.0 since for these devices, the two instructions issued every cycle are one instruction for two different warps.

  • +
+

The most common reason a warp is not ready to execute its next instruction is that the instruction’s input operands are not available yet.

+

If all input operands are registers, latency is caused by register dependencies, i.e., some of the input operands are written by some previous instruction(s) whose execution has not completed yet. In this case, the latency is equal to the execution time of the previous instruction and the warp schedulers must schedule instructions of other warps during that time. Execution time varies depending on the instruction. On devices of compute capability 7.x, for most arithmetic instructions, it is typically 4 clock cycles. This means that 16 active warps per multiprocessor (4 cycles, 4 warp schedulers) are required to hide arithmetic instruction latencies (assuming that warps execute instructions with maximum throughput, otherwise fewer warps are needed). If the individual warps exhibit instruction-level parallelism, i.e. have multiple independent instructions in their instruction stream, fewer warps are needed because multiple independent instructions from a single warp can be issued back to back.

+

If some input operand resides in off-chip memory, the latency is much higher: typically hundreds of clock cycles. The number of warps required to keep the warp schedulers busy during such high latency periods depends on the kernel code and its degree of instruction-level parallelism. In general, more warps are required if the ratio of the number of instructions with no off-chip memory operands (i.e., arithmetic instructions most of the time) to the number of instructions with off-chip memory operands is low (this ratio is commonly called the arithmetic intensity of the program).

+

Another reason a warp is not ready to execute its next instruction is that it is waiting at some memory fence (Memory Fence Functions) or synchronization point (Synchronization Functions). A synchronization point can force the multiprocessor to idle as more and more warps wait for other warps in the same block to complete execution of instructions prior to the synchronization point. Having multiple resident blocks per multiprocessor can help reduce idling in this case, as warps from different blocks do not need to wait for each other at synchronization points.

+

The number of blocks and warps residing on each multiprocessor for a given kernel call depends on the execution configuration of the call (Execution Configuration), the memory resources of the multiprocessor, and the resource requirements of the kernel as described in Hardware Multithreading. Register and shared memory usage are reported by the compiler when compiling with the --ptxas-options=-v option.

+

The total amount of shared memory required for a block is equal to the sum of the amount of statically allocated shared memory and the amount of dynamically allocated shared memory.

+

The number of registers used by a kernel can have a significant impact on the number of resident warps. For example, for devices of compute capability 6.x, if a kernel uses 64 +registers and each block has 512 threads and requires very little shared memory, then two blocks (i.e., 32 warps) can reside on the multiprocessor since they require 2x512x64 +registers, which exactly matches the number of registers available on the multiprocessor. But as soon as the kernel uses one more register, only one block (i.e., 16 warps) can be +resident since two blocks would require 2x512x65 registers, which are more registers than are available on the multiprocessor. Therefore, the compiler attempts to minimize register +usage while keeping register spilling (see Device Memory Accesses) and the number of instructions to a minimum. Register usage can be +controlled using the maxrregcount compiler option, the __launch_bounds__() qualifier as described in Launch Bounds, or the __maxnreg__() +qualifier as described in Maximum Number of Registers per Thread.

+

The register file is organized as 32-bit registers. So, each variable stored in a register needs at least one 32-bit register, for example, a double variable uses two 32-bit registers.

+

The effect of execution configuration on performance for a given kernel call generally depends on the kernel code. Experimentation is therefore recommended. Applications can also parametrize execution configurations based on register file size and shared memory size, which depends on the compute capability of the device, as well as on the number of multiprocessors and memory bandwidth of the device, all of which can be queried using the runtime (see reference manual).

+

The number of threads per block should be chosen as a multiple of the warp size to avoid wasting computing resources with under-populated warps as much as possible.

+
+

+8.2.3.1. Occupancy Calculator +

+

Several API functions exist to assist programmers in choosing thread block size and cluster size based on register and shared memory requirements.

+
    +
  • +

    The occupancy calculator API, cudaOccupancyMaxActiveBlocksPerMultiprocessor, can provide an occupancy prediction based on the block size and shared memory usage of a kernel. This function reports occupancy in terms of the number of concurrent thread blocks per multiprocessor.

    +
      +
    • Note that this value can be converted to other metrics. Multiplying by the number of warps per block yields the number of concurrent warps per multiprocessor; further dividing concurrent warps by max warps per multiprocessor gives the occupancy as a percentage.

    • +
    +
  • +
  • The occupancy-based launch configurator APIs, cudaOccupancyMaxPotentialBlockSize and cudaOccupancyMaxPotentialBlockSizeVariableSMem, heuristically calculate an execution configuration that achieves the maximum multiprocessor-level occupancy.

  • +
  • The occupancy calculator API, cudaOccupancyMaxActiveClusters, can provided occupancy prediction based on the cluster size, block size and shared memory usage of a kernel. This function reports occupancy in terms of number of max active clusters of a given size on the GPU present in the system.

  • +
+

The following code sample calculates the occupancy of MyKernel. It then reports the occupancy level with the ratio between concurrent warps versus maximum warps per multiprocessor.

+
+
// Device code
+__global__ void MyKernel(int *d, int *a, int *b)
+{
+    int idx = threadIdx.x + blockIdx.x * blockDim.x;
+    d[idx] = a[idx] * b[idx];
+}
+
+// Host code
+int main()
+{
+    int numBlocks;        // Occupancy in terms of active blocks
+    int blockSize = 32;
+
+    // These variables are used to convert occupancy to warps
+    int device;
+    cudaDeviceProp prop;
+    int activeWarps;
+    int maxWarps;
+
+    cudaGetDevice(&device);
+    cudaGetDeviceProperties(&prop, device);
+
+    cudaOccupancyMaxActiveBlocksPerMultiprocessor(
+        &numBlocks,
+        MyKernel,
+        blockSize,
+        0);
+
+    activeWarps = numBlocks * blockSize / prop.warpSize;
+    maxWarps = prop.maxThreadsPerMultiProcessor / prop.warpSize;
+
+    std::cout << "Occupancy: " << (double)activeWarps / maxWarps * 100 << "%" << std::endl;
+
+    return 0;
+}
+
+
+

The following code sample configures an occupancy-based kernel launch of MyKernel according to the user input.

+
+
// Device code
+__global__ void MyKernel(int *array, int arrayCount)
+{
+    int idx = threadIdx.x + blockIdx.x * blockDim.x;
+    if (idx < arrayCount) {
+        array[idx] *= array[idx];
+    }
+}
+
+// Host code
+int launchMyKernel(int *array, int arrayCount)
+{
+    int blockSize;      // The launch configurator returned block size
+    int minGridSize;    // The minimum grid size needed to achieve the
+                        // maximum occupancy for a full device
+                        // launch
+    int gridSize;       // The actual grid size needed, based on input
+                        // size
+
+    cudaOccupancyMaxPotentialBlockSize(
+        &minGridSize,
+        &blockSize,
+        (void*)MyKernel,
+        0,
+        arrayCount);
+
+    // Round up according to array size
+    gridSize = (arrayCount + blockSize - 1) / blockSize;
+
+    MyKernel<<<gridSize, blockSize>>>(array, arrayCount);
+    cudaDeviceSynchronize();
+
+    // If interested, the occupancy can be calculated with
+    // cudaOccupancyMaxActiveBlocksPerMultiprocessor
+
+    return 0;
+}
+
+
+

The following code sample shows how to use the cluster occupancy API to find the max number of active clusters of a given size. Example code below calcualtes occupancy for cluster of size 2 and 128 threads per block.

+

Cluster size of 8 is forward compatible starting compute capability 9.0, except on GPU hardware or MIG configurations which are too small to support 8 multiprocessors in which case the maximum cluster size will be reduced. But it is recommended that the users query the maximum cluster size before launching a cluster kernel. Max cluster size can be queried using cudaOccupancyMaxPotentialClusterSize API.

+
+
{
+  cudaLaunchConfig_t config = {0};
+  config.gridDim = number_of_blocks;
+  config.blockDim = 128; // threads_per_block = 128
+  config.dynamicSmemBytes = dynamic_shared_memory_size;
+
+  cudaLaunchAttribute attribute[1];
+  attribute[0].id = cudaLaunchAttributeClusterDimension;
+  attribute[0].val.clusterDim.x = 2; // cluster_size = 2
+  attribute[0].val.clusterDim.y = 1;
+  attribute[0].val.clusterDim.z = 1;
+  config.attrs = attribute;
+  config.numAttrs = 1;
+
+  int max_cluster_size = 0;
+  cudaOccupancyMaxPotentialClusterSize(&max_cluster_size, (void *)kernel, &config);
+
+  int max_active_clusters = 0;
+  cudaOccupancyMaxActiveClusters(&max_active_clusters, (void *)kernel, &config);
+
+  std::cout << "Max Active Clusters of size 2: " << max_active_clusters << std::endl;
+}
+
+
+

The CUDA Nsight Compute User Interface also provides a standalone occupancy calculator and launch configurator implementation in <CUDA_Toolkit_Path>/include/cuda_occupancy.h for any use cases that cannot depend on the CUDA software stack. The Nsight Compute version of the occupancy calculator is particularly useful as a learning tool that visualizes the impact of changes to the parameters that affect occupancy (block size, registers per thread, and shared memory per thread).

+
+
+
+
+

+8.3. Maximize Memory Throughput +

+

The first step in maximizing overall memory throughput for the application is to minimize data transfers with low bandwidth.

+

That means minimizing data transfers between the host and the device, as detailed in Data Transfer between Host and Device, since these have much lower bandwidth than data transfers between global memory and the device.

+

That also means minimizing data transfers between global memory and the device by maximizing use of on-chip memory: shared memory and caches (i.e., L1 cache and L2 cache available on devices of compute capability 2.x and higher, texture cache and constant cache available on all devices).

+

Shared memory is equivalent to a user-managed cache: The application explicitly allocates and accesses it. As illustrated in CUDA Runtime, a typical programming pattern is to stage data coming from device memory into shared memory; in other words, to have each thread of a block:

+
    +
  • Load data from device memory to shared memory,

  • +
  • Synchronize with all the other threads of the block so that each thread can safely read shared memory locations that were populated by different threads,

  • +
  • Process the data in shared memory,

  • +
  • Synchronize again if necessary to make sure that shared memory has been updated with the results,

  • +
  • Write the results back to device memory.

  • +
+

For some applications (for example, for which global memory access patterns are data-dependent), a traditional hardware-managed cache is more appropriate to exploit data locality. As mentioned in Compute Capability 7.x, Compute Capability 8.x and Compute Capability 9.0, for devices of compute capability 7.x, 8.x and 9.0, the same on-chip memory is used for both L1 and shared memory, and how much of it is dedicated to L1 versus shared memory is configurable for each kernel call.

+

The throughput of memory accesses by a kernel can vary by an order of magnitude depending on access pattern for each type of memory. The next step in maximizing memory throughput is therefore to organize memory accesses as optimally as possible based on the optimal memory access patterns described in Device Memory Accesses. This optimization is especially important for global memory accesses as global memory bandwidth is low compared to available on-chip bandwidths and arithmetic instruction throughput, so non-optimal global memory accesses generally have a high impact on performance.

+
+

+8.3.1. Data Transfer between Host and Device +

+

Applications should strive to minimize data transfer between the host and the device. One way to accomplish this is to move more code from the host to the device, even if that means running kernels that do not expose enough parallelism to execute on the device with full efficiency. Intermediate data structures may be created in device memory, operated on by the device, and destroyed without ever being mapped by the host or copied to host memory.

+

Also, because of the overhead associated with each transfer, batching many small transfers into a single large transfer always performs better than making each transfer separately.

+

On systems with a front-side bus, higher performance for data transfers between host and device is achieved by using page-locked host memory as described in Page-Locked Host Memory.

+

In addition, when using mapped page-locked memory (Mapped Memory), there is no need to allocate any device memory and explicitly copy data between device and host memory. Data transfers are implicitly performed each time the kernel accesses the mapped memory. For maximum performance, these memory accesses must be coalesced as with accesses to global memory (see Device Memory Accesses). Assuming that they are and that the mapped memory is read or written only once, using mapped page-locked memory instead of explicit copies between device and host memory can be a win for performance.

+

On integrated systems where device memory and host memory are physically the same, any copy between host and device memory is superfluous and mapped page-locked memory should be used instead. Applications may query a device is integrated by checking that the integrated device property (see Device Enumeration) is equal to 1.

+
+
+

+8.3.2. Device Memory Accesses +

+

An instruction that accesses addressable memory (i.e., global, local, shared, constant, or texture memory) might need to be re-issued multiple times depending on the distribution of the memory addresses across the threads within the warp. How the distribution affects the instruction throughput this way is specific to each type of memory and described in the following sections. For example, for global memory, as a general rule, the more scattered the addresses are, the more reduced the throughput is.

+

Global Memory

+

Global memory resides in device memory and device memory is accessed via 32-, 64-, or 128-byte memory transactions. These memory transactions must be naturally aligned: Only the 32-, 64-, or 128-byte segments of device memory that are aligned to their size (i.e., whose first address is a multiple of their size) can be read or written by memory transactions.

+

When a warp executes an instruction that accesses global memory, it coalesces the memory accesses of the threads within the warp into one or more of these memory transactions depending on the size of the word accessed by each thread and the distribution of the memory addresses across the threads. In general, the more transactions are necessary, the more unused words are transferred in addition to the words accessed by the threads, reducing the instruction throughput accordingly. For example, if a 32-byte memory transaction is generated for each thread’s 4-byte access, throughput is divided by 8.

+

How many transactions are necessary and how much throughput is ultimately affected varies with the compute capability of the device. Compute Capability 5.x, Compute Capability 6.x, Compute Capability 7.x, Compute Capability 8.x, Compute Capability 9.0, Compute Capability 10.0, and Compute Capability 12.0 give more details on how global memory accesses are handled for various compute capabilities.

+

To maximize global memory throughput, it is therefore important to maximize coalescing by:

+ +

Size and Alignment Requirement

+

Global memory instructions support reading or writing words of size equal to 1, 2, 4, 8, or 16 bytes. Any access (via a variable or a pointer) to data residing in global memory compiles to a single global memory instruction if and only if the size of the data type is 1, 2, 4, 8, or 16 bytes and the data is naturally aligned (i.e., its address is a multiple of that size).

+

If this size and alignment requirement is not fulfilled, the access compiles to multiple instructions with interleaved access patterns that prevent these instructions from fully coalescing. It is therefore recommended to use types that meet this requirement for data that resides in global memory.

+

The alignment requirement is automatically fulfilled for the Built-in Vector Types.

+

For structures, the size and alignment requirements can be enforced by the compiler using the alignment specifiers__align__(8) or __align__(16), such as

+
+
struct __align__(8) {
+    float x;
+    float y;
+};
+
+
+

or

+
+
struct __align__(16) {
+    float x;
+    float y;
+    float z;
+};
+
+
+

Any address of a variable residing in global memory or returned by one of the memory allocation routines from the driver or runtime API is always aligned to at least 256 bytes.

+

Reading non-naturally aligned 8-byte or 16-byte words produces incorrect results (off by a few words), so special care must be taken to maintain alignment of the starting address of any value or array of values of these types. A typical case where this might be easily overlooked is when using some custom global memory allocation scheme, whereby the allocations of multiple arrays (with multiple calls to cudaMalloc() or cuMemAlloc()) is replaced by the allocation of a single large block of memory partitioned into multiple arrays, in which case the starting address of each array is offset from the block’s starting address.

+

Two-Dimensional Arrays

+

A common global memory access pattern is when each thread of index (tx,ty) uses the following address to access one element of a 2D array of width width, located at address BaseAddress of type type* (where type meets the requirement described in Maximize Utilization):

+
+
BaseAddress + width * ty + tx
+
+
+

For these accesses to be fully coalesced, both the width of the thread block and the width of the array must be a multiple of the warp size.

+

In particular, this means that an array whose width is not a multiple of this size will be accessed much more efficiently if it is actually allocated with a width rounded up to the closest multiple of this size and its rows padded accordingly. The cudaMallocPitch() and cuMemAllocPitch() functions and associated memory copy functions described in the reference manual enable programmers to write non-hardware-dependent code to allocate arrays that conform to these constraints.

+

Local Memory

+

Local memory accesses only occur for some automatic variables as mentioned in Variable Memory Space Specifiers. Automatic variables that the compiler is likely to place in local memory are:

+
    +
  • Arrays for which it cannot determine that they are indexed with constant quantities,

  • +
  • Large structures or arrays that would consume too much register space,

  • +
  • Any variable if the kernel uses more registers than available (this is also known as register spilling).

  • +
+

Inspection of the PTX assembly code (obtained by compiling with the -ptx or-keep option) will tell if a variable has been placed in local memory during the first compilation phases as it will be declared using the .local mnemonic and accessed using the ld.local and st.local mnemonics. Even if it has not, subsequent compilation phases might still decide otherwise though if they find it consumes too much register space for the targeted architecture: Inspection of the cubin object using cuobjdump will tell if this is the case. Also, the compiler reports total local memory usage per kernel (lmem) when compiling with the --ptxas-options=-v option. Note that some mathematical functions have implementation paths that might access local memory.

+

The local memory space resides in device memory, so local memory accesses have the same high latency and low bandwidth as global memory accesses and are subject to the same requirements for memory coalescing as described in Device Memory Accesses. Local memory is however organized such that consecutive 32-bit words are accessed by consecutive thread IDs. Accesses are therefore fully coalesced as long as all threads in a warp access the same relative address (for example, same index in an array variable, same member in a structure variable).

+

On devices of compute capability 5.x onwards, local memory accesses are always cached in L2 in the same way as global memory accesses (see Compute Capability 5.x and Compute Capability 6.x).

+

Shared Memory

+

Because it is on-chip, shared memory has much higher bandwidth and much lower latency than local or global memory.

+

To achieve high bandwidth, shared memory is divided into equally-sized memory modules, called banks, which can be accessed simultaneously. Any memory read or write request made of n addresses that fall in n distinct memory banks can therefore be serviced simultaneously, yielding an overall bandwidth that is n times as high as the bandwidth of a single module.

+

However, if two addresses of a memory request fall in the same memory bank, there is a bank conflict and the access has to be serialized. The hardware splits a memory request with bank conflicts into as many separate conflict-free requests as necessary, decreasing throughput by a factor equal to the number of separate memory requests. If the number of separate memory requests is n, the initial memory request is said to cause n-way bank conflicts.

+

To get maximum performance, it is therefore important to understand how memory addresses map to memory banks in order to schedule the memory requests so as to minimize bank conflicts. This is described in Compute Capability 5.x, Compute Capability 6.x, Compute Capability 7.x, Compute Capability 8.x, Compute Capability 9.0, Compute Capability 10.0, and Compute Capability 12.0 for devices of these compute capabilities respectively.

+

Constant Memory

+

The constant memory space resides in device memory and is cached in the constant cache.

+

A request is then split into as many separate requests as there are different memory addresses in the initial request, decreasing throughput by a factor equal to the number of separate requests.

+

The resulting requests are then serviced at the throughput of the constant cache in case of a cache hit, or at the throughput of device memory otherwise.

+

Texture and Surface Memory

+

The texture and surface memory spaces reside in device memory and are cached in texture cache, so a texture fetch or surface read costs one memory read from device memory only on a cache miss, otherwise it just costs one read from texture cache. The texture cache is optimized for 2D spatial locality, so threads of the same warp that read texture or surface addresses that are close together in 2D will achieve best performance. Also, it is designed for streaming fetches with a constant latency; a cache hit reduces DRAM bandwidth demand but not fetch latency.

+

Reading device memory through texture or surface fetching present some benefits that can make it an advantageous alternative to reading device memory from global or constant memory:

+
    +
  • If the memory reads do not follow the access patterns that global or constant memory reads must follow to get good performance, higher bandwidth can be achieved providing that there is locality in the texture fetches or surface reads;

  • +
  • Addressing calculations are performed outside the kernel by dedicated units;

  • +
  • Packed data may be broadcast to separate variables in a single operation;

  • +
  • 8-bit and 16-bit integer input data may be optionally converted to 32 bit floating-point values in the range [0.0, 1.0] or [-1.0, 1.0] (see Texture Memory).

  • +
+
+
+
+

+8.4. Maximize Instruction Throughput +

+

See the CUDA C++ Best Practices Guide +for more details on optimizing instruction throughput.

+
+
+

+8.5. Minimize Memory Thrashing +

+

Applications that constantly allocate and free memory too often may find that the allocation calls tend to get slower over time up to a limit. This is typically expected due to the nature of releasing memory back to the operating system for its own use. For best performance in this regard, we recommend the following:

+
    +
  • Try to size your allocation to the problem at hand. Don’t try to allocate all available memory with cudaMalloc / cudaMallocHost / cuMemCreate, as this forces memory to be resident immediately and prevents other applications from being able to use that memory. This can put more pressure on operating system schedulers, or just prevent other applications using the same GPU from running entirely.

  • +
  • Try to allocate memory in appropriately sized allocations early in the application and allocations only when the application does not have any use for it. Reduce the number of cudaMalloc+cudaFree calls in the application, especially in performance-critical regions.

  • +
  • If an application cannot allocate enough device memory, consider falling back on other memory types such as cudaMallocHost or cudaMallocManaged, which may not be as performant, but will enable the application to make progress.

  • +
  • For platforms that support the feature, cudaMallocManaged allows for oversubscription, and with the correct cudaMemAdvise policies enabled, will allow the application to retain most if not all the performance of cudaMalloc. cudaMallocManaged also won’t force an allocation to be resident until it is needed or prefetched, reducing the overall pressure on the operating system schedulers and better enabling multi-tenet use cases.

  • +
+
+
+
+

+9. CUDA-Enabled GPUs +

+
+

Warning

+

This document has been replaced by a new CUDA Programming Guide. The information in this document should be considered legacy, and this document is no longer being updated as of CUDA 13.0. Please refer to the CUDA Programming Guide for up-to-date information on CUDA.

+
+

https://developer.nvidia.com/cuda-gpus lists all CUDA-enabled devices with their compute capability.

+

The compute capability, number of multiprocessors, clock frequency, total amount of device memory, and other properties can be queried using the runtime (see reference manual).

+
+
+

+10. C++ Language Extensions +

+
+

Warning

+

This document has been replaced by a new CUDA Programming Guide. The information in this document should be considered legacy, and this document is no longer being updated as of CUDA 13.0. Please refer to the CUDA Programming Guide for up-to-date information on CUDA.

+
+
+

+10.1. Function Execution Space Specifiers +

+

Function execution space specifiers denote whether a function executes on the host or on the device and whether it is callable from the host or from the device.

+
+

+10.1.1. __global__ +

+

The __global__ execution space specifier declares a function as being a kernel. Such a function is:

+
    +
  • Executed on the device,

  • +
  • Callable from the host,

  • +
  • Callable from the device for devices of compute capability 5.0 or higher (see CUDA Dynamic Parallelism for more details).

  • +
+

A __global__ function must have void return type, and cannot be a member of a class.

+

Any call to a __global__ function must specify its execution configuration as described in Execution Configuration.

+

A call to a __global__ function is asynchronous, meaning it returns before the device has completed its execution.

+
+
+

+10.1.2. __device__ +

+

The __device__ execution space specifier declares a function that is:

+
    +
  • Executed on the device,

  • +
  • Callable from the device only.

  • +
+

The __global__ and __device__ execution space specifiers cannot be used together.

+
+
+

+10.1.3. __host__ +

+

The __host__ execution space specifier declares a function that is:

+
    +
  • Executed on the host,

  • +
  • Callable from the host only.

  • +
+

It is equivalent to declare a function with only the __host__ execution space specifier or to declare it without any of the __host__, __device__, or __global__ execution space specifier; in either case the function is compiled for the host only.

+

The __global__ and __host__ execution space specifiers cannot be used together.

+

The __device__ and __host__ execution space specifiers can be used together however, in which case the function is compiled for both the host and the device. The __CUDA_ARCH__ macro introduced in Application Compatibility can be used to differentiate code paths between host and device:

+
+
__host__ __device__ func()
+{
+#if __CUDA_ARCH__ >= 800
+   // Device code path for compute capability 8.x
+#elif __CUDA_ARCH__ >= 700
+   // Device code path for compute capability 7.x
+#elif __CUDA_ARCH__ >= 600
+   // Device code path for compute capability 6.x
+#elif __CUDA_ARCH__ >= 500
+   // Device code path for compute capability 5.x
+#elif !defined(__CUDA_ARCH__)
+   // Host code path
+#endif
+}
+
+
+
+
+

+10.1.4. Undefined behavior +

+

A ‘cross-execution space’ call has undefined behavior when:

+
    +
  • __CUDA_ARCH__ is defined, a call from within a __global__, __device__ or __host__ __device__ function to a __host__ function.

  • +
  • __CUDA_ARCH__ is undefined, a call from within a __host__ function to a __device__ function. 4

  • +
+
+
+

+10.1.5. __noinline__ and __forceinline__ +

+

The compiler inlines any __device__ function when deemed appropriate.

+

The __noinline__ function qualifier can be used as a hint for the compiler not to inline the function if possible.

+

The __forceinline__ function qualifier can be used to force the compiler to inline the function.

+

The __noinline__ and __forceinline__ function qualifiers cannot be used together, and neither function qualifier can be applied to an inline function.

+
+
+

+10.1.6. __inline_hint__ +

+

The __inline_hint__ qualifier enables more aggressive inlining in the compiler. Unlike __forceinline__, it does not imply that the function is inline. It can be used to improve inlining across modules when using LTO.

+

Neither the __noinline__ nor the __forceinline__ function qualifier can be used with the __inline_hint__ function qualifier.

+
+
+
+

+10.2. Variable Memory Space Specifiers +

+

Variable memory space specifiers denote the memory location on the device of a variable.

+

An automatic variable declared in device code without any of the __device__, __shared__ and __constant__ memory space specifiers described in this section generally resides in a register. However in some cases the compiler might choose to place it in local memory, which can have adverse performance consequences as detailed in Device Memory Accesses.

+
+

+10.2.1. __device__ +

+

The __device__ memory space specifier declares a variable that resides on the device.

+

At most one of the other memory space specifiers defined in the next three sections may be used together with __device__ to further denote which memory space the variable belongs to. If none of them is present, the variable:

+
    +
  • Resides in global memory space,

  • +
  • Has the lifetime of the CUDA context in which it is created,

  • +
  • Has a distinct object per device,

  • +
  • Is accessible from all the threads within the grid and from the host through the runtime library (cudaGetSymbolAddress() / cudaGetSymbolSize() / cudaMemcpyToSymbol() / cudaMemcpyFromSymbol()).

  • +
+
+
+

+10.2.2. __constant__ +

+

The __constant__ memory space specifier, optionally used together with __device__, declares a variable that:

+
    +
  • Resides in constant memory space,

  • +
  • Has the lifetime of the CUDA context in which it is created,

  • +
  • Has a distinct object per device,

  • +
  • Is accessible from all the threads within the grid and from the host through the runtime library (cudaGetSymbolAddress() / cudaGetSymbolSize() / cudaMemcpyToSymbol() / cudaMemcpyFromSymbol()).

  • +
+

The behavior of modifying a constant from the host while there is a concurrent grid that access that constant at any point of this grid’s lifetime is undefined.

+
+
+

+10.2.3. __shared__ +

+

The __shared__ memory space specifier, optionally used together with __device__, declares a variable that:

+
    +
  • Resides in the shared memory space of a thread block,

  • +
  • Has the lifetime of the block,

  • +
  • Has a distinct object per block,

  • +
  • Is only accessible from all the threads within the block,

  • +
  • Does not have a constant address.

  • +
+

When declaring a variable in shared memory as an external array such as

+
+
extern __shared__ float shared[];
+
+
+

the size of the array is determined at launch time (see Execution Configuration). All variables declared in this fashion, start at the same address in memory, so that the layout of the variables in the array must be explicitly managed through offsets. For example, if one wants the equivalent of

+
+
short array0[128];
+float array1[64];
+int   array2[256];
+
+
+

in dynamically allocated shared memory, one could declare and initialize the arrays the following way:

+
+
extern __shared__ float array[];
+__device__ void func()      // __device__ or __global__ function
+{
+    short* array0 = (short*)array;
+    float* array1 = (float*)&array0[128];
+    int*   array2 =   (int*)&array1[64];
+}
+
+
+

Note that pointers need to be aligned to the type they point to, so the following code, for example, does not work since array1 is not aligned to 4 bytes.

+
+
extern __shared__ float array[];
+__device__ void func()      // __device__ or __global__ function
+{
+    short* array0 = (short*)array;
+    float* array1 = (float*)&array0[127];
+}
+
+
+

Alignment requirements for the built-in vector types are listed in Table 7.

+
+
+

+10.2.4. __grid_constant__ +

+

The __grid_constant__ annotation for compute architectures greater or equal to 7.0 annotates a const-qualified __global__ function parameter of non-reference type that:

+
    +
  • Has the lifetime of the grid,

  • +
  • Is private to the grid, i.e., the object is not accessible to host threads and threads from other grids, including sub-grids,

  • +
  • Has a distinct object per grid, i.e., all threads in the grid see the same address,

  • +
  • Is read-only, i.e., modifying a __grid_constant__ object or any of its sub-objects is undefined behavior, including mutable members.

  • +
+

Requirements:

+
    +
  • Kernel parameters annotated with __grid_constant__ must have const-qualified non-reference types.

  • +
  • All function declarations must match with respect to any __grid_constant_ parameters.

  • +
  • A function template specialization must match the primary template declaration with respect to any __grid_constant__ parameters.

  • +
  • A function template instantiation directive must match the primary template declaration with respect to any __grid_constant__ parameters.

  • +
+

If the address of a __global__ function parameter is taken, the compiler will ordinarily make a copy of the kernel parameter in thread local memory and use the address of the copy, to partially support C++ semantics, which allow each thread to modify its own local copy of function parameters. Annotating a __global__ function parameter with __grid_constant__ ensures that the compiler will not create a copy of the kernel parameter in thread local memory, but will instead use the generic address of the parameter itself. Avoiding the local copy may result in improved performance.

+
+
__device__ void unknown_function(S const&);
+__global__ void kernel(const __grid_constant__ S s) {
+   s.x += threadIdx.x;  // Undefined Behavior: tried to modify read-only memory
+
+   // Compiler will _not_ create a per-thread thread local copy of "s":
+   unknown_function(s);
+}
+
+
+
+
+

+10.2.5. __managed__ +

+

The __managed__ memory space specifier, optionally used together with __device__, declares a variable that:

+
    +
  • Can be referenced from both device and host code, for example, its address can be taken or it can be read or written directly from a device or host function.

  • +
  • Has the lifetime of an application.

  • +
+

See __managed__ Memory Space Specifier for more details.

+
+
+

+10.2.6. __restrict__ +

+

nvcc supports restricted pointers via the __restrict__ keyword.

+

Restricted pointers were introduced in C99 to alleviate the aliasing problem that exists in C-type languages, and which inhibits all kind of optimization from code re-ordering to common sub-expression elimination.

+

Here is an example subject to the aliasing issue, where use of restricted pointer can help the compiler to reduce the number of instructions:

+
+
void foo(const float* a,
+         const float* b,
+         float* c)
+{
+    c[0] = a[0] * b[0];
+    c[1] = a[0] * b[0];
+    c[2] = a[0] * b[0] * a[1];
+    c[3] = a[0] * a[1];
+    c[4] = a[0] * b[0];
+    c[5] = b[0];
+    ...
+}
+
+
+

In C-type languages, the pointers a, b, and c may be aliased, so any write through c could modify elements of a or b. This means that to guarantee functional correctness, the compiler cannot load a[0] and b[0] into registers, multiply them, and store the result to both c[0] and c[1], because the results would differ from the abstract execution model if, say, a[0] is really the same location as c[0]. So the compiler cannot take advantage of the common sub-expression. Likewise, the compiler cannot just reorder the computation of c[4] into the proximity of the computation of c[0] and c[1] because the preceding write to c[3] could change the inputs to the computation of c[4].

+

By making a, b, and c restricted pointers, the programmer asserts to the compiler that the pointers are in fact not aliased, which in this case means writes through c would never overwrite elements of a or b. This changes the function prototype as follows:

+
+
void foo(const float* __restrict__ a,
+         const float* __restrict__ b,
+         float* __restrict__ c);
+
+
+

Note that all pointer arguments need to be made restricted for the compiler optimizer to derive any benefit. With the __restrict__ keywords added, the compiler can now reorder and do common sub-expression elimination at will, while retaining functionality identical with the abstract execution model:

+
+
void foo(const float* __restrict__ a,
+         const float* __restrict__ b,
+         float* __restrict__ c)
+{
+    float t0 = a[0];
+    float t1 = b[0];
+    float t2 = t0 * t1;
+    float t3 = a[1];
+    c[0] = t2;
+    c[1] = t2;
+    c[4] = t2;
+    c[2] = t2 * t3;
+    c[3] = t0 * t3;
+    c[5] = t1;
+    ...
+}
+
+
+

The effects here are a reduced number of memory accesses and reduced number of computations. This is balanced by an increase in register pressure due to “cached” loads and common sub-expressions.

+

Since register pressure is a critical issue in many CUDA codes, use of restricted pointers can have negative performance impact on CUDA code, due to reduced occupancy.

+
+
+
+

+10.3. Built-in Vector Types +

+
+

+10.3.1. char, short, int, long, longlong, float, double +

+

These are vector types derived from the basic integer and floating-point types. They are structures and the 1st, 2nd, 3rd, and 4th components are accessible through the fields x, y, z, and w, respectively. They all come with a constructor function of the form make_<type name>; for example,

+
+
int2 make_int2(int x, int y);
+
+
+

which creates a vector of type int2 with value(x, y).

+

The alignment requirements of the vector types are detailed in Table 7.

+ + ++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+Table 7 Alignment Requirements +

Type

Alignment

char1, uchar1

1

char2, uchar2

2

char3, uchar3

1

char4, uchar4

4

short1, ushort1

2

short2, ushort2

4

short3, ushort3

2

short4, ushort4

8

int1, uint1

4

int2, uint2

8

int3, uint3

4

int4, uint4

16

long1, ulong1

4 if sizeof(long) is equal to sizeof(int) 8, otherwise

long2, ulong2

8 if sizeof(long) is equal to sizeof(int), 16, otherwise

long3, ulong3

4 if sizeof(long) is equal to sizeof(int), 8, otherwise

long4 3

16

long4_16a

long4_32a

32

ulong4 3

16

ulong4_16a

ulong4_32a

32

longlong1, ulonglong1

8

longlong2, ulonglong2

16

longlong3, ulonglong3

8

longlong4 3

16

longlong4_16a

longlong4_32a

32

ulonglong4 3

16

ulonglong4_16a

ulonglong4_32a

32

float1

4

float2

8

float3

4

float4

16

double1

8

double2

16

double3

8

double4 3

16

double4_16a

double4_32a

32

+
+
+3(1,2,3,4,5) +
+
+

This vector type was deprecated in CUDA Toolkit 13.0. Please use the _16a or _32a variant instead, depending on your alignment requirements.

+
+
+
+
+

+10.3.2. dim3 +

+

This type is an integer vector type based on uint3 that is used to specify dimensions. When defining a variable of type dim3, any component left unspecified is initialized to 1.

+
+
+
+

+10.4. Built-in Variables +

+

Built-in variables specify the grid and block dimensions and the block and thread indices. They are only valid within functions that are executed on the device.

+
+

+10.4.1. gridDim +

+

This variable is of type dim3 (see dim3) and contains the dimensions of the grid.

+
+
+

+10.4.2. blockIdx +

+

This variable is of type uint3 (see char, short, int, long, longlong, float, double) and contains the block index within the grid.

+
+
+

+10.4.3. blockDim +

+

This variable is of type dim3 (see dim3) and contains the dimensions of the block.

+
+
+

+10.4.4. threadIdx +

+

This variable is of type uint3 (see char, short, int, long, longlong, float, double) and contains the thread index within the block.

+
+
+

+10.4.5. warpSize +

+

This variable is of type int and contains the warp size in threads (see SIMT Architecture for the definition of a warp).

+
+
+
+

+10.5. Memory Fence Functions +

+

The CUDA programming model assumes a device with a weakly-ordered memory model, that is the order in which a CUDA thread writes data to shared memory, global memory, page-locked host memory, or the memory of a peer device is not necessarily the order in which the data is observed being written by another CUDA or host thread. It is undefined behavior for two threads to read from or write to the same memory location without synchronization.

+

In the following example, thread 1 executes writeXY(), while thread 2 executes readXY().

+
+
__device__ int X = 1, Y = 2;
+
+__device__ void writeXY()
+{
+    X = 10;
+    Y = 20;
+}
+
+__device__ void readXY()
+{
+    int B = Y;
+    int A = X;
+}
+
+
+

The two threads read and write from the same memory locations X and Y simultaneously. Any data-race is undefined behavior, and has no defined semantics. The resulting values for A and B can be anything.

+

Memory fence functions can be used to enforce a sequentially-consistent ordering on memory accesses. The memory fence functions differ in the scope in which the orderings are enforced but they are independent of the accessed memory space (shared memory, global memory, page-locked host memory, and the memory of a peer device).

+
+
void __threadfence_block();
+
+
+

is equivalent to cuda::atomic_thread_fence(cuda::memory_order_seq_cst, cuda::thread_scope_block) and ensures that:

+
    +
  • All writes to all memory made by the calling thread before the call to __threadfence_block() are observed by all threads in the block of the calling thread as occurring before all writes to all memory made by the calling thread after the call to __threadfence_block();

  • +
  • All reads from all memory made by the calling thread before the call to __threadfence_block() are ordered before all reads from all memory made by the calling thread after the call to __threadfence_block().

  • +
+
+
void __threadfence();
+
+
+

is equivalent to cuda::atomic_thread_fence(cuda::memory_order_seq_cst, cuda::thread_scope_device) and ensures that no writes to all memory made by the calling thread after the call to __threadfence() are observed by any thread in the device as occurring before any write to all memory made by the calling thread before the call to __threadfence().

+
+
void __threadfence_system();
+
+
+

is equivalent to cuda::atomic_thread_fence(cuda::memory_order_seq_cst, cuda::thread_scope_system) and ensures that all writes to all memory made by the calling thread before the call to __threadfence_system() are observed by all threads in the device, host threads, and all threads in peer devices as occurring before all writes to all memory made by the calling thread after the call to __threadfence_system().

+

__threadfence_system() is only supported by devices of compute capability 2.x and higher.

+

In the previous code sample, we can insert fences in the codes as follows:

+
+
__device__ int X = 1, Y = 2;
+
+__device__ void writeXY()
+{
+    X = 10;
+    __threadfence();
+    Y = 20;
+}
+
+__device__ void readXY()
+{
+    int B = Y;
+    __threadfence();
+    int A = X;
+}
+
+
+

For this code, the following outcomes can be observed:

+
    +
  • A equal to 1 and B equal to 2,

  • +
  • A equal to 10 and B equal to 2,

  • +
  • A equal to 10 and B equal to 20.

  • +
+

The fourth outcome is not possible, because the first write must be visible before the second write. If thread 1 and 2 belong to the same block, it is enough to use __threadfence_block(). If thread 1 and 2 do not belong to the same block, __threadfence() must be used if they are CUDA threads from the same device and __threadfence_system() must be used if they are CUDA threads from two different devices.

+

A common use case is when threads consume some data produced by other threads as illustrated by the following code sample of a kernel that computes the sum of an array of N numbers in one call. Each block first sums a subset of the array and stores the result in global memory. When all blocks are done, the last block done reads each of these partial sums from global memory and sums them to obtain the final result. In order to determine which block is finished last, each block atomically increments a counter to signal that it is done with computing and storing its partial sum (see Atomic Functions about atomic functions). The last block is the one that receives the counter value equal to gridDim.x-1. If no fence is placed between storing the partial sum and incrementing the counter, the counter might increment before the partial sum is stored and therefore, might reach gridDim.x-1 and let the last block start reading partial sums before they have been actually updated in memory.

+

Memory fence functions only affect the ordering of memory operations by a thread; they do not, by themselves, ensure that these memory operations are visible to other threads (like __syncthreads() does for threads within a block; see Synchronization Functions). In the code sample below, the visibility of memory operations on the result variable is ensured by declaring it as volatile (see Volatile Qualifier).

+
+
__device__ unsigned int count = 0;
+__shared__ bool isLastBlockDone;
+__global__ void sum(const float* array, unsigned int N,
+                    volatile float* result)
+{
+    // Each block sums a subset of the input array.
+    float partialSum = calculatePartialSum(array, N);
+
+    if (threadIdx.x == 0) {
+
+        // Thread 0 of each block stores the partial sum
+        // to global memory. The compiler will use
+        // a store operation that bypasses the L1 cache
+        // since the "result" variable is declared as
+        // volatile. This ensures that the threads of
+        // the last block will read the correct partial
+        // sums computed by all other blocks.
+        result[blockIdx.x] = partialSum;
+
+        // Thread 0 makes sure that the incrementing
+        // of the "count" variable is only performed after
+        // the partial sum has been written to global memory.
+        __threadfence();
+
+        // Thread 0 signals that it is done.
+        unsigned int value = atomicInc(&count, gridDim.x);
+
+        // Thread 0 determines if its block is the last
+        // block to be done.
+        isLastBlockDone = (value == (gridDim.x - 1));
+    }
+
+    // Synchronize to make sure that each thread reads
+    // the correct value of isLastBlockDone.
+    __syncthreads();
+
+    if (isLastBlockDone) {
+
+        // The last block sums the partial sums
+        // stored in result[0 .. gridDim.x-1]
+        float totalSum = calculateTotalSum(result);
+
+        if (threadIdx.x == 0) {
+
+            // Thread 0 of last block stores the total sum
+            // to global memory and resets the count
+            // variable, so that the next kernel call
+            // works properly.
+            result[0] = totalSum;
+            count = 0;
+        }
+    }
+}
+
+
+
+
+

+10.6. Synchronization Functions +

+
+
void __syncthreads();
+
+
+

waits until all threads in the thread block have reached this point and all global and shared memory accesses made by these threads prior to __syncthreads() are visible to all threads in the block.

+

__syncthreads() is used to coordinate communication between the threads of the same block. When some threads within a block access the same addresses in shared or global memory, there are potential read-after-write, write-after-read, or write-after-write hazards for some of these memory accesses. These data hazards can be avoided by synchronizing threads in-between these accesses.

+

__syncthreads() is allowed in conditional code but only if the conditional evaluates identically across the entire thread block, otherwise the code execution is likely to hang or produce unintended side effects.

+

Devices of compute capability 2.x and higher support three variations of __syncthreads() described below.

+
+
int __syncthreads_count(int predicate);
+
+
+

is identical to __syncthreads() with the additional feature that it evaluates predicate for all threads of the block and returns the number of threads for which predicate evaluates to non-zero.

+
+
int __syncthreads_and(int predicate);
+
+
+

is identical to __syncthreads() with the additional feature that it evaluates predicate for all threads of the block and returns non-zero if and only if predicate evaluates to non-zero for all of them.

+
+
int __syncthreads_or(int predicate);
+
+
+

is identical to __syncthreads() with the additional feature that it evaluates predicate for all threads of the block and returns non-zero if and only if predicate evaluates to non-zero for any of them.

+
+
void __syncwarp(unsigned mask=0xffffffff);
+
+
+

will cause the executing thread to wait until all warp lanes named in mask have executed a __syncwarp() (with the same mask) before resuming execution. Each calling thread must have its own bit set in the mask and all non-exited threads named in mask must execute a corresponding __syncwarp() with the same mask, or the result is undefined.

+

Executing __syncwarp() guarantees memory ordering among threads participating in the barrier. Thus, threads within a warp that wish to communicate via memory can store to memory, execute __syncwarp(), and then safely read values stored by other threads in the warp.

+
+

Note

+

For .target sm_6x or below, all threads in mask must execute the same __syncwarp() in convergence, and the union of all values in mask must be equal to the active mask. Otherwise, the behavior is undefined.

+
+
+
+

+10.7. Mathematical Functions +

+

The reference manual lists all C/C++ standard library mathematical functions that are supported in device code and all intrinsic functions that are only supported in device code.

+

Mathematical Functions provides accuracy information for some of these functions when relevant.

+
+
+

+10.8. Texture Functions +

+

Texture objects are described in Texture Object API.

+

Texture fetching is described in Texture Fetching.

+
+

+10.8.1. Texture Object API +

+
+

+10.8.1.1. tex1Dfetch() +

+
+
template<class T>
+T tex1Dfetch(cudaTextureObject_t texObj, int x);
+
+
+

fetches from the region of linear memory specified by the one-dimensional texture object texObj using integer texture coordinate x. tex1Dfetch() only works with non-normalized coordinates, so only the border and clamp addressing modes are supported. It does not perform any texture filtering. For integer types, it may optionally promote the integer to single-precision floating point.

+
+
+

+10.8.1.2. tex1D() +

+
+
template<class T>
+T tex1D(cudaTextureObject_t texObj, float x);
+
+
+

fetches from the CUDA array specified by the one-dimensional texture object texObj using texture coordinate x.

+
+
+

+10.8.1.3. tex1DLod() +

+
+
template<class T>
+T tex1DLod(cudaTextureObject_t texObj, float x, float level);
+
+
+

fetches from the CUDA array specified by the one-dimensional texture object texObj using texture coordinate x at the level-of-detail level.

+
+
+

+10.8.1.4. tex1DGrad() +

+
+
template<class T>
+T tex1DGrad(cudaTextureObject_t texObj, float x, float dx, float dy);
+
+
+

fetches from the CUDA array specified by the one-dimensional texture object texObj using texture coordinate x. The level-of-detail is derived from the X-gradient dx and Y-gradient dy.

+
+
+

+10.8.1.5. tex2D() +

+
+
template<class T>
+T tex2D(cudaTextureObject_t texObj, float x, float y);
+
+
+

fetches from the CUDA array or the region of linear memory specified by the two-dimensional texture object texObj using texture coordinate (x,y).

+
+
+

+10.8.1.6. tex2D() for sparse CUDA arrays +

+
+
                template<class T>
+T tex2D(cudaTextureObject_t texObj, float x, float y, bool* isResident);
+
+
+

fetches from the CUDA array specified by the two-dimensional texture object texObj using texture coordinate (x,y). Also returns whether the texel is resident in memory via isResident pointer. If not, the values fetched will be zeros.

+
+
+

+10.8.1.7. tex2Dgather() +

+
+
template<class T>
+T tex2Dgather(cudaTextureObject_t texObj,
+              float x, float y, int comp = 0);
+
+
+

fetches from the CUDA array specified by the 2D texture object texObj using texture coordinates x and y and the comp parameter as described in Texture Gather.

+
+
+

+10.8.1.8. tex2Dgather() for sparse CUDA arrays +

+
+
                template<class T>
+T tex2Dgather(cudaTextureObject_t texObj,
+            float x, float y, bool* isResident, int comp = 0);
+
+
+

fetches from the CUDA array specified by the 2D texture object texObj using texture coordinates x and y and the comp parameter as described in Texture Gather. Also returns whether the texel is resident in memory via isResident pointer. If not, the values fetched will be zeros.

+
+
+

+10.8.1.9. tex2DGrad() +

+
+
template<class T>
+T tex2DGrad(cudaTextureObject_t texObj, float x, float y,
+            float2 dx, float2 dy);
+
+
+

fetches from the CUDA array specified by the two-dimensional texture object texObj using texture coordinate (x,y). The level-of-detail is derived from the dx and dy gradients.

+
+
+

+10.8.1.10. tex2DGrad() for sparse CUDA arrays +

+
+
                template<class T>
+T tex2DGrad(cudaTextureObject_t texObj, float x, float y,
+        float2 dx, float2 dy, bool* isResident);
+
+
+

fetches from the CUDA array specified by the two-dimensional texture object texObj using texture coordinate (x,y). The level-of-detail is derived from the dx and dy gradients. Also returns whether the texel is resident in memory via isResident pointer. If not, the values fetched will be zeros.

+
+
+

+10.8.1.11. tex2DLod() +

+
+
template<class T>
+tex2DLod(cudaTextureObject_t texObj, float x, float y, float level);
+
+
+

fetches from the CUDA array or the region of linear memory specified by the two-dimensional texture object texObj using texture coordinate (x,y) at level-of-detail level.

+
+
+

+10.8.1.12. tex2DLod() for sparse CUDA arrays +

+
+
        template<class T>
+tex2DLod(cudaTextureObject_t texObj, float x, float y, float level, bool* isResident);
+
+
+

fetches from the CUDA array specified by the two-dimensional texture object texObj using texture coordinate (x,y) at level-of-detail level. Also returns whether the texel is resident in memory via isResident pointer. If not, the values fetched will be zeros.

+
+
+

+10.8.1.13. tex3D() +

+
+
template<class T>
+T tex3D(cudaTextureObject_t texObj, float x, float y, float z);
+
+
+

fetches from the CUDA array specified by the three-dimensional texture object texObj using texture coordinate (x,y,z).

+
+
+

+10.8.1.14. tex3D() for sparse CUDA arrays +

+
+
                template<class T>
+T tex3D(cudaTextureObject_t texObj, float x, float y, float z, bool* isResident);
+
+
+

fetches from the CUDA array specified by the three-dimensional texture object texObj using texture coordinate (x,y,z). Also returns whether the texel is resident in memory via isResident pointer. If not, the values fetched will be zeros.

+
+
+

+10.8.1.15. tex3DLod() +

+
+
template<class T>
+T tex3DLod(cudaTextureObject_t texObj, float x, float y, float z, float level);
+
+
+

fetches from the CUDA array or the region of linear memory specified by the three-dimensional texture object texObj using texture coordinate (x,y,z) at level-of-detail level.

+
+
+

+10.8.1.16. tex3DLod() for sparse CUDA arrays +

+
+
                template<class T>
+T tex3DLod(cudaTextureObject_t texObj, float x, float y, float z, float level, bool* isResident);
+
+
+

fetches from the CUDA array or the region of linear memory specified by the three-dimensional texture object texObj using texture coordinate (x,y,z) at level-of-detail level. Also returns whether the texel is resident in memory via isResident pointer. If not, the values fetched will be zeros.

+
+
+

+10.8.1.17. tex3DGrad() +

+
+
template<class T>
+T tex3DGrad(cudaTextureObject_t texObj, float x, float y, float z,
+            float4 dx, float4 dy);
+
+
+

fetches from the CUDA array specified by the three-dimensional texture object texObj using texture coordinate (x,y,z) at a level-of-detail derived from the X and Y gradients dx and dy.

+
+
+

+10.8.1.18. tex3DGrad() for sparse CUDA arrays +

+
+
                template<class T>
+T tex3DGrad(cudaTextureObject_t texObj, float x, float y, float z,
+        float4 dx, float4 dy, bool* isResident);
+
+
+

fetches from the CUDA array specified by the three-dimensional texture object texObj using texture coordinate (x,y,z) at a level-of-detail derived from the X and Y gradients dx and dy. Also returns whether the texel is resident in memory via isResident pointer. If not, the values fetched will be zeros.

+
+
+

+10.8.1.19. tex1DLayered() +

+
+
template<class T>
+T tex1DLayered(cudaTextureObject_t texObj, float x, int layer);
+
+
+

fetches from the CUDA array specified by the one-dimensional texture object texObj using texture coordinate x and index layer, as described in Layered Textures.

+
+
+

+10.8.1.20. tex1DLayeredLod() +

+
+
template<class T>
+T tex1DLayeredLod(cudaTextureObject_t texObj, float x, int layer, float level);
+
+
+

fetches from the CUDA array specified by the one-dimensional Layered Textures at layer layer using texture coordinate x and level-of-detail level.

+
+
+

+10.8.1.21. tex1DLayeredGrad() +

+
+
template<class T>
+T tex1DLayeredGrad(cudaTextureObject_t texObj, float x, int layer,
+                   float dx, float dy);
+
+
+

fetches from the CUDA array specified by the one-dimensional layered texture at layer layer using texture coordinate x and a level-of-detail derived from the dx and dy gradients.

+
+
+

+10.8.1.22. tex2DLayered() +

+
+
template<class T>
+T tex2DLayered(cudaTextureObject_t texObj,
+               float x, float y, int layer);
+
+
+

fetches from the CUDA array specified by the two-dimensional texture object texObj using texture coordinate (x,y) and index layer, as described in Layered Textures.

+
+
+

+10.8.1.23. tex2DLayered() for Sparse CUDA Arrays +

+
+
                template<class T>
+T tex2DLayered(cudaTextureObject_t texObj,
+            float x, float y, int layer, bool* isResident);
+
+
+

fetches from the CUDA array specified by the two-dimensional texture object texObj using texture coordinate (x,y) and index layer, as described in Layered Textures. Also returns whether the texel is resident in memory via isResident pointer. If not, the values fetched will be zeros.

+
+
+

+10.8.1.24. tex2DLayeredLod() +

+
+
template<class T>
+T tex2DLayeredLod(cudaTextureObject_t texObj, float x, float y, int layer,
+                  float level);
+
+
+

fetches from the CUDA array specified by the two-dimensional layered texture at layer layer using texture coordinate (x,y).

+
+
+

+10.8.1.25. tex2DLayeredLod() for sparse CUDA arrays +

+
+
                template<class T>
+T tex2DLayeredLod(cudaTextureObject_t texObj, float x, float y, int layer,
+                float level, bool* isResident);
+
+
+

fetches from the CUDA array specified by the two-dimensional layered texture at layer layer using texture coordinate (x,y). Also returns whether the texel is resident in memory via isResident pointer. If not, the values fetched will be zeros.

+
+
+

+10.8.1.26. tex2DLayeredGrad() +

+
+
template<class T>
+T tex2DLayeredGrad(cudaTextureObject_t texObj, float x, float y, int layer,
+                   float2 dx, float2 dy);
+
+
+

fetches from the CUDA array specified by the two-dimensional layered texture at layer layer using texture coordinate (x,y) and a level-of-detail derived from the dx and dy gradients.

+
+
+

+10.8.1.27. tex2DLayeredGrad() for sparse CUDA arrays +

+
+
                template<class T>
+T tex2DLayeredGrad(cudaTextureObject_t texObj, float x, float y, int layer,
+                float2 dx, float2 dy, bool* isResident);
+
+
+

fetches from the CUDA array specified by the two-dimensional layered texture at layer layer using texture coordinate (x,y) and a level-of-detail derived from the dx and dy gradients. Also returns whether the texel is resident in memory via isResident pointer. If not, the values fetched will be zeros.

+
+
+

+10.8.1.28. texCubemap() +

+
+
template<class T>
+T texCubemap(cudaTextureObject_t texObj, float x, float y, float z);
+
+
+

fetches the CUDA array specified by the cubemap texture object texObj using texture coordinate (x,y,z), as described in Cubemap Textures.

+
+
+

+10.8.1.29. texCubemapGrad() +

+
+
template<class T>
+T texCubemapGrad(cudaTextureObject_t texObj, float x, float, y, float z,
+                float4 dx, float4 dy);
+
+
+

fetches from the CUDA array specified by the cubemap texture object texObj using texture coordinate (x,y,z) as described in Cubemap Textures. The level-of-detail used is derived from the dx and dy gradients.

+
+
+

+10.8.1.30. texCubemapLod() +

+
+
template<class T>
+T texCubemapLod(cudaTextureObject_t texObj, float x, float, y, float z,
+                float level);
+
+
+

fetches from the CUDA array specified by the cubemap texture object texObj using texture coordinate (x,y,z) as described in Cubemap Textures. The level-of-detail used is given by level.

+
+
+

+10.8.1.31. texCubemapLayered() +

+
+
template<class T>
+T texCubemapLayered(cudaTextureObject_t texObj,
+                    float x, float y, float z, int layer);
+
+
+

fetches from the CUDA array specified by the cubemap layered texture object texObj using texture coordinates (x,y,z), and index layer, as described in Cubemap Layered Textures.

+
+
+

+10.8.1.32. texCubemapLayeredGrad() +

+
+
template<class T>
+T texCubemapLayeredGrad(cudaTextureObject_t texObj, float x, float y, float z,
+                       int layer, float4 dx, float4 dy);
+
+
+

fetches from the CUDA array specified by the cubemap layered texture object texObj using texture coordinate (x,y,z) and index layer, as described in Cubemap Layered Textures, at level-of-detail derived from the dx and dy gradients.

+
+
+

+10.8.1.33. texCubemapLayeredLod() +

+
+
template<class T>
+T texCubemapLayeredLod(cudaTextureObject_t texObj, float x, float y, float z,
+                       int layer, float level);
+
+
+

fetches from the CUDA array specified by the cubemap layered texture object texObj using texture coordinate (x,y,z) and index layer, as described in Cubemap Layered Textures, at level-of-detail level level.

+
+
+
+
+

+10.9. Surface Functions +

+

Surface functions are only supported by devices of compute capability 2.0 and higher.

+

Surface objects are described in described in Surface Object API.

+

In the sections below, boundaryMode specifies the boundary mode, that is how out-of-range surface coordinates are handled; it is equal to either cudaBoundaryModeClamp, in which case out-of-range coordinates are clamped to the valid range, or cudaBoundaryModeZero, in which case out-of-range reads return zero and out-of-range writes are ignored, or cudaBoundaryModeTrap, in which case out-of-range accesses cause the kernel execution to fail.

+
+

+10.9.1. Surface Object API +

+
+

+10.9.1.1. surf1Dread() +

+
+
template<class T>
+T surf1Dread(cudaSurfaceObject_t surfObj, int x,
+               boundaryMode = cudaBoundaryModeTrap);
+
+
+

reads the CUDA array specified by the one-dimensional surface object surfObj using byte coordinate x.

+
+
+

+10.9.1.2. surf1Dwrite +

+
+
template<class T>
+void surf1Dwrite(T data,
+                  cudaSurfaceObject_t surfObj,
+                  int x,
+                  boundaryMode = cudaBoundaryModeTrap);
+
+
+

writes value data to the CUDA array specified by the one-dimensional surface object surfObj at byte coordinate x.

+
+
+

+10.9.1.3. surf2Dread() +

+
+
template<class T>
+T surf2Dread(cudaSurfaceObject_t surfObj,
+              int x, int y,
+              boundaryMode = cudaBoundaryModeTrap);
+template<class T>
+void surf2Dread(T* data,
+                 cudaSurfaceObject_t surfObj,
+                 int x, int y,
+                 boundaryMode = cudaBoundaryModeTrap);
+
+
+

reads the CUDA array specified by the two-dimensional surface object surfObj using byte coordinates x and y.

+
+
+

+10.9.1.4. surf2Dwrite() +

+
+
template<class T>
+void surf2Dwrite(T data,
+                  cudaSurfaceObject_t surfObj,
+                  int x, int y,
+                  boundaryMode = cudaBoundaryModeTrap);
+
+
+

writes value data to the CUDA array specified by the two-dimensional surface object surfObj at byte coordinate x and y.

+
+
+

+10.9.1.5. surf3Dread() +

+
+
template<class T>
+T surf3Dread(cudaSurfaceObject_t surfObj,
+              int x, int y, int z,
+              boundaryMode = cudaBoundaryModeTrap);
+template<class T>
+void surf3Dread(T* data,
+                 cudaSurfaceObject_t surfObj,
+                 int x, int y, int z,
+                 boundaryMode = cudaBoundaryModeTrap);
+
+
+

reads the CUDA array specified by the three-dimensional surface object surfObj using byte coordinates x, y, and z.

+
+
+

+10.9.1.6. surf3Dwrite() +

+
+
template<class T>
+void surf3Dwrite(T data,
+                  cudaSurfaceObject_t surfObj,
+                  int x, int y, int z,
+                  boundaryMode = cudaBoundaryModeTrap);
+
+
+

writes value data to the CUDA array specified by the three-dimensional object surfObj at byte coordinate x, y, and z.

+
+
+

+10.9.1.7. surf1DLayeredread() +

+
+
template<class T>
+T surf1DLayeredread(
+                 cudaSurfaceObject_t surfObj,
+                 int x, int layer,
+                 boundaryMode = cudaBoundaryModeTrap);
+template<class T>
+void surf1DLayeredread(T data,
+                 cudaSurfaceObject_t surfObj,
+                 int x, int layer,
+                 boundaryMode = cudaBoundaryModeTrap);
+
+
+

reads the CUDA array specified by the one-dimensional layered surface object surfObj using byte coordinate x and index layer.

+
+
+

+10.9.1.8. surf1DLayeredwrite() +

+
+
template<class Type>
+void surf1DLayeredwrite(T data,
+                 cudaSurfaceObject_t surfObj,
+                 int x, int layer,
+                 boundaryMode = cudaBoundaryModeTrap);
+
+
+

writes value data to the CUDA array specified by the two-dimensional layered surface object surfObj at byte coordinate x and index layer.

+
+
+

+10.9.1.9. surf2DLayeredread() +

+
+
template<class T>
+T surf2DLayeredread(
+                 cudaSurfaceObject_t surfObj,
+                 int x, int y, int layer,
+                 boundaryMode = cudaBoundaryModeTrap);
+template<class T>
+void surf2DLayeredread(T data,
+                         cudaSurfaceObject_t surfObj,
+                         int x, int y, int layer,
+                         boundaryMode = cudaBoundaryModeTrap);
+
+
+

reads the CUDA array specified by the two-dimensional layered surface object surfObj using byte coordinate x and y, and index layer.

+
+
+

+10.9.1.10. surf2DLayeredwrite() +

+
+
template<class T>
+void surf2DLayeredwrite(T data,
+                          cudaSurfaceObject_t surfObj,
+                          int x, int y, int layer,
+                          boundaryMode = cudaBoundaryModeTrap);
+
+
+

writes value data to the CUDA array specified by the one-dimensional layered surface object surfObj at byte coordinate x and y, and index layer.

+
+
+

+10.9.1.11. surfCubemapread() +

+
+
template<class T>
+T surfCubemapread(
+                 cudaSurfaceObject_t surfObj,
+                 int x, int y, int face,
+                 boundaryMode = cudaBoundaryModeTrap);
+template<class T>
+void surfCubemapread(T data,
+                 cudaSurfaceObject_t surfObj,
+                 int x, int y, int face,
+                 boundaryMode = cudaBoundaryModeTrap);
+
+
+

reads the CUDA array specified by the cubemap surface object surfObj using byte coordinate x and y, and face index face.

+
+
+

+10.9.1.12. surfCubemapwrite() +

+
+
template<class T>
+void surfCubemapwrite(T data,
+                 cudaSurfaceObject_t surfObj,
+                 int x, int y, int face,
+                 boundaryMode = cudaBoundaryModeTrap);
+
+
+

writes value data to the CUDA array specified by the cubemap object surfObj at byte coordinate x and y, and face index face.

+
+
+

+10.9.1.13. surfCubemapLayeredread() +

+
+
template<class T>
+T surfCubemapLayeredread(
+             cudaSurfaceObject_t surfObj,
+             int x, int y, int layerFace,
+             boundaryMode = cudaBoundaryModeTrap);
+template<class T>
+void surfCubemapLayeredread(T data,
+             cudaSurfaceObject_t surfObj,
+             int x, int y, int layerFace,
+             boundaryMode = cudaBoundaryModeTrap);
+
+
+

reads the CUDA array specified by the cubemap layered surface object surfObj using byte coordinate x and y, and index layerFace.

+
+
+

+10.9.1.14. surfCubemapLayeredwrite() +

+
+
template<class T>
+void surfCubemapLayeredwrite(T data,
+             cudaSurfaceObject_t surfObj,
+             int x, int y, int layerFace,
+             boundaryMode = cudaBoundaryModeTrap);
+
+
+

writes value data to the CUDA array specified by the cubemap layered object surfObj at byte coordinate x and y, and index layerFace.

+
+
+
+
+

+10.10. Read-Only Data Cache Load Function +

+

The read-only data cache load function is only supported by devices of compute capability 5.0 and higher.

+
+
T __ldg(const T* address);
+
+
+

returns the data of type T located at address address, where T is char, signed char, short, int, long, long longunsigned char, unsigned short, unsigned int, unsigned long, unsigned long long, char2, char4, short2, short4, int2, int4, longlong2uchar2, uchar4, ushort2, ushort4, uint2, uint4, ulonglong2float, float2, float4, double, or double2. With the cuda_fp16.h header included, T can be __half or __half2. Similarly, with the cuda_bf16.h header included, T can also be __nv_bfloat16 or __nv_bfloat162. The operation is cached in the read-only data cache (see Global Memory).

+
+
+

+10.11. Load Functions Using Cache Hints +

+

These load functions are only supported by devices of compute capability 5.0 and higher.

+
+
T __ldcg(const T* address);
+T __ldca(const T* address);
+T __ldcs(const T* address);
+T __ldlu(const T* address);
+T __ldcv(const T* address);
+
+
+

returns the data of type T located at address address, where T is char, signed char, short, int, long, long longunsigned char, unsigned short, unsigned int, unsigned long, unsigned long long, char2, char4, short2, short4, int2, int4, longlong2uchar2, uchar4, ushort2, ushort4, uint2, uint4, ulonglong2float, float2, float4, double, or double2. With the cuda_fp16.h header included, T can be __half or __half2. Similarly, with the cuda_bf16.h header included, T can also be __nv_bfloat16 or __nv_bfloat162. The operation is using the corresponding cache operator (see PTX ISA)

+
+
+

+10.12. Store Functions Using Cache Hints +

+

These store functions are only supported by devices of compute capability 5.0 and higher.

+
+
void __stwb(T* address, T value);
+void __stcg(T* address, T value);
+void __stcs(T* address, T value);
+void __stwt(T* address, T value);
+
+
+

stores the value argument of type T to the location at address address, where T is char, signed char, short, int, long, long longunsigned char, unsigned short, unsigned int, unsigned long, unsigned long long, char2, char4, short2, short4, int2, int4, longlong2uchar2, uchar4, ushort2, ushort4, uint2, uint4, ulonglong2float, float2, float4, double, or double2. With the cuda_fp16.h header included, T can be __half or __half2. Similarly, with the cuda_bf16.h header included, T can also be __nv_bfloat16 or __nv_bfloat162. The operation is using the corresponding cache operator (see PTX ISA )

+
+
+

+10.13. Time Function +

+
+
clock_t clock();
+long long int clock64();
+
+
+

when executed in device code, returns the value of a per-multiprocessor counter that is incremented every clock cycle. Sampling this counter at the beginning and at the end of a kernel, taking the difference of the two samples, and recording the result per thread provides a measure for each thread of the number of clock cycles taken by the device to completely execute the thread, but not of the number of clock cycles the device actually spent executing thread instructions. The former number is greater than the latter since threads are time sliced.

+
+
+

+10.14. Atomic Functions +

+

An atomic function performs a read-modify-write atomic operation on one 32-bit, 64-bit, or 128-bit word residing in global or shared memory. In the case of float2 or float4, the read-modify-write operation is performed on each element of the vector residing in global memory. For example, atomicAdd() reads a word at some address in global or shared memory, adds a number to it, and writes the result back to the same address. Atomic functions can only be used in device functions.

+

The atomic functions described in this section have ordering cuda::memory_order_relaxed and are only atomic at a particular scope:

+
    +
  • Atomic APIs with _system suffix (example: atomicAdd_system) are atomic at scope cuda::thread_scope_system if they meet particular conditions.

  • +
  • Atomic APIs without a suffix (example: atomicAdd) are atomic at scope cuda::thread_scope_device.

  • +
  • Atomic APIs with _block suffix (example: atomicAdd_block) are atomic at scope cuda::thread_scope_block.

  • +
+

In the following example both the CPU and the GPU atomically update an integer value at address addr:

+
+
__global__ void mykernel(int *addr) {
+  atomicAdd_system(addr, 10);       // only available on devices with compute capability 6.x
+}
+
+void foo() {
+  int *addr;
+  cudaMallocManaged(&addr, 4);
+  *addr = 0;
+
+   mykernel<<<...>>>(addr);
+   __sync_fetch_and_add(addr, 10);  // CPU atomic operation
+}
+
+
+

Note that any atomic operation can be implemented based on atomicCAS() (Compare And Swap). For example, atomicAdd() for double-precision floating-point numbers is not available on devices with compute capability lower than 6.0 but it can be implemented as follows:

+
+
#if __CUDA_ARCH__ < 600
+__device__ double atomicAdd(double* address, double val)
+{
+    unsigned long long int* address_as_ull =
+                              (unsigned long long int*)address;
+    unsigned long long int old = *address_as_ull, assumed;
+
+    do {
+        assumed = old;
+        old = atomicCAS(address_as_ull, assumed,
+                        __double_as_longlong(val +
+                               __longlong_as_double(assumed)));
+
+    // Note: uses integer comparison to avoid hang in case of NaN (since NaN != NaN)
+    } while (assumed != old);
+
+    return __longlong_as_double(old);
+}
+#endif
+
+
+

There are system-wide and block-wide variants of the following device-wide atomic APIs, with the following exceptions:

+
    +
  • Devices with compute capability less than 6.0 only support device-wide atomic operations,

  • +
  • Tegra devices with compute capability less than 7.2 do not support system-wide atomic operations.

  • +
+

CUDA 12.8 and later support CUDA compiler builtin functions for atomic operations with memory order and thread scope. We follows the GNU’s atomic built-in function signature with an extra argument of thread scope. +We use the following atomic operation memory orders and thread scopes:

+
+
enum {
+   __NV_ATOMIC_RELAXED,
+   __NV_ATOMIC_CONSUME,
+   __NV_ATOMIC_ACQUIRE,
+   __NV_ATOMIC_RELEASE,
+   __NV_ATOMIC_ACQ_REL,
+   __NV_ATOMIC_SEQ_CST
+};
+
+enum {
+   __NV_THREAD_SCOPE_THREAD,
+   __NV_THREAD_SCOPE_BLOCK,
+   __NV_THREAD_SCOPE_CLUSTER,
+   __NV_THREAD_SCOPE_DEVICE,
+   __NV_THREAD_SCOPE_SYSTEM
+};
+
+
+

Example:

+
+
__device__ T __nv_atomic_load_n(T* ptr, int order, int scope = __NV_THREAD_SCOPE_SYSTEM);
+
+
+

T can be any integral type that is size of 1, 2, 4, 8 and 16 bytes.

+

These atomic functions cannot operate on local memory. For example:

+
+
__device__ void foo() {
+   int a = 1; // defined in local memory
+   int b;
+   __nv_atomic_load(&a, &b, __NV_ATOMIC_RELAXED, __NV_THREAD_SCOPE_SYSTEM);
+}
+
+
+

These functions must only be used within the block scope of a __device__ function. For example:

+
+
__device__ void foo() {
+   __shared__ unsigned int u1 = 1;
+   __shared__ unsigned int u2 = 2;
+   __nv_atomic_load(&u1, &u2, __NV_ATOMIC_RELAXED, __NV_THREAD_SCOPE_SYSTEM);
+}
+
+
+

And these functions’ address cannot be taken. Here are three unsupported examples:

+
+
// Not permitted to be used in a host function
+__host__ void bar() {
+   __shared__ unsigned int u1 = 1;
+   __shared__ unsigned int u2 = 2;
+   __nv_atomic_load(&u1, &u2, __NV_ATOMIC_RELAXED, __NV_THREAD_SCOPE_SYSTEM);
+}
+
+// Not permitted to be used as a template default argument.
+// The function address cannot be taken.
+template<void *F = __nv_atomic_load_n>
+class X {
+   void *f = F;
+};
+
+// Not permitted to be called in a constructor initialization list.
+class Y {
+   int a;
+public:
+   __device__ Y(int *b): a(__nv_atomic_load_n(b, __NV_ATOMIC_RELAXED)) {}
+};
+
+
+

The memory order corresponds to C++ standard atomic operation’s memory order. And for thread scope, we follows cuda::thread_scope’s definition.

+

__NV_ATOMIC_CONSUME memory order is currently implemented using stronger __NV_ATOMIC_ACQUIRE memory order.

+

__NV_THREAD_SCOPE_THREAD thread scope is currently implemented using wider __NV_THREAD_SCOPE_BLOCK thread scope.

+

For the supported data types, please refer to the corresponding section of different atomic operations.

+
+

+10.14.1. Arithmetic Functions +

+
+

+10.14.1.1. atomicAdd() +

+
+
int atomicAdd(int* address, int val);
+unsigned int atomicAdd(unsigned int* address,
+                       unsigned int val);
+unsigned long long int atomicAdd(unsigned long long int* address,
+                                 unsigned long long int val);
+float atomicAdd(float* address, float val);
+double atomicAdd(double* address, double val);
+__half2 atomicAdd(__half2 *address, __half2 val);
+__half atomicAdd(__half *address, __half val);
+__nv_bfloat162 atomicAdd(__nv_bfloat162 *address, __nv_bfloat162 val);
+__nv_bfloat16 atomicAdd(__nv_bfloat16 *address, __nv_bfloat16 val);
+float2 atomicAdd(float2* address, float2 val);
+float4 atomicAdd(float4* address, float4 val);
+
+
+

reads the 16-bit, 32-bit or 64-bit old located at the address address in global or shared memory, computes (old + val), and stores the result back to memory at the same address. These three operations are performed in one atomic transaction. The function returns old.

+

The 32-bit floating-point version of atomicAdd() is only supported by devices of compute capability 2.x and higher.

+

The 64-bit floating-point version of atomicAdd() is only supported by devices of compute capability 6.x and higher.

+

The 32-bit __half2 floating-point version of atomicAdd() is only supported by devices of compute capability 6.x and higher. The atomicity of the __half2 or __nv_bfloat162 add operation is guaranteed separately for each of the two __half or __nv_bfloat16 elements; the entire __half2 or __nv_bfloat162 is not guaranteed to be atomic as a single 32-bit access.

+

The float2 and float4 floating-point vector versions of atomicAdd() are only supported by devices of compute capability 9.x and higher. The atomicity of the float2 or float4 add operation is guaranteed separately for each of the two or four float elements; the entire float2 or float4 is not guaranteed to be atomic as a single 64-bit or 128-bit access.

+

The 16-bit __half floating-point version of atomicAdd() is only supported by devices of compute capability 7.x and higher.

+

The 16-bit __nv_bfloat16 floating-point version of atomicAdd() is only supported by devices of compute capability 8.x and higher.

+

The float2 and float4 floating-point vector versions of atomicAdd() are only supported by devices of compute capability 9.x and higher.

+

The float2 and float4 floating-point vector versions of atomicAdd() are only supported for global memory addresses.

+
+
+

+10.14.1.2. atomicSub() +

+
+
int atomicSub(int* address, int val);
+unsigned int atomicSub(unsigned int* address,
+                       unsigned int val);
+
+
+

reads the 32-bit word old located at the address address in global or shared memory, computes (old - val), and stores the result back to memory at the same address. These three operations are performed in one atomic transaction. The function returns old.

+
+
+

+10.14.1.3. atomicExch() +

+
+
int atomicExch(int* address, int val);
+unsigned int atomicExch(unsigned int* address,
+                        unsigned int val);
+unsigned long long int atomicExch(unsigned long long int* address,
+                                  unsigned long long int val);
+float atomicExch(float* address, float val);
+
+
+

reads the 32-bit or 64-bit word old located at the address address in global or shared memory and stores val back to memory at the same address. These two operations are performed in one atomic transaction. The function returns old.

+
+
template<typename T> T atomicExch(T* address, T val);
+
+
+

reads the 128-bit word old located at the address address in global or shared memory and stores val back to memory at the same address. These two operations are performed in one atomic transaction. The function returns old. The type T must meet the following requirements:

+
+
sizeof(T) == 16
+alignof(T) >= 16
+std::is_trivially_copyable<T>::value == true
+// for C++03 and older
+std::is_default_constructible<T>::value == true
+
+
+

So, T must be 128-bit and properly aligned, be trivially copyable, and on C++03 or older, it must also be default constructible.

+

The 128-bit atomicExch() is only supported by devices of compute capability 9.x and higher.

+
+
+

+10.14.1.4. atomicMin() +

+
+
int atomicMin(int* address, int val);
+unsigned int atomicMin(unsigned int* address,
+                       unsigned int val);
+unsigned long long int atomicMin(unsigned long long int* address,
+                                 unsigned long long int val);
+long long int atomicMin(long long int* address,
+                                long long int val);
+
+
+

reads the 32-bit or 64-bit word old located at the address address in global or shared memory, computes the minimum of old and val, and stores the result back to memory at the same address. These three operations are performed in one atomic transaction. The function returns old.

+

The 64-bit version of atomicMin() is only supported by devices of compute capability 5.0 and higher.

+
+
+

+10.14.1.5. atomicMax() +

+
+
int atomicMax(int* address, int val);
+unsigned int atomicMax(unsigned int* address,
+                       unsigned int val);
+unsigned long long int atomicMax(unsigned long long int* address,
+                                 unsigned long long int val);
+long long int atomicMax(long long int* address,
+                                 long long int val);
+
+
+

reads the 32-bit or 64-bit word old located at the address address in global or shared memory, computes the maximum of old and val, and stores the result back to memory at the same address. These three operations are performed in one atomic transaction. The function returns old.

+

The 64-bit version of atomicMax() is only supported by devices of compute capability 5.0 and higher.

+
+
+

+10.14.1.6. atomicInc() +

+
+
unsigned int atomicInc(unsigned int* address,
+                       unsigned int val);
+
+
+

reads the 32-bit word old located at the address address in global or shared memory, computes ((old >= val) ? 0 : (old+1)), and stores the result back to memory at the same address. These three operations are performed in one atomic transaction. The function returns old.

+
+
+

+10.14.1.7. atomicDec() +

+
+
unsigned int atomicDec(unsigned int* address,
+                       unsigned int val);
+
+
+

reads the 32-bit word old located at the address address in global or shared memory, computes (((old == 0) || (old > val)) ? val : (old-1) ), and stores the result back to memory at the same address. These three operations are performed in one atomic transaction. The function returns old.

+
+
+

+10.14.1.8. atomicCAS() +

+
+
int atomicCAS(int* address, int compare, int val);
+unsigned int atomicCAS(unsigned int* address,
+                       unsigned int compare,
+                       unsigned int val);
+unsigned long long int atomicCAS(unsigned long long int* address,
+                                 unsigned long long int compare,
+                                 unsigned long long int val);
+unsigned short int atomicCAS(unsigned short int *address,
+                             unsigned short int compare,
+                             unsigned short int val);
+
+
+

reads the 16-bit, 32-bit or 64-bit word old located at the address address in global or shared memory, computes (old == compare ? val : old), and stores the result back to memory at the same address. These three operations are performed in one atomic transaction. The function returns old (Compare And Swap).

+
+
template<typename T> T atomicCAS(T* address, T compare, T val);
+
+
+

reads the 128-bit word old located at the address address in global or shared memory, computes (old == compare ? val : old), and stores the result back to memory at the same address. These three operations are performed in one atomic transaction. The function returns old (Compare And Swap). The type T must meet the following requirements:

+
+
sizeof(T) == 16
+alignof(T) >= 16
+std::is_trivially_copyable<T>::value == true
+// for C++03 and older
+std::is_default_constructible<T>::value == true
+
+
+

So, T must be 128-bit and properly aligned, be trivially copyable, and on C++03 or older, it must also be default constructible.

+

The 128-bit atomicCAS() is only supported by devices of compute capability 9.x and higher.

+
+
+

+10.14.1.9. __nv_atomic_exchange() +

+
+
__device__ void __nv_atomic_exchange(T* ptr, T* val, T *ret, int order, int scope = __NV_THREAD_SCOPE_SYSTEM);
+
+
+

This atomic function is introduced in CUDA 12.8. It reads the value where ptr points to and stores the value to where ret points to. And it reads the value where val points to and stores the value to where ptr points to.

+

This is a generic atomic exchange, which means that T can be any data type that is size of 4, 8 or 16 bytes.

+

The atomic operation with memory order and thread scope is supported on the architecture sm_60 and higher.

+

16-byte data type is supported on the architecture sm_90 and higher.

+

The thread scope of cluster is supported on the architecture sm_90 and higher.

+

The arguments order and scope need to be integer literals, i.e., the arguments cannot be variables.

+
+
+

+10.14.1.10. __nv_atomic_exchange_n() +

+
+
__device__ T __nv_atomic_exchange_n(T* ptr, T val, int order, int scope = __NV_THREAD_SCOPE_SYSTEM);
+
+
+

This atomic function is introduced in CUDA 12.8. It reads the value where ptr points to and use this value as the return value. And it stores val to where ptr points to.

+

This is a non-generic atomic exchange, which means that T can only be an integral type that is size of 4, 8 or 16 bytes.

+

The atomic operation with memory order and thread scope is supported on the architecture sm_60 and higher.

+

16-byte data type is supported on the architecture sm_90 and higher.

+

The thread scope of cluster is supported on the architecture sm_90 and higher.

+

The arguments order and scope need to be integer literals, i.e., the arguments cannot be variables.

+
+
+

+10.14.1.11. __nv_atomic_compare_exchange() +

+
+
__device__ bool __nv_atomic_compare_exchange (T* ptr, T* expected, T* desired, bool weak, int success_order, int failure_order, int scope = __NV_THREAD_SCOPE_SYSTEM);
+
+
+

This atomic function is introduced in CUDA 12.8. It reads the value where ptr points to and compare it with the value where expected points to. If they are equal, the return value is true and the value where desired points to is stored to where ptr points to. Otherwise, it returns false and the value where ptr points to is stored to where expected points to. The parameter weak is ignored and it picks the stronger memory order between success_order and failure_order to execute the compare-and-exchange operation.

+

This is a generic atomic compare-and-exchange, which means that T can be any data type that is size of 2, 4, 8 or 16 bytes.

+

The atomic operation with memory order and thread scope is supported on the architecture sm_60 and higher.

+

16-byte data type is supported on the architecture sm_90 and higher.

+

2-byte data type is supported on the architecture sm_70 and higher.

+

The thread scope of cluster is supported on the architecture sm_90 and higher.

+

The arguments order and scope need to be integer literals, i.e., the arguments cannot be variables.

+
+
+

+10.14.1.12. __nv_atomic_compare_exchange_n() +

+
+
__device__ bool __nv_atomic_compare_exchange_n (T* ptr, T* expected, T desired, bool weak, int success_order, int failure_order, int scope = __NV_THREAD_SCOPE_SYSTEM);
+
+
+

This atomic function is introduced in CUDA 12.8. It reads the value where ptr points to and compare it with the value where expected points to. If they are equal, the return value is true and desired is stored to where ptr points to. Otherwise, it returns false and the value where ptr points to is stored to where expected points to. The parameter weak is ignored and it picks the stronger memory order between success_order and failure_order to execute the compare-and-exchange operation.

+

This is a non-generic atomic compare-and-exchange, which means that T can only be an integral type that is size of 2, 4, 8 or 16 bytes.

+

The atomic operation with memory order and thread scope is supported on the architecture sm_60 and higher.

+

16-byte data type is supported on the architecture sm_90 and higher.

+

2-byte data type is supported on the architecture sm_70 and higher.

+

The thread scope of cluster is supported on the architecture sm_90 and higher.

+

The arguments order and scope need to be integer literals, i.e., the arguments cannot be variables.

+
+
+

+10.14.1.13. __nv_atomic_fetch_add() and __nv_atomic_add() +

+
+
__device__ T __nv_atomic_fetch_add (T* ptr, T val, int order, int scope = __NV_THREAD_SCOPE_SYSTEM);
+__device__ void __nv_atomic_add (T* ptr, T val, int order, int scope = __NV_THREAD_SCOPE_SYSTEM);
+
+
+

These two atomic functions are introduced in CUDA 12.8. It reads the value where ptr points to, adds with val, and stores the result back to where ptr points to. __nv_atomic_fetch_add returns the old value where ptr points to. __nv_atomic_add does not have return value.

+

T can only be unsigned int, int, unsigned long long, float or double.

+

The atomic operation with memory order and thread scope is supported on the architecture sm_60 and higher.

+

The thread scope of cluster is supported on the architecture sm_90 and higher.

+

The arguments order and scope need to be integer literals, i.e., the arguments cannot be variables.

+
+
+

+10.14.1.14. __nv_atomic_fetch_sub() and __nv_atomic_sub() +

+
+
__device__ T __nv_atomic_fetch_sub (T* ptr, T val, int order, int scope = __NV_THREAD_SCOPE_SYSTEM);
+__device__ void __nv_atomic_sub (T* ptr, T val, int order, int scope = __NV_THREAD_SCOPE_SYSTEM);
+
+
+

These two atomic functions are introduced in CUDA 12.8. It reads the value where ptr points to, subtracts with val, and stores the result back to where ptr points to. __nv_atomic_fetch_sub returns the old value where ptr points to. __nv_atomic_sub does not have return value.

+

T can only be unsigned int, int, unsigned long long, float or double.

+

The atomic operation with memory order and thread scope is supported on the architecture sm_60 and higher.

+

The thread scope of cluster is supported on the architecture sm_90 and higher.

+

The arguments order and scope need to be integer literals, i.e., the arguments cannot be variables.

+
+
+

+10.14.1.15. __nv_atomic_fetch_min() and __nv_atomic_min() +

+
+
__device__ T __nv_atomic_fetch_min (T* ptr, T val, int order, int scope = __NV_THREAD_SCOPE_SYSTEM);
+__device__ void __nv_atomic_min (T* ptr, T val, int order, int scope = __NV_THREAD_SCOPE_SYSTEM);
+
+
+

These two atomic functions are introduced in CUDA 12.8. It reads the value where ptr points to, compares with val, and stores the smaller value back to where ptr points to. __nv_atomic_fetch_min returns the old value where ptr points to. __nv_atomic_min does not have return value.

+

T can only be unsigned int, int, unsigned long long or long long.

+

The atomic operation with memory order and thread scope is supported on the architecture sm_60 and higher.

+

The thread scope of cluster is supported on the architecture sm_90 and higher.

+

The arguments order and scope need to be integer literals, i.e., the arguments cannot be variables.

+
+
+

+10.14.1.16. __nv_atomic_fetch_max() and __nv_atomic_max() +

+
+
__device__ T __nv_atomic_fetch_max (T* ptr, T val, int order, int scope = __NV_THREAD_SCOPE_SYSTEM);
+__device__ void __nv_atomic_max (T* ptr, T val, int order, int scope = __NV_THREAD_SCOPE_SYSTEM);
+
+
+

These two atomic functions are introduced in CUDA 12.8. It reads the value where ptr points to, compares with val, and stores the bigger value back to where ptr points to. __nv_atomic_fetch_max returns the old value where ptr points to. __nv_atomic_max does not have return value.

+

T can only be unsigned int, int, unsigned long long or long long.

+

The atomic operation with memory order and thread scope is supported on the architecture sm_60 and higher.

+

The thread scope of cluster is supported on the architecture sm_90 and higher.

+

The arguments order and scope need to be integer literals, i.e., the arguments cannot be variables.

+
+
+
+

+10.14.2. Bitwise Functions +

+
+

+10.14.2.1. atomicAnd() +

+
+
int atomicAnd(int* address, int val);
+unsigned int atomicAnd(unsigned int* address,
+                       unsigned int val);
+unsigned long long int atomicAnd(unsigned long long int* address,
+                                 unsigned long long int val);
+
+
+

reads the 32-bit or 64-bit word old located at the address address in global or shared memory, computes (old & val), and stores the result back to memory at the same address. These three operations are performed in one atomic transaction. The function returns old.

+

The 64-bit version of atomicAnd() is only supported by devices of compute capability 5.0 and higher.

+
+
+

+10.14.2.2. atomicOr() +

+
+
int atomicOr(int* address, int val);
+unsigned int atomicOr(unsigned int* address,
+                      unsigned int val);
+unsigned long long int atomicOr(unsigned long long int* address,
+                                unsigned long long int val);
+
+
+

reads the 32-bit or 64-bit word old located at the address address in global or shared memory, computes (old | val), and stores the result back to memory at the same address. These three operations are performed in one atomic transaction. The function returns old.

+

The 64-bit version of atomicOr() is only supported by devices of compute capability 5.0 and higher.

+
+
+

+10.14.2.3. atomicXor() +

+
+
int atomicXor(int* address, int val);
+unsigned int atomicXor(unsigned int* address,
+                       unsigned int val);
+unsigned long long int atomicXor(unsigned long long int* address,
+                                 unsigned long long int val);
+
+
+

reads the 32-bit or 64-bit word old located at the address address in global or shared memory, computes (old ^ val), and stores the result back to memory at the same address. These three operations are performed in one atomic transaction. The function returns old.

+

The 64-bit version of atomicXor() is only supported by devices of compute capability 5.0 and higher.

+
+
+

+10.14.2.4. __nv_atomic_fetch_or() and __nv_atomic_or() +

+
+
__device__ T __nv_atomic_fetch_or (T* ptr, T val, int order, int scope = __NV_THREAD_SCOPE_SYSTEM);
+__device__ void __nv_atomic_or (T* ptr, T val, int order, int scope = __NV_THREAD_SCOPE_SYSTEM);
+
+
+

These two atomic functions are introduced in CUDA 12.8. It reads the value where ptr points to, or with val, and stores the result back to where ptr points to. __nv_atomic_fetch_or returns the old value where ptr points to. __nv_atomic_or does not have return value.

+

T can only be an integral type that is size of 4 or 8 bytes.

+

The atomic operation with memory order and thread scope is supported on the architecture sm_60 and higher.

+

The thread scope of cluster is supported on the architecture sm_90 and higher.

+

The arguments order and scope need to be integer literals, i.e., the arguments cannot be variables.

+
+
+

+10.14.2.5. __nv_atomic_fetch_xor() and __nv_atomic_xor() +

+
+
__device__ T __nv_atomic_fetch_xor (T* ptr, T val, int order, int scope = __NV_THREAD_SCOPE_SYSTEM);
+__device__ void __nv_atomic_xor (T* ptr, T val, int order, int scope = __NV_THREAD_SCOPE_SYSTEM);
+
+
+

These two atomic functions are introduced in CUDA 12.8. It reads the value where ptr points to, xor with val, and stores the result back to where ptr points to. __nv_atomic_fetch_xor returns the old value where ptr points to. __nv_atomic_xor does not have return value.

+

T can only be an integral type that is size of 4 or 8 bytes.

+

The atomic operation with memory order and thread scope is supported on the architecture sm_60 and higher.

+

The thread scope of cluster is supported on the architecture sm_90 and higher.

+

The arguments order and scope need to be integer literals, i.e., the arguments cannot be variables.

+
+
+

+10.14.2.6. __nv_atomic_fetch_and() and __nv_atomic_and() +

+
+
__device__ T __nv_atomic_fetch_and (T* ptr, T val, int order, int scope = __NV_THREAD_SCOPE_SYSTEM);
+__device__ void __nv_atomic_and (T* ptr, T val, int order, int scope = __NV_THREAD_SCOPE_SYSTEM);
+
+
+

These two atomic functions are introduced in CUDA 12.8. It reads the value where ptr points to, and with val, and stores the result back to where ptr points to. __nv_atomic_fetch_and returns the old value where ptr points to. __nv_atomic_and does not have return value.

+

T can only be an integral type that is size of 4 or 8 bytes.

+

The atomic operation with memory order and thread scope is supported on the architecture sm_60 and higher.

+

The thread scope of cluster is supported on the architecture sm_90 and higher.

+

The arguments order and scope need to be integer literals, i.e., the arguments cannot be variables.

+
+
+
+

+10.14.3. Other atomic functions +

+
+

+10.14.3.1. __nv_atomic_load() +

+
+
__device__ void __nv_atomic_load(T* ptr, T* ret, int order, int scope = __NV_THREAD_SCOPE_SYSTEM);
+
+
+

This atomic function is introduced in CUDA 12.8. It loads the value where ptr points to and writes the value to where ret points to.

+

This is a generic atomic load, which means that T can be any data type that is size of 1, 2, 4, 8 or 16 bytes.

+

The atomic operation with memory order and thread scope is supported on the architecture sm_60 and higher.

+

16-byte data type is supported on the architecture sm_70 and higher.

+

The thread scope of cluster is supported on the architecture sm_90 and higher.

+

The arguments order and scope need to be integer literals, i.e., the arguments cannot be variables. order cannot be __NV_ATOMIC_RELEASE or __NV_ATOMIC_ACQ_REL.

+
+
+

+10.14.3.2. __nv_atomic_load_n() +

+
+
__device__ T __nv_atomic_load_n(T* ptr, int order, int scope = __NV_THREAD_SCOPE_SYSTEM);
+
+
+

This atomic function is introduced in CUDA 12.8. It loads the value where ptr points to and returns this value.

+

This is a non-generic atomic load, which means that T can only be an integral type that is size of 1, 2, 4, 8 or 16 bytes.

+

The atomic operation with memory order and thread scope is supported on the architecture sm_60 and higher.

+

16-byte data type is supported on the architecture sm_70 and higher.

+

The thread scope of cluster is supported on the architecture sm_90 and higher.

+

The arguments order and scope need to be integer literals, i.e., the arguments cannot be variables. order cannot be __NV_ATOMIC_RELEASE or __NV_ATOMIC_ACQ_REL.

+
+
+

+10.14.3.3. __nv_atomic_store() +

+
+
__device__ void __nv_atomic_store(T* ptr, T* val, int order, int scope = __NV_THREAD_SCOPE_SYSTEM);
+
+
+

This atomic function is introduced in CUDA 12.8. It reads the value where val points to and stores to where ptr points to.

+

This is a generic atomic load, which means that T can be any data type that is size of 1, 2, 4, 8 or 16 bytes.

+

The atomic operation with memory order and thread scope is supported on the architecture sm_60 and higher.

+

16-byte data type is supported on the architecture sm_70 and higher.

+

The thread scope of cluster is supported on the architecture sm_90 and higher.

+

The arguments order and scope need to be integer literals, i.e., the arguments cannot be variables. order cannot be __NV_ATOMIC_CONSUME, __NV_ATOMIC_ACQUIRE or __NV_ATOMIC_ACQ_REL.

+
+
+

+10.14.3.4. __nv_atomic_store_n() +

+
+
__device__ void __nv_atomic_store_n(T* ptr, T val, int order, int scope = __NV_THREAD_SCOPE_SYSTEM);
+
+
+

This atomic function is introduced in CUDA 12.8. It stores val to where ptr points to.

+

This is a non-generic atomic load, which means that T can only be an integral type that is size of 1, 2, 4, 8 or 16 bytes.

+

The atomic operation with memory order and thread scope is supported on the architecture sm_60 and higher.

+

16-byte data type is supported on the architecture sm_70 and higher.

+

The thread scope of cluster is supported on the architecture sm_90 and higher.

+

The arguments order and scope need to be integer literals, i.e., the arguments cannot be variables. order cannot be __NV_ATOMIC_CONSUME, __NV_ATOMIC_ACQUIRE or __NV_ATOMIC_ACQ_REL.

+
+
+

+10.14.3.5. __nv_atomic_thread_fence() +

+
+
__device__ void __nv_atomic_thread_fence (int order, int scope = __NV_THREAD_SCOPE_SYSTEM);
+
+
+

This atomic function establishes an ordering between memory accesses requested by this thread based on the specified memory order. And the thread scope parameter specifies the set of threads that may observe the ordering effect of this operation.

+

The thread scope of cluster is supported on the architecture sm_90 and higher.

+

The arguments order and scope need to be integer literals, i.e., the arguments cannot be variables.

+
+
+
+
+

+10.15. Address Space Predicate Functions +

+

The functions described in this section have unspecified behavior if the argument is a null pointer.

+
+

+10.15.1. __isGlobal() +

+
+
__device__ unsigned int __isGlobal(const void *ptr);
+
+
+

Returns 1 if ptr contains the generic address of an object in global memory space, otherwise returns 0.

+
+
+

+10.15.2. __isShared() +

+
+
__device__ unsigned int __isShared(const void *ptr);
+
+
+

Returns 1 if ptr contains the generic address of an object in shared memory space, otherwise returns 0.

+
+
+

+10.15.3. __isConstant() +

+
+
__device__ unsigned int __isConstant(const void *ptr);
+
+
+

Returns 1 if ptr contains the generic address of an object in constant memory space, otherwise returns 0.

+
+
+

+10.15.4. __isGridConstant() +

+
+
__device__ unsigned int __isGridConstant(const void *ptr);
+
+
+

Returns 1 if ptr contains the generic address of a kernel parameter annotated with __grid_constant__, otherwise returns 0. Only supported for compute architectures greater than or equal to 7.x or later.

+
+
+

+10.15.5. __isLocal() +

+
+
__device__ unsigned int __isLocal(const void *ptr);
+
+
+

Returns 1 if ptr contains the generic address of an object in local memory space, otherwise returns 0.

+
+
+
+

+10.16. Address Space Conversion Functions +

+

CUDA C++ pointers (T*) can access CUDA C++ objects independently of where these objects are stored. +For example, an int* can access int objects independently of whether they are stored in global or shared memory.

+

The Address Space Conversion Functions below enable converting CUDA C++ pointers from and to other representations. +This is required, among others, to interoperate with certain PTX instructions, or to exploit properties of these other representations for performance optimizations.

+

As an example of interoperating with certain PTX instructions, an ld.shared.u32 r0, [addr]; PTX instruction expects addr to refer to the shared space. +A CUDA C++ program with a CUDA C++ uint32_t* pointer to an object in __shared__ memory, needs to convert this pointer to the shared space before passing it to such a PTX instruction by calling __cvta_generic_to_shared as follows:

+
+
__shared__ uint32_t x;
+x = 42;
+void* p = &x;
+size_t sp = __cvta_generic_to_shared(p);
+uint32_t o;
+asm volatile("ld.shared.u32 %0, [%1];" : "=r"(o) : "l"(sp) : "memory");
+assert(o == 42);
+
+
+

A common program optimization that exploits properties of these other address representations is reducing data-structure size by leveraging that the address ranges of shared, local, and const spaces is smaller than 32-bit, which allows programs to store 32-bit addresses instead of 64-bit pointers. +To obtain the 32-bit integer representation of these addresses, it suffices to truncate it by performing an unsigned 64-bit integer to unsigned 32-bit integer cast:

+
+
__shared__ int x;
+void* p = &x;
+uint32_t smem32 = __cvta_generic_to_shared(p);
+
+
+

To obtain a generic address from such a 32-bit integer representation, it suffices to zero-extend the address back to an unsigned 64-bit integer before calling the corresponding address space conversion function:

+
+
size_t smem64 = smem32;
+void* q = __cvta_shared_to_generic(smem64);
+assert(p == q);
+
+
+

A roundtrip from an input generic space pointer to its 32-bit integer representation and back to an output generic space pointer is guaranteed to return an output pointer that is equivalent to the input pointer of the roundtrip for the spaces listed above.

+
+

+10.16.1. __cvta_generic_to_global() +

+
+
__device__ size_t __cvta_generic_to_global(const void *ptr);
+
+
+

Returns the result of executing the PTXcvta.to.global instruction on the generic address denoted by ptr.

+
+
+

+10.16.2. __cvta_generic_to_shared() +

+
+
__device__ size_t __cvta_generic_to_shared(const void *ptr);
+
+
+

Returns the result of executing the PTXcvta.to.shared instruction on the generic address denoted by ptr.

+
+
+

+10.16.3. __cvta_generic_to_constant() +

+
+
__device__ size_t __cvta_generic_to_constant(const void *ptr);
+
+
+

Returns the result of executing the PTXcvta.to.const instruction on the generic address denoted by ptr.

+
+
+

+10.16.4. __cvta_generic_to_local() +

+
+
__device__ size_t __cvta_generic_to_local(const void *ptr);
+
+
+

Returns the result of executing the PTXcvta.to.local instruction on the generic address denoted by ptr.

+
+
+

+10.16.5. __cvta_global_to_generic() +

+
+
__device__ void * __cvta_global_to_generic(size_t rawbits);
+
+
+

Returns the generic pointer obtained by executing the PTXcvta.global instruction on the value provided by rawbits.

+
+
+

+10.16.6. __cvta_shared_to_generic() +

+
+
__device__ void * __cvta_shared_to_generic(size_t rawbits);
+
+
+

Returns the generic pointer obtained by executing the PTXcvta.shared instruction on the value provided by rawbits.

+
+
+

+10.16.7. __cvta_constant_to_generic() +

+
+
__device__ void * __cvta_constant_to_generic(size_t rawbits);
+
+
+

Returns the generic pointer obtained by executing the PTXcvta.const instruction on the value provided by rawbits.

+
+
+

+10.16.8. __cvta_local_to_generic() +

+
+
__device__ void * __cvta_local_to_generic(size_t rawbits);
+
+
+

Returns the generic pointer obtained by executing the PTXcvta.local instruction on the value provided by rawbits.

+
+
+
+

+10.17. Alloca Function +

+
+

+10.17.1. Synopsis +

+
+
__host__ __device__ void * alloca(size_t size);
+
+
+
+
+

+10.17.2. Description +

+

The alloca() function allocates size bytes of memory in the stack frame of the caller. The returned value is a pointer to allocated memory, the beginning of the memory is 16 bytes aligned when the function is invoked from device code. The allocated memory is automatically freed when the caller to alloca() is returned.

+
+

Note

+

On Windows platform, <malloc.h> must be included before using alloca(). Using alloca() may cause the stack to overflow, user needs to adjust stack size accordingly.

+
+

It is supported with compute capability 5.2 or higher.

+
+
+

+10.17.3. Example +

+
+
__device__ void foo(unsigned int num) {
+    int4 *ptr = (int4 *)alloca(num * sizeof(int4));
+    // use of ptr
+    ...
+}
+
+
+
+
+
+

+10.18. Compiler Optimization Hint Functions +

+

The functions described in this section can be used to provide additional information to the compiler optimizer.

+
+

+10.18.1. __builtin_assume_aligned() +

+
+
void * __builtin_assume_aligned (const void *exp, size_t align)
+
+
+

Allows the compiler to assume that the argument pointer is aligned to at least align bytes, and returns the argument pointer.

+

Example:

+
+
void *res = __builtin_assume_aligned(ptr, 32); // compiler can assume 'res' is
+                                               // at least 32-byte aligned
+
+
+

Three parameter version:

+
+
void * __builtin_assume_aligned (const void *exp, size_t align,
+                                 <integral type> offset)
+
+
+

Allows the compiler to assume that (char *)exp - offset is aligned to at least align bytes, and returns the argument pointer.

+

Example:

+
+
void *res = __builtin_assume_aligned(ptr, 32, 8); // compiler can assume
+                                                  // '(char *)res - 8' is
+                                                  // at least 32-byte aligned.
+
+
+
+
+

+10.18.2. __builtin_assume() +

+
+
void __builtin_assume(bool exp)
+
+
+

Allows the compiler to assume that the Boolean argument is true. If the argument is not true at run time, then the behavior is undefined. Note that if the argument has side effects, the behavior is unspecified.

+

Example:

+
+
 __device__ int get(int *ptr, int idx) {
+   __builtin_assume(idx <= 2);
+   return ptr[idx];
+}
+
+
+
+
+

+10.18.3. __assume() +

+
+
void __assume(bool exp)
+
+
+

Allows the compiler to assume that the Boolean argument is true. If the argument is not true at run time, then the behavior is undefined. Note that if the argument has side effects, the behavior is unspecified.

+

Example:

+
+
 __device__ int get(int *ptr, int idx) {
+   __assume(idx <= 2);
+   return ptr[idx];
+}
+
+
+
+
+

+10.18.4. __builtin_expect() +

+
+
long __builtin_expect (long exp, long c)
+
+
+

Indicates to the compiler that it is expected that exp == c, and returns the value of exp. Typically used to indicate branch prediction information to the compiler.

+

Example:

+
+
// indicate to the compiler that likely "var == 0",
+// so the body of the if-block is unlikely to be
+// executed at run time.
+if (__builtin_expect (var, 0))
+  doit ();
+
+
+
+
+

+10.18.5. __builtin_unreachable() +

+
+
void __builtin_unreachable(void)
+
+
+

Indicates to the compiler that control flow never reaches the point where this function is being called from. The program has undefined behavior if the control flow does actually reach this point at run time.

+

Example:

+
+
// indicates to the compiler that the default case label is never reached.
+switch (in) {
+case 1: return 4;
+case 2: return 10;
+default: __builtin_unreachable();
+}
+
+
+
+
+

+10.18.6. Restrictions +

+

__assume() is only supported when using cl.exe host compiler. The other functions are supported on all platforms, subject to the following restrictions:

+
    +
  • If the host compiler supports the function, the function can be invoked from anywhere in translation unit.

  • +
  • Otherwise, the function must be invoked from within the body of a __device__/ __global__function, or only when the __CUDA_ARCH__ macro is defined5.

  • +
+
+
+
+

+10.19. Warp Vote Functions +

+
+
int __all_sync(unsigned mask, int predicate);
+int __any_sync(unsigned mask, int predicate);
+unsigned __ballot_sync(unsigned mask, int predicate);
+unsigned __activemask();
+
+
+

Deprecation notice: __any, __all, and __ballot have been deprecated in CUDA 9.0 for all devices.

+

Removal notice: When targeting devices with compute capability 7.x or higher, __any, __all, and __ballot are no longer available and their sync variants should be used instead.

+

The warp vote functions allow the threads of a given warp to perform a reduction-and-broadcast operation. These functions take as input an integer predicate from each thread in the warp and compare those values with zero. The results of the comparisons are combined (reduced) across the active threads of the warp in one of the following ways, broadcasting a single return value to each participating thread:

+
+
+__all_sync(unsigned mask, predicate):
+
+

Evaluate predicate for all non-exited threads in mask and return non-zero if and only if predicate evaluates to non-zero for all of them.

+
+
+__any_sync(unsigned mask, predicate):
+
+

Evaluate predicate for all non-exited threads in mask and return non-zero if and only if predicate evaluates to non-zero for any of them.

+
+
+__ballot_sync(unsigned mask, predicate):
+
+

Evaluate predicate for all non-exited threads in mask and return an integer whose Nth bit is set if and only if predicate evaluates to non-zero for the Nth thread of the warp and the Nth thread is active.

+
+
+__activemask():
+
+

Returns a 32-bit integer mask of all currently active threads in the calling warp. The Nth bit is set if the Nth lane in the warp is active when __activemask() is called. Inactive threads are represented by 0 bits in the returned mask. Threads which have exited the program are always marked as inactive. Note that threads that are convergent at an __activemask() call are not guaranteed to be convergent at subsequent instructions unless those instructions are synchronizing warp-builtin functions.

+
+
+

For __all_sync, __any_sync, and __ballot_sync, a mask must be passed that specifies the threads participating in the call. A bit, representing the thread’s lane ID, must be set for each participating thread to ensure they are properly converged before the intrinsic is executed by the hardware. Each calling thread must have its own bit set in the mask and all non-exited threads named in mask must execute the same intrinsic with the same mask, or the result is undefined.

+

These intrinsics do not imply a memory barrier. They do not guarantee any memory ordering.

+
+
+

+10.20. Warp Match Functions +

+

__match_any_sync and __match_all_sync perform a broadcast-and-compare operation of a variable between threads within a warp.

+

Supported by devices of compute capability 7.x or higher.

+
+

+10.20.1. Synopsis +

+
+
unsigned int __match_any_sync(unsigned mask, T value);
+unsigned int __match_all_sync(unsigned mask, T value, int *pred);
+
+
+

T can be int, unsigned int, long, unsigned long, long long, unsigned long long, float or double.

+
+
+

+10.20.2. Description +

+

The __match_sync() intrinsics permit a broadcast-and-compare of a value value across threads in a warp after synchronizing threads named in mask.

+
+
__match_any_sync
+
+

Returns mask of threads that have same value of value in mask

+
+
__match_all_sync
+
+

Returns mask if all threads in mask have the same value for value; otherwise 0 is returned. Predicate pred is set to true if all threads in mask have the same value of value; otherwise the predicate is set to false.

+
+
+

The new *_sync match intrinsics take in a mask indicating the threads participating in the call. A bit, representing the thread’s lane id, must be set for each participating thread to ensure they are properly converged before the intrinsic is executed by the hardware. Each calling thread must have its own bit set in the mask and all non-exited threads named in mask must execute the same intrinsic with the same mask, or the result is undefined.

+

These intrinsics do not imply a memory barrier. They do not guarantee any memory ordering.

+
+
+
+

+10.21. Warp Reduce Functions +

+

The __reduce_sync(unsigned mask, T value) intrinsics perform a reduction operation on the data provided in value after synchronizing threads named in mask. T can be unsigned or signed for {add, min, max} and unsigned only for {and, or, xor} operations.

+

Supported by devices of compute capability 8.x or higher.

+
+

+10.21.1. Synopsis +

+
+
// add/min/max
+unsigned __reduce_add_sync(unsigned mask, unsigned value);
+unsigned __reduce_min_sync(unsigned mask, unsigned value);
+unsigned __reduce_max_sync(unsigned mask, unsigned value);
+int __reduce_add_sync(unsigned mask, int value);
+int __reduce_min_sync(unsigned mask, int value);
+int __reduce_max_sync(unsigned mask, int value);
+
+// and/or/xor
+unsigned __reduce_and_sync(unsigned mask, unsigned value);
+unsigned __reduce_or_sync(unsigned mask, unsigned value);
+unsigned __reduce_xor_sync(unsigned mask, unsigned value);
+
+
+
+
+

+10.21.2. Description +

+
+
+__reduce_add_sync, __reduce_min_sync, __reduce_max_sync +
+
+

Returns the result of applying an arithmetic add, min, or max reduction operation on the values provided in value by each thread named in mask.

+
+
+__reduce_and_sync, __reduce_or_sync, __reduce_xor_sync +
+
+

Returns the result of applying a logical AND, OR, or XOR reduction operation on the values provided in value by each thread named in mask.

+
+
+

The mask indicates the threads participating in the call. A bit, representing the thread’s lane id, must be set for each participating thread to ensure they are properly converged before the intrinsic is executed by the hardware. Each calling thread must have its own bit set in the mask and all non-exited threads named in mask must execute the same intrinsic with the same mask, or the result is undefined.

+

These intrinsics do not imply a memory barrier. They do not guarantee any memory ordering.

+
+
+
+

+10.22. Warp Shuffle Functions +

+

__shfl_sync, __shfl_up_sync, __shfl_down_sync, and __shfl_xor_sync exchange a variable between threads within a warp.

+

Supported by devices of compute capability 5.0 or higher.

+

Deprecation Notice: __shfl, __shfl_up, __shfl_down, and __shfl_xor have been deprecated in CUDA 9.0 for all devices.

+

Removal Notice: When targeting devices with compute capability 7.x or higher, __shfl, __shfl_up, __shfl_down, and __shfl_xor are no longer available and their sync variants should be used instead.

+
+

+10.22.1. Synopsis +

+
+
T __shfl_sync(unsigned mask, T var, int srcLane, int width=warpSize);
+T __shfl_up_sync(unsigned mask, T var, unsigned int delta, int width=warpSize);
+T __shfl_down_sync(unsigned mask, T var, unsigned int delta, int width=warpSize);
+T __shfl_xor_sync(unsigned mask, T var, int laneMask, int width=warpSize);
+
+
+

T can be int, unsigned int, long, unsigned long, long long, unsigned long long, float or double. With the cuda_fp16.h header included, T can also be __half or __half2. Similarly, with the cuda_bf16.h header included, T can also be __nv_bfloat16 or __nv_bfloat162.

+
+
+

+10.22.2. Description +

+

The __shfl_sync() intrinsics permit exchanging of a variable between threads within a warp without use of shared memory. The exchange occurs simultaneously for all active threads within the warp (and named in mask), moving 4 or 8 bytes of data per thread depending on the type.

+

Threads within a warp are referred to as lanes, and may have an index between 0 and warpSize-1 (inclusive). Four source-lane addressing modes are supported:

+
+
__shfl_sync()
+
+

Direct copy from indexed lane

+
+
__shfl_up_sync()
+
+

Copy from a lane with lower ID relative to caller

+
+
__shfl_down_sync()
+
+

Copy from a lane with higher ID relative to caller

+
+
__shfl_xor_sync()
+
+

Copy from a lane based on bitwise XOR of own lane ID

+
+
+

Threads may only read data from another thread which is actively participating in the __shfl_sync() command. If the target thread is inactive, the retrieved value is undefined.

+

All of the __shfl_sync() intrinsics take an optional width parameter which alters the behavior of the intrinsic. width must have a value which is a power of two in the range [1, warpSize] (i.e., 1, 2, 4, 8, 16 or 32). Results are undefined for other values.

+

__shfl_sync() returns the value of var held by the thread whose ID is given by srcLane. If width is less than warpSize then each subsection of the warp behaves as a separate entity with a starting logical lane ID of 0. If srcLane is outside the range [0:width-1], the value returned corresponds to the value of var held by the srcLane modulo width (i.e. within the same subsection).

+

__shfl_up_sync() calculates a source lane ID by subtracting delta from the caller’s lane ID. The value of var held by the resulting lane ID is returned: in effect, var is shifted up the warp by delta lanes. If width is less than warpSize then each subsection of the warp behaves as a separate entity with a starting logical lane ID of 0. The source lane index will not wrap around the value of width, so effectively the lower delta lanes will be unchanged.

+

__shfl_down_sync() calculates a source lane ID by adding delta to the caller’s lane ID. The value of var held by the resulting lane ID is returned: this has the effect of shifting var down the warp by delta lanes. If width is less than warpSize then each subsection of the warp behaves as a separate entity with a starting logical lane ID of 0. As for __shfl_up_sync(), the ID number of the source lane will not wrap around the value of width and so the upper delta lanes will remain unchanged.

+

__shfl_xor_sync() calculates a source line ID by performing a bitwise XOR of the caller’s lane ID with laneMask: the value of var held by the resulting lane ID is returned. If width is less than warpSize then each group of width consecutive threads are able to access elements from earlier groups of threads, however if they attempt to access elements from later groups of threads their own value of var will be returned. This mode implements a butterfly addressing pattern such as is used in tree reduction and broadcast.

+

The new *_sync shfl intrinsics take in a mask indicating the threads participating in the call. A bit, representing the thread’s lane id, must be set for each participating thread to ensure they are properly converged before the intrinsic is executed by the hardware. Each calling thread must have its own bit set in the mask and all non-exited threads named in mask must execute the same intrinsic with the same mask, or the result is undefined.

+

Threads may only read data from another thread which is actively participating in the __shfl_sync() command. If the target thread is inactive, the retrieved value is undefined.

+

These intrinsics do not imply a memory barrier. They do not guarantee any memory ordering.

+
+
+

+10.22.3. Examples +

+
+

+10.22.3.1. Broadcast of a single value across a warp +

+
+
#include <stdio.h>
+
+__global__ void bcast(int arg) {
+    int laneId = threadIdx.x & 0x1f;
+    int value;
+    if (laneId == 0)        // Note unused variable for
+        value = arg;        // all threads except lane 0
+    value = __shfl_sync(0xffffffff, value, 0);   // Synchronize all threads in warp, and get "value" from lane 0
+    if (value != arg)
+        printf("Thread %d failed.\n", threadIdx.x);
+}
+
+int main() {
+    bcast<<< 1, 32 >>>(1234);
+    cudaDeviceSynchronize();
+
+    return 0;
+}
+
+
+
+
+

+10.22.3.2. Inclusive plus-scan across sub-partitions of 8 threads +

+
+
#include <stdio.h>
+
+__global__ void scan4() {
+    int laneId = threadIdx.x & 0x1f;
+    // Seed sample starting value (inverse of lane ID)
+    int value = 31 - laneId;
+
+    // Loop to accumulate scan within my partition.
+    // Scan requires log2(n) == 3 steps for 8 threads
+    // It works by an accumulated sum up the warp
+    // by 1, 2, 4, 8 etc. steps.
+    for (int i=1; i<=4; i*=2) {
+        // We do the __shfl_sync unconditionally so that we
+        // can read even from threads which won't do a
+        // sum, and then conditionally assign the result.
+        int n = __shfl_up_sync(0xffffffff, value, i, 8);
+        if ((laneId & 7) >= i)
+            value += n;
+    }
+
+    printf("Thread %d final value = %d\n", threadIdx.x, value);
+}
+
+int main() {
+    scan4<<< 1, 32 >>>();
+    cudaDeviceSynchronize();
+
+    return 0;
+}
+
+
+
+
+

+10.22.3.3. Reduction across a warp +

+
+
#include <stdio.h>
+
+__global__ void warpReduce() {
+    int laneId = threadIdx.x & 0x1f;
+    // Seed starting value as inverse lane ID
+    int value = 31 - laneId;
+
+    // Use XOR mode to perform butterfly reduction
+    for (int i=16; i>=1; i/=2)
+        value += __shfl_xor_sync(0xffffffff, value, i, 32);
+
+    // "value" now contains the sum across all threads
+    printf("Thread %d final value = %d\n", threadIdx.x, value);
+}
+
+int main() {
+    warpReduce<<< 1, 32 >>>();
+    cudaDeviceSynchronize();
+
+    return 0;
+}
+
+
+
+
+
+
+

+10.23. Nanosleep Function +

+
+

+10.23.1. Synopsis +

+
+
void __nanosleep(unsigned ns);
+
+
+
+
+

+10.23.2. Description +

+

__nanosleep(ns) suspends the thread for a sleep duration of approximately ns nanoseconds. The maximum sleep duration is approximately 1 millisecond.

+

It is supported with compute capability 7.0 or higher.

+
+
+

+10.23.3. Example +

+

The following code implements a mutex with exponential back-off.

+
+
__device__ void mutex_lock(unsigned int *mutex) {
+    unsigned int ns = 8;
+    while (atomicCAS(mutex, 0, 1) == 1) {
+        __nanosleep(ns);
+        if (ns < 256) {
+            ns *= 2;
+        }
+    }
+}
+
+__device__ void mutex_unlock(unsigned int *mutex) {
+    atomicExch(mutex, 0);
+}
+
+
+
+
+
+

+10.24. Warp Matrix Functions +

+

C++ warp matrix operations leverage Tensor Cores to accelerate matrix problems of the form D=A*B+C. These operations are supported on mixed-precision floating point data for devices of compute capability 7.0 or higher. This requires co-operation from all threads in a warp. In addition, these operations are allowed in conditional code only if the condition evaluates identically across the entire warp, otherwise the code execution is likely to hang.

+
+

+10.24.1. Description +

+

All following functions and types are defined in the namespace nvcuda::wmma. Sub-byte operations are considered preview, i.e. the data structures and APIs for them are subject to change and may not be compatible with future releases. This extra functionality is defined in the nvcuda::wmma::experimental namespace.

+
+
template<typename Use, int m, int n, int k, typename T, typename Layout=void> class fragment;
+
+void load_matrix_sync(fragment<...> &a, const T* mptr, unsigned ldm);
+void load_matrix_sync(fragment<...> &a, const T* mptr, unsigned ldm, layout_t layout);
+void store_matrix_sync(T* mptr, const fragment<...> &a, unsigned ldm, layout_t layout);
+void fill_fragment(fragment<...> &a, const T& v);
+void mma_sync(fragment<...> &d, const fragment<...> &a, const fragment<...> &b, const fragment<...> &c, bool satf=false);
+
+
+
+
fragment
+
+

An overloaded class containing a section of a matrix distributed across all threads in the warp. The mapping of matrix elements into fragment internal storage is unspecified and subject to change in future architectures.

+
+
+

Only certain combinations of template arguments are allowed. The first template parameter specifies how the fragment will participate in the matrix operation. Acceptable values for Use are:

+
    +
  • matrix_a when the fragment is used as the first multiplicand, A,

  • +
  • matrix_b when the fragment is used as the second multiplicand, B, or

  • +
  • +

    accumulator when the fragment is used as the source or destination accumulators (C or D, respectively).

    +

    The m, n and k sizes describe the shape of the warp-wide matrix tiles participating in the multiply-accumulate operation. The dimension of each tile depends on its role. For matrix_a the tile takes dimension m x k; for matrix_b the dimension is k x n, and accumulator tiles are m x n.

    +

    The data type, T, may be double, float, __half, __nv_bfloat16, char, or unsigned char for multiplicands and double, float, int, or __half for accumulators. As documented in Element Types and Matrix Sizes, limited combinations of accumulator and multiplicand types are supported. The Layout parameter must be specified for matrix_a and matrix_b fragments. row_major or col_major indicate that elements within a matrix row or column are contiguous in memory, respectively. The Layout parameter for an accumulator matrix should retain the default value of void. A row or column layout is specified only when the accumulator is loaded or stored as described below.

    +
  • +
+
+
load_matrix_sync
+
+

Waits until all warp lanes have arrived at load_matrix_sync and then loads the matrix fragment a from memory. mptr must be a 256-bit aligned pointer pointing to the first element of the matrix in memory. ldm describes the stride in elements between consecutive rows (for row major layout) or columns (for column major layout) and must be a multiple of 8 for __half element type or multiple of 4 for float element type. (i.e., multiple of 16 bytes in both cases). If the fragment is an accumulator, the layout argument must be specified as either mem_row_major or mem_col_major. For matrix_a and matrix_b fragments, the layout is inferred from the fragment’s layout parameter. The values of mptr, ldm, layout and all template parameters for a must be the same for all threads in the warp. This function must be called by all threads in the warp, or the result is undefined.

+
+
store_matrix_sync
+
+

Waits until all warp lanes have arrived at store_matrix_sync and then stores the matrix fragment a to memory. mptr must be a 256-bit aligned pointer pointing to the first element of the matrix in memory. ldm describes the stride in elements between consecutive rows (for row major layout) or columns (for column major layout) and must be a multiple of 8 for __half element type or multiple of 4 for float element type. (i.e., multiple of 16 bytes in both cases). The layout of the output matrix must be specified as either mem_row_major or mem_col_major. The values of mptr, ldm, layout and all template parameters for a must be the same for all threads in the warp.

+
+
fill_fragment
+
+

Fill a matrix fragment with a constant value v. Because the mapping of matrix elements to each fragment is unspecified, this function is ordinarily called by all threads in the warp with a common value for v.

+
+
mma_sync
+
+

Waits until all warp lanes have arrived at mma_sync, and then performs the warp-synchronous matrix multiply-accumulate operation D=A*B+C. The in-place operation, C=A*B+C, is also supported. The value of satf and template parameters for each matrix fragment must be the same for all threads in the warp. Also, the template parameters m, n and k must match between fragments A, B, C and D. This function must be called by all threads in the warp, or the result is undefined.

+
+
+

If satf (saturate to finite value) mode is true, the following additional numerical properties apply for the destination accumulator:

+
    +
  • If an element result is +Infinity, the corresponding accumulator will contain +MAX_NORM

  • +
  • If an element result is -Infinity, the corresponding accumulator will contain -MAX_NORM

  • +
  • If an element result is NaN, the corresponding accumulator will contain +0

  • +
+

Because the map of matrix elements into each thread’s fragment is unspecified, individual matrix elements must be accessed from memory (shared or global) after calling store_matrix_sync. In the special case where all threads in the warp will apply an element-wise operation uniformly to all fragment elements, direct element access can be implemented using the following fragment class members.

+
+
enum fragment<Use, m, n, k, T, Layout>::num_elements;
+T fragment<Use, m, n, k, T, Layout>::x[num_elements];
+
+
+

As an example, the following code scales an accumulator matrix tile by half.

+
+
wmma::fragment<wmma::accumulator, 16, 16, 16, float> frag;
+float alpha = 0.5f; // Same value for all threads in warp
+/*...*/
+for(int t=0; t<frag.num_elements; t++)
+frag.x[t] *= alpha;
+
+
+
+
+

+10.24.2. Alternate Floating Point +

+

Tensor Cores support alternate types of floating point operations on devices with compute capability 8.0 and higher.

+
+
__nv_bfloat16
+
+

This data format is an alternate fp16 format that has the same range as f32 but reduced precision (7 bits). You can use this data format directly with the __nv_bfloat16 type available in cuda_bf16.h. Matrix fragments with __nv_bfloat16 data types are required to be composed with accumulators of float type. The shapes and operations supported are the same as with __half.

+
+
tf32
+
+

This data format is a special floating point format supported by Tensor Cores, with the same range as f32 and reduced precision (>=10 bits). The internal layout of this format is implementation defined. In order to use this floating point format with WMMA operations, the input matrices must be manually converted to tf32 precision.

+

To facilitate conversion, a new intrinsic __float_to_tf32 is provided. While the input and output arguments to the intrinsic are of float type, the output will be tf32 numerically. This new precision is intended to be used with Tensor Cores only, and if mixed with other floattype operations, the precision and range of the result will be undefined.

+

Once an input matrix (matrix_a or matrix_b) is converted to tf32 precision, the combination of a fragment with precision::tf32 precision, and a data type of float to load_matrix_sync will take advantage of this new capability. Both the accumulator fragments must have float data types. The only supported matrix size is 16x16x8 (m-n-k).

+

The elements of the fragment are represented as float, hence the mapping from element_type<T> to storage_element_type<T> is:

+
+
precision::tf32 -> float
+
+
+
+
+
+
+

+10.24.3. Double Precision +

+

Tensor Cores support double-precision floating point operations on devices with compute capability 8.0 and higher. To use this new functionality, a fragment with the double type must be used. The mma_sync operation will be performed with the .rn (rounds to nearest even) rounding modifier.

+
+
+

+10.24.4. Sub-byte Operations +

+

Sub-byte WMMA operations provide a way to access the low-precision capabilities of Tensor Cores. They are considered a preview feature i.e. the data structures and APIs for them are subject to change and may not be compatible with future releases. This functionality is available via the nvcuda::wmma::experimental namespace:

+
+
namespace experimental {
+    namespace precision {
+        struct u4; // 4-bit unsigned
+        struct s4; // 4-bit signed
+        struct b1; // 1-bit
+   }
+    enum bmmaBitOp {
+        bmmaBitOpXOR = 1, // compute_75 minimum
+        bmmaBitOpAND = 2  // compute_80 minimum
+    };
+    enum bmmaAccumulateOp { bmmaAccumulateOpPOPC = 1 };
+}
+
+
+

For 4 bit precision, the APIs available remain the same, but you must specify experimental::precision::u4 or experimental::precision::s4 as the fragment data type. Since the elements of the fragment are packed together, num_storage_elements will be smaller than num_elements for that fragment. The num_elements variable for a sub-byte fragment, hence returns the number of elements of sub-byte type element_type<T>. This is true for single bit precision as well, in which case, the mapping from element_type<T> to storage_element_type<T> is as follows:

+
+
experimental::precision::u4 -> unsigned (8 elements in 1 storage element)
+experimental::precision::s4 -> int (8 elements in 1 storage element)
+experimental::precision::b1 -> unsigned (32 elements in 1 storage element)
+T -> T  //all other types
+
+
+

The allowed layouts for sub-byte fragments is always row_major for matrix_a and col_major for matrix_b.

+

For sub-byte operations the value of ldm in load_matrix_sync should be a multiple of 32 for element type experimental::precision::u4 and experimental::precision::s4 or a multiple of 128 for element type experimental::precision::b1 (i.e., multiple of 16 bytes in both cases).

+
+

Note

+

Support for the following variants for MMA instructions is deprecated and will be removed in sm_90:

+
+
+
    +
  • experimental::precision::u4

  • +
  • experimental::precision::s4

  • +
  • experimental::precision::b1 with bmmaBitOp set to bmmaBitOpXOR

  • +
+
+
+
+
+
bmma_sync
+
+

Waits until all warp lanes have executed bmma_sync, and then performs the warp-synchronous bit matrix multiply-accumulate operation D = (A op B) + C, where op consists of a logical operation bmmaBitOp followed by the accumulation defined by bmmaAccumulateOp. The available operations are:

+

bmmaBitOpXOR, a 128-bit XOR of a row in matrix_a with the 128-bit column of matrix_b

+

bmmaBitOpAND, a 128-bit AND of a row in matrix_a with the 128-bit column of matrix_b, available on devices with compute capability 8.0 and higher.

+

The accumulate op is always bmmaAccumulateOpPOPC which counts the number of set bits.

+
+
+
+
+

+10.24.5. Restrictions +

+

The special format required by tensor cores may be different for each major and minor device architecture. This is further complicated by threads holding only a fragment (opaque architecture-specific ABI data structure) of the overall matrix, with the developer not allowed to make assumptions on how the individual parameters are mapped to the registers participating in the matrix multiply-accumulate.

+

Since fragments are architecture-specific, it is unsafe to pass them from function A to function B if the functions have been compiled for different link-compatible architectures and linked together into the same device executable. In this case, the size and layout of the fragment will be specific to one architecture and using WMMA APIs in the other will lead to incorrect results or potentially, corruption.

+

An example of two link-compatible architectures, where the layout of the fragment differs, is sm_70 and sm_75.

+
+
fragA.cu: void foo() { wmma::fragment<...> mat_a; bar(&mat_a); }
+fragB.cu: void bar(wmma::fragment<...> *mat_a) { // operate on mat_a }
+
+
+
+
// sm_70 fragment layout
+$> nvcc -dc -arch=compute_70 -code=sm_70 fragA.cu -o fragA.o
+// sm_75 fragment layout
+$> nvcc -dc -arch=compute_75 -code=sm_75 fragB.cu -o fragB.o
+// Linking the two together
+$> nvcc -dlink -arch=sm_75 fragA.o fragB.o -o frag.o
+
+
+

This undefined behavior might also be undetectable at compilation time and by tools at runtime, so extra care is needed to make sure the layout of the fragments is consistent. This linking hazard is most likely to appear when linking with a legacy library that is both built for a different link-compatible architecture and expecting to be passed a WMMA fragment.

+

Note that in the case of weak linkages (for example, a CUDA C++ inline function), the linker may choose any available function definition which may result in implicit passes between compilation units.

+

To avoid these sorts of problems, the matrix should always be stored out to memory for transit through external interfaces (e.g. wmma::store_matrix_sync(dst, …);) and then it can be safely passed to bar() as a pointer type [e.g. float *dst].

+

Note that since sm_70 can run on sm_75, the above example sm_75 code can be changed to sm_70 and correctly work on sm_75. However, it is recommended to have sm_75 native code in your application when linking with other sm_75 separately compiled binaries.

+
+
+

+10.24.6. Element Types and Matrix Sizes +

+

Tensor Cores support a variety of element types and matrix sizes. The following table presents the various combinations of matrix_a, matrix_b and accumulator matrix supported:

+ ++++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +

Matrix A

Matrix B

Accumulator

Matrix Size (m-n-k)

__half

__half

float

16x16x16

__half

__half

float

32x8x16

__half

__half

float

8x32x16

__half

__half

__half

16x16x16

__half

__half

__half

32x8x16

__half

__half

__half

8x32x16

unsigned char

unsigned char

int

16x16x16

unsigned char

unsigned char

int

32x8x16

unsigned char

unsigned char

int

8x32x16

signed char

signed char

int

16x16x16

signed char

signed char

int

32x8x16

signed char

signed char

int

8x32x16

+

Alternate Floating Point support:

+ ++++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +

Matrix A

Matrix B

Accumulator

Matrix Size (m-n-k)

__nv_bfloat16

__nv_bfloat16

float

16x16x16

__nv_bfloat16

__nv_bfloat16

float

32x8x16

__nv_bfloat16

__nv_bfloat16

float

8x32x16

precision::tf32

precision::tf32

float

16x16x8

+

Double Precision Support:

+ ++++++ + + + + + + + + + + + + + + + + +

Matrix A

Matrix B

Accumulator

Matrix Size (m-n-k)

double

double

double

8x8x4

+

Experimental support for sub-byte operations:

+ ++++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + +

Matrix A

Matrix B

Accumulator

Matrix Size (m-n-k)

precision::u4

precision::u4

int

8x8x32

precision::s4

precision::s4

int

8x8x32

precision::b1

precision::b1

int

8x8x128

+
+
+

+10.24.7. Example +

+

The following code implements a 16x16x16 matrix multiplication in a single warp.

+
+
#include <mma.h>
+using namespace nvcuda;
+
+__global__ void wmma_ker(half *a, half *b, float *c) {
+   // Declare the fragments
+   wmma::fragment<wmma::matrix_a, 16, 16, 16, half, wmma::col_major> a_frag;
+   wmma::fragment<wmma::matrix_b, 16, 16, 16, half, wmma::row_major> b_frag;
+   wmma::fragment<wmma::accumulator, 16, 16, 16, float> c_frag;
+
+   // Initialize the output to zero
+   wmma::fill_fragment(c_frag, 0.0f);
+
+   // Load the inputs
+   wmma::load_matrix_sync(a_frag, a, 16);
+   wmma::load_matrix_sync(b_frag, b, 16);
+
+   // Perform the matrix multiplication
+   wmma::mma_sync(c_frag, a_frag, b_frag, c_frag);
+
+   // Store the output
+   wmma::store_matrix_sync(c, c_frag, 16, wmma::mem_row_major);
+}
+
+
+
+
+
+

+10.25. DPX +

+

DPX is a set of functions that enable finding min and max values, as well as fused addition and min/max, for up to three 16 and 32-bit signed or unsigned integer parameters, with optional ReLU (clamping to zero):

+
    +
  • three parameters: __vimax3_s32, __vimax3_s16x2, __vimax3_u32, __vimax3_u16x2, __vimin3_s32, __vimin3_s16x2, __vimin3_u32, __vimin3_u16x2

  • +
  • two parameters, with ReLU: __vimax_s32_relu, __vimax_s16x2_relu, __vimin_s32_relu, __vimin_s16x2_relu

  • +
  • three parameters, with ReLU: __vimax3_s32_relu, __vimax3_s16x2_relu, __vimin3_s32_relu, __vimin3_s16x2_relu

  • +
  • two parameters, also returning which parameter was smaller/larger: __vibmax_s32, __vibmax_u32, __vibmin_s32, __vibmin_u32, __vibmax_s16x2, __vibmax_u16x2, __vibmin_s16x2, __vibmin_u16x2

  • +
  • three parameters, comparing (first + second) with the third: __viaddmax_s32, __viaddmax_s16x2, __viaddmax_u32, __viaddmax_u16x2, __viaddmin_s32, __viaddmin_s16x2, __viaddmin_u32, __viaddmin_u16x2

  • +
  • three parameters, with ReLU, comparing (first + second) with the third and a zero: __viaddmax_s32_relu, __viaddmax_s16x2_relu, __viaddmin_s32_relu, __viaddmin_s16x2_relu

  • +
+

These instructions are hardware-accelerated on devices with compute capability 9 and higher, and software emulation on older devices.

+

Full API can be found in CUDA Math API documentation.

+

DPX is exceptionally useful when implementing dynamic programming algorithms, such as Smith-Waterman or Needleman–Wunsch in genomics and Floyd-Warshall in route optimization.

+
+

+10.25.1. Examples +

+

Max value of three signed 32-bit integers, with ReLU

+
+
const int a = -15;
+const int b = 8;
+const int c = 5;
+int max_value_0 = __vimax3_s32_relu(a, b, c); // max(-15, 8, 5, 0) = 8
+const int d = -2;
+const int e = -4;
+int max_value_1 = __vimax3_s32_relu(a, d, e); // max(-15, -2, -4, 0) = 0
+
+
+

Min value of the sum of two 32-bit signed integers, another 32-bit signed integer and a zero (ReLU)

+
+
const int a = -5;
+const int b = 6;
+const int c = -2;
+int max_value_0 = __viaddmax_s32_relu(a, b, c); // max(-5 + 6, -2, 0) = max(1, -2, 0) = 1
+const int d = 4;
+int max_value_1 = __viaddmax_s32_relu(a, d, c); // max(-5 + 4, -2, 0) = max(-1, -2, 0) = 0
+
+
+

Min value of two unsigned 32-bit integers and determining which value is smaller

+
+
const unsigned int a = 9;
+const unsigned int b = 6;
+bool smaller_value;
+unsigned int min_value = __vibmin_u32(a, b, &smaller_value); // min_value is 6, smaller_value is true
+
+
+

Max values of three pairs of unsigned 16-bit integers

+
+
const unsigned a = 0x00050002;
+const unsigned b = 0x00070004;
+const unsigned c = 0x00020006;
+unsigned int max_value = __vimax3_u16x2(a, b, c); // max(5, 7, 2) and max(2, 4, 6), so max_value is 0x00070006
+
+
+
+
+
+

+10.26. Asynchronous Barrier +

+

The NVIDIA C++ standard library introduces a GPU implementation of std::barrier. Along with the implementation of std::barrier the library provides extensions that allow users to specify the scope of barrier objects. The barrier API scopes are documented under Thread Scopes. Devices of compute capability 8.0 or higher provide hardware acceleration for barrier operations and integration of these barriers with the memcpy_async feature. On devices with compute capability below 8.0 but starting 7.0, these barriers are available without hardware acceleration.

+

nvcuda::experimental::awbarrier is deprecated in favor of cuda::barrier.

+
+

+10.26.1. Simple Synchronization Pattern +

+

Without the arrive/wait barrier, synchronization is achieved using __syncthreads() (to synchronize all threads in a block) or group.sync() when using Cooperative Groups.

+
+
#include <cooperative_groups.h>
+
+__global__ void simple_sync(int iteration_count) {
+    auto block = cooperative_groups::this_thread_block();
+
+    for (int i = 0; i < iteration_count; ++i) {
+        /* code before arrive */
+        block.sync(); /* wait for all threads to arrive here */
+        /* code after wait */
+    }
+}
+
+
+

Threads are blocked at the synchronization point (block.sync()) until all threads have reached the synchronization point. In addition, memory updates that happened before the synchronization point are guaranteed to be visible to all threads in the block after the synchronization point, i.e., equivalent to atomic_thread_fence(memory_order_seq_cst, thread_scope_block) as well as the sync.

+

This pattern has three stages:

+
    +
  • Code before sync performs memory updates that will be read after the sync.

  • +
  • Synchronization point

  • +
  • Code after sync point with visibility of memory updates that happened before sync point.

  • +
+
+
+

+10.26.2. Temporal Splitting and Five Stages of Synchronization +

+

The temporally-split synchronization pattern with the std::barrier is as follows.

+
+
#include <cuda/barrier>
+#include <cooperative_groups.h>
+
+__device__ void compute(float* data, int curr_iteration);
+
+__global__ void split_arrive_wait(int iteration_count, float *data) {
+    using barrier = cuda::barrier<cuda::thread_scope_block>;
+    __shared__  barrier bar;
+    auto block = cooperative_groups::this_thread_block();
+
+    if (block.thread_rank() == 0) {
+        init(&bar, block.size()); // Initialize the barrier with expected arrival count
+    }
+    block.sync();
+
+    for (int curr_iter = 0; curr_iter < iteration_count; ++curr_iter) {
+        /* code before arrive */
+       barrier::arrival_token token = bar.arrive(); /* this thread arrives. Arrival does not block a thread */
+       compute(data, curr_iter);
+       bar.wait(std::move(token)); /* wait for all threads participating in the barrier to complete bar.arrive()*/
+        /* code after wait */
+    }
+}
+
+
+

In this pattern, the synchronization point (block.sync()) is split into an arrive point (bar.arrive()) and a wait point (bar.wait(std::move(token))). A thread begins participating in a cuda::barrier with its first call to bar.arrive(). When a thread calls bar.wait(std::move(token)) it will be blocked until participating threads have completed bar.arrive() the expected number of times as specified by the expected arrival count argument passed to init(). Memory updates that happen before participating threads’ call to bar.arrive() are guaranteed to be visible to participating threads after their call to bar.wait(std::move(token)). Note that the call to bar.arrive() does not block a thread, it can proceed with other work that does not depend upon memory updates that happen before other participating threads’ call to bar.arrive().

+

The arrive and then wait pattern has five stages which may be iteratively repeated:

+
    +
  • Code before arrive performs memory updates that will be read after the wait.

  • +
  • Arrive point with implicit memory fence (i.e., equivalent to atomic_thread_fence(memory_order_seq_cst, thread_scope_block)).

  • +
  • Code between arrive and wait.

  • +
  • Wait point.

  • +
  • Code after the wait, with visibility of updates that were performed before the arrive.

  • +
+
+
+

+10.26.3. Bootstrap Initialization, Expected Arrival Count, and Participation +

+

Initialization must happen before any thread begins participating in a cuda::barrier.

+
+
#include <cuda/barrier>
+#include <cooperative_groups.h>
+
+__global__ void init_barrier() {
+    __shared__ cuda::barrier<cuda::thread_scope_block> bar;
+    auto block = cooperative_groups::this_thread_block();
+
+    if (block.thread_rank() == 0) {
+        init(&bar, block.size()); // Single thread initializes the total expected arrival count.
+    }
+    block.sync();
+}
+
+
+

Before any thread can participate in cuda::barrier, the barrier must be initialized using init() with an expected arrival count, block.size() in this example. Initialization must happen before any thread calls bar.arrive(). This poses a bootstrapping challenge in that threads must synchronize before participating in the cuda::barrier, but threads are creating a cuda::barrier in order to synchronize. In this example, threads that will participate are part of a cooperative group and use block.sync() to bootstrap initialization. In this example a whole thread block is participating in initialization, hence __syncthreads() could also be used.

+

The second parameter of init() is the expected arrival count, i.e., the number of times bar.arrive() will be called by participating threads before a participating thread is unblocked from its call to bar.wait(std::move(token)). In the prior example the cuda::barrier is initialized with the number of threads in the thread block i.e., cooperative_groups::this_thread_block().size(), and all threads within the thread block participate in the barrier.

+

A cuda::barrier is flexible in specifying how threads participate (split arrive/wait) and which threads participate. In contrast this_thread_block.sync() from cooperative groups or __syncthreads() is applicable to whole-thread-block and __syncwarp(mask) is a specified subset of a warp. If the intention of the user is to synchronize a full thread block or a full warp we recommend using __syncthreads() and __syncwarp(mask) respectively for performance reasons.

+
+
+

+10.26.4. A Barrier’s Phase: Arrival, Countdown, Completion, and Reset +

+

A cuda::barrier counts down from the expected arrival count to zero as participating threads call bar.arrive(). When the countdown reaches zero, a cuda::barrier is complete for the current phase. When the last call to bar.arrive() causes the countdown to reach zero, the countdown is automatically and atomically reset. The reset assigns the countdown to the expected arrival count, and moves the cuda::barrier to the next phase.

+

A token object of class cuda::barrier::arrival_token, as returned from token=bar.arrive(), is associated with the current phase of the barrier. A call to bar.wait(std::move(token)) blocks the calling thread while the cuda::barrier is in the current phase, i.e., while the phase associated with the token matches the phase of the cuda::barrier. If the phase is advanced (because the countdown reaches zero) before the call to bar.wait(std::move(token)) then the thread does not block; if the phase is advanced while the thread is blocked in bar.wait(std::move(token)), the thread is unblocked.

+

It is essential to know when a reset could or could not occur, especially in non-trivial arrive/wait synchronization patterns.

+
    +
  • A thread’s calls to token=bar.arrive() and bar.wait(std::move(token)) must be sequenced such that token=bar.arrive() occurs during the cuda::barrier’s current phase, and bar.wait(std::move(token)) occurs during the same or next phase.

  • +
  • A thread’s call to bar.arrive() must occur when the barrier’s counter is non-zero. After barrier initialization, if a thread’s call to bar.arrive() causes the countdown to reach zero then a call to bar.wait(std::move(token)) must happen before the barrier can be reused for a subsequent call to bar.arrive().

  • +
  • bar.wait() must only be called using a token object of the current phase or the immediately preceding phase. For any other values of the token object, the behavior is undefined.

  • +
+

For simple arrive/wait synchronization patterns, compliance with these usage rules is straightforward.

+
+
+

+10.26.5. Spatial Partitioning (also known as Warp Specialization) +

+

A thread block can be spatially partitioned such that warps are specialized to perform independent computations. Spatial partitioning is used in a producer or consumer pattern, where one subset of threads produces data that is concurrently consumed by the other (disjoint) subset of threads.

+

A producer/consumer spatial partitioning pattern requires two one sided synchronizations to manage a data buffer between the producer and consumer.

+ ++++ + + + + + + + + + + + + + + + + + + + + + + + + +

Producer

Consumer

wait for buffer to be ready to be filled

signal buffer is ready to be filled

produce data and fill the buffer

signal buffer is filled

wait for buffer to be filled

consume data in filled buffer

+

Producer threads wait for consumer threads to signal that the buffer is ready to be filled; however, consumer threads do not wait for this signal. Consumer threads wait for producer threads to signal that the buffer is filled; however, producer threads do not wait for this signal. For full producer/consumer concurrency this pattern has (at least) double buffering where each buffer requires two cuda::barriers.

+
+
#include <cuda/barrier>
+#include <cooperative_groups.h>
+
+using barrier = cuda::barrier<cuda::thread_scope_block>;
+
+__device__ void producer(barrier ready[], barrier filled[], float* buffer, float* in, int N, int buffer_len)
+{
+    for (int i = 0; i < (N/buffer_len); ++i) {
+        ready[i%2].arrive_and_wait(); /* wait for buffer_(i%2) to be ready to be filled */
+        /* produce, i.e., fill in, buffer_(i%2)  */
+        barrier::arrival_token token = filled[i%2].arrive(); /* buffer_(i%2) is filled */
+    }
+}
+
+__device__ void consumer(barrier ready[], barrier filled[], float* buffer, float* out, int N, int buffer_len)
+{
+    barrier::arrival_token token1 = ready[0].arrive(); /* buffer_0 is ready for initial fill */
+    barrier::arrival_token token2 = ready[1].arrive(); /* buffer_1 is ready for initial fill */
+    for (int i = 0; i < (N/buffer_len); ++i) {
+        filled[i%2].arrive_and_wait(); /* wait for buffer_(i%2) to be filled */
+        /* consume buffer_(i%2) */
+        barrier::arrival_token token = ready[i%2].arrive(); /* buffer_(i%2) is ready to be re-filled */
+    }
+}
+
+//N is the total number of float elements in arrays in and out
+__global__ void producer_consumer_pattern(int N, int buffer_len, float* in, float* out) {
+
+    // Shared memory buffer declared below is of size 2 * buffer_len
+    // so that we can alternatively work between two buffers.
+    // buffer_0 = buffer and buffer_1 = buffer + buffer_len
+    __shared__ extern float buffer[];
+
+    // bar[0] and bar[1] track if buffers buffer_0 and buffer_1 are ready to be filled,
+    // while bar[2] and bar[3] track if buffers buffer_0 and buffer_1 are filled-in respectively
+    __shared__ barrier bar[4];
+
+
+    auto block = cooperative_groups::this_thread_block();
+    if (block.thread_rank() < 4)
+        init(bar + block.thread_rank(), block.size());
+    block.sync();
+
+    if (block.thread_rank() < warpSize)
+        producer(bar, bar+2, buffer, in, N, buffer_len);
+    else
+        consumer(bar, bar+2, buffer, out, N, buffer_len);
+}
+
+
+

In this example the first warp is specialized as the producer and the remaining warps are specialized as the consumer. All producer and consumer threads participate (call bar.arrive() or bar.arrive_and_wait()) in each of the four cuda::barriers so the expected arrival counts are equal to block.size().

+

A producer thread waits for the consumer threads to signal that the shared memory buffer can be filled. In order to wait for a cuda::barrier a producer thread must first arrive on that ready[i%2].arrive() to get a token and then ready[i%2].wait(token) with that token. For simplicity ready[i%2].arrive_and_wait() combines these operations.

+
+
bar.arrive_and_wait();
+/* is equivalent to */
+bar.wait(bar.arrive());
+
+
+

Producer threads compute and fill the ready buffer, they then signal that the buffer is filled by arriving on the filled barrier, filled[i%2].arrive(). A producer thread does not wait at this point, instead it waits until the next iteration’s buffer (double buffering) is ready to be filled.

+

A consumer thread begins by signaling that both buffers are ready to be filled. A consumer thread does not wait at this point, instead it waits for this iteration’s buffer to be filled, filled[i%2].arrive_and_wait(). After the consumer threads consume the buffer they signal that the buffer is ready to be filled again, ready[i%2].arrive(), and then wait for the next iteration’s buffer to be filled.

+
+
+

+10.26.6. Early Exit (Dropping out of Participation) +

+

When a thread that is participating in a sequence of synchronizations must exit early from that sequence, that thread must explicitly drop out of participation before exiting. The remaining participating threads can proceed normally with subsequent cuda::barrier arrive and wait operations.

+
+
#include <cuda/barrier>
+#include <cooperative_groups.h>
+
+__device__ bool condition_check();
+
+__global__ void early_exit_kernel(int N) {
+    using barrier = cuda::barrier<cuda::thread_scope_block>;
+    __shared__ barrier bar;
+    auto block = cooperative_groups::this_thread_block();
+
+    if (block.thread_rank() == 0)
+        init(&bar , block.size());
+    block.sync();
+
+    for (int i = 0; i < N; ++i) {
+        if (condition_check()) {
+          bar.arrive_and_drop();
+          return;
+        }
+        /* other threads can proceed normally */
+        barrier::arrival_token token = bar.arrive();
+        /* code between arrive and wait */
+        bar.wait(std::move(token)); /* wait for all threads to arrive */
+        /* code after wait */
+    }
+}
+
+
+

This operation arrives on the cuda::barrier to fulfill the participating thread’s obligation to arrive in the current phase, and then decrements the expected arrival count for the next phase so that this thread is no longer expected to arrive on the barrier.

+
+
+

+10.26.7. Completion Function +

+

The CompletionFunction of cuda::barrier<Scope, CompletionFunction> is executed once per phase, after the last thread arrives and before any thread is unblocked from the wait. Memory operations performed by the threads that arrived at the barrier during the phase are visible to the thread executing the CompletionFunction, and all memory operations performed within the CompletionFunction are visible to all threads waiting at the barrier once they are unblocked from the wait.

+
+
#include <cuda/barrier>
+#include <cooperative_groups.h>
+#include <functional>
+namespace cg = cooperative_groups;
+
+__device__ int divergent_compute(int*, int);
+__device__ int independent_computation(int*, int);
+
+__global__ void psum(int* data, int n, int* acc) {
+  auto block = cg::this_thread_block();
+
+  constexpr int BlockSize = 128;
+  __shared__ int smem[BlockSize];
+  assert(BlockSize == block.size());
+  assert(n % 128 == 0);
+
+  auto completion_fn = [&] {
+    int sum = 0;
+    for (int i = 0; i < 128; ++i) sum += smem[i];
+    *acc += sum;
+  };
+
+  // Barrier storage
+  // Note: the barrier is not default-constructible because
+  //       completion_fn is not default-constructible due
+  //       to the capture.
+  using completion_fn_t = decltype(completion_fn);
+  using barrier_t = cuda::barrier<cuda::thread_scope_block,
+                                  completion_fn_t>;
+  __shared__ std::aligned_storage<sizeof(barrier_t),
+                                  alignof(barrier_t)> bar_storage;
+
+  // Initialize barrier:
+  barrier_t* bar = (barrier_t*)&bar_storage;
+  if (block.thread_rank() == 0) {
+    assert(*acc == 0);
+    assert(blockDim.x == blockDim.y == blockDim.y == 1);
+    new (bar) barrier_t{block.size(), completion_fn};
+    // equivalent to: init(bar, block.size(), completion_fn);
+  }
+  block.sync();
+
+  // Main loop
+  for (int i = 0; i < n; i += block.size()) {
+    smem[block.thread_rank()] = data[i] + *acc;
+    auto t = bar->arrive();
+    // We can do independent computation here
+    bar->wait(std::move(t));
+    // shared-memory is safe to re-use in the next iteration
+    // since all threads are done with it, including the one
+    // that did the reduction
+  }
+}
+
+
+
+
+

+10.26.8. Memory Barrier Primitives Interface +

+

Memory barrier primitives are C-like interfaces to cuda::barrier functionality. These primitives are available through including the <cuda_awbarrier_primitives.h> header.

+
+

+10.26.8.1. Data Types +

+
+
typedef /* implementation defined */ __mbarrier_t;
+typedef /* implementation defined */ __mbarrier_token_t;
+
+
+
+
+

+10.26.8.2. Memory Barrier Primitives API +

+
+
uint32_t __mbarrier_maximum_count();
+void __mbarrier_init(__mbarrier_t* bar, uint32_t expected_count);
+
+
+
    +
  • bar must be a pointer to __shared__ memory.

  • +
  • expected_count <= __mbarrier_maximum_count()

  • +
  • Initialize *bar expected arrival count for the current and next phase to expected_count.

  • +
+
+
void __mbarrier_inval(__mbarrier_t* bar);
+
+
+
    +
  • bar must be a pointer to the mbarrier object residing in shared memory.

  • +
  • Invalidation of *bar is required before the corresponding shared memory can be repurposed.

  • +
+
+
__mbarrier_token_t __mbarrier_arrive(__mbarrier_t* bar);
+
+
+
    +
  • Initialization of *bar must happen before this call.

  • +
  • Pending count must not be zero.

  • +
  • Atomically decrement the pending count for the current phase of the barrier.

  • +
  • Return an arrival token associated with the barrier state immediately prior to the decrement.

  • +
+
+
__mbarrier_token_t __mbarrier_arrive_and_drop(__mbarrier_t* bar);
+
+
+
    +
  • Initialization of *bar must happen before this call.

  • +
  • Pending count must not be zero.

  • +
  • Atomically decrement the pending count for the current phase and expected count for the next phase of the barrier.

  • +
  • Return an arrival token associated with the barrier state immediately prior to the decrement.

  • +
+
+
bool __mbarrier_test_wait(__mbarrier_t* bar, __mbarrier_token_t token);
+
+
+
    +
  • token must be associated with the immediately preceding phase or current phase of *this.

  • +
  • Returns true if token is associated with the immediately preceding phase of *bar, otherwise returns false.

  • +
+
+
//Note: This API has been deprecated in CUDA 11.1
+uint32_t __mbarrier_pending_count(__mbarrier_token_t token);
+
+
+
+
+
+
+

+10.27. Asynchronous Data Copies +

+

CUDA 11 introduces Asynchronous Data operations with memcpy_async API to allow device code to explicitly manage the asynchronous copying of data. The memcpy_async feature enables CUDA kernels to overlap computation with data movement.

+
+

+10.27.1. memcpy_async API +

+

The memcpy_async APIs are provided in the cuda/barrier, cuda/pipeline, and cooperative_groups/memcpy_async.h header files.

+

The cuda::memcpy_async APIs work with cuda::barrier and cuda::pipeline synchronization primitives, while the cooperative_groups::memcpy_async synchronizes using cooperative_groups::wait.

+

These APIs have very similar semantics: copy objects from src to dst as-if performed by another thread which, on completion of the copy, can be synchronized through cuda::pipeline, cuda::barrier, or cooperative_groups::wait.

+

The complete API documentation of the cuda::memcpy_async overloads for cuda::barrier and cuda::pipeline is provided in the libcudacxx API documentation along with some examples.

+

The API documentation of cooperative_groups::memcpy_async is provided in the Cooperative Groups section.

+

The memcpy_async APIs that use cuda::barrier and cuda::pipeline require compute capability 7.0 or higher. On devices with compute capability 8.0 or higher, memcpy_async operations from global to shared memory can benefit from hardware acceleration.

+
+
+

+10.27.2. Copy and Compute Pattern - Staging Data Through Shared Memory +

+

CUDA applications often employ a copy and compute pattern that:

+
    +
  • fetches data from global memory,

  • +
  • stores data to shared memory, and

  • +
  • performs computations on shared memory data, and potentially writes results back to global memory.

  • +
+

The following sections illustrate how this pattern can be expressed without and with the memcpy_async feature:

+ +
+
+

+10.27.3. Without memcpy_async +

+

Without memcpy_async, the copy phase of the copy and compute pattern is expressed as shared[local_idx] = global[global_idx]. This global to shared memory copy is expanded to a read from global memory into a register, followed by a write to shared memory from the register.

+

When this pattern occurs within an iterative algorithm, each thread block needs to synchronize after the shared[local_idx] = global[global_idx] assignment, to ensure all writes to shared memory have completed before the compute phase can begin. The thread block also needs to synchronize again after the compute phase, to prevent overwriting shared memory before all threads have completed their computations. This pattern is illustrated in the following code snippet.

+
+
#include <cooperative_groups.h>
+__device__ void compute(int* global_out, int const* shared_in) {
+    // Computes using all values of current batch from shared memory.
+    // Stores this thread's result back to global memory.
+}
+
+__global__ void without_memcpy_async(int* global_out, int const* global_in, size_t size, size_t batch_sz) {
+  auto grid = cooperative_groups::this_grid();
+  auto block = cooperative_groups::this_thread_block();
+  assert(size == batch_sz * grid.size()); // Exposition: input size fits batch_sz * grid_size
+
+  extern __shared__ int shared[]; // block.size() * sizeof(int) bytes
+
+  size_t local_idx = block.thread_rank();
+
+  for (size_t batch = 0; batch < batch_sz; ++batch) {
+    // Compute the index of the current batch for this block in global memory:
+    size_t block_batch_idx = block.group_index().x * block.size() + grid.size() * batch;
+    size_t global_idx = block_batch_idx + threadIdx.x;
+    shared[local_idx] = global_in[global_idx];
+
+    block.sync(); // Wait for all copies to complete
+
+    compute(global_out + block_batch_idx, shared); // Compute and write result to global memory
+
+    block.sync(); // Wait for compute using shared memory to finish
+  }
+}
+
+
+
+
+

+10.27.4. With memcpy_async +

+

With memcpy_async, the assignment of shared memory from global memory

+
+
shared[local_idx] = global_in[global_idx];
+
+
+

is replaced with an asynchronous copy operation from cooperative groups

+
+
cooperative_groups::memcpy_async(group, shared, global_in + batch_idx, sizeof(int) * block.size());
+
+
+

The cooperative_groups::memcpy_async API copies sizeof(int) * block.size() bytes from global memory starting at global_in + batch_idx to the shared data. This operation happens as-if performed by another thread, which synchronizes with the current thread’s call to cooperative_groups::wait after the copy has completed. Until the copy operation completes, modifying the global data or reading or writing the shared data introduces a data race.

+

On devices with compute capability 8.0 or higher, memcpy_async transfers from global to shared memory can benefit from hardware acceleration, which avoids transferring the data through an intermediate register.

+
+
#include <cooperative_groups.h>
+#include <cooperative_groups/memcpy_async.h>
+
+__device__ void compute(int* global_out, int const* shared_in);
+
+__global__ void with_memcpy_async(int* global_out, int const* global_in, size_t size, size_t batch_sz) {
+  auto grid = cooperative_groups::this_grid();
+  auto block = cooperative_groups::this_thread_block();
+  assert(size == batch_sz * grid.size()); // Exposition: input size fits batch_sz * grid_size
+
+  extern __shared__ int shared[]; // block.size() * sizeof(int) bytes
+
+  for (size_t batch = 0; batch < batch_sz; ++batch) {
+    size_t block_batch_idx = block.group_index().x * block.size() + grid.size() * batch;
+    // Whole thread-group cooperatively copies whole batch to shared memory:
+    cooperative_groups::memcpy_async(block, shared, global_in + block_batch_idx, sizeof(int) * block.size());
+
+    cooperative_groups::wait(block); // Joins all threads, waits for all copies to complete
+
+    compute(global_out + block_batch_idx, shared);
+
+    block.sync();
+  }
+}}
+
+
+
+
+

+10.27.5. Asynchronous Data Copies using cuda::barrier +

+

The cuda::memcpy_async overload for cuda::barrier enables synchronizing asynchronous data transfers using a barrier. This overloads executes the copy operation as-if performed by another thread bound to the barrier by: incrementing the expected count of the current phase on creation, and decrementing it on completion of the copy operation, such that the phase of the barrier will only advance when all threads participating in the barrier have arrived, and all memcpy_async bound to the current phase of the barrier have completed. The following example uses a block-wide barrier, where all block threads participate, and swaps the wait operation with a barrier arrive_and_wait, while providing the same functionality as the previous example:

+
+
#include <cooperative_groups.h>
+#include <cuda/barrier>
+__device__ void compute(int* global_out, int const* shared_in);
+
+__global__ void with_barrier(int* global_out, int const* global_in, size_t size, size_t batch_sz) {
+  auto grid = cooperative_groups::this_grid();
+  auto block = cooperative_groups::this_thread_block();
+  assert(size == batch_sz * grid.size()); // Assume input size fits batch_sz * grid_size
+
+  extern __shared__ int shared[]; // block.size() * sizeof(int) bytes
+
+  // Create a synchronization object (C++20 barrier)
+  __shared__ cuda::barrier<cuda::thread_scope::thread_scope_block> barrier;
+  if (block.thread_rank() == 0) {
+    init(&barrier, block.size()); // Friend function initializes barrier
+  }
+  block.sync();
+
+  for (size_t batch = 0; batch < batch_sz; ++batch) {
+    size_t block_batch_idx = block.group_index().x * block.size() + grid.size() * batch;
+    cuda::memcpy_async(block, shared, global_in + block_batch_idx, sizeof(int) * block.size(), barrier);
+
+    barrier.arrive_and_wait(); // Waits for all copies to complete
+
+    compute(global_out + block_batch_idx, shared);
+
+    block.sync();
+  }
+}
+
+
+
+
+

+10.27.6. Performance Guidance for memcpy_async +

+

For compute capability 8.x, the pipeline mechanism is shared among CUDA threads in the same CUDA warp. This sharing causes batches of memcpy_async to be entangled within a warp, which can impact performance under certain circumstances.

+

This section highlights the warp-entanglement effect on commit, wait, and arrive operations. Please refer to Pipeline Interface and the Pipeline Primitives Interface for an overview of the individual operations.

+
+

+10.27.6.1. Alignment +

+

On devices with compute capability 8.0, the cp.async family of instructions allows copying data from global to shared memory asynchronously. These instructions support copying 4, 8, and 16 bytes at a time. If the size provided to memcpy_async is a multiple of 4, 8, or 16, and both pointers passed to memcpy_async are aligned to a 4, 8, or 16 alignment boundary, then memcpy_async can be implemented using exclusively asynchronous memory operations.

+

Additionally for achieving best performance when using memcpy_async API, an alignment of 128 Bytes for both shared memory and global memory is required.

+

For pointers to values of types with an alignment requirement of 1 or 2, it is often not possible to prove that the pointers are always aligned to a higher alignment boundary. Determining whether the cp.async instructions can or cannot be used must be delayed until run-time. Performing such a runtime alignment check increases code-size and adds runtime overhead.

+

The cuda::aligned_size_t<size_t Align>(size_t size)Shape can be used to supply a proof that both pointers passed to memcpy_async are aligned to an Align alignment boundary and that size is a multiple of Align, by passing it as an argument where the memcpy_async APIs expect a Shape:

+
+
cuda::memcpy_async(group, dst, src, cuda::aligned_size_t<16>(N * block.size()), pipeline);
+
+
+

If the proof is incorrect, the behavior is undefined.

+
+
+

+10.27.6.2. Trivially copyable +

+

On devices with compute capability 8.0, the cp.async family of instructions allows copying data from global to shared memory asynchronously. If the pointer types passed to memcpy_async do not point to TriviallyCopyable types, the copy constructor of each output element needs to be invoked, and these instructions cannot be used to accelerate memcpy_async.

+
+
+

+10.27.6.3. Warp Entanglement - Commit +

+

The sequence of memcpy_async batches is shared across the warp. The commit operation is coalesced such that the sequence is incremented once for all converged threads that invoke the commit operation. If the warp is fully converged, the sequence is incremented by one; if the warp is fully diverged, the sequence is incremented by 32.

+
    +
  • +

    Let PB be the warp-shared pipeline’s actual sequence of batches.

    +

    PB = {BP0, BP1, BP2, …, BPL}

    +
  • +
  • +

    Let TB be a thread’s perceived sequence of batches, as if the sequence were only incremented by this thread’s invocation of the commit operation.

    +

    TB = {BT0, BT1, BT2, …, BTL}

    +

    The pipeline::producer_commit() return value is from the thread’s perceived batch sequence.

    +
  • +
  • +

    An index in a thread’s perceived sequence always aligns to an equal or larger index in the actual warp-shared sequence. The sequences are equal only when all commit operations are invoked from converged threads.

    +

    BTn ≡ BPm where n <= m

    +
  • +
+

For example, when a warp is fully diverged:

+
    +
  • The warp-shared pipeline’s actual sequence would be: PB = {0, 1, 2, 3, ..., 31} (PL=31).

  • +
  • +

    The perceived sequence for each thread of this warp would be:

    +
      +
    • Thread 0: TB = {0} (TL=0)

    • +
    • Thread 1: TB = {0} (TL=0)

    • +
    • …

    • +
    • Thread 31: TB = {0} (TL=0)

    • +
    +
  • +
+
+
+

+10.27.6.4. Warp Entanglement - Wait +

+

A CUDA thread invokes either pipeline_consumer_wait_prior<N>() or pipeline::consumer_wait() to wait for batches in the perceived sequence TB to complete. Note that pipeline::consumer_wait() is equivalent to pipeline_consumer_wait_prior<N>(), where N =                                        PL.

+

The pipeline_consumer_wait_prior<N>() function waits for batches in the actual sequence at least up to and including PL-N. Since TL <= PL, waiting for batch up to and including PL-N includes waiting for batch TL-N. Thus, when TL < PL, the thread will unintentionally wait for additional, more recent batches.

+

In the extreme fully-diverged warp example above, each thread could wait for all 32 batches.

+
+
+

+10.27.6.5. Warp Entanglement - Arrive-On +

+

Warp-divergence affects the number of times an arrive_on(bar) operation updates the barrier. If the invoking warp is fully converged, then the barrier is updated once. If the invoking warp is fully diverged, then 32 individual updates are applied to the barrier.

+
+
+

+10.27.6.6. Keep Commit and Arrive-On Operations Converged +

+

It is recommended that commit and arrive-on invocations are by converged threads:

+
    +
  • to not over-wait, by keeping threads’ perceived sequence of batches aligned with the actual sequence, and

  • +
  • to minimize updates to the barrier object.

  • +
+

When code preceding these operations diverges threads, then the warp should be re-converged, via __syncwarp before invoking commit or arrive-on operations.

+
+
+
+
+

+10.28. Asynchronous Data Copies using cuda::pipeline +

+

CUDA provides the cuda::pipeline synchronization object to manage and overlap asynchronous data movement with computation.

+

The API documentation for cuda::pipeline is provided in the libcudacxx API. A pipeline object is a double-ended N stage queue with a head and a tail, and is used to process work in a first-in first-out (FIFO) order. The pipeline object has following member functions to manage the stages of the pipeline.

+ ++++ + + + + + + + + + + + + + + + + + + + + + + + + +

Pipeline Class Member Function

Description

producer_acquire

Acquires an available stage in the pipeline internal queue.

producer_commit

Commits the asynchronous operations issued after the producer_acquire call on the currently acquired stage of the pipeline.

consumer_wait

Wait for completion of all asynchronous operations on the oldest stage of the pipeline.

consumer_release

Release the oldest stage of the pipeline to the pipeline object for reuse. The released stage can be then acquired by the producer.

+
+

+10.28.1. Single-Stage Asynchronous Data Copies using cuda::pipeline +

+

In previous examples we showed how to use cooperative_groups and cuda::barrier to do asynchronous data transfers. In this section, we will use the cuda::pipeline API with a single stage to schedule asynchronous copies. And later we will expand this example to show multi staged overlapped compute and copy.

+
+
#include <cooperative_groups/memcpy_async.h>
+#include <cuda/pipeline>
+
+__device__ void compute(int* global_out, int const* shared_in);
+__global__ void with_single_stage(int* global_out, int const* global_in, size_t size, size_t batch_sz) {
+    auto grid = cooperative_groups::this_grid();
+    auto block = cooperative_groups::this_thread_block();
+    assert(size == batch_sz * grid.size()); // Assume input size fits batch_sz * grid_size
+
+    constexpr size_t stages_count = 1; // Pipeline with one stage
+    // One batch must fit in shared memory:
+    extern __shared__ int shared[];  // block.size() * sizeof(int) bytes
+
+    // Allocate shared storage for a single stage cuda::pipeline:
+    __shared__ cuda::pipeline_shared_state<
+        cuda::thread_scope::thread_scope_block,
+        stages_count
+    > shared_state;
+    auto pipeline = cuda::make_pipeline(block, &shared_state);
+
+    // Each thread processes `batch_sz` elements.
+    // Compute offset of the batch `batch` of this thread block in global memory:
+    auto block_batch = [&](size_t batch) -> int {
+      return block.group_index().x * block.size() + grid.size() * batch;
+    };
+
+    for (size_t batch = 0; batch < batch_sz; ++batch) {
+        size_t global_idx = block_batch(batch);
+
+        // Collectively acquire the pipeline head stage from all producer threads:
+        pipeline.producer_acquire();
+
+        // Submit async copies to the pipeline's head stage to be
+        // computed in the next loop iteration
+        cuda::memcpy_async(block, shared, global_in + global_idx, sizeof(int) * block.size(), pipeline);
+        // Collectively commit (advance) the pipeline's head stage
+        pipeline.producer_commit();
+
+        // Collectively wait for the operations committed to the
+        // previous `compute` stage to complete:
+        pipeline.consumer_wait();
+
+        // Computation overlapped with the memcpy_async of the "copy" stage:
+        compute(global_out + global_idx, shared);
+
+        // Collectively release the stage resources
+        pipeline.consumer_release();
+    }
+}
+
+
+
+
+

+10.28.2. Multi-Stage Asynchronous Data Copies using cuda::pipeline +

+

In the previous examples with cooperative_groups::wait and cuda::barrier, the kernel threads immediately wait for the data transfer to shared memory to complete. This avoids data transfers from global memory into registers, but does not hide the latency of the memcpy_async operation by overlapping computation.

+

For that we use the CUDA pipeline feature in the following example. It provides a mechanism for managing a sequence of memcpy_async batches, enabling CUDA kernels to overlap memory transfers with computation. The following example implements a two-stage pipeline that overlaps data-transfer with computation. It:

+
    +
  • Initializes the pipeline shared state (more below)

  • +
  • Kickstarts the pipeline by scheduling a memcpy_async for the first batch.

  • +
  • Loops over all the batches: it schedules memcpy_async for the next batch, blocks all threads on the completion of the memcpy_async for the previous batch, and then overlaps the computation on the previous batch with the asynchronous copy of the memory for the next batch.

  • +
  • Finally, it drains the pipeline by performing the computation on the last batch.

  • +
+

Note that, for interoperability with cuda::pipeline, cuda::memcpy_async from the cuda/pipeline header is used here.

+
+
#include <cooperative_groups/memcpy_async.h>
+#include <cuda/pipeline>
+
+__device__ void compute(int* global_out, int const* shared_in);
+__global__ void with_staging(int* global_out, int const* global_in, size_t size, size_t batch_sz) {
+    auto grid = cooperative_groups::this_grid();
+    auto block = cooperative_groups::this_thread_block();
+    assert(size == batch_sz * grid.size()); // Assume input size fits batch_sz * grid_size
+
+    constexpr size_t stages_count = 2; // Pipeline with two stages
+    // Two batches must fit in shared memory:
+    extern __shared__ int shared[];  // stages_count * block.size() * sizeof(int) bytes
+    size_t shared_offset[stages_count] = { 0, block.size() }; // Offsets to each batch
+
+    // Allocate shared storage for a two-stage cuda::pipeline:
+    __shared__ cuda::pipeline_shared_state<
+        cuda::thread_scope::thread_scope_block,
+        stages_count
+    > shared_state;
+    auto pipeline = cuda::make_pipeline(block, &shared_state);
+
+    // Each thread processes `batch_sz` elements.
+    // Compute offset of the batch `batch` of this thread block in global memory:
+    auto block_batch = [&](size_t batch) -> int {
+      return block.group_index().x * block.size() + grid.size() * batch;
+    };
+
+    // Initialize first pipeline stage by submitting a `memcpy_async` to fetch a whole batch for the block:
+    if (batch_sz == 0) return;
+    pipeline.producer_acquire();
+    cuda::memcpy_async(block, shared + shared_offset[0], global_in + block_batch(0), sizeof(int) * block.size(), pipeline);
+    pipeline.producer_commit();
+
+    // Pipelined copy/compute:
+    for (size_t batch = 1; batch < batch_sz; ++batch) {
+        // Stage indices for the compute and copy stages:
+        size_t compute_stage_idx = (batch - 1) % 2;
+        size_t copy_stage_idx = batch % 2;
+
+        size_t global_idx = block_batch(batch);
+
+        // Collectively acquire the pipeline head stage from all producer threads:
+        pipeline.producer_acquire();
+
+        // Submit async copies to the pipeline's head stage to be
+        // computed in the next loop iteration
+        cuda::memcpy_async(block, shared + shared_offset[copy_stage_idx], global_in + global_idx, sizeof(int) * block.size(), pipeline);
+        // Collectively commit (advance) the pipeline's head stage
+        pipeline.producer_commit();
+
+        // Collectively wait for the operations committed to the
+        // previous `compute` stage to complete:
+        pipeline.consumer_wait();
+
+        // Computation overlapped with the memcpy_async of the "copy" stage:
+        compute(global_out + global_idx, shared + shared_offset[compute_stage_idx]);
+
+        // Collectively release the stage resources
+        pipeline.consumer_release();
+    }
+
+    // Compute the data fetch by the last iteration
+    pipeline.consumer_wait();
+    compute(global_out + block_batch(batch_sz-1), shared + shared_offset[(batch_sz - 1) % 2]);
+    pipeline.consumer_release();
+}
+
+
+

A pipeline object is a double-ended queue with a head and a tail, and is used to process work in a first-in first-out (FIFO) order. Producer threads commit work to the pipeline’s head, while consumer threads pull work from the pipeline’s tail. In the example above, all threads are both producer and consumer threads. The threads first commitmemcpy_async operations to fetch the next batch while they wait on the previous batch of memcpy_async operations to complete.

+
    +
  • +

    Committing work to a pipeline stage involves:

    +
      +
    • Collectively acquiring the pipeline head from a set of producer threads using pipeline.producer_acquire().

    • +
    • Submitting memcpy_async operations to the pipeline head.

    • +
    • Collectively committing (advancing) the pipeline head using pipeline.producer_commit().

    • +
    +
  • +
  • +

    Using a previously commited stage involves:

    +
      +
    • Collectively waiting for the stage to complete, e.g., using pipeline.consumer_wait() to wait on the tail (oldest) stage.

    • +
    • Collectively releasing the stage using pipeline.consumer_release().

    • +
    +
  • +
+

cuda::pipeline_shared_state<scope, count> encapsulates the finite resources that allow a pipeline to process up to count concurrent stages. If all resources are in use, pipeline.producer_acquire() blocks producer threads until the resources of the next pipeline stage are released by consumer threads.

+

This example can be written in a more concise manner by merging the prolog and epilog of the loop with the loop itself as follows:

+
+
template <size_t stages_count = 2 /* Pipeline with stages_count stages */>
+__global__ void with_staging_unified(int* global_out, int const* global_in, size_t size, size_t batch_sz) {
+    auto grid = cooperative_groups::this_grid();
+    auto block = cooperative_groups::this_thread_block();
+    assert(size == batch_sz * grid.size()); // Assume input size fits batch_sz * grid_size
+
+    extern __shared__ int shared[]; // stages_count * block.size() * sizeof(int) bytes
+    size_t shared_offset[stages_count];
+    for (int s = 0; s < stages_count; ++s) shared_offset[s] = s * block.size();
+
+    __shared__ cuda::pipeline_shared_state<
+        cuda::thread_scope::thread_scope_block,
+        stages_count
+    > shared_state;
+    auto pipeline = cuda::make_pipeline(block, &shared_state);
+
+    auto block_batch = [&](size_t batch) -> int {
+        return block.group_index().x * block.size() + grid.size() * batch;
+    };
+
+    // compute_batch: next batch to process
+    // fetch_batch:  next batch to fetch from global memory
+    for (size_t compute_batch = 0, fetch_batch = 0; compute_batch < batch_sz; ++compute_batch) {
+        // The outer loop iterates over the computation of the batches
+        for (; fetch_batch < batch_sz && fetch_batch < (compute_batch + stages_count); ++fetch_batch) {
+            // This inner loop iterates over the memory transfers, making sure that the pipeline is always full
+            pipeline.producer_acquire();
+            size_t shared_idx = fetch_batch % stages_count;
+            size_t batch_idx = fetch_batch;
+            size_t block_batch_idx = block_batch(batch_idx);
+            cuda::memcpy_async(block, shared + shared_offset[shared_idx], global_in + block_batch_idx, sizeof(int) * block.size(), pipeline);
+            pipeline.producer_commit();
+        }
+        pipeline.consumer_wait();
+        int shared_idx = compute_batch % stages_count;
+        int batch_idx = compute_batch;
+        compute(global_out + block_batch(batch_idx), shared + shared_offset[shared_idx]);
+        pipeline.consumer_release();
+    }
+}
+
+
+

The pipeline<thread_scope_block> primitive used above is very flexible, and supports two features that our examples above are not using: any arbitrary subset of threads in the block can participate in the pipeline, and from the threads that participate, any subsets can be producers, consumers, or both. In the following example, threads with an “even” thread rank are producers, while other threads are consumers:

+
+
__device__ void compute(int* global_out, int shared_in);
+
+template <size_t stages_count = 2>
+__global__ void with_specialized_staging_unified(int* global_out, int const* global_in, size_t size, size_t batch_sz) {
+    auto grid = cooperative_groups::this_grid();
+    auto block = cooperative_groups::this_thread_block();
+
+    // In this example, threads with "even" thread rank are producers, while threads with "odd" thread rank are consumers:
+    const cuda::pipeline_role thread_role
+      = block.thread_rank() % 2 == 0? cuda::pipeline_role::producer : cuda::pipeline_role::consumer;
+
+    // Each thread block only has half of its threads as producers:
+    auto producer_threads = block.size() / 2;
+
+    // Map adjacent even and odd threads to the same id:
+    const int thread_idx = block.thread_rank() / 2;
+
+    auto elements_per_batch = size / batch_sz;
+    auto elements_per_batch_per_block = elements_per_batch / grid.group_dim().x;
+
+    extern __shared__ int shared[]; // stages_count * elements_per_batch_per_block * sizeof(int) bytes
+    size_t shared_offset[stages_count];
+    for (int s = 0; s < stages_count; ++s) shared_offset[s] = s * elements_per_batch_per_block;
+
+    __shared__ cuda::pipeline_shared_state<
+        cuda::thread_scope::thread_scope_block,
+        stages_count
+    > shared_state;
+    cuda::pipeline pipeline = cuda::make_pipeline(block, &shared_state, thread_role);
+
+    // Each thread block processes `batch_sz` batches.
+    // Compute offset of the batch `batch` of this thread block in global memory:
+    auto block_batch = [&](size_t batch) -> int {
+      return elements_per_batch * batch + elements_per_batch_per_block * blockIdx.x;
+    };
+
+    for (size_t compute_batch = 0, fetch_batch = 0; compute_batch < batch_sz; ++compute_batch) {
+        // The outer loop iterates over the computation of the batches
+        for (; fetch_batch < batch_sz && fetch_batch < (compute_batch + stages_count); ++fetch_batch) {
+            // This inner loop iterates over the memory transfers, making sure that the pipeline is always full
+            if (thread_role == cuda::pipeline_role::producer) {
+                // Only the producer threads schedule asynchronous memcpys:
+                pipeline.producer_acquire();
+                size_t shared_idx = fetch_batch % stages_count;
+                size_t batch_idx = fetch_batch;
+                size_t global_batch_idx = block_batch(batch_idx) + thread_idx;
+                size_t shared_batch_idx = shared_offset[shared_idx] + thread_idx;
+                cuda::memcpy_async(shared + shared_batch_idx, global_in + global_batch_idx, sizeof(int), pipeline);
+                pipeline.producer_commit();
+            }
+        }
+        if (thread_role == cuda::pipeline_role::consumer) {
+            // Only the consumer threads compute:
+            pipeline.consumer_wait();
+            size_t shared_idx = compute_batch % stages_count;
+            size_t global_batch_idx = block_batch(compute_batch) + thread_idx;
+            size_t shared_batch_idx = shared_offset[shared_idx] + thread_idx;
+            compute(global_out + global_batch_idx, *(shared + shared_batch_idx));
+            pipeline.consumer_release();
+        }
+    }
+}
+
+
+

There are some optimizations that pipeline performs, for example, when all threads are both producers and consumers, but in general, the cost of supporting all these features cannot be fully eliminated. For example, pipeline stores and uses a set of barriers in shared memory for synchronization, which is not really necessary if all threads in the block participate in the pipeline.

+

For the particular case in which all threads in the block participate in the pipeline, we can do better than pipeline<thread_scope_block> by using a pipeline<thread_scope_thread> combined with __syncthreads():

+
+
template<size_t stages_count>
+__global__ void with_staging_scope_thread(int* global_out, int const* global_in, size_t size, size_t batch_sz) {
+    auto grid = cooperative_groups::this_grid();
+    auto block = cooperative_groups::this_thread_block();
+    auto thread = cooperative_groups::this_thread();
+    assert(size == batch_sz * grid.size()); // Assume input size fits batch_sz * grid_size
+
+    extern __shared__ int shared[]; // stages_count * block.size() * sizeof(int) bytes
+    size_t shared_offset[stages_count];
+    for (int s = 0; s < stages_count; ++s) shared_offset[s] = s * block.size();
+
+    // No pipeline::shared_state needed
+    cuda::pipeline<cuda::thread_scope_thread> pipeline = cuda::make_pipeline();
+
+    auto block_batch = [&](size_t batch) -> int {
+        return block.group_index().x * block.size() + grid.size() * batch;
+    };
+
+    for (size_t compute_batch = 0, fetch_batch = 0; compute_batch < batch_sz; ++compute_batch) {
+        for (; fetch_batch < batch_sz && fetch_batch < (compute_batch + stages_count); ++fetch_batch) {
+            pipeline.producer_acquire();
+            size_t shared_idx = fetch_batch % stages_count;
+            size_t batch_idx = fetch_batch;
+            // Each thread fetches its own data:
+            size_t thread_batch_idx = block_batch(batch_idx) + threadIdx.x;
+            // The copy is performed by a single `thread` and the size of the batch is now that of a single element:
+            cuda::memcpy_async(thread, shared + shared_offset[shared_idx] + threadIdx.x, global_in + thread_batch_idx, sizeof(int), pipeline);
+            pipeline.producer_commit();
+        }
+        pipeline.consumer_wait();
+        block.sync(); // __syncthreads: All memcpy_async of all threads in the block for this stage have completed here
+        int shared_idx = compute_batch % stages_count;
+        int batch_idx = compute_batch;
+        compute(global_out + block_batch(batch_idx), shared + shared_offset[shared_idx]);
+        pipeline.consumer_release();
+    }
+}
+
+
+

If the compute operation only reads shared memory written to by other threads in the same warp as the current thread, __syncwarp() suffices.

+
+
+

+10.28.3. Pipeline Interface +

+

The complete API documentation for cuda::memcpy_async is provided in the libcudacxx API documentation along with some examples.

+

The pipeline interface requires

+
    +
  • at least CUDA 11.0,

  • +
  • at least ISO C++ 2011 compatibility, e.g., to be compiled with -std=c++11, and

  • +
  • #include <cuda/pipeline>.

  • +
+

For a C-like interface, when compiling without ISO C++ 2011 compatibility, see Pipeline Primitives Interface.

+
+
+

+10.28.4. Pipeline Primitives Interface +

+

Pipeline primitives are a C-like interface for memcpy_async functionality. The pipeline primitives interface is available by including the <cuda_pipeline.h> header. When compiling without ISO C++ 2011 compatibility, include the <cuda_pipeline_primitives.h> header.

+
+

+10.28.4.1. memcpy_async Primitive +

+
+
void __pipeline_memcpy_async(void* __restrict__ dst_shared,
+                             const void* __restrict__ src_global,
+                             size_t size_and_align,
+                             size_t zfill=0);
+
+
+
    +
  • +

    Request that the following operation be submitted for asynchronous evaluation:

    +
    +
    size_t i = 0;
    +for (; i < size_and_align - zfill; ++i) ((char*)dst_shared)[i] = ((char*)src_global)[i]; /* copy */
    +for (; i < size_and_align; ++i) ((char*)dst_shared)[i] = 0; /* zero-fill */
    +
    +
    +
  • +
  • +

    Requirements:

    +
      +
    • dst_shared must be a pointer to the shared memory destination for the memcpy_async.

    • +
    • src_global must be a pointer to the global memory source for the memcpy_async.

    • +
    • size_and_align must be 4, 8, or 16.

    • +
    • zfill <= size_and_align.

    • +
    • size_and_align must be the alignment of dst_shared and src_global.

    • +
    +
  • +
  • +

    It is a race condition for any thread to modify the source memory or observe the destination memory prior to waiting for the memcpy_async operation to complete. Between submitting a memcpy_async operation and waiting for its completion, any of the following actions introduces a race condition:

    +
      +
    • Loading from dst_shared.

    • +
    • Storing to dst_shared or src_global.

    • +
    • Applying an atomic update to dst_shared or src_global.

    • +
    +
  • +
+
+
+

+10.28.4.2. Commit Primitive +

+
+
void __pipeline_commit();
+
+
+
    +
  • Commit submitted memcpy_async to the pipeline as the current batch.

  • +
+
+
+

+10.28.4.3. Wait Primitive +

+
+
void __pipeline_wait_prior(size_t N);
+
+
+
    +
  • Let {0, 1, 2, ..., L} be the sequence of indices associated with invocations of __pipeline_commit() by a given thread.

  • +
  • Wait for completion of batches at least up to and including L-N.

  • +
+
+
+

+10.28.4.4. Arrive On Barrier Primitive +

+
+
void __pipeline_arrive_on(__mbarrier_t* bar);
+
+
+
    +
  • bar points to a barrier in shared memory.

  • +
  • Increments the barrier arrival count by one, when all memcpy_async operations sequenced before this call have completed, the arrival count is decremented by one and hence the net effect on the arrival count is zero. It is user’s responsibility to make sure that the increment on the arrival count does not exceed __mbarrier_maximum_count().

  • +
+
+
+
+
+

+10.29. Asynchronous Data Copies using the Tensor Memory Accelerator (TMA) +

+

Many applications require movement of large amounts of data from and to global +memory. Often, the data is laid out in global memory as a multi-dimensional +array with non-sequential data acess patterns. To reduce global memory usage, +sub-tiles of such arrays are copied to shared memory before use in computations. +The loading and storing involves address-calculations that can be error-prone +and repetitive. To offload these computations, Compute Capability 9.0 introduces the +Tensor Memory Accelerator (TMA). The primary goal of TMA is to provide an efficient +data transfer mechanism from global memory to shared memory for +multi-dimensional arrays.

+

Naming. Tensor memory accelerator (TMA) is a broad term used to refer to the +features described in this section. For the purpose of forward-compatibility and +to reduce discrepancies with the PTX ISA, the text in this section refers to TMA +operations as either bulk-asynchronous copies or bulk tensor asynchronous +copies, depending on the specific type of copy used. The term “bulk” is used to +contrast these operations with the asynchronous memory operations described in +the previous sections.

+

Dimensions. TMA supports copying both one-dimensional and multi-dimensional +arrays (up to 5-dimensional). The programming model for bulk-asynchronous +copies of one-dimensional contiguous arrays is different from the programming +model for bulk tensor asynchronous copies of multi-dimensional arrays. To +perform a bulk tensor asynchronous copy of a multi-dimensional array, the +hardware requires a tensor map. +This object describes the layout of the multi-dimensional array in global and +shared memory. A tensor map is typically created on the host using the cuTensorMapEncode +API +and then transferred from host to device as a const kernel parameter annotated +with __grid_constant__. +The tensor map is transferred from host to device as a const kernel +parameter annotated with __grid_constant__, and can be used on the device to +copy a tile of data between shared and global memory. In contrast, performing a +bulk-asynchronous copy of a contiguous one-dimensional array does not require a +tensor map: it can be performed on-device with a pointer and size parameter.

+

Source and destination. The source and destination addresses of bulk-asynchronous copy +operations can be in shared or global memory. The operations can read data from global to +shared memory, write data from shared to global memory, and also copy from +shared memory to Distributed Shared Memory of another block in the same cluster. +In addition, when in a cluster, a bulk-asynchronous operation can be specified as being +multicast. In this case, data can be transferred from global memory to the +shared memory of multiple blocks within the cluster. The multicast feature is +optimized for target architecture sm_90a and may have significantly reduced performance on +other targets. Hence, it is advised to be used with compute architecture +sm_90a.

+

Asynchronous. Data transfers using TMA are asynchronous. This allows the initiating +thread to continue computing while the hardware asynchronously copies the data. +Whether the data transfer occurs asynchronously in practice is up to the hardware implementation and may change in the future. +There are several completion mechanisms +that bulk-asynchronous operations can use to signal that they have completed. +When the operation reads from global to shared memory, any +thread in the block can wait for the data to be readable in shared memory by +waiting on a Shared Memory Barrier. When the bulk-asynchronous +operation writes data from shared memory to global or distributed shared memory, +only the initiating thread can wait for the operation to have completed. +This is accomplished using a bulk async-group based completion mechanism. A +table describing the completion mechanisms can be found below and in the PTX ISA.

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+Table 8 Asynchronous copies with possible source and destinations memory spaces and completion mechanisms. An empty cell indicates that a source-destination pair is not supported. +

Direction

Completion mechanism

Destination

Source

Asynchronous copy

Bulk-asynchronous copy (TMA)

Global

Global

Global

Shared::cta

Bulk async-group

Shared::cta

Global

Async-group, mbarrier

Mbarrier

Shared::cluster

Global

Mbarrier (multicast)

Shared::cta

Shared::cluster

Mbarrier

Shared::cta

Shared::cta

+
+

+10.29.1. Using TMA to transfer one-dimensional arrays +

+

This section demonstrates how to write a simple kernel that read-modify-writes a +one-dimensional array using TMA. This shows how to how to load and store data +using bulk-asynchronous copies, as well as how to synchronize threads of +execution with those copies.

+

The code of the kernel is included below. Some functionality requires inline PTX +assembly that is currently made available through libcu++. +The availability of these wrappers can be +checked with the following code:

+
+
#if defined(__CUDA_MINIMUM_ARCH__) && __CUDA_MINIMUM_ARCH__ < 900
+static_assert(false, "Device code is being compiled with older architectures that are incompatible with TMA.");
+#endif // __CUDA_MINIMUM_ARCH__
+
+
+

The kernel goes through the following stages:

+
    +
  1. Initialize shared memory barrier.

  2. +
  3. Initiate bulk-asynchronous copy of a block of memory from global to shared memory.

  4. +
  5. Arrive and wait on the shared memory barrier.

  6. +
  7. Increment the shared memory buffer values.

  8. +
  9. Wait for shared memory writes to be visible to the subsequent bulk-asynchronous copy, i.e., order the shared memory writes in the async proxy before the next step.

  10. +
  11. Initiate bulk-asynchronous copy of the buffer in shared memory to global memory.

  12. +
  13. Wait at end of kernel for bulk-asynchronous copy to have finished reading shared memory.

  14. +
+
+
#include <cuda/barrier>
+#include <cuda/ptx>
+using barrier = cuda::barrier<cuda::thread_scope_block>;
+namespace ptx = cuda::ptx;
+
+static constexpr size_t buf_len = 1024;
+__global__ void add_one_kernel(int* data, size_t offset)
+{
+  // Shared memory buffer. The destination shared memory buffer of
+  // a bulk operations should be 16 byte aligned.
+  __shared__ alignas(16) int smem_data[buf_len];
+
+  // 1. a) Initialize shared memory barrier with the number of threads participating in the barrier.
+  //    b) Make initialized barrier visible in async proxy.
+  #pragma nv_diag_suppress static_var_with_dynamic_init
+  __shared__ barrier bar;
+  if (threadIdx.x == 0) { 
+    init(&bar, blockDim.x);                      // a)
+    ptx::fence_proxy_async(ptx::space_shared);   // b)
+  }
+  __syncthreads();
+
+  // 2. Initiate TMA transfer to copy global to shared memory.
+  if (threadIdx.x == 0) {
+    // 3a. cuda::memcpy_async arrives on the barrier and communicates
+    //     how many bytes are expected to come in (the transaction count)
+    cuda::memcpy_async(
+        smem_data, 
+        data + offset, 
+        cuda::aligned_size_t<16>(sizeof(smem_data)),
+        bar
+    );
+  }
+  // 3b. All threads arrive on the barrier
+  barrier::arrival_token token = bar.arrive();
+  
+  // 3c. Wait for the data to have arrived.
+  bar.wait(std::move(token));
+
+  // 4. Compute saxpy and write back to shared memory
+  for (int i = threadIdx.x; i < buf_len; i += blockDim.x) {
+    smem_data[i] += 1;
+  }
+
+  // 5. Wait for shared memory writes to be visible to TMA engine.
+  ptx::fence_proxy_async(ptx::space_shared);   // b)
+  __syncthreads();
+  // After syncthreads, writes by all threads are visible to TMA engine.
+
+  // 6. Initiate TMA transfer to copy shared memory to global memory
+  if (threadIdx.x == 0) {
+    ptx::cp_async_bulk(
+        ptx::space_global,
+        ptx::space_shared,
+        data + offset, smem_data, sizeof(smem_data));
+    // 7. Wait for TMA transfer to have finished reading shared memory.
+    // Create a "bulk async-group" out of the previous bulk copy operation.
+    ptx::cp_async_bulk_commit_group();
+    // Wait for the group to have completed reading from shared memory.
+    ptx::cp_async_bulk_wait_group_read(ptx::n32_t<0>());
+  }
+}
+
+
+

Barrier initialization. The barrier is initialized with the number of +threads participating in the block. As a result, the barrier will flip only if +all threads have arrived on this barrier. Shared memory barriers are described +in more detail in Asynchronous Data Copies using cuda::barrier. +To make the initialized barrier visible to subsequent bulk-asynchronous copies, the +fence.proxy.async.shared::cta instruction is used. This instruction ensures that +subsequent bulk-asynchronous copy operations operate on the initialized barrier.

+

TMA read. The bulk-asynchronous copy instruction directs the +hardware to copy a large chunk of data into shared memory, and to update the +transaction count +of the shared memory barrier after completing the read. In general, issuing as +few bulk copies with as big a size as possible results in the best performance. +Because the copy can be performed asynchronously by the hardware, it is not +necessary to split the copy into smaller chunks.

+

The thread that initiates the bulk-asynchronous copy operation arrives at the barrier +using mbarrier.expect_tx. This is automatically performed by cuda::memcpy_async. This tells the barrier that the thread has +arrived and also how many bytes (tx / transactions) are expected to arrive. Only +a single thread has to update the expected transaction count. If multiple +threads update the transaction count, the expected transaction will be the sum +of the updates. The barrier will only flip once all threads have arrived and +all bytes have arrived. Once the barrier has flipped, the bytes are safe to read +from shared memory, both by the threads as well as by subsequent +bulk-asynchronous copies. More information about barrier transaction accounting +can be found in the PTX ISA.

+

Barrier wait. Waiting for the barrier to flip is done using +mbarrier.try_wait. It can either return true, indicating that the wait is +over, or return false, which may mean that the wait timed out. The while loop +waits for completion, and retries on time-out.

+

SMEM write and sync. The increment of the buffer values reads and writes to shared +memory. To make the writes visible to subsequent bulk-asynchronous copies, the +fence.proxy.async.shared::cta instruction is used. This orders the writes to +shared memory before subsequent reads from bulk-asynchronous copy operations, +which read through the async proxy. So each thread first orders the writes to +objects in shared memory in the async proxy via the +fence.proxy.async.shared::cta, and these operations by all threads are +ordered before the async operation performed in thread 0 using +__syncthreads().

+

TMA write and sync. The write from shared to global memory is again +initiated by a single thread. The completion of the write is not tracked by a +shared memory barrier. Instead, a thread-local mechanism is used. Multiple +writes can be batched into a so-called bulk async-group. Afterwards, the +thread can wait for all operations in this group to have completed reading from +shared memory (as in the code above) or to have completed writing to global +memory, making the writes visible to the initiating thread. For more information, +refer to the PTX ISA documentation of cp.async.bulk.wait_group. +Note that the bulk-asynchronous and non-bulk asynchronous copy instructions have +different async-groups: there exist both cp.async.wait_group and +cp.async.bulk.wait_group instructions.

+

The bulk-asynchronous instructions have specific alignment requirements on their source and +destination addresses. More information can be found in the table below.

+ + ++++ + + + + + + + + + + + + + + + + + + + + + + + + +
+Table 9 Alignment requirements for one-dimensional bulk-asynchronous operations in Compute Capability 9.0. +

Address / Size

Alignment

Global memory address

Must be 16 byte aligned.

Shared memory address

Must be 16 byte aligned.

Shared memory barrier address

Must be 8 byte aligned (this is guaranteed by cuda::barrier).

Size of transfer

Must be a multiple of 16 bytes.

+
+
+

+10.29.2. Using TMA to transfer multi-dimensional arrays +

+

The primary difference between the one-dimensional and multi-dimensional case is +that a tensor map must be created on the host and passed to the CUDA kernel. +This section describes how to create a tensor map using the CUDA driver API, how +to pass it to device, and how to use it on device.

+

Driver API. A tensor map is created using the cuTensorMapEncodeTiled +driver API. This API can be accessed by linking to the driver directly +(-lcuda) or by using the cudaGetDriverEntryPointByVersion +API. Below, we show how to get a pointer to the cuTensorMapEncodeTiled API. +For more information, refer to Driver Entry Point Access.

+
+
#include <cudaTypedefs.h> // PFN_cuTensorMapEncodeTiled, CUtensorMap
+
+PFN_cuTensorMapEncodeTiled_v12000 get_cuTensorMapEncodeTiled() {
+  // Get pointer to cuTensorMapEncodeTiled
+  cudaDriverEntryPointQueryResult driver_status;
+  void* cuTensorMapEncodeTiled_ptr = nullptr;
+  CUDA_CHECK(cudaGetDriverEntryPointByVersion("cuTensorMapEncodeTiled", &cuTensorMapEncodeTiled_ptr, 12000, cudaEnableDefault, &driver_status));
+  assert(driver_status == cudaDriverEntryPointSuccess);
+
+  return reinterpret_cast<PFN_cuTensorMapEncodeTiled_v12000>(cuTensorMapEncodeTiled_ptr);
+}
+
+
+

Creation. Creating a tensor map requires many parameters. Among +them are the base pointer to an array in global memory, the size of the array +(in number of elements), the stride from one row to the next (in bytes), the +size of the shared memory buffer (in number of elements). The code below creates +a tensor map to describe a two-dimensional row-major array of size GMEM_HEIGHT +x GMEM_WIDTH. Note the order of the parameters: the fastest moving dimension +comes first.

+
+
  CUtensorMap tensor_map{};
+  // rank is the number of dimensions of the array.
+  constexpr uint32_t rank = 2;
+  uint64_t size[rank] = {GMEM_WIDTH, GMEM_HEIGHT};
+  // The stride is the number of bytes to traverse from the first element of one row to the next.
+  // It must be a multiple of 16.
+  uint64_t stride[rank - 1] = {GMEM_WIDTH * sizeof(int)};
+  // The box_size is the size of the shared memory buffer that is used as the
+  // destination of a TMA transfer.
+  uint32_t box_size[rank] = {SMEM_WIDTH, SMEM_HEIGHT};
+  // The distance between elements in units of sizeof(element). A stride of 2
+  // can be used to load only the real component of a complex-valued tensor, for instance.
+  uint32_t elem_stride[rank] = {1, 1};
+
+  // Get a function pointer to the cuTensorMapEncodeTiled driver API.
+  auto cuTensorMapEncodeTiled = get_cuTensorMapEncodeTiled();
+
+  // Create the tensor descriptor.
+  CUresult res = cuTensorMapEncodeTiled(
+    &tensor_map,                // CUtensorMap *tensorMap,
+    CUtensorMapDataType::CU_TENSOR_MAP_DATA_TYPE_INT32,
+    rank,                       // cuuint32_t tensorRank,
+    tensor_ptr,                 // void *globalAddress,
+    size,                       // const cuuint64_t *globalDim,
+    stride,                     // const cuuint64_t *globalStrides,
+    box_size,                   // const cuuint32_t *boxDim,
+    elem_stride,                // const cuuint32_t *elementStrides,
+    // Interleave patterns can be used to accelerate loading of values that
+    // are less than 4 bytes long.
+    CUtensorMapInterleave::CU_TENSOR_MAP_INTERLEAVE_NONE,
+    // Swizzling can be used to avoid shared memory bank conflicts.
+    CUtensorMapSwizzle::CU_TENSOR_MAP_SWIZZLE_NONE,
+    // L2 Promotion can be used to widen the effect of a cache-policy to a wider
+    // set of L2 cache lines.
+    CUtensorMapL2promotion::CU_TENSOR_MAP_L2_PROMOTION_NONE,
+    // Any element that is outside of bounds will be set to zero by the TMA transfer.
+    CUtensorMapFloatOOBfill::CU_TENSOR_MAP_FLOAT_OOB_FILL_NONE
+  );
+
+
+

Host-to-device transfer. There are three ways to make a tensor map accessible to +device code. The recommended approach is to pass the tensor map as a const __grid_constant__ +parameter to a kernel. The other possibilities are copying the tensor map into device __constant__ +memory using cudaMemcpyToSymbol or accessing it via global memory. When passing the tensor map as a parameter, some versions of the +GCC C++ compiler issue the warning “the ABI for passing parameters with 64-byte +alignment has changed in GCC 4.6”. This warning can be ignored.

+
+
#include <cuda.h>
+
+__global__ void kernel(const __grid_constant__ CUtensorMap tensor_map)
+{
+   // Use tensor_map here.
+}
+int main() {
+  CUtensorMap map;
+  // [ ..Initialize map.. ]
+  kernel<<<1, 1>>>(map);
+}
+
+
+

As an alternative to the __grid_constant__ kernel parameter, a global +constant variable can be used. An example is included +below.

+
+
#include <cuda.h>
+
+__constant__ CUtensorMap global_tensor_map;
+__global__ void kernel()
+{
+  // Use global_tensor_map here.
+}
+int main() {
+  CUtensorMap local_tensor_map;
+  // [ ..Initialize map.. ]
+  cudaMemcpyToSymbol(global_tensor_map, &local_tensor_map, sizeof(CUtensorMap));
+  kernel<<<1, 1>>>();
+}
+
+
+

Finally, it is possible to copy the tensor map to global memory. Using a pointer to a +tensor map in global device memory requires a fence in each thread block before any thread +in the block uses the updated tensor map. Further uses of the tensor map by that thread block +do not need to be fenced unless the tensor map is modified again. Note that this mechanism +may be slower than the two mechanisms described above.

+
+
#include <cuda.h>
+#include <cuda/ptx>
+namespace ptx = cuda::ptx;
+
+__device__ CUtensorMap global_tensor_map;
+__global__ void kernel(CUtensorMap *tensor_map)
+{
+  // Fence acquire tensor map:
+  ptx::n32_t<128> size_bytes;
+  // Since the tensor map was modified from the host using cudaMemcpy,
+  // the scope should be .sys.
+  ptx::fence_proxy_tensormap_generic(
+     ptx::sem_acquire, ptx::scope_sys, tensor_map, size_bytes
+ );
+ // Safe to use tensor_map after fence inside this thread..
+}
+int main() {
+  CUtensorMap local_tensor_map;
+  // [ ..Initialize map.. ]
+  cudaMemcpy(&global_tensor_map, &local_tensor_map, sizeof(CUtensorMap), cudaMemcpyHostToDevice);
+  kernel<<<1, 1>>>(global_tensor_map);
+}
+
+
+

Use. The kernel below loads a 2D tile of size SMEM_HEIGHT x SMEM_WIDTH +from a larger 2D array. The top-left corner of the tile is indicated by the +indices x and y. The tile is loaded into shared memory, modified, and +written back to global memory.

+
+
#include <cuda.h>         // CUtensormap
+#include <cuda/barrier>
+using barrier = cuda::barrier<cuda::thread_scope_block>;
+namespace cde = cuda::device::experimental;
+
+__global__ void kernel(const __grid_constant__ CUtensorMap tensor_map, int x, int y) {
+  // The destination shared memory buffer of a bulk tensor operation should be
+  // 128 byte aligned.
+  __shared__ alignas(128) int smem_buffer[SMEM_HEIGHT][SMEM_WIDTH];
+
+  // Initialize shared memory barrier with the number of threads participating in the barrier.
+  #pragma nv_diag_suppress static_var_with_dynamic_init
+  __shared__ barrier bar;
+
+  if (threadIdx.x == 0) {
+    // Initialize barrier. All `blockDim.x` threads in block participate.
+    init(&bar, blockDim.x);
+    // Make initialized barrier visible in async proxy.
+    cde::fence_proxy_async_shared_cta();
+  }
+  // Syncthreads so initialized barrier is visible to all threads.
+  __syncthreads();
+
+  barrier::arrival_token token;
+  if (threadIdx.x == 0) {
+    // Initiate bulk tensor copy.
+    cde::cp_async_bulk_tensor_2d_global_to_shared(&smem_buffer, &tensor_map, x, y, bar);
+    // Arrive on the barrier and tell how many bytes are expected to come in.
+    token = cuda::device::barrier_arrive_tx(bar, 1, sizeof(smem_buffer));
+  } else {
+    // Other threads just arrive.
+    token = bar.arrive();
+  }
+  // Wait for the data to have arrived.
+  bar.wait(std::move(token));
+
+  // Symbolically modify a value in shared memory.
+  smem_buffer[0][threadIdx.x] += threadIdx.x;
+
+  // Wait for shared memory writes to be visible to TMA engine.
+  cde::fence_proxy_async_shared_cta();
+  __syncthreads();
+  // After syncthreads, writes by all threads are visible to TMA engine.
+
+  // Initiate TMA transfer to copy shared memory to global memory
+  if (threadIdx.x == 0) {
+    cde::cp_async_bulk_tensor_2d_shared_to_global(&tensor_map, x, y, &smem_buffer);
+    // Wait for TMA transfer to have finished reading shared memory.
+    // Create a "bulk async-group" out of the previous bulk copy operation.
+    cde::cp_async_bulk_commit_group();
+    // Wait for the group to have completed reading from shared memory.
+    cde::cp_async_bulk_wait_group_read<0>();
+  }
+
+  // Destroy barrier. This invalidates the memory region of the barrier. If
+  // further computations were to take place in the kernel, this allows the
+  // memory location of the shared memory barrier to be reused.
+  if (threadIdx.x == 0) {
+    (&bar)->~barrier();
+  }
+}
+
+
+

Negative indices and out of bounds. When part of the tile that is being +read from global to shared memory is out of bounds, the shared memory that +corresponds to the out of bounds area is zero-filled. The top-left corner +indices of the tile may also be negative. When writing from shared to global +memory, parts of the tile may be out of bounds, but the top left corner cannot +have any negative indices.

+

Size and stride. The size of a tensor is the number of elements along one +dimension. All sizes must be greater than one. The stride is the number of bytes +between elements of the same dimension. For instance, a 4 x 4 matrix of +integers has sizes 4 and 4. Since it has 4 bytes per element, the strides are 4 +and 16 bytes. Due to alignment requirements, a 4 x 3 row-major matrix of +integers must have strides of 4 and 16 bytes as well. Each row is padded with 4 +extra bytes to ensure that the start of the next row is aligned to 16 bytes. For +more information regarding alignment, refer to Table 10.

+ + ++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+Table 10 Alignment requirements for multi-dimensional bulk tensor asynchronous copy operations in Compute Capability 9.0. +

Address / Size

Alignment

Global memory address

Must be 16 byte aligned.

Global memory sizes

Must be greater than or equal to one. Does not have to be a multiple of 16 bytes.

Global memory strides

Must be multiples of 16 bytes.

Shared memory address

Must be 128 byte aligned.

Shared memory barrier address

Must be 8 byte aligned (this is guaranteed by cuda::barrier).

Size of transfer

Must be a multiple of 16 bytes.

+
+

+10.29.2.1. Multi-dimensional TMA PTX wrappers +

+

Below, the PTX instructions are ordered by their use in the example code above.

+

The cp.async.bulk.tensor +instructions initiate a bulk tensor asynchronous copy between global and shared memory. The +wrappers below read from global to shared memory and write from shared to global +memory.

+
+
// https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#data-movement-and-conversion-instructions-cp-async-bulk-tensor
+inline __device__
+void cuda::device::experimental::cp_async_bulk_tensor_1d_global_to_shared(
+    void *dest, const CUtensorMap *tensor_map , int c0, cuda::barrier<cuda::thread_scope_block> &bar
+);
+
+// https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#data-movement-and-conversion-instructions-cp-async-bulk-tensor
+inline __device__
+void cuda::device::experimental::cp_async_bulk_tensor_2d_global_to_shared(
+    void *dest, const CUtensorMap *tensor_map , int c0, int c1, cuda::barrier<cuda::thread_scope_block> &bar
+);
+
+// https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#data-movement-and-conversion-instructions-cp-async-bulk-tensor
+inline __device__
+void cuda::device::experimental::cp_async_bulk_tensor_3d_global_to_shared(
+    void *dest, const CUtensorMap *tensor_map, int c0, int c1, int c2, cuda::barrier<cuda::thread_scope_block> &bar
+);
+
+// https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#data-movement-and-conversion-instructions-cp-async-bulk-tensor
+inline __device__
+void cuda::device::experimental::cp_async_bulk_tensor_4d_global_to_shared(
+    void *dest, const CUtensorMap *tensor_map , int c0, int c1, int c2, int c3, cuda::barrier<cuda::thread_scope_block> &bar
+);
+
+// https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#data-movement-and-conversion-instructions-cp-async-bulk-tensor
+inline __device__
+void cuda::device::experimental::cp_async_bulk_tensor_5d_global_to_shared(
+    void *dest, const CUtensorMap *tensor_map , int c0, int c1, int c2, int c3, int c4, cuda::barrier<cuda::thread_scope_block> &bar
+);
+
+
+
+
// https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#data-movement-and-conversion-instructions-cp-async-bulk-tensor
+inline __device__
+void cuda::device::experimental::cp_async_bulk_tensor_1d_shared_to_global(
+    const CUtensorMap *tensor_map, int c0, const void *src
+);
+
+// https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#data-movement-and-conversion-instructions-cp-async-bulk-tensor
+inline __device__
+void cuda::device::experimental::cp_async_bulk_tensor_2d_shared_to_global(
+    const CUtensorMap *tensor_map, int c0, int c1, const void *src
+);
+
+// https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#data-movement-and-conversion-instructions-cp-async-bulk-tensor
+inline __device__
+void cuda::device::experimental::cp_async_bulk_tensor_3d_shared_to_global(
+    const CUtensorMap *tensor_map, int c0, int c1, int c2, const void *src
+);
+
+// https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#data-movement-and-conversion-instructions-cp-async-bulk-tensor
+inline __device__
+void cuda::device::experimental::cp_async_bulk_tensor_4d_shared_to_global(
+    const CUtensorMap *tensor_map, int c0, int c1, int c2, int c3, const void *src
+);
+
+// https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#data-movement-and-conversion-instructions-cp-async-bulk-tensor
+inline __device__
+void cuda::device::experimental::cp_async_bulk_tensor_5d_shared_to_global(
+    const CUtensorMap *tensor_map, int c0, int c1, int c2, int c3, int c4, const void *src
+);
+
+
+
+
+
+

+10.29.3. TMA Swizzle +

+

By default, the TMA engine loads data to shared memory in the same order as it is laid out in global memory. However, this +layout may not be optimal for certain shared memory access patterns, as it could cause shared memory bank conflicts. To +improve performance and reduce bank conflicts, we can change the shared memory layout by applying a ‘swizzle pattern’.

+

Shared memory has 32 banks that are organized such that successive 32-bit words map to successive banks. Each bank has a +bandwidth of 32 bits per clock cycle. When loading and storing shared memory, bank conflicts arise if the same bank is +used multiple times within a transaction, resulting in reduced bandwidth. See Shared Memory, bank conflicts.

+

To ensure that data is laid out in shared memory in such a way that user code can avoid shared memory bank conflicts, +the TMA engine can be instructed to ‘swizzle’ the data before storing it in shared memory and ‘unswizzle’ it when copying +the data back from shared memory to global memory. The tensor map encodes the ‘swizzle mode’ indicating which swizzle pattern is used.

+
+

+10.29.3.1. Example ‘Matrix Transpose’ +

+

An example is the transpose of a matrix where data is mapped from row to column first access. The data is stored row major in +global memory, but we want to also access it column wise in shared memory, which leads to bank conflicts. However, by using +the 128 bytes ‘swizzle’ mode and new shared memory indices, they are eliminated.

+

In the example, we load an 8x8 matrix of type int4, stored as row major in global memory to shared memory. Then, each set of eight +threads loads a row from the shared memory buffer and stores it to a column in a separate transpose shared memory buffer. This +results in an eight-way bank conflict when storing. Finally, the transpose buffer is written back to global memory.

+

To avoid bank conflicts, the CU_TENSOR_MAP_SWIZZLE_128B layout can be used. This layout matches the 128 bytes row length and +changes the shared memory layout in a way that both the column wise and row wise access don’t require the same banks per transaction.

+

The two tables, Figure 27 and Figure 28, below show the normal and the swizzled shared memory layout of the 8x8 matrix of type int4 and +its transpose matrix. The colors indicate which of the eight groups of four banks the matrix element is mapped to, and +the margin row and margin column list the global memory row and column indices. The entries show the shared memory +indices of the 16-byte matrix elements.

+
+The shared memory data layout without swizzle +
+

Figure 27 In the shared memory data layout without swizzle, the shared memory indices are equivalent to the global memory indices. +Per load instruction, one row is read and stored in a column of the transpose buffer. Since all matrix elements of the +column in the transpose fall in the same bank, the store must be serialized, resulting in eight store transactions, giving +an eight-way bank conflict per stored column.

+
+
+
+The shared memory data layout with CU_TENSOR_MAP_SWIZZLE_128B swizzle. +
+

Figure 28 The shared memory data layout with CU_TENSOR_MAP_SWIZZLE_128B swizzle. One row is stored in a column, each matrix +element is from a different bank for both the rows and columns, and so without any bank conflicts.

+
+
+
+
__global__ void kernel_tma(const __grid_constant__ CUtensorMap tensor_map) {
+   // The destination shared memory buffer of a bulk tensor operation
+   // with the 128-byte swizzle mode, it should be 1024 bytes aligned.
+   __shared__ alignas(1024) int4 smem_buffer[8][8];
+   __shared__ alignas(1024) int4 smem_buffer_tr[8][8];
+
+   // Initialize shared memory barrier
+   #pragma nv_diag_suppress static_var_with_dynamic_init
+   __shared__ barrier bar;
+
+   if (threadIdx.x == 0) {
+     init(&bar, blockDim.x);
+     cde::fence_proxy_async_shared_cta();
+   }
+
+   __syncthreads();
+
+   barrier::arrival_token token;
+   if (threadIdx.x == 0) {
+     // Initiate bulk tensor copy from global to shared memory,
+     // in the same way as without swizzle.
+     cde::cp_async_bulk_tensor_2d_global_to_shared(&smem_buffer, &tensor_map, 0, 0, bar);
+     token = cuda::device::barrier_arrive_tx(bar, 1, sizeof(smem_buffer));
+   } else {
+     token = bar.arrive();
+   }
+
+   bar.wait(std::move(token));
+
+   /* Matrix transpose
+    *  When using the normal shared memory layout, there are eight
+    *  8-way shared memory bank conflict when storing to the transpose.
+    *  When enabling the 128-byte swizzle pattern and using the according access pattern,
+    *  they are eliminated both for load and store. */
+   for(int sidx_j =threadIdx.x; sidx_j < 8; sidx_j+= blockDim.x){
+      for(int sidx_i = 0; sidx_i < 8; ++sidx_i){
+         const int swiz_j_idx = (sidx_i % 8) ^ sidx_j;
+         const int swiz_i_idx_tr = (sidx_j % 8) ^ sidx_i;
+         smem_buffer_tr[sidx_j][swiz_i_idx_tr] = smem_buffer[sidx_i][swiz_j_idx];
+      }
+   }
+
+   // Wait for shared memory writes to be visible to TMA engine.
+   cde::fence_proxy_async_shared_cta();
+   __syncthreads();
+
+   /* Initiate TMA transfer to copy the transposed shared memory buffer back to global memory,
+    * it will 'unswizzle' the data. */
+   if (threadIdx.x == 0) {
+      cde::cp_async_bulk_tensor_2d_shared_to_global(&tensor_map, 0, 0, &smem_buffer_tr);
+      cde::cp_async_bulk_commit_group();
+      cde::cp_async_bulk_wait_group_read<0>();
+   }
+
+   // Destroy barrier
+   if (threadIdx.x == 0) {
+     (&bar)->~barrier();
+   }
+}
+
+// --------------------------------- main ----------------------------------------
+
+int main(){
+
+...
+   void* tensor_ptr = d_data;
+
+   CUtensorMap tensor_map{};
+   // rank is the number of dimensions of the array.
+   constexpr uint32_t rank = 2;
+   // global memory size
+   uint64_t size[rank] = {4*8, 8};
+   // global memory stride, must be a multiple of 16.
+   uint64_t stride[rank - 1] = {8 * sizeof(int4)};
+   // The inner shared memory box dimension in bytes, equal to the swizzle span.
+   uint32_t box_size[rank] = {4*8, 8};
+
+   uint32_t elem_stride[rank] = {1, 1};
+
+   // Create the tensor descriptor.
+   CUresult res = cuTensorMapEncodeTiled(
+       &tensor_map,                // CUtensorMap *tensorMap,
+       CUtensorMapDataType::CU_TENSOR_MAP_DATA_TYPE_INT32,
+       rank,                       // cuuint32_t tensorRank,
+       tensor_ptr,                 // void *globalAddress,
+       size,                       // const cuuint64_t *globalDim,
+       stride,                     // const cuuint64_t *globalStrides,
+       box_size,                   // const cuuint32_t *boxDim,
+       elem_stride,                // const cuuint32_t *elementStrides,
+       CUtensorMapInterleave::CU_TENSOR_MAP_INTERLEAVE_NONE,
+       // Using a swizzle pattern of 128 bytes.
+       CUtensorMapSwizzle::CU_TENSOR_MAP_SWIZZLE_128B,
+       CUtensorMapL2promotion::CU_TENSOR_MAP_L2_PROMOTION_NONE,
+       CUtensorMapFloatOOBfill::CU_TENSOR_MAP_FLOAT_OOB_FILL_NONE
+   );
+
+   kernel_tma<<<1, 8>>>(tensor_map);
+ ...
+}
+
+
+

Remark. This example is supposed to show the use of swizzle and ‘as-is’ is not performant nor does it scale beyond the given dimensions.

+

Explanation. During data transfer, the TMA engine shuffles the data according to the swizzle pattern, as described in the following +tables. These swizzle patterns define the mapping of the 16-byte chunks along the swizzle width to subgroups of four banks. +It is of type CUtensorMapSwizzle and has four options: none, 32 bytes, 64 bytes and 128 bytes. Note that the shared memory box’s +inner dimension must be less or equal to the span of the swizzle pattern.

+
+
+

+10.29.3.2. The Swizzle Modes +

+

As previously mentioned, there are four swizzle modes. The following tables show the different swizzle patterns, including the relation of the new +shared memory indices. The tables define the mapping of the 16-byte chunks along the 128 bytes to eight subgroups of four banks.

+
+An Overview of TMA Swizzle Patterns +
+

Figure 29 An Overview of TMA Swizzle Patterns

+
+
+

Considerations. When applying a TMA swizzle pattern, it is crucial to adhere to specific memory requirements:

+
    +
  • Global memory alignment: +Global memory must be aligned to 128 bytes.

  • +
  • Shared memory alignment: +For simplicity shared memory should be aligned according to the number of bytes after which the swizzle pattern repeats. When the shared memory buffer is not aligned by the number of bytes by which the swizzle pattern repeats itself, there is an offset between the swizzle pattern and the shared memory. +See comment, below.

  • +
  • Inner dimension: +The inner dimension of the shared memory block must meet the size requirements specified in Table 12. If these +requirements are not met, the instruction is considered invalid. Additionally, if the swizzle width exceeds the inner dimension, +ensure that the shared memory is allocated to accommodate the full swizzle width.

  • +
  • Granularity: +The granularity of swizzle mapping is fixed at 16 bytes. This means that data is organized and accessed in chunks +of 16 bytes, which must be considered when planning memory layout and access patterns.

  • +
+

Swizzle Pattern Pointer Offset Computation. Here, we describe how to determine the offset between the swizzle pattern and the shared memory, when the shared memory buffer is not aligned by the number of bytes by which the swizzle pattern repeats itself. +When using TMA, the shared memory is required to be aligned to 128 bytes. To find how many times the shared memory buffer relative to the swizzle pattern is shifted by that, apply the corresponding offset formula.

+ + + + + + + + + + + + + + + + + + + + + + + + + + +
+Table 11 Swizzle Pattern Pointer Offset Formula and Index Relation +

Swizzle Mode

Offset Formula

Index Relation

CU_TENSOR_MAP_SWIZZLE_128B

(reinterpret_cast <uintptr_t>(smem_ptr)/128)%8

smem[y][x] <-> smem[y][((y+offset)%8)^x]

CU_TENSOR_MAP_SWIZZLE_64B

(reinterpret_cast <uintptr_t>(smem_ptr)/128)%4

smem[y][x] <-> smem[y][((y+offset)%4)^x]

CU_TENSOR_MAP_SWIZZLE_32B

(reinterpret_cast <uintptr_t>(smem_ptr)/128)%2

smem[y][x] <-> smem[y][((y+offset)%2)^x]

+

In Figure 29, this offset represents the initial row offset, thus, in the swizzle index calculation, it is added to the row index y. +The following snippet shows how to access the swizzled shared memory in the CU_TENSOR_MAP_SWIZZLE_128B mode.

+
+
data_t* smem_ptr = &smem[0][0];
+int offset = (reinterpret_cast<uintptr_t>(smem_ptr)/128)%8;
+smem[y][((y+offset)%8)^x] = ...
+
+
+

Summary. The following Table 12 summarizes the requirements and properties of the different swizzle patterns for Compute Capability 9.

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+Table 12 Requirements and properties of the different swizzle patterns for Compute Capability 9 +

Pattern

Swizzle width

Shared box’s +inner dimension

Repeats +after

Shared +memory +alignment

Global +memory +alignment

CU_TENSOR_MAP_SWIZZLE_128B

128 bytes

<=128 bytes

1024 bytes

128 bytes

128 bytes

CU_TENSOR_MAP_SWIZZLE_64B

64 bytes

<=64 bytes

512 bytes

128 bytes

128 bytes

CU_TENSOR_MAP_SWIZZLE_32B

32 bytes

<=32 bytes

256 bytes

128 bytes

128 bytes

CU_TENSOR_MAP_SWIZZLE_NONE +(default)

128 bytes

16 bytes

+
+
+
+
+

+10.30. Encoding a Tensor Map on Device +

+

Previous sections have described how to create a tensor map on the host using the CUDA driver API.

+

This section explains how to encode a tiled-type tensor map on device. This is useful in situations where the typical +way of transferring the tensor map (using const __grid_constant__ kernel parameters) is undesirable, for instance, +when processing a batch of tensors of various sizes in a single kernel launch.

+

The recommended pattern is as follows:

+
    +
  1. Create a tensor map “template”, template_tensor_map, using the Driver API on the host.

  2. +
  3. In a device kernel, copy the template_tensor_map, modify the copy, store in global memory, and appropriately fence.

  4. +
  5. Use the tensor map in a kernel with appropriate fencing.

  6. +
+

The high-level code structure is as follows:

+
+
// Initialize device context:
+CUDA_CHECK(cudaDeviceSynchronize());
+
+// Create a tensor map template using the cuTensorMapEncodeTiled driver function
+CUtensorMap template_tensor_map = make_tensormap_template();
+
+// Allocate tensor map and tensor in global memory
+CUtensorMap* global_tensor_map;
+CUDA_CHECK(cudaMalloc(&global_tensor_map, sizeof(CUtensorMap)));
+char* global_buf;
+CUDA_CHECK(cudaMalloc(&global_buf, 8 * 256));
+
+// Fill global buffer with data.
+fill_global_buf<<<1, 1>>>(global_buf);
+
+// Define the parameters of the tensor map that will be created on device.
+tensormap_params p{};
+p.global_address    = global_buf;
+p.rank              = 2;
+p.box_dim[0]        = 128; // The box in shared memory has half the width of the full buffer
+p.box_dim[1]        = 4;   // The box in shared memory has half the height of the full buffer
+p.global_dim[0]     = 256; //
+p.global_dim[1]     = 8;   //
+p.global_stride[0]  = 256; //
+p.element_stride[0] = 1;   //
+p.element_stride[1] = 1;   //
+
+// Encode global_tensor_map on device:
+encode_tensor_map<<<1, 32>>>(template_tensor_map, p, global_tensor_map);
+
+// Use it from another kernel:
+consume_tensor_map<<<1, 1>>>(global_tensor_map);
+
+// Check for errors:
+CUDA_CHECK(cudaDeviceSynchronize());
+
+
+

The following sections describe the high-level steps. Throughout the examples, the following tensormap_params +struct contains the new values of the fields to be updated. It is included here to reference when reading the examples.

+
+
struct tensormap_params {
+  void* global_address;
+  int rank;
+  uint32_t box_dim[5];
+  uint64_t global_dim[5];
+  size_t global_stride[4];
+  uint32_t element_stride[5];
+};
+
+
+
+

+10.30.1. Device-side Encoding and Modification of a Tensor Map +

+

The recommended process of encoding a tensor map in global memory proceeds as follows.

+
    +
  1. Pass an existing tensor map, the template_tensor_map, to the kernel. In contrast to kernels that use +the tensor map in a cp.async.bulk.tensor instruction, this may be done in any way: a pointer to global +memory, kernel parameter, a __const___ variable, and so on.

  2. +
  3. Copy-initialize a tensor map in shared memory with the template_tensor_map value.

  4. +
  5. Modify the tensor map in shared memory using the cuda::ptx::tensormap_replace +functions. These functions wrap the tensormap.replace +PTX instruction, which can be used to modify any field of a tiled-type tensor map, including the +base address, size, stride, and so on.

  6. +
  7. Using the cuda::ptx::tensormap_copy_fenceproxy +function, copy the modified tensor map from shared memory to global memory and perform any necessary fencing.

  8. +
+

The following code contains a kernel that follows these steps. For completeness, it modifies all the fields +of the tensor map. Typically, a kernel will modify just a few fields.

+

In this kernel, template_tensor_map is passed as a kernel parameter. This is the preferred way of moving template_tensor_map +from the host to the device. If the kernel is intended to update an existing tensor map in device memory, it can take a +pointer to the existing tensor map to modify.

+
+

Note

+

The format of the tensor map may change over time. Therefore, the cuda::ptx::tensormap_replace +functions and corresponding tensormap.replace.tile +PTX instructions are marked as specific to sm_90a. To use them, compile using nvcc -arch sm_90a .....

+
+
+

Tip

+

On sm_90a, a zero-initialized buffer in shared memory may also be used as the initial tensor map value. This +enables encoding a tensor map purely on device, without using the driver API to encode the template_tensor_map value.

+
+
+

Note

+

On-device modification is only supported for tiled-type tensor maps; other tensor map types cannot be modified on device. For more +information on the tensor map types, refer to the Driver API reference.

+
+
+
#include <cuda/ptx>
+
+namespace ptx = cuda::ptx;
+
+// launch with 1 warp.
+__launch_bounds__(32)
+__global__ void encode_tensor_map(const __grid_constant__ CUtensorMap template_tensor_map, tensormap_params p, CUtensorMap* out) {
+   __shared__ alignas(128) CUtensorMap smem_tmap;
+   if (threadIdx.x == 0) {
+      // Copy template to shared memory:
+      smem_tmap = template_tensor_map;
+
+      const auto space_shared = ptx::space_shared;
+      ptx::tensormap_replace_global_address(space_shared, &smem_tmap, p.global_address);
+      // For field .rank, the operand new_val must be ones less than the desired
+      // tensor rank as this field uses zero-based numbering.
+      ptx::tensormap_replace_rank(space_shared, &smem_tmap, p.rank - 1);
+
+      // Set box dimensions:
+      if (0 < p.rank) { ptx::tensormap_replace_box_dim(space_shared, &smem_tmap, ptx::n32_t<0>{}, p.box_dim[0]); }
+      if (1 < p.rank) { ptx::tensormap_replace_box_dim(space_shared, &smem_tmap, ptx::n32_t<1>{}, p.box_dim[1]); }
+      if (2 < p.rank) { ptx::tensormap_replace_box_dim(space_shared, &smem_tmap, ptx::n32_t<2>{}, p.box_dim[2]); }
+      if (3 < p.rank) { ptx::tensormap_replace_box_dim(space_shared, &smem_tmap, ptx::n32_t<3>{}, p.box_dim[3]); }
+      if (4 < p.rank) { ptx::tensormap_replace_box_dim(space_shared, &smem_tmap, ptx::n32_t<4>{}, p.box_dim[4]); }
+      // Set global dimensions:
+      if (0 < p.rank) { ptx::tensormap_replace_global_dim(space_shared, &smem_tmap, ptx::n32_t<0>{}, (uint32_t) p.global_dim[0]); }
+      if (1 < p.rank) { ptx::tensormap_replace_global_dim(space_shared, &smem_tmap, ptx::n32_t<1>{}, (uint32_t) p.global_dim[1]); }
+      if (2 < p.rank) { ptx::tensormap_replace_global_dim(space_shared, &smem_tmap, ptx::n32_t<2>{}, (uint32_t) p.global_dim[2]); }
+      if (3 < p.rank) { ptx::tensormap_replace_global_dim(space_shared, &smem_tmap, ptx::n32_t<3>{}, (uint32_t) p.global_dim[3]); }
+      if (4 < p.rank) { ptx::tensormap_replace_global_dim(space_shared, &smem_tmap, ptx::n32_t<4>{}, (uint32_t) p.global_dim[4]); }
+      // Set global stride:
+      if (1 < p.rank) { ptx::tensormap_replace_global_stride(space_shared, &smem_tmap, ptx::n32_t<0>{}, p.global_stride[0]); }
+      if (2 < p.rank) { ptx::tensormap_replace_global_stride(space_shared, &smem_tmap, ptx::n32_t<1>{}, p.global_stride[1]); }
+      if (3 < p.rank) { ptx::tensormap_replace_global_stride(space_shared, &smem_tmap, ptx::n32_t<2>{}, p.global_stride[2]); }
+      if (4 < p.rank) { ptx::tensormap_replace_global_stride(space_shared, &smem_tmap, ptx::n32_t<3>{}, p.global_stride[3]); }
+      // Set element stride:
+      if (0 < p.rank) { ptx::tensormap_replace_element_size(space_shared, &smem_tmap, ptx::n32_t<0>{}, p.element_stride[0]); }
+      if (1 < p.rank) { ptx::tensormap_replace_element_size(space_shared, &smem_tmap, ptx::n32_t<1>{}, p.element_stride[1]); }
+      if (2 < p.rank) { ptx::tensormap_replace_element_size(space_shared, &smem_tmap, ptx::n32_t<2>{}, p.element_stride[2]); }
+      if (3 < p.rank) { ptx::tensormap_replace_element_size(space_shared, &smem_tmap, ptx::n32_t<3>{}, p.element_stride[3]); }
+      if (4 < p.rank) { ptx::tensormap_replace_element_size(space_shared, &smem_tmap, ptx::n32_t<4>{}, p.element_stride[4]); }
+
+      // These constants are documented in this table:
+      // https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#tensormap-new-val-validity
+      auto u8_elem_type = ptx::n32_t<0>{};
+      ptx::tensormap_replace_elemtype(space_shared, &smem_tmap, u8_elem_type);
+      auto no_interleave = ptx::n32_t<0>{};
+      ptx::tensormap_replace_interleave_layout(space_shared, &smem_tmap, no_interleave);
+      auto no_swizzle = ptx::n32_t<0>{};
+      ptx::tensormap_replace_swizzle_mode(space_shared, &smem_tmap, no_swizzle);
+      auto zero_fill = ptx::n32_t<0>{};
+      ptx::tensormap_replace_fill_mode(space_shared, &smem_tmap, zero_fill);
+   }
+   // Synchronize the modifications with other threads in warp
+   __syncwarp();
+   // Copy the tensor map to global memory collectively with threads in the warp.
+   // In addition: make the updated tensor map visible to other threads on device that
+   // for use with cp.async.bulk.
+   ptx::n32_t<128> bytes_128;
+   ptx::tensormap_cp_fenceproxy(ptx::sem_release, ptx::scope_gpu, out, &smem_tmap, bytes_128);
+}
+
+
+
+
+

+10.30.2. Usage of a Modified Tensor Map +

+

In contrast to using a tensor map that is passed as a const __grid_constant__ kernel parameter, using a tensor map in +global memory requires explicitly establishing a release-acquire pattern in the tensor map proxy between the threads +that modify the tensor map and the threads that use it.

+

The release part of the pattern was shown in the previous section. It is accomplished using +the cuda::ptx::tensormap.cp_fenceproxy function.

+

The acquire part is accomplished using the cuda::ptx::fence_proxy_tensormap_generic +function that wraps the fence.proxy.tensormap::generic.acquire +instruction. If the two threads participating in the release-acquire pattern are on the same device, the .gpu scope suffices. If the threads are on +different devices, the .sys scope must be used. Once a tensor map has been acquired by one thread, it can be used by other threads in the block +after sufficient synchronization, for example, using __syncthreads(). The thread that uses the tensor map and the thread that performs the fence +must be in the same block. That is, if the threads are in, for example, two different thread blocks of the same cluster, the same grid, or a +different kernel, synchronization APIs such as cooperative_groups::cluster or grid_group::sync() or stream-order synchronization do not +suffice to establish ordering for tensor map updates, that is, threads in these other thread blocks still need to acquire the tensor map proxy +at the right scope before using the updated tensor map. If there are no intermediate modifications, the fence does not have to be repeated +before each cp.async.bulk.tensor instruction.

+

The fence and subsequent use of the tensor map is shown in the following example.

+
+
// Consumer of tensor map in global memory:
+__global__ void consume_tensor_map(CUtensorMap* tensor_map) {
+  // Fence acquire tensor map:
+  ptx::n32_t<128> size_bytes;
+  ptx::fence_proxy_tensormap_generic(ptx::sem_acquire, ptx::scope_sys, tensor_map, size_bytes);
+  // Safe to use tensor_map after fence..
+
+  __shared__ uint64_t bar;
+  __shared__ alignas(128) char smem_buf[4][128];
+
+  if (threadIdx.x == 0) {
+    // Initialize barrier
+    ptx::mbarrier_init(&bar, 1);
+    // Make barrier init visible in async proxy, i.e., to TMA engine
+    ptx::fence_proxy_async(ptx::space_shared);
+    // Issue TMA request
+    ptx::cp_async_bulk_tensor(ptx::space_cluster, ptx::space_global, smem_buf, tensor_map, {0, 0}, &bar);
+
+    // Arrive on barrier. Expect 4 * 128 bytes.
+    ptx::mbarrier_arrive_expect_tx(ptx::sem_release, ptx::scope_cta, ptx::space_shared, &bar, sizeof(smem_buf));
+  }
+  const int parity = 0;
+  // Wait for load to have completed
+  while (!ptx::mbarrier_try_wait_parity(&bar, parity)) {}
+
+  // print items:
+  printf("Got:\n\n");
+  for (int j = 0; j < 4; ++j) {
+    for (int i = 0; i < 128; ++i) {
+      printf("%3d ", smem_buf[j][i]);
+      if (i % 32 == 31) { printf("\n"); };
+    }
+    printf("\n");
+  }
+}
+
+
+
+
+

+10.30.3. Creating a Template Tensor Map Value Using the Driver API +

+

The following code creates a minimal tiled-type tensor map that can be subsequently modified on device.

+
+
CUtensorMap make_tensormap_template() {
+  CUtensorMap template_tensor_map{};
+  auto cuTensorMapEncodeTiled = get_cuTensorMapEncodeTiled();
+
+  uint32_t dims_32         = 16;
+  uint64_t dims_strides_64 = 16;
+  uint32_t elem_strides    = 1;
+
+  // Create the tensor descriptor.
+  CUresult res = cuTensorMapEncodeTiled(
+    &template_tensor_map, // CUtensorMap *tensorMap,
+    CUtensorMapDataType::CU_TENSOR_MAP_DATA_TYPE_UINT8,
+    1,                // cuuint32_t tensorRank,
+    nullptr,          // void *globalAddress,
+    &dims_strides_64, // const cuuint64_t *globalDim,
+    &dims_strides_64, // const cuuint64_t *globalStrides,
+    &dims_32,         // const cuuint32_t *boxDim,
+    &elem_strides,    // const cuuint32_t *elementStrides,
+    CUtensorMapInterleave::CU_TENSOR_MAP_INTERLEAVE_NONE,
+    CUtensorMapSwizzle::CU_TENSOR_MAP_SWIZZLE_NONE,
+    CUtensorMapL2promotion::CU_TENSOR_MAP_L2_PROMOTION_NONE,
+    CUtensorMapFloatOOBfill::CU_TENSOR_MAP_FLOAT_OOB_FILL_NONE);
+
+  CU_CHECK(res);
+  return template_tensor_map;
+}
+
+
+
+
+
+

+10.31. Profiler Counter Function +

+

Each multiprocessor has a set of sixteen hardware counters that an application can increment with a single instruction by calling the __prof_trigger() function.

+
+
void __prof_trigger(int counter);
+
+
+

increments by one per warp the per-multiprocessor hardware counter of index counter. Counters 8 to 15 are reserved and should not be used by applications.

+

The value of counters 0, 1, …, 7 can be obtained via nvprof by nvprof --events prof_trigger_0x where x is 0, 1, …, 7. All counters are reset before each kernel launch (note that when collecting counters, kernel launches are synchronous as mentioned in Concurrent Execution between Host and Device).

+
+
+

+10.32. Assertion +

+

Assertion is only supported by devices of compute capability 2.x and higher.

+
+
void assert(int expression);
+
+
+

stops the kernel execution if expression is equal to zero. If the program is run within a debugger, this triggers a breakpoint and the debugger can be used to inspect the current state of the device. Otherwise, each thread for which expression is equal to zero prints a message to stderr after synchronization with the host via cudaDeviceSynchronize(), cudaStreamSynchronize(), or cudaEventSynchronize(). The format of this message is as follows:

+
+
<filename>:<line number>:<function>:
+block: [blockId.x,blockId.x,blockIdx.z],
+thread: [threadIdx.x,threadIdx.y,threadIdx.z]
+Assertion `<expression>` failed.
+
+
+

Any subsequent host-side synchronization calls made for the same device will return cudaErrorAssert. No more commands can be sent to this device until cudaDeviceReset() is called to reinitialize the device.

+

If expression is different from zero, the kernel execution is unaffected.

+

For example, the following program from source file test.cu

+
+
#include <assert.h>
+
+__global__ void testAssert(void)
+{
+    int is_one = 1;
+    int should_be_one = 0;
+
+    // This will have no effect
+    assert(is_one);
+
+    // This will halt kernel execution
+    assert(should_be_one);
+}
+
+int main(int argc, char* argv[])
+{
+    testAssert<<<1,1>>>();
+    cudaDeviceSynchronize();
+
+    return 0;
+}
+
+
+

will output:

+
+
test.cu:19: void testAssert(): block: [0,0,0], thread: [0,0,0] Assertion `should_be_one` failed.
+
+
+

Assertions are for debugging purposes. They can affect performance and it is therefore recommended to disable them in production code. They can be disabled at compile time by defining the NDEBUG preprocessor macro before including assert.h. Note that expression should not be an expression with side effects (something like(++i > 0), for example), otherwise disabling the assertion will affect the functionality of the code.

+
+
+

+10.33. Trap function +

+

A trap operation can be initiated by calling the __trap() function from any device thread.

+
+
void __trap();
+
+
+

The execution of the kernel is aborted and an interrupt is raised in the host program.

+
+
+

+10.34. Breakpoint Function +

+

Execution of a kernel function can be suspended by calling the __brkpt() function from any device thread.

+
+
void __brkpt();
+
+
+
+
+

+10.35. Formatted Output +

+

Formatted output is only supported by devices of compute capability 2.x and higher.

+
+
int printf(const char *format[, arg, ...]);
+
+
+

prints formatted output from a kernel to a host-side output stream.

+

The in-kernel printf() function behaves in a similar way to the standard C-library printf() function, and the user is referred to the host system’s manual pages for a complete description of printf() behavior. In essence, the string passed in as format is output to a stream on the host, with substitutions made from the argument list wherever a format specifier is encountered. Supported format specifiers are listed below.

+

The printf() command is executed as any other device-side function: per-thread, and in the context of the calling thread. From a multi-threaded kernel, this means that a straightforward call to printf() will be executed by every thread, using that thread’s data as specified. Multiple versions of the output string will then appear at the host stream, once for each thread which encountered the printf().

+

It is up to the programmer to limit the output to a single thread if only a single output string is desired (see Examples for an illustrative example).

+

Unlike the C-standard printf(), which returns the number of characters printed, CUDA’s printf() returns the number of arguments parsed. If no arguments follow the format string, 0 is returned. If the format string is NULL, -1 is returned. If an internal error occurs, -2 is returned.

+
+

+10.35.1. Format Specifiers +

+

As for standard printf(), format specifiers take the form: %[flags][width][.precision][size]type

+

The following fields are supported (see widely-available documentation for a complete description of all behaviors):

+
    +
  • Flags: '#' ' ' '0' '+' '-'

  • +
  • Width: '*' '0-9'

  • +
  • Precision: '0-9'

  • +
  • Size: 'h' 'l' 'll'

  • +
  • Type: "%cdiouxXpeEfgGaAs"

  • +
+

Note that CUDA’s printf()will accept any combination of flag, width, precision, size and type, whether or not overall they form a valid format specifier. In other words, “%hd” will be accepted and printf will expect a double-precision variable in the corresponding location in the argument list.

+
+
+

+10.35.2. Limitations +

+

Final formatting of the printf()output takes place on the host system. This means that the format string must be understood by the host-system’s compiler and C library. Every effort has been made to ensure that the format specifiers supported by CUDA’s printf function form a universal subset from the most common host compilers, but exact behavior will be host-OS-dependent.

+

As described in Format Specifiers, printf() will accept all combinations of valid flags and types. This is because it cannot determine what will and will not be valid on the host system where the final output is formatted. The effect of this is that output may be undefined if the program emits a format string which contains invalid combinations.

+

The printf() command can accept at most 32 arguments in addition to the format string. Additional arguments beyond this will be ignored, and the format specifier output as-is.

+

Owing to the differing size of the long type on 64-bit Windows platforms (four bytes on 64-bit Windows platforms, eight bytes on other 64-bit platforms), a kernel which is compiled on a non-Windows 64-bit machine but then run on a win64 machine will see corrupted output for all format strings which include “%ld”. It is recommended that the compilation platform matches the execution platform to ensure safety.

+

The output buffer for printf() is set to a fixed size before kernel launch (see Associated Host-Side API). It is circular and if more output is produced during kernel execution than can fit in the buffer, older output is overwritten. It is flushed only when one of these actions is performed:

+
    +
  • Kernel launch via <<<>>> or cuLaunchKernel() (at the start of the launch, and if the CUDA_LAUNCH_BLOCKING environment variable is set to 1, at the end of the launch as well),

  • +
  • Synchronization via cudaDeviceSynchronize(), cuCtxSynchronize(), cudaStreamSynchronize(), cuStreamSynchronize(), cudaEventSynchronize(), or cuEventSynchronize(),

  • +
  • Memory copies via any blocking version of cudaMemcpy*() or cuMemcpy*(),

  • +
  • Module loading/unloading via cuModuleLoad() or cuModuleUnload(),

  • +
  • Context destruction via cudaDeviceReset() or cuCtxDestroy().

  • +
  • Prior to executing a stream callback added by cudaLaunchHostFunc or cuLaunchHostFunc.

  • +
+

Note that the buffer is not flushed automatically when the program exits. The user must call cudaDeviceReset() or cuCtxDestroy() explicitly, as shown in the examples below.

+

Internally printf() uses a shared data structure and so it is possible that calling printf() might change the order of execution of threads. In particular, a thread which calls printf() might take a longer execution path than one which does not call printf(), and that path length is dependent upon the parameters of the printf(). Note, however, that CUDA makes no guarantees of thread execution order except at explicit __syncthreads() barriers, so it is impossible to tell whether execution order has been modified by printf() or by other scheduling behavior in the hardware.

+
+
+

+10.35.3. Associated Host-Side API +

+

The following API functions get and set the size of the buffer used to transfer the printf() arguments and internal metadata to the host (default is 1 megabyte):

+
    +
  • cudaDeviceGetLimit(size_t* size,cudaLimitPrintfFifoSize)

  • +
  • cudaDeviceSetLimit(cudaLimitPrintfFifoSize, size_t size)

  • +
+
+
+

+10.35.4. Examples +

+

The following code sample:

+
+
#include <stdio.h>
+
+__global__ void helloCUDA(float f)
+{
+    printf("Hello thread %d, f=%f\n", threadIdx.x, f);
+}
+
+int main()
+{
+    helloCUDA<<<1, 5>>>(1.2345f);
+    cudaDeviceSynchronize();
+    return 0;
+}
+
+
+

will output:

+
+
Hello thread 2, f=1.2345
+Hello thread 1, f=1.2345
+Hello thread 4, f=1.2345
+Hello thread 0, f=1.2345
+Hello thread 3, f=1.2345
+
+
+

Notice how each thread encounters the printf() command, so there are as many lines of output as there were threads launched in the grid. As expected, global values (i.e., float f) are common between all threads, and local values (i.e., threadIdx.x) are distinct per-thread.

+

The following code sample:

+
+
#include <stdio.h>
+
+__global__ void helloCUDA(float f)
+{
+    if (threadIdx.x == 0)
+        printf("Hello thread %d, f=%f\n", threadIdx.x, f) ;
+}
+
+int main()
+{
+    helloCUDA<<<1, 5>>>(1.2345f);
+    cudaDeviceSynchronize();
+    return 0;
+}
+
+
+

will output:

+
+
Hello thread 0, f=1.2345
+
+
+

Self-evidently, the if() statement limits which threads will call printf, so that only a single line of output is seen.

+
+
+
+

+10.36. Dynamic Global Memory Allocation and Operations +

+

Dynamic global memory allocation and operations are only supported by devices of compute capability 2.x and higher.

+
+
__host__ __device__ void* malloc(size_t size);
+__device__ void *__nv_aligned_device_malloc(size_t size, size_t align);
+__host__ __device__  void free(void* ptr);
+
+
+

allocate and free memory dynamically from a fixed-size heap in global memory.

+
+
__host__ __device__ void* memcpy(void* dest, const void* src, size_t size);
+
+
+

copy size bytes from the memory location pointed by src to the memory location pointed by dest.

+
+
__host__ __device__ void* memset(void* ptr, int value, size_t size);
+
+
+

set size bytes of memory block pointed by ptr to value (interpreted as an unsigned char).

+

The CUDA in-kernel malloc()function allocates at least size bytes from the device heap and returns a pointer to the allocated memory or NULL if insufficient memory exists to fulfill the request. The returned pointer is guaranteed to be aligned to a 16-byte boundary.

+

The CUDA in-kernel __nv_aligned_device_malloc() function allocates at least size bytes from the device heap and returns a pointer to the allocated memory or NULL if insufficient memory exists to fulfill the requested size or alignment. The address of the allocated memory will be a multiple of align. align must be a non-zero power of 2.

+

The CUDA in-kernel free() function deallocates the memory pointed to by ptr, which must have been returned by a previous call to malloc() or __nv_aligned_device_malloc(). If ptr is NULL, the call to free() is ignored. Repeated calls to free() with the same ptr has undefined behavior.

+

The memory allocated by a given CUDA thread via malloc() or __nv_aligned_device_malloc() remains allocated for the lifetime of the CUDA context, or until it is explicitly released by a call to free(). It can be used by any other CUDA threads even from subsequent kernel launches. Any CUDA thread may free memory allocated by another thread, but care should be taken to ensure that the same pointer is not freed more than once.

+
+

+10.36.1. Heap Memory Allocation +

+

The device memory heap has a fixed size that must be specified before any program using malloc(), __nv_aligned_device_malloc() or free() is loaded into the context. A default heap of eight megabytes is allocated if any program uses malloc() or __nv_aligned_device_malloc() without explicitly specifying the heap size.

+

The following API functions get and set the heap size:

+
    +
  • cudaDeviceGetLimit(size_t* size, cudaLimitMallocHeapSize)

  • +
  • cudaDeviceSetLimit(cudaLimitMallocHeapSize, size_t size)

  • +
+

The heap size granted will be at least size bytes. cuCtxGetLimit()and cudaDeviceGetLimit() return the currently requested heap size.

+

The actual memory allocation for the heap occurs when a module is loaded into the context, either explicitly via the CUDA driver API (see Module), or implicitly via the CUDA runtime API (see CUDA Runtime). If the memory allocation fails, the module load will generate a CUDA_ERROR_SHARED_OBJECT_INIT_FAILED error.

+

Heap size cannot be changed once a module load has occurred and it does not resize dynamically according to need.

+

Memory reserved for the device heap is in addition to memory allocated through host-side CUDA API calls such as cudaMalloc().

+
+
+

+10.36.2. Interoperability with Host Memory API +

+

Memory allocated via device malloc() or __nv_aligned_device_malloc() cannot be freed using the runtime (i.e., by calling any of the free memory functions from Device Memory).

+

Similarly, memory allocated via the runtime (i.e., by calling any of the memory allocation functions from Device Memory) cannot be freed via free().

+

In addition, memory allocated by a call to malloc() or __nv_aligned_device_malloc() in device code cannot be used in any runtime or driver API calls (i.e. cudaMemcpy, cudaMemset, etc).

+
+
+

+10.36.3. Examples +

+
+

+10.36.3.1. Per Thread Allocation +

+

The following code sample:

+
+
#include <stdlib.h>
+#include <stdio.h>
+
+__global__ void mallocTest()
+{
+    size_t size = 123;
+    char* ptr = (char*)malloc(size);
+    memset(ptr, 0, size);
+    printf("Thread %d got pointer: %p\n", threadIdx.x, ptr);
+    free(ptr);
+}
+
+int main()
+{
+    // Set a heap size of 128 megabytes. Note that this must
+    // be done before any kernel is launched.
+    cudaDeviceSetLimit(cudaLimitMallocHeapSize, 128*1024*1024);
+    mallocTest<<<1, 5>>>();
+    cudaDeviceSynchronize();
+    return 0;
+}
+
+
+

will output:

+
+
Thread 0 got pointer: 00057020
+Thread 1 got pointer: 0005708c
+Thread 2 got pointer: 000570f8
+Thread 3 got pointer: 00057164
+Thread 4 got pointer: 000571d0
+
+
+

Notice how each thread encounters the malloc() and memset() commands and so receives and initializes its own allocation. (Exact pointer values will vary: these are illustrative.)

+
+
+

+10.36.3.2. Per Thread Block Allocation +

+
+
#include <stdlib.h>
+
+__global__ void mallocTest()
+{
+    __shared__ int* data;
+
+    // The first thread in the block does the allocation and then
+    // shares the pointer with all other threads through shared memory,
+    // so that access can easily be coalesced.
+    // 64 bytes per thread are allocated.
+    if (threadIdx.x == 0) {
+        size_t size = blockDim.x * 64;
+        data = (int*)malloc(size);
+    }
+    __syncthreads();
+
+    // Check for failure
+    if (data == NULL)
+        return;
+
+    // Threads index into the memory, ensuring coalescence
+    int* ptr = data;
+    for (int i = 0; i < 64; ++i)
+        ptr[i * blockDim.x + threadIdx.x] = threadIdx.x;
+
+    // Ensure all threads complete before freeing
+    __syncthreads();
+
+    // Only one thread may free the memory!
+    if (threadIdx.x == 0)
+        free(data);
+}
+
+int main()
+{
+    cudaDeviceSetLimit(cudaLimitMallocHeapSize, 128*1024*1024);
+    mallocTest<<<10, 128>>>();
+    cudaDeviceSynchronize();
+    return 0;
+}
+
+
+
+
+

+10.36.3.3. Allocation Persisting Between Kernel Launches +

+
+
#include <stdlib.h>
+#include <stdio.h>
+
+#define NUM_BLOCKS 20
+
+__device__ int* dataptr[NUM_BLOCKS]; // Per-block pointer
+
+__global__ void allocmem()
+{
+    // Only the first thread in the block does the allocation
+    // since we want only one allocation per block.
+    if (threadIdx.x == 0)
+        dataptr[blockIdx.x] = (int*)malloc(blockDim.x * 4);
+    __syncthreads();
+
+    // Check for failure
+    if (dataptr[blockIdx.x] == NULL)
+        return;
+
+    // Zero the data with all threads in parallel
+    dataptr[blockIdx.x][threadIdx.x] = 0;
+}
+
+// Simple example: store thread ID into each element
+__global__ void usemem()
+{
+    int* ptr = dataptr[blockIdx.x];
+    if (ptr != NULL)
+        ptr[threadIdx.x] += threadIdx.x;
+}
+
+// Print the content of the buffer before freeing it
+__global__ void freemem()
+{
+    int* ptr = dataptr[blockIdx.x];
+    if (ptr != NULL)
+        printf("Block %d, Thread %d: final value = %d\n",
+                      blockIdx.x, threadIdx.x, ptr[threadIdx.x]);
+
+    // Only free from one thread!
+    if (threadIdx.x == 0)
+        free(ptr);
+}
+
+int main()
+{
+    cudaDeviceSetLimit(cudaLimitMallocHeapSize, 128*1024*1024);
+
+    // Allocate memory
+    allocmem<<< NUM_BLOCKS, 10 >>>();
+
+    // Use memory
+    usemem<<< NUM_BLOCKS, 10 >>>();
+    usemem<<< NUM_BLOCKS, 10 >>>();
+    usemem<<< NUM_BLOCKS, 10 >>>();
+
+    // Free memory
+    freemem<<< NUM_BLOCKS, 10 >>>();
+
+    cudaDeviceSynchronize();
+
+    return 0;
+}
+
+
+
+
+
+
+

+10.37. Execution Configuration +

+

Any call to a __global__ function must specify the execution configuration for that call. The execution configuration defines the dimension of the grid and blocks that will be used to execute the function on the device, as well as the associated stream (see CUDA Runtime for a description of streams).

+

The execution configuration is specified by inserting an expression of the form <<< Dg, Db, Ns, S >>> between the function name and the parenthesized argument list, where:

+
    +
  • Dg is of type dim3 (see dim3) and specifies the dimension and size of the grid, such that Dg.x * Dg.y * Dg.z equals the number of blocks being launched;

  • +
  • Db is of type dim3 (see dim3) and specifies the dimension and size of each block, such that Db.x * Db.y * Db.z equals the number of threads per block;

  • +
  • Ns is of type size_t and specifies the number of bytes in shared memory that is dynamically allocated per block for this call in addition to the statically allocated memory; this dynamically allocated memory is used by any of the variables declared as an external array as mentioned in __shared__; Ns is an optional argument which defaults to 0;

  • +
  • S is of type cudaStream_t and specifies the associated stream; S is an optional argument which defaults to 0.

  • +
+

As an example, a function declared as

+
+
__global__ void Func(float* parameter);
+
+
+

must be called like this:

+
+
Func<<< Dg, Db, Ns >>>(parameter);
+
+
+

The arguments to the execution configuration are evaluated before the actual function arguments.

+

The function call will fail if Dg or Db are greater than the maximum sizes allowed for the device as specified in Compute Capabilities, or if Ns is greater than the maximum amount of shared memory available on the device, minus the amount of shared memory required for static allocation.

+

Compute capability 9.0 and above allows users to specify compile time thread block cluster dimensions, so that the kernel can use the cluster hierarchy in CUDA. Compile time cluster dimension can be specified using __cluster_dims__([x, [y, [z]]]). The example below shows compile time cluster size of 2 in X dimension and 1 in Y and Z dimension.

+
+
__global__ void __cluster_dims__(2, 1, 1) Func(float* parameter);
+
+
+

The default form of __cluster_dims__() specifies that a kernel is to be launched as a cluster grid. By not specifying a cluster dimension, the user is free to specify the dimension at launch time. Not specifying a dimension at launch time will result in a launch time error.

+

Thread block cluster dimensions can also be specified at runtime and kernel with the cluster can be launched using cudaLaunchKernelEx API. The API takes a configuration argument of type cudaLaunchConfig_t, kernel function pointer and kernel arguments. Runtime kernel configuration is shown in the example below.

+
+
__global__ void Func(float* parameter);
+
+
+// Kernel invocation with runtime cluster size
+{
+    cudaLaunchConfig_t config = {0};
+    // The grid dimension is not affected by cluster launch, and is still enumerated
+    // using number of blocks.
+    // The grid dimension should be a multiple of cluster size.
+    config.gridDim = Dg;
+    config.blockDim = Db;
+    config.dynamicSmemBytes = Ns;
+
+    cudaLaunchAttribute attribute[1];
+    attribute[0].id = cudaLaunchAttributeClusterDimension;
+    attribute[0].val.clusterDim.x = 2; // Cluster size in X-dimension
+    attribute[0].val.clusterDim.y = 1;
+    attribute[0].val.clusterDim.z = 1;
+    config.attrs = attribute;
+    config.numAttrs = 1;
+
+    float* parameter;
+    cudaLaunchKernelEx(&config, Func, parameter);
+}
+
+
+
+
+

+10.38. Launch Bounds +

+

As discussed in detail in Multiprocessor Level, the fewer registers a kernel uses, the more threads and thread blocks are likely to reside on a multiprocessor, which can improve performance.

+

Therefore, the compiler uses heuristics to minimize register usage while keeping register spilling (see Device Memory Accesses) and instruction count to a minimum. An application can optionally aid these heuristics by providing additional information to the compiler in the form of launch bounds that are specified using the __launch_bounds__() qualifier in the definition of a __global__ function:

+
+
__global__ void
+__launch_bounds__(maxThreadsPerBlock, minBlocksPerMultiprocessor, maxBlocksPerCluster)
+MyKernel(...)
+{
+    ...
+}
+
+
+
    +
  • maxThreadsPerBlock specifies the maximum number of threads per block with which the application will ever launch MyKernel(); it compiles to the .maxntidPTX directive.

  • +
  • minBlocksPerMultiprocessor is optional and specifies the desired minimum number of resident blocks per multiprocessor; it compiles to the .minnctapersmPTX directive.

  • +
  • maxBlocksPerCluster is optional and specifies the desired maximum number thread blocks per cluster with which the application will ever launch MyKernel(); it compiles to the .maxclusterrankPTX directive.

  • +
+

If launch bounds are specified, the compiler first derives from them the upper limit L on the number of registers the kernel should use to ensure that minBlocksPerMultiprocessor blocks (or a single block if minBlocksPerMultiprocessor is not specified) of maxThreadsPerBlock threads can reside on the multiprocessor (see Hardware Multithreading for the relationship between the number of registers used by a kernel and the number of registers allocated per block). The compiler then optimizes register usage in the following way:

+
    +
  • If the initial register usage is higher than L, the compiler reduces it further until it becomes less or equal to L, usually at the expense of more local memory usage and/or higher number of instructions;

  • +
  • +

    If the initial register usage is lower than L

    +
      +
    • If maxThreadsPerBlock is specified and minBlocksPerMultiprocessor is not, the compiler uses maxThreadsPerBlock to determine the register usage thresholds for the transitions between n and n+1 resident blocks (i.e., when using one less register makes room for an additional resident block as in the example of Multiprocessor Level) and then applies similar heuristics as when no launch bounds are specified;

    • +
    • If both minBlocksPerMultiprocessor and maxThreadsPerBlock are specified, the compiler may increase register usage as high as L to reduce the number of instructions and better hide single thread instruction latency.

    • +
    +
  • +
+

A kernel will fail to launch if it is executed with more threads per block than its launch bound maxThreadsPerBlock.

+

A kernel will fail to launch if it is executed with more thread blocks per cluster than its launch bound maxBlocksPerCluster.

+

Per thread resources required by a CUDA kernel might limit the maximum block size in an unwanted way. In order to maintain forward compatibility to future hardware and toolkits and to ensure that at least one thread block can run on an SM, developers should include the single argument __launch_bounds__(maxThreadsPerBlock) which specifies the largest block size that the kernel will be launched with. Failure to do so could lead to “too many resources requested for launch” errors. Providing the two argument version of __launch_bounds__(maxThreadsPerBlock,minBlocksPerMultiprocessor) can improve performance in some cases. The right value for minBlocksPerMultiprocessor should be determined using a detailed per kernel analysis.

+

Optimal launch bounds for a given kernel will usually differ across major architecture revisions. The sample code below shows how this is typically handled in device code using the __CUDA_ARCH__ macro introduced in Application Compatibility.

+
+
#define THREADS_PER_BLOCK          256
+#if __CUDA_ARCH__ >= 200
+    #define MY_KERNEL_MAX_THREADS  (2 * THREADS_PER_BLOCK)
+    #define MY_KERNEL_MIN_BLOCKS   3
+#else
+    #define MY_KERNEL_MAX_THREADS  THREADS_PER_BLOCK
+    #define MY_KERNEL_MIN_BLOCKS   2
+#endif
+
+// Device code
+__global__ void
+__launch_bounds__(MY_KERNEL_MAX_THREADS, MY_KERNEL_MIN_BLOCKS)
+MyKernel(...)
+{
+    ...
+}
+
+
+

In the common case where MyKernel is invoked with the maximum number of threads per block (specified as the first parameter of __launch_bounds__()), it is tempting to use MY_KERNEL_MAX_THREADS as the number of threads per block in the execution configuration:

+
+
// Host code
+MyKernel<<<blocksPerGrid, MY_KERNEL_MAX_THREADS>>>(...);
+
+
+

This will not work however since __CUDA_ARCH__ is undefined in host code as mentioned in Application Compatibility, so MyKernel will launch with 256 threads per block even when __CUDA_ARCH__ is greater or equal to 200. Instead the number of threads per block should be determined:

+
    +
  • +

    Either at compile time using a macro that does not depend on __CUDA_ARCH__, for example

    +
    +
    // Host code
    +MyKernel<<<blocksPerGrid, THREADS_PER_BLOCK>>>(...);
    +
    +
    +
  • +
  • +

    Or at runtime based on the compute capability

    +
    +
    // Host code
    +cudaGetDeviceProperties(&deviceProp, device);
    +int threadsPerBlock =
    +          (deviceProp.major >= 2 ?
    +                    2 * THREADS_PER_BLOCK : THREADS_PER_BLOCK);
    +MyKernel<<<blocksPerGrid, threadsPerBlock>>>(...);
    +
    +
    +
  • +
+

Register usage is reported by the --ptxas-options=-v compiler option. The number of resident blocks can be derived from the occupancy reported by the CUDA profiler (see Device Memory Accesses for a definition of occupancy).

+
+
+

+10.39. Maximum Number of Registers per Thread +

+

To provide a mechanism for low-level performance tuning, CUDA C++ provides the __maxnreg__() function qualifier to pass performance tuning information to the backend optimizing compiler. The +__maxnreg__() qualifier specifies the maximum number of registers to be allocated to a single thread in a thread block. In the definition of a __global__ function:

+
+
__global__ void
+__maxnreg__(maxNumberRegistersPerThread)
+MyKernel(...)
+{
+    ...
+}
+
+
+
    +
  • maxNumberRegistersPerThread specifies the maximum number of registers to be allocated to a single thread in a thread block of the kernel MyKernel(); it compiles to the .maxnregPTX directive.

  • +
+

The __launch_bounds__() and __maxnreg__() qualifiers cannot be applied to the same kernel.

+

Register usage can also be controlled for all __global__ functions in a file using the maxrregcount compiler option. The value of maxrregcount is ignored for functions with the __maxnreg__ qualifier.

+
+
+

+10.40. #pragma unroll +

+

By default, the compiler unrolls small loops with a known trip count. The #pragma unroll directive however can be used to control unrolling of any given loop. It must be placed immediately before the loop and only applies to that loop. It is optionally followed by an integral constant expression (ICE)6. If the ICE is absent, the loop will be completely unrolled if its trip count is constant. If the ICE evaluates to 1, the compiler will not unroll the loop. The pragma will be ignored if the ICE evaluates to a non-positive integer or to an integer greater than the maximum value representable by the int data type.

+

Examples:

+
+
struct S1_t { static const int value = 4; };
+template <int X, typename T2>
+__device__ void foo(int *p1, int *p2) {
+
+// no argument specified, loop will be completely unrolled
+#pragma unroll
+for (int i = 0; i < 12; ++i)
+  p1[i] += p2[i]*2;
+
+// unroll value = 8
+#pragma unroll (X+1)
+for (int i = 0; i < 12; ++i)
+  p1[i] += p2[i]*4;
+
+// unroll value = 1, loop unrolling disabled
+#pragma unroll 1
+for (int i = 0; i < 12; ++i)
+  p1[i] += p2[i]*8;
+
+// unroll value = 4
+#pragma unroll (T2::value)
+for (int i = 0; i < 12; ++i)
+  p1[i] += p2[i]*16;
+}
+
+__global__ void bar(int *p1, int *p2) {
+foo<7, S1_t>(p1, p2);
+}
+
+
+
+
+

+10.41. SIMD Video Instructions +

+

PTX ISA version 3.0 includes SIMD (Single Instruction, Multiple Data) video instructions which operate on pairs of 16-bit values and quads of 8-bit values. These are available on devices of compute capability 3.0.

+

The SIMD video instructions are:

+
    +
  • vadd2, vadd4

  • +
  • vsub2, vsub4

  • +
  • vavrg2, vavrg4

  • +
  • vabsdiff2, vabsdiff4

  • +
  • vmin2, vmin4

  • +
  • vmax2, vmax4

  • +
  • vset2, vset4

  • +
+

PTX instructions, such as the SIMD video instructions, can be included in CUDA programs by way of the assembler, asm(), statement.

+

The basic syntax of an asm() statement is:

+
+
asm("template-string" : "constraint"(output) : "constraint"(input)"));
+
+
+

An example of using the vabsdiff4 PTX instruction is:

+
+
asm("vabsdiff4.u32.u32.u32.add" " %0, %1, %2, %3;": "=r" (result):"r" (A), "r" (B), "r" (C));
+
+
+

This uses the vabsdiff4 instruction to compute an integer quad byte SIMD sum of absolute differences. The absolute difference value is computed for each byte of the unsigned integers A and B in SIMD fashion. The optional accumulate operation (.add) is specified to sum these differences.

+

Refer to the document “Using Inline PTX Assembly in CUDA” for details on using the assembly statement in your code. Refer to the PTX ISA documentation (“Parallel Thread Execution ISA Version 3.0” for example) for details on the PTX instructions for the version of PTX that you are using.

+
+
+

+10.42. Diagnostic Pragmas +

+

The following pragmas may be used to control the error severity used when a given diagnostic message is issued.

+
+
#pragma nv_diag_suppress
+#pragma nv_diag_warning
+#pragma nv_diag_error
+#pragma nv_diag_default
+#pragma nv_diag_once
+
+
+

Uses of these pragmas have the following form:

+
+
#pragma nv_diag_xxx error_number, error_number ...
+
+
+

The diagnostic affected is specified using an error number showed in a warning message. Any diagnostic may be overridden to be an error, but only warnings may have their severity suppressed or be restored to a warning after being promoted to an error. The nv_diag_default pragma is used to return the severity of a diagnostic to the one that was in effect before any pragmas were issued (i.e., the normal severity of the message as modified by any command-line options). The following example suppresses the "declared but never referenced" warning on the declaration of foo:

+
+
#pragma nv_diag_suppress 177
+void foo()
+{
+  int i=0;
+}
+#pragma nv_diag_default 177
+void bar()
+{
+  int i=0;
+}
+
+
+

The following pragmas may be used to save and restore the current diagnostic pragma state:

+
+
#pragma nv_diagnostic push
+#pragma nv_diagnostic pop
+
+
+

Examples:

+
+
#pragma nv_diagnostic push
+#pragma nv_diag_suppress 177
+void foo()
+{
+  int i=0;
+}
+#pragma nv_diagnostic pop
+void bar()
+{
+  int i=0;
+}
+
+
+

Note that the pragmas only affect the nvcc CUDA frontend compiler; they have no effect on the host compiler.

+

Removal Notice: The support of diagnostic pragmas without nv_ prefix are removed from CUDA 12.0, if the pragmas are inside the device code, warning unrecognized #pragma in device code will be emitted, otherwise they will be passed to the host compiler. If they are intended for CUDA code, use the pragmas with nv_ prefix instead.

+
+
4
+
+

When the enclosing __host__ function is a template, nvcc may currently fail to issue a diagnostic message in some cases; this behavior may change in the future.

+
+
5
+
+

The intent is to prevent the host compiler from encountering the call to the function if the host compiler does not support it.

+
+
+6(1,2,3) +
+
+

See the C++ Standard for definition of integral constant expression.

+
+
+
+
+

+10.43. Custom ABI Pragmas +

+

The #pragma nv_abi directive enables applications compiled in separate compilation mode to achieve performance similar to that of whole program compilation.

+

The syntax for using this pragma is as follows, where ICE refers to any integral constant expression (ICE): 6.

+
+
#pragma nv_abi preserve_n_data(ICE) preserve_n_control(ICE)
+
+
+

Note, the arguments that follow #pragma nv_abi are optional and can be provided in any order; however, at least one argument is required.

+

The preserve_n arguments set a limit on the number of registers preserved during a function call:

+
    +
  • preserve_n_data(ICE) limits the number of data registers, and

  • +
  • preserve_n_control(ICE) limits the number of control registers.

  • +
+

#pragma nv_abi can be placed immediately before a device function declaration or definition. Alternatively, it can be placed directly before an indirect function call within a C++ expression statement inside a device function. Note, indirect function calls to free functions are supported, but indirect calls through function argument references or class member functions are not.

+

When the pragma is applied to a device function declaration or definition, it modifies the custom ABI properties for any calls to that function. When placed at an indirect function call site, the pragma affects the ABI properties for that indirect function call. The key point is that unlike direct function calls, where you can place the pragma before a function declaration or definition, #pragma nv_abi only affects indirect function calls when the pragma is placed before a call site.

+

As shown in the following example, we have two device functions, foo() and bar(). In this example the pragma is placed before the call site of the function pointer fptr to modify the ABI properties of the indirect function call. Notice that placing the pragma before the direct call does not affect the ABI properties of the call. To alter the ABI properties of a direct function call, the pragma must be placed before the function declaration or definition.

+
+
__device__ int foo()
+{
+  int value{0};
+  ...
+  return value;
+}
+
+__device__ int bar()
+{
+  int value{0};
+  ...
+  return value;
+}
+
+__device__ void baz()
+{
+  int result{0};
+  int (*fptr)() = foo;  // function pointer
+
+  #pragma nv_abi preserve_n_data(16) preserve_n_control(8)
+  result = fptr();      // The pragma affects the indirect call to foo() via fptr
+
+  #pragma nv_abi preserve_n_data(16) preserve_n_control(8)
+  result = (*fptr)();   // Alternate syntax for the indirect call to foo()
+
+  #pragma nv_abi preserve_n_data(16) preserve_n_control(8)
+  result += bar();      // The pragma does NOT affect the direct call to bar()
+}
+
+
+

As shown in the following example, to modify direct function calls, you must apply the pragma to the function declaration or definition.

+
+
#pragma nv_abi preserve_n_data(16)
+__device__ void foo();
+
+
+

Note that a program is ill-formed if the pragma arguments for a function declaration and its corresponding definition do not match.

+
+
+

+10.44. CUDA C++ Memory Model +

+

The CUDA C++ Memory Model extends the ISO C++ Memory Model as documented in the CUDA C++ Memory Model documentation.

+
+
+

+10.45. CUDA C++ Execution Model +

+

The CUDA C++ Exeuction Model extends the ISO C++ execution model as documented in the CUDA C++ Execution Model documentation.

+
+
+
+

+11. Cooperative Groups +

+
+

Warning

+

This document has been replaced by a new CUDA Programming Guide. The information in this document should be considered legacy, and this document is no longer being updated as of CUDA 13.0. Please refer to the CUDA Programming Guide for up-to-date information on CUDA.

+
+
+

+11.1. Introduction +

+

Cooperative Groups is an extension to the CUDA programming model, introduced in CUDA 9, for organizing groups of communicating threads. Cooperative Groups allows developers to express the granularity at which threads are communicating, helping them to express richer, more efficient parallel decompositions.

+

Historically, the CUDA programming model has provided a single, simple construct for synchronizing cooperating threads: a barrier across all threads of a thread block, as implemented with the __syncthreads() intrinsic function. However, programmers would like to define and synchronize groups of threads at other granularities to enable greater performance, design flexibility, and software reuse in the form of “collective” group-wide function interfaces. In an effort to express broader patterns of parallel interaction, many performance-oriented programmers have resorted to writing their own ad hoc and unsafe primitives for synchronizing threads within a single warp, or across sets of thread blocks running on a single GPU. Whilst the performance improvements achieved have often been valuable, this has resulted in an ever-growing collection of brittle code that is expensive to write, tune, and maintain over time and across GPU generations. Cooperative Groups addresses this by providing a safe and future-proof mechanism to enable performant code.

+
+
+

+11.2. What’s New in Cooperative Groups +

+
+

+11.2.1. CUDA 13.0 +

+
    +
  • multi_grid_group was removed.

  • +
+
+
+

+11.2.2. CUDA 12.2 +

+
    +
  • barrier_arrive and barrier_wait member functions were added for grid_group and thread_block. Description of the API is available here.

  • +
+
+
+

+11.2.3. CUDA 12.1 +

+ +
+
+

+11.2.4. CUDA 12.0 +

+
    +
  • +

    The following experimental APIs are now moved to the main namespace:

    +
      +
    • asynchronous reduce and scan update added in CUDA 11.7

    • +
    • thread_block_tile larger than 32 added in CUDA 11.1

    • +
    +
  • +
  • It is no longer required to provide memory using the block_tile_memory object in order to create these large tiles on Compute Capability 8.0 or higher.

  • +
+
+
+
+

+11.3. Programming Model Concept +

+

The Cooperative Groups programming model describes synchronization patterns both within and across CUDA thread blocks. It provides both the means for applications to define their own groups of threads, and the interfaces to synchronize them. It also provides new launch APIs that enforce certain restrictions and therefore can guarantee the synchronization will work. These primitives enable new patterns of cooperative parallelism within CUDA, including producer-consumer parallelism, opportunistic parallelism, and global synchronization across the entire Grid.

+

The Cooperative Groups programming model consists of the following elements:

+
    +
  • Data types for representing groups of cooperating threads;

  • +
  • Operations to obtain implicit groups defined by the CUDA launch API (e.g., thread blocks);

  • +
  • Collectives for partitioning existing groups into new groups;

  • +
  • Collective Algorithms for data movement and manipulation (e.g. memcpy_async, reduce, scan);

  • +
  • An operation to synchronize all threads within the group;

  • +
  • Operations to inspect the group properties;

  • +
  • Collectives that expose low-level, group-specific and often HW accelerated, operations.

  • +
+

The main concept in Cooperative Groups is that of objects naming the set of threads that are part of it. This expression of groups as first-class program objects improves software composition, since collective functions can receive an explicit object representing the group of participating threads. This object also makes programmer intent explicit, which eliminates unsound architectural assumptions that result in brittle code, undesirable restrictions upon compiler optimizations, and better compatibility with new GPU generations.

+

To write efficient code, its best to use specialized groups (going generic loses a lot of compile time optimizations), and pass these group objects by reference to functions that intend to use these threads in some cooperative fashion.

+

Cooperative Groups requires CUDA 9.0 or later. To use Cooperative Groups, include the header file:

+
+
// Primary header is compatible with pre-C++11, collective algorithm headers require C++11
+#include <cooperative_groups.h>
+// Optionally include for memcpy_async() collective
+#include <cooperative_groups/memcpy_async.h>
+// Optionally include for reduce() collective
+#include <cooperative_groups/reduce.h>
+// Optionally include for inclusive_scan() and exclusive_scan() collectives
+#include <cooperative_groups/scan.h>
+
+
+

and use the Cooperative Groups namespace:

+
+
using namespace cooperative_groups;
+// Alternatively use an alias to avoid polluting the namespace with collective algorithms
+namespace cg = cooperative_groups;
+
+
+

The code can be compiled in a normal way using nvcc, however if you wish to use memcpy_async, reduce or scan functionality and your host compiler’s default dialect is not C++11 or higher, then you must add --std=c++11 to the command line.

+
+

+11.3.1. Composition Example +

+

To illustrate the concept of groups, this example attempts to perform a block-wide sum reduction. Previously, there were hidden constraints on the implementation when writing this code:

+
+
__device__ int sum(int *x, int n) {
+    // ...
+    __syncthreads();
+    return total;
+}
+
+__global__ void parallel_kernel(float *x) {
+    // ...
+    // Entire thread block must call sum
+    sum(x, n);
+}
+
+
+

All threads in the thread block must arrive at the __syncthreads() barrier, however, this constraint is hidden from the developer who might want to use sum(…). With Cooperative Groups, a better way of writing this would be:

+
+
__device__ int sum(const thread_block& g, int *x, int n) {
+    // ...
+    g.sync()
+    return total;
+}
+
+__global__ void parallel_kernel(...) {
+    // ...
+    // Entire thread block must call sum
+    thread_block tb = this_thread_block();
+    sum(tb, x, n);
+    // ...
+}
+
+
+
+
+
+

+11.4. Group Types +

+
+

+11.4.1. Implicit Groups +

+

Implicit groups represent the launch configuration of the kernel. Regardless of how your kernel is written, it always has a set number of threads, blocks and block dimensions, a single grid and grid dimensions. In addition, if the multi-device cooperative launch API is used, it can have multiple grids (single grid per device). These groups provide a starting point for decomposition into finer grained groups which are typically HW accelerated and are more specialized for the problem the developer is solving.

+

Although you can create an implicit group anywhere in the code, it is dangerous to do so. Creating a handle for an implicit group is a collective operation—all threads in the group must participate. If the group was created in a conditional branch that not all threads reach, this can lead to deadlocks or data corruption. For this reason, it is recommended that you create a handle for the implicit group upfront (as early as possible, before any branching has occurred) and use that handle throughout the kernel. Group handles must be initialized at declaration time (there is no default constructor) for the same reason and copy-constructing them is discouraged.

+
+

+11.4.1.1. Thread Block Group +

+

Any CUDA programmer is already familiar with a certain group of threads: the thread block. The Cooperative Groups extension introduces a new datatype, thread_block, to explicitly represent this concept within the kernel.

+

class thread_block;

+

Constructed via:

+
+
thread_block g = this_thread_block();
+
+
+

Public Member Functions:

+

static void sync(): Synchronize the threads named in the group, equivalent to g.barrier_wait(g.barrier_arrive())

+

thread_block::arrival_token barrier_arrive(): Arrive on the thread_block barrier, returns a token that needs to be passed into barrier_wait(). More details here

+

void barrier_wait(thread_block::arrival_token&& t): Wait on the thread_block barrier, takes arrival token returned from barrier_arrive() as an rvalue reference. More details here

+

static unsigned int thread_rank(): Rank of the calling thread within [0, num_threads)

+

static dim3 group_index(): 3-Dimensional index of the block within the launched grid

+

static dim3 thread_index(): 3-Dimensional index of the thread within the launched block

+

static dim3 dim_threads(): Dimensions of the launched block in units of threads

+

static unsigned int num_threads(): Total number of threads in the group

+

Legacy member functions (aliases):

+

static unsigned int size(): Total number of threads in the group (alias of num_threads())

+

static dim3 group_dim(): Dimensions of the launched block (alias of dim_threads())

+

Example:

+
+
/// Loading an integer from global into shared memory
+__global__ void kernel(int *globalInput) {
+    __shared__ int x;
+    thread_block g = this_thread_block();
+    // Choose a leader in the thread block
+    if (g.thread_rank() == 0) {
+        // load from global into shared for all threads to work with
+        x = (*globalInput);
+    }
+    // After loading data into shared memory, you want to synchronize
+    // if all threads in your thread block need to see it
+    g.sync(); // equivalent to __syncthreads();
+}
+
+
+

Note: that all threads in the group must participate in collective operations, or the behavior is undefined.

+

Related: The thread_block datatype is derived from the more generic thread_group datatype, which can be used to represent a wider class of groups.

+
+
+

+11.4.1.2. Cluster Group +

+

This group object represents all the threads launched in a single cluster. Refer to Thread Block Clusters. The APIs are available on all hardware with Compute Capability 9.0+. In such cases, when a non-cluster grid is launched, the APIs assume a 1x1x1 cluster.

+

class cluster_group;

+

Constructed via:

+
+
cluster_group g = this_cluster();
+
+
+

Public Member Functions:

+

static void sync(): Synchronize the threads named in the group, equivalent to g.barrier_wait(g.barrier_arrive())

+

static cluster_group::arrival_token barrier_arrive(): Arrive on the cluster barrier, returns a token that needs to be passed into barrier_wait(). More details here

+

static void barrier_wait(cluster_group::arrival_token&& t): Wait on the cluster barrier, takes arrival token returned from barrier_arrive() as a rvalue reference. More details here

+

static unsigned int thread_rank(): Rank of the calling thread within [0, num_threads)

+

static unsigned int block_rank(): Rank of the calling block within [0, num_blocks)

+

static unsigned int num_threads(): Total number of threads in the group

+

static unsigned int num_blocks(): Total number of blocks in the group

+

static dim3 dim_threads(): Dimensions of the launched cluster in units of threads

+

static dim3 dim_blocks(): Dimensions of the launched cluster in units of blocks

+

static dim3 block_index(): 3-Dimensional index of the calling block within the launched cluster

+

static unsigned int query_shared_rank(const void *addr): Obtain the block rank to which a shared memory address belongs

+

static T* map_shared_rank(T *addr, int rank): Obtain the address of a shared memory variable of another block in the cluster

+

Legacy member functions (aliases):

+

static unsigned int size(): Total number of threads in the group (alias of num_threads())

+
+
+

+11.4.1.3. Grid Group +

+

This group object represents all the threads launched in a single grid. APIs other than sync() are available at all times, but to be able to synchronize across the grid, you need to use the cooperative launch API.

+

class grid_group;

+

Constructed via:

+
+
grid_group g = this_grid();
+
+
+

Public Member Functions:

+

bool is_valid() const: Returns whether the grid_group can synchronize

+

void sync() const: Synchronize the threads named in the group, equivalent to g.barrier_wait(g.barrier_arrive())

+

grid_group::arrival_token barrier_arrive(): Arrive on the grid barrier, returns a token that needs to be passed into barrier_wait(). More details here

+

void barrier_wait(grid_group::arrival_token&& t): Wait on the grid barrier, takes arrival token returned from barrier_arrive() as a rvalue reference. More details here

+

static unsigned long long thread_rank(): Rank of the calling thread within [0, num_threads)

+

static unsigned long long block_rank(): Rank of the calling block within [0, num_blocks)

+

static unsigned long long cluster_rank(): Rank of the calling cluster within [0, num_clusters)

+

static unsigned long long num_threads(): Total number of threads in the group

+

static unsigned long long num_blocks(): Total number of blocks in the group

+

static unsigned long long num_clusters(): Total number of clusters in the group

+

static dim3 dim_blocks(): Dimensions of the launched grid in units of blocks

+

static dim3 dim_clusters(): Dimensions of the launched grid in units of clusters

+

static dim3 block_index(): 3-Dimensional index of the block within the launched grid

+

static dim3 cluster_index(): 3-Dimensional index of the cluster within the launched grid

+

Legacy member functions (aliases):

+

static unsigned long long size(): Total number of threads in the group (alias of num_threads())

+

static dim3 group_dim(): Dimensions of the launched grid (alias of dim_blocks())

+
+
+
+

+11.4.2. Explicit Groups +

+
+

+11.4.2.1. Thread Block Tile +

+

A templated version of a tiled group, where a template parameter is used to specify the size of the tile - with this known at compile time there is the potential for more optimal execution.

+
+
template <unsigned int Size, typename ParentT = void>
+class thread_block_tile;
+
+
+

Constructed via:

+
+
template <unsigned int Size, typename ParentT>
+_CG_QUALIFIER thread_block_tile<Size, ParentT> tiled_partition(const ParentT& g)
+
+
+

Size must be a power of 2 and less than or equal to 1024. Notes section describes extra steps needed to create tiles of size larger than 32 on hardware with Compute Capability 7.5 or lower.

+

ParentT is the parent-type from which this group was partitioned. It is automatically inferred, but a value of void will store this information in the group handle rather than in the type.

+

Public Member Functions:

+

void sync() const: Synchronize the threads named in the group

+

unsigned long long num_threads() const: Total number of threads in the group

+

unsigned long long thread_rank() const: Rank of the calling thread within [0, num_threads)

+

unsigned long long meta_group_size() const: Returns the number of groups created when the parent group was partitioned.

+

unsigned long long meta_group_rank() const: Linear rank of the group within the set of tiles partitioned from a parent group (bounded by meta_group_size)

+

T shfl(T var, unsigned int src_rank) const: Refer to Warp Shuffle Functions, Note: For sizes larger than 32 all threads in the group have to specify the same src_rank, otherwise the behavior is undefined.

+

T shfl_up(T var, int delta) const: Refer to Warp Shuffle Functions, available only for sizes lower or equal to 32.

+

T shfl_down(T var, int delta) const: Refer to Warp Shuffle Functions, available only for sizes lower or equal to 32.

+

T shfl_xor(T var, int delta) const: Refer to Warp Shuffle Functions, available only for sizes lower or equal to 32.

+

int any(int predicate) const: Refer to Warp Vote Functions

+

int all(int predicate) const: Refer to Warp Vote Functions

+

unsigned int ballot(int predicate) const: Refer to Warp Vote Functions, available only for sizes lower or equal to 32.

+

unsigned int match_any(T val) const: Refer to Warp Match Functions, available only for sizes lower or equal to 32.

+

unsigned int match_all(T val, int &pred) const: Refer to Warp Match Functions, available only for sizes lower or equal to 32.

+

Legacy member functions (aliases):

+

unsigned long long size() const: Total number of threads in the group (alias of num_threads())

+

Notes:

+
    +
  • thread_block_tile templated data structure is being used here, the size of the group is passed to the tiled_partition call as a template parameter rather than an argument.

  • +
  • +

    shfl, shfl_up, shfl_down, and shfl_xor functions accept objects of any type when compiled with C++11 or later. This means it’s possible to shuffle non-integral types as long as they satisfy the below constraints:

    +
      +
    • Qualifies as trivially copyable i.e., is_trivially_copyable<T>::value == true

    • +
    • sizeof(T) <= 32 for tile sizes lower or equal 32, sizeof(T) <= 8 for larger tiles

    • +
    +
  • +
  • +

    On hardware with Compute Capability 7.5 or lower tiles of size larger than 32 need small amount of memory reserved for them. This can be done using cooperative_groups::block_tile_memory struct template that has to reside in either shared or global memory.

    +
    +
    template <unsigned int MaxBlockSize = 1024>
    +struct block_tile_memory;
    +
    +
    +

    MaxBlockSize Specifies the maximal number of threads in the current thread block. This parameter can be used to minimize the shared memory usage of block_tile_memory in kernels launched only with smaller thread counts.

    +

    This block_tile_memory needs be then passed into cooperative_groups::this_thread_block, allowing the resulting thread_block to be partitioned into tiles of sizes larger than 32. Overload of this_thread_block accepting block_tile_memory argument is a collective operation and has to be called with all threads in the thread_block.

    +

    block_tile_memory can be used on hardware with Compute Capability 8.0 or higher in order to be able to write one source targeting multiple different Compute Capabilities. It should consume no memory when instantiated in shared memory in cases where its not required.

    +
  • +
+

Examples:

+
+
/// The following code will create two sets of tiled groups, of size 32 and 4 respectively:
+/// The latter has the provenance encoded in the type, while the first stores it in the handle
+thread_block block = this_thread_block();
+thread_block_tile<32> tile32 = tiled_partition<32>(block);
+thread_block_tile<4, thread_block> tile4 = tiled_partition<4>(block);
+
+
+
+
/// The following code will create tiles of size 128 on all Compute Capabilities.
+/// block_tile_memory can be omitted on Compute Capability 8.0 or higher.
+__global__ void kernel(...) {
+    // reserve shared memory for thread_block_tile usage,
+    //   specify that block size will be at most 256 threads.
+    __shared__ block_tile_memory<256> shared;
+    thread_block thb = this_thread_block(shared);
+
+    // Create tiles with 128 threads.
+    auto tile = tiled_partition<128>(thb);
+
+    // ...
+}
+
+
+
+
+11.4.2.1.1. Warp-Synchronous Code Pattern +
+

Developers might have had warp-synchronous codes that they previously made implicit assumptions about the warp size and would code around that number. Now this needs to be specified explicitly.

+
+
__global__ void cooperative_kernel(...) {
+    // obtain default "current thread block" group
+    thread_block my_block = this_thread_block();
+
+    // subdivide into 32-thread, tiled subgroups
+    // Tiled subgroups evenly partition a parent group into
+    // adjacent sets of threads - in this case each one warp in size
+    auto my_tile = tiled_partition<32>(my_block);
+
+    // This operation will be performed by only the
+    // first 32-thread tile of each block
+    if (my_tile.meta_group_rank() == 0) {
+        // ...
+        my_tile.sync();
+    }
+}
+
+
+
+
+
+11.4.2.1.2. Single Thread Group +
+

Group representing the current thread can be obtained from this_thread function:

+
+
thread_block_tile<1> this_thread();
+
+
+

The following memcpy_async API uses a thread_group, to copy an int element from source to destination:

+
+
#include <cooperative_groups.h>
+#include <cooperative_groups/memcpy_async.h>
+
+cooperative_groups::memcpy_async(cooperative_groups::this_thread(), dest, src, sizeof(int));
+
+
+

More detailed examples of using this_thread to perform asynchronous copies can be found in the Single-Stage Asynchronous Data Copies using cuda::pipeline and Multi-Stage Asynchronous Data Copies using cuda::pipeline sections.

+
+
+
+

+11.4.2.2. Coalesced Groups +

+

In CUDA’s SIMT architecture, at the hardware level the multiprocessor executes threads in groups of 32 called warps. If there exists a data-dependent conditional branch in the application code such that threads within a warp diverge, then the warp serially executes each branch disabling threads not on that path. The threads that remain active on the path are referred to as coalesced. Cooperative Groups has functionality to discover, and create, a group containing all coalesced threads.

+

Constructing the group handle via coalesced_threads() is opportunistic. It returns the set of active threads at that point in time, and makes no guarantee about which threads are returned (as long as they are active) or that they will stay coalesced throughout execution (they will be brought back together for the execution of a collective but can diverge again afterwards).

+

class coalesced_group;

+

Constructed via:

+
+
coalesced_group active = coalesced_threads();
+
+
+

Public Member Functions:

+

void sync() const: Synchronize the threads named in the group

+

unsigned long long num_threads() const: Total number of threads in the group

+

unsigned long long thread_rank() const: Rank of the calling thread within [0, num_threads)

+

unsigned long long meta_group_size() const: Returns the number of groups created when the parent group was partitioned. If this group was created by querying the set of active threads, for example coalesced_threads() the value of meta_group_size() will be 1.

+

unsigned long long meta_group_rank() const: Linear rank of the group within the set of tiles partitioned from a parent group (bounded by meta_group_size). If this group was created by querying the set of active threads, e.g. coalesced_threads() the value of meta_group_rank() will always be 0.

+

T shfl(T var, unsigned int src_rank) const: Refer to Warp Shuffle Functions

+

T shfl_up(T var, int delta) const: Refer to Warp Shuffle Functions

+

T shfl_down(T var, int delta) const: Refer to Warp Shuffle Functions

+

int any(int predicate) const: Refer to Warp Vote Functions

+

int all(int predicate) const: Refer to Warp Vote Functions

+

unsigned int ballot(int predicate) const: Refer to Warp Vote Functions

+

unsigned int match_any(T val) const: Refer to Warp Match Functions

+

unsigned int match_all(T val, int &pred) const: Refer to Warp Match Functions

+

Legacy member functions (aliases):

+

unsigned long long size() const: Total number of threads in the group (alias of num_threads())

+

Notes:

+

shfl, shfl_up, and shfl_down functions accept objects of any type when compiled with C++11 or later. This means it’s possible to shuffle non-integral types as long as they satisfy the below constraints:

+
    +
  • Qualifies as trivially copyable i.e. is_trivially_copyable<T>::value == true

  • +
  • sizeof(T) <= 32

  • +
+

Example:

+
+
/// Consider a situation whereby there is a branch in the
+/// code in which only the 2nd, 4th and 8th threads in each warp are
+/// active. The coalesced_threads() call, placed in that branch, will create (for each
+/// warp) a group, active, that has three threads (with
+/// ranks 0-2 inclusive).
+__global__ void kernel(int *globalInput) {
+    // Lets say globalInput says that threads 2, 4, 8 should handle the data
+    if (threadIdx.x == *globalInput) {
+        coalesced_group active = coalesced_threads();
+        // active contains 0-2 inclusive
+        active.sync();
+    }
+}
+
+
+
+
+11.4.2.2.1. Discovery Pattern +
+

Commonly developers need to work with the current active set of threads. No assumption is made about the threads that are present, and instead developers work with the threads that happen to be there. This is seen in the following “aggregating atomic increment across threads in a warp” example (written using the correct CUDA 9.0 set of intrinsics):

+
+
{
+    unsigned int writemask = __activemask();
+    unsigned int total = __popc(writemask);
+    unsigned int prefix = __popc(writemask & __lanemask_lt());
+    // Find the lowest-numbered active lane
+    int elected_lane = __ffs(writemask) - 1;
+    int base_offset = 0;
+    if (prefix == 0) {
+        base_offset = atomicAdd(p, total);
+    }
+    base_offset = __shfl_sync(writemask, base_offset, elected_lane);
+    int thread_offset = prefix + base_offset;
+    return thread_offset;
+}
+
+
+

This can be re-written with Cooperative Groups as follows:

+
+
{
+    cg::coalesced_group g = cg::coalesced_threads();
+    int prev;
+    if (g.thread_rank() == 0) {
+        prev = atomicAdd(p, g.num_threads());
+    }
+    prev = g.thread_rank() + g.shfl(prev, 0);
+    return prev;
+}
+
+
+
+
+
+
+
+

+11.5. Group Partitioning +

+
+

+11.5.1. tiled_partition +

+
+
template <unsigned int Size, typename ParentT>
+thread_block_tile<Size, ParentT> tiled_partition(const ParentT& g);
+
+
+
+
thread_group tiled_partition(const thread_group& parent, unsigned int tilesz);
+
+
+

The tiled_partition method is a collective operation that partitions the parent group into a one-dimensional, row-major, tiling of subgroups. A total of ((size(parent)/tilesz) subgroups will be created, therefore the parent group size must be evenly divisible by the Size. The allowed parent groups are thread_block or thread_block_tile.

+

The implementation may cause the calling thread to wait until all the members of the parent group have invoked the operation before resuming execution. Functionality is limited to native hardware sizes, 1/2/4/8/16/32 and the cg::size(parent) must be greater than the Size parameter. The templated version of tiled_partition supports 64/128/256/512 sizes as well, but some additional steps are required on Compute Capability 7.5 or lower, refer to Thread Block Tile for details.

+

Codegen Requirements: Compute Capability 5.0 minimum, C++11 for sizes larger than 32

+

Example:

+
+
/// The following code will create a 32-thread tile
+thread_block block = this_thread_block();
+thread_block_tile<32> tile32 = tiled_partition<32>(block);
+
+
+

We can partition each of these groups into even smaller groups, each of size 4 threads:

+
+
auto tile4 = tiled_partition<4>(tile32);
+// or using a general group
+// thread_group tile4 = tiled_partition(tile32, 4);
+
+
+

If, for instance, if we were to then include the following line of code:

+
+
if (tile4.thread_rank()==0) printf("Hello from tile4 rank 0\n");
+
+
+

then the statement would be printed by every fourth thread in the block: the threads of rank 0 in each tile4 group, which correspond to those threads with ranks 0,4,8,12,etc. in the block group.

+
+
+

+11.5.2. labeled_partition +

+
+
template <typename Label>
+coalesced_group labeled_partition(const coalesced_group& g, Label label);
+
+
+
+
template <unsigned int Size, typename Label>
+coalesced_group labeled_partition(const thread_block_tile<Size>& g, Label label);
+
+
+

The labeled_partition method is a collective operation that partitions the parent group into one-dimensional subgroups within which the threads are coalesced. The implementation will evaluate a condition label and assign threads that have the same value for label into the same group.

+

Label can be any integral type.

+

The implementation may cause the calling thread to wait until all the members of the parent group have invoked the operation before resuming execution.

+

Note: This functionality is still being evaluated and may slightly change in the future.

+

Codegen Requirements: Compute Capability 7.0 minimum, C++11

+
+
+

+11.5.3. binary_partition +

+
+
coalesced_group binary_partition(const coalesced_group& g, bool pred);
+
+
+
+
template <unsigned int Size>
+coalesced_group binary_partition(const thread_block_tile<Size>& g, bool pred);
+
+
+

The binary_partition() method is a collective operation that partitions the parent group into one-dimensional subgroups within which the threads are coalesced. The implementation will evaluate a predicate and assign threads that have the same value into the same group. This is a specialized form of labeled_partition(), where the label can only be 0 or 1.

+

The implementation may cause the calling thread to wait until all the members of the parent group have invoked the operation before resuming execution.

+

Note: This functionality is still being evaluated and may slightly change in the future.

+

Codegen Requirements: Compute Capability 7.0 minimum, C++11

+

Example:

+
+
/// This example divides a 32-sized tile into a group with odd
+/// numbers and a group with even numbers
+_global__ void oddEven(int *inputArr) {
+    auto block = cg::this_thread_block();
+    auto tile32 = cg::tiled_partition<32>(block);
+
+    // inputArr contains random integers
+    int elem = inputArr[block.thread_rank()];
+    // after this, tile32 is split into 2 groups,
+    // a subtile where elem&1 is true and one where its false
+    auto subtile = cg::binary_partition(tile32, (elem & 1));
+}
+
+
+
+
+
+

+11.6. Group Collectives +

+

Cooperative Groups library provides a set of collective operations that can be performed by a group of threads. +These operations require participation of all threads in the specified group in order to complete the operation. +All threads in the group need to pass the same values for corresponding arguments to each collective call, unless +different values are explicitly allowed in the argument description. Otherwise the behavior of the call is undefined.

+
+

+11.6.1. Synchronization +

+
+

+11.6.1.1. barrier_arrive and barrier_wait +

+
+
T::arrival_token T::barrier_arrive();
+void T::barrier_wait(T::arrival_token&&);
+
+
+

barrier_arrive and barrier_wait member functions provide a synchronization API similar to cuda::barrier (read more). Cooperative Groups automatically initializes the group barrier, but arrive and wait operations have an additional restriction resulting from collective nature of those operations: All threads in the group must arrive and wait at the barrier once per phase. +When barrier_arrive is called with a group, result of calling any collective operation or another barrier arrival with that group is undefined until completion of the barrier phase is observed with barrier_wait call. Threads blocked on barrier_wait might be released from the synchronization before other threads call barrier_wait, but only after all threads in the group called barrier_arrive. +Group type T can be any of the implicit groups.This allows threads to do independent work after they arrive and before they wait for the synchronization to resolve, allowing to hide some of the synchronization latency. +barrier_arrive returns an arrival_token object that must be passed into the corresponding barrier_wait. Token is consumed this way and can not be used for another barrier_wait call.

+

Example of barrier_arrive and barrier_wait used to synchronize initalization of shared memory across the cluster:

+
+
#include <cooperative_groups.h>
+
+using namespace cooperative_groups;
+
+void __device__ init_shared_data(const thread_block& block, int *data);
+void __device__ local_processing(const thread_block& block);
+void __device__ process_shared_data(const thread_block& block, int *data);
+
+__global__ void cluster_kernel() {
+    extern __shared__ int array[];
+    auto cluster = this_cluster();
+    auto block   = this_thread_block();
+
+    // Use this thread block to initialize some shared state
+    init_shared_data(block, &array[0]);
+
+    auto token = cluster.barrier_arrive(); // Let other blocks know this block is running and data was initialized
+
+    // Do some local processing to hide the synchronization latency
+    local_processing(block);
+
+    // Map data in shared memory from the next block in the cluster
+    int *dsmem = cluster.map_shared_rank(&array[0], (cluster.block_rank() + 1) % cluster.num_blocks());
+
+    // Make sure all other blocks in the cluster are running and initialized shared data before accessing dsmem
+    cluster.barrier_wait(std::move(token));
+
+    // Consume data in distributed shared memory
+    process_shared_data(block, dsmem);
+    cluster.sync();
+}
+
+
+
+
+

+11.6.1.2. sync +

+
+
static void T::sync();
+
+template <typename T>
+void sync(T& group);
+
+
+

sync synchronizes the threads named in the group. Group type T can be any of the existing group types, as all of them support synchronization. Its available as a member function in every group type or as a free function taking a group as parameter. +If the group is a grid_group the kernel must have been launched using the appropriate cooperative launch APIs. Equivalent to T.barrier_wait(T.barrier_arrive()).

+
+
+
+

+11.6.2. Data Transfer +

+
+

+11.6.2.1. memcpy_async +

+

memcpy_async is a group-wide collective memcpy that utilizes hardware accelerated support for non-blocking memory transactions from global to shared memory. Given a set of threads named in the group, memcpy_async will move specified amount of bytes or elements of the input type through a single pipeline stage. Additionally for achieving best performance when using the memcpy_async API, an alignment of 16 bytes for both shared memory and global memory is required. It is important to note that while this is a memcpy in the general case, it is only asynchronous if the source is global memory and the destination is shared memory and both can be addressed with 16, 8, or 4 byte alignments. Asynchronously copied data should only be read following a call to wait or wait_prior which signals that the corresponding stage has completed moving data to shared memory.

+

Having to wait on all outstanding requests can lose some flexibility (but gain simplicity). In order to efficiently overlap data transfer and execution, its important to be able to kick off an N+1memcpy_async request while waiting on and operating on request N. To do so, use memcpy_async and wait on it using the collective stage-based wait_prior API. See wait and wait_prior for more details.

+

Usage 1

+
+
template <typename TyGroup, typename TyElem, typename TyShape>
+void memcpy_async(
+  const TyGroup &group,
+  TyElem *__restrict__ _dst,
+  const TyElem *__restrict__ _src,
+  const TyShape &shape
+);
+
+
+

Performs a copy of ``shape`` bytes.

+

Usage 2

+
+
template <typename TyGroup, typename TyElem, typename TyDstLayout, typename TySrcLayout>
+void memcpy_async(
+  const TyGroup &group,
+  TyElem *__restrict__ dst,
+  const TyDstLayout &dstLayout,
+  const TyElem *__restrict__ src,
+  const TySrcLayout &srcLayout
+);
+
+
+

Performs a copy of ``min(dstLayout, srcLayout)`` elements. If layouts are of type cuda::aligned_size_t<N>, both must specify the same alignment.

+

Errata +The memcpy_async API introduced in CUDA 11.1 with both src and dst input layouts, expects the layout to be provided in elements rather than bytes. The element type is inferred from TyElem and has the size sizeof(TyElem). If cuda::aligned_size_t<N> type is used as the layout, the number of elements specified times sizeof(TyElem) must be a multiple of N and it is recommended to use std::byte or char as the element type.

+

If specified shape or layout of the copy is of type cuda::aligned_size_t<N>, alignment will be guaranteed to be at least min(16, N). In that case both dst and src pointers need to be aligned to N bytes and the number of bytes copied needs to be a multiple of N.

+

Codegen Requirements: Compute Capability 5.0 minimum, Compute Capability 8.0 for asynchronicity, C++11

+

cooperative_groups/memcpy_async.h header needs to be included.

+

Example:

+
+
/// This example streams elementsPerThreadBlock worth of data from global memory
+/// into a limited sized shared memory (elementsInShared) block to operate on.
+#include <cooperative_groups.h>
+#include <cooperative_groups/memcpy_async.h>
+
+namespace cg = cooperative_groups;
+
+__global__ void kernel(int* global_data) {
+    cg::thread_block tb = cg::this_thread_block();
+    const size_t elementsPerThreadBlock = 16 * 1024;
+    const size_t elementsInShared = 128;
+    __shared__ int local_smem[elementsInShared];
+
+    size_t copy_count;
+    size_t index = 0;
+    while (index < elementsPerThreadBlock) {
+        cg::memcpy_async(tb, local_smem, elementsInShared, global_data + index, elementsPerThreadBlock - index);
+        copy_count = min(elementsInShared, elementsPerThreadBlock - index);
+        cg::wait(tb);
+        // Work with local_smem
+        index += copy_count;
+    }
+}
+
+
+
+
+

+11.6.2.2. wait and wait_prior +

+
+
template <typename TyGroup>
+void wait(TyGroup & group);
+
+template <unsigned int NumStages, typename TyGroup>
+void wait_prior(TyGroup & group);
+
+
+

wait and wait_prior collectives allow to wait for memcpy_async copies to complete. wait blocks calling threads until all previous copies are done. wait_prior allows that the latest NumStages are still not done and waits for all the previous requests. So with N total copies requested, it waits until the first N-NumStages are done and the last NumStages might still be in progress. Both wait and wait_prior will synchronize the named group.

+

Codegen Requirements: Compute Capability 5.0 minimum, Compute Capability 8.0 for asynchronicity, C++11

+

cooperative_groups/memcpy_async.h header needs to be included.

+

Example:

+
+
/// This example streams elementsPerThreadBlock worth of data from global memory
+/// into a limited sized shared memory (elementsInShared) block to operate on in
+/// multiple (two) stages. As stage N is kicked off, we can wait on and operate on stage N-1.
+#include <cooperative_groups.h>
+#include <cooperative_groups/memcpy_async.h>
+
+namespace cg = cooperative_groups;
+
+__global__ void kernel(int* global_data) {
+    cg::thread_block tb = cg::this_thread_block();
+    const size_t elementsPerThreadBlock = 16 * 1024 + 64;
+    const size_t elementsInShared = 128;
+    __align__(16) __shared__ int local_smem[2][elementsInShared];
+    int stage = 0;
+    // First kick off an extra request
+    size_t copy_count = elementsInShared;
+    size_t index = copy_count;
+    cg::memcpy_async(tb, local_smem[stage], elementsInShared, global_data, elementsPerThreadBlock - index);
+    while (index < elementsPerThreadBlock) {
+        // Now we kick off the next request...
+        cg::memcpy_async(tb, local_smem[stage ^ 1], elementsInShared, global_data + index, elementsPerThreadBlock - index);
+        // ... but we wait on the one before it
+        cg::wait_prior<1>(tb);
+
+        // Its now available and we can work with local_smem[stage] here
+        // (...)
+        //
+
+        // Calculate the amount fo data that was actually copied, for the next iteration.
+        copy_count = min(elementsInShared, elementsPerThreadBlock - index);
+        index += copy_count;
+
+        // A cg::sync(tb) might be needed here depending on whether
+        // the work done with local_smem[stage] can release threads to race ahead or not
+        // Wrap to the next stage
+        stage ^= 1;
+    }
+    cg::wait(tb);
+    // The last local_smem[stage] can be handled here
+}
+
+
+
+
+
+

+11.6.3. Data Manipulation +

+
+

+11.6.3.1. reduce +

+
+
template <typename TyGroup, typename TyArg, typename TyOp>
+auto reduce(const TyGroup& group, TyArg&& val, TyOp&& op) -> decltype(op(val, val));
+
+
+

reduce performs a reduction operation on the data provided by each thread named in the group passed in. This takes advantage of hardware acceleration (on compute 80 and higher devices) for the arithmetic add, min, or max operations and the logical AND, OR, or XOR, as well as providing a software fallback on older generation hardware. Only 4B types are accelerated by hardware.

+

group: Valid group types are coalesced_group and thread_block_tile.

+

val: Any type that satisfies the below requirements:

+
    +
  • Qualifies as trivially copyable i.e. is_trivially_copyable<TyArg>::value == true

  • +
  • sizeof(T) <= 32 for coalesced_group and tiles of size lower or equal 32, sizeof(T) <= 8 for larger tiles

  • +
  • Has suitable arithmetic or comparative operators for the given function object.

  • +
+

Note: Different threads in the group can pass different values for this argument.

+

op: Valid function objects that will provide hardware acceleration with integral types are plus(), less(), greater(), bit_and(), bit_xor(), bit_or(). These must be constructed, hence the TyVal template argument is required, i.e. plus<int>(). Reduce also supports lambdas and other function objects that can be invoked using operator()

+

Asynchronous reduce

+
+
template <typename TyGroup, typename TyArg, typename TyAtomic, typename TyOp>
+void reduce_update_async(const TyGroup& group, TyAtomic& atomic, TyArg&& val, TyOp&& op);
+
+template <typename TyGroup, typename TyArg, typename TyAtomic, typename TyOp>
+void reduce_store_async(const TyGroup& group, TyAtomic& atomic, TyArg&& val, TyOp&& op);
+
+template <typename TyGroup, typename TyArg, typename TyOp>
+void reduce_store_async(const TyGroup& group, TyArg* ptr, TyArg&& val, TyOp&& op);
+
+
+

*_async variants of the API are asynchronously calculating the result to either store to or update a specified destination by one of the participating threads, instead of returning it by each thread. To observe the effect of these asynchronous calls, calling group of threads or a larger group containing them need to be synchronized.

+
    +
  • In case of the atomic store or update variant, atomic argument can be either of cuda::atomic or cuda::atomic_ref available in CUDA C++ Standard Library. This variant of the API is available only on platforms and devices, where these types are supported by the CUDA C++ Standard Library. Result of the reduction is used to atomically update the atomic according to the specified op, eg. the result is atomically added to the atomic in case of cg::plus(). Type held by the atomic must match the type of TyArg. Scope of the atomic must include all the threads in the group and if multiple groups are using the same atomic concurrently, scope must include all threads in all groups using it. Atomic update is performed with relaxed memory ordering.

  • +
  • In case of the pointer store variant, result of the reduction will be weakly stored into the dst pointer.

  • +
+

Codegen Requirements: Compute Capability 5.0 minimum, Compute Capability 8.0 for HW acceleration, C++11.

+

cooperative_groups/reduce.h header needs to be included.

+

Example of approximate standard deviation for integer vector:

+
+
#include <cooperative_groups.h>
+#include <cooperative_groups/reduce.h>
+namespace cg = cooperative_groups;
+
+/// Calculate approximate standard deviation of integers in vec
+__device__ int std_dev(const cg::thread_block_tile<32>& tile, int *vec, int length) {
+    int thread_sum = 0;
+
+    // calculate average first
+    for (int i = tile.thread_rank(); i < length; i += tile.num_threads()) {
+        thread_sum += vec[i];
+    }
+    // cg::plus<int> allows cg::reduce() to know it can use hardware acceleration for addition
+    int avg = cg::reduce(tile, thread_sum, cg::plus<int>()) / length;
+
+    int thread_diffs_sum = 0;
+    for (int i = tile.thread_rank(); i < length; i += tile.num_threads()) {
+        int diff = vec[i] - avg;
+        thread_diffs_sum += diff * diff;
+    }
+
+    // temporarily use floats to calculate the square root
+    float diff_sum = static_cast<float>(cg::reduce(tile, thread_diffs_sum, cg::plus<int>())) / length;
+
+    return static_cast<int>(sqrtf(diff_sum));
+}
+
+
+

Example of block wide reduction:

+
+
#include <cooperative_groups.h>
+#include <cooperative_groups/reduce.h>
+namespace cg=cooperative_groups;
+
+/// The following example accepts input in *A and outputs a result into *sum
+/// It spreads the data equally within the block
+__device__ void block_reduce(const int* A, int count, cuda::atomic<int, cuda::thread_scope_block>& total_sum) {
+    auto block = cg::this_thread_block();
+    auto tile = cg::tiled_partition<32>(block);
+    int thread_sum = 0;
+
+    // Stride loop over all values, each thread accumulates its part of the array.
+    for (int i = block.thread_rank(); i < count; i += block.size()) {
+        thread_sum += A[i];
+    }
+
+    // reduce thread sums across the tile, add the result to the atomic
+    // cg::plus<int> allows cg::reduce() to know it can use hardware acceleration for addition
+ cg::reduce_update_async(tile, total_sum, thread_sum, cg::plus<int>());
+
+ // synchronize the block, to ensure all async reductions are ready
+    block.sync();
+}
+
+
+
+
+

+11.6.3.2. Reduce Operators +

+

Below are the prototypes of function objects for some of the basic operations that can be done with reduce

+
+
namespace cooperative_groups {
+  template <typename Ty>
+  struct cg::plus;
+
+  template <typename Ty>
+  struct cg::less;
+
+  template <typename Ty>
+  struct cg::greater;
+
+  template <typename Ty>
+  struct cg::bit_and;
+
+  template <typename Ty>
+  struct cg::bit_xor;
+
+  template <typename Ty>
+  struct cg::bit_or;
+}
+
+
+

Reduce is limited to the information available to the implementation at compile time. Thus in order to make use of intrinsics introduced in CC 8.0, the cg:: namespace exposes several functional objects that mirror the hardware. These objects appear similar to those presented in the C++ STL, with the exception of less/greater. The reason for any difference from the STL is that these function objects are designed to actually mirror the operation of the hardware intrinsics.

+

Functional description:

+
    +
  • cg::plus: Accepts two values and returns the sum of both using operator+.

  • +
  • cg::less: Accepts two values and returns the lesser using operator<. This differs in that the lower value is returned rather than a Boolean.

  • +
  • cg::greater: Accepts two values and returns the greater using operator<. This differs in that the greater value is returned rather than a Boolean.

  • +
  • cg::bit_and: Accepts two values and returns the result of operator&.

  • +
  • cg::bit_xor: Accepts two values and returns the result of operator^.

  • +
  • cg::bit_or: Accepts two values and returns the result of operator|.

  • +
+

Example:

+
+
{
+    // cg::plus<int> is specialized within cg::reduce and calls __reduce_add_sync(...) on CC 8.0+
+    cg::reduce(tile, (int)val, cg::plus<int>());
+
+    // cg::plus<float> fails to match with an accelerator and instead performs a standard shuffle based reduction
+    cg::reduce(tile, (float)val, cg::plus<float>());
+
+    // While individual components of a vector are supported, reduce will not use hardware intrinsics for the following
+    // It will also be necessary to define a corresponding operator for vector and any custom types that may be used
+    int4 vec = {...};
+    cg::reduce(tile, vec, cg::plus<int4>())
+
+    // Finally lambdas and other function objects cannot be inspected for dispatch
+    // and will instead perform shuffle based reductions using the provided function object.
+    cg::reduce(tile, (int)val, [](int l, int r) -> int {return l + r;});
+}
+
+
+
+
+

+11.6.3.3. inclusive_scan and exclusive_scan +

+
+
template <typename TyGroup, typename TyVal, typename TyFn>
+auto inclusive_scan(const TyGroup& group, TyVal&& val, TyFn&& op) -> decltype(op(val, val));
+
+template <typename TyGroup, typename TyVal>
+TyVal inclusive_scan(const TyGroup& group, TyVal&& val);
+
+template <typename TyGroup, typename TyVal, typename TyFn>
+auto exclusive_scan(const TyGroup& group, TyVal&& val, TyFn&& op) -> decltype(op(val, val));
+
+template <typename TyGroup, typename TyVal>
+TyVal exclusive_scan(const TyGroup& group, TyVal&& val);
+
+
+

inclusive_scan and exclusive_scan performs a scan operation on the data provided by each thread named in the group passed in. Result for each thread is a reduction of data from threads with lower thread_rank than that thread in case of exclusive_scan. inclusive_scan result also includes the calling thread data in the reduction.

+

group: Valid group types are coalesced_group and thread_block_tile.

+

val: Any type that satisfies the below requirements:

+
    +
  • Qualifies as trivially copyable i.e. is_trivially_copyable<TyArg>::value == true

  • +
  • sizeof(T) <= 32 for coalesced_group and tiles of size lower or equal 32, sizeof(T) <= 8 for larger tiles

  • +
  • Has suitable arithmetic or comparative operators for the given function object.

  • +
+

Note: Different threads in the group can pass different values for this argument.

+

op: Function objects defined for convenience are plus(), less(), greater(), bit_and(), bit_xor(), bit_or() described in Reduce Operators. These must be constructed, hence the TyVal template argument is required, i.e. plus<int>(). inclusive_scan and exclusive_scan also supports lambdas and other function objects that can be invoked using operator(). Overloads without this argument use cg::plus<TyVal>().

+

Scan update

+
+
template <typename TyGroup, typename TyAtomic, typename TyVal, typename TyFn>
+auto inclusive_scan_update(const TyGroup& group, TyAtomic& atomic, TyVal&& val, TyFn&& op) -> decltype(op(val, val));
+
+template <typename TyGroup, typename TyAtomic, typename TyVal>
+TyVal inclusive_scan_update(const TyGroup& group, TyAtomic& atomic, TyVal&& val);
+
+template <typename TyGroup, typename TyAtomic, typename TyVal, typename TyFn>
+auto exclusive_scan_update(const TyGroup& group, TyAtomic& atomic, TyVal&& val, TyFn&& op) -> decltype(op(val, val));
+
+template <typename TyGroup, typename TyAtomic, typename TyVal>
+TyVal exclusive_scan_update(const TyGroup& group, TyAtomic& atomic, TyVal&& val);
+
+
+

*_scan_update collectives take an additional argument atomic that can be either of cuda::atomic or cuda::atomic_ref available in CUDA C++ Standard Library. These variants of the API are available only on platforms and devices, where these types are supported by the CUDA C++ Standard Library. These variants will perform an update to the atomic according to op with value of the sum of input values of all threads in the group. Previous value of the atomic will be combined with the result of scan by each thread and returned. Type held by the atomic must match the type of TyVal. Scope of the atomic must include all the threads in the group and if multiple groups are using the same atomic concurrently, scope must include all threads in all groups using it. Atomic update is performed with relaxed memory ordering.

+

Following pseudocode illustrates how the update variant of scan works:

+
+
/*
+ inclusive_scan_update behaves as the following block,
+ except both reduce and inclusive_scan is calculated simultaneously.
+auto total = reduce(group, val, op);
+TyVal old;
+if (group.thread_rank() == selected_thread) {
+    atomically {
+        old = atomic.load();
+        atomic.store(op(old, total));
+    }
+}
+old = group.shfl(old, selected_thread);
+return op(inclusive_scan(group, val, op), old);
+*/
+
+
+

Codegen Requirements: Compute Capability 5.0 minimum, C++11.

+

cooperative_groups/scan.h header needs to be included.

+

Example:

+
+
#include <stdio.h>
+#include <cooperative_groups.h>
+#include <cooperative_groups/scan.h>
+namespace cg = cooperative_groups;
+
+__global__ void kernel() {
+    auto thread_block = cg::this_thread_block();
+    auto tile = cg::tiled_partition<8>(thread_block);
+    unsigned int val = cg::inclusive_scan(tile, tile.thread_rank());
+    printf("%u: %u\n", tile.thread_rank(), val);
+}
+
+/*  prints for each group:
+    0: 0
+    1: 1
+    2: 3
+    3: 6
+    4: 10
+    5: 15
+    6: 21
+    7: 28
+*/
+
+
+

Example of stream compaction using exclusive_scan:

+
+
#include <cooperative_groups.h>
+#include <cooperative_groups/scan.h>
+namespace cg = cooperative_groups;
+
+// put data from input into output only if it passes test_fn predicate
+template<typename Group, typename Data, typename TyFn>
+__device__ int stream_compaction(Group &g, Data *input, int count, TyFn&& test_fn, Data *output) {
+    int per_thread = count / g.num_threads();
+    int thread_start = min(g.thread_rank() * per_thread, count);
+    int my_count = min(per_thread, count - thread_start);
+
+    // get all passing items from my part of the input
+    //  into a contagious part of the array and count them.
+    int i = thread_start;
+    while (i < my_count + thread_start) {
+        if (test_fn(input[i])) {
+            i++;
+        }
+        else {
+            my_count--;
+            input[i] = input[my_count + thread_start];
+        }
+    }
+
+    // scan over counts from each thread to calculate my starting
+    //  index in the output
+    int my_idx = cg::exclusive_scan(g, my_count);
+
+    for (i = 0; i < my_count; ++i) {
+        output[my_idx + i] = input[thread_start + i];
+    }
+    // return the total number of items in the output
+    return g.shfl(my_idx + my_count, g.num_threads() - 1);
+}
+
+
+

Example of dynamic buffer space allocation using exclusive_scan_update:

+
+
#include <cooperative_groups.h>
+#include <cooperative_groups/scan.h>
+namespace cg = cooperative_groups;
+
+// Buffer partitioning is static to make the example easier to follow,
+// but any arbitrary dynamic allocation scheme can be implemented by replacing this function.
+__device__ int calculate_buffer_space_needed(cg::thread_block_tile<32>& tile) {
+    return tile.thread_rank() % 2 + 1;
+}
+
+__device__ int my_thread_data(int i) {
+    return i;
+}
+
+__global__ void kernel() {
+    __shared__ extern int buffer[];
+    __shared__ cuda::atomic<int, cuda::thread_scope_block> buffer_used;
+
+    auto block = cg::this_thread_block();
+    auto tile = cg::tiled_partition<32>(block);
+    buffer_used = 0;
+    block.sync();
+
+    // each thread calculates buffer size it needs
+    int buf_needed = calculate_buffer_space_needed(tile);
+
+    // scan over the needs of each thread, result for each thread is an offset
+    // of that thread’s part of the buffer. buffer_used is atomically updated with
+    // the sum of all thread's inputs, to correctly offset other tile’s allocations
+    int buf_offset =
+        cg::exclusive_scan_update(tile, buffer_used, buf_needed);
+
+    // each thread fills its own part of the buffer with thread specific data
+    for (int i = 0 ; i < buf_needed ; ++i) {
+        buffer[buf_offset + i] = my_thread_data(i);
+    }
+
+    block.sync();
+    // buffer_used now holds total amount of memory allocated
+    // buffer is {0, 0, 1, 0, 0, 1 ...};
+}
+
+
+
+
+
+

+11.6.4. Execution control +

+
+

+11.6.4.1. invoke_one and invoke_one_broadcast +

+
+
template<typename Group, typename Fn, typename... Args>
+void invoke_one(const Group& group, Fn&& fn, Args&&... args);
+
+template<typename Group, typename Fn, typename... Args>
+auto invoke_one_broadcast(const Group& group, Fn&& fn, Args&&... args) -> decltype(fn(args...));
+
+
+

invoke_one selects a single arbitrary thread from the calling group and uses that thread to call the supplied invocable fn with the supplied arguments args. +In case of invoke_one_broadcast the result of the call is also distributed to all threads in the group and returned from this collective.

+

Calling group can be synchronized with the selected thread before and/or after it calls the supplied invocable. It means that communication within the calling group +is not allowed inside the supplied invocable body, otherwise forward progress is not guaranteed. Communication with threads outside of the calling group is allowed in the +body of the supplied invocable. Thread selection mechanism is not guaranteed to be deterministic.

+

On devices with Compute Capability 9.0 or higher hardware acceleration might be used to select the thread when called with explicit group types.

+

group: All group types are valid for invoke_one, coalesced_group and thread_block_tile are valid for invoke_one_broadcast.

+

fn: Function or object that can be invoked using operator().

+

args: Parameter pack of types matching types of parameters of the supplied invocable fn.

+

In case of invoke_one_broadcast the return type of the supplied invocable fn must satisfy the below requirements:

+
    +
  • Qualifies as trivially copyable i.e. is_trivially_copyable<T>::value == true

  • +
  • sizeof(T) <= 32 for coalesced_group and tiles of size lower or equal 32, sizeof(T) <= 8 for larger tiles

  • +
+

Codegen Requirements: Compute Capability 5.0 minimum, Compute Capability 9.0 for hardware acceleration, C++11.

+

Aggregated atomic example from Discovery pattern section re-written to use invoke_one_broadcast:

+
+
#include <cooperative_groups.h>
+#include <cuda/atomic>
+namespace cg = cooperative_groups;
+
+template<cuda::thread_scope Scope>
+__device__ unsigned int atomicAddOneRelaxed(cuda::atomic<unsigned int, Scope>& atomic) {
+    auto g = cg::coalesced_threads();
+    auto prev = cg::invoke_one_broadcast(g, [&] () {
+        return atomic.fetch_add(g.num_threads(), cuda::memory_order_relaxed);
+    });
+    return prev + g.thread_rank();
+}
+
+
+
+
+
+
+

+11.7. Grid Synchronization +

+

Prior to the introduction of Cooperative Groups, the CUDA programming model only allowed synchronization between thread blocks at a kernel completion boundary. The kernel boundary carries with it an implicit invalidation of state, and with it, potential performance implications.

+

For example, in certain use cases, applications have a large number of small kernels, with each kernel representing a stage in a processing pipeline. The presence of these kernels is required by the current CUDA programming model to ensure that the thread blocks operating on one pipeline stage have produced data before the thread block operating on the next pipeline stage is ready to consume it. In such cases, the ability to provide global inter thread block synchronization would allow the application to be restructured to have persistent thread blocks, which are able to synchronize on the device when a given stage is complete.

+

To synchronize across the grid, from within a kernel, you would simply use the grid.sync() function:

+
+
grid_group grid = this_grid();
+grid.sync();
+
+
+

And when launching the kernel it is necessary to use, instead of the <<<...>>> execution configuration syntax, the cudaLaunchCooperativeKernel CUDA runtime launch API or the CUDA driver equivalent.

+

Example:

+

To guarantee co-residency of the thread blocks on the GPU, the number of blocks launched needs to be carefully considered. For example, as many blocks as there are SMs can be launched as follows:

+
+
int dev = 0;
+cudaDeviceProp deviceProp;
+cudaGetDeviceProperties(&deviceProp, dev);
+// initialize, then launch
+cudaLaunchCooperativeKernel((void*)my_kernel, deviceProp.multiProcessorCount, numThreads, args);
+
+
+

Alternatively, you can maximize the exposed parallelism by calculating how many blocks can fit simultaneously per-SM using the occupancy calculator as follows:

+
+
/// This will launch a grid that can maximally fill the GPU, on the default stream with kernel arguments
+int numBlocksPerSm = 0;
+ // Number of threads my_kernel will be launched with
+int numThreads = 128;
+cudaDeviceProp deviceProp;
+cudaGetDeviceProperties(&deviceProp, dev);
+cudaOccupancyMaxActiveBlocksPerMultiprocessor(&numBlocksPerSm, my_kernel, numThreads, 0);
+// launch
+void *kernelArgs[] = { /* add kernel args */ };
+dim3 dimBlock(numThreads, 1, 1);
+dim3 dimGrid(deviceProp.multiProcessorCount*numBlocksPerSm, 1, 1);
+cudaLaunchCooperativeKernel((void*)my_kernel, dimGrid, dimBlock, kernelArgs);
+
+
+

It is good practice to first ensure the device supports cooperative launches by querying the device attribute cudaDevAttrCooperativeLaunch:

+
+
int dev = 0;
+int supportsCoopLaunch = 0;
+cudaDeviceGetAttribute(&supportsCoopLaunch, cudaDevAttrCooperativeLaunch, dev);
+
+
+

which will set supportsCoopLaunch to 1 if the property is supported on device 0. Only devices with compute capability of 6.0 and higher are supported. In addition, you need to be running on either of these:

+
    +
  • The Linux platform without MPS

  • +
  • The Linux platform with MPS and on a device with compute capability 7.0 or higher

  • +
  • The latest Windows platform

  • +
+
+
+
+

+12. Cluster Launch Control +

+
+

Warning

+

This document has been replaced by a new CUDA Programming Guide. The information in this document should be considered legacy, and this document is no longer being updated as of CUDA 13.0. Please refer to the CUDA Programming Guide for up-to-date information on CUDA.

+
+
+

+12.1. Introduction +

+

Compute Capability 10.0 introduces Cluster Launch Control, a new +feature that provides developer with more control over thread block scheduling +by cancelling thread blocks or thread block clusters.

+

When dealing with problems of variable size, there are two main +approaches to determining the number of kernel thread blocks.

+

Approach 1: Fixed Work per Thread Block:

+

In this approach, the number of thread blocks is determined by the problem +size, while the amount of work done by each thread block remains constant +or is limited.

+

Key advantages of this approach:

+
    +
  • +

    Load balancing between SMs.

    +

    In particular, when thread block run-times exhibit variability +and/or when the number of thread blocks is much larger than what +the GPU can execute simultaneously (resulting in a low-tail effect), +this approach allows the GPU scheduler to run more thread blocks +on some SMs than others.

    +
  • +
  • +

    Preemption.

    +

    The GPU scheduler can start executing a +higher-priority kernel, +even if it is launched after the execution of a lower-priority kernel has +already begun, by scheduling the higher-priority kernel’s thread blocks +as the currently running lower-priority kernel’s thread blocks finish. +It can then return to the lower-priority kernel once the higher-priority +kernel has finished.

    +
  • +
+

Approach 2: Fixed Number of Thread Blocks:

+

In this approach, often implemented as a block-stride or grid-stride loop, +the number of thread blocks does not directly depend +on the problem size. Instead, the amount of work done by each thread block +is a function of the problem size. Typically, the number of thread blocks is +based on the number of SMs on the GPU where the kernel is executed +and the desired occupancy.

+

Key advantage of this approach:

+
    +
  • +

    Reduced thread block overheads.

    +

    This approach not only reduces amortized thread block launch latency +but also minimizes the computational overhead associated with shared +operations across all thread blocks. +These overheads can be significantly higher than launch latency overheads.

    +

    For example, in convolution kernels, a prologue for calculating +convolution coefficients – independent of the thread block index – +can be computed fewer times due to the fixed number of thread blocks, +thus reducing redundant computations.

    +
  • +
+

Cluster Launch Control Approach:

+

Cluster Launch Control allows a kernel to request (cancel) +the thread block index of a block that has not yet started execution.

+

This mechanism enables work-stealing among thread blocks: +a thread block attempts to cancel the launch of another thread block +that has not started running yet. If cancellation succeeds, +it “steals” the other thread block’s work by using cancelled block index +to perform the task.

+

The cancellation will fail if there are no more thread block indices +available and may fail for other reasons, such as a higher-priority kernel +being scheduled. In the latter case, if a thread block exits after a +cancellation failure, the scheduler can start executing the higher-priority +kernel, after which it will continue scheduling the remaining thread blocks +of the current kernel for execution.

+

The table below summarizes advantages and disadvantages of the three approaches:

+ ++++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + +

+Fixed Work per Thread Block

+Fixed Number of Thread Blocks

+Cluster Launch Control

Reduced overheads

+\(\textcolor{red}{\textbf{X}}\)

+\(\textcolor{lime}{\textbf{V}}\)

+\(\textcolor{lime}{\textbf{V}}\)

Preemption

+\(\textcolor{lime}{\textbf{V}}\)

+\(\textcolor{red}{\textbf{X}}\)

+\(\textcolor{lime}{\textbf{V}}\)

Load balancing

+\(\textcolor{lime}{\textbf{V}}\)

+\(\textcolor{red}{\textbf{X}}\)

+\(\textcolor{lime}{\textbf{V}}\)

+
+
+

+12.2. Cluster Launch Control API Details +

+

Cancelling a thread block via the Cluster Launch Control API is done +asynchronously and synchronized using a memory barrier, +following a programming pattern similar +to asynchronous data copies.

+

The API, currently available through +libcu++, +provides a request instruction that writes the encoded cancellation result +into a __shared__ variable, along with instructions to decode the result +into a Success/Fail flag and the index of the cancelled thread block +in case of Success.

+
+

+12.2.1. Thread block cancellation steps +

+

The preferred way to use Cluster Launch Control is from a single thread, +i.e., one request at a time.

+

The following are the five steps of the thread block cancellation process. +The first two steps are declarations and initialization of cancellation result +and synchronization variables, which are done before the work-stealing. +The last three steps are typically executed inside a work-stealing loop +over thread block indices.

+
    +
  1. +

    Declare variables for thread block cancellation:

    +
    +
    __shared__ uint4 result; // Request result.
    +__shared__ uint64_t bar; // Synchronization barrier.
    +int phase = 0;           // Synchronization barrier phase.
    +
    +
    +
  2. +
  3. +

    Initialize shared memory barrier with a single arrival count:

    +
    +
    if (cg::thread_block::thread_rank() == 0)
    +    ptx::mbarrier_init(&bar, 1);
    +__syncthreads();
    +
    +
    +
  4. +
  5. +

    Submit asynchronous cancellation request by a single thread and +set transaction count:

    +
    +
    if (cg::thread_block::thread_rank() == 0) {
    +    cg::invoke_one(cg::coalesced_threads(), ptx::clusterlaunchcontrol_try_cancel, &result, &bar);
    +    ptx::mbarrier_arrive_expect_tx(ptx::sem_relaxed, ptx::scope_cta, ptx::space_shared, &bar, sizeof(uint4));
    +}
    +
    +
    +
    +

    Note

    +

    Since thread block cancellation is a uniform instruction, +it is recommended to submit it inside +invoke_one thread selector. +This allows the compliler to optimize out the peeling loop.

    +
    +
  6. +
  7. +

    Synchronize (complete) asynchronous cancellation request:

    +
    +
    while (!ptx::mbarrier_try_wait_parity(&bar, phase))
    +{}
    +phase ^= 1;
    +
    +
    +
  8. +
  9. +

    Retrieve cancellation status and cancelled thread block index:

    +
    +
    bool success = ptx::clusterlaunchcontrol_query_cancel_is_canceled(result);
    +if (success) {
    +    // Don't need all three for 1D/2D thread blocks:
    +    int bx = ptx::clusterlaunchcontrol_query_cancel_get_first_ctaid_x(result);
    +    int by = ptx::clusterlaunchcontrol_query_cancel_get_first_ctaid_y(result);
    +    int bz = ptx::clusterlaunchcontrol_query_cancel_get_first_ctaid_z(result);
    +}
    +
    +
    +
  10. +
  11. Ensure visibility of shared memory operations between async and generic +proxies, +and protect against data races between iterations of the work-stealing loop.

  12. +
+
+
+

+12.2.2. Thread block cancellation constraints +

+

The constraints are related to failed cancellation requests:

+
    +
  • +

    Submitting another cancellation request after observing a +previously failed request is Undefined Behavior.

    +

    In the two code examples below, assuming the first cancellation request +fails, only the first example exhibits undefined behavior. +The second example is correct because there is no observation between the +cancellation requests:

    +

    Invalid code:

    +
    +
    // First request:
    +ptx::clusterlaunchcontrol_try_cancel(&result0, &bar0);
    +
    +// First request query:
    +[Synchronize bar0 code here.]
    +bool success0 = ptx::clusterlaunchcontrol_query_cancel_is_canceled(result0);
    +assert(!success0); // Observed failure; second cacellation will be invalid.
    +
    +// Second request - next line is Undefined Behavior:
    +ptx::clusterlaunchcontrol_try_cancel(&result1, &bar1);
    +
    +
    +

    Valid code:

    +
    +
    // First request:
    +ptx::clusterlaunchcontrol_try_cancel(&result0, &bar0);
    +
    +// Second request:
    +ptx::clusterlaunchcontrol_try_cancel(&result1, &bar1);
    +
    +// First request query:
    +[Synchronize bar0 code here.]
    +bool success0 = ptx::clusterlaunchcontrol_query_cancel_is_canceled(result0);
    +assert(!success0); // Observed failure; second cacellation was valid.
    +
    +
    +
  • +
  • Retrieving the thread block index of a failed cancellation request +is Undefined Behavior.

  • +
  • +

    Submitting a cancellation request from multiple threads is not recommended. +It results in the cancellation of multiple thread blocks +and requires careful handling, +such as:

    +
      +
    • Each submitting thread must proivde a unique __shared__ result +pointer to avoid data races.

    • +
    • If the same barrier is used for synchronization, the arrival and +transaction counts must be adjusted accordingly.

    • +
    +
  • +
+
+
+

+12.2.3. Kernel Example: Vector-Scalar Multiplication +

+

The three kernels below demonstrate the Fixed Work per Thread Block, +Fixed Number of Thread Blocks, and Cluster Launch Control approaches +for vector-scalar multiplication +\(\overline{v} := \alpha \overline{v}\).

+
    +
  • +

    Fixed Work per Thread Block:

    +
    +
    __global__
    +void kernel_fixed_work (float* data, int n)
    +{
    +    // Prologue:
    +    float alpha = compute_scalar();
    +
    +    // Computation:
    +    int i = blockIdx.x * blockDim.x + threadIdx.x;
    +    if (i < n)
    +        data[i] *= alpha;
    +}
    +
    +// Launch: kernel_fixed_work<<<1024, (n + 1023) / 1024>>>(data, n);
    +
    +
    +
  • +
  • +

    Fixed Number of Thread Blocks:

    +
    +
    __global__
    +void kernel_fixed_blocks (float* data, int n)
    +{
    +    // Prologue:
    +    float alpha = compute_scalar();
    +
    +    // Computation:
    +    int i = blockIdx.x * blockDim.x + threadIdx.x;
    +    while (i < n) {
    +        data[i] *= alpha;
    +        i += gridDim.x * blockDim.x;
    +    }
    +}
    +
    +// Launch: kernel_fixed_blocks<<<1024, SM_COUNT>>>(data, n);
    +
    +
    +
  • +
  • +

    Cluster Launch Control:

    +
    +
    #include <cooperative_groups.h>
    +#include <cuda/ptx>
    +
    +namespace cg = cooperative_groups;
    +namespace ptx = cuda::ptx;
    +
    +__global__
    +void kernel_cluster_launch_control (float* data, int n)
    +{
    +    // Cluster launch control initialization:
    +    __shared__ uint4 result;
    +    __shared__ uint64_t bar;
    +    int phase = 0;
    +
    +    if (cg::thread_block::thread_rank() == 0)
    +        ptx::mbarrier_init(&bar, 1);
    +
    +    // Prologue:
    +    float alpha = compute_scalar(); // Device function not shown in this code snippet.
    +
    +    // Work-stealing loop:
    +    int bx = blockIdx.x; // Assuming 1D x-axis thread blocks.
    +
    +    while (true) {
    +        // Protect result from overwrite in the next iteration,
    +        // (also ensure barrier initialization at 1st iteration):
    +        __syncthreads();
    +
    +        // Cancellation request:
    +        if (cg::thread_block::thread_rank() == 0) {
    +            // Acquire write of result in the async proxy:
    +            ptx::fence_proxy_async_generic_sync_restrict(ptx::sem_acquire, ptx::space_cluster, ptx::scope_cluster);
    +
    +            cg::invoke_one(cg::coalesced_threads(), [&](){ptx::clusterlaunchcontrol_try_cancel(&result, &bar);});
    +            ptx::mbarrier_arrive_expect_tx(ptx::sem_relaxed, ptx::scope_cta, ptx::space_shared, &bar, sizeof(uint4));
    +        }
    +
    +        // Computation:
    +        int i = bx * blockDim.x + threadIdx.x;
    +        if (i < n)
    +            data[i] *= alpha;
    +
    +        // Cancellation request synchronization:
    +        while (!ptx::mbarrier_try_wait_parity(ptx::sem_acquire, ptx::scope_cta, &bar, phase))
    +        {}
    +        phase ^= 1;
    +
    +        // Cancellation request decoding:
    +        bool success = ptx::clusterlaunchcontrol_query_cancel_is_canceled(result);
    +        if (!success)
    +            break;
    +
    +        bx = ptx::clusterlaunchcontrol_query_cancel_get_first_ctaid_x<int>(result);
    +
    +        // Release read of result to the async proxy:
    +        ptx::fence_proxy_async_generic_sync_restrict(ptx::sem_release, ptx::space_shared, ptx::scope_cluster);
    +    }
    +}
    +
    +// Launch: kernel_cluster_launch_control<<<1024, (n + 1023) / 1024>>>(data, n);
    +
    +
    +
  • +
+
+
+

+12.2.4. Cluster Launch Control for Thread Block Clusters +

+

In the case of a thread block clusters, +the thread block cancellation steps are the same as in a non-cluster setting, +with minor adjustments. +As in the non-cluster case, submitting a cancellation request from multiple +threads within a cluster is not recommended, +as this will attempt to cancel multiple clusters.

+
    +
  • The cancellation is submitted by a single cluster thread.

  • +
  • The shared memory result of each cluster’s thread block will receive +the same (encoded) value of the cancelled thread block index +(i.e., the result value is multicasted). +The result received by all thread blocks corresponds to the local +block index {0, 0, 0} within a cluster. +Therefore, thread blocks within the cluster need to add +the local block index.

  • +
  • Synchronization is performed by each cluster’s thread block using +a local __shared__ memory barrier. Barrier operations must be performed +with the ptx::scope_cluster scope.

  • +
  • Cancelling in the cluster case requires all the thread blocks to exist. +A user can guarantee that all thread blocks are running by using +cg::cluster_group::sync() from Cluster Group API.

  • +
+

The kernel below demonstrates Cluster Launch Control approach +in a thread block cluster case:

+
+
#include <cooperative_groups.h>
+#include <cuda/ptx>
+
+namespace cg = cooperative_groups;
+namespace ptx = cuda::ptx;
+
+__global__ __cluster_dims__(2, 1, 1)
+void kernel_cluster_launch_control (float* data, int n)
+{
+    // Cluster launch control initialization:
+    __shared__ uint4 result;
+    __shared__ uint64_t bar;
+    int phase = 0;
+
+    if (cg::thread_block::thread_rank() == 0) {
+        ptx::mbarrier_init(&bar, 1);
+        ptx::fence_mbarrier_init(ptx::sem_release, ptx::scope_cluster); // CGA-level fence.
+    }
+
+    // Prologue:
+    float alpha = compute_scalar(); // Device function not shown in this code snippet.
+
+    // Work-stealing loop:
+    int bx = blockIdx.x; // Assuming 1D x-axis thread blocks.
+
+    while (true) {
+        // Protect result from overwrite in the next iteration,
+        // (also ensure all thread blocks have started at 1st iteration):
+        cg::cluster_group::sync();
+
+        // Cancellation request by a single cluster thread:
+        if (cg::cluster_group::thread_rank() == 0) {
+            // Acquire write of result in the async proxy:
+            ptx::fence_proxy_async_generic_sync_restrict(ptx::sem_acquire, ptx::space_cluster, ptx::scope_cluster);
+
+            cg::invoke_one(cg::coalesced_threads(), [&](){ptx::clusterlaunchcontrol_try_cancel_multicast(&result, &bar);});
+        }
+
+        // Cancellation completion tracked by each thread block:
+        if (cg::thread_block::thread_rank() == 0)
+            ptx::mbarrier_arrive_expect_tx(ptx::sem_relaxed, ptx::scope_cluster, ptx::space_shared, &bar, sizeof(uint4));
+
+        // Computation:
+        int i = bx * blockDim.x + threadIdx.x;
+        if (i < n)
+            data[i] *= alpha;
+
+        // Cancellation request synchronization:
+        while (!ptx::mbarrier_try_wait_parity(ptx::sem_acquire, ptx::scope_cluster, &bar, phase))
+        {}
+        phase ^= 1;
+
+        // Cancellation request decoding:
+        bool success = ptx::clusterlaunchcontrol_query_cancel_is_canceled(result);
+        if (!success)
+            break;
+
+        bx = ptx::clusterlaunchcontrol_query_cancel_get_first_ctaid_x<int>(result);
+        bx += cg::cluster_group::block_index().x; // Add local offset.
+
+        // Release read of result to the async proxy:
+        ptx::fence_proxy_async_generic_sync_restrict(ptx::sem_release, ptx::space_shared, ptx::scope_cluster);
+    }
+}
+
+// Launch: kernel_cluster_launch_control<<<1024, (n + 1023) / 1024>>>(data, n);
+
+
+
+
+
+
+

+13. CUDA Dynamic Parallelism +

+
+

Warning

+

This document has been replaced by a new CUDA Programming Guide. The information in this document should be considered legacy, and this document is no longer being updated as of CUDA 13.0. Please refer to the CUDA Programming Guide for up-to-date information on CUDA.

+
+
+

+13.1. Introduction +

+
+

+13.1.1. Overview +

+

Dynamic Parallelism is an extension to the CUDA programming model enabling a CUDA kernel to create and synchronize with new work directly on the GPU. The creation of parallelism dynamically at whichever point in a program that it is needed offers exciting capabilities.

+

The ability to create work directly from the GPU can reduce the need to transfer execution control and data between host and device, as launch configuration decisions can now be made at runtime by threads executing on the device. Additionally, data-dependent parallel work can be generated inline within a kernel at run-time, taking advantage of the GPU’s hardware schedulers and load balancers dynamically and adapting in response to data-driven decisions or workloads. Algorithms and programming patterns that had previously required modifications to eliminate recursion, irregular loop structure, or other constructs that do not fit a flat, single-level of parallelism may more transparently be expressed.

+

This document describes the extended capabilities of CUDA which enable Dynamic Parallelism, including the modifications and additions to the CUDA programming model necessary to take advantage of these, as well as guidelines and best practices for exploiting this added capacity.

+

Dynamic Parallelism is only supported by devices of compute capability 3.5 and higher.

+
+
+

+13.1.2. Glossary +

+

Definitions for terms used in this guide.

+
+
Grid
+
+

A Grid is a collection of Threads. Threads in a Grid execute a Kernel Function and are divided into Thread Blocks.

+
+
Thread Block
+
+

A Thread Block is a group of threads which execute on the same multiprocessor (SM). Threads within a Thread Block have access to shared memory and can be explicitly synchronized.

+
+
Kernel Function
+
+

A Kernel Function is an implicitly parallel subroutine that executes under the CUDA execution and memory model for every Thread in a Grid.

+
+
Host
+
+

The Host refers to the execution environment that initially invoked CUDA. Typically the thread running on a system’s CPU processor.

+
+
Parent
+
+

A Parent Thread, Thread Block, or Grid is one that has launched new grid(s), the Child Grid(s). The Parent is not considered completed until all of its launched Child Grids have also completed.

+
+
Child
+
+

A Child thread, block, or grid is one that has been launched by a Parent grid. A Child grid must complete before the Parent Thread, Thread Block, or Grid are considered complete.

+
+
Thread Block Scope
+
+

Objects with Thread Block Scope have the lifetime of a single Thread Block. They only have defined behavior when operated on by Threads in the Thread Block that created the object and are destroyed when the Thread Block that created them is complete.

+
+
Device Runtime
+
+

The Device Runtime refers to the runtime system and APIs available to enable Kernel Functions to use Dynamic Parallelism.

+
+
+
+
+
+

+13.2. Execution Environment and Memory Model +

+
+

+13.2.1. Execution Environment +

+

The CUDA execution model is based on primitives of threads, thread blocks, and grids, with kernel functions defining the program executed by individual threads within a thread block and grid. When a kernel function is invoked the grid’s properties are described by an execution configuration, which has a special syntax in CUDA. Support for dynamic parallelism in CUDA extends the ability to configure, launch, and implicitly synchronize upon new grids to threads that are running on the device.

+
+

+13.2.1.1. Parent and Child Grids +

+

A device thread that configures and launches a new grid belongs to the parent grid, and the grid created by the invocation is a child grid.

+

The invocation and completion of child grids is properly nested, meaning that the parent grid is not considered complete until all child grids created by its threads have completed, and the runtime guarantees an implicit synchronization between the parent and child.

+
+Parent-Child Launch Nesting +
+

Figure 30 Parent-Child Launch Nesting

+
+
+
+
+

+13.2.1.2. Scope of CUDA Primitives +

+

On both host and device, the CUDA runtime offers an API for launching kernels and for tracking dependencies between launches via streams and events. On the host system, the state of launches and the CUDA primitives referencing streams and events are shared by all threads within a process; however processes execute independently and may not share CUDA objects.

+

On the device, launched kernels and CUDA objects are visible to all threads in a grid. This means, for example, that a stream may be created by one thread and used by any other thread in the grid.

+
+
+

+13.2.1.3. Synchronization +

+
+

Warning

+

Explicit synchronization with child kernels from a parent block (i.e. using cudaDeviceSynchronize() in device code) is deprecated in CUDA 11.6 and removed for compute_90+ compilation. For compute capability < 9.0, compile-time opt-in by specifying -DCUDA_FORCE_CDP1_IF_SUPPORTED is required to continue using cudaDeviceSynchronize() in device code. Note that this is slated for full removal in a future CUDA release.

+
+

CUDA runtime operations from any thread, including kernel launches, are visible across all the threads in a grid. This means that an invoking thread in the parent grid may perform synchronization to control the launch order of grids launched by any thread in the grid on streams created by any thread in the grid. Execution of a grid is not considered complete until all launches by all threads in the grid have completed. If all threads in a grid exit before all child launches have completed, an implicit synchronization operation will automatically be triggered.

+
+
+

+13.2.1.4. Streams and Events +

+

CUDA Streams and Events allow control over dependencies between grid launches: grids launched into the same stream execute in-order, and events may be used to create dependencies between streams. Streams and events created on the device serve this exact same purpose.

+

Streams and events created within a grid exist within grid scope, but have undefined behavior when used outside of the grid where they were created. As described above, all work launched by a grid is implicitly synchronized when the grid exits; work launched into streams is included in this, with all dependencies resolved appropriately. The behavior of operations on a stream that has been modified outside of grid scope is undefined.

+

Streams and events created on the host have undefined behavior when used within any kernel, just as streams and events created by a parent grid have undefined behavior if used within a child grid.

+
+
+

+13.2.1.5. Ordering and Concurrency +

+

The ordering of kernel launches from the device runtime follows CUDA Stream ordering semantics. Within a grid, all kernel launches into the same stream (with the exception of the fire-and-forget stream discussed later) are executed in-order. With multiple threads in the same grid launching into the same stream, the ordering within the stream is dependent on the thread scheduling within the grid, which may be controlled with synchronization primitives such as __syncthreads().

+

Note that while named streams are shared by all threads within a grid, the implicit NULL stream is only shared by all threads within a thread block. If multiple threads in a thread block launch into the implicit stream, then these launches will be executed in-order. If multiple threads in different thread blocks launch into the implicit stream, then these launches may be executed concurrently. If concurrency is desired for launches by multiple threads within a thread block, explicit named streams should be used.

+

Dynamic Parallelism enables concurrency to be expressed more easily within a program; however, the device runtime introduces no new concurrency guarantees within the CUDA execution model. There is no guarantee of concurrent execution between any number of different thread blocks on a device.

+

The lack of concurrency guarantee extends to a parent grid and their child grids. When a parent grid launches a child grid, the child may start to execute once stream dependencies are satisfied and hardware resources are available to host the child, but is not guaranteed to begin execution until the parent grid reaches an implicit synchronization point.

+

While concurrency will often easily be achieved, it may vary as a function of device configuration, application workload, and runtime scheduling. It is therefore unsafe to depend upon any concurrency between different thread blocks.

+
+
+

+13.2.1.6. Device Management +

+

There is no multi-GPU support from the device runtime; the device runtime is only capable of operating on the device upon which it is currently executing. It is permitted, however, to query properties for any CUDA capable device in the system.

+
+
+
+

+13.2.2. Memory Model +

+

Parent and child grids share the same global and constant memory storage, but have distinct local and shared memory.

+
+

+13.2.2.1. Coherence and Consistency +

+
+
+13.2.2.1.1. Global Memory +
+

Parent and child grids have coherent access to global memory, with weak consistency guarantees between child and parent. There is only one point of time in the execution of a child grid when its view of memory is fully consistent with the parent thread: at the point when the child grid is invoked by the parent.

+

All global memory operations in the parent thread prior to the child grid’s invocation are visible to the child grid. With the removal of cudaDeviceSynchronize(), it is no longer possible to access the modifications made by the threads in the child grid from the parent grid. The only way to access the modifications made by the threads in the child grid before the parent grid exits is via a kernel launched into the cudaStreamTailLaunch stream.

+

In the following example, the child grid executing child_launch is only guaranteed to see the modifications to data made before the child grid was launched. Since thread 0 of the parent is performing the launch, the child will be consistent with the memory seen by thread 0 of the parent. Due to the first __syncthreads() call, the child will see data[0]=0, data[1]=1, …, data[255]=255 (without the __syncthreads() call, only data[0]=0 would be guaranteed to be seen by the child). The child grid is only guaranteed to return at an implicit synchronization. This means that the modifications made by the threads in the child grid are never guaranteed to become available to the parent grid. To access modifications made by child_launch, a tail_launch kernel is launched into the cudaStreamTailLaunch stream.

+
+
__global__ void tail_launch(int *data) {
+   data[threadIdx.x] = data[threadIdx.x]+1;
+}
+
+__global__ void child_launch(int *data) {
+   data[threadIdx.x] = data[threadIdx.x]+1;
+}
+
+__global__ void parent_launch(int *data) {
+   data[threadIdx.x] = threadIdx.x;
+
+   __syncthreads();
+
+   if (threadIdx.x == 0) {
+       child_launch<<< 1, 256 >>>(data);
+       tail_launch<<< 1, 256, 0, cudaStreamTailLaunch >>>(data);
+   }
+}
+
+void host_launch(int *data) {
+    parent_launch<<< 1, 256 >>>(data);
+}
+
+
+
+
+
+13.2.2.1.2. Zero Copy Memory +
+

Zero-copy system memory has identical coherence and consistency guarantees to global memory, and follows the semantics detailed above. A kernel may not allocate or free zero-copy memory, but may use pointers to zero-copy passed in from the host program.

+
+
+
+13.2.2.1.3. Constant Memory +
+

Constants may not be modified from the device. They may only be modified from the host, but the behavior of modifying a constant from the host while there is a concurrent grid that access that constant at any point during its lifetime is undefined.

+
+
+
+13.2.2.1.4. Shared and Local Memory +
+

Shared and Local memory is private to a thread block or thread, respectively, and is not visible or coherent between parent and child. Behavior is undefined when an object in one of these locations is referenced outside of the scope within which it belongs, and may cause an error.

+

The NVIDIA compiler will attempt to warn if it can detect that a pointer to local or shared memory is being passed as an argument to a kernel launch. At runtime, the programmer may use the __isGlobal() intrinsic to determine whether a pointer references global memory and so may safely be passed to a child launch.

+

Note that calls to cudaMemcpy*Async() or cudaMemset*Async() may invoke new child kernels on the device in order to preserve stream semantics. As such, passing shared or local memory pointers to these APIs is illegal and will return an error.

+
+
+
+13.2.2.1.5. Local Memory +
+

Local memory is private storage for an executing thread, and is not visible outside of that thread. It is illegal to pass a pointer to local memory as a launch argument when launching a child kernel. The result of dereferencing such a local memory pointer from a child will be undefined.

+

For example the following is illegal, with undefined behavior if x_array is accessed by child_launch:

+
+
int x_array[10];       // Creates x_array in parent's local memory
+child_launch<<< 1, 1 >>>(x_array);
+
+
+

It is sometimes difficult for a programmer to be aware of when a variable is placed into local memory by the compiler. As a general rule, all storage passed to a child kernel should be allocated explicitly from the global-memory heap, either with cudaMalloc(), new() or by declaring __device__ storage at global scope. For example:

+
+
// Correct - "value" is global storage
+__device__ int value;
+__device__ void x() {
+    value = 5;
+    child<<< 1, 1 >>>(&value);
+}
+
+
+
+
// Invalid - "value" is local storage
+__device__ void y() {
+    int value = 5;
+    child<<< 1, 1 >>>(&value);
+}
+
+
+
+
+
+13.2.2.1.6. Texture Memory +
+

Writes to the global memory region over which a texture is mapped are incoherent with respect to texture accesses. Coherence for texture memory is enforced at the invocation of a child grid and when a child grid completes. This means that writes to memory prior to a child kernel launch are reflected in texture memory accesses of the child. Similarly to Global Memory above, writes to memory by a child are never guaranteed to be reflected in the texture memory accesses by a parent. The only way to access the modifications made by the threads in the child grid before the parent grid exits is via a kernel launched into the cudaStreamTailLaunch stream. Concurrent accesses by parent and child may result in inconsistent data.

+
+
+
+
+
+

+13.3. Programming Interface +

+
+

+13.3.1. CUDA C++ Reference +

+

This section describes changes and additions to the CUDA C++ language extensions for supporting Dynamic Parallelism.

+

The language interface and API available to CUDA kernels using CUDA C++ for Dynamic Parallelism, referred to as the Device Runtime, is substantially like that of the CUDA Runtime API available on the host. Where possible the syntax and semantics of the CUDA Runtime API have been retained in order to facilitate ease of code reuse for routines that may run in either the host or device environments.

+

As with all code in CUDA C++, the APIs and code outlined here is per-thread code. This enables each thread to make unique, dynamic decisions regarding what kernel or operation to execute next. There are no synchronization requirements between threads within a block to execute any of the provided device runtime APIs, which enables the device runtime API functions to be called in arbitrarily divergent kernel code without deadlock.

+
+

+13.3.1.1. Device-Side Kernel Launch +

+

Kernels may be launched from the device using the standard CUDA <<< >>> syntax:

+
+
kernel_name<<< Dg, Db, Ns, S >>>([kernel arguments]);
+
+
+
    +
  • Dg is of type dim3 and specifies the dimensions and size of the grid

  • +
  • Db is of type dim3 and specifies the dimensions and size of each thread block

  • +
  • Ns is of type size_t and specifies the number of bytes of shared memory that is dynamically allocated per thread block for this call in addition to statically allocated memory. Ns is an optional argument that defaults to 0.

  • +
  • S is of type cudaStream_t and specifies the stream associated with this call. The stream must have been allocated in the same grid where the call is being made. S is an optional argument that defaults to the NULL stream.

  • +
+
+
+13.3.1.1.1. Launches are Asynchronous +
+

Identical to host-side launches, all device-side kernel launches are asynchronous with respect to the launching thread. That is to say, the <<<>>> launch command will return immediately and the launching thread will continue to execute until it hits an implicit launch-synchronization point (such as at a kernel launched into the cudaStreamTailLaunch stream).

+

The child grid launch is posted to the device and will execute independently of the parent thread. The child grid may begin execution at any time after launch, but is not guaranteed to begin execution until the launching thread reaches an implicit launch-synchronization point.

+
+
+
+13.3.1.1.2. Launch Environment Configuration +
+

All global device configuration settings (for example, shared memory and L1 cache size as returned from cudaDeviceGetCacheConfig(), and device limits returned from cudaDeviceGetLimit()) will be inherited from the parent. Likewise, device limits such as stack size will remain as-configured.

+

For host-launched kernels, per-kernel configurations set from the host will take precedence over the global setting. These configurations will be used when the kernel is launched from the device as well. It is not possible to reconfigure a kernel’s environment from the device.

+
+
+
+

+13.3.1.2. Streams +

+

Both named and unnamed (NULL) streams are available from the device runtime. Named streams may be used by any thread within a grid, but stream handles may not be passed to other child/parent kernels. In other words, a stream should be treated as private to the grid in which it is created.

+

Similar to host-side launch, work launched into separate streams may run concurrently, but actual concurrency is not guaranteed. Programs that depend upon concurrency between child kernels are not supported by the CUDA programming model and will have undefined behavior.

+

The host-side NULL stream’s cross-stream barrier semantic is not supported on the device (see below for details). In order to retain semantic compatibility with the host runtime, all device streams must be created using the cudaStreamCreateWithFlags() API, passing the cudaStreamNonBlocking flag. The cudaStreamCreate() call is a host-runtime- only API and will fail to compile for the device.

+

As cudaStreamSynchronize() and cudaStreamQuery() are unsupported by the device runtime, a kernel launched into the cudaStreamTailLaunch stream should be used instead when the application needs to know that stream-launched child kernels have completed.

+
+
+13.3.1.2.1. The Implicit (NULL) Stream +
+

Within a host program, the unnamed (NULL) stream has additional barrier synchronization semantics with other streams (see Default Stream for details). The device runtime offers a single implicit, unnamed stream shared between all threads in a thread block, but as all named streams must be created with the cudaStreamNonBlocking flag, work launched into the NULL stream will not insert an implicit dependency on pending work in any other streams (including NULL streams of other thread blocks).

+
+
+
+13.3.1.2.2. The Fire-and-Forget Stream +
+

The fire-and-forget named stream (cudaStreamFireAndForget) allows the user to launch fire-and-forget work with less boilerplate and without stream tracking overhead. It is functionally identical to, but faster than, creating a new stream per launch, and launching into that stream.

+

Fire-and-forget launches are immediately scheduled for launch without any dependency on the completion of previously launched grids. No other grid launches can depend on the completion of a fire-and-forget launch, except through the implicit synchronization at the end of the parent grid. So a tail launch or the next grid in parent grid’s stream won’t launch before a parent grid’s fire-and-forget work has completed.

+
+
// In this example, C2's launch will not wait for C1's completion
+__global__ void P( ... ) {
+   C1<<< ... , cudaStreamFireAndForget >>>( ... );
+   C2<<< ... , cudaStreamFireAndForget >>>( ... );
+}
+
+
+

The fire-and-forget stream cannot be used to record or wait on events. Attempting to do so results in cudaErrorInvalidValue. The fire-and-forget stream is not supported when compiled with CUDA_FORCE_CDP1_IF_SUPPORTED defined. Fire-and-forget stream usage requires compilation to be in 64-bit mode.

+
+
+
+13.3.1.2.3. The Tail Launch Stream +
+

The tail launch named stream (cudaStreamTailLaunch) allows a grid to schedule a new grid for launch after its completion. It should be possible to to use a tail launch to achieve the same functionality as a cudaDeviceSynchronize() in most cases.

+

Each grid has its own tail launch stream. All non-tail launch work launched by a grid is implicitly synchronized before the tail stream is kicked off. I.e. A parent grid’s tail launch does not launch until the parent grid and all work launched by the parent grid to ordinary streams or per-thread or fire-and-forget streams have completed. If two grids are launched to the same grid’s tail launch stream, the later grid does not launch until the earlier grid and all its descendent work has completed.

+
+
// In this example, C2 will only launch after C1 completes.
+__global__ void P( ... ) {
+   C1<<< ... , cudaStreamTailLaunch >>>( ... );
+   C2<<< ... , cudaStreamTailLaunch >>>( ... );
+}
+
+
+

Grids launched into the tail launch stream will not launch until the completion of all work by the parent grid, including all other grids (and their descendants) launched by the parent in all non-tail launched streams, including work executed or launched after the tail launch.

+
+
// In this example, C will only launch after all X, F and P complete.
+__global__ void P( ... ) {
+   C<<< ... , cudaStreamTailLaunch >>>( ... );
+   X<<< ... , cudaStreamPerThread >>>( ... );
+   F<<< ... , cudaStreamFireAndForget >>>( ... )
+}
+
+
+

The next grid in the parent grid’s stream will not be launched before a parent grid’s tail launch work has completed. In other words, the tail launch stream behaves as if it were inserted between its parent grid and the next grid in its parent grid’s stream.

+
+
// In this example, P2 will only launch after C completes.
+__global__ void P1( ... ) {
+   C<<< ... , cudaStreamTailLaunch >>>( ... );
+}
+
+__global__ void P2( ... ) {
+}
+
+int main ( ... ) {
+   ...
+   P1<<< ... >>>( ... );
+   P2<<< ... >>>( ... );
+   ...
+}
+
+
+

Each grid only gets one tail launch stream. To tail launch concurrent grids, it can be done like the example below.

+
+
// In this example,  C1 and C2 will launch concurrently after P's completion
+__global__ void T( ... ) {
+   C1<<< ... , cudaStreamFireAndForget >>>( ... );
+   C2<<< ... , cudaStreamFireAndForget >>>( ... );
+}
+
+__global__ void P( ... ) {
+   ...
+   T<<< ... , cudaStreamTailLaunch >>>( ... );
+}
+
+
+

The tail launch stream cannot be used to record or wait on events. Attempting to do so results in cudaErrorInvalidValue. The tail launch stream is not supported when compiled with CUDA_FORCE_CDP1_IF_SUPPORTED defined. Tail launch stream usage requires compilation to be in 64-bit mode.

+
+
+
+

+13.3.1.3. Events +

+

Only the inter-stream synchronization capabilities of CUDA events are supported. This means that cudaStreamWaitEvent() is supported, but cudaEventSynchronize(), cudaEventElapsedTime(), and cudaEventQuery() are not. As cudaEventElapsedTime() is not supported, cudaEvents must be created via cudaEventCreateWithFlags(), passing the cudaEventDisableTiming flag.

+

As with named streams, event objects may be shared between all threads within the grid which created them but are local to that grid and may not be passed to other kernels. Event handles are not guaranteed to be unique between grids, so using an event handle within a grid that did not create it will result in undefined behavior.

+
+
+

+13.3.1.4. Synchronization +

+

It is up to the program to perform sufficient inter-thread synchronization, for example via a CUDA Event, if the calling thread is intended to synchronize with child grids invoked from other threads.

+

As it is not possible to explicitly synchronize child work from a parent thread, there is no way to guarantee that changes occurring in child grids are visible to threads within the parent grid.

+
+
+

+13.3.1.5. Device Management +

+

Only the device on which a kernel is running will be controllable from that kernel. This means that device APIs such as cudaSetDevice() are not supported by the device runtime. The active device as seen from the GPU (returned from cudaGetDevice()) will have the same device number as seen from the host system. The cudaDeviceGetAttribute() call may request information about another device as this API allows specification of a device ID as a parameter of the call. Note that the catch-all cudaGetDeviceProperties() API is not offered by the device runtime - properties must be queried individually.

+
+
+

+13.3.1.6. Memory Declarations +

+
+
+13.3.1.6.1. Device and Constant Memory +
+

Memory declared at file scope with __device__ or __constant__ memory space specifiers behaves identically when using the device runtime. All kernels may read or write device variables, whether the kernel was initially launched by the host or device runtime. Equivalently, all kernels will have the same view of __constant__s as declared at the module scope.

+
+
+
+13.3.1.6.2. Textures and Surfaces +
+

CUDA supports dynamically created texture and surface objects7, where a texture object may be created on the host, passed to a kernel, used by that kernel, and then destroyed from the host. The device runtime does not allow creation or destruction of texture or surface objects from within device code, but texture and surface objects created from the host may be used and passed around freely on the device. Regardless of where they are created, dynamically created texture objects are always valid and may be passed to child kernels from a parent.

+
+

Note

+

The device runtime does not support legacy module-scope (i.e., Fermi-style) textures and surfaces within a kernel launched from the device. Module-scope (legacy) textures may be created from the host and used in device code as for any kernel, but may only be used by a top-level kernel (i.e., the one which is launched from the host).

+
+
+
+
+13.3.1.6.3. Shared Memory Variable Declarations +
+

In CUDA C++ shared memory can be declared either as a statically sized file-scope or function-scoped variable, or as an extern variable with the size determined at runtime by the kernel’s caller via a launch configuration argument. Both types of declarations are valid under the device runtime.

+
+
__global__ void permute(int n, int *data) {
+   extern __shared__ int smem[];
+   if (n <= 1)
+       return;
+
+   smem[threadIdx.x] = data[threadIdx.x];
+   __syncthreads();
+
+   permute_data(smem, n);
+   __syncthreads();
+
+   // Write back to GMEM since we can't pass SMEM to children.
+   data[threadIdx.x] = smem[threadIdx.x];
+   __syncthreads();
+
+   if (threadIdx.x == 0) {
+       permute<<< 1, 256, n/2*sizeof(int) >>>(n/2, data);
+       permute<<< 1, 256, n/2*sizeof(int) >>>(n/2, data+n/2);
+   }
+}
+
+void host_launch(int *data) {
+    permute<<< 1, 256, 256*sizeof(int) >>>(256, data);
+}
+
+
+
+
+
+13.3.1.6.4. Symbol Addresses +
+

Device-side symbols (i.e., those marked __device__) may be referenced from within a kernel simply via the & operator, as all global-scope device variables are in the kernel’s visible address space. This also applies to __constant__ symbols, although in this case the pointer will reference read-only data.

+

Given that device-side symbols can be referenced directly, those CUDA runtime APIs which reference symbols (e.g., cudaMemcpyToSymbol() or cudaGetSymbolAddress()) are redundant and hence not supported by the device runtime. Note this implies that constant data cannot be altered from within a running kernel, even ahead of a child kernel launch, as references to __constant__ space are read-only.

+
+
+
+

+13.3.1.7. API Errors and Launch Failures +

+

As usual for the CUDA runtime, any function may return an error code. The last error code returned is recorded and may be retrieved via the cudaGetLastError() call. Errors are recorded per-thread, so that each thread can identify the most recent error that it has generated. The error code is of type cudaError_t.

+

Similar to a host-side launch, device-side launches may fail for many reasons (invalid arguments, etc). The user must call cudaGetLastError() to determine if a launch generated an error, however lack of an error after launch does not imply the child kernel completed successfully.

+

For device-side exceptions, e.g., access to an invalid address, an error in a child grid will be returned to the host.

+
+
+13.3.1.7.1. Launch Setup APIs +
+

Kernel launch is a system-level mechanism exposed through the device runtime library, and as such is available directly from PTX via the underlying cudaGetParameterBuffer() and cudaLaunchDevice() APIs. It is permitted for a CUDA application to call these APIs itself, with the same requirements as for PTX. In both cases, the user is then responsible for correctly populating all necessary data structures in the correct format according to specification. Backwards compatibility is guaranteed in these data structures.

+

As with host-side launch, the device-side operator <<<>>> maps to underlying kernel launch APIs. This is so that users targeting PTX will be able to enact a launch, and so that the compiler front-end can translate <<<>>> into these calls.

+ + ++++ + + + + + + + + + + + + + + + + +
+Table 13 New Device-only Launch Implementation Functions +

Runtime API Launch Functions

Description of Difference From Host Runtime Behaviour (behavior is identical if no description)

cudaGetParameterBuffer

Generated automatically from <<<>>>. Note different API to host equivalent.

cudaLaunchDevice

Generated automatically from <<<>>>. Note different API to host equivalent.

+

The APIs for these launch functions are different to those of the CUDA Runtime API, and are defined as follows:

+
+
extern   device   cudaError_t cudaGetParameterBuffer(void **params);
+extern __device__ cudaError_t cudaLaunchDevice(void *kernel,
+                                        void *params, dim3 gridDim,
+                                        dim3 blockDim,
+                                        unsigned int sharedMemSize = 0,
+                                        cudaStream_t stream = 0);
+
+
+
+
+
+

+13.3.1.8. API Reference +

+

The portions of the CUDA Runtime API supported in the device runtime are detailed here. Host and device runtime APIs have identical syntax; semantics are the same except where indicated. The following table provides an overview of the API relative to the version available from the host.

+ + ++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+Table 14 Supported API Functions +

Runtime API Functions

Details

cudaDeviceGetCacheConfig

cudaDeviceGetLimit

cudaGetLastError

Last error is per-thread state, not per-block state

cudaPeekAtLastError

cudaGetErrorString

cudaGetDeviceCount

cudaDeviceGetAttribute

Will return attributes for any device

cudaGetDevice

Always returns current device ID as would be seen from host

cudaStreamCreateWithFlags

Must pass cudaStreamNonBlocking flag

cudaStreamDestroy

cudaStreamWaitEvent

cudaEventCreateWithFlags

Must pass cudaEventDisableTiming flag

cudaEventRecord

cudaEventDestroy

cudaFuncGetAttributes

cudaMemcpyAsync

+

Notes about all memcpy/memset functions:

+
    +
  • Only async memcpy/set functions are supported

  • +
  • Only device-to-device memcpy is permitted

  • +
  • May not pass in local or shared memory pointers

  • +
+

cudaMemcpy2DAsync

cudaMemcpy3DAsync

cudaMemsetAsync

cudaMemset2DAsync

cudaMemset3DAsync

cudaRuntimeGetVersion

cudaMalloc

May not call cudaFree on the device on a pointer created on the host, and vice-versa

cudaFree

cudaOccupancyMaxActiveBlocksPerMultiprocessor

cudaOccupancyMaxPotentialBlockSize

cudaOccupancyMaxPotentialBlockSizeVariableSMem

+
+
+
+

+13.3.2. Device-side Launch from PTX +

+

This section is for the programming language and compiler implementers who target Parallel Thread Execution (PTX) and plan to support Dynamic Parallelism in their language. It provides the low-level details related to supporting kernel launches at the PTX level.

+
+

+13.3.2.1. Kernel Launch APIs +

+

Device-side kernel launches can be implemented using the following two APIs accessible from PTX: cudaLaunchDevice() and cudaGetParameterBuffer(). cudaLaunchDevice() launches the specified kernel with the parameter buffer that is obtained by calling cudaGetParameterBuffer() and filled with the parameters to the launched kernel. The parameter buffer can be NULL, i.e., no need to invoke cudaGetParameterBuffer(), if the launched kernel does not take any parameters.

+
+
+13.3.2.1.1. cudaLaunchDevice +
+

At the PTX level, cudaLaunchDevice()needs to be declared in one of the two forms shown below before it is used.

+
+
// PTX-level Declaration of cudaLaunchDevice() when .address_size is 64
+.extern .func(.param .b32 func_retval0) cudaLaunchDevice
+(
+  .param .b64 func,
+  .param .b64 parameterBuffer,
+  .param .align 4 .b8 gridDimension[12],
+  .param .align 4 .b8 blockDimension[12],
+  .param .b32 sharedMemSize,
+  .param .b64 stream
+)
+;
+
+
+

The CUDA-level declaration below is mapped to one of the aforementioned PTX-level declarations and is found in the system header file cuda_device_runtime_api.h. The function is defined in the cudadevrt system library, which must be linked with a program in order to use device-side kernel launch functionality.

+
+
// CUDA-level declaration of cudaLaunchDevice()
+extern "C" __device__
+cudaError_t cudaLaunchDevice(void *func, void *parameterBuffer,
+                             dim3 gridDimension, dim3 blockDimension,
+                             unsigned int sharedMemSize,
+                             cudaStream_t stream);
+
+
+

The first parameter is a pointer to the kernel to be is launched, and the second parameter is the parameter buffer that holds the actual parameters to the launched kernel. The layout of the parameter buffer is explained in Parameter Buffer Layout, below. Other parameters specify the launch configuration, i.e., as grid dimension, block dimension, shared memory size, and the stream associated with the launch (please refer to Execution Configuration for the detailed description of launch configuration.

+
+
+
+13.3.2.1.2. cudaGetParameterBuffer +
+

cudaGetParameterBuffer() needs to be declared at the PTX level before it’s used. The PTX-level declaration must be in one of the two forms given below, depending on address size:

+
+
// PTX-level Declaration of cudaGetParameterBuffer() when .address_size is 64
+.extern .func(.param .b64 func_retval0) cudaGetParameterBuffer
+(
+  .param .b64 alignment,
+  .param .b64 size
+)
+;
+
+
+

The following CUDA-level declaration of cudaGetParameterBuffer() is mapped to the aforementioned PTX-level declaration:

+
+
// CUDA-level Declaration of cudaGetParameterBuffer()
+extern "C" __device__
+void *cudaGetParameterBuffer(size_t alignment, size_t size);
+
+
+

The first parameter specifies the alignment requirement of the parameter buffer and the second parameter the size requirement in bytes. In the current implementation, the parameter buffer returned by cudaGetParameterBuffer() is always guaranteed to be 64- byte aligned, and the alignment requirement parameter is ignored. However, it is recommended to pass the correct alignment requirement value - which is the largest alignment of any parameter to be placed in the parameter buffer - to cudaGetParameterBuffer() to ensure portability in the future.

+
+
+
+

+13.3.2.2. Parameter Buffer Layout +

+

Parameter reordering in the parameter buffer is prohibited, and each individual parameter placed in the parameter buffer is required to be aligned. That is, each parameter must be placed at the nth byte in the parameter buffer, where n is the smallest multiple of the parameter size that is greater than the offset of the last byte taken by the preceding parameter. The maximum size of the parameter buffer is 4KB.

+

For a more detailed description of PTX code generated by the CUDA compiler, please refer to the PTX-3.5 specification.

+
+
+
+

+13.3.3. Toolkit Support for Dynamic Parallelism +

+
+

+13.3.3.1. Including Device Runtime API in CUDA Code +

+

Similar to the host-side runtime API, prototypes for the CUDA device runtime API are included automatically during program compilation. There is no need to includecuda_device_runtime_api.h explicitly.

+
+
+

+13.3.3.2. Compiling and Linking +

+

When compiling and linking CUDA programs using dynamic parallelism with nvcc, the program will automatically link against the static device runtime library libcudadevrt.

+

The device runtime is offered as a static library (cudadevrt.lib on Windows, libcudadevrt.a under Linux), against which a GPU application that uses the device runtime must be linked. Linking of device libraries can be accomplished through nvcc and/or nvlink. Two simple examples are shown below.

+

A device runtime program may be compiled and linked in a single step, if all required source files can be specified from the command line:

+
+
$ nvcc -arch=sm_75 -rdc=true hello_world.cu -o hello -lcudadevrt
+
+
+

It is also possible to compile CUDA .cu source files first to object files, and then link these together in a two-stage process:

+
+
$ nvcc -arch=sm_75 -dc hello_world.cu -o hello_world.o
+$ nvcc -arch=sm_75 -rdc=true hello_world.o -o hello -lcudadevrt
+
+
+

Please see the Using Separate Compilation section of The CUDA Driver Compiler NVCC guide for more details.

+
+
+
+
+

+13.4. Programming Guidelines +

+
+

+13.4.1. Basics +

+

The device runtime is a functional subset of the host runtime. API level device management, kernel launching, device memcpy, stream management, and event management are exposed from the device runtime.

+

Programming for the device runtime should be familiar to someone who already has experience with CUDA. Device runtime syntax and semantics are largely the same as that of the host API, with any exceptions detailed earlier in this document.

+

The following example shows a simple Hello World program incorporating dynamic parallelism:

+
+
#include <stdio.h>
+
+__global__ void childKernel()
+{
+    printf("Hello ");
+}
+
+__global__ void tailKernel()
+{
+    printf("World!\n");
+}
+
+__global__ void parentKernel()
+{
+    // launch child
+    childKernel<<<1,1>>>();
+    if (cudaSuccess != cudaGetLastError()) {
+        return;
+    }
+
+    // launch tail into cudaStreamTailLaunch stream
+    // implicitly synchronizes: waits for child to complete
+    tailKernel<<<1,1,0,cudaStreamTailLaunch>>>();
+
+}
+
+int main(int argc, char *argv[])
+{
+    // launch parent
+    parentKernel<<<1,1>>>();
+    if (cudaSuccess != cudaGetLastError()) {
+        return 1;
+    }
+
+    // wait for parent to complete
+    if (cudaSuccess != cudaDeviceSynchronize()) {
+        return 2;
+    }
+
+    return 0;
+}
+
+
+

This program may be built in a single step from the command line as follows:

+
+
$ nvcc -arch=sm_75 -rdc=true hello_world.cu -o hello -lcudadevrt
+
+
+
+
+

+13.4.2. Performance +

+
+

+13.4.2.1. Dynamic-parallelism-enabled Kernel Overhead +

+

System software which is active when controlling dynamic launches may impose an overhead on any kernel which is running at the time, whether or not it invokes kernel launches of its own. This overhead arises from the device runtime’s execution tracking and management software and may result in decreased performance. This overhead is, in general, incurred for applications that link against the device runtime library.

+
+
+
+

+13.4.3. Implementation Restrictions and Limitations +

+

Dynamic Parallelism guarantees all semantics described in this document, however, certain hardware and software resources are implementation-dependent and limit the scale, performance and other properties of a program which uses the device runtime.

+
+

+13.4.3.1. Runtime +

+
+
+13.4.3.1.1. Memory Footprint +
+

The device runtime system software reserves memory for various management purposes, in particular a reservation for tracking pending grid launches. Configuration controls are available to reduce the size of this reservation in exchange for certain launch limitations. See Configuration Options, below, for details.

+
+
+
+13.4.3.1.2. Pending Kernel Launches +
+

When a kernel is launched, all associated configuration and parameter data is tracked until the kernel completes. This data is stored within a system-managed launch pool.

+

The size of the fixed-size launch pool is configurable by calling cudaDeviceSetLimit() from the host and specifying cudaLimitDevRuntimePendingLaunchCount.

+
+
+
+13.4.3.1.3. Configuration Options +
+

Resource allocation for the device runtime system software is controlled via the cudaDeviceSetLimit() API from the host program. Limits must be set before any kernel is launched, and may not be changed while the GPU is actively running programs.

+

The following named limits may be set:

+ ++++ + + + + + + + + + + + + + + + + +

Limit

Behavior

cudaLimitDevRuntimePendingLaunchCount

Controls the amount of memory set aside for buffering kernel launches and events which have not yet begun to execute, due either to unresolved dependencies or lack of execution resources. When the buffer is full, an attempt to allocate a launch slot during a device side kernel launch will fail and return cudaErrorLaunchOutOfResources, while an attempt to allocate an event slot will fail and return cudaErrorMemoryAllocation. The default number of launch slots is 2048. Applications may increase the number of launch and/or event slots by setting cudaLimitDevRuntimePendingLaunchCount. The number of event slots allocated is twice the value of that limit.

cudaLimitStackSize

Controls the stack size in bytes of each GPU thread. The CUDA driver automatically increases the per-thread stack size for each kernel launch as needed. This size isn’t reset back to the original value after each launch. To set the per-thread stack size to a different value, cudaDeviceSetLimit() can be called to set this limit. The stack will be immediately resized, and if necessary, the device will block until all preceding requested tasks are complete. cudaDeviceGetLimit() can be called to get the current per-thread stack size.

+
+
+
+13.4.3.1.4. Memory Allocation and Lifetime +
+

cudaMalloc() and cudaFree() have distinct semantics between the host and device environments. When invoked from the host, cudaMalloc() allocates a new region from unused device memory. When invoked from the device runtime these functions map to device-side malloc() and free(). This implies that within the device environment the total allocatable memory is limited to the device malloc() heap size, which may be smaller than the available unused device memory. Also, it is an error to invoke cudaFree() from the host program on a pointer which was allocated by cudaMalloc() on the device or vice-versa.

+ +++++ + + + + + + + + + + + + + + + + + + + + + + + + +

cudaMalloc() on Host

cudaMalloc() on Device

cudaFree() on Host

Supported

Not Supported

cudaFree() on Device

Not Supported

Supported

Allocation limit

Free device memory

cudaLimitMallocHeapSize

+
+
+
+13.4.3.1.5. SM Id and Warp Id +
+

Note that in PTX %smid and %warpid are defined as volatile values. The device runtime may reschedule thread blocks onto different SMs in order to more efficiently manage resources. As such, it is unsafe to rely upon %smid or %warpid remaining unchanged across the lifetime of a thread or thread block.

+
+
+
+13.4.3.1.6. ECC Errors +
+

No notification of ECC errors is available to code within a CUDA kernel. ECC errors are reported at the host side once the entire launch tree has completed. Any ECC errors which arise during execution of a nested program will either generate an exception or continue execution (depending upon error and configuration).

+
+
+
+
+
+

+13.5. CDP2 vs CDP1 +

+

This section summarises the differences between, and the compatibility and interoperability of, the new (CDP2) and legacy (CDP1) CUDA Dynamic Parallelism interfaces. It also shows how to opt-out of the CDP2 interface on devices of compute capability less than 9.0.

+
+

+13.5.1. Differences Between CDP1 and CDP2 +

+

Explicit device-side synchronization is no longer possible with CDP2 or on devices of compute capability 9.0 or higher. Implicit synchronization (such as tail launches) must be used instead.

+

Attempting to query or set cudaLimitDevRuntimeSyncDepth (or CU_LIMIT_DEV_RUNTIME_SYNC_DEPTH) with CDP2 or on devices of compute capability 9.0 or higher results in cudaErrorUnsupportedLimit.

+

CDP2 no longer has a virtualized pool for pending launches that don’t fit in the fixed-sized pool. cudaLimitDevRuntimePendingLaunchCount must be set to be large enough to avoid running out of launch slots.

+

For CDP2, there is a limit to the total number of events existing at once (note that events are destroyed only after a launch completes), equal to twice the pending launch count. cudaLimitDevRuntimePendingLaunchCount must be set to be large enough to avoid running out of event slots.

+

Streams are tracked per grid with CDP2 or on devices of compute capability 9.0 or higher, not per thread block. This allows work to be launched into a stream created by another thread block. Attempting to do so with the CDP1 results in cudaErrorInvalidValue.

+

CDP2 introduces the tail launch (cudaStreamTailLaunch) and fire-and-forget (cudaStreamFireAndForget) named streams.

+

CDP2 is supported only under 64-bit compilation mode.

+
+
+

+13.5.2. Compatibility and Interoperability +

+

CDP2 is the default. Functions can be compiled with -DCUDA_FORCE_CDP1_IF_SUPPORTED to opt-out of using CDP2 on devices of compute capability less than 9.0.

+ +++++ + + + + + + + + + + + + + + + + + + + + + + + + +

Function compiler with CUDA 12.0 and newer (default)

Function compiled with pre-CUDA 12.0 or with CUDA 12.0 and newer with -DCUDA_FORCE_CDP1_IF_SUPPORTED specified

Compilation

Compile error if device code references cudaDeviceSynchronize.

Compile error if code references cudaStreamTailLaunch or cudaStreamFireAndForget. Compile error if device code references cudaDeviceSynchronize and code is compiled for sm_90 or newer.

Compute capability < 9.0

New interface is used.

Legacy interface is used.

Compute capability 9.0 and higher

New interface is used.

New interface is used. If function references cudaDeviceSynchronize in device code, function load returns cudaErrorSymbolNotFound (this could happen if the code is compiled for devices of compute capability less than 9.0, but run on devices of compute capability 9.0 or higher using JIT).

+

Functions using CDP1 and CDP2 may be loaded and run simultaneously in the same context. The CDP1 functions are able to use CDP1-specific features (e.g. cudaDeviceSynchronize) and CDP2 functions are able to use CDP2-specific features (e.g. tail launch and fire-and-forget launch).

+

A function using CDP1 cannot launch a function using CDP2, and vice versa. If a function that would use CDP1 contains in its call graph a function that would use CDP2, or vice versa, cudaErrorCdpVersionMismatch would result during function load.

+
+
+
+

+13.6. Legacy CUDA Dynamic Parallelism (CDP1) +

+

See CUDA Dynamic Parallelism, above, for CDP2 version of document.

+
+

+13.6.1. Execution Environment and Memory Model (CDP1) +

+

See Execution Environment and Memory Model, above, for CDP2 version of document.

+
+

+13.6.1.1. Execution Environment (CDP1) +

+

See Execution Environment, above, for CDP2 version of document.

+

The CUDA execution model is based on primitives of threads, thread blocks, and grids, with kernel functions defining the program executed by individual threads within a thread block and grid. When a kernel function is invoked the grid’s properties are described by an execution configuration, which has a special syntax in CUDA. Support for dynamic parallelism in CUDA extends the ability to configure, launch, and synchronize upon new grids to threads that are running on the device.

+
+

Warning

+

Explicit synchronization with child kernels from a parent block (i.e. using cudaDeviceSynchronize() in device code) block is deprecated in CUDA 11.6, removed for compute_90+ compilation, and is slated for full removal in a future CUDA release.

+
+
+
+13.6.1.1.1. Parent and Child Grids (CDP1) +
+

See Parent and Child Grids, above, for CDP2 version of document.

+

A device thread that configures and launches a new grid belongs to the parent grid, and the grid created by the invocation is a child grid.

+

The invocation and completion of child grids is properly nested, meaning that the parent grid is not considered complete until all child grids created by its threads have completed. Even if the invoking threads do not explicitly synchronize on the child grids launched, the runtime guarantees an implicit synchronization between the parent and child.

+
+

Warning

+

Explicit synchronization with child kernels from a parent block (i.e. using cudaDeviceSynchronize() in device code) is deprecated in CUDA 11.6, removed for compute_90+ compilation, and is slated for full removal in a future CUDA release.

+
+
+The GPU Devotes More Transistors to Data Processing +
+

Figure 31 Parent-Child Launch Nesting

+
+
+
+
+
+13.6.1.1.2. Scope of CUDA Primitives (CDP1) +
+

See Scope of CUDA Primitives, above, for CDP2 version of document.

+

On both host and device, the CUDA runtime offers an API for launching kernels, for waiting for launched work to complete, and for tracking dependencies between launches via streams and events. On the host system, the state of launches and the CUDA primitives referencing streams and events are shared by all threads within a process; however processes execute independently and may not share CUDA objects.

+

A similar hierarchy exists on the device: launched kernels and CUDA objects are visible to all threads in a thread block, but are independent between thread blocks. This means for example that a stream may be created by one thread and used by any other thread in the same thread block, but may not be shared with threads in any other thread block.

+
+
+
+13.6.1.1.3. Synchronization (CDP1) +
+

See Synchronization, above, for CDP2 version of document.

+
+

Warning

+

Explicit synchronization with child kernels from a parent block (i.e. using cudaDeviceSynchronize() in device code) is deprecated in CUDA 11.6, removed for compute_90+ compilation, and is slated for full removal in a future CUDA release.

+
+

CUDA runtime operations from any thread, including kernel launches, are visible across a thread block. This means that an invoking thread in the parent grid may perform synchronization on the grids launched by that thread, by other threads in the thread block, or on streams created within the same thread block. Execution of a thread block is not considered complete until all launches by all threads in the block have completed. If all threads in a block exit before all child launches have completed, a synchronization operation will automatically be triggered.

+
+
+
+13.6.1.1.4. Streams and Events (CDP1) +
+

See Streams and Events, above, for CDP2 version of document.

+

CUDA Streams and Events allow control over dependencies between grid launches: grids launched into the same stream execute in-order, and events may be used to create dependencies between streams. Streams and events created on the device serve this exact same purpose.

+

Streams and events created within a grid exist within thread block scope but have undefined behavior when used outside of the thread block where they were created. As described above, all work launched by a thread block is implicitly synchronized when the block exits; work launched into streams is included in this, with all dependencies resolved appropriately. The behavior of operations on a stream that has been modified outside of thread block scope is undefined.

+

Streams and events created on the host have undefined behavior when used within any kernel, just as streams and events created by a parent grid have undefined behavior if used within a child grid.

+
+
+
+13.6.1.1.5. Ordering and Concurrency (CDP1) +
+

See Ordering and Concurrency, above, for CDP2 version of document.

+

The ordering of kernel launches from the device runtime follows CUDA Stream ordering semantics. Within a thread block, all kernel launches into the same stream are executed in-order. With multiple threads in the same thread block launching into the same stream, the ordering within the stream is dependent on the thread scheduling within the block, which may be controlled with synchronization primitives such as __syncthreads().

+

Note that because streams are shared by all threads within a thread block, the implicit NULL stream is also shared. If multiple threads in a thread block launch into the implicit stream, then these launches will be executed in-order. If concurrency is desired, explicit named streams should be used.

+

Dynamic Parallelism enables concurrency to be expressed more easily within a program; however, the device runtime introduces no new concurrency guarantees within the CUDA execution model. There is no guarantee of concurrent execution between any number of different thread blocks on a device.

+

The lack of concurrency guarantee extends to parent thread blocks and their child grids. When a parent thread block launches a child grid, the child is not guaranteed to begin execution until the parent thread block reaches an explicit synchronization point (such as cudaDeviceSynchronize()).

+
+

Warning

+

Explicit synchronization with child kernels from a parent block (i.e. using cudaDeviceSynchronize() in device code) is deprecated in CUDA 11.6, removed for compute_90+ compilation, and is slated for full removal in a future CUDA release.

+
+

While concurrency will often easily be achieved, it may vary as a function of deviceconfiguration, application workload, and runtime scheduling. It is therefore unsafe to depend upon any concurrency between different thread blocks.

+
+
+
+13.6.1.1.6. Device Management (CDP1) +
+

See Device Management, above, for CDP2 version of document.

+

There is no multi-GPU support from the device runtime; the device runtime is only capable of operating on the device upon which it is currently executing. It is permitted, however, to query properties for any CUDA capable device in the system.

+
+
+
+

+13.6.1.2. Memory Model (CDP1) +

+

See Memory Model, above, for CDP2 version of document.

+

Parent and child grids share the same global and constant memory storage, but have distinct local and shared memory.

+
+
+13.6.1.2.1. Coherence and Consistency (CDP1) +
+

See Coherence and Consistency, above, for CDP2 version of document.

+
+
+13.6.1.2.1.1. Global Memory (CDP1) +
+

See Global Memory, above, for CDP2 version of document.

+

Parent and child grids have coherent access to global memory, with weak consistency guarantees between child and parent. There are two points in the execution of a child grid when its view of memory is fully consistent with the parent thread: when the child grid is invoked by the parent, and when the child grid completes as signaled by a synchronization API invocation in the parent thread.

+
+

Warning

+

Explicit synchronization with child kernels from a parent block (i.e. using cudaDeviceSynchronize() in device code) is deprecated in CUDA 11.6, removed for compute_90+ compilation, and is slated for full removal in a future CUDA release.

+
+

All global memory operations in the parent thread prior to the child grid’s invocation are visible to the child grid. All memory operations of the child grid are visible to the parent after the parent has synchronized on the child grid’s completion.

+

In the following example, the child grid executing child_launch is only guaranteed to see the modifications to data made before the child grid was launched. Since thread 0 of the parent is performing the launch, the child will be consistent with the memory seen by thread 0 of the parent. Due to the first __syncthreads() call, the child will see data[0]=0, data[1]=1, …, data[255]=255 (without the __syncthreads() call, only data[0] would be guaranteed to be seen by the child). When the child grid returns, thread 0 is guaranteed to see modifications made by the threads in its child grid. Those modifications become available to the other threads of the parent grid only after the second __syncthreads() call:

+
+
__global__ void child_launch(int *data) {
+   data[threadIdx.x] = data[threadIdx.x]+1;
+}
+
+__global__ void parent_launch(int *data) {
+   data[threadIdx.x] = threadIdx.x;
+
+   __syncthreads();
+
+   if (threadIdx.x == 0) {
+       child_launch<<< 1, 256 >>>(data);
+       cudaDeviceSynchronize();
+   }
+
+   __syncthreads();
+}
+
+void host_launch(int *data) {
+    parent_launch<<< 1, 256 >>>(data);
+}
+
+
+
+
+
+13.6.1.2.1.2. Zero Copy Memory (CDP1) +
+

See Zero Copy Memory, above, for CDP2 version of document.

+

Zero-copy system memory has identical coherence and consistency guarantees to global memory, and follows the semantics detailed above. A kernel may not allocate or free zero-copy memory, but may use pointers to zero-copy passed in from the host program.

+
+
+
+13.6.1.2.1.3. Constant Memory (CDP1) +
+

See Constant Memory, above, for CDP2 version of document.

+

Constants are immutable and may not be modified from the device, even between parent and child launches. That is to say, the value of all __constant__ variables must be set from the host prior to launch. Constant memory is inherited automatically by all child kernels from their respective parents.

+

Taking the address of a constant memory object from within a kernel thread has the same semantics as for all CUDA programs, and passing that pointer from parent to child or from a child to parent is naturally supported.

+
+
+
+13.6.1.2.1.4. Shared and Local Memory (CDP1) +
+

See Shared and Local Memory, above, for CDP2 version of document.

+

Shared and Local memory is private to a thread block or thread, respectively, and is not visible or coherent between parent and child. Behavior is undefined when an object in one of these locations is referenced outside of the scope within which it belongs, and may cause an error.

+

The NVIDIA compiler will attempt to warn if it can detect that a pointer to local or shared memory is being passed as an argument to a kernel launch. At runtime, the programmer may use the __isGlobal() intrinsic to determine whether a pointer references global memory and so may safely be passed to a child launch.

+

Note that calls to cudaMemcpy*Async() or cudaMemset*Async() may invoke new child kernels on the device in order to preserve stream semantics. As such, passing shared or local memory pointers to these APIs is illegal and will return an error.

+
+
+
+13.6.1.2.1.5. Local Memory (CDP1) +
+

See Local Memory, above, for CDP2 version of document.

+

Local memory is private storage for an executing thread, and is not visible outside of that thread. It is illegal to pass a pointer to local memory as a launch argument when launching a child kernel. The result of dereferencing such a local memory pointer from a child will be undefined.

+

For example the following is illegal, with undefined behavior if x_array is accessed by child_launch:

+
+
int x_array[10];       // Creates x_array in parent's local memory
+child_launch<<< 1, 1 >>>(x_array);
+
+
+

It is sometimes difficult for a programmer to be aware of when a variable is placed into local memory by the compiler. As a general rule, all storage passed to a child kernel should be allocated explicitly from the global-memory heap, either with cudaMalloc(), new() or by declaring __device__ storage at global scope. For example:

+
+
// Correct - "value" is global storage
+__device__ int value;
+__device__ void x() {
+    value = 5;
+    child<<< 1, 1 >>>(&value);
+}
+
+
+
+
// Invalid - "value" is local storage
+__device__ void y() {
+    int value = 5;
+    child<<< 1, 1 >>>(&value);
+}
+
+
+
+
+
+13.6.1.2.1.6. Texture Memory (CDP1) +
+

See Texture Memory, above, for CDP2 version of document.

+

Writes to the global memory region over which a texture is mapped are incoherent with respect to texture accesses. Coherence for texture memory is enforced at the invocation of a child grid and when a child grid completes. This means that writes to memory prior to a child kernel launch are reflected in texture memory accesses of the child. Similarly, writes to memory by a child will be reflected in the texture memory accesses by a parent, but only after the parent synchronizes on the child’s completion. Concurrent accesses by parent and child may result in inconsistent data.

+
+

Warning

+

Explicit synchronization with child kernels from a parent block (i.e. using cudaDeviceSynchronize() in device code) is deprecated in CUDA 11.6, removed for compute_90+ compilation, and is slated for full removal in a future CUDA release.

+
+
+
+
+
+
+

+13.6.2. Programming Interface (CDP1) +

+

See Programming Interface, above, for CDP2 version of document.

+
+

+13.6.2.1. CUDA C++ Reference (CDP1) +

+

See CUDA C++ Reference, above, for CDP2 version of document.

+

This section describes changes and additions to the CUDA C++ language extensions for supporting Dynamic Parallelism.

+

The language interface and API available to CUDA kernels using CUDA C++ for Dynamic Parallelism, referred to as the Device Runtime, is substantially like that of the CUDA Runtime API available on the host. Where possible the syntax and semantics of the CUDA Runtime API have been retained in order to facilitate ease of code reuse for routines that may run in either the host or device environments.

+

As with all code in CUDA C++, the APIs and code outlined here is per-thread code. This enables each thread to make unique, dynamic decisions regarding what kernel or operation to execute next. There are no synchronization requirements between threads within a block to execute any of the provided device runtime APIs, which enables the device runtime API functions to be called in arbitrarily divergent kernel code without deadlock.

+
+
+13.6.2.1.1. Device-Side Kernel Launch (CDP1) +
+

See Kernel Launch APIs, above, for CDP2 version of document.

+

Kernels may be launched from the device using the standard CUDA <<< >>> syntax:

+
+
kernel_name<<< Dg, Db, Ns, S >>>([kernel arguments]);
+
+
+
    +
  • Dg is of type dim3 and specifies the dimensions and size of the grid

  • +
  • Db is of type dim3 and specifies the dimensions and size of each thread block

  • +
  • Ns is of type size_t and specifies the number of bytes of shared memory that is dynamically allocated per thread block for this call and addition to statically allocated memory. Ns is an optional argument that defaults to 0.

  • +
  • S is of type cudaStream_t and specifies the stream associated with this call. The stream must have been allocated in the same thread block where the call is being made. S is an optional argument that defaults to 0.

  • +
+
+
+13.6.2.1.1.1. Launches are Asynchronous (CDP1) +
+

See Launches are Asynchronous, above, for CDP2 version of document.

+

Identical to host-side launches, all device-side kernel launches are asynchronous with respect to the launching thread. That is to say, the <<<>>> launch command will return immediately and the launching thread will continue to execute until it hits an explicit launch-synchronization point such as cudaDeviceSynchronize().

+
+

Warning

+

Explicit synchronization with child kernels from a parent block (i.e. using cudaDeviceSynchronize() in device code) is deprecated in CUDA 11.6, removed for compute_90+ compilation, and is slated for full removal in a future CUDA release.

+
+

The grid launch is posted to the device and will execute independently of the parent thread. The child grid may begin execution at any time after launch, but is not guaranteed to begin execution until the launching thread reaches an explicit launch-synchronization point.

+
+
+
+13.6.2.1.1.2. Launch Environment Configuration (CDP1) +
+

See Launch Environment Configuration, above, for CDP2 version of document.

+

All global device configuration settings (for example, shared memory and L1 cache size as returned from cudaDeviceGetCacheConfig(), and device limits returned from cudaDeviceGetLimit()) will be inherited from the parent. Likewise, device limits such as stack size will remain as-configured.

+

For host-launched kernels, per-kernel configurations set from the host will take precedence over the global setting. These configurations will be used when the kernel is launched from the device as well. It is not possible to reconfigure a kernel’s environment from the device.

+
+
+
+
+13.6.2.1.2. Streams (CDP1) +
+

See Streams, above, for CDP2 version of document.

+

Both named and unnamed (NULL) streams are available from the device runtime. Named streams may be used by any thread within a thread-block, but stream handles may not be passed to other blocks or child/parent kernels. In other words, a stream should be treated as private to the block in which it is created. Stream handles are not guaranteed to be unique between blocks, so using a stream handle within a block that did not allocate it will result in undefined behavior.

+

Similar to host-side launch, work launched into separate streams may run concurrently, but actual concurrency is not guaranteed. Programs that depend upon concurrency between child kernels are not supported by the CUDA programming model and will have undefined behavior.

+

The host-side NULL stream’s cross-stream barrier semantic is not supported on the device (see below for details). In order to retain semantic compatibility with the host runtime, all device streams must be created using the cudaStreamCreateWithFlags() API, passing the cudaStreamNonBlocking flag. The cudaStreamCreate() call is a host-runtime- only API and will fail to compile for the device.

+

As cudaStreamSynchronize() and cudaStreamQuery() are unsupported by the device runtime, cudaDeviceSynchronize() should be used instead when the application needs to know that stream-launched child kernels have completed.

+
+

Warning

+

Explicit synchronization with child kernels from a parent block (i.e. using cudaDeviceSynchronize() in device code) is deprecated in CUDA 11.6, removed for compute_90+ compilation, and is slated for full removal in a future CUDA release.

+
+
+
+13.6.2.1.2.1. The Implicit (NULL) Stream (CDP1) +
+

See The Implicit (NULL) Stream, above, for CDP2 version of document.

+

Within a host program, the unnamed (NULL) stream has additional barrier synchronization semantics with other streams (see Default Stream for details). The device runtime offers a single implicit, unnamed stream shared between all threads in a block, but as all named streams must be created with the cudaStreamNonBlocking flag, work launched into the NULL stream will not insert an implicit dependency on pending work in any other streams (including NULL streams of other thread blocks).

+
+
+
+
+13.6.2.1.3. Events (CDP1) +
+

See Events, above, for CDP2 version of document.

+

Only the inter-stream synchronization capabilities of CUDA events are supported. This means that cudaStreamWaitEvent() is supported, but cudaEventSynchronize(), cudaEventElapsedTime(), and cudaEventQuery() are not. As cudaEventElapsedTime() is not supported, cudaEvents must be created via cudaEventCreateWithFlags(), passing the cudaEventDisableTiming flag.

+

As for all device runtime objects, event objects may be shared between all threads within the thread-block which created them but are local to that block and may not be passed to other kernels, or between blocks within the same kernel. Event handles are not guaranteed to be unique between blocks, so using an event handle within a block that did not create it will result in undefined behavior.

+
+
+
+13.6.2.1.4. Synchronization (CDP1) +
+

See Synchronization, above, for CDP2 version of document.

+
+

Warning

+

Explicit synchronization with child kernels from a parent block (i.e. using cudaDeviceSynchronize() in device code) is deprecated in CUDA 11.6, removed for compute_90+ compilation, and is slated for full removal in a future CUDA release.

+
+

The cudaDeviceSynchronize() function will synchronize on all work launched by any thread in the thread-block up to the point where cudaDeviceSynchronize() was called. Note that cudaDeviceSynchronize() may be called from within divergent code (see Block Wide Synchronization (CDP1)).

+

It is up to the program to perform sufficient additional inter-thread synchronization, for example via a call to __syncthreads(), if the calling thread is intended to synchronize with child grids invoked from other threads.

+
+
+13.6.2.1.4.1. Block Wide Synchronization (CDP1) +
+

See CUDA Dynamic Parallelism, above, for CDP2 version of document.

+

The cudaDeviceSynchronize() function does not imply intra-block synchronization. In particular, without explicit synchronization via a __syncthreads() directive the calling thread can make no assumptions about what work has been launched by any thread other than itself. For example if multiple threads within a block are each launching work and synchronization is desired for all this work at once (perhaps because of event-based dependencies), it is up to the program to guarantee that this work is submitted by all threads before calling cudaDeviceSynchronize().

+

Because the implementation is permitted to synchronize on launches from any thread in the block, it is quite possible that simultaneous calls to cudaDeviceSynchronize() by multiple threads will drain all work in the first call and then have no effect for the later calls.

+
+
+
+
+13.6.2.1.5. Device Management (CDP1) +
+

See Device Management, above, for CDP2 version of document.

+

Only the device on which a kernel is running will be controllable from that kernel. This means that device APIs such as cudaSetDevice() are not supported by the device runtime. The active device as seen from the GPU (returned from cudaGetDevice()) will have the same device number as seen from the host system. The cudaDeviceGetAttribute() call may request information about another device as this API allows specification of a device ID as a parameter of the call. Note that the catch-all cudaGetDeviceProperties() API is not offered by the device runtime - properties must be queried individually.

+
+
+
+13.6.2.1.6. Memory Declarations (CDP1) +
+

See Memory Declarations, above, for CDP2 version of document.

+
+
+13.6.2.1.6.1. Device and Constant Memory (CDP1) +
+

See Device and Constant Memory, above, for CDP2 version of document.

+

Memory declared at file scope with __device__ or __constant__ memory space specifiers behaves identically when using the device runtime. All kernels may read or write device variables, whether the kernel was initially launched by the host or device runtime. Equivalently, all kernels will have the same view of __constant__s as declared at the module scope.

+
+
+
+13.6.2.1.6.2. Textures and Surfaces (CDP1) +
+

See Textures and Surfaces, above, for CDP2 version of document.

+

CUDA supports dynamically created texture and surface objects7, where a texture object may be created on the host, passed to a kernel, used by that kernel, and then destroyed from the host. The device runtime does not allow creation or destruction of texture or surface objects from within device code, but texture and surface objects created from the host may be used and passed around freely on the device. Regardless of where they are created, dynamically created texture objects are always valid and may be passed to child kernels from a parent.

+
+

Note

+

The device runtime does not support legacy module-scope (i.e., Fermi-style) textures and surfaces within a kernel launched from the device. Module-scope (legacy) textures may be created from the host and used in device code as for any kernel, but may only be used by a top-level kernel (i.e., the one which is launched from the host).

+
+
+
+
+13.6.2.1.6.3. Shared Memory Variable Declarations (CDP1) +
+

See Shared Memory Variable Declarations, above, for CDP2 version of document.

+

In CUDA C++ shared memory can be declared either as a statically sized file-scope or function-scoped variable, or as an extern variable with the size determined at runtime by the kernel’s caller via a launch configuration argument. Both types of declarations are valid under the device runtime.

+
+
__global__ void permute(int n, int *data) {
+   extern __shared__ int smem[];
+   if (n <= 1)
+       return;
+
+   smem[threadIdx.x] = data[threadIdx.x];
+   __syncthreads();
+
+   permute_data(smem, n);
+   __syncthreads();
+
+   // Write back to GMEM since we can't pass SMEM to children.
+   data[threadIdx.x] = smem[threadIdx.x];
+   __syncthreads();
+
+   if (threadIdx.x == 0) {
+       permute<<< 1, 256, n/2*sizeof(int) >>>(n/2, data);
+       permute<<< 1, 256, n/2*sizeof(int) >>>(n/2, data+n/2);
+   }
+}
+
+void host_launch(int *data) {
+    permute<<< 1, 256, 256*sizeof(int) >>>(256, data);
+}
+
+
+
+
+
+13.6.2.1.6.4. Symbol Addresses (CDP1) +
+

See Symbol Addresses, above, for CDP2 version of document.

+

Device-side symbols (i.e., those marked __device__) may be referenced from within a kernel simply via the & operator, as all global-scope device variables are in the kernel’s visible address space. This also applies to __constant__ symbols, although in this case the pointer will reference read-only data.

+

Given that device-side symbols can be referenced directly, those CUDA runtime APIs which reference symbols (for example, cudaMemcpyToSymbol() or cudaGetSymbolAddress()) are redundant and hence not supported by the device runtime. Note this implies that constant data cannot be altered from within a running kernel, even ahead of a child kernel launch, as references to __constant__ space are read-only.

+
+
+
+
+13.6.2.1.7. API Errors and Launch Failures (CDP1) +
+

See API Errors and Launch Failures, above, for CDP2 version of document.

+

As usual for the CUDA runtime, any function may return an error code. The last error code returned is recorded and may be retrieved via the cudaGetLastError() call. Errors are recorded per-thread, so that each thread can identify the most recent error that it has generated. The error code is of type cudaError_t.

+

Similar to a host-side launch, device-side launches may fail for many reasons (invalid arguments, and so on). The user must call cudaGetLastError() to determine if a launch generated an error, however lack of an error after launch does not imply the child kernel completed successfully.

+

For device-side exceptions, for example, access to an invalid address, an error in a child grid will be returned to the host instead of being returned by the parent’s call to cudaDeviceSynchronize().

+
+
+13.6.2.1.7.1. Launch Setup APIs (CDP1) +
+

See Launch Setup APIs, above, for CDP2 version of document.

+

Kernel launch is a system-level mechanism exposed through the device runtime library, and as such is available directly from PTX via the underlying cudaGetParameterBuffer() and cudaLaunchDevice() APIs. It is permitted for a CUDA application to call these APIs itself, with the same requirements as for PTX. In both cases, the user is then responsible for correctly populating all necessary data structures in the correct format according to specification. Backwards compatibility is guaranteed in these data structures.

+

As with host-side launch, the device-side operator <<<>>> maps to underlying kernel launch APIs. This is so that users targeting PTX will be able to enact a launch, and so that the compiler front-end can translate <<<>>> into these calls.

+ + ++++ + + + + + + + + + + + + + + + + +
+Table 15 New Device-only Launch Implementation Functions +

Runtime API Launch Functions

Description of Difference From Host Runtime Behaviour (behavior is identical if no description)

cudaGetParameterBuffer

Generated automatically from <<<>>>. Note different API to host equivalent.

cudaLaunchDevice

Generated automatically from <<<>>>. Note different API to host equivalent.

+

The APIs for these launch functions are different to those of the CUDA Runtime API, and are defined as follows:

+
+
extern   device   cudaError_t cudaGetParameterBuffer(void **params);
+extern __device__ cudaError_t cudaLaunchDevice(void *kernel,
+                                        void *params, dim3 gridDim,
+                                        dim3 blockDim,
+                                        unsigned int sharedMemSize = 0,
+                                        cudaStream_t stream = 0);
+
+
+
+
+
+
+13.6.2.1.8. API Reference (CDP1) +
+

See API Reference, above, for CDP2 version of document.

+

The portions of the CUDA Runtime API supported in the device runtime are detailed here. Host and device runtime APIs have identical syntax; semantics are the same except where indicated. The table below provides an overview of the API relative to the version available from the host.

+ + ++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+Table 16 Supported API Functions +

Runtime API Functions

Details

cudaDeviceSynchronize

+

Synchronizes on work launched from thread’s own block only.

+

Warning: Note that calling this API from device code is deprecated in CUDA 11.6, removed for compute_90+ compilation, and is slated for full removal in a future CUDA release.

+

cudaDeviceGetCacheConfig

cudaDeviceGetLimit

cudaGetLastError

Last error is per-thread state, not per-block state

cudaPeekAtLastError

cudaGetErrorString

cudaGetDeviceCount

cudaDeviceGetAttribute

Will return attributes for any device

cudaGetDevice

Always returns current device ID as would be seen from host

cudaStreamCreateWithFlags

Must pass cudaStreamNonBlocking flag

cudaStreamDestroy

cudaStreamWaitEvent

cudaEventCreateWithFlags

Must pass cudaEventDisableTiming flag

cudaEventRecord

cudaEventDestroy

cudaFuncGetAttributes

cudaMemcpyAsync

+

Notes about all memcpy/memset functions:

+
    +
  • Only async memcpy/set functions are supported

  • +
  • Only device-to-device memcpy is permitted

  • +
  • May not pass in local or shared memory pointers

  • +
+

cudaMemcpy2DAsync

cudaMemcpy3DAsync

cudaMemsetAsync

cudaMemset2DAsync

cudaMemset3DAsync

cudaRuntimeGetVersion

cudaMalloc

May not call cudaFree on the device on a pointer created on the host, and vice-versa

cudaFree

cudaOccupancyMaxActiveBlocksPerMultiprocessor

cudaOccupancyMaxPotentialBlockSize

cudaOccupancyMaxPotentialBlockSizeVariableSMem

+
+
+
+

+13.6.2.2. Device-side Launch from PTX (CDP1) +

+

See Device-side Launch from PTX, above, for CDP2 version of document.

+

This section is for the programming language and compiler implementers who target Parallel Thread Execution (PTX) and plan to support Dynamic Parallelism in their language. It provides the low-level details related to supporting kernel launches at the PTX level.

+
+
+13.6.2.2.1. Kernel Launch APIs (CDP1) +
+

See Kernel Launch APIs, above, for CDP2 version of document.

+

Device-side kernel launches can be implemented using the following two APIs accessible from PTX: cudaLaunchDevice() and cudaGetParameterBuffer(). cudaLaunchDevice() launches the specified kernel with the parameter buffer that is obtained by calling cudaGetParameterBuffer() and filled with the parameters to the launched kernel. The parameter buffer can be NULL, i.e., no need to invoke cudaGetParameterBuffer(), if the launched kernel does not take any parameters.

+
+
+13.6.2.2.1.1. cudaLaunchDevice (CDP1) +
+

See cudaLaunchDevice, above, for CDP2 version of document.

+

At the PTX level, cudaLaunchDevice()needs to be declared in one of the two forms shown below before it is used.

+
+
// PTX-level Declaration of cudaLaunchDevice() when .address_size is 64
+.extern .func(.param .b32 func_retval0) cudaLaunchDevice
+(
+  .param .b64 func,
+  .param .b64 parameterBuffer,
+  .param .align 4 .b8 gridDimension[12],
+  .param .align 4 .b8 blockDimension[12],
+  .param .b32 sharedMemSize,
+  .param .b64 stream
+)
+;
+
+
+
+
// PTX-level Declaration of cudaLaunchDevice() when .address_size is 32
+.extern .func(.param .b32 func_retval0) cudaLaunchDevice
+(
+  .param .b32 func,
+  .param .b32 parameterBuffer,
+  .param .align 4 .b8 gridDimension[12],
+  .param .align 4 .b8 blockDimension[12],
+  .param .b32 sharedMemSize,
+  .param .b32 stream
+)
+;
+
+
+

The CUDA-level declaration below is mapped to one of the aforementioned PTX-level declarations and is found in the system header file cuda_device_runtime_api.h. The function is defined in the cudadevrt system library, which must be linked with a program in order to use device-side kernel launch functionality.

+
+
// CUDA-level declaration of cudaLaunchDevice()
+extern "C" __device__
+cudaError_t cudaLaunchDevice(void *func, void *parameterBuffer,
+                             dim3 gridDimension, dim3 blockDimension,
+                             unsigned int sharedMemSize,
+                             cudaStream_t stream);
+
+
+

The first parameter is a pointer to the kernel to be is launched, and the second parameter is the parameter buffer that holds the actual parameters to the launched kernel. The layout of the parameter buffer is explained in Parameter Buffer Layout (CDP1), below. Other parameters specify the launch configuration, i.e., as grid dimension, block dimension, shared memory size, and the stream associated with the launch (please refer to Execution Configuration for the detailed description of launch configuration.

+
+
+
+13.6.2.2.1.2. cudaGetParameterBuffer (CDP1) +
+

See cudaGetParameterBuffer, above, for CDP2 version of document.

+

cudaGetParameterBuffer() needs to be declared at the PTX level before it’s used. The PTX-level declaration must be in one of the two forms given below, depending on address size:

+
+
// PTX-level Declaration of cudaGetParameterBuffer() when .address_size is 64
+// When .address_size is 64
+.extern .func(.param .b64 func_retval0) cudaGetParameterBuffer
+(
+  .param .b64 alignment,
+  .param .b64 size
+)
+;
+
+
+
+
// PTX-level Declaration of cudaGetParameterBuffer() when .address_size is 32
+.extern .func(.param .b32 func_retval0) cudaGetParameterBuffer
+(
+  .param .b32 alignment,
+  .param .b32 size
+)
+;
+
+
+

The following CUDA-level declaration of cudaGetParameterBuffer() is mapped to the aforementioned PTX-level declaration:

+
+
// CUDA-level Declaration of cudaGetParameterBuffer()
+extern "C" __device__
+void *cudaGetParameterBuffer(size_t alignment, size_t size);
+
+
+

The first parameter specifies the alignment requirement of the parameter buffer and the second parameter the size requirement in bytes. In the current implementation, the parameter buffer returned by cudaGetParameterBuffer() is always guaranteed to be 64- byte aligned, and the alignment requirement parameter is ignored. However, it is recommended to pass the correct alignment requirement value - which is the largest alignment of any parameter to be placed in the parameter buffer - to cudaGetParameterBuffer() to ensure portability in the future.

+
+
+
+
+13.6.2.2.2. Parameter Buffer Layout (CDP1) +
+

See Parameter Buffer Layout, above, for CDP2 version of document.

+

Parameter reordering in the parameter buffer is prohibited, and each individual parameter placed in the parameter buffer is required to be aligned. That is, each parameter must be placed at the nth byte in the parameter buffer, where n is the smallest multiple of the parameter size that is greater than the offset of the last byte taken by the preceding parameter. The maximum size of the parameter buffer is 4KB.

+

For a more detailed description of PTX code generated by the CUDA compiler, please refer to the PTX-3.5 specification.

+
+
+
+

+13.6.2.3. Toolkit Support for Dynamic Parallelism (CDP1) +

+

See Toolkit Support for Dynamic Parallelism, above, for CDP2 version of document.

+
+
+13.6.2.3.1. Including Device Runtime API in CUDA Code (CDP1) +
+

See Including Device Runtime API in CUDA Code, above, for CDP2 version of document.

+

Similar to the host-side runtime API, prototypes for the CUDA device runtime API are included automatically during program compilation. There is no need to include cuda_device_runtime_api.h explicitly.

+
+
+
+13.6.2.3.2. Compiling and Linking (CDP1) +
+

See Compiling and Linking, above, for CDP2 version of document.

+

When compiling and linking CUDA programs using dynamic parallelism with nvcc, the program will automatically link against the static device runtime library libcudadevrt.

+

The device runtime is offered as a static library (cudadevrt.lib on Windows, libcudadevrt.a under Linux), against which a GPU application that uses the device runtime must be linked. Linking of device libraries can be accomplished through nvcc and/or nvlink. Two simple examples are shown below.

+

A device runtime program may be compiled and linked in a single step, if all required source files can be specified from the command line:

+
+
$ nvcc -arch=sm_75 -rdc=true hello_world.cu -o hello -lcudadevrt
+
+
+

It is also possible to compile CUDA .cu source files first to object files, and then link these together in a two-stage process:

+
+
$ nvcc -arch=sm_75 -dc hello_world.cu -o hello_world.o
+$ nvcc -arch=sm_75 -rdc=true hello_world.o -o hello -lcudadevrt
+
+
+

Please see the Using Separate Compilation section of The CUDA Driver Compiler NVCC guide for more details.

+
+
+
+
+

+13.6.3. Programming Guidelines (CDP1) +

+

See Programming Guidelines, above, for CDP2 version of document.

+
+

+13.6.3.1. Basics (CDP1) +

+

See Basics, above, for CDP2 version of document.

+

The device runtime is a functional subset of the host runtime. API level device management, kernel launching, device memcpy, stream management, and event management are exposed from the device runtime.

+

Programming for the device runtime should be familiar to someone who already has experience with CUDA. Device runtime syntax and semantics are largely the same as that of the host API, with any exceptions detailed earlier in this document.

+
+

Warning

+

Explicit synchronization with child kernels from a parent block (i.e. using cudaDeviceSynchronize() in device code) is deprecated in CUDA 11.6, removed for compute_90+ compilation, and is slated for full removal in a future CUDA release.

+
+

The following example shows a simple Hello World program incorporating dynamic parallelism:

+
+
#include <stdio.h>
+
+__global__ void childKernel()
+{
+    printf("Hello ");
+}
+
+__global__ void parentKernel()
+{
+    // launch child
+    childKernel<<<1,1>>>();
+    if (cudaSuccess != cudaGetLastError()) {
+        return;
+    }
+
+    // wait for child to complete
+    if (cudaSuccess != cudaDeviceSynchronize()) {
+        return;
+    }
+
+    printf("World!\n");
+}
+
+int main(int argc, char *argv[])
+{
+    // launch parent
+    parentKernel<<<1,1>>>();
+    if (cudaSuccess != cudaGetLastError()) {
+        return 1;
+    }
+
+    // wait for parent to complete
+    if (cudaSuccess != cudaDeviceSynchronize()) {
+        return 2;
+    }
+
+    return 0;
+}
+
+
+

This program may be built in a single step from the command line as follows:

+
+
$ nvcc -arch=sm_75 -rdc=true hello_world.cu -o hello -lcudadevrt
+
+
+
+
+

+13.6.3.2. Performance (CDP1) +

+

See Performance, above, for CDP2 version of document.

+
+
+13.6.3.2.1. Synchronization (CDP1) +
+

See CUDA Dynamic Parallelism, above, for CDP2 version of document.

+
+

Warning

+

Explicit synchronization with child kernels from a parent block (such as using cudaDeviceSynchronize() in device code) is deprecated in CUDA 11.6, removed for compute_90+ compilation, and is slated for full removal in a future CUDA release.

+
+

Synchronization by one thread may impact the performance of other threads in the same Thread Block, even when those other threads do not call cudaDeviceSynchronize() themselves. This impact will depend upon the underlying implementation. In general the implicit synchronization of child kernels done when a thread block ends is more efficient compared to calling cudaDeviceSynchronize() explicitly. It is therefore recommended to only call cudaDeviceSynchronize() if it is needed to synchronize with a child kernel before a thread block ends.

+
+
+
+13.6.3.2.2. Dynamic-parallelism-enabled Kernel Overhead (CDP1) +
+

See Dynamic-parallelism-enabled Kernel Overhead, above, for CDP2 version of document.

+

System software which is active when controlling dynamic launches may impose an overhead on any kernel which is running at the time, whether or not it invokes kernel launches of its own. This overhead arises from the device runtime’s execution tracking and management software and may result in decreased performance for example, library calls when made from the device compared to from the host side. This overhead is, in general, incurred for applications that link against the device runtime library.

+
+
+
+

+13.6.3.3. Implementation Restrictions and Limitations (CDP1) +

+

See Implementation Restrictions and Limitations, above, for CDP2 version of document.

+

Dynamic Parallelism guarantees all semantics described in this document, however, certain hardware and software resources are implementation-dependent and limit the scale, performance and other properties of a program which uses the device runtime.

+
+
+13.6.3.3.1. Runtime (CDP1) +
+

See Runtime, above, for CDP2 version of document.

+
+
+13.6.3.3.1.1. Memory Footprint (CDP1) +
+

See Memory Footprint, above, for CDP2 version of document.

+

The device runtime system software reserves memory for various management purposes, in particular one reservation which is used for saving parent-grid state during synchronization, and a second reservation for tracking pending grid launches. Configuration controls are available to reduce the size of these reservations in exchange for certain launch limitations. See Configuration Options (CDP1), below, for details.

+

The majority of reserved memory is allocated as backing-store for parent kernel state, for use when synchronizing on a child launch. Conservatively, this memory must support storing of state for the maximum number of live threads possible on the device. This means that each parent generation at which cudaDeviceSynchronize() is callable may require up to 860MB of device memory, depending on the device configuration, which will be unavailable for program use even if it is not all consumed.

+
+
+
+13.6.3.3.1.2. Nesting and Synchronization Depth (CDP1) +
+

See CUDA Dynamic Parallelism, above, for CDP2 version of document.

+

Using the device runtime, one kernel may launch another kernel, and that kernel may launch another, and so on. Each subordinate launch is considered a new nesting level, and the total number of levels is the nesting depth of the program. The synchronization depth is defined as the deepest level at which the program will explicitly synchronize on a child launch. Typically this is one less than the nesting depth of the program, but if the program does not need to call cudaDeviceSynchronize() at all levels then the synchronization depth might be substantially different to the nesting depth.

+
+

Warning

+

Explicit synchronization with child kernels from a parent block (i.e. using cudaDeviceSynchronize() in device code) is deprecated in CUDA 11.6, removed for compute_90+ compilation, and is slated for full removal in a future CUDA release.

+
+

The overall maximum nesting depth is limited to 24, but practically speaking the real limit will be the amount of memory required by the system for each new level (see Memory Footprint (CDP1) above). Any launch which would result in a kernel at a deeper level than the maximum will fail. Note that this may also apply to cudaMemcpyAsync(), which might itself generate a kernel launch. See Configuration Options (CDP1) for details.

+

By default, sufficient storage is reserved for two levels of synchronization. This maximum synchronization depth (and hence reserved storage) may be controlled by calling cudaDeviceSetLimit() and specifying cudaLimitDevRuntimeSyncDepth. The number of levels to be supported must be configured before the top-level kernel is launched from the host, in order to guarantee successful execution of a nested program. Calling cudaDeviceSynchronize() at a depth greater than the specified maximum synchronization depth will return an error.

+

An optimization is permitted where the system detects that it need not reserve space for the parent’s state in cases where the parent kernel never calls cudaDeviceSynchronize(). In this case, because explicit parent/child synchronization never occurs, the memory footprint required for a program will be much less than the conservative maximum. Such a program could specify a shallower maximum synchronization depth to avoid over-allocation of backing store.

+
+
+
+13.6.3.3.1.3. Pending Kernel Launches (CDP1) +
+

See Pending Kernel Launches, above, for CDP2 version of document.

+

When a kernel is launched, all associated configuration and parameter data is tracked until the kernel completes. This data is stored within a system-managed launch pool.

+

The launch pool is divided into a fixed-size pool and a virtualized pool with lower performance. The device runtime system software will try to track launch data in the fixed-size pool first. The virtualized pool will be used to track new launches when the fixed-size pool is full.

+

The size of the fixed-size launch pool is configurable by calling cudaDeviceSetLimit() from the host and specifying cudaLimitDevRuntimePendingLaunchCount.

+
+
+
+13.6.3.3.1.4. Configuration Options (CDP1) +
+

See Configuration Options, above, for CDP2 version of document.

+

Resource allocation for the device runtime system software is controlled via the cudaDeviceSetLimit() API from the host program. Limits must be set before any kernel is launched, and may not be changed while the GPU is actively running programs.

+
+

Warning

+

Explicit synchronization with child kernels from a parent block (i.e. using cudaDeviceSynchronize() in device code) is deprecated in CUDA 11.6, removed for compute_90+ compilation, and is slated for full removal in a future CUDA release.

+
+

The following named limits may be set:

+ ++++ + + + + + + + + + + + + + + + + + + + + +

Limit

Behavior

cudaLimitDevRuntimeSyncDepth

Sets the maximum depth at which cudaDeviceSynchronize() may be called. Launches may be performed deeper than this, but explicit synchronization deeper than this limit will return the cudaErrorLaunchMaxDepthExceeded. The default maximum sync depth is 2.

cudaLimitDevRuntimePendingLaunchCount

Controls the amount of memory set aside for buffering kernel launches which have not yet begun to execute, due either to unresolved dependencies or lack of execution resources. When the buffer is full, the device runtime system software will attempt to track new pending launches in a lower performance virtualized buffer. If the virtualized buffer is also full, i.e. when all available heap space is consumed, launches will not occur, and the thread’s last error will be set to cudaErrorLaunchPendingCountExceeded. The default pending launch count is 2048 launches.

cudaLimitStackSize

Controls the stack size in bytes of each GPU thread. The CUDA driver automatically increases the per-thread stack size for each kernel launch as needed. This size isn’t reset back to the original value after each launch. To set the per-thread stack size to a different value, cudaDeviceSetLimit() can be called to set this limit. The stack will be immediately resized, and if necessary, the device will block until all preceding requested tasks are complete. cudaDeviceGetLimit() can be called to get the current per-thread stack size.

+
+
+
+13.6.3.3.1.5. Memory Allocation and Lifetime (CDP1) +
+

See Memory Allocation and Lifetime, above, for CDP2 version of document.

+

cudaMalloc() and cudaFree() have distinct semantics between the host and device environments. When invoked from the host, cudaMalloc() allocates a new region from unused device memory. When invoked from the device runtime these functions map to device-side malloc() and free(). This implies that within the device environment the total allocatable memory is limited to the device malloc() heap size, which may be smaller than the available unused device memory. Also, it is an error to invoke cudaFree() from the host program on a pointer which was allocated by cudaMalloc() on the device or vice-versa.

+ +++++ + + + + + + + + + + + + + + + + + + + + + + + + +

cudaMalloc() on Host

cudaMalloc() on Device

cudaFree() on Host

Supported

Not Supported

cudaFree() on Device

Not Supported

Supported

Allocation limit

Free device memory

cudaLimitMallocHeapSize

+
+
+
+13.6.3.3.1.6. SM Id and Warp Id (CDP1) +
+

See SM Id and Warp Id, above, for CDP2 version of document.

+

Note that in PTX %smid and %warpid are defined as volatile values. The device runtime may reschedule thread blocks onto different SMs in order to more efficiently manage resources. As such, it is unsafe to rely upon %smid or %warpid remaining unchanged across the lifetime of a thread or thread block.

+
+
+
+13.6.3.3.1.7. ECC Errors (CDP1) +
+

See ECC Errors, above, for CDP2 version of document.

+

No notification of ECC errors is available to code within a CUDA kernel. ECC errors are reported at the host side once the entire launch tree has completed. Any ECC errors which arise during execution of a nested program will either generate an exception or continue execution (depending upon error and configuration).

+
+
+7(1,2,3) +
+
+

Dynamically created texture and surface objects are an addition to the CUDA memory model introduced with CUDA 5.0. Please see the CUDA Programming Guide for details.

+
+
+
+
+
+
+
+
+
+

+14. Virtual Memory Management +

+
+

Warning

+

This document has been replaced by a new CUDA Programming Guide. The information in this document should be considered legacy, and this document is no longer being updated as of CUDA 13.0. Please refer to the CUDA Programming Guide for up-to-date information on CUDA.

+
+
+

+14.1. Introduction +

+

The Virtual Memory Management APIs provide a way for the application to directly manage the unified virtual address space that CUDA provides to map physical memory to virtual addresses accessible by the GPU. Introduced in CUDA 10.2, these APIs additionally provide a new way to interop with other processes and graphics APIs like OpenGL and Vulkan, as well as provide newer memory attributes that a user can tune to fit their applications.

+

Historically, memory allocation calls (such as cudaMalloc()) in the CUDA programming model have returned a memory address that points to the GPU memory. The address thus obtained could be used with any CUDA API or inside a device kernel. However, the memory allocated could not be resized depending on the user’s memory needs. In order to increase an allocation’s size, the user had to explicitly allocate a larger buffer, copy data from the initial allocation, free it and then continue to keep track of the newer allocation’s address. This often leads to lower performance and higher peak memory utilization for applications. Essentially, users had a malloc-like interface for allocating GPU memory, but did not have a corresponding realloc to complement it. The Virtual Memory Management APIs decouple the idea of an address and memory and allow the application to handle them separately. The APIs allow applications to map and unmap memory from a virtual address range as they see fit.

+

In the case of enabling peer device access to memory allocations by using cudaEnablePeerAccess, all past and future user allocations are mapped to the target peer device. This lead to users unwittingly paying runtime cost of mapping all cudaMalloc allocations to peer devices. However, in most situations applications communicate by sharing only a few allocations with another device and not all allocations are required to be mapped to all the devices. With Virtual Memory Management, applications can specifically choose certain allocations to be accessible from target devices.

+

The CUDA Virtual Memory Management APIs expose fine grained control to the user for managing the GPU memory in applications. It provides APIs that let users:

+
    +
  • Place memory allocated on different devices into a contiguous VA range.

  • +
  • Perform interprocess communication for memory sharing using platform-specific mechanisms.

  • +
  • Opt into newer memory types on the devices that support them.

  • +
+

In order to allocate memory, the Virtual Memory Management programming model exposes the following functionality:

+
    +
  • Allocating physical memory.

  • +
  • Reserving a VA range.

  • +
  • Mapping allocated memory to the VA range.

  • +
  • Controlling access rights on the mapped range.

  • +
+

Note that the suite of APIs described in this section require a system that supports UVA.

+
+
+

+14.2. Query for Support +

+

Before attempting to use Virtual Memory Management APIs, applications must ensure that the devices they want to use support CUDA Virtual Memory Management. The following code sample shows querying for Virtual Memory Management support:

+
+
int deviceSupportsVmm;
+CUresult result = cuDeviceGetAttribute(&deviceSupportsVmm, CU_DEVICE_ATTRIBUTE_VIRTUAL_MEMORY_MANAGEMENT_SUPPORTED, device);
+if (deviceSupportsVmm != 0) {
+    // `device` supports Virtual Memory Management
+}
+
+
+
+
+

+14.3. Allocating Physical Memory +

+

The first step in memory allocation using Virtual Memory Management APIs is to create a physical memory chunk that will provide a backing for the allocation. In order to allocate physical memory, applications must use the cuMemCreate API. The allocation created by this function does not have any device or host mappings. The function argument CUmemGenericAllocationHandle describes the properties of the memory to allocate such as the location of the allocation, if the allocation is going to be shared to another process (or other Graphics APIs), or the physical attributes of the memory to be allocated. Users must ensure the requested allocation’s size must be aligned to appropriate granularity. Information regarding an allocation’s granularity requirements can be queried using cuMemGetAllocationGranularity. The following code snippet shows allocating physical memory with cuMemCreate:

+
+
CUmemGenericAllocationHandle allocatePhysicalMemory(int device, size_t size) {
+    CUmemAllocationProp prop = {};
+    prop.type = CU_MEM_ALLOCATION_TYPE_PINNED;
+    prop.location.type = CU_MEM_LOCATION_TYPE_DEVICE;
+    prop.location.id = device;
+
+    size_t granularity = 0;
+    cuMemGetAllocationGranularity(&granularity, &prop, CU_MEM_ALLOC_GRANULARITY_MINIMUM);
+
+    // Ensure size matches granularity requirements for the allocation
+    size_t padded_size = ROUND_UP(size, granularity);
+
+    // Allocate physical memory
+    CUmemGenericAllocationHandle allocHandle;
+    cuMemCreate(&allocHandle, padded_size, &prop, 0);
+
+    return allocHandle;
+}
+
+
+

The memory allocated by cuMemCreate is referenced by the CUmemGenericAllocationHandle it returns. This is a departure from the cudaMalloc-style of allocation, which returns a pointer to the GPU memory, which was directly accessible by CUDA kernel executing on the device. The memory allocated cannot be used for any operations other than querying properties using cuMemGetAllocationPropertiesFromHandle. In order to make this memory accessible, applications must map this memory into a VA range reserved by cuMemAddressReserve and provide suitable access rights to it. Applications must free the allocated memory using the cuMemRelease API.

+
+

+14.3.1. Shareable Memory Allocations +

+

With cuMemCreate users now have the facility to indicate to CUDA, at allocation time, that they have earmarked a particular allocation for Inter process communication and graphics interop purposes. Applications can do this by setting CUmemAllocationProp::requestedHandleTypes to a platform-specific field. On Windows, when CUmemAllocationProp::requestedHandleTypes is set to CU_MEM_HANDLE_TYPE_WIN32 applications must also specify an LPSECURITYATTRIBUTES attribute in CUmemAllocationProp::win32HandleMetaData. This security attribute defines the scope of which exported allocations may be transferred to other processes.

+

The CUDA Virtual Memory Management API functions do not support the legacy interprocess communication functions with their memory. Instead, they expose a new mechanism for interprocess communication that uses OS-specific handles. Applications can obtain these OS-specific handles corresponding to the allocations by using cuMemExportToShareableHandle. The handles thus obtained can be transferred by using the usual OS native mechanisms for inter process communication. The recipient process should import the allocation by using cuMemImportFromShareableHandle.

+

Users must ensure they query for support of the requested handle type before attempting to export memory allocated with cuMemCreate. The following code snippet illustrates query for handle type support in a platform-specific way.

+
+
int deviceSupportsIpcHandle;
+#if defined(__linux__)
+    cuDeviceGetAttribute(&deviceSupportsIpcHandle, CU_DEVICE_ATTRIBUTE_HANDLE_TYPE_POSIX_FILE_DESCRIPTOR_SUPPORTED, device));
+#else
+    cuDeviceGetAttribute(&deviceSupportsIpcHandle, CU_DEVICE_ATTRIBUTE_HANDLE_TYPE_WIN32_HANDLE_SUPPORTED, device));
+#endif
+
+
+

Users should set the CUmemAllocationProp::requestedHandleTypes appropriately as shown below:

+
+
#if defined(__linux__)
+    prop.requestedHandleTypes = CU_MEM_HANDLE_TYPE_POSIX_FILE_DESCRIPTOR;
+#else
+    prop.requestedHandleTypes = CU_MEM_HANDLE_TYPE_WIN32;
+    prop.win32HandleMetaData = // Windows specific LPSECURITYATTRIBUTES attribute.
+#endif
+
+
+

The memMapIpcDrv sample can be used as an example for using IPC with Virtual Memory Management allocations.

+
+
+

+14.3.2. Memory Type +

+

Before CUDA 10.2, applications had no user-controlled way of allocating any special type of memory that certain devices may support. With cuMemCreate, applications can additionally specify memory type requirements using the CUmemAllocationProp::allocFlags to opt into any specific memory features. Applications must also ensure that the requested memory type is supported on the device of allocation.

+
+

+14.3.2.1. Compressible Memory +

+

Compressible memory can be used to accelerate accesses to data with unstructured sparsity and other compressible data patterns. Compression can save DRAM bandwidth, L2 read bandwidth and L2 capacity depending on the data being operated on. Applications that want to allocate compressible memory on devices that support Compute Data Compression can do so by setting CUmemAllocationProp::allocFlags::compressionType to CU_MEM_ALLOCATION_COMP_GENERIC. Users must query if device supports Compute Data Compression by using CU_DEVICE_ATTRIBUTE_GENERIC_COMPRESSION_SUPPORTED. The following code snippet illustrates querying compressible memory support cuDeviceGetAttribute.

+
+
int compressionSupported = 0;
+cuDeviceGetAttribute(&compressionSupported, CU_DEVICE_ATTRIBUTE_GENERIC_COMPRESSION_SUPPORTED, device);
+
+
+

On devices that support Compute Data Compression, users must opt in at allocation time as shown below:

+
+
prop.allocFlags.compressionType = CU_MEM_ALLOCATION_COMP_GENERIC;
+
+
+

Due to various reasons such as limited HW resources, the allocation may not have compression attributes, the user is expected to query back the properties of the allocated memory using cuMemGetAllocationPropertiesFromHandle and check for compression attribute.

+
+
CUmemAllocationProp allocationProp = {};
+cuMemGetAllocationPropertiesFromHandle(&allocationProp, allocationHandle);
+
+if (allocationProp.allocFlags.compressionType == CU_MEM_ALLOCATION_COMP_GENERIC)
+{
+    // Obtained compressible memory allocation
+}
+
+
+
+
+
+
+

+14.4. Reserving a Virtual Address Range +

+

Since with Virtual Memory Management the notions of address and memory are distinct, applications must carve out an address range that can hold the memory allocations made by cuMemCreate. The address range reserved must be at least as large as the sum of the sizes of all the physical memory allocations the user plans to place in them.

+

Applications can reserve a virtual address range by passing appropriate parameters to cuMemAddressReserve. The address range obtained will not have any device or host physical memory associated with it. The reserved virtual address range can be mapped to memory chunks belonging to any device in the system, thus providing the application a continuous VA range backed and mapped by memory belonging to different devices. Applications are expected to return the virtual address range back to CUDA using cuMemAddressFree. Users must ensure that the entire VA range is unmapped before calling cuMemAddressFree. These functions are conceptually similar to mmap/munmap (on Linux) or VirtualAlloc/VirtualFree (on Windows) functions. The following code snippet illustrates the usage for the function:

+
+
CUdeviceptr ptr;
+// `ptr` holds the returned start of virtual address range reserved.
+CUresult result = cuMemAddressReserve(&ptr, size, 0, 0, 0); // alignment = 0 for default alignment
+
+
+
+
+

+14.5. Virtual Aliasing Support +

+

The Virtual Memory Management APIs provide a way to create multiple virtual memory mappings or “proxies” to the same allocation using multiple calls to cuMemMap with different virtual addresses, so-called virtual aliasing. Unless otherwise noted in the PTX ISA, writes to one proxy of the allocation are considered inconsistent and incoherent with any other proxy of the same memory until the writing device operation (grid launch, memcpy, memset, and so on) completes. Grids present on the GPU prior to a writing device operation but reading after the writing device operation completes are also considered to have inconsistent and incoherent proxies.

+

For example, the following snippet is considered undefined, assuming device pointers A and B are virtual aliases of the same memory allocation:

+
+
__global__ void foo(char *A, char *B) {
+  *A = 0x1;
+  printf("%d\n", *B);    // Undefined behavior!  *B can take on either
+// the previous value or some value in-between.
+}
+
+
+

The following is defined behavior, assuming these two kernels are ordered monotonically (by streams or events).

+
+
__global__ void foo1(char *A) {
+  *A = 0x1;
+}
+
+__global__ void foo2(char *B) {
+  printf("%d\n", *B);    // *B == *A == 0x1 assuming foo2 waits for foo1
+// to complete before launching
+}
+
+cudaMemcpyAsync(B, input, size, stream1);    // Aliases are allowed at
+// operation boundaries
+foo1<<<1,1,0,stream1>>>(A);                  // allowing foo1 to access A.
+cudaEventRecord(event, stream1);
+cudaStreamWaitEvent(stream2, event);
+foo2<<<1,1,0,stream2>>>(B);
+cudaStreamWaitEvent(stream3, event);
+cudaMemcpyAsync(output, B, size, stream3);  // Both launches of foo2 and
+                                            // cudaMemcpy (which both
+                                            // read) wait for foo1 (which writes)
+                                            // to complete before proceeding
+
+
+

If accessing same allocation through different “proxies” is required in the same kernel a fence.proxy.alias can be used between the two accesses. The above example can thus be made legal with inline PTX assembly:

+
+
__global__ void foo(char *A, char *B) {
+  *A = 0x1;
+  asm volatile ("fence.proxy.alias;" ::: "memory");
+  printf("%d\n", *B);    // *B == *A == 0x1
+}
+
+
+
+
+

+14.6. Mapping Memory +

+

The allocated physical memory and the carved out virtual address space from the previous two sections represent the memory and address distinction introduced by the Virtual Memory Management APIs. For the allocated memory to be useable, the user must first place the memory in the address space. The address range obtained from cuMemAddressReserve and the physical allocation obtained from cuMemCreate or cuMemImportFromShareableHandle must be associated with each other by using cuMemMap.

+

Users can associate allocations from multiple devices to reside in contiguous virtual address ranges as long as they have carved out enough address space. In order to decouple the physical allocation and the address range, users must unmap the address of the mapping by using cuMemUnmap. Users can map and unmap memory to the same address range as many times as they want, as long as they ensure that they don’t attempt to create mappings on VA range reservations that are already mapped. The following code snippet illustrates the usage for the function:

+
+
CUdeviceptr ptr;
+// `ptr`: address in the address range previously reserved by cuMemAddressReserve.
+// `allocHandle`: CUmemGenericAllocationHandle obtained by a previous call to cuMemCreate.
+CUresult result = cuMemMap(ptr, size, 0, allocHandle, 0);
+
+
+
+
+

+14.7. Controlling Access Rights +

+

The Virtual Memory Management APIs enable applications to explicitly protect their VA ranges with access control mechanisms. Mapping the allocation to a region of the address range using cuMemMap does not make the address accessible, and would result in a program crash if accessed by a CUDA kernel. Users must specifically select access control using the cuMemSetAccess function, which allows or restricts access for specific devices to a mapped address range. The following code snippet illustrates the usage for the function:

+
+
void setAccessOnDevice(int device, CUdeviceptr ptr, size_t size) {
+    CUmemAccessDesc accessDesc = {};
+    accessDesc.location.type = CU_MEM_LOCATION_TYPE_DEVICE;
+    accessDesc.location.id = device;
+    accessDesc.flags = CU_MEM_ACCESS_FLAGS_PROT_READWRITE;
+
+    // Make the address accessible
+    cuMemSetAccess(ptr, size, &accessDesc, 1);
+}
+
+
+

The access control mechanism exposed with Virtual Memory Management allows users to be explicit about which allocations they want to share with other peer devices on the system. As specified earlier, cudaEnablePeerAccess forces all prior and future cudaMalloc’d allocations to be mapped to the target peer device. This can be convenient in many cases as user doesn’t have to worry about tracking the mapping state of every allocation to every device in the system. But for users concerned with performance of their applications this approach has performance implications. With access control at allocation granularity Virtual Memory Management exposes a mechanism to have peer mappings with minimal overhead.

+

The vectorAddMMAP sample can be used as an example for using the Virtual Memory Management APIs.

+
+
+

+14.8. Fabric Memory +

+

CUDA 12.4 introduced a new VMM allocation handle type CU_MEM_HANDLE_TYPE_FABRIC. On supported platforms and provided the NVIDIA IMEX daemon +is running this allocation handle type enables sharing allocations not only intra node with any communication mechanism, e.g. MPI, but also inter +node. This allows GPUs in a Multi Node NVLINK System to map the memory of all other GPUs part of the same NVLINK fabric even if they are in +different nodes greatly increasing the scale of multi-GPU Programming with NVLINK.

+
+

+14.8.1. Query for Support +

+

Before attempting to use Fabric Memory, applications must ensure that the devices they want to use support Fabric Memory. The following code +sample shows querying for Fabric Memory support:

+
+
int deviceSupportsFabricMem;
+CUresult result = cuDeviceGetAttribute(&deviceSupportsFabricMem, CU_DEVICE_ATTRIBUTE_HANDLE_TYPE_FABRIC_SUPPORTED, device);
+if (deviceSupportsFabricMem != 0) {
+    // `device` supports Fabric Memory
+}
+
+
+

Aside from using CU_MEM_HANDLE_TYPE_FABRIC as handle type and not requiring OS native mechanisms for inter process communication to exchange +sharable handles there is no difference in using Fabric Memory compared to other allocation handle types.

+
+
+
+

+14.9. Multicast Support +

+

The Multicast Object Management APIs +provide a way for the application to create Multicast Objects and in combination with the Virtual Memory Management APIs +described above allow applications to leverage NVLINK SHARP on supported NVLINK connected GPUs if they are connected with NVSWITCH. NVLINK SHARP +allows CUDA applications to leverage in fabric computing to accelerate operations like broadcast and reductions between GPUs connected with +NVSWITCH. For this to work multiple NVLINK connected GPUs form a Multicast Team and each GPU from the team backs up a Multicast Object with +physical memory. So a Multicast Team of N GPUs has N physical replicas, each local to one participating GPU, of a Multicast Object. +The multimem PTX instructions +using mappings of Multicast Objects work with all replicas of the Multicast Object.

+

To work with Multicast Objects an application needs to

+
    +
  • Query Multicast Support

  • +
  • Create a Multicast Handle with cuMulticastCreate.

  • +
  • Share the Multicast Handle with all processes that control a GPU which should participate in a Multicast Team. This works with cuMemExportToShareableHandle as described above.

  • +
  • Add all GPUs that should participate in the Multicast Team with cuMulticastAddDevice.

  • +
  • For each participating GPU bind physical memory allocated with cuMemCreate as described above to the Multicast Handle. All devices need to be added to the Multicast Team before binding memory on any device.

  • +
  • Reserve an address range, map the Multicast Handle and set Access Rights as described above for regular Unicast mappings. Unicast and Multicast mappings to the same physical memory are possible. See the Virtual Aliasing Support section above how to ensure consistency between multiple mappings to the same physical memory.

  • +
  • Use the multimem PTX instructions with the multicast mappings.

  • +
+

The multi_node_p2p example in the Multi GPU Programming Models GitHub +repository contains a complete example using Fabric Memory including Multicast Objects to leverage NVLINK SHARP. Please note that this example is +for developers of libraries like NCCL or NVSHMEM. It shows how higher-level programming models like NVSHMEM work internally within a (multinode) +NVLINK domain. Application developers generally should use the higher-level MPI, NCCL, or NVSHMEM interfaces instead of this API.

+
+

+14.9.1. Query for Support +

+

Before attempting to use Multicast Objects, applications must ensure that the devices they want to use support them. The following code sample +shows querying for Fabric Memory support:

+
+
int deviceSupportsMultiCast;
+CUresult result = cuDeviceGetAttribute(&deviceSupportsMultiCast, CU_DEVICE_ATTRIBUTE_MULTICAST_SUPPORTED, device);
+if (deviceSupportsMultiCast != 0) {
+    // `device` supports Multicast Objects
+}
+
+
+
+
+

+14.9.2. Allocating Multicast Objects +

+

Multicast Objects can be created with cuMulticastCreate:

+
+
CUmemGenericAllocationHandle createMCHandle(int numDevices, size_t size) {
+    CUmemAllocationProp mcProp = {};
+    mcProp.numDevices = numDevices;
+    mcProp.handleTypes = CU_MEM_HANDLE_TYPE_FABRIC; // or on single node CU_MEM_HANDLE_TYPE_POSIX_FILE_DESCRIPTOR
+
+    size_t granularity = 0;
+    cuMulticastGetGranularity(&granularity, &mcProp, CU_MEM_ALLOC_GRANULARITY_MINIMUM);
+
+    // Ensure size matches granularity requirements for the allocation
+    size_t padded_size = ROUND_UP(size, granularity);
+
+    mcProp.size = padded_size;
+
+    // Create Multicast Object this has no devices and no physical memory associated yet
+    CUmemGenericAllocationHandle mcHandle;
+    cuMulticastCreate(&mcHandle, &mcProp);
+
+    return mcHandle;
+}
+
+
+
+
+

+14.9.3. Add Devices to Multicast Objects +

+

Devices can be added to a Multicast Team with cuMulticastAddDevice:

+
+
cuMulticastAddDevice(&mcHandle, device);
+
+
+

This step needs to be completed on all processes controlling devices that should participate in a Multicast Team before memory on any device is +bound to the Multicast Object.

+
+
+

+14.9.4. Bind Memory to Multicast Objects +

+

After a Multicast Object has been created and all participating devices have been added to the Multicast Object it needs to be backed with +physical memory allocated with cuMemCreate for each device:

+
+
cuMulticastBindMem(mcHandle, mcOffset, memHandle, memOffset, size, 0 /*flags*/);
+
+
+
+
+

+14.9.5. Use Multicast Mappings +

+

To use Multicast Mappings in CUDA C++ it is required to use the multimem PTX instructions +with Inline PTX Assembly:

+
+
__global__ void all_reduce_norm_barrier_kernel(float* l2_norm,
+                                               float* partial_l2_norm_mc,
+                                               unsigned int* arrival_counter_uc, unsigned int* arrival_counter_mc,
+                                               const unsigned int expected_count) {
+    assert( 1 == blockDim.x * blockDim.y * blockDim.z * gridDim.x * gridDim.y * gridDim.z );
+    float l2_norm_sum = 0.0;
+#if __CUDA_ARCH__ >= 900
+
+    // atomic reduction to all replicas
+    // this can be conceptually thought of as __threadfence_system(); atomicAdd_system(arrival_counter_mc, 1);
+    asm volatile ("multimem.red.release.sys.global.add.u32 [%0], %1;" :: "l"(arrival_counter_mc), "n"(1) : "memory");
+
+    // Need a fence between Multicast (mc) and Unicast (uc) access to the same memory `arrival_counter_uc` and `arrival_counter_mc`:
+    // - fence.proxy instructions establish an ordering between memory accesses that may happen through different proxies
+    // - Value .alias of the .proxykind qualifier refers to memory accesses performed using virtually aliased addresses to the same memory location.
+    // from https://docs.nvidia.com/cuda/parallel-thread-execution/#parallel-synchronization-and-communication-instructions-membar
+    asm volatile ("fence.proxy.alias;" ::: "memory");
+
+    // spin wait with acquire ordering on UC mapping till all peers have arrived in this iteration
+    // Note: all ranks need to reach another barrier after this kernel, such that it is not possible for the barrier to be unblocked by an
+    // arrival of a rank for the next iteration if some other rank is slow.
+    cuda::atomic_ref<unsigned int,cuda::thread_scope_system> ac(arrival_counter_uc);
+    while (expected_count > ac.load(cuda::memory_order_acquire));
+
+    // Atomic load reduction from all replicas. It does not provide ordering so it can be relaxed.
+    asm volatile ("multimem.ld_reduce.relaxed.sys.global.add.f32 %0, [%1];" : "=f"(l2_norm_sum) : "l"(partial_l2_norm_mc) : "memory");
+
+#else
+    #error "ERROR: multimem instructions require compute capability 9.0 or larger."
+#endif
+
+    *l2_norm = std::sqrt(l2_norm_sum);
+}
+
+
+
+
+
+
+

+15. Stream Ordered Memory Allocator +

+
+

Warning

+

This document has been replaced by a new CUDA Programming Guide. The information in this document should be considered legacy, and this document is no longer being updated as of CUDA 13.0. Please refer to the CUDA Programming Guide for up-to-date information on CUDA.

+
+
+

+15.1. Introduction +

+

Managing memory allocations using cudaMalloc and cudaFree causes GPU to synchronize across all executing CUDA streams. The Stream Order Memory Allocator enables applications to order memory allocation and deallocation with other work launched into a CUDA stream such as kernel launches and asynchronous copies. This improves application memory use by taking advantage of stream-ordering semantics to reuse memory allocations. The allocator also allows applications to control the allocator’s memory caching behavior. When set up with an appropriate release threshold, the caching behavior allows the allocator to avoid expensive calls into the OS when the application indicates it is willing to accept a bigger memory footprint. The allocator also supports the easy and secure sharing of allocations between processes.

+

For many applications, the Stream Ordered Memory Allocator reduces the need for custom memory management abstractions, and makes it easier to create high-performance custom memory management for applications that need it. For applications and libraries that already have custom memory allocators, adopting the Stream Ordered Memory Allocator enables multiple libraries to share a common pool of memory managed by the driver, thus reducing excess memory consumption. Additionally, the driver can perform optimizations based on its awareness of the allocator and other stream management APIs. Finally, Nsight Compute and the Next-Gen CUDA debugger is aware of the allocator as part of their CUDA 11.3 toolkit support.

+
+
+

+15.2. Query for Support +

+

The user can determine whether or not a device supports the stream ordered memory allocator by calling cudaDeviceGetAttribute() with the device attribute cudaDevAttrMemoryPoolsSupported.

+

Starting with CUDA 11.3, IPC memory pool support can be queried with the cudaDevAttrMemoryPoolSupportedHandleTypes device attribute. Previous drivers will return cudaErrorInvalidValue as those drivers are unaware of the attribute enum.

+
+
int driverVersion = 0;
+int deviceSupportsMemoryPools = 0;
+int poolSupportedHandleTypes = 0;
+cudaDriverGetVersion(&driverVersion);
+if (driverVersion >= 11020) {
+    cudaDeviceGetAttribute(&deviceSupportsMemoryPools,
+                           cudaDevAttrMemoryPoolsSupported, device);
+}
+if (deviceSupportsMemoryPools != 0) {
+    // `device` supports the Stream Ordered Memory Allocator
+}
+
+if (driverVersion >= 11030) {
+    cudaDeviceGetAttribute(&poolSupportedHandleTypes,
+              cudaDevAttrMemoryPoolSupportedHandleTypes, device);
+}
+if (poolSupportedHandleTypes & cudaMemHandleTypePosixFileDescriptor) {
+   // Pools on the specified device can be created with posix file descriptor-based IPC
+}
+
+
+

Performing the driver version check before the query avoids hitting a cudaErrorInvalidValue error on drivers where the attribute was not yet defined. One can use cudaGetLastError to clear the error instead of avoiding it.

+
+
+

+15.3. API Fundamentals (cudaMallocAsync and cudaFreeAsync) +

+

The APIs cudaMallocAsync and cudaFreeAsync form the core of the allocator. cudaMallocAsync returns an allocation and cudaFreeAsync frees an allocation. Both APIs accept stream arguments to define when the allocation will become and stop being available for use. The pointer value returned by cudaMallocAsync is determined synchronously and is available for constructing future work. It is important to note that cudaMallocAsync ignores the current device/context when determining where the allocation will reside. Instead, cudaMallocAsync determines the resident device based on the specified memory pool or the supplied stream. The simplest use pattern is when the memory is allocated, used, and freed back into the same stream.

+
+
void *ptr;
+size_t size = 512;
+cudaMallocAsync(&ptr, size, cudaStreamPerThread);
+// do work using the allocation
+kernel<<<..., cudaStreamPerThread>>>(ptr, ...);
+// An asynchronous free can be specified without synchronizing the cpu and GPU
+cudaFreeAsync(ptr, cudaStreamPerThread);
+
+
+

When using an allocation in a stream other than the allocating stream, the user must guarantee that the access will happen after the allocation operation, otherwise the behavior is undefined. The user may make this guarantee either by synchronizing the allocating stream, or by using CUDA events to synchronize the producing and consuming streams.

+

cudaFreeAsync() inserts a free operation into the stream. The user must guarantee that the free operation happens after the allocation operation and any use of the allocation. Also, any use of the allocation after the free operation starts results in undefined behavior. Events and/or stream synchronizing operations should be used to guarantee any access to the allocation on other streams is complete before the freeing stream begins the free operation.

+
+
cudaMallocAsync(&ptr, size, stream1);
+cudaEventRecord(event1, stream1);
+//stream2 must wait for the allocation to be ready before accessing
+cudaStreamWaitEvent(stream2, event1);
+kernel<<<..., stream2>>>(ptr, ...);
+cudaEventRecord(event2, stream2);
+// stream3 must wait for stream2 to finish accessing the allocation before
+// freeing the allocation
+cudaStreamWaitEvent(stream3, event2);
+cudaFreeAsync(ptr, stream3);
+
+
+

The user can free allocations allocated with cudaMalloc() with cudaFreeAsync(). The user must make the same guarantees about accesses being complete before the free operation begins.

+
+
cudaMalloc(&ptr, size);
+kernel<<<..., stream>>>(ptr, ...);
+cudaFreeAsync(ptr, stream);
+
+
+

The user can free memory allocated with cudaMallocAsync with cudaFree(). When freeing such allocations through the cudaFree() API, the driver assumes that all accesses to the allocation are complete and performs no further synchronization. The user can use cudaStreamQuery / cudaStreamSynchronize / cudaEventQuery / cudaEventSynchronize / cudaDeviceSynchronize to guarantee that the appropriate asynchronous work is complete and that the GPU will not try to access the allocation.

+
+
cudaMallocAsync(&ptr, size,stream);
+kernel<<<..., stream>>>(ptr, ...);
+// synchronize is needed to avoid prematurely freeing the memory
+cudaStreamSynchronize(stream);
+cudaFree(ptr);
+
+
+
+
+

+15.4. Memory Pools and the cudaMemPool_t +

+

Memory pools encapsulate virtual address and physical memory resources that are allocated and managed according to the pools attributes and properties. The primary aspect of a memory pool is the kind and location of memory it manages.

+

All calls to cudaMallocAsync use the resources of a memory pool. In the absence of a specified memory pool, cudaMallocAsync uses the current memory pool of the supplied stream’s device. The current memory pool for a device may be set with cudaDeviceSetMempool and queried with cudaDeviceGetMempool. By default (in the absence of a cudaDeviceSetMempool call), the current memory pool is the default memory pool of a device. The API cudaMallocFromPoolAsync and c++ overloads of cudaMallocAsync allow a user to specify the pool to be used for an allocation without setting it as the current pool. The APIs cudaDeviceGetDefaultMempool and cudaMemPoolCreate give users handles to memory pools.

+
+

Note

+

The mempool current to a device will be local to that device. So allocating without specifying a memory pool will always yield an allocation local to the stream’s device.

+
+
+

Note

+

cudaMemPoolSetAttribute and cudaMemPoolGetAttribute control the attributes of the memory pools.

+
+
+
+

+15.5. Default/Implicit Pools +

+

The default memory pool of a device may be retrieved with the cudaDeviceGetDefaultMempool API. Allocations from the default memory pool of a device are non-migratable device allocation located on that device. These allocations will always be accessible from that device. The accessibility of the default memory pool may be modified with cudaMemPoolSetAccess and queried by cudaMemPoolGetAccess. Since the default pools do not need to be explicitly created, they are sometimes referred to as implicit pools. The default memory pool of a device does not support IPC.

+
+
+

+15.6. Explicit Pools +

+

The API cudaMemPoolCreate creates an explicit pool. This allows applications to request properties for their allocation beyond what is provided by the default/implict pools. These include properties such as IPC capability, maximum pool size, allocations resident on a specific CPU NUMA node on supported platforms etc.

+
+
// create a pool similar to the implicit pool on device 0
+int device = 0;
+cudaMemPoolProps poolProps = { };
+poolProps.allocType = cudaMemAllocationTypePinned;
+poolProps.location.id = device;
+poolProps.location.type = cudaMemLocationTypeDevice;
+
+cudaMemPoolCreate(&memPool, &poolProps));
+
+
+

The following code snippet illustrates an example of creating an IPC capable memory pool on a valid CPU NUMA node.

+
+
// create a pool resident on a CPU NUMA node that is capable of IPC sharing (via a file descriptor).
+int cpu_numa_id = 0;
+cudaMemPoolProps poolProps = { };
+poolProps.allocType = cudaMemAllocationTypePinned;
+poolProps.location.id = cpu_numa_id;
+poolProps.location.type = cudaMemLocationTypeHostNuma;
+poolProps.handleType = cudaMemHandleTypePosixFileDescriptor;
+
+cudaMemPoolCreate(&ipcMemPool, &poolProps));
+
+
+
+
+

+15.7. Physical Page Caching Behavior +

+

By default, the allocator tries to minimize the physical memory owned by a pool. To minimize the OS calls to allocate and free physical memory, applications must configure a memory footprint for each pool. Applications can do this with the release threshold attribute (cudaMemPoolAttrReleaseThreshold).

+

The release threshold is the amount of memory in bytes a pool should hold onto before trying to release memory back to the OS. When more than the release threshold bytes of memory are held by the memory pool, the allocator will try to release memory back to the OS on the next call to stream, event or device synchronize. Setting the release threshold to UINT64_MAX will prevent the driver from attempting to shrink the pool after every synchronization.

+
+
Cuuint64_t setVal = UINT64_MAX;
+cudaMemPoolSetAttribute(memPool, cudaMemPoolAttrReleaseThreshold, &setVal);
+
+
+

Applications that set cudaMemPoolAttrReleaseThreshold high enough to effectively disable memory pool shrinking may wish to explicitly shrink a memory pool’s memory footprint. cudaMemPoolTrimTo allows such applications to do so. When trimming a memory pool’s footprint, the minBytesToKeep parameter allows an application to hold onto an amount of memory it expects to need in a subsequent phase of execution.

+
+
Cuuint64_t setVal = UINT64_MAX;
+cudaMemPoolSetAttribute(memPool, cudaMemPoolAttrReleaseThreshold, &setVal);
+
+// application phase needing a lot of memory from the stream ordered allocator
+for (i=0; i<10; i++) {
+    for (j=0; j<10; j++) {
+        cudaMallocAsync(&ptrs[j],size[j], stream);
+    }
+    kernel<<<...,stream>>>(ptrs,...);
+    for (j=0; j<10; j++) {
+        cudaFreeAsync(ptrs[j], stream);
+    }
+}
+
+// Process does not need as much memory for the next phase.
+// Synchronize so that the trim operation will know that the allocations are no
+// longer in use.
+cudaStreamSynchronize(stream);
+cudaMemPoolTrimTo(mempool, 0);
+
+// Some other process/allocation mechanism can now use the physical memory
+// released by the trimming operation.
+
+
+
+
+

+15.8. Resource Usage Statistics +

+

In CUDA 11.3, the pool attributes cudaMemPoolAttrReservedMemCurrent, cudaMemPoolAttrReservedMemHigh, cudaMemPoolAttrUsedMemCurrent, and cudaMemPoolAttrUsedMemHigh were added to query the memory usage of a pool.

+

Querying the cudaMemPoolAttrReservedMemCurrent attribute of a pool reports the current total physical GPU memory consumed by the pool. Querying the cudaMemPoolAttrUsedMemCurrent of a pool returns the total size of all of the memory allocated from the pool and not available for reuse.

+

ThecudaMemPoolAttr*MemHigh attributes are watermarks recording the max value achieved by the respective cudaMemPoolAttr*MemCurrent attribute since last reset. They can be reset to the current value by using the cudaMemPoolSetAttribute API.

+
+
// sample helper functions for getting the usage statistics in bulk
+struct usageStatistics {
+    cuuint64_t reserved;
+    cuuint64_t reservedHigh;
+    cuuint64_t used;
+    cuuint64_t usedHigh;
+};
+
+void getUsageStatistics(cudaMemoryPool_t memPool, struct usageStatistics *statistics)
+{
+    cudaMemPoolGetAttribute(memPool, cudaMemPoolAttrReservedMemCurrent, statistics->reserved);
+    cudaMemPoolGetAttribute(memPool, cudaMemPoolAttrReservedMemHigh, statistics->reservedHigh);
+    cudaMemPoolGetAttribute(memPool, cudaMemPoolAttrUsedMemCurrent, statistics->used);
+    cudaMemPoolGetAttribute(memPool, cudaMemPoolAttrUsedMemHigh, statistics->usedHigh);
+}
+
+
+// resetting the watermarks will make them take on the current value.
+void resetStatistics(cudaMemoryPool_t memPool)
+{
+    cuuint64_t value = 0;
+    cudaMemPoolSetAttribute(memPool, cudaMemPoolAttrReservedMemHigh, &value);
+    cudaMemPoolSetAttribute(memPool, cudaMemPoolAttrUsedMemHigh, &value);
+}
+
+
+
+
+

+15.9. Memory Reuse Policies +

+

In order to service an allocation request, the driver attempts to reuse memory that was previously freed via cudaFreeAsync() before attempting to allocate more memory from the OS. For example, memory freed in a stream can immediately be reused for a subsequent allocation request in the same stream. Similarly, when a stream is synchronized with the CPU, the memory that was previously freed in that stream becomes available for reuse for an allocation in any stream.

+

The stream ordered allocator has a few controllable allocation policies. The pool attributes cudaMemPoolReuseFollowEventDependencies, cudaMemPoolReuseAllowOpportunistic, and cudaMemPoolReuseAllowInternalDependencies control these policies. Upgrading to a newer CUDA driver may change, enhance, augment and/or reorder the reuse policies.

+
+

+15.9.1. cudaMemPoolReuseFollowEventDependencies +

+

Before allocating more physical GPU memory, the allocator examines dependency information established by CUDA events and tries to allocate from memory freed in another stream.

+
+
cudaMallocAsync(&ptr, size, originalStream);
+kernel<<<..., originalStream>>>(ptr, ...);
+cudaFreeAsync(ptr, originalStream);
+cudaEventRecord(event,originalStream);
+
+// waiting on the event that captures the free in another stream
+// allows the allocator to reuse the memory to satisfy
+// a new allocation request in the other stream when
+// cudaMemPoolReuseFollowEventDependencies is enabled.
+cudaStreamWaitEvent(otherStream, event);
+cudaMallocAsync(&ptr2, size, otherStream);
+
+
+
+
+

+15.9.2. cudaMemPoolReuseAllowOpportunistic +

+

According to the cudaMemPoolReuseAllowOpportunistic policy, the allocator examines freed allocations to see if the free’s stream order semantic has been met (such as the stream has passed the point of execution indicated by the free). When this is disabled, the allocator will still reuse memory made available when a stream is synchronized with the CPU. Disabling this policy does not stop the cudaMemPoolReuseFollowEventDependencies from applying.

+
+
cudaMallocAsync(&ptr, size, originalStream);
+kernel<<<..., originalStream>>>(ptr, ...);
+cudaFreeAsync(ptr, originalStream);
+
+
+// after some time, the kernel finishes running
+wait(10);
+
+// When cudaMemPoolReuseAllowOpportunistic is enabled this allocation request
+// can be fulfilled with the prior allocation based on the progress of originalStream.
+cudaMallocAsync(&ptr2, size, otherStream);
+
+
+
+
+

+15.9.3. cudaMemPoolReuseAllowInternalDependencies +

+

Failing to allocate and map more physical memory from the OS, the driver will look for memory whose availability depends on another stream’s pending progress. If such memory is found, the driver will insert the required dependency into the allocating stream and reuse the memory.

+
+
cudaMallocAsync(&ptr, size, originalStream);
+kernel<<<..., originalStream>>>(ptr, ...);
+cudaFreeAsync(ptr, originalStream);
+
+// When cudaMemPoolReuseAllowInternalDependencies is enabled
+// and the driver fails to allocate more physical memory, the driver may
+// effectively perform a cudaStreamWaitEvent in the allocating stream
+// to make sure that future work in ‘otherStream’ happens after the work
+// in the original stream that would be allowed to access the original allocation.
+cudaMallocAsync(&ptr2, size, otherStream);
+
+
+
+
+

+15.9.4. Disabling Reuse Policies +

+

While the controllable reuse policies improve memory reuse, users may want to disable them. Allowing opportunistic reuse (such as cudaMemPoolReuseAllowOpportunistic) introduces run to run variance in allocation patterns based on the interleaving of CPU and GPU execution. Internal dependency insertion (such as cudaMemPoolReuseAllowInternalDependencies) can serialize work in unexpected and potentially non-deterministic ways when the user would rather explicitly synchronize an event or stream on allocation failure.

+
+
+
+

+15.10. Device Accessibility for Multi-GPU Support +

+

Just like allocation accessibility controlled through the virtual memory management APIs, memory pool allocation accessibility does not follow cudaDeviceEnablePeerAccess or cuCtxEnablePeerAccess. Instead, the API cudaMemPoolSetAccess modifies what devices can access allocations from a pool. By default, allocations are accessible from the device where the allocations are located. This access cannot be revoked. To enable access from other devices, the accessing device must be peer capable with the memory pool’s device; check with cudaDeviceCanAccessPeer. If the peer capability is not checked, the set access may fail with cudaErrorInvalidDevice. If no allocations had been made from the pool, the cudaMemPoolSetAccess call may succeed even when the devices are not peer capable; in this case, the next allocation from the pool will fail.

+

It is worth noting that cudaMemPoolSetAccess affects all allocations from the memory pool, not just future ones. Also the accessibility reported by cudaMemPoolGetAccess applies to all allocations from the pool, not just future ones. It is recommended that the accessibility settings of a pool for a given GPU not be changed frequently; once a pool is made accessible from a given GPU, it should remain accessible from that GPU for the lifetime of the pool.

+
+
// snippet showing usage of cudaMemPoolSetAccess:
+cudaError_t setAccessOnDevice(cudaMemPool_t memPool, int residentDevice,
+              int accessingDevice) {
+    cudaMemAccessDesc accessDesc = {};
+    accessDesc.location.type = cudaMemLocationTypeDevice;
+    accessDesc.location.id = accessingDevice;
+    accessDesc.flags = cudaMemAccessFlagsProtReadWrite;
+
+    int canAccess = 0;
+    cudaError_t error = cudaDeviceCanAccessPeer(&canAccess, accessingDevice,
+              residentDevice);
+    if (error != cudaSuccess) {
+        return error;
+    } else if (canAccess == 0) {
+        return cudaErrorPeerAccessUnsupported;
+    }
+
+    // Make the address accessible
+    return cudaMemPoolSetAccess(memPool, &accessDesc, 1);
+}
+
+
+
+
+

+15.11. IPC Memory Pools +

+

IPC capable memory pools allow easy, efficient and secure sharing of GPU memory between processes. CUDA’s IPC memory pools provide the same security benefits as CUDA’s virtual memory management APIs.

+

There are two phases to sharing memory between processes with memory pools. The processes first need to share access to the pool, then share specific allocations from that pool. The first phase establishes and enforces security. The second phase coordinates what virtual addresses are used in each process and when mappings need to be valid in the importing process.

+
+

+15.11.1. Creating and Sharing IPC Memory Pools +

+

Sharing access to a pool involves retrieving an OS native handle to the pool (with the cudaMemPoolExportToShareableHandle() API), transferring the handle to the importing process using the usual OS native IPC mechanisms, and creating an imported memory pool (with the cudaMemPoolImportFromShareableHandle() API). For cudaMemPoolExportToShareableHandle to succeed, the memory pool had to be created with the requested handle type specified in the pool properties structure. Please reference samples for the appropriate IPC mechanisms to transfer the OS native handle between processes. The rest of the procedure can be found in the following code snippets.

+
+
// in exporting process
+// create an exportable IPC capable pool on device 0
+cudaMemPoolProps poolProps = { };
+poolProps.allocType = cudaMemAllocationTypePinned;
+poolProps.location.id = 0;
+poolProps.location.type = cudaMemLocationTypeDevice;
+
+// Setting handleTypes to a non zero value will make the pool exportable (IPC capable)
+poolProps.handleTypes = CU_MEM_HANDLE_TYPE_POSIX_FILE_DESCRIPTOR;
+
+cudaMemPoolCreate(&memPool, &poolProps));
+
+// FD based handles are integer types
+int fdHandle = 0;
+
+
+// Retrieve an OS native handle to the pool.
+// Note that a pointer to the handle memory is passed in here.
+cudaMemPoolExportToShareableHandle(&fdHandle,
+             memPool,
+             CU_MEM_HANDLE_TYPE_POSIX_FILE_DESCRIPTOR,
+             0);
+
+// The handle must be sent to the importing process with the appropriate
+// OS specific APIs.
+
+
+
+
// in importing process
+ int fdHandle;
+// The handle needs to be retrieved from the exporting process with the
+// appropriate OS specific APIs.
+// Create an imported pool from the shareable handle.
+// Note that the handle is passed by value here.
+cudaMemPoolImportFromShareableHandle(&importedMemPool,
+          (void*)fdHandle,
+          CU_MEM_HANDLE_TYPE_POSIX_FILE_DESCRIPTOR,
+          0);
+
+
+
+
+

+15.11.2. Set Access in the Importing Process +

+

Imported memory pools are initially only accessible from their resident device. The imported memory pool does not inherit any accessibility set by the exporting process. The importing process needs to enable access (with cudaMemPoolSetAccess) from any GPU it plans to access the memory from.

+

If the imported memory pool belongs to a non-visible device in the importing process, the user must use the cudaMemPoolSetAccess API to enable access from the GPUs the allocations will be used on.

+
+
+

+15.11.3. Creating and Sharing Allocations from an Exported Pool +

+

Once the pool has been shared, allocations made with cudaMallocAsync() from the pool in the exporting process can be shared with other processes that have imported the pool. Since the pool’s security policy is established and verified at the pool level, the OS does not need extra bookkeeping to provide security for specific pool allocations; In other words, the opaque cudaMemPoolPtrExportData required to import a pool allocation may be sent to the importing process using any mechanism.

+

While allocations may be exported and even imported without synchronizing with the allocating stream in any way, the importing process must follow the same rules as the exporting process when accessing the allocation. Namely, access to the allocation must happen after the stream ordering of the allocation operation in the allocating stream. The two following code snippets show cudaMemPoolExportPointer() and cudaMemPoolImportPointer() sharing the allocation with an IPC event used to guarantee that the allocation isn’t accessed in the importing process before the allocation is ready.

+
+
// preparing an allocation in the exporting process
+cudaMemPoolPtrExportData exportData;
+cudaEvent_t readyIpcEvent;
+cudaIpcEventHandle_t readyIpcEventHandle;
+
+// ipc event for coordinating between processes
+// cudaEventInterprocess flag makes the event an ipc event
+// cudaEventDisableTiming  is set for performance reasons
+
+cudaEventCreate(
+        &readyIpcEvent, cudaEventDisableTiming | cudaEventInterprocess)
+
+// allocate from the exporting mem pool
+cudaMallocAsync(&ptr, size,exportMemPool, stream);
+
+// event for sharing when the allocation is ready.
+cudaEventRecord(readyIpcEvent, stream);
+cudaMemPoolExportPointer(&exportData, ptr);
+cudaIpcGetEventHandle(&readyIpcEventHandle, readyIpcEvent);
+
+// Share IPC event and pointer export data with the importing process using
+//  any mechanism. Here we copy the data into shared memory
+shmem->ptrData = exportData;
+shmem->readyIpcEventHandle = readyIpcEventHandle;
+// signal consumers data is ready
+
+
+
+
// Importing an allocation
+cudaMemPoolPtrExportData *importData = &shmem->prtData;
+cudaEvent_t readyIpcEvent;
+cudaIpcEventHandle_t *readyIpcEventHandle = &shmem->readyIpcEventHandle;
+
+// Need to retrieve the ipc event handle and the export data from the
+// exporting process using any mechanism.  Here we are using shmem and just
+// need synchronization to make sure the shared memory is filled in.
+
+cudaIpcOpenEventHandle(&readyIpcEvent, readyIpcEventHandle);
+
+// import the allocation. The operation does not block on the allocation being ready.
+cudaMemPoolImportPointer(&ptr, importedMemPool, importData);
+
+// Wait for the prior stream operations in the allocating stream to complete before
+// using the allocation in the importing process.
+cudaStreamWaitEvent(stream, readyIpcEvent);
+kernel<<<..., stream>>>(ptr, ...);
+
+
+

When freeing the allocation, the allocation needs to be freed in the importing process before it is freed in the exporting process. The following code snippet demonstrates the use of CUDA IPC events to provide the required synchronization between the cudaFreeAsync operations in both processes. Access to the allocation from the importing process is obviously restricted by the free operation in the importing process side. It is worth noting that cudaFree can be used to free the allocation in both processes and that other stream synchronization APIs may be used instead of CUDA IPC events.

+
+
// The free must happen in importing process before the exporting process
+kernel<<<..., stream>>>(ptr, ...);
+
+// Last access in importing process
+cudaFreeAsync(ptr, stream);
+
+// Access not allowed in the importing process after the free
+cudaIpcEventRecord(finishedIpcEvent, stream);
+
+
+
+
// Exporting process
+// The exporting process needs to coordinate its free with the stream order
+// of the importing process’s free.
+cudaStreamWaitEvent(stream, finishedIpcEvent);
+kernel<<<..., stream>>>(ptrInExportingProcess, ...);
+
+// The free in the importing process doesn’t stop the exporting process
+// from using the allocation.
+cudFreeAsync(ptrInExportingProcess,stream);
+
+
+
+
+

+15.11.4. IPC Export Pool Limitations +

+

IPC pools currently do not support releasing physical blocks back to the OS. As a result the cudaMemPoolTrimTo API acts as a no-op and the cudaMemPoolAttrReleaseThreshold effectively gets ignored. This behavior is controlled by the driver, not the runtime and may change in a future driver update.

+
+
+

+15.11.5. IPC Import Pool Limitations +

+

Allocating from an import pool is not allowed; specifically, import pools cannot be set current and cannot be used in the cudaMallocFromPoolAsync API. As such, the allocation reuse policy attributes are meaningless for these pools.

+

IPC pools currently do not support releasing physical blocks back to the OS. As a result the cudaMemPoolTrimTo API acts as a no-op and the cudaMemPoolAttrReleaseThreshold effectively gets ignored.

+

The resource usage stat attribute queries only reflect the allocations imported into the process and the associated physical memory.

+
+
+
+

+15.12. Synchronization API Actions +

+

One of the optimizations that comes with the allocator being part of the CUDA driver is integration with the synchronize APIs. When the user requests that the CUDA driver synchronize, the driver waits for asynchronous work to complete. Before returning, the driver will determine what frees the synchronization guaranteed to be completed. These allocations are made available for allocation regardless of specified stream or disabled allocation policies. The driver also checks cudaMemPoolAttrReleaseThreshold here and releases any excess physical memory that it can.

+
+
+

+15.13. Addendums +

+
+

+15.13.1. cudaMemcpyAsync Current Context/Device Sensitivity +

+

In the current CUDA driver, any async memcpy involving memory from cudaMallocAsync should be done using the specified stream’s context as the calling thread’s current context. This is not necessary for cudaMemcpyPeerAsync, as the device primary contexts specified in the API are referenced instead of the current context.

+
+
+

+15.13.2. cuPointerGetAttribute Query +

+

Invoking cuPointerGetAttribute on an allocation after invoking cudaFreeAsync on it results in undefined behavior. Specifically, it does not matter if an allocation is still accessible from a given stream: the behavior is still undefined.

+
+
+

+15.13.3. cuGraphAddMemsetNode +

+

cuGraphAddMemsetNode does not work with memory allocated via the stream ordered allocator. However, memsets of the allocations can be stream captured.

+
+
+

+15.13.4. Pointer Attributes +

+

The cuPointerGetAttributes query works on stream ordered allocations. Since stream ordered allocations are not context associated, querying CU_POINTER_ATTRIBUTE_CONTEXT will succeed but return NULL in *data. The attribute CU_POINTER_ATTRIBUTE_DEVICE_ORDINAL can be used to determine the location of the allocation: this can be useful when selecting a context for making p2h2p copies using cudaMemcpyPeerAsync. The attribute CU_POINTER_ATTRIBUTE_MEMPOOL_HANDLE was added in CUDA 11.3 and can be useful for debugging and for confirming which pool an allocation comes from before doing IPC.

+
+
+

+15.13.5. CPU Virtual Memory +

+

When using CUDA stream-ordered memory allocator APIs, avoid setting VRAM limitations with “ulimit -v” as this is not supported.

+
+
+
+
+

+16. Graph Memory Nodes +

+
+

Warning

+

This document has been replaced by a new CUDA Programming Guide. The information in this document should be considered legacy, and this document is no longer being updated as of CUDA 13.0. Please refer to the CUDA Programming Guide for up-to-date information on CUDA.

+
+
+

+16.1. Introduction +

+

Graph memory nodes allow graphs to create and own memory allocations. Graph memory nodes have GPU ordered lifetime semantics, which dictate when memory is allowed to be accessed on the device. These GPU ordered lifetime semantics enable driver-managed memory reuse, and match those of the stream ordered allocation APIs cudaMallocAsync and cudaFreeAsync, which may be captured when creating a graph.

+

Graph allocations have fixed addresses over the life of a graph including repeated instantiations and launches. This allows the memory to be directly referenced by other operations within the graph without the need of a graph update, even when CUDA changes the backing physical memory. Within a graph, allocations whose graph ordered lifetimes do not overlap may use the same underlying physical memory.

+

CUDA may reuse the same physical memory for allocations across multiple graphs, aliasing virtual address mappings according to the GPU ordered lifetime semantics. For example when different graphs are launched into the same stream, CUDA may virtually alias the same physical memory to satisfy the needs of allocations which have single-graph lifetimes.

+
+
+

+16.2. Support and Compatibility +

+

Graph memory nodes require an 11.4 capable CUDA driver and support for the stream ordered allocator on the GPU. The following snippet shows how to check for support on a given device.

+
+
int driverVersion = 0;
+int deviceSupportsMemoryPools = 0;
+int deviceSupportsMemoryNodes = 0;
+cudaDriverGetVersion(&driverVersion);
+if (driverVersion >= 11020) { // avoid invalid value error in cudaDeviceGetAttribute
+    cudaDeviceGetAttribute(&deviceSupportsMemoryPools, cudaDevAttrMemoryPoolsSupported, device);
+}
+deviceSupportsMemoryNodes = (driverVersion >= 11040) && (deviceSupportsMemoryPools != 0);
+
+
+

Doing the attribute query inside the driver version check avoids an invalid value return code on 11.0 and 11.1 drivers. Be aware that the compute sanitizer emits warnings when it detects CUDA returning error codes, and a version check before reading the attribute will avoid this. Graph memory nodes are only supported on driver versions 11.4 and newer.

+
+
+

+16.3. API Fundamentals +

+

Graph memory nodes are graph nodes representing either memory allocation or free actions. As a shorthand, nodes that allocate memory are called allocation nodes. Likewise, nodes that free memory are called free nodes. Allocations created by allocation nodes are called graph allocations. CUDA assigns virtual addresses for the graph allocation at node creation time. While these virtual addresses are fixed for the lifetime of the allocation node, the allocation contents are not persistent past the freeing operation and may be overwritten by accesses referring to a different allocation.

+

Graph allocations are considered recreated every time a graph runs. A graph allocation’s lifetime, which differs from the node’s lifetime, begins when GPU execution reaches the allocating graph node and ends when one of the following occurs:

+
    +
  • GPU execution reaches the freeing graph node

  • +
  • GPU execution reaches the freeing cudaFreeAsync() stream call

  • +
  • immediately upon the freeing call to cudaFree()

  • +
+
+

Note

+

Graph destruction does not automatically free any live graph-allocated memory, even though it ends the lifetime of the allocation node. The allocation must subsequently be freed in another graph, or using cudaFreeAsync()/cudaFree().

+
+

Just like other Graph Structure, graph memory nodes are ordered within a graph by dependency edges. A program must guarantee that operations accessing graph memory:

+
    +
  • are ordered after the allocation node

  • +
  • are ordered before the operation freeing the memory

  • +
+

Graph allocation lifetimes begin and usually end according to GPU execution (as opposed to API invocation). GPU ordering is the order that work runs on the GPU as opposed to the order that the work is enqueued or described. Thus, graph allocations are considered ‘GPU ordered.’

+
+

+16.3.1. Graph Node APIs +

+

Graph memory nodes may be explicitly created with the memory node creation APIs, cudaGraphAddMemAllocNode and cudaGraphAddMemFreeNode. The address allocated by cudaGraphAddMemAllocNode is returned to the user in the dptr field of the passed CUDA_MEM_ALLOC_NODE_PARAMS structure. All operations using graph allocations inside the allocating graph must be ordered after the allocating node. Similarly, any free nodes must be ordered after all uses of the allocation within the graph. cudaGraphAddMemFreeNode creates free nodes.

+

In the following figure, there is an example graph with an alloc and a free node. Kernel nodes a, b, and c are ordered after the allocation node and before the free node such that the kernels can access the allocation. Kernel node e is not ordered after the alloc node and therefore cannot safely access the memory. Kernel node d is not ordered before the free node, therefore it cannot safely access the memory.

+
+Kernel Nodes +
+

Figure 32 Kernel Nodes

+
+
+

The following code snippet establishes the graph in this figure:

+
+
// Create the graph - it starts out empty
+cudaGraphCreate(&graph, 0);
+
+// parameters for a basic allocation
+cudaMemAllocNodeParams params = {};
+params.poolProps.allocType = cudaMemAllocationTypePinned;
+params.poolProps.location.type = cudaMemLocationTypeDevice;
+// specify device 0 as the resident device
+params.poolProps.location.id = 0;
+params.bytesize = size;
+
+cudaGraphAddMemAllocNode(&allocNode, graph, NULL, 0, &params);
+nodeParams->kernelParams[0] = params.dptr;
+cudaGraphAddKernelNode(&a, graph, &allocNode, 1, &nodeParams);
+cudaGraphAddKernelNode(&b, graph, &a, 1, &nodeParams);
+cudaGraphAddKernelNode(&c, graph, &a, 1, &nodeParams);
+cudaGraphNode_t dependencies[2];
+// kernel nodes b and c are using the graph allocation, so the freeing node must depend on them.  Since the dependency of node b on node a establishes an indirect dependency, the free node does not need to explicitly depend on node a.
+dependencies[0] = b;
+dependencies[1] = c;
+cudaGraphAddMemFreeNode(&freeNode, graph, dependencies, 2, params.dptr);
+// free node does not depend on kernel node d, so it must not access the freed graph allocation.
+cudaGraphAddKernelNode(&d, graph, &c, 1, &nodeParams);
+
+// node e does not depend on the allocation node, so it must not access the allocation.  This would be true even if the freeNode depended on kernel node e.
+cudaGraphAddKernelNode(&e, graph, NULL, 0, &nodeParams);
+
+
+
+
+

+16.3.2. Stream Capture +

+

Graph memory nodes can be created by capturing the corresponding stream ordered allocation and free calls cudaMallocAsync and cudaFreeAsync. In this case, the virtual addresses returned by the captured allocation API can be used by other operations inside the graph. Since the stream ordered dependencies will be captured into the graph, the ordering requirements of the stream ordered allocation APIs guarantee that the graph memory nodes will be properly ordered with respect to the captured stream operations (for correctly written stream code).

+

Ignoring kernel nodes d and e, for clarity, the following code snippet shows how to use stream capture to create the graph from the previous figure:

+
+
cudaMallocAsync(&dptr, size, stream1);
+kernel_A<<< ..., stream1 >>>(dptr, ...);
+
+// Fork into stream2
+cudaEventRecord(event1, stream1);
+cudaStreamWaitEvent(stream2, event1);
+
+kernel_B<<< ..., stream1 >>>(dptr, ...);
+// event dependencies translated into graph dependencies, so the kernel node created by the capture of kernel C will depend on the allocation node created by capturing the cudaMallocAsync call.
+kernel_C<<< ..., stream2 >>>(dptr, ...);
+
+// Join stream2 back to origin stream (stream1)
+cudaEventRecord(event2, stream2);
+cudaStreamWaitEvent(stream1, event2);
+
+// Free depends on all work accessing the memory.
+cudaFreeAsync(dptr, stream1);
+
+// End capture in the origin stream
+cudaStreamEndCapture(stream1, &graph);
+
+
+
+
+

+16.3.3. Accessing and Freeing Graph Memory Outside of the Allocating Graph +

+

Graph allocations do not have to be freed by the allocating graph. When a graph does not free an allocation, that allocation persists beyond the execution of the graph and can be accessed by subsequent CUDA operations. These allocations may be accessed in another graph or directly using a stream operation as long as the accessing operation is ordered after the allocation through CUDA events and other stream ordering mechanisms. An allocation may subsequently be freed by regular calls to cudaFree, cudaFreeAsync, or by the launch of another graph with a corresponding free node, or a subsequent launch of the allocating graph (if it was instantiated with the cudaGraphInstantiateFlagAutoFreeOnLaunch flag). It is illegal to access memory after it has been freed - the free operation must be ordered after all operations accessing the memory using graph dependencies, CUDA events, and other stream ordering mechanisms.

+
+

Note

+

Because graph allocations may share underlying physical memory with each other, the Virtual Aliasing Support rules relating to consistency and coherency must be considered. Simply put, the free operation must be ordered after the full device operation (for example, compute kernel / memcpy) completes. Specifically, out of band synchronization - for example a handshake through memory as part of a compute kernel that accesses the graph-allocated memory - is not sufficient for providing ordering guarantees between the memory writes to graph memory and the free operation of that graph memory.

+
+

The following code snippets demonstrate accessing graph allocations outside of the allocating graph with ordering properly established by: using a single stream, using events between streams, and using events baked into the allocating and freeing graph.

+

Ordering established by using a single stream:

+
+
void *dptr;
+cudaGraphAddMemAllocNode(&allocNode, allocGraph, NULL, 0, &params);
+dptr = params.dptr;
+
+cudaGraphInstantiate(&allocGraphExec, allocGraph, NULL, NULL, 0);
+
+cudaGraphLaunch(allocGraphExec, stream);
+kernel<<< …, stream >>>(dptr, …);
+cudaFreeAsync(dptr, stream);
+
+
+

Ordering established by recording and waiting on CUDA events:

+
+
void *dptr;
+
+// Contents of allocating graph
+cudaGraphAddMemAllocNode(&allocNode, allocGraph, NULL, 0, &params);
+dptr = params.dptr;
+
+// contents of consuming/freeing graph
+nodeParams->kernelParams[0] = params.dptr;
+cudaGraphAddKernelNode(&a, graph, NULL, 0, &nodeParams);
+cudaGraphAddMemFreeNode(&freeNode, freeGraph, &a, 1, dptr);
+
+cudaGraphInstantiate(&allocGraphExec, allocGraph, NULL, NULL, 0);
+cudaGraphInstantiate(&freeGraphExec, freeGraph, NULL, NULL, 0);
+
+cudaGraphLaunch(allocGraphExec, allocStream);
+
+// establish the dependency of stream2 on the allocation node
+// note: the dependency could also have been established with a stream synchronize operation
+cudaEventRecord(allocEvent, allocStream)
+cudaStreamWaitEvent(stream2, allocEvent);
+
+kernel<<< …, stream2 >>> (dptr, …);
+
+// establish the dependency between the stream 3 and the allocation use
+cudaStreamRecordEvent(streamUseDoneEvent, stream2);
+cudaStreamWaitEvent(stream3, streamUseDoneEvent);
+
+// it is now safe to launch the freeing graph, which may also access the memory
+cudaGraphLaunch(freeGraphExec, stream3);
+
+
+

Ordering established by using graph external event nodes:

+
+
void *dptr;
+cudaEvent_t allocEvent; // event indicating when the allocation will be ready for use.
+cudaEvent_t streamUseDoneEvent; // event indicating when the stream operations are done with the allocation.
+
+// Contents of allocating graph with event record node
+cudaGraphAddMemAllocNode(&allocNode, allocGraph, NULL, 0, &params);
+dptr = params.dptr;
+// note: this event record node depends on the alloc node
+cudaGraphAddEventRecordNode(&recordNode, allocGraph, &allocNode, 1, allocEvent);
+cudaGraphInstantiate(&allocGraphExec, allocGraph, NULL, NULL, 0);
+
+// contents of consuming/freeing graph with event wait nodes
+cudaGraphAddEventWaitNode(&streamUseDoneEventNode, waitAndFreeGraph, NULL, 0, streamUseDoneEvent);
+cudaGraphAddEventWaitNode(&allocReadyEventNode, waitAndFreeGraph, NULL, 0, allocEvent);
+nodeParams->kernelParams[0] = params.dptr;
+
+// The allocReadyEventNode provides ordering with the alloc node for use in a consuming graph.
+cudaGraphAddKernelNode(&kernelNode, waitAndFreeGraph, &allocReadyEventNode, 1, &nodeParams);
+
+// The free node has to be ordered after both external and internal users.
+// Thus the node must depend on both the kernelNode and the
+// streamUseDoneEventNode.
+dependencies[0] = kernelNode;
+dependencies[1] = streamUseDoneEventNode;
+cudaGraphAddMemFreeNode(&freeNode, waitAndFreeGraph, &dependencies, 2, dptr);
+cudaGraphInstantiate(&waitAndFreeGraphExec, waitAndFreeGraph, NULL, NULL, 0);
+
+cudaGraphLaunch(allocGraphExec, allocStream);
+
+// establish the dependency of stream2 on the event node satisfies the ordering requirement
+cudaStreamWaitEvent(stream2, allocEvent);
+kernel<<< …, stream2 >>> (dptr, …);
+cudaStreamRecordEvent(streamUseDoneEvent, stream2);
+
+// the event wait node in the waitAndFreeGraphExec establishes the dependency on the “readyForFreeEvent” that is needed to prevent the kernel running in stream two from accessing the allocation after the free node in execution order.
+cudaGraphLaunch(waitAndFreeGraphExec, stream3);
+
+
+
+
+

+16.3.4. cudaGraphInstantiateFlagAutoFreeOnLaunch +

+

Under normal circumstances, CUDA will prevent a graph from being relaunched if it has unfreed memory allocations because multiple allocations at the same address will leak memory. Instantiating a graph with the cudaGraphInstantiateFlagAutoFreeOnLaunch flag allows the graph to be relaunched while it still has unfreed allocations. In this case, the launch automatically inserts an asynchronous free of the unfreed allocations.

+

Auto free on launch is useful for single-producer multiple-consumer algorithms. At each iteration, a producer graph creates several allocations, and, depending on runtime conditions, a varying set of consumers accesses those allocations. This type of variable execution sequence means that consumers cannot free the allocations because a subsequent consumer may require access. Auto free on launch means that the launch loop does not need to track the producer’s allocations - instead, that information remains isolated to the producer’s creation and destruction logic. In general, auto free on launch simplifies an algorithm which would otherwise need to free all the allocations owned by a graph before each relaunch.

+
+

Note

+

The cudaGraphInstantiateFlagAutoFreeOnLaunch flag does not change the behavior of graph destruction. The application must explicitly free the unfreed memory in order to avoid memory leaks, even for graphs instantiated with the flag. +The following code shows the use of cudaGraphInstantiateFlagAutoFreeOnLaunch to simplify a single-producer / multiple-consumer algorithm:

+
+
+
// Create producer graph which allocates memory and populates it with data
+cudaStreamBeginCapture(cudaStreamPerThread, cudaStreamCaptureModeGlobal);
+cudaMallocAsync(&data1, blocks * threads, cudaStreamPerThread);
+cudaMallocAsync(&data2, blocks * threads, cudaStreamPerThread);
+produce<<<blocks, threads, 0, cudaStreamPerThread>>>(data1, data2);
+...
+cudaStreamEndCapture(cudaStreamPerThread, &graph);
+cudaGraphInstantiateWithFlags(&producer,
+                              graph,
+                              cudaGraphInstantiateFlagAutoFreeOnLaunch);
+cudaGraphDestroy(graph);
+
+// Create first consumer graph by capturing an asynchronous library call
+cudaStreamBeginCapture(cudaStreamPerThread, cudaStreamCaptureModeGlobal);
+consumerFromLibrary(data1, cudaStreamPerThread);
+cudaStreamEndCapture(cudaStreamPerThread, &graph);
+cudaGraphInstantiateWithFlags(&consumer1, graph, 0); //regular instantiation
+cudaGraphDestroy(graph);
+
+// Create second consumer graph
+cudaStreamBeginCapture(cudaStreamPerThread, cudaStreamCaptureModeGlobal);
+consume2<<<blocks, threads, 0, cudaStreamPerThread>>>(data2);
+...
+cudaStreamEndCapture(cudaStreamPerThread, &graph);
+cudaGraphInstantiateWithFlags(&consumer2, graph, 0);
+cudaGraphDestroy(graph);
+
+// Launch in a loop
+bool launchConsumer2 = false;
+do {
+    cudaGraphLaunch(producer, myStream);
+    cudaGraphLaunch(consumer1, myStream);
+    if (launchConsumer2) {
+        cudaGraphLaunch(consumer2, myStream);
+    }
+} while (determineAction(&launchConsumer2));
+
+cudaFreeAsync(data1, myStream);
+cudaFreeAsync(data2, myStream);
+
+cudaGraphExecDestroy(producer);
+cudaGraphExecDestroy(consumer1);
+cudaGraphExecDestroy(consumer2);
+
+
+
+
+
+

+16.4. Optimized Memory Reuse +

+

CUDA reuses memory in two ways:

+
    +
  • Virtual and physical memory reuse within a graph is based on virtual address assignment, like in the stream ordered allocator.

  • +
  • Physical memory reuse between graphs is done with virtual aliasing: different graphs can map the same physical memory to their unique virtual addresses.

  • +
+
+

+16.4.1. Address Reuse within a Graph +

+

CUDA may reuse memory within a graph by assigning the same virtual address ranges to different allocations whose lifetimes do not overlap. Since virtual addresses may be reused, pointers to different allocations with disjoint lifetimes are not guaranteed to be unique.

+

The following figure shows adding a new allocation node (2) that can reuse the address freed by a dependent node (1).

+
+Adding New Alloc Node 2 +
+

Figure 33 Adding New Alloc Node 2

+
+

The following figure shows adding a new alloc node (4). The new alloc node is not dependent on the free node (2) so cannot reuse the address from the associated alloc node (2). If the alloc node (2) used the address freed by free node (1), the new alloc node 3 would need a new address.

+
+
+
+
+Adding New Alloc Node 3 +
+

Figure 34 Adding New Alloc Node 3

+
+
+
+
+

+16.4.2. Physical Memory Management and Sharing +

+

CUDA is responsible for mapping physical memory to the virtual address before the allocating node is reached in GPU order. As an optimization for memory footprint and mapping overhead, multiple graphs may use the same physical memory for distinct allocations if they will not run simultaneously; however, physical pages cannot be reused if they are bound to more than one executing graph at the same time, or to a graph allocation which remains unfreed.

+

CUDA may update physical memory mappings at any time during graph instantiation, launch, or execution. CUDA may also introduce synchronization between future graph launches in order to prevent live graph allocations from referring to the same physical memory. As for any allocate-free-allocate pattern, if a program accesses a pointer outside of an allocation’s lifetime, the erroneous access may silently read or write live data owned by another allocation (even if the virtual address of the allocation is unique). Use of compute sanitizer tools can catch this error.

+

The following figure shows graphs sequentially launched in the same stream. In this example, each graph frees all the memory it allocates. Since the graphs in the same stream never run concurrently, CUDA can and should use the same physical memory to satisfy all the allocations.

+
+Sequentially Launched Graphs +
+

Figure 35 Sequentially Launched Graphs

+
+
+
+
+
+

+16.5. Performance Considerations +

+

When multiple graphs are launched into the same stream, CUDA attempts to allocate the same physical memory to them because the execution of these graphs cannot overlap. Physical mappings for a graph are retained between launches as an optimization to avoid the cost of remapping. If, at a later time, one of the graphs is launched such that its execution may overlap with the others (for example if it is launched into a different stream) then CUDA must perform some remapping because concurrent graphs require distinct memory to avoid data corruption.

+

In general, remapping of graph memory in CUDA is likely caused by these operations:

+
    +
  • Changing the stream into which a graph is launched

  • +
  • A trim operation on the graph memory pool, which explicitly frees unused memory (discussed in Physical Memory Footprint)

  • +
  • Relaunching a graph while an unfreed allocation from another graph is mapped to the same memory will cause a remap of memory before relaunch

  • +
+

Remapping must happen in execution order, but after any previous execution of that graph is complete (otherwise memory that is still in use could be unmapped). Due to this ordering dependency, as well as because mapping operations are OS calls, mapping operations can be relatively expensive. Applications can avoid this cost by launching graphs containing allocation memory nodes consistently into the same stream.

+
+

+16.5.1. First Launch / cudaGraphUpload +

+

Physical memory cannot be allocated or mapped during graph instantiation because the stream in which the graph will execute is unknown. Mapping is done instead during graph launch. Calling cudaGraphUpload can separate out the cost of allocation from the launch by performing all mappings for that graph immediately and associating the graph with the upload stream. If the graph is then launched into the same stream, it will launch without any additional remapping.

+

Using different streams for graph upload and graph launch behaves similarly to switching streams, likely resulting in remap operations. In addition, unrelated memory pool management is permitted to pull memory from an idle stream, which could negate the impact of the uploads.

+
+
+
+

+16.6. Physical Memory Footprint +

+

The pool-management behavior of asynchronous allocation means that destroying a graph which contains memory nodes (even if their allocations are free) will not immediately return physical memory to the OS for use by other processes. To explicitly release memory back to the OS, an application should use the cudaDeviceGraphMemTrim API.

+

cudaDeviceGraphMemTrim will unmap and release any physical memory reserved by graph memory nodes that is not actively in use. Allocations that have not been freed and graphs that are scheduled or running are considered to be actively using the physical memory and will not be impacted. Use of the trim API will make physical memory available to other allocation APIs and other applications or processes, but will cause CUDA to reallocate and remap memory when the trimmed graphs are next launched. Note that cudaDeviceGraphMemTrim operates on a different pool from cudaMemPoolTrimTo(). The graph memory pool is not exposed to the steam ordered memory allocator. CUDA allows applications to query their graph memory footprint through the cudaDeviceGetGraphMemAttribute API. Querying the attribute cudaGraphMemAttrReservedMemCurrent returns the amount of physical memory reserved by the driver for graph allocations in the current process. Querying cudaGraphMemAttrUsedMemCurrent returns the amount of physical memory currently mapped by at least one graph. Either of these attributes can be used to track when new physical memory is acquired by CUDA for the sake of an allocating graph. Both of these attributes are useful for examining how much memory is saved by the sharing mechanism.

+
+
+

+16.7. Peer Access +

+

Graph allocations can be configured for access from multiple GPUs, in which case CUDA will map the allocations onto the peer GPUs as required. CUDA allows graph allocations requiring different mappings to reuse the same virtual address. When this occurs, the address range is mapped onto all GPUs required by the different allocations. This means an allocation may sometimes allow more peer access than was requested during its creation; however, relying on these extra mappings is still an error.

+
+

+16.7.1. Peer Access with Graph Node APIs +

+

The cudaGraphAddMemAllocNode API accepts mapping requests in the accessDescs array field of the node parameters structures. The poolProps.location embedded structure specifies the resident device for the allocation. Access from the allocating GPU is assumed to be needed, thus the application does not need to specify an entry for the resident device in the accessDescs array.

+
+
cudaMemAllocNodeParams params = {};
+params.poolProps.allocType = cudaMemAllocationTypePinned;
+params.poolProps.location.type = cudaMemLocationTypeDevice;
+// specify device 1 as the resident device
+params.poolProps.location.id = 1;
+params.bytesize = size;
+
+// allocate an allocation resident on device 1 accessible from device 1
+cudaGraphAddMemAllocNode(&allocNode, graph, NULL, 0, &params);
+
+accessDescs[2];
+// boilerplate for the access descs (only ReadWrite and Device access supported by the add node api)
+accessDescs[0].flags = cudaMemAccessFlagsProtReadWrite;
+accessDescs[0].location.type = cudaMemLocationTypeDevice;
+accessDescs[1].flags = cudaMemAccessFlagsProtReadWrite;
+accessDescs[1].location.type = cudaMemLocationTypeDevice;
+
+// access being requested for device 0 & 2.  Device 1 access requirement left implicit.
+accessDescs[0].location.id = 0;
+accessDescs[1].location.id = 2;
+
+// access request array has 2 entries.
+params.accessDescCount = 2;
+params.accessDescs = accessDescs;
+
+// allocate an allocation resident on device 1 accessible from devices 0, 1 and 2. (0 & 2 from the descriptors, 1 from it being the resident device).
+cudaGraphAddMemAllocNode(&allocNode, graph, NULL, 0, &params);
+
+
+
+
+

+16.7.2. Peer Access with Stream Capture +

+

For stream capture, the allocation node records the peer accessibility of the allocating pool at the time of the capture. Altering the peer accessibility of the allocating pool after a cudaMallocFromPoolAsync call is captured does not affect the mappings that the graph will make for the allocation.

+
+
// boilerplate for the access descs (only ReadWrite and Device access supported by the add node api)
+accessDesc.flags = cudaMemAccessFlagsProtReadWrite;
+accessDesc.location.type = cudaMemLocationTypeDevice;
+accessDesc.location.id = 1;
+
+// let memPool be resident and accessible on device 0
+
+cudaStreamBeginCapture(stream);
+cudaMallocAsync(&dptr1, size, memPool, stream);
+cudaStreamEndCapture(stream, &graph1);
+
+cudaMemPoolSetAccess(memPool, &accessDesc, 1);
+
+cudaStreamBeginCapture(stream);
+cudaMallocAsync(&dptr2, size, memPool, stream);
+cudaStreamEndCapture(stream, &graph2);
+
+//The graph node allocating dptr1 would only have the device 0 accessibility even though memPool now has device 1 accessibility.
+//The graph node allocating dptr2 will have device 0 and device 1 accessibility, since that was the pool accessibility at the time of the cudaMallocAsync call.
+
+
+
+
+
+

+16.8. Memory Nodes in Child Graphs +

+

CUDA 12.9 introduces the ability to move child graph ownership to a parent graph. Child graphs which are moved to the parent are allowed to contain memory allocation and free nodes. This allows a child graph containing allocation or free nodes to be independently constructed prior to its addition in a parent graph.

+

The following restrictions apply to child graphs after they have been moved:

+
    +
  • Cannot be independently instantiated or destroyed.

  • +
  • Cannot be added as a child graph of a separate parent graph.

  • +
  • Cannot be used as an argument to cuGraphExecUpdate.

  • +
  • Cannot have additional memory allocation or free nodes added.

  • +
+
+
// Create the child graph
+cudaGraphCreate(&child, 0);
+
+// parameters for a basic allocation
+cudaMemAllocNodeParams params = {};
+params.poolProps.allocType = cudaMemAllocationTypePinned;
+params.poolProps.location.type = cudaMemLocationTypeDevice;
+// specify device 0 as the resident device
+params.poolProps.location.id = 0;
+params.bytesize = size;
+
+cudaGraphAddMemAllocNode(&allocNode, graph, NULL, 0, &params);
+// Additional nodes using the allocation could be added here
+cudaGraphAddMemFreeNode(&freeNode, graph, &allocNode, 1, params.dptr);
+
+// Create the parent graph
+cudaGraphCreate(&parent, 0);
+
+// Move the child graph to the parent graph
+cudaGraphNodeParams childNodeParams = { cudaGraphNodeTypeGraph };
+childNodeParams.graph.graph = child;
+childNodeParams.graph.ownership = cudaGraphChildGraphOwnershipMove;
+cudaGraphAddNode(&parentNode, parent, NULL, NULL, 0, &childNodeParams);
+
+
+
+
+
+

+17. Mathematical Functions +

+
+

Warning

+

This document has been replaced by a new CUDA Programming Guide. The information in this document should be considered legacy, and this document is no longer being updated as of CUDA 13.0. Please refer to the CUDA Programming Guide for up-to-date information on CUDA.

+
+

The reference manual lists, along with their description, all the functions of the C/C++ standard library mathematical functions that are supported in device code, as well as all intrinsic functions (that are only supported in device code).

+

This section provides accuracy information for some of these functions when applicable. It uses ULP for quantification. For further information on the definition of the Unit in the Last Place (ULP), please see Jean-Michel Muller’s paper On the definition of ulp(x), RR-5504, LIP RR-2005-09, INRIA, LIP. 2005, pp.16 at https://hal.inria.fr/inria-00070503/document.

+

Mathematical functions supported in device code do not set the global errno variable, nor report any floating-point exceptions to indicate errors; thus, if error diagnostic mechanisms are required, the user should implement additional screening for inputs and outputs of the functions. The user is responsible for the validity of pointer arguments. The user must not pass uninitialized parameters to the Mathematical functions as this may result in undefined behavior: functions are inlined in the user program and thus are subject to compiler optimizations.

+
+

+17.1. Standard Functions +

+

The functions from this section can be used in both host and device code.

+

This section specifies the error bounds of each function when executed on the device and also when executed on the host in the case where the host does not supply the function.

+

The error bounds are generated from extensive but not exhaustive tests, so they are not guaranteed bounds.

+

Single-Precision Floating-Point Functions

+

Addition and multiplication are IEEE-compliant, so have a maximum error of 0.5 ulp.

+

The recommended way to round a single-precision floating-point operand to an integer, with the result being a single-precision floating-point number is rintf(), not roundf(). The reason is that roundf() maps to a 4-instruction sequence on the device, whereas rintf() maps to a single instruction. truncf(), ceilf(), and floorf() each map to a single instruction as well.

+ + ++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+Table 17 Single-Precision Mathematical Standard Library Functions with Maximum ULP Error. The maximum error is stated as the absolute value of the difference in ulps between the result returned by the CUDA library function and a correctly rounded single-precision result obtained according to the round-to-nearest ties-to-even rounding mode. +

Function

Maximum ulp error

x+y

0 (IEEE-754 round-to-nearest-even)

x*y

0 (IEEE-754 round-to-nearest-even)

x/y

+

0 for compute capability \(\ge 2\) when compiled with -prec-div=true

+

2 (full range), otherwise

+

1/x

+

0 for compute capability \(\ge 2\) when compiled with -prec-div=true

+

1 (full range), otherwise

+
+

rsqrtf(x)

+

1/sqrtf(x)

+
+

2 (full range)

+

Applies to 1/sqrtf(x) only when it is converted to rsqrtf(x) by the compiler.

+

sqrtf(x)

+

0 when compiled with -prec-sqrt=true

+

Otherwise 1 for compute capability \(\ge 5.2\)

+

and 3 for older architectures

+

cbrtf(x)

1 (full range)

rcbrtf(x)

1 (full range)

hypotf(x,y)

3 (full range)

rhypotf(x,y)

2 (full range)

norm3df(x,y,z)

3 (full range)

rnorm3df(x,y,z)

2 (full range)

norm4df(x,y,z,t)

3 (full range)

rnorm4df(x,y,z,t)

2 (full range)

normf(dim,arr)

An error bound cannot be provided because a fast algorithm is used with accuracy loss due to round-off. .

rnormf(dim,arr)

An error bound cannot be provided because a fast algorithm is used with accuracy loss due to round-off. .

expf(x)

2 (full range)

exp2f(x)

2 (full range)

exp10f(x)

2 (full range)

expm1f(x)

1 (full range)

logf(x)

1 (full range)

log2f(x)

1 (full range)

log10f(x)

2 (full range)

log1pf(x)

1 (full range)

sinf(x)

2 (full range)

cosf(x)

2 (full range)

tanf(x)

4 (full range)

sincosf(x,sptr,cptr)

2 (full range)

sinpif(x)

1 (full range)

cospif(x)

1 (full range)

sincospif(x,sptr,cptr)

1 (full range)

asinf(x)

2 (full range)

acosf(x)

2 (full range)

atanf(x)

2 (full range)

atan2f(y,x)

3 (full range)

sinhf(x)

3 (full range)

coshf(x)

2 (full range)

tanhf(x)

2 (full range)

asinhf(x)

3 (full range)

acoshf(x)

4 (full range)

atanhf(x)

3 (full range)

powf(x,y)

4 (full range)

erff(x)

2 (full range)

erfcf(x)

4 (full range)

erfinvf(x)

2 (full range)

erfcinvf(x)

4 (full range)

erfcxf(x)

4 (full range)

normcdff(x)

5 (full range)

normcdfinvf(x)

5 (full range)

lgammaf(x)

6 (outside interval -10.001 … -2.264; larger inside)

tgammaf(x)

5 (full range)

fmaf(x,y,z)

0 (full range)

frexpf(x,exp)

0 (full range)

ldexpf(x,exp)

0 (full range)

scalbnf(x,n)

0 (full range)

scalblnf(x,l)

0 (full range)

logbf(x)

0 (full range)

ilogbf(x)

0 (full range)

j0f(x)

+

9 for |x| < 8

+

otherwise, the maximum absolute error is 2.2 x 10-6

+

j1f(x)

+

9 for |x| < 8

+

otherwise, the maximum absolute error is 2.2 x 10-6

+

jnf(n,x)

For n = 128, the maximum absolute error is 2.2 x 10-6

y0f(x)

+

9 for |x| < 8

+

otherwise, the maximum absolute error is 2.2 x 10-6

+

y1f(x)

+

9 for |x| < 8

+

otherwise, the maximum absolute error is 2.2 x 10-6

+

ynf(n,x)

+

ceil(2 + 2.5n) for |x| < n

+

otherwise, the maximum absolute error is 2.2 x 10-6

+

cyl_bessel_i0f(x)

6 (full range)

cyl_bessel_i1f(x)

6 (full range)

fmodf(x,y)

0 (full range)

remainderf(x,y)

0 (full range)

remquof(x,y,iptr)

0 (full range)

modff(x,iptr)

0 (full range)

fdimf(x,y)

0 (full range)

truncf(x)

0 (full range)

roundf(x)

0 (full range)

rintf(x)

0 (full range)

nearbyintf(x)

0 (full range)

ceilf(x)

0 (full range)

floorf(x)

0 (full range)

lrintf(x)

0 (full range)

lroundf(x)

0 (full range)

llrintf(x)

0 (full range)

llroundf(x)

0 (full range)

+

Double-Precision Floating-Point Functions

+

The recommended way to round a double-precision floating-point operand to an integer, with the result being a double-precision floating-point number is rint(), not round(). The reason is that round() maps to a 5-instruction sequence on the device, whereas rint() maps to a single instruction. trunc(), ceil(), and floor() each map to a single instruction as well.

+ + ++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+Table 18 Double-Precision Mathematical Standard Library Functions with Maximum ULP Error. The maximum error is stated as the absolute value of the difference in ulps between the result returned by the CUDA library function and a correctly rounded double-precision result obtained according to the round-to-nearest ties-to-even rounding mode. +

Function

Maximum ulp error

x+y

0 (IEEE-754 round-to-nearest-even)

x*y

0 (IEEE-754 round-to-nearest-even)

x/y

0 (IEEE-754 round-to-nearest-even)

1/x

0 (IEEE-754 round-to-nearest-even)

sqrt(x)

0 (IEEE-754 round-to-nearest-even)

rsqrt(x)

1 (full range)

cbrt(x)

1 (full range)

rcbrt(x)

1 (full range)

hypot(x,y)

2 (full range)

rhypot(x,y)

1 (full range)

norm3d(x,y,z)

2 (full range)

rnorm3d(x,y,z)

1 (full range)

norm4d(x,y,z,t)

2 (full range)

rnorm4d(x,y,z,t)

1 (full range)

norm(dim,arr)

An error bound cannot be provided because a fast algorithm is used with accuracy loss due to round-off.

rnorm(dim,arr)

An error bound cannot be provided because a fast algorithm is used with accuracy loss due to round-off.

exp(x)

1 (full range)

exp2(x)

1 (full range)

exp10(x)

1 (full range)

expm1(x)

1 (full range)

log(x)

1 (full range)

log2(x)

1 (full range)

log10(x)

1 (full range)

log1p(x)

1 (full range)

sin(x)

2 (full range)

cos(x)

2 (full range)

tan(x)

2 (full range)

sincos(x,sptr,cptr)

2 (full range)

sinpi(x)

2 (full range)

cospi(x)

2 (full range)

sincospi(x,sptr,cptr)

2 (full range)

asin(x)

2 (full range)

acos(x)

2 (full range)

atan(x)

2 (full range)

atan2(y,x)

2 (full range)

sinh(x)

2 (full range)

cosh(x)

1 (full range)

tanh(x)

1 (full range)

asinh(x)

3 (full range)

acosh(x)

3 (full range)

atanh(x)

2 (full range)

pow(x,y)

2 (full range)

erf(x)

2 (full range)

erfc(x)

5 (full range)

erfinv(x)

5 (full range)

erfcinv(x)

6 (full range)

erfcx(x)

4 (full range)

normcdf(x)

5 (full range)

normcdfinv(x)

8 (full range)

lgamma(x)

4 (outside interval -23.0001 … -2.2637; larger inside)

tgamma(x)

10 (full range)

fma(x,y,z)

0 (IEEE-754 round-to-nearest-even)

frexp(x,exp)

0 (full range)

ldexp(x,exp)

0 (full range)

scalbn(x,n)

0 (full range)

scalbln(x,l)

0 (full range)

logb(x)

0 (full range)

ilogb(x)

0 (full range)

j0(x)

+

7 for |x| < 8

+

otherwise, the maximum absolute error is 5 x 10-12

+

j1(x)

+

7 for |x| < 8

+

otherwise, the maximum absolute error is 5 x 10-12

+

jn(n,x)

For n = 128, the maximum absolute error is 5 x 10-12

y0(x)

+

7 for |x| < 8

+

otherwise, the maximum absolute error is 5 x 10-12

+

y1(x)

+

7 for |x| < 8

+

otherwise, the maximum absolute error is 5 x 10-12

+

yn(n,x)

For |x| > 1.5n, the maximum absolute error is 5 x 10-12

cyl_bessel_i0(x)

6 (full range)

cyl_bessel_i1(x)

6 (full range)

fmod(x,y)

0 (full range)

remainder(x,y)

0 (full range)

remquo(x,y,iptr)

0 (full range)

modf(x,iptr)

0 (full range)

fdim(x,y)

0 (full range)

trunc(x)

0 (full range)

round(x)

0 (full range)

rint(x)

0 (full range)

nearbyint(x)

0 (full range)

ceil(x)

0 (full range)

floor(x)

0 (full range)

lrint(x)

0 (full range)

lround(x)

0 (full range)

llrint(x)

0 (full range)

llround(x)

0 (full range)

+

Quad-Precision Floating-Point Functions

+

Note that the quad-precision mathematical functions are currently only available to devices with compute capability 10.0 and later. +Due to the specifics of implementation, the support of __float128 and _Float128 types in device code is also limited to select combinations of host platforms, see also Host Compiler Extensions.

+ + ++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+Table 19 Quad-Precision Mathematical Standard Library Functions with Maximum ULP Error. The maximum error is stated as the absolute value of the difference in ulps between the result returned by the CUDA library function and a correctly rounded quad-precision result obtained according to the round-to-nearest ties-to-even rounding mode. +

Function

Maximum ulp error

x+y +__nv_fp128_add(x, y)

0 (IEEE-754 round-to-nearest-even)

x-y +__nv_fp128_sub(x, y)

0 (IEEE-754 round-to-nearest-even)

x*y +__nv_fp128_mul(x, y)

0 (IEEE-754 round-to-nearest-even)

x/y +__nv_fp128_div(x, y)

0 (IEEE-754 round-to-nearest-even)

__nv_fp128_sqrt(x)

0 (IEEE-754 round-to-nearest-even)

__nv_fp128_fma(x, y, z)

0 (IEEE-754 round-to-nearest-even)

__nv_fp128_sin(x)

1 (full range)

__nv_fp128_cos(x)

1 (full range)

__nv_fp128_tan(x)

1 (full range)

__nv_fp128_asin(x)

1 (full range)

__nv_fp128_acos(x)

1 (full range)

__nv_fp128_atan(x)

1 (full range)

__nv_fp128_exp(x)

1 (full range)

__nv_fp128_exp2(x)

1 (full range)

__nv_fp128_exp10(x)

1 (full range)

__nv_fp128_expm1(x)

1 (full range)

__nv_fp128_log(x)

1 (full range)

__nv_fp128_log2(x)

1 (full range)

__nv_fp128_log10(x)

1 (full range)

__nv_fp128_log1p(x)

1 (full range)

__nv_fp128_pow(x, y)

1 (full range)

__nv_fp128_sinh(x)

1 (full range)

__nv_fp128_cosh(x)

1 (full range)

__nv_fp128_tanh(x)

1 (full range)

__nv_fp128_asinh(x)

1 (full range)

__nv_fp128_acosh(x)

1 (full range)

__nv_fp128_atanh(x)

1 (full range)

__nv_fp128_hypot(x, y)

1 (full range)

__nv_fp128_ceil(x)

0 (full range)

__nv_fp128_trunc(x)

0 (full range)

__nv_fp128_floor(x)

0 (full range)

__nv_fp128_round(x)

0 (full range)

__nv_fp128_rint(x)

0 (full range)

__nv_fp128_fabs(x)

0 (full range)

__nv_fp128_copysign(x, y)

0 (full range)

__nv_fp128_fmax(x, y)

0 (full range)

__nv_fp128_fmin(x, y)

0 (full range)

__nv_fp128_fdim(x, y)

0 (full range)

__nv_fp128_fmod(x, y)

0 (full range)

__nv_fp128_remainder(x, y)

0 (full range)

__nv_fp128_frexp(x, nptr)

0 (full range)

__nv_fp128_modf(x, iptr)

0 (full range)

__nv_fp128_ldexp(x, exp)

0 (full range)

__nv_fp128_ilogb(x)

0 (full range)

+
+
+

+17.2. Intrinsic Functions +

+

The functions from this section can only be used in device code.

+

Among these functions are the less accurate, but faster versions of some of the functions of Standard Functions. +They have the same name prefixed with __ (such as __sinf(x)). +They are faster as they map to fewer native instructions. +The compiler has an option (-use_fast_math) that forces each function in Table 20 +to compile to its intrinsic counterpart. In addition to reducing the accuracy of the affected functions, +it may also cause some differences in special case handling. A more robust approach is to selectively replace +mathematical function calls by calls to intrinsic functions only where it is merited by the performance gains +and where changed properties such as reduced accuracy and different special case handling can be tolerated.

+ + ++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+Table 20 Functions Affected by -use_fast_math +

Operator/Function

Device Function

x/y

__fdividef(x,y)

sinf(x)

__sinf(x)

cosf(x)

__cosf(x)

tanf(x)

__tanf(x)

sincosf(x,sptr,cptr)

__sincosf(x,sptr,cptr)

logf(x)

__logf(x)

log2f(x)

__log2f(x)

log10f(x)

__log10f(x)

expf(x)

__expf(x)

exp10f(x)

__exp10f(x)

powf(x,y)

__powf(x,y)

tanhf(x)

__tanhf(x)

+

Single-Precision Floating-Point Functions

+

__fadd_[rn,rz,ru,rd]() and __fmul_[rn,rz,ru,rd]() map to addition and multiplication operations that the compiler never merges into FMADs. By contrast, additions and multiplications generated from the ‘*’ and ‘+’ operators will frequently be combined into FMADs.

+

Functions suffixed with _rn operate using the round to nearest even rounding mode.

+

Functions suffixed with _rz operate using the round towards zero rounding mode.

+

Functions suffixed with _ru operate using the round up (to positive infinity) rounding mode.

+

Functions suffixed with _rd operate using the round down (to negative infinity) rounding mode.

+

The accuracy of floating-point division varies depending on whether the code is compiled with -prec-div=false +or -prec-div=true. When the code is compiled with -prec-div=false, both the regular division / +operator and __fdividef(x,y) have the same accuracy, but for 2126 < |y| < 2128, +__fdividef(x,y) delivers a result of zero, whereas the / operator delivers the correct result to +within the accuracy stated in Table 21. +Also, for 2126 < |y| < 2128, if x is infinity, __fdividef(x,y) delivers +a NaN (as a result of multiplying infinity by zero), while the / operator returns infinity. +On the other hand, the / operator is IEEE-compliant when the code is compiled with -prec-div=true +or without any -prec-div option at all since its default value is true.

+ + ++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+Table 21 Single-Precision Floating-Point Intrinsic Functions. (Supported by the CUDA Runtime Library with Respective Error Bounds) +

Function

Error bounds

__fadd_[rn,rz,ru,rd](x,y)

IEEE-compliant.

__fsub_[rn,rz,ru,rd](x,y)

IEEE-compliant.

__fmul_[rn,rz,ru,rd](x,y)

IEEE-compliant.

__fmaf_[rn,rz,ru,rd](x,y,z)

IEEE-compliant.

__frcp_[rn,rz,ru,rd](x)

IEEE-compliant.

__fsqrt_[rn,rz,ru,rd](x)

IEEE-compliant.

__frsqrt_rn(x)

IEEE-compliant.

__fdiv_[rn,rz,ru,rd](x,y)

IEEE-compliant.

__fdividef(x,y)

For |y| in [\(2^{-126}, 2^{126}\)], the maximum ulp error is 2.

__expf(x)

The maximum ulp error is 2 + floor(abs(1.173 * x)).

__exp10f(x)

The maximum ulp error is 2 + floor(abs(2.97 * x)).

__logf(x)

For x in [0.5, 2], the maximum absolute error is \(2^{-21.41}\), otherwise, the maximum ulp error is 3.

__log2f(x)

For x in [0.5, 2], the maximum absolute error is \(2^{-22}\), otherwise, the maximum ulp error is 2.

__log10f(x)

For x in [0.5, 2], the maximum absolute error is \(2^{-24}\), otherwise, the maximum ulp error is 3.

__sinf(x)

For x in [\(-\pi, \pi\)], the maximum absolute error is \(2^{-21.41}\), and larger otherwise.

__cosf(x)

For x in [\(-\pi, \pi\)], the maximum absolute error is \(2^{-21.19}\), and larger otherwise.

__sincosf(x,sptr,cptr)

Same as __sinf(x) and __cosf(x).

__tanf(x)

Derived from its implementation as __sinf(x) * (1/__cosf(x)).

__powf(x, y)

Derived from its implementation as exp2f(y * __log2f(x)).

__tanhf(x)

The maximum relative error of the current implementation is \(2^{-11}\). +Subnormal results of this fast intrinsic are not flushed to zero even under -ftz=true compiler setting. +Available for devices with compute capability of at least 7.5; +and defaults to regular tanhf() function behavior on other devices.

+

Double-Precision Floating-Point Functions

+

__dadd_rn() and __dmul_rn() map to addition and multiplication operations that the compiler never merges into FMADs. By contrast, additions and multiplications generated from the ‘*’ and ‘+’ operators will frequently be combined into FMADs.

+ + ++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+Table 22 Double-Precision Floating-Point Intrinsic Functions. (Supported by the CUDA Runtime Library with Respective Error Bounds) +

Function

Error bounds

__dadd_[rn,rz,ru,rd](x,y)

IEEE-compliant.

__dsub_[rn,rz,ru,rd](x,y)

IEEE-compliant.

__dmul_[rn,rz,ru,rd](x,y)

IEEE-compliant.

__fma_[rn,rz,ru,rd](x,y,z)

IEEE-compliant.

__ddiv_[rn,rz,ru,rd](x,y)(x,y)

+

IEEE-compliant.

+

Requires compute capability > 2.

+

__drcp_[rn,rz,ru,rd](x)

+

IEEE-compliant.

+

Requires compute capability > 2.

+

__dsqrt_[rn,rz,ru,rd](x)

+

IEEE-compliant.

+

Requires compute capability > 2.

+
+
+
+
+

+18. C++ Language Support +

+
+

Warning

+

This document has been replaced by a new CUDA Programming Guide. The information in this document should be considered legacy, and this document is no longer being updated as of CUDA 13.0. Please refer to the CUDA Programming Guide for up-to-date information on CUDA.

+
+

As described in Compilation with NVCC, CUDA source files compiled with nvcc can include a mix of host code and device code. The CUDA front-end compiler aims to emulate the host compiler behavior with respect to C++ input code. The input source code is processed according to the C++ ISO/IEC 14882:2003, C++ ISO/IEC 14882:2011, C++ ISO/IEC 14882:2014 or C++ ISO/IEC 14882:2017 specifications, and the CUDA front-end compiler aims to emulate any host compiler divergences from the ISO specification. In addition, the supported language is extended with CUDA-specific constructs described in this document 6, and is subject to the restrictions described below.

+

C++11 Language Features, C++14 Language Features and C++17 Language Features provide support matrices for the C++11, C++14, C++17 and C++20 features, respectively. Restrictions lists the language restrictions. Polymorphic Function Wrappers and Extended Lambdas describe additional features. Code Samples gives code samples.

+
+

+18.1. C++11 Language Features +

+

The following table lists new language features that have been accepted into the C++11 standard. The “Proposal” column provides a link to the ISO C++ committee proposal that describes the feature, while the “Available in nvcc (device code)” column indicates the first version of nvcc that contains an implementation of this feature (if it has been implemented) for device code.

+ + +++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+Table 23 C++11 Language Features +

Language Feature

C++11 Proposal

Available in nvcc (device code)

Rvalue references

N2118

7.0

  Rvalue references for *this

N2439

7.0

Initialization of class objects by rvalues

N1610

7.0

Non-static data member initializers

N2756

7.0

Variadic templates

N2242

7.0

  Extending variadic template template parameters

N2555

7.0

Initializer lists

N2672

7.0

Static assertions

N1720

7.0

auto-typed variables

N1984

7.0

        Multi-declarator auto

N1737

7.0

        Removal of auto as a storage-class specifier

N2546

7.0

        New function declarator syntax

N2541

7.0

Lambda expressions

N2927

7.0

Declared type of an expression

N2343

7.0

        Incomplete return types

N3276

7.0

Right angle brackets

N1757

7.0

Default template arguments for function templates

DR226

7.0

Solving the SFINAE problem for expressions

DR339

7.0

Alias templates

N2258

7.0

Extern templates

N1987

7.0

Null pointer constant

N2431

7.0

Strongly-typed enums

N2347

7.0

Forward declarations for enums

N2764 +DR1206

7.0

Standardized attribute syntax

N2761

7.0

Generalized constant expressions

N2235

7.0

Alignment support

N2341

7.0

Conditionally-support behavior

N1627

7.0

Changing undefined behavior into diagnosable errors

N1727

7.0

Delegating constructors

N1986

7.0

Inheriting constructors

N2540

7.0

Explicit conversion operators

N2437

7.0

New character types

N2249

7.0

Unicode string literals

N2442

7.0

Raw string literals

N2442

7.0

Universal character names in literals

N2170

7.0

User-defined literals

N2765

7.0

Standard Layout Types

N2342

7.0

Defaulted functions

N2346

7.0

Deleted functions

N2346

7.0

Extended friend declarations

N1791

7.0

Extending sizeof

N2253 +DR850

7.0

Inline namespaces

N2535

7.0

Unrestricted unions

N2544

7.0

Local and unnamed types as template arguments

N2657

7.0

Range-based for

N2930

7.0

Explicit virtual overrides

N2928 +N3206 +N3272

7.0

Minimal support for garbage collection and reachability-based leak detection

N2670

N/A (see Restrictions)

Allowing move constructors to throw [noexcept]

N3050

7.0

Defining move special member functions

N3053

7.0

Concurrency

Sequence points

N2239

Atomic operations

N2427

Strong Compare and Exchange

N2748

Bidirectional Fences

N2752

Memory model

N2429

Data-dependency ordering: atomics and memory model

N2664

Propagating exceptions

N2179

Allow atomics use in signal handlers

N2547

Thread-local storage

N2659

Dynamic initialization and destruction with concurrency

N2660

C99 Features in C++11

__func__ predefined identifier

N2340

7.0

C99 preprocessor

N1653

7.0

long long

N1811

7.0

Extended integral types

N1988

+
+
+

+18.2. C++14 Language Features +

+

The following table lists new language features that have been accepted into the C++14 standard.

+ + +++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+Table 24 C++14 Language Features +

Language Feature

C++14 Proposal

Available in nvcc (device code)

Tweak to certain C++ contextual conversions

N3323

9.0

Binary literals

N3472

9.0

Functions with deduced return type

N3638

9.0

Generalized lambda capture (init-capture)

N3648

9.0

Generic (polymorphic) lambda expressions

N3649

9.0

Variable templates

N3651

9.0

Relaxing requirements on constexpr functions

N3652

9.0

Member initializers and aggregates

N3653

9.0

Clarifying memory allocation

N3664

Sized deallocation

N3778

[[deprecated]] attribute

N3760

9.0

Single-quotation-mark as a digit separator

N3781

9.0

+
+
+

+18.3. C++17 Language Features +

+

All C++17 language features are supported in nvcc version 11.0 and later, subject to restrictions described here.

+
+
+

+18.4. C++20 Language Features +

+

All C++20 language features are supported in nvcc version 12.0 and later, subject to restrictions described here.

+
+
+

+18.5. Restrictions +

+
+

+18.5.1. Host Compiler Extensions +

+

Host compiler specific language extensions are not supported in device code.

+

__Complex types are only supported in host code.

+

__int128 type is supported in device code when compiled in conjunction with a host compiler that supports it.

+

__float128 type is supported for devices with compute capability 10.0 and later, when compiled in conjunction with a host compiler that supports the type. A constant expression of __float128 type may be processed by the compiler in a floating point representation with lower precision.

+
+
+

+18.5.2. Preprocessor Symbols +

+
+

+18.5.2.1. __CUDA_ARCH__ +

+
    +
  1. +

    The type signature of the following entities shall not depend on whether __CUDA_ARCH__ is defined or not, or on a particular value of __CUDA_ARCH__:

    +
      +
    • __global__ functions and function templates

    • +
    • __device__ and __constant__ variables

    • +
    • textures and surfaces

    • +
    +

    Example:

    +
    +
    #if !defined(__CUDA_ARCH__)
    +typedef int mytype;
    +#else
    +typedef double mytype;
    +#endif
    +
    +__device__ mytype xxx;         // error: xxx's type depends on __CUDA_ARCH__
    +__global__ void foo(mytype in, // error: foo's type depends on __CUDA_ARCH__
    +                    mytype *ptr)
    +{
    +  *ptr = in;
    +}
    +
    +
    +
  2. +
  3. +

    If a __global__ function template is instantiated and launched from the host, then the function template must be instantiated with the same template arguments irrespective of whether __CUDA_ARCH__ is defined and regardless of the value of __CUDA_ARCH__.

    +

    Example:

    +
    +
    __device__ int result;
    +template <typename T>
    +__global__ void kern(T in)
    +{
    +  result = in;
    +}
    +
    +__host__ __device__ void foo(void)
    +{
    +#if !defined(__CUDA_ARCH__)
    +  kern<<<1,1>>>(1);      // error: "kern<int>" instantiation only
    +                         // when __CUDA_ARCH__ is undefined!
    +#endif
    +}
    +
    +int main(void)
    +{
    +  foo();
    +  cudaDeviceSynchronize();
    +  return 0;
    +}
    +
    +
    +
  4. +
  5. +

    In separate compilation mode, the presence or absence of a definition of a function or variable with external linkage shall not depend on whether __CUDA_ARCH__ is defined or on a particular value of __CUDA_ARCH__7.

    +

    Example:

    +
    +
    #if !defined(__CUDA_ARCH__)
    +void foo(void) { }                  // error: The definition of foo()
    +                                    // is only present when __CUDA_ARCH__
    +                                    // is undefined
    +#endif
    +
    +
    +
  6. +
  7. +

    In separate compilation, __CUDA_ARCH__ must not be used in headers such that different objects could contain different behavior. Or, it must be guaranteed that all objects will compile for the same compute_arch. If a weak function or template function is defined in a header and its behavior depends on __CUDA_ARCH__, then the instances of that function in the objects could conflict if the objects are compiled for different compute arch.

    +

    For example, if an a.h contains:

    +
    +
    template<typename T>
    +__device__ T* getptr(void)
    +{
    +#if __CUDA_ARCH__ == 700
    +  return NULL; /* no address */
    +#else
    +  __shared__ T arr[256];
    +  return arr;
    +#endif
    +}
    +
    +
    +

    Then if a.cu and b.cu both include a.h and instantiate getptr for the same type, and b.cu expects a non-NULL address, and compile with:

    +
    +
    nvcc –arch=compute_70 –dc a.cu
    +nvcc –arch=compute_80 –dc b.cu
    +nvcc –arch=sm_80 a.o b.o
    +
    +
    +

    At link time only one version of the getptr is used, so the behavior would depend on which version is chosen. To avoid this, either a.cu and b.cu must be compiled for the same compute arch, or __CUDA_ARCH__ should not be used in the shared header function.

    +
  8. +
+

The compiler does not guarantee that a diagnostic will be generated for the unsupported uses of __CUDA_ARCH__ described above.

+
+
+
+

+18.5.3. Qualifiers +

+
+

+18.5.3.1. Device Memory Space Specifiers +

+

The __device__, __shared__, __managed__ and __constant__ memory space specifiers are not allowed on:

+
    +
  • class, struct, and union data members,

  • +
  • formal parameters,

  • +
  • non-extern variable declarations within a function that executes on the host.

  • +
+

The __device__, __constant__ and __managed__ memory space specifiers are not allowed on variable declarations that are neither extern nor static within a function that executes on the device.

+

A __device__, __constant__, __managed__ or __shared__ variable definition cannot have a class type with a non-empty constructor or a non-empty destructor. A constructor for a class type is considered empty at a point in the translation unit, if it is either a trivial constructor or it satisfies all of the following conditions:

+
    +
  • The constructor function has been defined.

  • +
  • The constructor function has no parameters, the initializer list is empty and the function body is an empty compound statement.

  • +
  • Its class has no virtual functions, no virtual base classes and no non-static data member initializers.

  • +
  • The default constructors of all base classes of its class can be considered empty.

  • +
  • For all the nonstatic data members of its class that are of class type (or array thereof), the default constructors can be considered empty.

  • +
+

A destructor for a class is considered empty at a point in the translation unit, if it is either a trivial destructor or it satisfies all of the following conditions:

+
    +
  • The destructor function has been defined.

  • +
  • The destructor function body is an empty compound statement.

  • +
  • Its class has no virtual functions and no virtual base classes.

  • +
  • The destructors of all base classes of its class can be considered empty.

  • +
  • For all the nonstatic data members of its class that are of class type (or array thereof), the destructor can be considered empty.

  • +
+

When compiling in the whole program compilation mode (see the nvcc user manual for a description of this mode), __device__, __shared__, __managed__ and __constant__ variables cannot be defined as external using the extern keyword. The only exception is for dynamically allocated __shared__ variables as described in __shared__.

+

When compiling in the separate compilation mode (see the nvcc user manual for a description of this mode), __device__, __shared__, __managed__ and __constant__ variables can be defined as external using the extern keyword. nvlink will generate an error when it cannot find a definition for an external variable (unless it is a dynamically allocated __shared__ variable).

+
+
+

+18.5.3.2. __managed__ Memory Space Specifier +

+

Variables marked with the __managed__ memory space specifier (“managed” variables) have the following restrictions:

+
    +
  • The address of a managed variable is not a constant expression.

  • +
  • A managed variable shall not have a const qualified type.

  • +
  • A managed variable shall not have a reference type.

  • +
  • +

    The address or value of a managed variable shall not be used when the CUDA runtime may not be in a valid state, including the following cases:

    +
      +
    • In static/dynamic initialization or destruction of an object with static or thread local storage duration.

    • +
    • In code that executes after exit() has been called (for example, a function marked with gcc’s “__attribute__((destructor))”).

    • +
    • In code that executes when CUDA runtime may not be initialized (for example, a function marked with gcc’s “__attribute__((constructor))”).

    • +
    +
  • +
  • A managed variable cannot be used as an unparenthesized id-expression argument to a decltype() expression.

  • +
  • Managed variables have the same coherence and consistency behavior as specified for dynamically allocated managed memory.

  • +
  • When a CUDA program containing managed variables is run on an execution platform with multiple GPUs, the variables are allocated only once, and not per GPU.

  • +
  • A managed variable declaration without the extern linkage is not allowed within a function that executes on the host.

  • +
  • A managed variable declaration without the extern or static linkage is not allowed within a function that executes on the device.

  • +
+

Here are examples of legal and illegal uses of managed variables:

+
+
__device__ __managed__ int xxx = 10;         // OK
+
+int *ptr = &xxx;                             // error: use of managed variable
+                                             // (xxx) in static initialization
+struct S1_t {
+  int field;
+  S1_t(void) : field(xxx) { };
+};
+struct S2_t {
+  ~S2_t(void) { xxx = 10; }
+};
+
+S1_t temp1;                                 // error: use of managed variable
+                                            // (xxx) in dynamic initialization
+
+S2_t temp2;                                 // error: use of managed variable
+                                            // (xxx) in the destructor of
+                                            // object with static storage
+                                            // duration
+
+__device__ __managed__ const int yyy = 10;  // error: const qualified type
+
+__device__ __managed__ int &zzz = xxx;      // error: reference type
+
+template <int *addr> struct S3_t { };
+S3_t<&xxx> temp;                            // error: address of managed
+                                            // variable(xxx) not a
+                                            // constant expression
+
+__global__ void kern(int *ptr)
+{
+  assert(ptr == &xxx);                      // OK
+  xxx = 20;                                 // OK
+}
+int main(void)
+{
+  int *ptr = &xxx;                          // OK
+  kern<<<1,1>>>(ptr);
+  cudaDeviceSynchronize();
+  xxx++;                                    // OK
+  decltype(xxx) qqq;                        // error: managed variable(xxx) used
+                                            // as unparenthized argument to
+                                            // decltype
+
+  decltype((xxx)) zzz = yyy;                // OK
+}
+
+
+
+
+

+18.5.3.3. Volatile Qualifier +

+
+

Note

+

The volatile keyword is supported to maintain compatibility with ISO C++; however, few if any of its remaining non-deprecated uses apply to GPUs.

+
+

Reads and writes to volatile qualified objects are not atomic, and are compiled to one or more .volatile instructions which do NOT guarantee:

+
    +
  • ordering of memory operations, or

  • +
  • that the number of memory operations performed by the HW matches the number of PTX instructions.

  • +
+

That is, CUDA C++ volatile is not suitable for:

+
    +
  • +

    Inter-Thread Synchronization: use atomic operations via cuda::atomic_ref, cuda::atomic, or Atomic Functions instead. +Atomic memory operations provide inter-thread synchronization guarantees and deliver much better performance than volatile operations. +CUDA C++ volatile operations do not provide any inter-thread synchronization guarantees and are therefore not correct for inter-thread synchronization. +The following example shows how to pass a message across two threads using atomic operations.

    +
    +
    +
    + +
    + +++ + + + + + +
    +
    +
    __global__ void kernel(int* flag, int* data) {
    +  cuda::atomic_ref<int, cuda::thread_scope_device> f{*flag};
    +  if (threadIdx.x == 0) {
    +    // Consumer: blocks until flag is set by producer, then reads data
    +    while(f.load(cuda::memory_order_acquire) == 0);
    +    if (*data != 42) __trap(); // Errors if wrong data read
    +  } else if (threadIdx.x == 1) {
    +    // Producer: writes data then sets flag
    +    *data = 42;
    +    f.store(1, cuda::memory_order_release);
    +  }
    +}
    +
    +
    +
    +
    + +
    + +++ + + + + + +
    +
    +
    __global__ void kernel(cuda::atomic<int, cuda::thread_scope_device>* flag, int* data) {
    +  if (threadIdx.x == 0) {
    +    // Consumer: blocks until flag is set by producer, then reads data
    +    while(flag->load(cuda::memory_order_acquire) == 0);
    +    if (*data != 42) __trap(); // Errors if wrong data read
    +  } else if (threadIdx.x == 1) {
    +    // Producer: writes data then sets flag
    +    *data = 42;
    +    flag->store(1, cuda::memory_order_release);
    +  }
    +}
    +
    +
    +
    +
    + +
    + +++ + + + + + +
    +
    +
    __global__ void kernel(int* flag, int* data) {
    +  if (threadIdx.x == 0) {
    +    // Consumer: blocks until flag is set by producer, then reads data
    +    while(atomicAdd(flag, 0) == 0); // Load with Relaxed Read-Modify-Write
    +    __threadfence();                // SequentiallyConsistent fence
    +    if (*data != 42) __trap();      // Errors if wrong data read
    +  } else if (threadIdx.x == 1) {
    +    // Producer: writes data then sets flag
    +    *data = 42;
    +    __threadfence();     // SequentiallyConsistent fence
    +    atomicExch(flag, 1); // Store with Relaxed Read-Modify-Write
    +  }
    +}
    +
    +
    +
    +
    +
    +
    +
    +
  • +
  • +

    Memory Mapped IO (MMIO): use PTX MMIO operations via inline PTX instead. +PTX MMIO operations strictly preserve the number of memory accesses performed. +CUDA C++ volatile operations do not preserve the number of memory accesses performed, and may perform more or less accesses than requested in a non-deterministic way, making them incorrect for MMIO. +The following example shows how to read and write from a register using PTX mmio operations.

    +
    +
    +
    +
    __global__ void kernel(int* mmio_reg0, int* mmio_reg1) {
    +  // Write to MMIO register:
    +  int value = 13;
    +  asm volatile("st.relaxed.mmio.sys.u32 [%0], %1;" :: "l"(mmio_reg0), "r"(value) : "memory");
    +
    +  // Read MMIO register:
    +  asm volatile("ld.relaxed.mmio.sys.u32 %0, [%1];" : "=r"(value) : "l"(mmio_reg1) : "memory");
    +  
    +  if (value != 42) __trap(); // Errors if wrong data read
    +}
    +
    +
    +
    +
    +
  • +
+
+
+
+

+18.5.4. Pointers +

+

Dereferencing a pointer either to global or shared memory in code that is executed on the host, or to host memory in code that is executed on the device results in an undefined behavior, most often in a segmentation fault and application termination.

+

The address obtained by taking the address of a __device__, __shared__ or __constant__ variable can only be used in device code. The address of a __device__ or __constant__ variable obtained through cudaGetSymbolAddress() as described in Device Memory can only be used in host code.

+
+
+

+18.5.5. Operators +

+
+

+18.5.5.1. Assignment Operator +

+

__constant__ variables can only be assigned from the host code through runtime functions (Device Memory); they cannot be assigned from the device code.

+

__shared__ variables cannot have an initialization as part of their declaration.

+

It is not allowed to assign values to any of the built-in variables defined in Built-in Variables.

+
+
+

+18.5.5.2. Address Operator +

+

It is not allowed to take the address of any of the built-in variables defined in Built-in Variables.

+
+
+
+

+18.5.6. Run Time Type Information (RTTI) +

+

The following RTTI-related features are supported in host code, but not in device code.

+
    +
  • typeid operator

  • +
  • std::type_info

  • +
  • dynamic_cast operator

  • +
+
+
+

+18.5.7. Exception Handling +

+

Exception handling is only supported in host code, but not in device code.

+

Exception specification is not supported for __global__ functions.

+
+
+

+18.5.8. Standard Library +

+

Standard libraries are only supported in host code, but not in device code, unless specified otherwise.

+
+
+

+18.5.9. Namespace Reservations +

+

Unless an exception is otherwise noted, it is undefined behavior to add any declarations or definitions to cuda::, nv::, cooperative_groups:: or any namespace nested within.

+

Examples:

+
+
namespace cuda{
+   // Bad: class declaration added to namespace cuda
+   struct foo{};
+
+   // Bad: function definition added to namespace cuda
+   cudaStream_t make_stream(){
+      cudaStream_t s;
+      cudaStreamCreate(&s);
+      return s;
+   }
+} // namespace cuda
+
+namespace cuda{
+   namespace utils{
+      // Bad: function definition added to namespace nested within cuda
+      cudaStream_t make_stream(){
+          cudaStream_t s;
+          cudaStreamCreate(&s);
+          return s;
+      }
+   } // namespace utils
+} // namespace cuda
+
+namespace utils{
+   namespace cuda{
+     // Okay: namespace cuda may be used nested within a non-reserved namespace
+     cudaStream_t make_stream(){
+          cudaStream_t s;
+          cudaStreamCreate(&s);
+          return s;
+      }
+   } // namespace cuda
+} // namespace utils
+
+// Bad: Equivalent to adding symbols to namespace cuda at global scope
+using namespace utils;
+
+
+
+
+

+18.5.10. Functions +

+
+

+18.5.10.1. External Linkage +

+

A call within some device code of a function declared with the extern qualifier is only allowed if the function is defined within the same compilation unit as the device code, i.e., a single file or several files linked together with relocatable device code and nvlink.

+
+
+

+18.5.10.2. Implicitly-declared and non-virtual explicitly-defaulted functions +

+

Let F denote a function that is either implicitly-declared or is a non-virtual function that is explicitly-defaulted on its first declaration. The execution space specifiers (__host__, __device__) for F are the union of the execution space specifiers of all the functions that invoke it (note that a __global__ caller will be treated as a __device__ caller for this analysis). For example:

+
+
class Base {
+  int x;
+public:
+  __host__ __device__ Base(void) : x(10) {}
+};
+
+class Derived : public Base {
+  int y;
+};
+
+class Other: public Base {
+  int z;
+};
+
+__device__ void foo(void)
+{
+  Derived D1;
+  Other D2;
+}
+
+__host__ void bar(void)
+{
+  Other D3;
+}
+
+
+

Here, the implicitly-declared constructor function “Derived::Derived” will be treated as a __device__ function, since it is invoked only from the __device__ function “foo”. The implicitly-declared constructor function “Other::Other” will be treated as a __host__ __device__ function, since it is invoked both from a __device__ function “foo” and a __host__ function “bar”.

+

In addition, if F is an implicitly declared virtual function (e.g.,a virtual destructor), then the execution spaces of each virtual function D overridden by F are added to the set of execution spaces for F, if D is not implicitly declared.

+

For example:

+
+
struct Base1 { virtual __host__ __device__ ~Base1() { } };
+struct Derived1 : Base1 { }; // implicitly-declared virtual destructor
+                             // ~Derived1 has __host__ __device__
+                             // execution space specifiers
+
+struct Base2 { virtual __device__ ~Base2() = default; };
+struct Derived2 : Base2 { }; // implicitly-declared virtual destructor
+                             // ~Derived2 has __device__ execution
+                             // space specifiers
+
+
+
+
+

+18.5.10.3. Function Parameters +

+

__global__ function parameters are passed to the device via constant memory and are limited to 32,764 bytes starting with Volta, and 4 KB on older architectures.

+

__global__ functions cannot have a variable number of arguments.

+

__global__ function parameters cannot be pass-by-reference.

+

In separate compilation mode, if a __device__ or __global__ function is ODR-used in a particular translation unit, then the parameter and return types of the function must be complete in that translation unit.

+

Example:

+
+
//first.cu:
+struct S;
+__device__ void foo(S); // error: type 'S' is incomplete
+__device__ auto *ptr = foo;
+
+int main() { }
+
+//second.cu:
+struct S { int x; };
+__device__ void foo(S) { }
+
+
+
+
//compiler invocation
+$nvcc -std=c++14 -rdc=true first.cu second.cu -o first
+nvlink error   : Prototype doesn't match for '_Z3foo1S' in '/tmp/tmpxft_00005c8c_00000000-18_second.o', first defined in '/tmp/tmpxft_00005c8c_00000000-18_second.o'
+nvlink fatal   : merge_elf failed
+
+
+
+
+18.5.10.3.1. __global__ Function Argument Processing +
+

When a __global__ function is launched from device code, each argument must be trivially copyable and trivially destructible.

+

When a __global__ function is launched from host code, each argument type is allowed to be non-trivially copyable or non-trivially-destructible, but the processing for such types does not follow the standard C++ model, as described below. User code must ensure that this workflow does not affect program correctness. The workflow diverges from standard C++ in two areas:

+
    +
  1. +

    Memcpy instead of copy constructor invocation

    +

    When lowering a __global__ function launch from host code, the compiler generates stub functions that copy the parameters one or more times by value, before eventually using memcpy to copy the arguments to the __global__ function’s parameter memory on the device. This occurs even if an argument was non-trivially-copyable, and therefore may break programs where the copy constructor has side effects.

    +

    Example:

    +
    +
    #include <cassert>
    +struct S {
    + int x;
    + int *ptr;
    + __host__ __device__ S() { }
    + __host__ __device__ S(const S &) { ptr = &x; }
    +};
    +
    +__global__ void foo(S in) {
    + // this assert may fail, because the compiler
    + // generated code will memcpy the contents of "in"
    + // from host to kernel parameter memory, so the
    + // "in.ptr" is not initialized to "&in.x" because
    + // the copy constructor is skipped.
    + assert(in.ptr == &in.x);
    +}
    +
    +int main() {
    +  S tmp;
    +  foo<<<1,1>>>(tmp);
    +  cudaDeviceSynchronize();
    +}
    +
    +
    +

    Example:

    +
    +
    #include <cassert>
    +
    +__managed__ int counter;
    +struct S1 {
    +S1() { }
    +S1(const S1 &) { ++counter; }
    +};
    +
    +__global__ void foo(S1) {
    +
    +/* this assertion may fail, because
    +   the compiler generates stub
    +   functions on the host for a kernel
    +   launch, and they may copy the
    +   argument by value more than once.
    +*/
    +assert(counter == 1);
    +}
    +
    +int main() {
    +S1 V;
    +foo<<<1,1>>>(V);
    +cudaDeviceSynchronize();
    +}
    +
    +
    +
  2. +
  3. +

    Destructor may be invoked before the ``__global__`` function has finished

    +

    Kernel launches are asynchronous with host execution. As a result, if a __global__ function argument has a non-trivial destructor, the destructor may execute in host code even before the __global__ function has finished execution. This may break programs where the destructor has side effects.

    +

    Example:

    +
    +
    struct S {
    + int *ptr;
    + S() : ptr(nullptr) { }
    + S(const S &) { cudaMallocManaged(&ptr, sizeof(int)); }
    + ~S() { cudaFree(ptr); }
    +};
    +
    +__global__ void foo(S in) {
    +
    +  //error: This store may write to memory that has already been
    +  //       freed (see below).
    +  *(in.ptr) = 4;
    +
    +}
    +
    +int main() {
    + S V;
    +
    + /* The object 'V' is first copied by value to a compiler-generated
    +  * stub function that does the kernel launch, and the stub function
    +  * bitwise copies the contents of the argument to kernel parameter
    +  * memory.
    +  * However, GPU kernel execution is asynchronous with host
    +  * execution.
    +  * As a result, S::~S() will execute when the stub function   returns, releasing allocated memory, even though the kernel may not have finished execution.
    +  */
    + foo<<<1,1>>>(V);
    + cudaDeviceSynchronize();
    +}
    +
    +
    +
  4. +
+
+
+
+18.5.10.3.2. Toolkit and Driver Compatibility +
+

Developers must use the 12.1 Toolkit and r530 driver or higher to compile, launch, and debug kernels that accept parameters larger than 4KB. If such kernels are launched on older drivers, CUDA will issue the error CUDA_ERROR_NOT_SUPPORTED.

+
+ +
+
+

+18.5.10.4. Static Variables within Function +

+

Variable memory space specifiers are allowed in the declaration of a static variable V within the immediate or nested block scope of a function F where:

+
    +
  • F is a __global__ or __device__-only function.

  • +
  • F is a __host__ __device__ function and __CUDA_ARCH__ is defined 11.

  • +
+

If no explicit memory space specifier is present in the declaration of V, an implicit __device__ specifier is assumed during device compilation.

+

V has the same initialization restrictions as a variable with the same memory space specifiers declared in namespace scope for example a __device__ variable cannot have a ‘non-empty’ constructor (see Device Memory Space Specifiers).

+

Examples of legal and illegal uses of function-scope static variables are shown below.

+
+
struct S1_t {
+  int x;
+};
+
+struct S2_t {
+  int x;
+  __device__ S2_t(void) { x = 10; }
+};
+
+struct S3_t {
+  int x;
+  __device__ S3_t(int p) : x(p) { }
+};
+
+__device__ void f1() {
+  static int i1;              // OK, implicit __device__ memory space specifier
+  static int i2 = 11;         // OK, implicit __device__ memory space specifier
+  static __managed__ int m1;  // OK
+  static __device__ int d1;   // OK
+  static __constant__ int c1; // OK
+
+  static S1_t i3;             // OK, implicit __device__ memory space specifier
+  static S1_t i4 = {22};      // OK, implicit __device__ memory space specifier
+
+  static __shared__ int i5;   // OK
+
+  int x = 33;
+  static int i6 = x;          // error: dynamic initialization is not allowed
+  static S1_t i7 = {x};       // error: dynamic initialization is not allowed
+
+  static S2_t i8;             // error: dynamic initialization is not allowed
+  static S3_t i9(44);         // error: dynamic initialization is not allowed
+}
+
+__host__ __device__ void f2() {
+  static int i1;              // OK, implicit __device__ memory space specifier
+                              // during device compilation.
+#ifdef __CUDA_ARCH__
+  static __device__ int d1;   // OK, declaration is only visible during device
+                              // compilation  (__CUDA_ARCH__ is defined)
+#else
+  static int d0;              // OK, declaration is only visible during host
+                              // compilation (__CUDA_ARCH__ is not defined)
+#endif
+
+  static __device__ int d2;   // error: __device__ variable inside
+                              // a host function during host compilation
+                              // i.e. when __CUDA_ARCH__ is not defined
+
+  static __shared__ int i2;  // error: __shared__ variable inside
+                             // a host function during host compilation
+                             // i.e. when __CUDA_ARCH__ is not defined
+}
+
+
+
+
+

+18.5.10.5. Function Pointers +

+

The address of a __global__ function taken in host code cannot be used in device code (e.g. to launch the kernel). Similarly, the address of a __global__ function taken in device code cannot be used in host code.

+

It is not allowed to take the address of a __device__ function in host code.

+
+
+

+18.5.10.6. Function Recursion +

+

__global__ functions do not support recursion.

+
+
+

+18.5.10.7. Friend Functions +

+

A __global__ function or function template cannot be defined in a friend declaration.

+

Example:

+
+
struct S1_t {
+  friend __global__
+  void foo1(void);  // OK: not a definition
+  template<typename T>
+  friend __global__
+  void foo2(void); // OK: not a definition
+
+  friend __global__
+  void foo3(void) { } // error: definition in friend declaration
+
+  template<typename T>
+  friend __global__
+  void foo4(void) { } // error: definition in friend declaration
+};
+
+
+
+
+

+18.5.10.8. Operator Function +

+

An operator function cannot be a __global__ function.

+
+
+

+18.5.10.9. Allocation and Deallocation Functions +

+

A user-defined operator new, operator new[], operator delete, or operator delete[] cannot be used to replace the corresponding __host__ or __device__ builtins provided by the compiler.

+
+
+
+

+18.5.11. Classes +

+
+

+18.5.11.1. Data Members +

+

Static data members are not supported except for those that are also const-qualified (see Const-qualified variables).

+
+
+

+18.5.11.2. Function Members +

+

Static member functions cannot be __global__ functions.

+
+
+

+18.5.11.3. Virtual Functions +

+

When a function in a derived class overrides a virtual function in a base class, the execution space specifiers (i.e., __host__, __device__) on the overridden and overriding functions must match.

+

It is not allowed to pass as an argument to a __global__ function an object of a class with virtual functions.

+

If an object is created in host code, invoking a virtual function for that object in device code has undefined behavior.

+

If an object is created in device code, invoking a virtual function for that object in host code has undefined behavior.

+

See Windows-Specific for additional constraints when using the Microsoft host compiler.

+

Example:

+
+
struct S1 { virtual __host__ __device__ void foo() { } };
+
+__managed__ S1 *ptr1, *ptr2;
+
+__managed__ __align__(16) char buf1[128];
+__global__ void kern() {
+  ptr1->foo();     // error: virtual function call on a object
+                   //        created in host code.
+  ptr2 = new(buf1) S1();
+}
+
+int main(void) {
+  void *buf;
+  cudaMallocManaged(&buf, sizeof(S1), cudaMemAttachGlobal);
+  ptr1 = new (buf) S1();
+  kern<<<1,1>>>();
+  cudaDeviceSynchronize();
+  ptr2->foo();  // error: virtual function call on an object
+                //        created in device code.
+}
+
+
+
+
+

+18.5.11.4. Virtual Base Classes +

+

It is not allowed to pass as an argument to a __global__ function an object of a class derived from virtual base classes.

+

See Windows-Specific for additional constraints when using the Microsoft host compiler.

+
+
+

+18.5.11.5. Anonymous Unions +

+

Member variables of a namespace scope anonymous union cannot be referenced in a __global__ or __device__ function.

+
+
+

+18.5.11.6. Windows-Specific +

+

The CUDA compiler follows the IA64 ABI for class layout, while the Microsoft host compiler does not. Let T denote a pointer to member type, or a class type that satisfies any of the following conditions:

+
    +
  • T has virtual functions.

  • +
  • T has a virtual base class.

  • +
  • T has multiple inheritance with more than one direct or indirect empty base class.

  • +
  • All direct and indirect base classes B of T are empty and the type of the first field F of T uses B in its definition, such that B is laid out at offset 0 in the definition of F.

  • +
+

Let C denote T or a class type that has T as a field type or as a base class type. The CUDA compiler may compute the class layout and size differently than the Microsoft host compiler for the type C.

+

As long as the type C is used exclusively in host or device code, the program should work correctly.

+

Passing an object of type C between host and device code has undefined behavior, for example, as an argument to a __global__ function or through cudaMemcpy*() calls.

+

Accessing an object of type C or any subobject in device code, or invoking a member function in device code, has undefined behavior if the object is created in host code.

+

Accessing an object of type C or any subobject in host code, or invoking a member function in host code, has undefined behavior if the object is created in device code 12.

+
+
+
+

+18.5.12. Templates +

+

A type or template cannot be used in the type, non-type or template template argument of a __global__ function template instantiation or a __device__/__constant__ variable instantiation if either:

+
    +
  • The type or template is defined within a __host__ or __host__ __device__.

  • +
  • The type or template is a class member with private or protected access and its parent class is not defined within a __device__ or __global__ function.

  • +
  • The type is unnamed.

  • +
  • The type is compounded from any of the types above.

  • +
+

Example:

+
+
template <typename T>
+__global__ void myKernel(void) { }
+
+class myClass {
+private:
+    struct inner_t { };
+public:
+    static void launch(void)
+    {
+       // error: inner_t is used in template argument
+       // but it is private
+       myKernel<inner_t><<<1,1>>>();
+    }
+};
+
+// C++14 only
+template <typename T> __device__ T d1;
+
+template <typename T1, typename T2> __device__ T1 d2;
+
+void fn() {
+  struct S1_t { };
+  // error (C++14 only): S1_t is local to the function fn
+  d1<S1_t> = {};
+
+  auto lam1 = [] { };
+  // error (C++14 only): a closure type cannot be used for
+  // instantiating a variable template
+  d2<int, decltype(lam1)> = 10;
+}
+
+
+
+
+

+18.5.13. Trigraphs and Digraphs +

+

Trigraphs are not supported on any platform. Digraphs are not supported on Windows.

+
+
+

+18.5.14. Const-qualified variables +

+

Let ‘V’ denote a namespace scope variable or a class static member variable that has const qualified type and does not have execution space annotations (for example, __device__, __constant__, __shared__). V is considered to be a host code variable.

+

The value of V may be directly used in device code, if

+
    +
  • V has been initialized with a constant expression before the point of use,

  • +
  • the type of V is not volatile-qualified, and

  • +
  • +

    it has one of the following types:

    +
      +
    • built-in floating point type except when the Microsoft compiler is used as the host compiler,

    • +
    • built-in integral type.

    • +
    +
  • +
+

Device source code cannot contain a reference to V or take the address of V.

+

Example:

+
+
const int xxx = 10;
+struct S1_t {  static const int yyy = 20; };
+
+extern const int zzz;
+const float www = 5.0;
+__device__ void foo(void) {
+  int local1[xxx];          // OK
+  int local2[S1_t::yyy];    // OK
+
+  int val1 = xxx;           // OK
+
+  int val2 = S1_t::yyy;     // OK
+
+  int val3 = zzz;           // error: zzz not initialized with constant
+                            // expression at the point of use.
+
+  const int &val3 = xxx;    // error: reference to host variable
+  const int *val4 = &xxx;   // error: address of host variable
+  const float val5 = www;   // OK except when the Microsoft compiler is used as
+                            // the host compiler.
+}
+const int zzz = 20;
+
+
+
+
+

+18.5.15. Long Double +

+

The use of long double type is not supported in device code.

+
+
+

+18.5.16. Deprecation Annotation +

+

nvcc supports the use of deprecated attribute when using gcc, clang, xlC, icc or pgcc host compilers, and the use of deprecated declspec when using the cl.exe host compiler. It also supports the [[deprecated]] standard attribute when the C++14 dialect has been enabled. The CUDA frontend compiler will generate a deprecation diagnostic for a reference to a deprecated entity from within the body of a __device__, __global__ or __host__ __device__ function when __CUDA_ARCH__ is defined (i.e., during device compilation phase). Other references to deprecated entities will be handled by the host compiler, e.g., a reference from within a __host__ function.

+

The CUDA frontend compiler does not support the #pragma gcc diagnostic or #pragma warning mechanisms supported by various host compilers. Therefore, deprecation diagnostics generated by the CUDA frontend compiler are not affected by these pragmas, but diagnostics generated by the host compiler will be affected. To suppress the warning for device-code, user can use NVIDIA specific pragma #pragma nv_diag_suppress. The nvcc flag -Wno-deprecated-declarations can be used to suppress all deprecation warnings, and the flag -Werror=deprecated-declarations can be used to turn deprecation warnings into errors.

+
+
+

+18.5.17. Noreturn Annotation +

+

nvcc supports the use of noreturn attribute when using gcc, clang, xlC, icc or pgcc host compilers, and the use of noreturn declspec when using the cl.exe host compiler. It also supports the [[noreturn]] standard attribute when the C++11 dialect has been enabled.

+

The attribute/declspec can be used in both host and device code.

+
+
+

+18.5.18. [[likely]] / [[unlikely]] Standard Attributes +

+

These attributes are accepted in all configurations that support the C++ standard attribute syntax. The attributes can be used to hint to the device compiler optimizer whether a statement is more or less likely to be executed compared to any alternative path that does not include the statement.

+

Example:

+
+
__device__ int foo(int x) {
+
+ if (i < 10) [[likely]] { // the 'if' block will likely be entered
+  return 4;
+ }
+ if (i < 20) [[unlikely]] { // the 'if' block will not likely be entered
+  return 1;
+ }
+ return 0;
+}
+
+
+

If these attributes are used in host code when __CUDA_ARCH__ is undefined, then they will be present in the code parsed by the host compiler, which may generate a warning if the attributes are not supported. For example, clang11 host compiler will generate an ‘unknown attribute’ warning.

+
+
+

+18.5.19. const and pure GNU Attributes +

+

These attributes are supported for both host and device functions, when using a language dialect and host compiler that also supports these attributes e.g. with g++ host compiler.

+

For a device function annotated with the pure attribute, the device code optimizer assumes that the function does not change any mutable state visible to caller functions (e.g. memory).

+

For a device function annotated with the const attribute, the device code optimizer assumes that the function does not access or change any mutable state visible to caller functions (e.g. memory).

+

Example:

+
+
__attribute__((const)) __device__ int get(int in);
+
+__device__ int doit(int in) {
+int sum = 0;
+
+//because 'get' is marked with 'const' attribute
+//device code optimizer can recognize that the
+//second call to get() can be commoned out.
+sum = get(in);
+sum += get(in);
+
+return sum;
+}
+
+
+
+
+

+18.5.20. __nv_pure__ Attribute +

+

The __nv_pure__ attributed is supported for both host and device functions. For host functions, when using a language dialect that supports the pure GNU attribute, the __nv_pure__ attribute is translated to the pure GNU attribute. Similarly when using MSVC as the host compiler, the attribute is translated to the MSVC noalias attribute.

+

When a device function is annotated with the __nv_pure__ attribute, the device code optimizer assumes that the function does not change any mutable state visible to caller functions (e.g. memory).

+
+
+

+18.5.21. Intel Host Compiler Specific +

+

The CUDA frontend compiler parser does not recognize some of the intrinsic functions supported by the Intel compiler (e.g. icc). When using the Intel compiler as a host compiler, nvcc will therefore enable the macro __INTEL_COMPILER_USE_INTRINSIC_PROTOTYPES during preprocessing. This macro enables explicit declarations of the Intel compiler intrinsic functions in the associated header files, allowing nvcc to support use of such functions in host code13.

+
+
+

+18.5.22. C++11 Features +

+

C++11 features that are enabled by default by the host compiler are also supported by nvcc, subject to the restrictions described in this document. In addition, invoking nvcc with -std=c++11 flag turns on all C++11 features and also invokes the host preprocessor, compiler and linker with the corresponding C++11 dialect option 14.

+
+

+18.5.22.1. Lambda Expressions +

+

The execution space specifiers for all member functions15 of the closure class associated with a lambda expression are derived by the compiler as follows. As described in the C++11 standard, the compiler creates a closure type in the smallest block scope, class scope or namespace scope that contains the lambda expression. The innermost function scope enclosing the closure type is computed, and the corresponding function’s execution space specifiers are assigned to the closure class member functions. If there is no enclosing function scope, the execution space specifier is __host__.

+

Examples of lambda expressions and computed execution space specifiers are shown below (in comments).

+
+
auto globalVar = [] { return 0; }; // __host__
+
+void f1(void) {
+  auto l1 = [] { return 1; };      // __host__
+}
+
+__device__ void f2(void) {
+  auto l2 = [] { return 2; };      // __device__
+}
+
+__host__ __device__ void f3(void) {
+  auto l3 = [] { return 3; };      // __host__ __device__
+}
+
+__device__ void f4(int (*fp)() = [] { return 4; } /* __host__ */) {
+}
+
+__global__ void f5(void) {
+  auto l5 = [] { return 5; };      // __device__
+}
+
+__device__ void f6(void) {
+  struct S1_t {
+    static void helper(int (*fp)() = [] {return 6; } /* __device__ */) {
+    }
+  };
+}
+
+
+

The closure type of a lambda expression cannot be used in the type or non-type argument of a __global__ function template instantiation, unless the lambda is defined within a __device__ or __global__ function.

+

Example:

+
+
template <typename T>
+__global__ void foo(T in) { };
+
+template <typename T>
+struct S1_t { };
+
+void bar(void) {
+  auto temp1 = [] { };
+
+  foo<<<1,1>>>(temp1);                    // error: lambda closure type used in
+                                          // template type argument
+  foo<<<1,1>>>( S1_t<decltype(temp1)>()); // error: lambda closure type used in
+                                          // template type argument
+}
+
+
+
+
+

+18.5.22.2. std::initializer_list +

+

By default, the CUDA compiler will implicitly consider the member functions of std::initializer_list to have __host__ __device__ execution space specifiers, and therefore they can be invoked directly from device code. The nvcc flag --no-host-device-initializer-list will disable this behavior; member functions of std::initializer_list will then be considered as __host__ functions and will not be directly invokable from device code.

+

Example:

+
+
#include <initializer_list>
+
+__device__ int foo(std::initializer_list<int> in);
+
+__device__ void bar(void)
+  {
+    foo({4,5,6});   // (a) initializer list containing only
+                    // constant expressions.
+
+    int i = 4;
+    foo({i,5,6});   // (b) initializer list with at least one
+                    // non-constant element.
+                    // This form may have better performance than (a).
+  }
+
+
+
+
+

+18.5.22.3. Rvalue references +

+

By default, the CUDA compiler will implicitly consider std::move and std::forward function templates to have __host__ __device__ execution space specifiers, and therefore they can be invoked directly from device code. The nvcc flag --no-host-device-move-forward will disable this behavior; std::move and std::forward will then be considered as __host__ functions and will not be directly invokable from device code.

+
+
+

+18.5.22.4. Constexpr functions and function templates +

+

By default, a constexpr function cannot be called from a function with incompatible execution space 16. The experimental nvcc flag --expt-relaxed-constexpr removes this restriction 17. When this flag is specified, host code can invoke a __device__ constexpr function and device code can invoke a __host__ constexpr function. nvcc will define the macro __CUDACC_RELAXED_CONSTEXPR__ when --expt-relaxed-constexpr has been specified. Note that a function template instantiation may not be a constexpr function even if the corresponding template is marked with the keyword constexpr (C++11 Standard Section [dcl.constexpr.p6]).

+
+
+

+18.5.22.5. Constexpr variables +

+

Let ‘V’ denote a namespace scope variable or a class static member variable that has been marked constexpr and that does not have execution space annotations (e.g., __device__, __constant__, __shared__). V is considered to be a host code variable.

+

If V is of scalar type 18 other than long double and the type is not volatile-qualified, the value of V can be directly used in device code. In addition, if V is of a non-scalar type then scalar elements of V can be used inside a constexpr __device__ or __host__ __device__ function, if the call to the function is a constant expression 19. Device source code cannot contain a reference to V or take the address of V.

+

Example:

+
+
constexpr int xxx = 10;
+constexpr int yyy = xxx + 4;
+struct S1_t { static constexpr int qqq = 100; };
+
+constexpr int host_arr[] = { 1, 2, 3};
+constexpr __device__ int get(int idx) { return host_arr[idx]; }
+
+__device__ int foo(int idx) {
+  int v1 = xxx + yyy + S1_t::qqq;  // OK
+  const int &v2 = xxx;             // error: reference to host constexpr
+                                   // variable
+  const int *v3 = &xxx;            // error: address of host constexpr
+                                   // variable
+  const int &v4 = S1_t::qqq;       // error: reference to host constexpr
+                                   // variable
+  const int *v5 = &S1_t::qqq;      // error: address of host constexpr
+                                   // variable
+
+  v1 += get(2);                    // OK: 'get(2)' is a constant
+                                   // expression.
+  v1 += get(idx);                  // error: 'get(idx)' is not a constant
+                                   // expression
+  v1 += host_arr[2];               // error: 'host_arr' does not have
+                                   // scalar type.
+  return v1;
+}
+
+
+
+
+

+18.5.22.6. Inline namespaces +

+

For an input CUDA translation unit, the CUDA compiler may invoke the host compiler for compiling the host code within the translation unit. In the code passed to the host compiler, the CUDA compiler will inject additional compiler generated code, if the input CUDA translation unit contained a definition of any of the following entities:

+
    +
  • __global__ function or function template instantiation

  • +
  • __device__, __constant__

  • +
  • variables with surface or texture type

  • +
+

The compiler generated code contains a reference to the defined entity. If the entity is defined within an inline namespace and another entity of the same name and type signature is defined in an enclosing namespace, this reference may be considered ambiguous by the host compiler and host compilation will fail.

+

This limitation can be avoided by using unique names for such entities defined within an inline namespace.

+

Example:

+
+
__device__ int Gvar;
+inline namespace N1 {
+  __device__ int Gvar;
+}
+
+// <-- CUDA compiler inserts a reference to "Gvar" at this point in the
+// translation unit. This reference will be considered ambiguous by the
+// host compiler and compilation will fail.
+
+
+

Example:

+
+
inline namespace N1 {
+  namespace N2 {
+    __device__ int Gvar;
+  }
+}
+
+namespace N2 {
+  __device__ int Gvar;
+}
+
+// <-- CUDA compiler inserts reference to "::N2::Gvar" at this point in
+// the translation unit. This reference will be considered ambiguous by
+// the host compiler and compilation will fail.
+
+
+
+
+18.5.22.6.1. Inline unnamed namespaces +
+

The following entities cannot be declared in namespace scope within an inline unnamed namespace:

+
    +
  • __managed__, __device__, __shared__ and __constant__ variables

  • +
  • __global__ function and function templates

  • +
  • variables with surface or texture type

  • +
+

Example:

+
+
inline namespace {
+  namespace N2 {
+    template <typename T>
+    __global__ void foo(void);            // error
+
+    __global__ void bar(void) { }         // error
+
+    template <>
+    __global__ void foo<int>(void) { }    // error
+
+    __device__ int x1b;                   // error
+    __constant__ int x2b;                 // error
+    __shared__ int x3b;                   // error
+
+    texture<int> q2;                      // error
+    surface<int> s2;                      // error
+  }
+};
+
+
+
+
+
+

+18.5.22.7. thread_local +

+

The thread_local storage specifier is not allowed in device code.

+
+
+

+18.5.22.8. __global__ functions and function templates +

+

If the closure type associated with a lambda expression is used in a template argument of a __global__ function template instantiation, the lambda expression must either be defined in the immediate or nested block scope of a __device__ or __global__ function, or must be an extended lambda.

+

Example:

+
+
template <typename T>
+__global__ void kernel(T in) { }
+
+__device__ void foo_device(void)
+{
+  // All kernel instantiations in this function
+  // are valid, since the lambdas are defined inside
+  // a __device__ function.
+
+  kernel<<<1,1>>>( [] __device__ { } );
+  kernel<<<1,1>>>( [] __host__ __device__ { } );
+  kernel<<<1,1>>>( []  { } );
+}
+
+auto lam1 = [] { };
+
+auto lam2 = [] __host__ __device__ { };
+
+void foo_host(void)
+{
+   // OK: instantiated with closure type of an extended __device__ lambda
+   kernel<<<1,1>>>( [] __device__ { } );
+
+   // OK: instantiated with closure type of an extended __host__ __device__
+   // lambda
+   kernel<<<1,1>>>( [] __host__ __device__ { } );
+
+   // error: unsupported: instantiated with closure type of a lambda
+   // that is not an extended lambda
+   kernel<<<1,1>>>( []  { } );
+
+   // error: unsupported: instantiated with closure type of a lambda
+   // that is not an extended lambda
+   kernel<<<1,1>>>( lam1);
+
+   // error: unsupported: instantiated with closure type of a lambda
+   // that is not an extended lambda
+   kernel<<<1,1>>>( lam2);
+}
+
+
+

A __global__ function or function template cannot be declared as constexpr.

+

A __global__ function or function template cannot have a parameter of type std::initializer_list or va_list.

+

A __global__ function cannot have a parameter of rvalue reference type.

+

A variadic __global__ function template has the following restrictions:

+
    +
  • Only a single pack parameter is allowed.

  • +
  • The pack parameter must be listed last in the template parameter list.

  • +
+

Example:

+
+
// ok
+template <template <typename...> class Wrapper, typename... Pack>
+__global__ void foo1(Wrapper<Pack...>);
+
+// error: pack parameter is not last in parameter list
+template <typename... Pack, template <typename...> class Wrapper>
+__global__ void foo2(Wrapper<Pack...>);
+
+// error: multiple parameter packs
+template <typename... Pack1, int...Pack2, template<typename...> class Wrapper1,
+          template<int...> class Wrapper2>
+__global__ void foo3(Wrapper1<Pack1...>, Wrapper2<Pack2...>);
+
+
+
+
+

+18.5.22.9. __managed__ and __shared__ variables +

+

`__managed__ and __shared__ variables cannot be marked with the keyword constexpr.

+
+
+

+18.5.22.10. Defaulted functions +

+

Execution space specifiers on a non-virtual function that is explicitly-defaulted on its first declaration are ignored by the CUDA compiler. Instead, the CUDA compiler will infer the execution space specifiers as described in Implicitly-declared and non-virtual explicitly-defaulted functions.

+

Execution space specifiers are not ignored if the function is either:

+
    +
  • Explicitly-defaulted but not on its first declaration.

  • +
  • Explicitly-defaulted and virtual.

  • +
+

Example:

+
+
 struct S1 {
+   // warning: __host__ annotation is ignored on a non-virtual function that
+   //          is explicitly-defaulted on its first declaration
+   __host__ S1() = default;
+ };
+
+ __device__ void foo1() {
+   //note: __device__ execution space is derived for S1::S1
+   //       based on implicit call from within __device__ function
+   //       foo1
+   S1 s1;
+ }
+
+ struct S2 {
+   __host__ S2();
+ };
+
+ //note: S2::S2 is not defaulted on its first declaration, and
+ //      its execution space is fixed to __host__  based on its
+ //      first declaration.
+ S2::S2() = default;
+
+ __device__ void foo2() {
+    // error: call from __device__ function 'foo2' to
+    //        __host__ function 'S2::S2'
+    S2 s2;
+ }
+
+struct S3 {
+  //note: S3::~S3 has __host__ execution space
+  virtual __host__ ~S3() = default;
+};
+
+__device__ void foo3() {
+  S3 qqq;
+}  /*(implicit destructor call for 'qqq'):
+      error: call from a __device__ fuction 'foo3' to a
+     __host__ function 'S3::~S3' */
+
+
+
+
+
+

+18.5.23. C++14 Features +

+

C++14 features enabled by default by the host compiler are also supported by nvcc. Passing nvcc -std=c++14 flag turns on all C++14 features and also invokes the host preprocessor, compiler and linker with the corresponding C++14 dialect option 20. This section describes the restrictions on the supported C++14 features.

+
+

+18.5.23.1. Functions with deduced return type +

+

A __global__ function cannot have a deduced return type.

+

If a __device__ function has deduced return type, the CUDA frontend compiler will change the function declaration to have a void return type, before invoking the host compiler. This may cause issues for introspecting the deduced return type of the __device__ function in host code. Thus, the CUDA compiler will issue compile-time errors for referencing such deduced return type outside device function bodies, except if the reference is absent when __CUDA_ARCH__ is undefined.

+

Examples:

+
+
__device__ auto fn1(int x) {
+  return x;
+}
+
+__device__ decltype(auto) fn2(int x) {
+  return x;
+}
+
+__device__ void device_fn1() {
+  // OK
+  int (*p1)(int) = fn1;
+}
+
+// error: referenced outside device function bodies
+decltype(fn1(10)) g1;
+
+void host_fn1() {
+  // error: referenced outside device function bodies
+  int (*p1)(int) = fn1;
+
+  struct S_local_t {
+    // error: referenced outside device function bodies
+    decltype(fn2(10)) m1;
+
+    S_local_t() : m1(10) { }
+  };
+}
+
+// error: referenced outside device function bodies
+template <typename T = decltype(fn2)>
+void host_fn2() { }
+
+template<typename T> struct S1_t { };
+
+// error: referenced outside device function bodies
+struct S1_derived_t : S1_t<decltype(fn1)> { };
+
+
+
+
+

+18.5.23.2. Variable templates +

+

A __device__/__constant__ variable template cannot have a const qualified type when using the Microsoft host compiler.

+

Examples:

+
+
// error: a __device__ variable template cannot
+// have a const qualified type on Windows
+template <typename T>
+__device__ const T d1(2);
+
+int *const x = nullptr;
+// error: a __device__ variable template cannot
+// have a const qualified type on Windows
+template <typename T>
+__device__ T *const d2(x);
+
+// OK
+template <typename T>
+__device__ const T *d3;
+
+__device__ void fn() {
+  int t1 = d1<int>;
+
+  int *const t2 = d2<int>;
+
+  const int *t3 = d3<int>;
+}
+
+
+
+
+
+

+18.5.24. C++17 Features +

+

C++17 features enabled by default by the host compiler are also supported by nvcc. Passing nvcc -std=c++17 flag turns on all C++17 features and also invokes the host preprocessor, compiler and linker with the corresponding C++17 dialect option 21. This section describes the restrictions on the supported C++17 features.

+
+

+18.5.24.1. Inline Variable +

+
    +
  • +

    A namespace scope inline variable declared with __device__ or __constant__ or __managed__ memory space specifier must have internal linkage, if the code is compiled with nvcc in whole program compilation mode.

    +

    Examples:

    +
    +
    inline __device__ int xxx; //error when compiled with nvcc in
    +                           //whole program compilation mode.
    +                           //ok when compiled with nvcc in
    +                           //separate compilation mode.
    +
    +inline __shared__ int yyy0; // ok.
    +
    +static inline __device__ int yyy; // ok: internal linkage
    +namespace {
    +inline __device__ int zzz; // ok: internal linkage
    +}
    +
    +
    +
  • +
  • When using g++ host compiler, an inline variable declared with __managed__ memory space specifier may not be visible to the debugger.

  • +
+
+
+

+18.5.24.2. Structured Binding +

+

A structured binding cannot be declared with a variable memory space specifier.

+

Example:

+
+
struct S { int x; int y; };
+__device__ auto [a1, b1] = S{4,5}; // error
+
+
+
+
+
+

+18.5.25. C++20 Features +

+

C++20 features enabled by default by the host compiler are also supported by nvcc. Passing nvcc -std=c++20 flag turns on all C++20 features and also invokes the host preprocessor, compiler and linker with the corresponding C++20 dialect option 22. This section describes the restrictions on the supported C++20 features.

+
+

+18.5.25.1. Module support +

+

Modules are not supported in CUDA C++, in either host or device code. Uses of the module, export and import keywords are diagnosed as errors.

+
+
+

+18.5.25.2. Coroutine support +

+

Coroutines are not supported in device code. Uses of the co_await, co_yield and co_return keywords in the scope of a device function are diagnosed as error during device compilation.

+
+
+

+18.5.25.3. Three-way comparison operator +

+

The three-way comparison operator is supported in both host and device code, but some uses implicitly rely on functionality from the Standard Template Library provided by the host implementation. Uses of those operators may require specifying the flag --expt-relaxed-constexpr to silence warnings and the functionality requires that the host implementation satisfies the requirements of device code.

+

Example:

+
+
#include<compare>
+struct S {
+  int x, y, z;
+  auto operator<=>(const S& rhs) const = default;
+  __host__ __device__ bool operator<=>(int rhs) const { return false; }
+};
+__host__ __device__ bool f(S a, S b) {
+  if (a <=> 1) // ok, calls a user-defined host-device overload
+    return true;
+  return a < b; // call to an implicitly-declared function and requires
+                // a device-compatible std::strong_ordering implementation
+}
+
+
+
+
+

+18.5.25.4. Consteval functions +

+

Ordinarily, cross execution space calls are not allowed, and cause a compiler diagnostic (warning or error). This restriction does not apply when the called function is declared with the consteval specifier. Thus, a __device__ or __global__ function can call a __host__consteval function, and a __host__ function can call a __device__ consteval function.

+

Example:

+
+
namespace N1 {
+//consteval host function
+consteval int hcallee() { return 10; }
+
+__device__ int dfunc() { return hcallee(); /* OK */ }
+__global__ void gfunc() { (void)hcallee(); /* OK */ }
+__host__ __device__ int hdfunc() { return hcallee();  /* OK */ }
+int hfunc() { return hcallee(); /* OK */ }
+} // namespace N1
+
+
+namespace N2 {
+//consteval device function
+consteval __device__ int dcallee() { return 10; }
+
+__device__ int dfunc() { return dcallee(); /* OK */ }
+__global__ void gfunc() { (void)dcallee(); /* OK */ }
+__host__ __device__ int hdfunc() { return dcallee();  /* OK */ }
+int hfunc() { return dcallee(); /* OK */ }
+}
+
+
+
+
+
+
+

+18.6. Polymorphic Function Wrappers +

+

A polymorphic function wrapper class template nvstd::function is provided in the nvfunctional header. Instances of this class template can be used to store, copy and invoke any callable target, e.g., lambda expressions. nvstd::function can be used in both host and device code.

+

Example:

+
+
#include <nvfunctional>
+
+__device__ int foo_d() { return 1; }
+__host__ __device__ int foo_hd () { return 2; }
+__host__ int foo_h() { return 3; }
+
+__global__ void kernel(int *result) {
+  nvstd::function<int()> fn1 = foo_d;
+  nvstd::function<int()> fn2 = foo_hd;
+  nvstd::function<int()> fn3 =  []() { return 10; };
+
+  *result = fn1() + fn2() + fn3();
+}
+
+__host__ __device__ void hostdevice_func(int *result) {
+  nvstd::function<int()> fn1 = foo_hd;
+  nvstd::function<int()> fn2 =  []() { return 10; };
+
+  *result = fn1() + fn2();
+}
+
+__host__ void host_func(int *result) {
+  nvstd::function<int()> fn1 = foo_h;
+  nvstd::function<int()> fn2 = foo_hd;
+  nvstd::function<int()> fn3 =  []() { return 10; };
+
+  *result = fn1() + fn2() + fn3();
+}
+
+
+

Instances of nvstd::function in host code cannot be initialized with the address of a __device__ function or with a functor whose operator() is a __device__ function. Instances of nvstd::function in device code cannot be initialized with the address of a __host__ function or with a functor whose operator() is a __host__ function.

+

nvstd::function instances cannot be passed from host code to device code (and vice versa) at run time. nvstd::function cannot be used in the parameter type of a __global__ function, if the __global__ function is launched from host code.

+

Example:

+
+
#include <nvfunctional>
+
+__device__ int foo_d() { return 1; }
+__host__ int foo_h() { return 3; }
+auto lam_h = [] { return 0; };
+
+__global__ void k(void) {
+  // error: initialized with address of __host__ function
+  nvstd::function<int()> fn1 = foo_h;
+
+  // error: initialized with address of functor with
+  // __host__ operator() function
+  nvstd::function<int()> fn2 = lam_h;
+}
+
+__global__ void kern(nvstd::function<int()> f1) { }
+
+void foo(void) {
+  // error: initialized with address of __device__ function
+  nvstd::function<int()> fn1 = foo_d;
+
+  auto lam_d = [=] __device__ { return 1; };
+
+  // error: initialized with address of functor with
+  // __device__ operator() function
+  nvstd::function<int()> fn2 = lam_d;
+
+  // error: passing nvstd::function from host to device
+  kern<<<1,1>>>(fn2);
+}
+
+
+

nvstd::function is defined in the nvfunctional header as follows:

+
+
namespace nvstd {
+  template <class _RetType, class ..._ArgTypes>
+  class function<_RetType(_ArgTypes...)>
+  {
+    public:
+      // constructors
+      __device__ __host__  function() noexcept;
+      __device__ __host__  function(nullptr_t) noexcept;
+      __device__ __host__  function(const function &);
+      __device__ __host__  function(function &&);
+
+      template<class _F>
+      __device__ __host__  function(_F);
+
+      // destructor
+      __device__ __host__  ~function();
+
+      // assignment operators
+      __device__ __host__  function& operator=(const function&);
+      __device__ __host__  function& operator=(function&&);
+      __device__ __host__  function& operator=(nullptr_t);
+      __device__ __host__  function& operator=(_F&&);
+
+      // swap
+      __device__ __host__  void swap(function&) noexcept;
+
+      // function capacity
+      __device__ __host__  explicit operator bool() const noexcept;
+
+      // function invocation
+      __device__ _RetType operator()(_ArgTypes...) const;
+  };
+
+  // null pointer comparisons
+  template <class _R, class... _ArgTypes>
+  __device__ __host__
+  bool operator==(const function<_R(_ArgTypes...)>&, nullptr_t) noexcept;
+
+  template <class _R, class... _ArgTypes>
+  __device__ __host__
+  bool operator==(nullptr_t, const function<_R(_ArgTypes...)>&) noexcept;
+
+  template <class _R, class... _ArgTypes>
+  __device__ __host__
+  bool operator!=(const function<_R(_ArgTypes...)>&, nullptr_t) noexcept;
+
+  template <class _R, class... _ArgTypes>
+  __device__ __host__
+  bool operator!=(nullptr_t, const function<_R(_ArgTypes...)>&) noexcept;
+
+  // specialized algorithms
+  template <class _R, class... _ArgTypes>
+  __device__ __host__
+  void swap(function<_R(_ArgTypes...)>&, function<_R(_ArgTypes...)>&);
+}
+
+
+
+
+

+18.7. Extended Lambdas +

+

The nvcc flag '--extended-lambda' allows explicit execution space annotations in a lambda expression 23. The execution space annotations should be present after the ‘lambda-introducer’ and before the optional ‘lambda-declarator’. nvcc will define the macro __CUDACC_EXTENDED_LAMBDA__ when the '--extended-lambda' flag has been specified.

+

An ‘extended __device__ lambda’ is a lambda expression that is annotated explicitly with ‘__device__’, and is defined within the immediate or nested block scope of a __host__ or __host__ __device__ function.

+

An ‘extended __host__ __device__ lambda’ is a lambda expression that is annotated explicitly with both ‘__host__’ and ‘__device__’, and is defined within the immediate or nested block scope of a __host__ or __host__ __device__ function.

+

An ‘extended lambda’ denotes either an extended __device__ lambda or an extended __host__ __device__ lambda. Extended lambdas can be used in the type arguments of __global__ function template instantiation.

+

If the execution space annotations are not explicitly specified, they are computed based on the scopes enclosing the closure class associated with the lambda, as described in the section on C++11 support. The execution space annotations are applied to all methods of the closure class associated with the lambda.

+

Example:

+
+
void foo_host(void) {
+  // not an extended lambda: no explicit execution space annotations
+  auto lam1 = [] { };
+
+  // extended __device__ lambda
+  auto lam2 = [] __device__ { };
+
+  // extended __host__ __device__ lambda
+  auto lam3 = [] __host__ __device__ { };
+
+  // not an extended lambda: explicitly annotated with only '__host__'
+  auto lam4 = [] __host__ { };
+}
+
+__host__ __device__ void foo_host_device(void) {
+  // not an extended lambda: no explicit execution space annotations
+  auto lam1 = [] { };
+
+  // extended __device__ lambda
+  auto lam2 = [] __device__ { };
+
+  // extended __host__ __device__ lambda
+  auto lam3 = [] __host__ __device__ { };
+
+  // not an extended lambda: explicitly annotated with only '__host__'
+  auto lam4 = [] __host__ { };
+}
+
+__device__ void foo_device(void) {
+  // none of the lambdas within this function are extended lambdas,
+  // because the enclosing function is not a __host__ or __host__ __device__
+  // function.
+  auto lam1 = [] { };
+  auto lam2 = [] __device__ { };
+  auto lam3 = [] __host__ __device__ { };
+  auto lam4 = [] __host__ { };
+}
+
+// lam1 and lam2 are not extended lambdas because they are not defined
+// within a __host__ or __host__ __device__ function.
+auto lam1 = [] { };
+auto lam2 = [] __host__ __device__ { };
+
+
+
+

+18.7.1. Extended Lambda Type Traits +

+

The compiler provides type traits to detect closure types for extended lambdas at compile time:

+

__nv_is_extended_device_lambda_closure_type(type): If ‘type’ is the closure class created for an extended __device__ lambda, then the trait is true, otherwise it is false.

+

__nv_is_extended_device_lambda_with_preserved_return_type(type): If ‘type’ is the closure class created for an extended __device__ lambda and the lambda is defined with trailing return type (with restriction), then the trait is true, otherwise it is false. If the trailing return type definition refers to any lambda parameter name, the return type is not preserved.

+

__nv_is_extended_host_device_lambda_closure_type(type): If ‘type’ is the closure class created for an extended __host__ __device__ lambda, then the trait is true, otherwise it is false.

+

These traits can be used in all compilation modes, irrespective of whether lambdas or extended lambdas are enabled24.

+

Example:

+
+
#define IS_D_LAMBDA(X) __nv_is_extended_device_lambda_closure_type(X)
+#define IS_DPRT_LAMBDA(X) __nv_is_extended_device_lambda_with_preserved_return_type(X)
+#define IS_HD_LAMBDA(X) __nv_is_extended_host_device_lambda_closure_type(X)
+
+auto lam0 = [] __host__ __device__ { };
+
+void foo(void) {
+  auto lam1 = [] { };
+  auto lam2 = [] __device__ { };
+  auto lam3 = [] __host__ __device__ { };
+  auto lam4 = [] __device__ () --> double { return 3.14; }
+  auto lam5 = [] __device__ (int x) --> decltype(&x) { return 0; }
+
+  // lam0 is not an extended lambda (since defined outside function scope)
+  static_assert(!IS_D_LAMBDA(decltype(lam0)), "");
+  static_assert(!IS_DPRT_LAMBDA(decltype(lam0)), "");
+  static_assert(!IS_HD_LAMBDA(decltype(lam0)), "");
+
+  // lam1 is not an extended lambda (since no execution space annotations)
+  static_assert(!IS_D_LAMBDA(decltype(lam1)), "");
+  static_assert(!IS_DPRT_LAMBDA(decltype(lam1)), "");
+  static_assert(!IS_HD_LAMBDA(decltype(lam1)), "");
+
+  // lam2 is an extended __device__ lambda
+  static_assert(IS_D_LAMBDA(decltype(lam2)), "");
+  static_assert(!IS_DPRT_LAMBDA(decltype(lam2)), "");
+  static_assert(!IS_HD_LAMBDA(decltype(lam2)), "");
+
+  // lam3 is an extended __host__ __device__ lambda
+  static_assert(!IS_D_LAMBDA(decltype(lam3)), "");
+  static_assert(!IS_DPRT_LAMBDA(decltype(lam3)), "");
+  static_assert(IS_HD_LAMBDA(decltype(lam3)), "");
+
+  // lam4 is an extended __device__ lambda with preserved return type
+  static_assert(IS_D_LAMBDA(decltype(lam4)), "");
+  static_assert(IS_DPRT_LAMBDA(decltype(lam4)), "");
+  static_assert(!IS_HD_LAMBDA(decltype(lam4)), "");
+
+  // lam5 is not an extended __device__ lambda with preserved return type
+  // because it references the operator()'s parameter types in the trailing return type.
+  static_assert(IS_D_LAMBDA(decltype(lam5)), "");
+  static_assert(!IS_DPRT_LAMBDA(decltype(lam5)), "");
+  static_assert(!IS_HD_LAMBDA(decltype(lam5)), "");
+}
+
+
+
+
+

+18.7.2. Extended Lambda Restrictions +

+

The CUDA compiler will replace an extended lambda expression with an instance of a placeholder type defined in namespace scope, before invoking the host compiler. The template argument of the placeholder type requires taking the address of a function enclosing the original extended lambda expression. This is required for the correct execution of any __global__ function template whose template argument involves the closure type of an extended lambda. The enclosing function is computed as follows.

+

By definition, the extended lambda is present within the immediate or nested block scope of a __host__ or __host__ __device__ function. If this function is not the operator() of a lambda expression, then it is considered the enclosing function for the extended lambda. Otherwise, the extended lambda is defined within the immediate or nested block scope of the operator() of one or more enclosing lambda expressions. If the outermost such lambda expression is defined in the immediate or nested block scope of a function F, then F is the computed enclosing function, else the enclosing function does not exist.

+

Example:

+
+
void foo(void) {
+  // enclosing function for lam1 is "foo"
+  auto lam1 = [] __device__ { };
+
+  auto lam2 = [] {
+     auto lam3 = [] {
+        // enclosing function for lam4 is "foo"
+        auto lam4 = [] __host__ __device__ { };
+     };
+  };
+}
+
+auto lam6 = [] {
+  // enclosing function for lam7 does not exist
+  auto lam7 = [] __host__ __device__ { };
+};
+
+
+

Here are the restrictions on extended lambdas:

+
    +
  1. +

    An extended lambda cannot be defined inside another extended lambda expression.

    +

    Example:

    +
    +
    void foo(void) {
    +  auto lam1 = [] __host__ __device__  {
    +    // error: extended lambda defined within another extended lambda
    +    auto lam2 = [] __host__ __device__ { };
    +  };
    +}
    +
    +
    +
  2. +
  3. +

    An extended lambda cannot be defined inside a generic lambda expression.

    +

    Example:

    +
    +
    void foo(void) {
    +  auto lam1 = [] (auto) {
    +    // error: extended lambda defined within a generic lambda
    +    auto lam2 = [] __host__ __device__ { };
    +  };
    +}
    +
    +
    +
  4. +
  5. +

    If an extended lambda is defined within the immediate or nested block scope of one or more nested lambda expression, the outermost such lambda expression must be defined inside the immediate or nested block scope of a function.

    +

    Example:

    +
    +
    auto lam1 = []  {
    +  // error: outer enclosing lambda is not defined within a
    +  // non-lambda-operator() function.
    +  auto lam2 = [] __host__ __device__ { };
    +};
    +
    +
    +
  6. +
  7. +

    The enclosing function for the extended lambda must be named and its address can be taken. If the enclosing function is a class member, then the following conditions must be satisfied:

    +
      +
    • All classes enclosing the member function must have a name.

    • +
    • The member function must not have private or protected access within its parent class.

    • +
    • All enclosing classes must not have private or protected access within their respective parent classes.

    • +
    +

    Example:

    +
    +
    void foo(void) {
    +  // OK
    +  auto lam1 = [] __device__ { return 0; };
    +  {
    +    // OK
    +    auto lam2 = [] __device__ { return 0; };
    +    // OK
    +    auto lam3 = [] __device__ __host__ { return 0; };
    +  }
    +}
    +
    +struct S1_t {
    +  S1_t(void) {
    +    // Error: cannot take address of enclosing function
    +    auto lam4 = [] __device__ { return 0; };
    +  }
    +};
    +
    +class C0_t {
    +  void foo(void) {
    +    // Error: enclosing function has private access in parent class
    +    auto temp1 = [] __device__ { return 10; };
    +  }
    +  struct S2_t {
    +    void foo(void) {
    +      // Error: enclosing class S2_t has private access in its
    +      // parent class
    +      auto temp1 = [] __device__ { return 10; };
    +    }
    +  };
    +};
    +
    +
    +
  8. +
  9. +

    It must be possible to take the address of the enclosing routine unambiguously, at the point where the extended lambda has been defined. This may not be feasible in some cases e.g. when a class typedef shadows a template type argument of the same name.

    +

    Example:

    +
    +
    template <typename> struct A {
    +  typedef void Bar;
    +  void test();
    +};
    +
    +template<> struct A<void> { };
    +
    +template <typename Bar>
    +void A<Bar>::test() {
    +  /* In code sent to host compiler, nvcc will inject an
    +     address expression here, of the form:
    +     (void (A< Bar> ::*)(void))(&A::test))
    +
    +     However, the class typedef 'Bar' (to void) shadows the
    +     template argument 'Bar', causing the address
    +     expression in A<int>::test to actually refer to:
    +     (void (A< void> ::*)(void))(&A::test))
    +
    +     ..which doesn't take the address of the enclosing
    +     routine 'A<int>::test' correctly.
    +  */
    +  auto lam1 = [] __host__ __device__ { return 4; };
    +}
    +
    +int main() {
    +  A<int> xxx;
    +  xxx.test();
    +}
    +
    +
    +
  10. +
  11. +

    An extended lambda cannot be defined in a class that is local to a function.

    +

    Example:

    +
    +
    void foo(void) {
    +  struct S1_t {
    +    void bar(void) {
    +      // Error: bar is member of a class that is local to a function.
    +      auto lam4 = [] __host__ __device__ { return 0; };
    +    }
    +  };
    +}
    +
    +
    +
  12. +
  13. +

    The enclosing function for an extended lambda cannot have deduced return type.

    +

    Example:

    +
    +
    auto foo(void) {
    +  // Error: the return type of foo is deduced.
    +  auto lam1 = [] __host__ __device__ { return 0; };
    +}
    +
    +
    +
  14. +
  15. +

    __host__ __device__ extended lambdas cannot be generic lambdas.

    +

    Example:

    +
    +
    void foo(void) {
    +  // Error: __host__ __device__ extended lambdas cannot be
    +  // generic lambdas.
    +  auto lam1 = [] __host__ __device__ (auto i) { return i; };
    +
    +  // Error: __host__ __device__ extended lambdas cannot be
    +  // generic lambdas.
    +  auto lam2 = [] __host__ __device__ (auto ...i) {
    +               return sizeof...(i);
    +              };
    +}
    +
    +
    +
  16. +
  17. +

    If the enclosing function is an instantiation of a function template or a member function template, and/or the function is a member of a class template, the template(s) must satisfy the following constraints:

    +
      +
    • The template must have at most one variadic parameter, and it must be listed last in the template parameter list.

    • +
    • The template parameters must be named.

    • +
    • The template instantiation argument types cannot involve types that are either local to a function (except for closure types for extended lambdas), or are private or protected class members.

    • +
    +

    Example:

    +
    +
    template <typename T>
    +__global__ void kern(T in) { in(); }
    +
    +template <typename... T>
    +struct foo {};
    +
    +template < template <typename...> class T, typename... P1,
    +          typename... P2>
    +void bar1(const T<P1...>, const T<P2...>) {
    +  // Error: enclosing function has multiple parameter packs
    +  auto lam1 =  [] __device__ { return 10; };
    +}
    +
    +template < template <typename...> class T, typename... P1,
    +          typename T2>
    +void bar2(const T<P1...>, T2) {
    +  // Error: for enclosing function, the
    +  // parameter pack is not last in the template parameter list.
    +  auto lam1 =  [] __device__ { return 10; };
    +}
    +
    +template <typename T, T>
    +void bar3(void) {
    +  // Error: for enclosing function, the second template
    +  // parameter is not named.
    +  auto lam1 =  [] __device__ { return 10; };
    +}
    +
    +int main() {
    +  foo<char, int, float> f1;
    +  foo<char, int> f2;
    +  bar1(f1, f2);
    +  bar2(f1, 10);
    +  bar3<int, 10>();
    +}
    +
    +
    +

    Example:

    +
    +
    template <typename T>
    +__global__ void kern(T in) { in(); }
    +
    +template <typename T>
    +void bar4(void) {
    +  auto lam1 =  [] __device__ { return 10; };
    +  kern<<<1,1>>>(lam1);
    +}
    +
    +struct C1_t { struct S1_t { }; friend int main(void); };
    +int main() {
    +  struct S1_t { };
    +  // Error: enclosing function for device lambda in bar4
    +  // is instantiated with a type local to main.
    +  bar4<S1_t>();
    +
    +  // Error: enclosing function for device lambda in bar4
    +  // is instantiated with a type that is a private member
    +  // of a class.
    +  bar4<C1_t::S1_t>();
    +}
    +
    +
    +
  18. +
  19. With Visual Studio host compilers, the enclosing function must have external linkage. The restriction is present because this host compiler does not support using the address of non-extern linkage functions as template arguments, which is needed by the CUDA compiler transformations to support extended lambdas.

  20. +
  21. With Visual Studio host compilers, an extended lambda shall not be defined within the body of an ‘if-constexpr’ block.

  22. +
  23. +

    An extended lambda has the following restrictions on captured variables:

    +
      +
    • In the code sent to the host compiler, the variable may be passed by value to a sequence of helper functions before being used to direct-initialize the field of the class type used to represent the closure type for the extended lambda25.

    • +
    • A variable can only be captured by value.

    • +
    • A variable of array type cannot be captured if the number of array dimensions is greater than 7.

    • +
    • For a variable of array type, in the code sent to the host compiler, the closure type’s array field is first default-initialized, and then each element of the array field is copy-assigned from the corresponding element of the captured array variable. Therefore, the array element type must be default-constructible and copy-assignable in host code.

    • +
    • A function parameter that is an element of a variadic argument pack cannot be captured.

    • +
    • The type of the captured variable cannot involve types that are either local to a function (except for closure types of extended lambdas), or are private or protected class members.

    • +
    • For a __host__ __device__ extended lambda, the types used in the return or parameter types of the lambda expression’s operator() cannot involve types that are either local to a function (except for closure types of extended lambdas), or are private or protected class members.

    • +
    • Init-capture is not supported for __host__ __device__ extended lambdas. Init-capture is supported for __device__ extended lambdas, except when the init-capture is of array type or of type std::initializer_list.

    • +
    • The function call operator for an extended lambda is not constexpr. The closure type for an extended lambda is not a literal type. The constexpr and consteval specifier cannot be used in the declaration of an extended lambda.

    • +
    • A variable cannot be implicitly captured inside an if-constexpr block lexically nested inside an extended lambda, unless it has already been implicitly captured earlier outside the if-constexpr block or appears in the explicit capture list for the extended lambda (see example below).

    • +
    +

    Example

    +
    +
    void foo(void) {
    +  // OK: an init-capture is allowed for an
    +  // extended __device__ lambda.
    +  auto lam1 = [x = 1] __device__ () { return x; };
    +
    +  // Error: an init-capture is not allowed for
    +  // an extended __host__ __device__ lambda.
    +  auto lam2 = [x = 1] __host__ __device__ () { return x; };
    +
    +  int a = 1;
    +  // Error: an extended __device__ lambda cannot capture
    +  // variables by reference.
    +  auto lam3 = [&a] __device__ () { return a; };
    +
    +  // Error: by-reference capture is not allowed
    +  // for an extended __device__ lambda.
    +  auto lam4 = [&x = a] __device__ () { return x; };
    +
    +  struct S1_t { };
    +  S1_t s1;
    +  // Error: a type local to a function cannot be used in the type
    +  // of a captured variable.
    +  auto lam6 = [s1] __device__ () { };
    +
    +  // Error: an init-capture cannot be of type std::initializer_list.
    +  auto lam7 = [x = {11}] __device__ () { };
    +
    +  std::initializer_list<int> b = {11,22,33};
    +  // Error: an init-capture cannot be of type std::initializer_list.
    +  auto lam8 = [x = b] __device__ () { };
    +
    +  // Error scenario (lam9) and supported scenarios (lam10, lam11)
    +  // for capture within 'if-constexpr' block
    +  int yyy = 4;
    +  auto lam9 = [=] __device__ {
    +    int result = 0;
    +    if constexpr(false) {
    +      //Error: An extended __device__ lambda cannot first-capture
    +      //      'yyy' in constexpr-if context
    +      result += yyy;
    +    }
    +    return result;
    +  };
    +
    +  auto lam10 = [yyy] __device__ {
    +    int result = 0;
    +    if constexpr(false) {
    +      //OK: 'yyy' already listed in explicit capture list for the extended lambda
    +      result += yyy;
    +    }
    +    return result;
    +  };
    +
    +  auto lam11 = [=] __device__ {
    +    int result = yyy;
    +    if constexpr(false) {
    +      //OK: 'yyy' already implicit captured outside the 'if-constexpr' block
    +      result += yyy;
    +    }
    +    return result;
    +  };
    +}
    +
    +
    +
  24. +
  25. +

    When parsing a function, the CUDA compiler assigns a counter value to each extended lambda within that function. This counter value is used in the substituted named type passed to the host compiler. Hence, whether or not an extended lambda is defined within a function should not depend on a particular value of __CUDA_ARCH__, or on __CUDA_ARCH__ being undefined.

    +

    Example

    +
    +
    template <typename T>
    +__global__ void kernel(T in) { in(); }
    +
    +__host__ __device__ void foo(void) {
    +  // Error: the number and relative declaration
    +  // order of extended lambdas depends on
    +  // __CUDA_ARCH__
    +#if defined(__CUDA_ARCH__)
    +  auto lam1 = [] __device__ { return 0; };
    +  auto lam1b = [] __host___ __device__ { return 10; };
    +#endif
    +  auto lam2 = [] __device__ { return 4; };
    +  kernel<<<1,1>>>(lam2);
    +}
    +
    +
    +
  26. +
  27. +

    As described above, the CUDA compiler replaces a __device__ extended lambda defined in a host function with a placeholder type defined in namespace scope. Unless the trait __nv_is_extended_device_lambda_with_preserved_return_type() returns true for the closure type of the extended lambda, the placeholder type does not define a operator() function equivalent to the original lambda declaration. An attempt to determine the return type or parameter types of the operator() function of such a lambda may therefore work incorrectly in host code, as the code processed by the host compiler will be semantically different than the input code processed by the CUDA compiler. However, it is OK to introspect the return type or parameter types of the operator() function within device code. Note that this restriction does not apply to __host__ __device__ extended lambdas, or to __device__ extended lambdas for which the trait __nv_is_extended_device_lambda_with_preserved_return_type() returns true.

    +

    Example

    +
    +
    #include <type_traits>
    +const char& getRef(const char* p) { return *p; }
    +
    +void foo(void) {
    +  auto lam1 = [] __device__ { return "10"; };
    +
    +  // Error: attempt to extract the return type
    +  // of a __device__ lambda in host code
    +  std::result_of<decltype(lam1)()>::type xx1 = "abc";
    +
    +
    +  auto lam2 = [] __host__ __device__  { return "10"; };
    +
    +  // OK : lam2 represents a __host__ __device__ extended lambda
    +  std::result_of<decltype(lam2)()>::type xx2 = "abc";
    +
    +  auto lam3 = []  __device__ () -> const char * { return "10"; };
    +
    +  // OK : lam3 represents a __device__ extended lambda with preserved return type
    +  std::result_of<decltype(lam3)()>::type xx2 = "abc";
    +  static_assert( std::is_same_v< std::result_of<decltype(lam3)()>::type, const char *>);
    +
    +  auto lam4 = [] __device__ (char x) -> decltype(getRef(&x)) { return 0; };
    +  // lam4's return type is not preserved because it references the operator()'s
    +  // parameter types in the trailing return type.
    +  static_assert( ! __nv_is_extended_device_lambda_with_preserved_return_type(decltype(lam4)), "" );
    +}
    +
    +
    +
  28. +
  29. For an extended device lambda: +- Introspecting the parameter type of operator() is only supported in device code. +- Introspecting the return type of operator() is supported only in device code, unless the trait function __nv_is_extended_device_lambda_with_preserved_return_type() returns true.

  30. +
  31. +

    If the functor object represented by an extended lambda is passed from host to device code (e.g., as the argument of a __global__ function), then any expression in the body of the lambda expression that captures variables must be remain unchanged irrespective of whether the __CUDA_ARCH__ macro is defined, and whether the macro has a particular value. This restriction arises because the lambda’s closure class layout depends on the order in which captured variables are encountered when the compiler processes the lambda expression; the program may execute incorrectly if the closure class layout differs in device and host compilation.

    +

    Example

    +
    +
    __device__ int result;
    +
    +template <typename T>
    +__global__ void kernel(T in) { result = in(); }
    +
    +void foo(void) {
    +  int x1 = 1;
    +  auto lam1 = [=] __host__ __device__ {
    +    // Error: "x1" is only captured when __CUDA_ARCH__ is defined.
    +#ifdef __CUDA_ARCH__
    +    return x1 + 1;
    +#else
    +    return 10;
    +#endif
    +  };
    +  kernel<<<1,1>>>(lam1);
    +}
    +
    +
    +
  32. +
  33. +

    As described previously, the CUDA compiler replaces an extended __device__ lambda expression with an instance of a placeholder type in the code sent to the host compiler. This placeholder type does not define a pointer-to-function conversion operator in host code, however the conversion operator is provided in device code. Note that this restriction does not apply to __host__ __device__ extended lambdas.

    +

    Example

    +
    +
    template <typename T>
    +__global__ void kern(T in) {
    +  int (*fp)(double) = in;
    +
    +  // OK: conversion in device code is supported
    +  fp(0);
    +  auto lam1 = [](double) { return 1; };
    +
    +  // OK: conversion in device code is supported
    +  fp = lam1;
    +  fp(0);
    +}
    +
    +void foo(void) {
    +  auto lam_d = [] __device__ (double) { return 1; };
    +  auto lam_hd = [] __host__ __device__ (double) { return 1; };
    +  kern<<<1,1>>>(lam_d);
    +  kern<<<1,1>>>(lam_hd);
    +
    +  // OK : conversion for __host__ __device__ lambda is supported
    +  // in host code
    +  int (*fp)(double) = lam_hd;
    +
    +  // Error: conversion for __device__ lambda is not supported in
    +  // host code.
    +  int (*fp2)(double) = lam_d;
    +}
    +
    +
    +
  34. +
  35. +

    As described previously, the CUDA compiler replaces an extended __device__ or __host__ __device__ lambda expression with an instance of a placeholder type in the code sent to the host compiler. This placeholder type may define C++ special member functions (e.g. constructor, destructor). As a result, some standard C++ type traits may return different results for the closure type of the extended lambda, in the CUDA frontend compiler versus the host compiler. The following type traits are affected: std::is_trivially_copyable, std::is_trivially_constructible, std::is_trivially_copy_constructible, std::is_trivially_move_constructible, std::is_trivially_destructible.

    +

    Care must be taken that the results of these type traits are not used in __global__ function template instantiation or in __device__ / __constant__ / __managed__ variable template instantiation.

    +

    Example

    +
    +
    template <bool b>
    +void __global__ foo() { printf("hi"); }
    +
    +template <typename T>
    +void dolaunch() {
    +
    +// ERROR: this kernel launch may fail, because CUDA frontend compiler
    +// and host compiler may disagree on the result of
    +// std::is_trivially_copyable() trait on the closure type of the
    +// extended lambda
    +foo<std::is_trivially_copyable<T>::value><<<1,1>>>();
    +cudaDeviceSynchronize();
    +}
    +
    +int main() {
    +int x = 0;
    +auto lam1 = [=] __host__ __device__ () { return x; };
    +dolaunch<decltype(lam1)>();
    +}
    +
    +
    +
  36. +
+

The CUDA compiler will generate compiler diagnostics for a subset of cases described in 1-12; no diagnostic will be generated for cases 13-17, but the host compiler may fail to compile the generated code.

+
+
+

+18.7.3. Notes on __host__ __device__ lambdas +

+

Unlike __device__ lambdas, __host__ __device__ lambdas can be called from host code. As described earlier, the CUDA compiler replaces an extended lambda expression defined in host code with an instance of a named placeholder type. The placeholder type for an extended __host__ __device__ lambda invokes the original lambda’s operator() with an indirect function call 24.

+

The presence of the indirect function call may cause an extended __host__ __device__ lambda to be less optimized by the host compiler than lambdas that are implicitly or explicitly __host__ only. In the latter case, the host compiler can easily inline the body of the lambda into the calling context. But in case of an extended __host__                                  __device__ lambda, the host compiler encounters the indirect function call and may not be able to easily inline the original __host__ __device__ lambda body.

+
+
+

+18.7.4. *this Capture By Value +

+

When a lambda is defined within a non-static class member function, and the body of the lambda refers to a class member variable, C++11/C++14 rules require that the this pointer of the class is captured by value, instead of the referenced member variable. If the lambda is an extended __device__ or __host____device__ lambda defined in a host function, and the lambda is executed on the GPU, accessing the referenced member variable on the GPU will cause a run time error if the this pointer points to host memory.

+

Example:

+
+
#include <cstdio>
+
+template <typename T>
+__global__ void foo(T in) { printf("\n value = %d", in()); }
+
+struct S1_t {
+  int xxx;
+  __host__ __device__ S1_t(void) : xxx(10) { };
+
+  void doit(void) {
+
+    auto lam1 = [=] __device__ {
+       // reference to "xxx" causes
+       // the 'this' pointer (S1_t*) to be captured by value
+       return xxx + 1;
+
+    };
+
+    // Kernel launch fails at run time because 'this->xxx'
+    // is not accessible from the GPU
+    foo<<<1,1>>>(lam1);
+    cudaDeviceSynchronize();
+  }
+};
+
+int main(void) {
+  S1_t s1;
+  s1.doit();
+}
+
+
+

C++17 solves this problem by adding a new “*this” capture mode. In this mode, the compiler makes a copy of the object denoted by “*this” instead of capturing the pointer this by value. The “*this” capture mode is described in more detail here: http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2016/p0018r3.html .

+

The CUDA compiler supports the “*this” capture mode for lambdas defined within __device__ and __global__ functions and for extended __device__ lambdas defined in host code, when the --extended-lambda nvcc flag is used.

+

Here’s the above example modified to use “*this” capture mode:

+
+
#include <cstdio>
+
+template <typename T>
+__global__ void foo(T in) { printf("\n value = %d", in()); }
+
+struct S1_t {
+  int xxx;
+  __host__ __device__ S1_t(void) : xxx(10) { };
+
+  void doit(void) {
+
+    // note the "*this" capture specification
+    auto lam1 = [=, *this] __device__ {
+
+       // reference to "xxx" causes
+       // the object denoted by '*this' to be captured by
+       // value, and the GPU code will access copy_of_star_this->xxx
+       return xxx + 1;
+
+    };
+
+    // Kernel launch succeeds
+    foo<<<1,1>>>(lam1);
+    cudaDeviceSynchronize();
+  }
+};
+
+int main(void) {
+  S1_t s1;
+  s1.doit();
+}
+
+
+

“*this” capture mode is not allowed for unannotated lambdas defined in host code, or for extended __host____device__ lambdas, unless “*this” capture is enabled by the selected language dialect. Examples of supported and unsupported usage:

+
+
struct S1_t {
+  int xxx;
+  __host__ __device__ S1_t(void) : xxx(10) { };
+
+  void host_func(void) {
+
+    // OK: use in an extended __device__ lambda
+    auto lam1 = [=, *this] __device__ { return xxx; };
+
+    // Use in an extended __host__ __device__ lambda
+    // Error if *this capture not enabled by language dialect
+    auto lam2 = [=, *this] __host__ __device__ { return xxx; };
+
+    // Use in an unannotated lambda in host function
+    // Error if *this capture not enabled by language dialect
+    auto lam3 = [=, *this]  { return xxx; };
+  }
+
+  __device__ void device_func(void) {
+
+    // OK: use in a lambda defined in a __device__ function
+    auto lam1 = [=, *this] __device__ { return xxx; };
+
+    // OK: use in a lambda defined in a __device__ function
+    auto lam2 = [=, *this] __host__ __device__ { return xxx; };
+
+    // OK: use in a lambda defined in a __device__ function
+    auto lam3 = [=, *this]  { return xxx; };
+  }
+
+   __host__ __device__ void host_device_func(void) {
+
+    // OK: use in an extended __device__ lambda
+    auto lam1 = [=, *this] __device__ { return xxx; };
+
+    // Use in an extended __host__ __device__ lambda
+    // Error if *this capture not enabled by language dialect
+    auto lam2 = [=, *this] __host__ __device__ { return xxx; };
+
+    // Use in an unannotated lambda in a __host__ __device__ function
+    // Error if *this capture not enabled by language dialect
+    auto lam3 = [=, *this]  { return xxx; };
+  }
+};
+
+
+
+
+

+18.7.5. Additional Notes +

+
    +
  1. +

    ADL Lookup: As described earlier, the CUDA compiler will replace an extended lambda expression with an instance of a placeholder type, before invoking the host compiler. One template argument of the placeholder type uses the address of the function enclosing the original lambda expression. This may cause additional namespaces to participate in argument dependent lookup (ADL), for any host function call whose argument types involve the closure type of the extended lambda expression. This may cause an incorrect function to be selected by the host compiler.

    +

    Example:

    +
    +
    namespace N1 {
    +  struct S1_t { };
    +  template <typename T>  void foo(T);
    +};
    +
    +namespace N2 {
    +  template <typename T> int foo(T);
    +
    +  template <typename T>  void doit(T in) {     foo(in);  }
    +}
    +
    +void bar(N1::S1_t in) {
    +  /* extended __device__ lambda. In the code sent to the host compiler, this
    +     is replaced with the placeholder type instantiation expression
    +     ' __nv_dl_wrapper_t< __nv_dl_tag<void (*)(N1::S1_t in),(&bar),1> > { }'
    +
    +     As a result, the namespace 'N1' participates in ADL lookup of the
    +     call to "foo" in the body of N2::doit, causing ambiguity.
    +  */
    +  auto lam1 = [=] __device__ { };
    +  N2::doit(lam1);
    +}
    +
    +
    +

    In the example above, the CUDA compiler replaced the extended lambda with a placeholder type that involves the N1 namespace. As a result, the namespace N1 participates in the ADL lookup for foo(in) in the body of N2::doit, and host compilation fails because multiple overload candidates N1::foo and N2::foo are found.

    +
  2. +
+
+
+
+

+18.8. Relaxed Constexpr (-expt-relaxed-constexpr) +

+

By default, the following cross-execution space calls are not supported:

+
    +
  1. +

    Calling a __device__-only constexpr function from a __host__ function during host code generation phase (i.e, when __CUDA_ARCH__ macro is undefined). Example:

    +
    +
    +
    +
    constexpr __device__ int D() { return 0; }
    +int main() {
    +    int x = D();  //ERROR: calling a __device__-only constexpr function from host code
    +}
    +
    +
    +
    +
    +
  2. +
  3. +

    Calling a __host__-only constexpr function from a __device__ or __global__ function, during device code generation phase (i.e. when __CUDA_ARCH__ macro is defined). Example:

    +
    +
    +
    +
    constexpr  int H() { return 0; }
    +__device__ void dmain()
    +{
    +    int x = H();  //ERROR: calling a __host__-only constexpr function from device code
    +}
    +
    +
    +
    +
    +
  4. +
+

The experimental flag -expt-relaxed-constexpr can be used to relax this constraint. When this flag is specified, the compiler will support cross execution space calls described above, as follows:

+
    +
  1. +

    A cross-execution space call to a constexpr function is supported if it occurs in a context that requires constant evaluation, e.g., in the initializer of a constexpr variable. Example:

    +
    +
    +
    +
    constexpr __host__ int H(int x) { return x+1; };
    +__global__ void doit() {
    +constexpr int val = H(1); // OK: call is in a context that
    +                          // requires constant evaluation.
    +}
    +
    +constexpr __device__ int D(int x) { return x+1; }
    +int main() {
    +constexpr int val = D(1); // OK: call is in a context that
    +                          // requires constant evaluation.
    +}
    +
    +
    +
    +
    +
  2. +
  3. +

    Otherwise:

    +
    +
    +
      +
    1. +

      During device code generation, device code is generated for the body of a __host__-only constexpr function H, unless H is not used or is only called in a constexpr context. Example:

      +
      +
      +
      +
      // NOTE: "H" is emitted in generated device code because it is
      +// called from device code in a non-constexpr context
      +constexpr __host__ int H(int x) { return x+1; }
      +
      +__device__ int doit(int in) {
      +  in = H(in);  // OK, even though argument is not a constant expression
      +  return in;
      +}
      +
      +
      +
      +
      +
    2. +
    3. +

      All code restrictions applicable to a ``__device__`` function are also applicable to the ``constexpr host``-only function ``H`` that is called from device code. However, compiler may not emit any build time diagnostics for ``H`` for these restrictions 8 .

      +
      +
      +

      For example, the following code patterns are unsupported in the body of H (as with any __device__ function), but no compiler diagnostic may be generated:

      +
      +
      +
        +
      • +

        ODR-use of a host variable or __host__-only non-constexpr function. Example:

        +
        +
        +
        +
        int qqq, www;
        +
        +constexpr __host__ int* H(bool b) { return b ? &qqq : &www; };
        +
        +__device__ int doit(bool flag) {
        +  int *ptr;
        +  ptr = H(flag); // ERROR: H() attempts to refer to host variables 'qqq' and 'www'.
        +                 // code will compile, but will NOT execute correctly.
        +  return *ptr;
        +}
        +
        +
        +
        +
        +
      • +
      • +

        Use of exceptions (throw/catch) and RTTI (typeid, dynamic_cast). Example:

        +
        +
        +
        +
        struct Base { };
        +struct Derived : public Base { };
        +
        +// NOTE: "H" is emitted in generated device code
        +constexpr int H(bool b, Base *ptr) {
        +  if (b) {
        +    return 1;
        +  } else if (typeid(ptr) == typeid(Derived)) { // ERROR: use of typeid in code executing on the GPU
        +    return 2;
        +  } else {
        +    throw int{4}; // ERROR: use of throw in code executing on the GPU
        +  }
        +}
        +__device__ void doit(bool flag) {
        +  int val;
        +  Derived d;
        +  val = H(flag, &d); //ERROR: H() attempts use typeid and throw(), which are not allowed in code that executes on the GPU
        +}
        +
        +
        +
        +
        +
      • +
      +
      +
      +
      +
      +
    4. +
    5. +

      During host code generation, the body of a __device__-only constexpr function D is preserved in the code sent to the host compiler. If the body of D attempts to ODR-use a namespace scope device variable or a __device__-only non-constexpr function, then the call to D from host code is not supported (code may build without compiler diagnostics, but may behave incorrectly at run time). Example:

      +
      +
      +
      +
      __device__ int qqq, www;
      +constexpr __device__ int* D(bool b) { return b ? &qqq : &www; };
      +
      +int doit(bool flag) {
      +  int *ptr;
      +  ptr = D(flag); // ERROR: D() attempts to refer to device variables 'qqq' and 'www'
      +                 // code will compile, but will NOT execute correctly.
      +  return *ptr;
      +}
      +
      +
      +
      +
      +
    6. +
    7. +

      Note: Given above restrictions and lack of compiler diagnostics for incorrect usage, be careful when calling a constexpr __host__ function in the standard C++ headers from device code, since the implementation of the function will vary depending on the host platform, e.g., based on the libstdc++ version for gcc host compiler. Such code may break silently when being ported to a different platform or host compiler version (if the target C++ library implementation odr-uses a host code variable or function, as described earlier).

      +
      +
      +

      Example:

      +
      +
      __device__ int get(int in) {
      + int val = std::foo(in); // "std::foo" is constexpr function defined in the host compiler's standard library header
      +                         // WARNING: if std::foo implementation ODR-uses host variables or functions,
      +                         // code will not work correctly
      +}
      +
      +
      +
      +
      +
    8. +
    +
    +
    +
  4. +
+
+
8
+
+

Diagnostics are usually generated during parsing, but the host-only function H may already have been parsed before the call to H from device code is encountered later in the translation unit.

+
+
+
+
+

+18.9. Code Samples +

+
+

+18.9.1. Data Aggregation Class +

+
+
class PixelRGBA {
+public:
+    __device__ PixelRGBA(): r_(0), g_(0), b_(0), a_(0) { }
+
+    __device__ PixelRGBA(unsigned char r, unsigned char g,
+                         unsigned char b, unsigned char a = 255):
+                         r_(r), g_(g), b_(b), a_(a) { }
+
+private:
+    unsigned char r_, g_, b_, a_;
+
+    friend PixelRGBA operator+(const PixelRGBA&, const PixelRGBA&);
+};
+
+__device__
+PixelRGBA operator+(const PixelRGBA& p1, const PixelRGBA& p2)
+{
+    return PixelRGBA(p1.r_ + p2.r_, p1.g_ + p2.g_,
+                     p1.b_ + p2.b_, p1.a_ + p2.a_);
+}
+
+__device__ void func(void)
+{
+    PixelRGBA p1, p2;
+    // ...      // Initialization of p1 and p2 here
+    PixelRGBA p3 = p1 + p2;
+}
+
+
+
+
+

+18.9.2. Derived Class +

+
+
__device__ void* operator new(size_t bytes, MemoryPool& p);
+__device__ void operator delete(void*, MemoryPool& p);
+class Shape {
+public:
+    __device__ Shape(void) { }
+    __device__ void putThis(PrintBuffer *p) const;
+    __device__ virtual void Draw(PrintBuffer *p) const {
+         p->put("Shapeless");
+    }
+    __device__ virtual ~Shape() {}
+};
+class Point : public Shape {
+public:
+    __device__ Point() : x(0), y(0) {}
+    __device__ Point(int ix, int iy) : x(ix), y(iy) { }
+    __device__ void PutCoord(PrintBuffer *p) const;
+    __device__ void Draw(PrintBuffer *p) const;
+    __device__ ~Point() {}
+private:
+    int x, y;
+};
+__device__ Shape* GetPointObj(MemoryPool& pool)
+{
+    Shape* shape = new(pool) Point(rand(-20,10), rand(-100,-20));
+    return shape;
+}
+
+
+
+
+

+18.9.3. Class Template +

+
+
template <class T>
+class myValues {
+    T values[MAX_VALUES];
+public:
+    __device__ myValues(T clear) { ... }
+    __device__ void setValue(int Idx, T value) { ... }
+    __device__ void putToMemory(T* valueLocation) { ... }
+};
+
+template <class T>
+void __global__ useValues(T* memoryBuffer) {
+    myValues<T> myLocation(0);
+    ...
+}
+
+__device__ void* buffer;
+
+int main()
+{
+    ...
+    useValues<int><<<blocks, threads>>>(buffer);
+    ...
+}
+
+
+
+
+

+18.9.4. Function Template +

+
+
template <typename T>
+__device__ bool func(T x)
+{
+   ...
+   return (...);
+}
+
+template <>
+__device__ bool func<int>(T x) // Specialization
+{
+   return true;
+}
+
+// Explicit argument specification
+bool result = func<double>(0.5);
+
+// Implicit argument deduction
+int x = 1;
+bool result = func(x);
+
+
+
+
+

+18.9.5. Functor Class +

+
+
class Add {
+public:
+    __device__  float operator() (float a, float b) const
+    {
+        return a + b;
+    }
+};
+
+class Sub {
+public:
+    __device__  float operator() (float a, float b) const
+    {
+        return a - b;
+    }
+};
+
+// Device code
+template<class O> __global__
+void VectorOperation(const float * A, const float * B, float * C,
+                     unsigned int N, O op)
+{
+    unsigned int iElement = blockDim.x * blockIdx.x + threadIdx.x;
+    if (iElement < N)
+        C[iElement] = op(A[iElement], B[iElement]);
+}
+
+// Host code
+int main()
+{
+    ...
+    VectorOperation<<<blocks, threads>>>(v1, v2, v3, N, Add());
+    ...
+}
+
+
+
+
9
+
+

e.g., the <<<...>>> syntax for launching kernels.

+
+
10
+
+

This does not apply to entities that may be defined in more than one translation unit, such as compiler generated template instantiations.

+
+
11
+
+

The intent is to allow variable memory space specifiers for static variables in a __host__ __device__ function during device compilation, but disallow it during host compilation

+
+
12
+
+

One way to debug suspected layout mismatch of a type C is to use printf to output the values of sizeof(C) and offsetof(C, field) in host and device code.

+
+
13
+
+

Note that this may negatively impact compile time due to presence of extra declarations.

+
+
14
+
+

At present, the -std=c++11 flag is supported only for the following host compilers : gcc version >= 4.7, clang, icc >= 15, and xlc >= 13.1

+
+
15
+
+

including operator()

+
+
16
+
+

The restrictions are the same as with a non-constexpr callee function.

+
+
17
+
+

Note that the behavior of experimental flags may change in future compiler releases.

+
+
18
+
+

C++ Standard Section [basic.types]

+
+
19
+
+

C++ Standard Section [expr.const]

+
+
20
+
+

At present, the -std=c++14 flag is supported only for the following host compilers : gcc version >= 5.1, clang version >= 3.7 and icc version >= 17

+
+
21
+
+

At present, the -std=c++17 flag is supported only for the following host compilers : gcc version >= 7.0, clang version >= 8.0, Visual Studio version >= 2017, pgi compiler version >= 19.0, icc compiler version >= 19.0

+
+
22
+
+

At present, the -std=c++20 flag is supported only for the following host compilers : gcc version >= 10.0, clang version >= 10.0, Visual Studio Version >= 2022 and nvc++ version >= 20.7.

+
+
23
+
+

When using the icc host compiler, this flag is only supported for icc >= 1800.

+
+
+24(1,2) +
+
+

The traits will always return false if extended lambda mode is not active.

+
+
25
+
+

In contrast, the C++ standard specifies that the captured variable is used to direct-initialize the field of the closure type.

+
+
+
+
+
+
+

+19. Texture Fetching +

+
+

Warning

+

This document has been replaced by a new CUDA Programming Guide. The information in this document should be considered legacy, and this document is no longer being updated as of CUDA 13.0. Please refer to the CUDA Programming Guide for up-to-date information on CUDA.

+
+

This section gives the formula used to compute the value returned by the texture functions of Texture Functions depending on the various attributes of the texture object (see Texture and Surface Memory).

+

The texture bound to the texture object is represented as an array T of

+
    +
  • N texels for a one-dimensional texture,

  • +
  • N x M texels for a two-dimensional texture,

  • +
  • N x M x L texels for a three-dimensional texture.

  • +
+

It is fetched using non-normalized texture coordinates x, y, and z, or the normalized texture coordinates x/N, y/M, and z/L as described in Texture Memory. In this section, the coordinates are assumed to be in the valid range. Texture Memory explained how out-of-range coordinates are remapped to the valid range based on the addressing mode.

+
+

+19.1. Nearest-Point Sampling +

+

In this filtering mode, the value returned by the texture fetch is

+
    +
  • tex(x)=T[i] for a one-dimensional texture,

  • +
  • tex(x,y)=T[i,j] for a two-dimensional texture,

  • +
  • tex(x,y,z)=T[i,j,k] for a three-dimensional texture,

  • +
+

where i=floor(x), j=floor(y), and k=floor(z).

+

Figure 36 illustrates nearest-point sampling for a one-dimensional texture with N=4.

+
+_images/nearest-point-sampling-of-1-d-texture-of-4-texels.png +
+

Figure 36 Nearest-Point Sampling Filtering Mode

+
+
+

For integer textures, the value returned by the texture fetch can be optionally remapped to [0.0, 1.0] (see Texture Memory).

+
+
+

+19.2. Linear Filtering +

+

In this filtering mode, which is only available for floating-point textures, the value returned by the texture fetch is

+
    +
  • \(tex(x)=(1-\alpha)T[i]+{\alpha}T[i+1]\) for a one-dimensional texture,

  • +
  • \(tex(x)=(1-\alpha)T[i]+{\alpha}T[i+1]\) for a one-dimensional texture,

  • +
  • \(tex(x,y)=(1-\alpha)(1-\beta)T[i,j]+\alpha(1-\beta)T[i+1,j]+(1-\alpha){\beta}T[i,j+1]+\alpha{\beta}T[i+1,j+1]\) for a two-dimensional texture,

  • +
  • +

    \(tex(x,y,z)\) =

    +

    \((1-\alpha)(1-\beta)(1-\gamma)T[i,j,k]+\alpha(1-\beta)(1-\gamma)T[i+1,j,k]+\)

    +

    \((1-\alpha)\beta(1-\gamma)T[i,j+1,k]+\alpha\beta(1-\gamma)T[i+1,j+1,k]+\)

    +

    \((1-\alpha)(1-\beta){\gamma}T[i,j,k+1]+\alpha(1-\beta){\gamma}T[i+1,j,k+1]+\)

    +

    \((1-\alpha)\beta{\gamma}T[i,j+1,k+1]+\alpha\beta{\gamma}T[i+1,j+1,k+1]\)

    +

    for a three-dimensional texture,

    +
  • +
+

where:

+
    +
  • \(i=floor(x\ B)*, \alpha=frac(x\ B)*, *x\ B\ =x-0.5,\)

  • +
  • \(j=floor(y\ B)*, \beta=frac(y\ B)*, *y\ B\ =y-0.5,\)

  • +
  • \(k=floor(z\ B)*, \gamma=frac(z\ B)*, *z\ B\ = z-0.5,\)

  • +
+

\(\alpha\), \(\beta\), and \(\gamma\) are stored in 9-bit fixed point format with 8 bits of fractional value (so 1.0 is exactly represented).

+

Figure 37 illustrates linear filtering of a one-dimensional texture with N=4.

+
+_images/linear-filtering-of-1-d-texture-of-4-texels.png +
+

Figure 37 Linear Filtering Mode

+
+
+
+
+

+19.3. Table Lookup +

+

A table lookup TL(x) where x spans the interval [0,R] can be implemented as TL(x)=tex((N-1)/R)x+0.5) in order to ensure that TL(0)=T[0] and TL(R)=T[N-1].

+

Figure 38 illustrates the use of texture filtering to implement a table lookup with R=4 or R=1 from a one-dimensional texture with N=4.

+
+_images/1-d-table-lookup-using-linear-filtering.png +
+

Figure 38 One-Dimensional Table Lookup Using Linear Filtering

+
+
+
+
+
+

+20. Compute Capabilities +

+
+

Warning

+

This document has been replaced by a new CUDA Programming Guide. The information in this document should be considered legacy, and this document is no longer being updated as of CUDA 13.0. Please refer to the CUDA Programming Guide for up-to-date information on CUDA.

+
+

The general specifications and features of a compute device depend on its compute capability (see Compute Capability).

+

Table 26 and +Table 27 show the features and technical specifications associated with each compute capability that is currently supported.

+

Section Floating-Point Standard reviews compliance with the IEEE floating-point standard.

+

Sections Compute Capability 5.x, Compute Capability 6.x, Compute Capability 7.x, Compute Capability 8.x, Compute Capability 9.0, Compute Capability 10.0, and Compute Capability 12.0 give more details on the architecture of devices with these respective compute capabilities.

+
+

+20.1. Feature Availability +

+

Most compute features introduced with a compute architecture are intended to be available on all subsequent architectures. This is shown in Table 26 by the “yes” for availability of a feature on compute capabilities subsequent to its introduction.

+
+

+20.1.1. Architecture-Specific Features +

+

Beginning with devices of Compute Capability 9.0, specialized compute features that are introduced with an architecture may not be guaranteed to be available on all subsequent compute capabilities. These features are called architecture-specific features and target acceleration of specialized operations, such as Tensor Core operations, which are not intended for all classes of compute capabilities or may significantly change on future generations. Code must be compiled with an architecture-specific compiler target (see Feature Set Compiler Targets) to enable architecture-specific features. Code compiled with an architecture-specific compiler target can only be run on the exact compute capability it was compiled for.

+
+
+

+20.1.2. Family-Specific Features +

+

Beginning with devices of Compute Capability 10.0, some architecture-specific features are common to devices of more than one compute capability. The devices that contain these features are part of the same family and these features can also be called family-specific features. Family-specific features are guaranteed to be available on all devices in the same family. A family-specific compiler target is required to enable family-specific features. See Section 20.1.3. Code compiled for a family-specific target can only be run on GPUs which are members of that family.

+
+
+

+20.1.3. Feature Set Compiler Targets +

+

There are three sets of compute features which the compiler can target:

+

Baseline Feature Set: The predominant set of compute features that are introduced with the intent to be available for subsequent compute architectures. These features and their availability are summarized in Table 26.

+

Architecture-Specific Feature Set: A small and highly specialized set of features called architecture-specific, that are introduced to accelerate specialized operations, which are not guaranteed to be available or might change significantly on subsequent compute architectures. These features are summarized in the respective “Compute Capability #.#” subsections. The architecture-specific feature set is a superset of the family-specific feature set. Architecture-specific compiler targets were introduced with Compute Capability 9.0 devices and are selected by using an a suffix in the compilation target, for example by specifying compute_100a or compute_120a as the compute target.

+

Family-Specific Feature Set: Some architecture-specific features are common to GPUs of more than one compute capability. These features are summarized in the respective “Compute Capability #.#” subsections. With a few exceptions, later generation devices with the same major compute capability are in the same family. Table 25 indicates the compatibility of family-specific targets with device compute capability, including exceptions. The family-specific feature set is a superset of the baseline feature set. Family-specific compiler targets were introduced with Compute Capability 10.0 devices and are selected by using a f suffix in the compilation target, for example by specifying compute_100f or compute_120f as the compute target.

+

All devices starting from compute capability 9.0 have a set of features that are architecture-specific. To utilize the complete set of these features on a specific GPU, the architecture-specific compiler target with the suffix a must be used. Additionally, starting from compute capability 10.0, there are sets of features that appear in multiple devices with different minor compute capability. These sets of instructions are called family-specific features, and the devices which share these features are said to be part of the same family. The family-specific features are a subset of the architecture-specific features that are shared by all members of that GPU family. The family-specific compiler target with the suffix f allows the compiler to generate code which uses this common subset of architecture-specific features.

+

For example:

+
    +
  • The compute_100 compilation target does not allow use of architecture-specific features. This target will be compatible with all devices of compute capability 10.0 and later.

  • +
  • The compute_100f family-specific compilation target allows the use of the subset of architecture-specific features that are common across the GPU family. This target will only be compatible with devices that are part of the GPU family. In this example it is compatible with devices of Compute Capability 10.0 and Compute Capability 10.3. The features available in the family-specific compute_100f target is a superset of the features available in the baseline compute_100 target.

  • +
  • The compute_100a architecture-specific compilation target allows use of the complete set of architecture-specific features in Compute Capability 10.0 devices. This target will only be compatible with devices of Compute Capability 10.0 and no others. The features available in the compute_100a target form a superset of the features available in the compute_100f target.

  • +
+ + +++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+Table 25 Family-Specific Compatibility +

Compilation Target

Compatible with Compute Capability

compute_100f

10.0

10.3

compute_103f

10.3 26

compute_110f

11.0 26

compute_120f

12.0

12.1

compute_121f

12.1 26

+
+
+26(1,2,3) +
+
+

Some families only contain a single member when they are created. They may be expanded in the future to include more devices.

+
+
+
+
+
+

+20.2. Features and Technical Specifications +

+ + +++++++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+Table 26 Feature Support per Compute Capability +

+Feature Support

+Compute Capability

(Unlisted features are supported for all compute capabilities)

7.x

8.x

9.0

10.0

11.0

12.0

Atomic functions operating on 128-bit integer values in global memory (Atomic Functions)

No

Yes

Atomic functions operating on 128-bit integer values in shared memory (Atomic Functions)

No

Yes

Atomic addition operating on float2 and float4 floating point vectors in global memory (atomicAdd())

No

Yes

Bfloat16-precision floating-point operations: addition, subtraction, multiplication, comparison, warp shuffle functions, conversion

No

Yes

Hardware-accelerated memcpy_async (Asynchronous Data Copies using cuda::pipeline)

No

Yes

Hardware-accelerated Split Arrive/Wait Barrier (Asynchronous Barrier)

No

Yes

L2 Cache Residency Management (Device Memory L2 Access Management)

No

Yes

DPX Instructions for Accelerated Dynamic Programming

No

Yes

Distributed Shared Memory

No

Yes

Thread Block Cluster

No

Yes

Tensor Memory Accelerator (TMA) unit

No

Yes

+

Note that the KB and K units used in the following table correspond to 1024 bytes (i.e., a KiB) and 1024 respectively.

+ + ++++++++++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+Table 27 Technical Specifications per Compute Capability +

+Compute Capability

Technical Specifications

7.5

8.0

8.6

8.7

8.9

9.0

10.0

11.0

12.0

Maximum number of +resident grids per device +(Concurrent Kernel +Execution)

128

Maximum dimensionality of +grid of thread blocks

3

Maximum x -dimension of a +grid of thread blocks

231-1

Maximum y- or z-dimension +of a grid of thread +blocks

65535

Maximum dimensionality of +thread block

3

Maximum x- or +y-dimensionality of a +block

1024

Maximum z-dimension +of a block

64

Maximum number of +threads per block

1024

Warp size

32

Maximum number of +resident blocks per SM

16

32

16

24

32

24

Maximum number of +resident warps per SM

32

64

48

64

48

Maximum number of +resident threads per SM

1024

2048

1536

2048

1536

Number of 32-bit +registers per SM

64 K

Maximum number of 32-bit +registers per thread +block

64 K

Maximum number of 32-bit +registers per thread

255

Maximum amount of shared +memory per SM

64 KB

164 +KB

100 +KB

164 +KB

100 +KB

228 +KB

100 +KB

Maximum amount of shared +memory per thread block +27

64 KB

163 +KB

99 KB

163 +KB

99 KB

227 +KB

99 KB

Number of shared +memory banks

32

Maximum amount of local +memory per thread

512 KB

Constant memory size

64 KB

Cache working set per SM +for constant memory

8 KB

Cache working set per SM +for texture memory

32 or +64 KB

28 KB +~ 192 +KB

28 KB +~ 128 +KB

28 KB +~ 192 +KB

28 KB +~ 128 +KB

28 KB +~ 256 +KB

28 KB +~ 128 +KB

Maximum width for a 1D +texture object using +a CUDA array

131072

Maximum width for a 1D +texture object using +linear memory

228

Maximum width and number +of layers for a 1D +layered texture object

32768 +x +2048

Maximum width and height +for a 2D texture +object using a CUDA +array

131072 +x +65536

Maximum width and height +for a 2D texture +object using +linear memory

131072 +x +65000

Maximum width and height +for a 2D texture +object using a CUDA +array supporting texture +gather

32768 +x +32768

Maximum width, height, +and number of layers for +a 2D layered texture +object

32768 x 32768 x 2048

Maximum width, height, +and depth for a 3D +texture object using +to a CUDA array

16384 x 16384 x 16384

Maximum width (and +height) for a cubemap +texture object

32768

Maximum width (and +height) and number of +layers for a cubemap +layered texture object

32768 +x +2046

Maximum number of +textures that can be +bound to a kernel

256

Maximum width for a 1D +surface object using +a CUDA array

32768

Maximum width and number +of layers for a 1D +layered surface object

32768 +x +2048

Maximum width and height +for a 2D surface +object using a +CUDA array

131072 +x +65536

Maximum width, height, +and number of layers for +a 2D layered surface +object

32768 +x +32768 x 1048

Maximum width, height, +and depth for a 3D +surface object using +a CUDA array

16384 +x +16384 x 16384

Maximum width (and +height) for a cubemap +surface object using +a CUDA array

32768

Maximum width (and +height) and number of +layers for a cubemap +layered surface object

32768 +x +2046

Maximum number of +surfaces that can use +a kernel

32

+
+
+

+20.3. Floating-Point Standard +

+

All compute devices follow the IEEE 754-2008 standard for binary floating-point arithmetic with the following deviations:

+
    +
  • There is no dynamically configurable rounding mode; however, most of the operations support multiple IEEE rounding modes, exposed via device intrinsics.

  • +
  • There is no mechanism for detecting that a floating-point exception has occurred and all operations behave as if the IEEE-754 exceptions are always masked, and deliver the masked response as defined by IEEE-754 if there is an exceptional event. For the same reason, while SNaN encodings are supported, they are not signaling and are handled as quiet.

  • +
  • The result of a single-precision floating-point operation involving one or more input NaNs is the quiet NaN of bit pattern 0x7fffffff.

  • +
  • Double-precision floating-point absolute value and negation are not compliant with IEEE-754 with respect to NaNs; these are passed through unchanged.

  • +
+

Code must be compiled with -ftz=false, -prec-div=true, and -prec-sqrt=true to ensure IEEE compliance (this is the default setting; see the nvcc user manual for description of these compilation flags).

+

Regardless of the setting of the compiler flag -ftz,

+
    +
  • atomic single-precision floating-point adds on global memory always operate in flush-to-zero mode, i.e., behave equivalent to FADD.F32.FTZ.RN,

  • +
  • atomic single-precision floating-point adds on shared memory always operate with denormal support, i.e., behave equivalent to FADD.F32.RN.

  • +
+

In accordance to the IEEE-754R standard, if one of the input parameters to fminf(), fmin(), fmaxf(), or fmax() is NaN, but not the other, the result is the non-NaN parameter.

+

The conversion of a floating-point value to an integer value in the case where the floating-point value falls outside the range of the integer format is left undefined by IEEE-754. For compute devices, the behavior is to clamp to the end of the supported range. This is unlike the x86 architecture behavior.

+

The behavior of integer division by zero and integer overflow is left undefined by IEEE-754. For compute devices, there is no mechanism for detecting that such integer operation exceptions have occurred. Integer division by zero yields an unspecified, machine-specific value.

+

https://developer.nvidia.com/content/precision-performance-floating-point-and-ieee-754-compliance-nvidia-gpus includes more information on the floating point accuracy and compliance of NVIDIA GPUs.

+
+
+

+20.4. Compute Capability 5.x +

+
+

+20.4.1. Architecture +

+

An SM consists of:

+
    +
  • 128 CUDA cores for arithmetic operations (see CUDA C++ Best Practices Guide for throughputs of arithmetic operations),

  • +
  • 32 special function units for single-precision floating-point transcendental functions,

  • +
  • 4 warp schedulers.

  • +
+

When an SM is given warps to execute, it first distributes them among the four schedulers. Then, at every instruction issue time, each scheduler issues one instruction for one of its assigned warps that is ready to execute, if any.

+

An SM has:

+
    +
  • a read-only constant cache that is shared by all functional units and speeds up reads from the constant memory space, which resides in device memory,

  • +
  • a unified L1/texture cache of 24 KB used to cache reads from global memory,

  • +
  • 64 KB of shared memory for devices of compute capability 5.0 or 96 KB of shared memory for devices of compute capability 5.2.

  • +
+

The unified L1/texture cache is also used by the texture unit that implements the various addressing modes and data filtering mentioned in Texture and Surface Memory.

+

There is also an L2 cache shared by all SMs that is used to cache accesses to local or global memory, including temporary register spills. Applications may query the L2 cache size by checking the l2CacheSize device property (see Device Enumeration).

+

The cache behavior (e.g., whether reads are cached in both the unified L1/texture cache and L2 or in L2 only) can be partially configured on a per-access basis using modifiers to the load instruction.

+
+
+

+20.4.2. Global Memory +

+

Global memory accesses are always cached in L2.

+

Data that is read-only for the entire lifetime of the kernel can also be cached in the unified L1/texture cache described in the previous section by reading it using the __ldg() function (see Read-Only Data Cache Load Function). When the compiler detects that the read-only condition is satisfied for some data, it will use __ldg() to read it. The compiler might not always be able to detect that the read-only condition is satisfied for some data. Marking pointers used for loading such data with both the const and __restrict__ qualifiers increases the likelihood that the compiler will detect the read-only condition.

+

Data that is not read-only for the entire lifetime of the kernel cannot be cached in the unified L1/texture cache for devices of compute capability 5.0. For devices of compute capability 5.2, it is, by default, not cached in the unified L1/texture cache, but caching may be enabled using the following mechanisms:

+
    +
  • Perform the read using inline assembly with the appropriate modifier as described in the PTX reference manual;

  • +
  • Compile with the -Xptxas -dlcm=ca compilation flag, in which case all reads are cached, except reads that are performed using inline assembly with a modifier that disables caching;

  • +
  • Compile with the -Xptxas -fscm=ca compilation flag, in which case all reads are cached, including reads that are performed using inline assembly regardless of the modifier used.

  • +
+

When caching is enabled using one of the three mechanisms listed above, devices of compute capability 5.2 will cache global memory reads in the unified L1/texture cache for all kernel launches except for the kernel launches for which thread blocks consume too much of the SM’s register file. These exceptions are reported by the profiler.

+
+
+

+20.4.3. Shared Memory +

+

Shared memory has 32 banks that are organized such that successive 32-bit words map to successive banks. Each bank has a bandwidth of 32 bits per clock cycle.

+

A shared memory request for a warp does not generate a bank conflict between two threads that access any address within the same 32-bit word (even though the two addresses fall in the same bank). In that case, for read accesses, the word is broadcast to the requesting threads and for write accesses, each address is written by only one of the threads (which thread performs the write is undefined).

+

Figure 39 shows some examples of strided access.

+

Figure 40 shows some examples of memory read accesses that involve the broadcast mechanism.

+
+Strided Shared Memory Accesses in 32 bit bank size mode. +
+

Figure 39 Strided Shared Memory Accesses in 32 bit bank size mode.

+
+
+
Left
+
+

Linear addressing with a stride of one 32-bit word (no bank conflict).

+
+
Middle
+
+

Linear addressing with a stride of two 32-bit words (two-way bank conflict).

+
+
Right
+
+

Linear addressing with a stride of three 32-bit words (no bank conflict).

+
+
+
+
+
+
+Irregular Shared Memory Accesses. +
+

Figure 40 Irregular Shared Memory Accesses.

+
+
+
Left
+
+

Conflict-free access via random permutation.

+
+
Middle
+
+

Conflict-free access since threads 3, 4, 6, 7, and 9 access the same word within bank 5.

+
+
Right
+
+

Conflict-free broadcast access (threads access the same word within a bank).

+
+
+
+
+
+
+
+
+

+20.5. Compute Capability 6.x +

+
+

+20.5.1. Architecture +

+

An SM consists of:

+
    +
  • 64 (compute capability 6.0) or 128 (6.1 and 6.2) CUDA cores for arithmetic operations,

  • +
  • 16 (6.0) or 32 (6.1 and 6.2) special function units for single-precision floating-point transcendental functions,

  • +
  • 2 (6.0) or 4 (6.1 and 6.2) warp schedulers.

  • +
+

When an SM is given warps to execute, it first distributes them among its schedulers. Then, at every instruction issue time, each scheduler issues one instruction for one of its assigned warps that is ready to execute, if any.

+

An SM has:

+
    +
  • a read-only constant cache that is shared by all functional units and speeds up reads from the constant memory space, which resides in device memory,

  • +
  • a unified L1/texture cache for reads from global memory of size 24 KB (6.0 and 6.2) or 48 KB (6.1),

  • +
  • a shared memory of size 64 KB (6.0 and 6.2) or 96 KB (6.1).

  • +
+

The unified L1/texture cache is also used by the texture unit that implements the various addressing modes and data filtering mentioned in Texture and Surface Memory.

+

There is also an L2 cache shared by all SMs that is used to cache accesses to local or global memory, including temporary register spills. Applications may query the L2 cache size by checking the l2CacheSize device property (see Device Enumeration).

+

The cache behavior (for example, whether reads are cached in both the unified L1/texture cache and L2 or in L2 only) can be partially configured on a per-access basis using modifiers to the load instruction.

+
+
+

+20.5.2. Global Memory +

+

Global memory behaves the same way as in devices of compute capability 5.x (See Global Memory).

+
+
+

+20.5.3. Shared Memory +

+

Shared memory behaves the same way as in devices of compute capability 5.x (See Shared Memory).

+
+
+
+

+20.6. Compute Capability 7.x +

+
+

+20.6.1. Architecture +

+

An SM consists of:

+
    +
  • 64 FP32 cores for single-precision arithmetic operations,

  • +
  • 32 FP64 cores for double-precision arithmetic operations,28

  • +
  • 64 INT32 cores for integer math,

  • +
  • 8 mixed-precision Tensor Cores for deep learning matrix arithmetic

  • +
  • 16 special function units for single-precision floating-point transcendental functions,

  • +
  • 4 warp schedulers.

  • +
+

An SM statically distributes its warps among its schedulers. Then, at every instruction issue time, each scheduler issues one instruction for one of its assigned warps that is ready to execute, if any.

+

An SM has:

+
    +
  • a read-only constant cache that is shared by all functional units and speeds up reads from the constant memory space, which resides in device memory,

  • +
  • a unified data cache and shared memory with a total size of 128 KB (Volta) or 96 KB (Turing).

  • +
+

Shared memory is partitioned out of unified data cache, and can be configured to various sizes (See Shared Memory.) The remaining data cache serves as an L1 cache and is also used by the texture unit that implements the various addressing and data filtering modes mentioned in Texture and Surface Memory.

+
+
+

+20.6.2. Independent Thread Scheduling +

+

The NVIDIA Volta GPU Architecture introduces Independent Thread Scheduling among threads in a warp, enabling intra-warp synchronization patterns previously unavailable and simplifying code changes when porting CPU code. However, this can lead to a rather different set of threads participating in the executed code than intended if the developer made assumptions about warp-synchronicity of previous hardware architectures.

+

Below are code patterns of concern and suggested corrective actions for Volta-safe code.

+
    +
  1. For applications using warp intrinsics (__shfl*, __any, __all, __ballot), it is necessary that developers port their code to the new, safe, synchronizing counterpart, with the *_sync suffix. The new warp intrinsics take in a mask of threads that explicitly define which lanes (threads of a warp) must participate in the warp intrinsic. See Warp Vote Functions and Warp Shuffle Functions for details.

  2. +
+

Since the intrinsics are available with CUDA 9.0+, (if necessary) code can be executed conditionally with the following preprocessor macro:

+
+
#if defined(CUDART_VERSION) && CUDART_VERSION >= 9000
+// *_sync intrinsic
+#endif
+
+
+

These intrinsics are available on all architectures, not just NVIDIA Volta GPU Architecture or NVIDIA Turing GPU Architecture, and in most cases a single code-base will suffice for all architectures. Note, however, that for Pascal and earlier architectures, all threads in mask must execute the same warp intrinsic instruction in convergence, and the union of all values in mask must be equal to the warp’s active mask. The following code pattern is valid on NVIDIA Volta GPU Architecture, but not on Pascal or earlier architectures.

+
+
+
+
if (tid % warpSize < 16) {
+    ...
+    float swapped = __shfl_xor_sync(0xffffffff, val, 16);
+    ...
+} else {
+    ...
+    float swapped = __shfl_xor_sync(0xffffffff, val, 16);
+    ...
+}
+
+
+
+
+

The replacement for __ballot(1) is __activemask(). Note that threads within a warp can diverge even within a single code path. As a result, __activemask() and __ballot(1) may return only a subset of the threads on the current code path. The following invalid code example sets bit i of output to 1 when data[i] is greater than threshold. __activemask() is used in an attempt to enable cases where dataLen is not a multiple of 32.

+
+
+
+
// Sets bit in output[] to 1 if the correspond element in data[i]
+// is greater than 'threshold', using 32 threads in a warp.
+
+for (int i = warpLane; i < dataLen; i += warpSize) {
+    unsigned active = __activemask();
+    unsigned bitPack = __ballot_sync(active, data[i] > threshold);
+    if (warpLane == 0) {
+        output[i / 32] = bitPack;
+    }
+}
+
+
+
+
+

This code is invalid because CUDA does not guarantee that the warp will diverge ONLY at the loop condition. When divergence happens for other reasons, conflicting results will be computed for the same 32-bit output element by different subsets of threads in the warp. A correct code might use a non-divergent loop condition together with __ballot_sync() to safely enumerate the set of threads in the warp participating in the threshold calculation as follows.

+
+
+
+
for (int i = warpLane; i - warpLane < dataLen; i += warpSize) {
+    unsigned active = __ballot_sync(0xFFFFFFFF, i < dataLen);
+    if (i < dataLen) {
+        unsigned bitPack = __ballot_sync(active, data[i] > threshold);
+        if (warpLane == 0) {
+            output[i / 32] = bitPack;
+        }
+    }
+}
+
+
+
+
+

Discovery Pattern demonstrates a valid use case for __activemask().

+
    +
  1. +

    If applications have warp-synchronous codes, they will need to insert the new __syncwarp() warp-wide barrier synchronization instruction between any steps where data is exchanged between threads via global or shared memory. Assumptions that code is executed in lockstep or that reads/writes from separate threads are visible across a warp without synchronization are invalid.

    +
    +
    __shared__ float s_buff[BLOCK_SIZE];
    +s_buff[tid] = val;
    +__syncthreads();
    +
    +// Inter-warp reduction
    +for (int i = BLOCK_SIZE / 2; i >= 32; i /= 2) {
    +    if (tid < i) {
    +        s_buff[tid] += s_buff[tid+i];
    +    }
    +    __syncthreads();
    +}
    +
    +// Intra-warp reduction
    +// Butterfly reduction simplifies syncwarp mask
    +if (tid < 32) {
    +    float temp;
    +    temp = s_buff[tid ^ 16]; __syncwarp();
    +    s_buff[tid] += temp;     __syncwarp();
    +    temp = s_buff[tid ^ 8];  __syncwarp();
    +    s_buff[tid] += temp;     __syncwarp();
    +    temp = s_buff[tid ^ 4];  __syncwarp();
    +    s_buff[tid] += temp;     __syncwarp();
    +    temp = s_buff[tid ^ 2];  __syncwarp();
    +    s_buff[tid] += temp;     __syncwarp();
    +}
    +
    +if (tid == 0) {
    +    *output = s_buff[0] + s_buff[1];
    +}
    +__syncthreads();
    +
    +
    +
  2. +
  3. Although __syncthreads() has been consistently documented as synchronizing all threads in the thread block, Pascal and prior architectures could only enforce synchronization at the warp level. In certain cases, this allowed a barrier to succeed without being executed by every thread as long as at least some thread in every warp reached the barrier. Starting with NVIDIA Volta GPU Architecture, the CUDA built-in __syncthreads() and PTX instruction bar.sync (and their derivatives) are enforced per thread and thus will not succeed until reached by all non-exited threads in the block. Code exploiting the previous behavior will likely deadlock and must be modified to ensure that all non-exited threads reach the barrier.

  4. +
+

The racecheck and synccheck tools provided by compute-saniter can help with locating violations.

+

To aid migration while implementing the above-mentioned corrective actions, developers can opt-in to the Pascal scheduling model that does not support independent thread scheduling. See Application Compatibility for details.

+
+
+

+20.6.3. Global Memory +

+

Global memory behaves the same way as in devices of compute capability 5.x (See Global Memory).

+
+
+

+20.6.4. Shared Memory +

+

The amount of the unified data cache reserved for shared memory is configurable on a per kernel basis. For the Volta architecture (compute capability 7.0), the unified data cache has a size of 128 KB, and the shared memory capacity can be set to 0, 8, 16, 32, 64 or 96 KB. For the Turing architecture (compute capability 7.5), the unified data cache has a size of 96 KB, and the shared memory capacity can be set to either 32 KB or 64 KB. Unlike Kepler, the driver automatically configures the shared memory capacity for each kernel to avoid shared memory occupancy bottlenecks while also allowing concurrent execution with already launched kernels where possible. In most cases, the driver’s default behavior should provide optimal performance.

+

Because the driver is not always aware of the full workload, it is sometimes useful for applications to provide additional hints regarding the desired shared memory configuration. For example, a kernel with little or no shared memory use may request a larger carveout in order to encourage concurrent execution with later kernels that require more shared memory. The new cudaFuncSetAttribute() API allows applications to set a preferred shared memory capacity, or carveout, as a percentage of the maximum supported shared memory capacity (96 KB for Volta, and 64 KB for Turing).

+

cudaFuncSetAttribute() relaxes enforcement of the preferred shared capacity compared to the legacy cudaFuncSetCacheConfig() API introduced with Kepler. The legacy API treated shared memory capacities as hard requirements for kernel launch. As a result, interleaving kernels with different shared memory configurations would needlessly serialize launches behind shared memory reconfigurations. With the new API, the carveout is treated as a hint. The driver may choose a different configuration if required to execute the function or to avoid thrashing.

+
+
// Device code
+__global__ void MyKernel(...)
+{
+    __shared__ float buffer[BLOCK_DIM];
+    ...
+}
+
+// Host code
+int carveout = 50; // prefer shared memory capacity 50% of maximum
+// Named Carveout Values:
+// carveout = cudaSharedmemCarveoutDefault;   //  (-1)
+// carveout = cudaSharedmemCarveoutMaxL1;     //   (0)
+// carveout = cudaSharedmemCarveoutMaxShared; // (100)
+cudaFuncSetAttribute(MyKernel, cudaFuncAttributePreferredSharedMemoryCarveout, carveout);
+MyKernel <<<gridDim, BLOCK_DIM>>>(...);
+
+
+

In addition to an integer percentage, several convenience enums are provided as listed in the code comments above. Where a chosen integer percentage does not map exactly to a supported capacity (SM 7.0 devices support shared capacities of 0, 8, 16, 32, 64, or 96 KB), the next larger capacity is used. For instance, in the example above, 50% of the 96 KB maximum is 48 KB, which is not a supported shared memory capacity. Thus, the preference is rounded up to 64 KB.

+

Compute capability 7.x devices allow a single thread block to address the full capacity of shared memory: 96 KB on Volta, 64 KB on Turing. Kernels relying on shared memory allocations over 48 KB per block are architecture-specific, as such they must use dynamic shared memory (rather than statically sized arrays) and require an explicit opt-in using cudaFuncSetAttribute() as follows.

+
+
// Device code
+__global__ void MyKernel(...)
+{
+    extern __shared__ float buffer[];
+    ...
+}
+
+// Host code
+int maxbytes = 98304; // 96 KB
+cudaFuncSetAttribute(MyKernel, cudaFuncAttributeMaxDynamicSharedMemorySize, maxbytes);
+MyKernel <<<gridDim, blockDim, maxbytes>>>(...);
+
+
+

Otherwise, shared memory behaves the same way as for devices of compute capability 5.x (See Shared Memory).

+
+
+
+

+20.7. Compute Capability 8.x +

+
+

+20.7.1. Architecture +

+

A Streaming Multiprocessor (SM) consists of:

+
    +
  • 64 FP32 cores for single-precision arithmetic operations in devices of compute capability 8.0 and 128 FP32 cores in devices of compute capability 8.6, 8.7 and 8.9,

  • +
  • 32 FP64 cores for double-precision arithmetic operations in devices of compute capability 8.0 and 2 FP64 cores in devices of compute capability 8.6, 8.7 and 8.9

  • +
  • 64 INT32 cores for integer math,

  • +
  • 4 mixed-precision Third-Generation Tensor Cores supporting half-precision (fp16), __nv_bfloat16, tf32, sub-byte and double precision (fp64) matrix arithmetic for compute capabilities 8.0, 8.6 and 8.7 (see Warp Matrix Functions for details),

  • +
  • 4 mixed-precision Fourth-Generation Tensor Cores supporting fp8, fp16, __nv_bfloat16, tf32, sub-byte and fp64 for compute capability 8.9 (see Warp Matrix Functions for details),

  • +
  • 16 special function units for single-precision floating-point transcendental functions,

  • +
  • 4 warp schedulers.

  • +
+

An SM statically distributes its warps among its schedulers. Then, at every instruction issue time, each scheduler issues one instruction for one of its assigned warps that is ready to execute, if any.

+

An SM has:

+
    +
  • a read-only constant cache that is shared by all functional units and speeds up reads from the constant memory space, which resides in device memory,

  • +
  • a unified data cache and shared memory with a total size of 192 KB for devices of compute capability 8.0 and 8.7 (1.5x Volta’s 128 KB capacity) and 128 KB for devices of compute capabilities 8.6 and 8.9.

  • +
+

Shared memory is partitioned out of the unified data cache, and can be configured to various sizes (see Shared Memory). The remaining data cache serves as an L1 cache and is also used by the texture unit that implements the various addressing and data filtering modes mentioned in Texture and Surface Memory.

+
+
+

+20.7.2. Global Memory +

+

Global memory behaves the same way as for devices of compute capability 5.x (See Global Memory).

+
+
+

+20.7.3. Shared Memory +

+

Similar to the Volta architecture, the amount of the unified data cache reserved for shared memory is configurable on a per kernel basis. For the NVIDIA Ampere GPU Architecture, the unified data cache has a size of 192 KB for devices of compute capability 8.0 and 8.7 and 128 KB for devices of compute capabilities 8.6 and 8.9. The shared memory capacity can be set to 0, 8, 16, 32, 64, 100, 132 or 164 KB for devices of compute capability 8.0 and 8.7, and to 0, 8, 16, 32, 64 or 100 KB for devices of compute capabilities 8.6 and 8.9.

+

An application can set the carveout, i.e., the preferred shared memory capacity, with the cudaFuncSetAttribute().

+
+
cudaFuncSetAttribute(kernel_name, cudaFuncAttributePreferredSharedMemoryCarveout, carveout);
+
+
+

The API can specify the carveout either as an integer percentage of the maximum supported shared memory capacity of 164 KB for devices of compute capability 8.0 and 8.7 and 100 KB for devices of compute capabilities 8.6 and 8.9 respectively, or as one of the following values: {cudaSharedmemCarveoutDefault, cudaSharedmemCarveoutMaxL1, or cudaSharedmemCarveoutMaxShared. When using a percentage, the carveout is rounded up to the nearest supported shared memory capacity. For example, for devices of compute capability 8.0, 50% will map to a 100 KB carveout instead of an 82 KB one. Setting the cudaFuncAttributePreferredSharedMemoryCarveout is considered a hint by the driver; the driver may choose a different configuration, if needed.

+

Devices of compute capability 8.0 and 8.7 allow a single thread block to address up to 163 KB of shared memory, while devices of compute capabilities 8.6 and 8.9 allow up to 99 KB of shared memory. Kernels relying on shared memory allocations over 48 KB per block are architecture-specific, and must use dynamic shared memory rather than statically sized shared memory arrays. These kernels require an explicit opt-in by using cudaFuncSetAttribute() to set the cudaFuncAttributeMaxDynamicSharedMemorySize; see Shared Memory for the NVIDIA Volta GPU Architecture.

+

Note that the maximum amount of shared memory per thread block is smaller than the maximum shared memory partition available per SM. The 1 KB of shared memory not made available to a thread block is reserved for system use.

+
+
+
+

+20.8. Compute Capability 9.0 +

+
+

+20.8.1. Architecture +

+

A Streaming Multiprocessor (SM) consists of:

+
    +
  • 128 FP32 cores for single-precision arithmetic operations,

  • +
  • 64 FP64 cores for double-precision arithmetic operations,

  • +
  • 64 INT32 cores for integer math,

  • +
  • 4 mixed-precision fourth-generation Tensor Cores supporting the new FP8 input type in either E4M3 or E5M2 for exponent (E) and mantissa (M), half-precision (fp16), __nv_bfloat16, tf32, INT8 and double precision (fp64) matrix arithmetic (see Warp Matrix Functions for details) with sparsity support,

  • +
  • 16 special function units for single-precision floating-point transcendental functions,

  • +
  • 4 warp schedulers.

  • +
+

An SM statically distributes its warps among its schedulers. Then, at every instruction issue time, each scheduler issues one instruction for one of its assigned warps that is ready to execute, if any.

+

An SM has:

+
    +
  • a read-only constant cache that is shared by all functional units and speeds up reads from the constant memory space, which resides in device memory,

  • +
  • a unified data cache and shared memory with a total size of 256 KB for devices of compute capability 9.0 (1.33x NVIDIA Ampere GPU Architecture’s 192 KB capacity).

  • +
+

Shared memory is partitioned out of the unified data cache, and can be configured to various sizes (see Shared Memory). The remaining data cache serves as an L1 cache and is also used by the texture unit that implements the various addressing and data filtering modes mentioned in Texture and Surface Memory.

+
+
+

+20.8.2. Global Memory +

+

Global memory behaves the same way as for devices of compute capability 5.x (See Global Memory).

+
+
+

+20.8.3. Shared Memory +

+

Similar to the NVIDIA Ampere GPU architecture, the amount of the unified data cache reserved for shared memory is configurable on a per kernel basis. For the NVIDIA H100 Tensor Core GPU architecture, the unified data cache has a size of 256 KB for devices of compute capability 9.0. The shared memory capacity can be set to 0, 8, 16, 32, 64, 100, 132, 164, 196 or 228 KB.

+

As with the NVIDIA Ampere GPU architecture, an application can configure its preferred shared memory capacity, i.e., the carveout. Devices of compute capability 9.0 allow a single thread block to address up to 227 KB of shared memory. Kernels relying on shared memory allocations over 48 KB per block are architecture-specific, and must use dynamic shared memory rather than statically sized shared memory arrays. These kernels require an explicit opt-in by using cudaFuncSetAttribute() to set the cudaFuncAttributeMaxDynamicSharedMemorySize; see Shared Memory for the NVIDIA Volta GPU Architecture.

+

Note that the maximum amount of shared memory per thread block is smaller than the maximum shared memory partition available per SM. The 1 KB of shared memory not made available to a thread block is reserved for system use.

+
+
+

+20.8.4. Features Accelerating Specialized Computations +

+

The NVIDIA Hopper GPU architecture includes features to accelerate matrix multiply-accumulate (MMA) computations with:

+
    +
  • asynchronous execution of MMA instructions

  • +
  • MMA instructions acting on large matrices spanning a warp-group

  • +
  • dynamic reassignment of register capacity among warp-groups to support even larger matrices, and

  • +
  • operand matrices accessed directly from shared memory

  • +
+

See the PTX ISA for more details.

+

This feature set is only available within the CUDA compilation toolchain through inline PTX.

+

It is strongly recommended that applications utilize this complex feature set through CUDA-X libraries such as cuBLAS, cuDNN, or cuFFT.

+

It is strongly recommended that device kernels utilize this complex feature set through CUTLASS, a collection of CUDA C++ template abstractions for implementing high-performance matrix-multiplication (GEMM) and related computations at all levels and scales within CUDA.

+
+
+
+

+20.9. Compute Capability 10.0 +

+
+

+20.9.1. Architecture +

+

A Streaming Multiprocessor (SM) consists of:

+
    +
  • 128 FP32 cores for single-precision arithmetic operations,

  • +
  • 64 FP64 cores for double-precision arithmetic operations,

  • +
  • 64 INT32 cores for integer math,

  • +
  • 4 mixed-precision fifth-generation Tensor Cores supporting FP8 input type in either E4M3 or E5M2 for exponent (E) and mantissa (M), half-precision (fp16), __nv_bfloat16, tf32, INT8 and double precision (fp64) matrix arithmetic (see Warp Matrix Functions for details) with sparsity support,

  • +
  • 16 special function units for single-precision floating-point transcendental functions,

  • +
  • 4 warp schedulers.

  • +
+

An SM statically distributes its warps among its schedulers. Then, at every instruction issue time, each scheduler issues one instruction for one of its assigned warps that is ready to execute, if any.

+

An SM has:

+
    +
  • a read-only constant cache that is shared by all functional units and speeds up reads from the constant memory space, which resides in device memory,

  • +
  • a unified data cache and shared memory with a total size of 256 KB for devices of compute capability 10.0

  • +
+

Shared memory is partitioned out of the unified data cache, and can be configured to various sizes (see Shared Memory). The remaining data cache serves as an L1 cache and is also used by the texture unit that implements the various addressing and data filtering modes mentioned in Texture and Surface Memory.

+
+
+

+20.9.2. Global Memory +

+

Global memory behaves the same way as for devices of compute capability 5.x (See Global Memory).

+
+
+

+20.9.3. Shared Memory +

+

The amount of the unified data cache reserved for shared memory is configurable on a per kernel basis and is identical to compute capability 9.0. The unified data cache has a size of 256 KB for devices of compute capability 10.0. The shared memory capacity can be set to 0, 8, 16, 32, 64, 100, 132, 164, 196 or 228 KB.

+

As with the NVIDIA Ampere GPU architecture, an application can configure its preferred shared memory capacity, i.e., the carveout. Devices of compute capability 10.0 allow a single thread block to address up to 227 KB of shared memory. Kernels relying on shared memory allocations over 48 KB per block are architecture-specific, and must use dynamic shared memory rather than statically sized shared memory arrays. These kernels require an explicit opt-in by using cudaFuncSetAttribute() to set the cudaFuncAttributeMaxDynamicSharedMemorySize; see Shared Memory for the Volta architecture.

+

Note that the maximum amount of shared memory per thread block is smaller than the maximum shared memory partition available per SM. The 1 KB of shared memory not made available to a thread block is reserved for system use.

+
+
+

+20.9.4. Features Accelerating Specialized Computations +

+

The NVIDIA Blackwell GPU architecture extends features to accelerate matrix multiply-accumulate (MMA) from the NVIDIA Hopper GPU architecture.

+

See the PTX ISA for more details.

+

This feature set is only available within the CUDA compilation toolchain through inline PTX.

+

It is strongly recommended that applications utilize this complex feature set through CUDA-X libraries such as cuBLAS, cuDNN, or cuFFT.

+

It is strongly recommended that device kernels utilize this complex feature set through CUTLASS, a collection of CUDA C++ template abstractions for implementing high-performance matrix-multiplication (GEMM) and related computations at all levels and scales within CUDA.

+
+
+
+

+20.10. Compute Capability 12.0 +

+
+

+20.10.1. Architecture +

+

A Streaming Multiprocessor (SM) consists of:

+
    +
  • 128 FP32 cores for single-precision arithmetic operations,

  • +
  • 2 FP64 cores for double-precision arithmetic operations,

  • +
  • 64 INT32 cores for integer math,

  • +
  • Mixed-precision fifth-generation Tensor Core(s) supporting FP8 input type in either E4M3 or E5M2 for exponent (E) and mantissa (M), half-precision (fp16), __nv_bfloat16, tf32, INT8 and double precision (fp64) matrix arithmetic (see Warp Matrix Functions for details) with sparsity support,

  • +
  • 16 special function units for single-precision floating-point transcendental functions,

  • +
  • 4 warp schedulers.

  • +
+

An SM statically distributes its warps among its schedulers. Then, at every instruction issue time, each scheduler issues one instruction for one of its assigned warps that is ready to execute, if any.

+

An SM has:

+
    +
  • a read-only constant cache that is shared by all functional units and speeds up reads from the constant memory space, which resides in device memory,

  • +
  • a unified data cache and shared memory with a total size of 100 KB for devices of compute capability 12.0

  • +
+

Shared memory is partitioned out of the unified data cache, and can be configured to various sizes (see Shared Memory). The remaining data cache serves as an L1 cache and is also used by the texture unit that implements the various addressing and data filtering modes mentioned in Texture and Surface Memory.

+
+
+

+20.10.2. Global Memory +

+

Global memory behaves the same way as for devices of compute capability 5.x (See Global Memory).

+
+
+

+20.10.3. Shared Memory +

+

The amount of the unified data cache reserved for shared memory is configurable on a per kernel basis. The unified data cache has a size of 100 KB for devices of compute capability 12.0. The shared memory capacity can be set to 0, 8, 16, 32, 64, or 100 KB.

+

As with the NVIDIA Ampere GPU architecture, an application can configure its preferred shared memory capacity, i.e., the carveout. Devices of compute capability 12.0 allow a single thread block to address up to 99 KB of shared memory. Kernels relying on shared memory allocations over 48 KB per block are architecture-specific, and must use dynamic shared memory rather than statically sized shared memory arrays. These kernels require an explicit opt-in by using cudaFuncSetAttribute() to set the cudaFuncAttributeMaxDynamicSharedMemorySize; see Shared Memory for the Volta architecture.

+

Note that the maximum amount of shared memory per thread block is smaller than the maximum shared memory partition available per SM. The 1 KB of shared memory not made available to a thread block is reserved for system use.

+
+
+

+20.10.4. Features Accelerating Specialized Computations +

+

The NVIDIA Blackwell GPU architecture extends features to accelerate matrix multiply-accumulate (MMA) from the NVIDIA Hopper GPU architecture.

+

See the PTX ISA for more details.

+

This feature set is only available within the CUDA compilation toolchain through inline PTX.

+

It is strongly recommended that applications utilize this complex feature set through CUDA-X libraries such as cuBLAS, cuDNN, or cuFFT.

+

It is strongly recommended that device kernels utilize this complex feature set through CUTLASS, a collection of CUDA C++ template abstractions for implementing high-performance matrix-multiplication (GEMM) and related computations at all levels and scales within CUDA.

+
+
27
+
+

above 48 KB requires dynamic shared memory

+
+
28
+
+

2 FP64 cores for double-precision arithmetic operations for devices of compute capabilities 7.5

+
+
+
+
+
+
+

+21. Driver API +

+
+

Warning

+

This document has been replaced by a new CUDA Programming Guide. The information in this document should be considered legacy, and this document is no longer being updated as of CUDA 13.0. Please refer to the CUDA Programming Guide for up-to-date information on CUDA.

+
+

This section assumes knowledge of the concepts described in CUDA Runtime.

+

The driver API is implemented in the cuda dynamic library (cuda.dll or cuda.so) which is copied on the system during the installation of the device driver. All its entry points are prefixed with cu.

+

It is a handle-based, imperative API: Most objects are referenced by opaque handles that may be specified to functions to manipulate the objects.

+

The objects available in the driver API are summarized in Table 28.

+ + +++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+Table 28 Objects Available in the CUDA Driver API +

Object

Handle

Description

Device

CUdevice

CUDA-enabled device

Context

CUcontext

Roughly equivalent to a CPU process

Module

CUmodule

Roughly equivalent to a dynamic library

Function

CUfunction

Kernel

Heap memory

CUdeviceptr

Pointer to device memory

CUDA array

CUarray

Opaque container for one-dimensional or two-dimensional data on the device, readable via texture or surface references

Texture object

CUtexref

Object that describes how to interpret texture memory data

Surface reference

CUsurfref

Object that describes how to read or write CUDA arrays

Stream

CUstream

Object that describes a CUDA stream

Event

CUevent

Object that describes a CUDA event

+

The driver API must be initialized with cuInit() before any function from the driver API is called. A CUDA context must then be created that is attached to a specific device and made current to the calling host thread as detailed in Context.

+

Within a CUDA context, kernels are explicitly loaded as PTX or binary objects by the host code as described in Module. Kernels written in C++ must therefore be compiled separately into PTX or binary objects. Kernels are launched using API entry points as described in Kernel Execution.

+

Any application that wants to run on future device architectures must load PTX, not binary code. This is because binary code is architecture-specific and therefore incompatible with future architectures, whereas PTX code is compiled to binary code at load time by the device driver.

+

Here is the host code of the sample from Kernels written using the driver API:

+
+
int main()
+{
+    int N = ...;
+    size_t size = N * sizeof(float);
+
+    // Allocate input vectors h_A and h_B in host memory
+    float* h_A = (float*)malloc(size);
+    float* h_B = (float*)malloc(size);
+
+    // Initialize input vectors
+    ...
+
+    // Initialize
+    cuInit(0);
+
+    // Get number of devices supporting CUDA
+    int deviceCount = 0;
+    cuDeviceGetCount(&deviceCount);
+    if (deviceCount == 0) {
+        printf("There is no device supporting CUDA.\n");
+        exit (0);
+    }
+
+    // Get handle for device 0
+    CUdevice cuDevice;
+    cuDeviceGet(&cuDevice, 0);
+
+    // Create context
+    CUcontext cuContext;
+    cuCtxCreate(&cuContext, NULL, 0, cuDevice);
+
+    // Create module from binary file
+    CUmodule cuModule;
+    cuModuleLoad(&cuModule, "VecAdd.ptx");
+
+    // Allocate vectors in device memory
+    CUdeviceptr d_A;
+    cuMemAlloc(&d_A, size);
+    CUdeviceptr d_B;
+    cuMemAlloc(&d_B, size);
+    CUdeviceptr d_C;
+    cuMemAlloc(&d_C, size);
+
+    // Copy vectors from host memory to device memory
+    cuMemcpyHtoD(d_A, h_A, size);
+    cuMemcpyHtoD(d_B, h_B, size);
+
+    // Get function handle from module
+    CUfunction vecAdd;
+    cuModuleGetFunction(&vecAdd, cuModule, "VecAdd");
+
+    // Invoke kernel
+    int threadsPerBlock = 256;
+    int blocksPerGrid =
+            (N + threadsPerBlock - 1) / threadsPerBlock;
+    void* args[] = { &d_A, &d_B, &d_C, &N };
+    cuLaunchKernel(vecAdd,
+                   blocksPerGrid, 1, 1, threadsPerBlock, 1, 1,
+                   0, 0, args, 0);
+
+    ...
+}
+
+
+

Full code can be found in the vectorAddDrv CUDA sample.

+
+

+21.1. Context +

+

A CUDA context is analogous to a CPU process. All resources and actions performed within the driver API are encapsulated inside a CUDA context, and the system automatically cleans up these resources when the context is destroyed. Besides objects such as modules and texture or surface references, each context has its own distinct address space. As a result, CUdeviceptr values from different contexts reference different memory locations.

+

A host thread may have only one device context current at a time. When a context is created with cuCtxCreate(), it is made current to the calling host thread. CUDA functions that operate in a context (most functions that do not involve device enumeration or context management) will return CUDA_ERROR_INVALID_CONTEXT if a valid context is not current to the thread.

+

Each host thread has a stack of current contexts. cuCtxCreate() pushes the new context onto the top of the stack. cuCtxPopCurrent() may be called to detach the context from the host thread. The context is then “floating” and may be pushed as the current context for any host thread. cuCtxPopCurrent() also restores the previous current context, if any.

+

A usage count is also maintained for each context. cuCtxCreate() creates a context with a usage count of 1. cuCtxAttach() increments the usage count and cuCtxDetach() decrements it. A context is destroyed when the usage count goes to 0 when calling cuCtxDetach() or cuCtxDestroy().

+

The driver API is interoperable with the runtime and it is possible to access the primary context (see Initialization) managed by the runtime from the driver API via cuDevicePrimaryCtxRetain().

+

Usage count facilitates interoperability between third party authored code operating in the same context. For example, if three libraries are loaded to use the same context, each library would call cuCtxAttach() to increment the usage count and cuCtxDetach() to decrement the usage count when the library is done using the context. For most libraries, it is expected that the application will have created a context before loading or initializing the library; that way, the application can create the context using its own heuristics, and the library simply operates on the context handed to it. Libraries that wish to create their own contexts - unbeknownst to their API clients who may or may not have created contexts of their own - would use cuCtxPushCurrent() and cuCtxPopCurrent() as illustrated in the following figure.

+
+Library Context Management +
+

Figure 41 Library Context Management

+
+
+
+
+

+21.2. Module +

+

Modules are dynamically loadable packages of device code and data, akin to DLLs in Windows, that are output by nvcc (see Compilation with NVCC). The names for all symbols, including functions, global variables, and texture or surface references, are maintained at module scope so that modules written by independent third parties may interoperate in the same CUDA context.

+

This code sample loads a module and retrieves a handle to some kernel:

+
+
CUmodule cuModule;
+cuModuleLoad(&cuModule, "myModule.ptx");
+CUfunction myKernel;
+cuModuleGetFunction(&myKernel, cuModule, "MyKernel");
+
+
+

This code sample compiles and loads a new module from PTX code and parses compilation errors:

+
+
#define BUFFER_SIZE 8192
+CUmodule cuModule;
+CUjit_option options[3];
+void* values[3];
+char* PTXCode = "some PTX code";
+char error_log[BUFFER_SIZE];
+int err;
+options[0] = CU_JIT_ERROR_LOG_BUFFER;
+values[0]  = (void*)error_log;
+options[1] = CU_JIT_ERROR_LOG_BUFFER_SIZE_BYTES;
+values[1]  = (void*)BUFFER_SIZE;
+options[2] = CU_JIT_TARGET_FROM_CUCONTEXT;
+values[2]  = 0;
+err = cuModuleLoadDataEx(&cuModule, PTXCode, 3, options, values);
+if (err != CUDA_SUCCESS)
+    printf("Link error:\n%s\n", error_log);
+
+
+

This code sample compiles, links, and loads a new module from multiple PTX codes and parses link and compilation errors:

+
+
#define BUFFER_SIZE 8192
+CUmodule cuModule;
+CUjit_option options[6];
+void* values[6];
+float walltime;
+char error_log[BUFFER_SIZE], info_log[BUFFER_SIZE];
+char* PTXCode0 = "some PTX code";
+char* PTXCode1 = "some other PTX code";
+CUlinkState linkState;
+int err;
+void* cubin;
+size_t cubinSize;
+options[0] = CU_JIT_WALL_TIME;
+values[0] = (void*)&walltime;
+options[1] = CU_JIT_INFO_LOG_BUFFER;
+values[1] = (void*)info_log;
+options[2] = CU_JIT_INFO_LOG_BUFFER_SIZE_BYTES;
+values[2] = (void*)BUFFER_SIZE;
+options[3] = CU_JIT_ERROR_LOG_BUFFER;
+values[3] = (void*)error_log;
+options[4] = CU_JIT_ERROR_LOG_BUFFER_SIZE_BYTES;
+values[4] = (void*)BUFFER_SIZE;
+options[5] = CU_JIT_LOG_VERBOSE;
+values[5] = (void*)1;
+cuLinkCreate(6, options, values, &linkState);
+err = cuLinkAddData(linkState, CU_JIT_INPUT_PTX,
+                    (void*)PTXCode0, strlen(PTXCode0) + 1, 0, 0, 0, 0);
+if (err != CUDA_SUCCESS)
+    printf("Link error:\n%s\n", error_log);
+err = cuLinkAddData(linkState, CU_JIT_INPUT_PTX,
+                    (void*)PTXCode1, strlen(PTXCode1) + 1, 0, 0, 0, 0);
+if (err != CUDA_SUCCESS)
+    printf("Link error:\n%s\n", error_log);
+cuLinkComplete(linkState, &cubin, &cubinSize);
+printf("Link completed in %fms. Linker Output:\n%s\n", walltime, info_log);
+cuModuleLoadData(cuModule, cubin);
+cuLinkDestroy(linkState);
+
+
+

Full code can be found in the ptxjit CUDA sample.

+
+
+

+21.3. Kernel Execution +

+

cuLaunchKernel() launches a kernel with a given execution configuration.

+

Parameters are passed either as an array of pointers (next to last parameter of cuLaunchKernel()) where the nth pointer corresponds to the nth parameter and points to a region of memory from which the parameter is copied, or as one of the extra options (last parameter of cuLaunchKernel()).

+

When parameters are passed as an extra option (the CU_LAUNCH_PARAM_BUFFER_POINTER option), they are passed as a pointer to a single buffer where parameters are assumed to be properly offset with respect to each other by matching the alignment requirement for each parameter type in device code.

+

Alignment requirements in device code for the built-in vector types are listed in Table 7. For all other basic types, the alignment requirement in device code matches the alignment requirement in host code and can therefore be obtained using __alignof(). The only exception is when the host compiler aligns double and long long (and long on a 64-bit system) on a one-word boundary instead of a two-word boundary (for example, using gcc’s compilation flag -mno-align-double) since in device code these types are always aligned on a two-word boundary.

+

CUdeviceptr is an integer, but represents a pointer, so its alignment requirement is __alignof(void*).

+

The following code sample uses a macro (ALIGN_UP()) to adjust the offset of each parameter to meet its alignment requirement and another macro (ADD_TO_PARAM_BUFFER()) to add each parameter to the parameter buffer passed to the CU_LAUNCH_PARAM_BUFFER_POINTER option.

+
+
#define ALIGN_UP(offset, alignment) \
+      (offset) = ((offset) + (alignment) - 1) & ~((alignment) - 1)
+
+char paramBuffer[1024];
+size_t paramBufferSize = 0;
+
+#define ADD_TO_PARAM_BUFFER(value, alignment)                   \
+    do {                                                        \
+        paramBufferSize = ALIGN_UP(paramBufferSize, alignment); \
+        memcpy(paramBuffer + paramBufferSize,                   \
+               &(value), sizeof(value));                        \
+        paramBufferSize += sizeof(value);                       \
+    } while (0)
+
+int i;
+ADD_TO_PARAM_BUFFER(i, __alignof(i));
+float4 f4;
+ADD_TO_PARAM_BUFFER(f4, 16); // float4's alignment is 16
+char c;
+ADD_TO_PARAM_BUFFER(c, __alignof(c));
+float f;
+ADD_TO_PARAM_BUFFER(f, __alignof(f));
+CUdeviceptr devPtr;
+ADD_TO_PARAM_BUFFER(devPtr, __alignof(devPtr));
+float2 f2;
+ADD_TO_PARAM_BUFFER(f2, 8); // float2's alignment is 8
+
+void* extra[] = {
+    CU_LAUNCH_PARAM_BUFFER_POINTER, paramBuffer,
+    CU_LAUNCH_PARAM_BUFFER_SIZE,    &paramBufferSize,
+    CU_LAUNCH_PARAM_END
+};
+cuLaunchKernel(cuFunction,
+               blockWidth, blockHeight, blockDepth,
+               gridWidth, gridHeight, gridDepth,
+               0, 0, 0, extra);
+
+
+

The alignment requirement of a structure is equal to the maximum of the alignment requirements of its fields. The alignment requirement of a structure that contains built-in vector types, CUdeviceptr, or non-aligned double and long long, might therefore differ between device code and host code. Such a structure might also be padded differently. The following structure, for example, is not padded at all in host code, but it is padded in device code with 12 bytes after field f since the alignment requirement for field f4 is 16.

+
+
typedef struct {
+    float  f;
+    float4 f4;
+} myStruct;
+
+
+
+
+

+21.4. Interoperability between Runtime and Driver APIs +

+

An application can mix runtime API code with driver API code.

+

If a context is created and made current via the driver API, subsequent runtime calls will pick up this context instead of creating a new one.

+

If the runtime is initialized (implicitly as mentioned in CUDA Runtime), cuCtxGetCurrent() can be used to retrieve the context created during initialization. This context can be used by subsequent driver API calls.

+

The implicitly created context from the runtime is called the primary context (see Initialization). It can be managed from the driver API with the Primary Context Management functions.

+

Device memory can be allocated and freed using either API. CUdeviceptr can be cast to regular pointers and vice-versa:

+
+
CUdeviceptr devPtr;
+float* d_data;
+
+// Allocation using driver API
+cuMemAlloc(&devPtr, size);
+d_data = (float*)devPtr;
+
+// Allocation using runtime API
+cudaMalloc(&d_data, size);
+devPtr = (CUdeviceptr)d_data;
+
+
+

In particular, this means that applications written using the driver API can invoke libraries written using the runtime API (such as cuFFT, cuBLAS, …).

+

All functions from the device and version management sections of the reference manual can be used interchangeably.

+
+
+

+21.5. Driver Entry Point Access +

+
+

+21.5.1. Introduction +

+

The Driver Entry Point Access APIs provide a way to retrieve the address of a CUDA driver function. Starting from CUDA 11.3, users can call into available CUDA driver APIs using function pointers obtained from these APIs.

+

These APIs provide functionality similar to their counterparts, dlsym on POSIX platforms and GetProcAddress on Windows. The provided APIs will let users:

+
    +
  • Retrieve the address of a driver function using the CUDA Driver API.

  • +
  • Retrieve the address of a driver function using the CUDA Runtime API.

  • +
  • Request per-thread default stream version of a CUDA driver function. For more details, see Retrieve Per-thread Default Stream Versions.

  • +
  • Access new CUDA features on older toolkits but with a newer driver.

  • +
+
+
+

+21.5.2. Driver Function Typedefs +

+

To help retrieve the CUDA Driver API entry points, the CUDA Toolkit provides access to headers containing the function pointer definitions for all CUDA driver APIs. These headers are installed with the CUDA Toolkit and are made available in the toolkit’s include/ directory. The table below summarizes the header files containing the typedefs for each CUDA API header file.

+ + ++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+Table 29 Typedefs header files for CUDA driver APIs +

API header file

API Typedef header file

cuda.h

cudaTypedefs.h

cudaGL.h

cudaGLTypedefs.h

cudaProfiler.h

cudaProfilerTypedefs.h

cudaVDPAU.h

cudaVDPAUTypedefs.h

cudaEGL.h

cudaEGLTypedefs.h

cudaD3D9.h

cudaD3D9Typedefs.h

cudaD3D10.h

cudaD3D10Typedefs.h

cudaD3D11.h

cudaD3D11Typedefs.h

+

The above headers do not define actual function pointers themselves; they define the typedefs for function pointers. For example, cudaTypedefs.h has the below typedefs for the driver API cuMemAlloc:

+
+
typedef CUresult (CUDAAPI *PFN_cuMemAlloc_v3020)(CUdeviceptr_v2 *dptr, size_t bytesize);
+typedef CUresult (CUDAAPI *PFN_cuMemAlloc_v2000)(CUdeviceptr_v1 *dptr, unsigned int bytesize);
+
+
+

CUDA driver symbols have a version based naming scheme with a _v* extension in its name except for the first version. When the signature or the semantics of a specific CUDA driver API changes, we increment the version number of the corresponding driver symbol. In the case of the cuMemAlloc driver API, the first driver symbol name is cuMemAlloc and the next symbol name is cuMemAlloc_v2. The typedef for the first version which was introduced in CUDA 2.0 (2000) is PFN_cuMemAlloc_v2000. The typedef for the next version which was introduced in CUDA 3.2 (3020) is PFN_cuMemAlloc_v3020.

+

The typedefs can be used to more easily define a function pointer of the appropriate type in code:

+
+
PFN_cuMemAlloc_v3020 pfn_cuMemAlloc_v2;
+PFN_cuMemAlloc_v2000 pfn_cuMemAlloc_v1;
+
+
+
+
+

+21.5.3. Driver Function Retrieval +

+

Using the Driver Entry Point Access APIs and the appropriate typedef, we can get the function pointer to any CUDA driver API.

+
+

+21.5.3.1. Using the Driver API +

+

The driver API requires CUDA version as an argument to get the ABI compatible version for the requested driver symbol. CUDA Driver APIs have a per-function ABI denoted with a _v* extension. For example, consider the versions of cuStreamBeginCapture and their corresponding typedefs from cudaTypedefs.h:

+
+
// cuda.h
+CUresult CUDAAPI cuStreamBeginCapture(CUstream hStream);
+CUresult CUDAAPI cuStreamBeginCapture_v2(CUstream hStream, CUstreamCaptureMode mode);
+
+// cudaTypedefs.h
+typedef CUresult (CUDAAPI *PFN_cuStreamBeginCapture_v10000)(CUstream hStream);
+typedef CUresult (CUDAAPI *PFN_cuStreamBeginCapture_v10010)(CUstream hStream, CUstreamCaptureMode mode);
+
+
+

From the above typedefs in the code snippet, version suffixes _v10000 and _v10010 indicate that the above APIs were introduced in CUDA 10.0 and CUDA 10.1 respectively.

+
+
#include <cudaTypedefs.h>
+
+// Declare the entry points for cuStreamBeginCapture
+PFN_cuStreamBeginCapture_v10000 pfn_cuStreamBeginCapture_v1;
+PFN_cuStreamBeginCapture_v10010 pfn_cuStreamBeginCapture_v2;
+
+// Get the function pointer to the cuStreamBeginCapture driver symbol
+cuGetProcAddress("cuStreamBeginCapture", &pfn_cuStreamBeginCapture_v1, 10000, CU_GET_PROC_ADDRESS_DEFAULT, &driverStatus);
+// Get the function pointer to the cuStreamBeginCapture_v2 driver symbol
+cuGetProcAddress("cuStreamBeginCapture", &pfn_cuStreamBeginCapture_v2, 10010, CU_GET_PROC_ADDRESS_DEFAULT, &driverStatus);
+
+
+

Referring to the code snippet above, to retrieve the address to the _v1 version of the driver API cuStreamBeginCapture, the CUDA version argument should be exactly 10.0 (10000). Similarly, the CUDA version for retrieving the address to the _v2 version of the API should be 10.1 (10010). Specifying a higher CUDA version for retrieving a specific version of a driver API might not always be portable. For example, using 11030 here would still return the _v2 symbol, but if a hypothetical _v3 version is released in CUDA 11.3, the cuGetProcAddress API would start returning the newer _v3 symbol instead when paired with a CUDA 11.3 driver. Since the ABI and function signatures of the _v2 and _v3 symbols might differ, calling the _v3 function using the _v10010 typedef intended for the _v2 symbol would exhibit undefined behavior.

+

Note that requesting a driver API with an invalid CUDA version will return an error CUDA_ERROR_NOT_FOUND. In the above code examples, passing in a version less than 10000 (CUDA 10.0) would be invalid.

+
+
+

+21.5.3.2. Using the Runtime API +

+

The runtime API cudaGetDriverEntryPointByVersion uses the provided CUDA version to get the ABI compatible version for the requested driver symbol in the same way cuGetProcAddress does. In the below code snippet, the minimum CUDA version required would be CUDA 11.2 as cuMemAllocAsync was introduced then.

+
+
#include <cudaTypedefs.h>
+
+int cudaVersion;
+// Ensure a CUDA driver >= 11.2 is installed or we will get an error from cuGetProcAddress
+status = cuDriverGetVersion(&cudaVersion);
+if (cudaVersion >= 11020) {
+
+   // Declare the entry point
+   PFN_cuMemAllocAsync_v11020 pfn_cuMemAllocAsync;
+
+   // Intialize the entry point
+   cudaGetDriverEntryPointByVersion("cuMemAllocAsync", &pfn_cuMemAllocAsync, 11020, cudaEnableDefault, &driverStatus);
+
+   // Call the entry point
+   if(driverStatus == cudaDriverEntryPointSuccess && pfn_cuMemAllocAsync) {
+       pfn_cuMemAllocAsync(...);
+   }
+}
+
+
+
+
+

+21.5.3.3. Retrieve Per-thread Default Stream Versions +

+

Some CUDA driver APIs can be configured to have default stream or per-thread default stream semantics. Driver APIs having per-thread default stream semantics are suffixed with _ptsz or _ptds in their name. For example, cuLaunchKernel has a per-thread default stream variant named cuLaunchKernel_ptsz. With the Driver Entry Point Access APIs, users can request for the per-thread default stream version of the driver API cuLaunchKernel instead of the default stream version. Configuring the CUDA driver APIs for default stream or per-thread default stream semantics affects the synchronization behavior. More details can be found here.

+

The default stream or per-thread default stream versions of a driver API can be obtained by one of the following ways:

+
    +
  • Use the compilation flag --default-stream per-thread or define the macro CUDA_API_PER_THREAD_DEFAULT_STREAM to get per-thread default stream behavior.

  • +
  • Force default stream or per-thread default stream behavior using the flags CU_GET_PROC_ADDRESS_LEGACY_STREAM/cudaEnableLegacyStream or CU_GET_PROC_ADDRESS_PER_THREAD_DEFAULT_STREAM/cudaEnablePerThreadDefaultStream respectively.

  • +
+
+
+

+21.5.3.4. Access New CUDA features +

+

It is always recommended to install the latest CUDA toolkit to access new CUDA driver features, but if for some reason, a user does not want to update or does not have access to the latest toolkit, the API can be used to access new CUDA features with only an updated CUDA driver. For discussion, let us assume the user is on CUDA 12.3 and wants to use a new driver API cuFoo available in the CUDA 12.5 driver. The below code snippet illustrates this use-case:

+
+
int main()
+{
+    // Manually define the prototype as cudaTypedefs.h in CUDA 12.3 does not have the cuFoo typedef
+    typedef CUresult (CUDAAPI *PFN_cuFoo_v12050)(...);
+    PFN_cuFoo_v12050 pfn_cuFoo = NULL;
+    CUdriverProcAddressQueryResult driverStatus;
+    int cudaVersion;
+
+    // Ensure a CUDA driver >= 12.5 is installed or we will get an error from cuGetProcAddress
+    status = cuDriverGetVersion(&cudaVersion);
+    if (cudaVersion >= 12050) {
+        // Get the address for cuFoo API using cuGetProcAddress. Specify CUDA version as
+        // 12050 since cuFoo was introduced then
+        CUresult status = cuGetProcAddress("cuFoo", &pfn_cuFoo, 12050, CU_GET_PROC_ADDRESS_DEFAULT, &driverStatus);
+
+        if (status == CUDA_SUCCESS && pfn_cuFoo) {
+            pfn_cuFoo(...);
+        }
+        else {
+            printf("Cannot retrieve the address to cuFoo - driverStatus = %d\n", driverStatus);
+            assert(0);
+        }
+    }
+
+    // rest of code here
+}
+
+
+

In the next example, we discuss how to get a new version of an API released in a minor version of the CUDA Toolkit. Note that in the cuda.h header the version macro that would bump cuDeviceGetUuid to _v2 is not done until a major boundary. So during the 11.4+ releases the following example illustrates how to get the _v2 version.

+

Note in this case the original (not the _v2 version) typedef looks like:

+
+
typedef CUresult (CUDAAPI *PFN_cuDeviceGetUuid_v9020)(CUuuid *uuid, CUdevice_v1 dev);
+
+
+

But the _v2 version typedef looks like:

+
+
typedef CUresult (CUDAAPI *PFN_cuDeviceGetUuid_v11040)(CUuuid *uuid, CUdevice_v1 dev);
+
+
+
+
#include <cudaTypedefs.h>
+
+CUuuid uuid;
+CUdevice dev;
+CUresult status;
+int cudaVersion;
+CUdriverProcAddressQueryResult driverStatus;
+
+status = cuDeviceGet(&dev, 0); // Get device 0
+// handle status
+
+status = cuDriverGetVersion(&cudaVersion);
+// handle status
+
+// Ensure a CUDA driver >= 11.4 is installed or we will get an error from cuGetProcAddress
+status = cuDriverGetVersion(&cudaVersion);
+if (cudaVersion >= 11040) {
+   PFN_cuDeviceGetUuid_v11040 pfn_cuDeviceGetUuid;
+   status = cuGetProcAddress("cuDeviceGetUuid", &pfn_cuDeviceGetUuid, 11040, CU_GET_PROC_ADDRESS_DEFAULT, &driverStatus);
+   if(CUDA_SUCCESS == status && pfn_cuDeviceGetUuid) {
+      pfn_cuDeviceGetUuid(&uuid, dev);
+   }
+}
+
+
+
+
+
+

+21.5.4. Guidelines for cuGetProcAddress +

+

Below are guidelines to keep in mind when using cuGetProcAddress.

+
    +
  • Code the CUDA version passed to cuGetProcAddress to match the typedef version (do not use a compile time constant such as CUDA_VERSION or a dynamic version such as returned from cuDriverGetVersion)

  • +
  • Check the current driver version (such as from cuDriverGetVersion) is sufficient before calling cuGetProcAddress or an error is expected or an unexpected symbol may be returned

  • +
+
+

+21.5.4.1. Guidelines for Runtime API Usage +

+

Unless specified otherwise, the CUDA runtime API cudaGetDriverEntryPointByVersion will have similar guidelines as the driver entry point cuGetProcAddress since it allows for the user to request a specific CUDA driver version.

+
+
+
+

+21.5.5. Determining cuGetProcAddress Failure Reasons +

+

There are two types of errors with cuGetProcAddress. Those are (1) API/usage errors and (2) inability to find the driver API requested. The first error type will return error codes from the API via the CUresult return value. Things like passing NULL as the pfn variable or passing invalid flags.

+

The second error type encodes in the CUdriverProcAddressQueryResult *symbolStatus and can be used to help distinguish potential issues with the driver not being able to find the symbol requested. Take the following example:

+
+
// cuDeviceGetExecAffinitySupport was introduced in release CUDA 11.4
+#include <cuda.h>
+CUdriverProcAddressQueryResult driverStatus;
+cudaVersion = ...;
+status = cuGetProcAddress("cuDeviceGetExecAffinitySupport", &pfn, cudaVersion, 0, &driverStatus);
+if (CUDA_SUCCESS == status) {
+    if (CU_GET_PROC_ADDRESS_VERSION_NOT_SUFFICIENT == driverStatus) {
+        printf("We can use the new feature when you upgrade cudaVersion to 11.4, but CUDA driver is good to go!\n");
+        // Indicating cudaVersion was < 11.4 but run against a CUDA driver >= 11.4
+    }
+    else if (CU_GET_PROC_ADDRESS_SYMBOL_NOT_FOUND == driverStatus) {
+        printf("Please update both CUDA driver and cudaVersion to at least 11.4 to use the new feature!\n");
+        // Indicating driver is < 11.4 since string not found, doesn't matter what cudaVersion was
+    }
+    else if (CU_GET_PROC_ADDRESS_SUCCESS == driverStatus && pfn) {
+        printf("You're using cudaVersion and CUDA driver >= 11.4, using new feature!\n");
+        pfn();
+    }
+}
+
+
+

The first case with the return code CU_GET_PROC_ADDRESS_VERSION_NOT_SUFFICIENT indicates that the symbol was found when searching in the CUDA driver but it was added later than the cudaVersion supplied. In the example, specifying cudaVersion as anything 11030 or less and when running against a CUDA driver >= CUDA 11.4 would give this result of CU_GET_PROC_ADDRESS_VERSION_NOT_SUFFICIENT. This is because cuDeviceGetExecAffinitySupport was added in CUDA 11.4 (11040).

+

The second case with the return code CU_GET_PROC_ADDRESS_SYMBOL_NOT_FOUND indicates that the symbol was not found when searching in the CUDA driver. This can be due to a few reasons such as unsupported CUDA function due to older driver as well as just having a typo. In the latter, similar to the last example if the user had put symbol as CUDeviceGetExecAffinitySupport - notice the capital CU to start the string - cuGetProcAddress would not be able to find the API because the string doesn’t match. In the former case an example might be the user developing an application against a CUDA driver supporting the new API, and deploying the application against an older CUDA driver. Using the last example, if the developer developed against CUDA 11.4 or later but was deployed against a CUDA 11.3 driver, during their development they may have had a succesful cuGetProcAddress, but when deploying an application running against a CUDA 11.3 driver the call would no longer work with the CU_GET_PROC_ADDRESS_SYMBOL_NOT_FOUND returned in driverStatus.

+
+
+
+
+

+22. CUDA Environment Variables +

+
+

Warning

+

This document has been replaced by a new CUDA Programming Guide. The information in this document should be considered legacy, and this document is no longer being updated as of CUDA 13.0. Please refer to the CUDA Programming Guide for up-to-date information on CUDA.

+
+

The following table lists the CUDA environment variables. Environment variables related to the Multi-Process Service are documented in the Multi-Process Service section of the GPU Deployment and Management guide.

+ + +++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+Table 30 CUDA Environment Variables +

Variable

Values

Description

Device Enumeration and Properties

CUDA_VISIBLE_DEVICES

A comma-separated sequence of GPU identifiers +MIG support: MIG-<GPU-UUID>/<GPU instance ID>/<compute instance ID>

GPU identifiers are given as integer indices or as UUID strings. GPU UUID strings should follow the same format as given by nvidia-smi, such as GPU-8932f937-d72c-4106-c12f-20bd9faed9f6. However, for convenience, abbreviated forms are allowed; simply specify enough digits from the beginning of the GPU UUID to uniquely identify that GPU in the target system. For example, CUDA_VISIBLE_DEVICES=GPU-8932f937 may be a valid way to refer to the above GPU UUID, assuming no other GPU in the system shares this prefix. +Only the devices whose index is present in the sequence are visible to CUDA applications and they are enumerated in the order of the sequence. If one of the indices is invalid, only the devices whose index precedes the invalid index are visible to CUDA applications. For example, setting CUDA_VISIBLE_DEVICES to 2,1 causes device 0 to be invisible and device 2 to be enumerated before device 1. Setting CUDA_VISIBLE_DEVICES to 0,2,-1,1 causes devices 0 and 2 to be visible and device 1 to be invisible. +MIG format starts with MIG keyword and GPU UUID should follow the same format as given by nvidia-smi. For example, MIG-GPU-8932f937-d72c-4106-c12f-20bd9faed9f6/1/2. Only single MIG instance enumeration is supported.

CUDA_MANAGED_FORCE_DEVICE_ALLOC

0 or 1 (default is 0)

Forces the driver to place all managed allocations in device memory.

CUDA_DEVICE_ORDER

FASTEST_FIRST, PCI_BUS_ID, (default is FASTEST_FIRST)

FASTEST_FIRST causes CUDA to enumerate the available devices in fastest to slowest order using a simple heuristic. PCI_BUS_ID orders devices by PCI bus ID in ascending order.

Compilation

CUDA_CACHE_DISABLE

0 or 1 (default is 0)

Disables caching (when set to 1) or enables caching (when set to 0) for just-in-time-compilation. When disabled, no binary code is added to or retrieved from the cache.

CUDA_CACHE_PATH

filepath

+

Specifies the folder where the just-in-time compiler caches binary codes; the default values are:

+
    +
  • on Windows, %APPDATA%\NVIDIA\ComputeCache

  • +
  • on Linux, ~/.nv/ComputeCache

  • +
+

CUDA_CACHE_MAXSIZE

integer (default is 1073741824 (1 GiB) for desktop/server platforms and 268435456 (256 MiB) for embedded platforms and the maximum is 4294967296 (4 GiB))

Specifies the size in bytes of the cache used by the just-in-time compiler. Binary codes whose size exceeds the cache size are not cached. Older binary codes are evicted from the cache to make room for newer binary codes if needed.

CUDA_FORCE_PTX_JIT

0 or 1 (default is 0)

When set to 1, forces the device driver to ignore any binary code embedded in an application (see Application Compatibility) and to just-in-time compile embedded PTX code instead. If a kernel does not have embedded PTX code, it will fail to load. This environment variable can be used to validate that PTX code is embedded in an application and that its just-in-time compilation works as expected to guarantee application forward compatibility with future architectures (see Just-in-Time Compilation).

CUDA_DISABLE_PTX_JIT

0 or 1 (default is 0)

When set to 1, disables the just-in-time compilation of embedded PTX code and use the compatible binary code embedded in an application (see Application Compatibility). If a kernel does not have embedded binary code or the embedded binary was compiled for an incompatible architecture, then it will fail to load. This environment variable can be used to validate that an application has the compatible SASS code generated for each kernel.(see Binary Compatibility).

CUDA_FORCE_JIT

0 or 1 (default is 0)

When set to 1, forces the device driver to ignore any binary code embedded in an application (see Application Compatibility) and to just-in-time compile embedded PTX code instead. If a kernel does not have embedded PTX code, it will fail to load. This environment variable can be used to validate that PTX code is embedded in an application and that its just-in-time compilation works as expected to guarantee application forward compatibility with future architectures (see Just-in-Time Compilation). The behavior can be overridden for embedded PTX by setting CUDA_FORCE_PTX_JIT=0.

CUDA_DISABLE_JIT

0 or 1 (default is 0)

When set to 1, disables the just-in-time compilation of embedded PTX code and use the compatible binary code embedded in an application (see Application Compatibility). If a kernel does not have embedded binary code or the embedded binary was compiled for an incompatible architecture, then it will fail to load. This environment variable can be used to validate that an application has the compatible SASS code generated for each kernel.(see Binary Compatibility). The behavior can be overridden for embedded PTX by setting CUDA_DISABLE_PTX_JIT=0.

Execution

CUDA_LAUNCH_BLOCKING

0 or 1 (default is 0)

Disables (when set to 1) or enables (when set to 0) asynchronous kernel launches.

CUDA_DEVICE_MAX_CONNECTIONS

1 to 32 (default is 8)

Sets the number of compute and copy engine concurrent connections (work queues) from the host to each device of compute capability 3.5 and above.

CUDA_DEVICE_MAX_COPY_CONNECTIONS

1 to 32 (default is 8)

Sets the number of copy engine concurrent connections (work queues) per async copy engine from the host to each device of compute capability 8.0 and above. When both CUDA_DEVICE_MAX_CONNECTIONS and CUDA_DEVICE_MAX_COPY_CONNECTIONS are set, only the number of copy connections set by CUDA_DEVICE_MAX_CONNECTIONS will be overwritten.

CUDA_AUTO_BOOST

0 or 1

Overrides the autoboost behavior set by the –auto-boost-default option of nvidia-smi. If an application requests via this environment variable a behavior that is different from nvidia-smi’s, its request is honored if there is no other application currently running on the same GPU that successfully requested a different behavior, otherwise it is ignored.

CUDA_SCALE_LAUNCH_QUEUES

“0.25x”, “0.5x”, “2x” or “4x”

Scales the size of the queues available for launching work by a fixed multiplier.

cuda-gdb (on Linux platform)

CUDA_DEVICE_WAITS_ON_EXCEPTION

0 or 1 (default is 0)

When set to 1, a CUDA application will halt when a device exception occurs, allowing a debugger to be attached for further debugging.

MPS service (on Linux platform)

CUDA_DEVICE_DEFAULT_PERSISTING_L2_CACHE_PERCENTAGE_LIMIT

Percentage value (between 0 - 100, default is 0)

Devices of compute capability 8.x allow, a portion of L2 cache to be set-aside for persisting data accesses to global memory. When using CUDA MPS service, the set-aside size can only be controlled using this environment variable, before starting CUDA MPS control daemon. I.e., the environment variable should be set before running the command nvidia-cuda-mps-control -d.

Module loading

CUDA_MODULE_LOADING

DEFAULT, LAZY, EAGER (default is LAZY)

Specifies the module loading mode for the application. When set to EAGER, all kernels and data from a cubin, fatbin or a PTX file are fully loaded upon corresponding cuModuleLoad* and cuLibraryLoad* API call. When set to LAZY, loading of specific kernels is delayed to the point a CUfunc handle is extracted with cuModuleGetFunction or cuKernelGetFunction API calls and data from the cubin is loaded at load of first kernel in the cubin or at first access of variables in the cubin. Default behavior may change in future CUDA releases.

CUDA_MODULE_DATA_LOADING

DEFAULT, LAZY, EAGER (default is LAZY)

Specifies the data loading mode for the application. When set to EAGER, all data from a cubin, fatbin or a PTX file are fully loaded to memory upon corresponding cuLibraryLoad*. This doesn’t affect the LAZY or EAGER loading of kernels. When set to LAZY, loading of data is delayed to the point at which a handle is required. Default behavior may change in future CUDA releases. Data loading behavior is inherited from CUDA_MODULE_LOADING if this environment variable is not set.

CUDA_BINARY_LOADER_THREAD_COUNT

integer (default is 0)

Sets the number of CPU threads to use when loading device binaries. When set to 0, the number of CPU threads used is set to a default value of 1. When this is set, CU_JIT_BINARY_LOADER_THREAD_COUNT is ignored.

Pre-loading dependent libraries

CUDA_FORCE_PRELOAD_LIBRARIES

0 or 1 (default is 0)

When set to 1, forces the driver to preload the libraries required for NVVM and PTX just-in-time compilation during driver initialization. This will increase the memory footprint and the time taken for CUDA driver initialization. This environment variable needs to be set to avoid certain deadlock situations involving multiple CUDA threads.

CUDA Graphs

CUDA_GRAPHS_USE_NODE_PRIORITY

0 or 1

Overrides the cudaGraphInstantiateFlagUseNodePriority flag on graph instantiation. When set to 1, the flag will be set for all graphs and when set to 0, the flag will be cleared for all graphs.

CUDA Error Log Management

CUDA_LOG_FILE

stdout, stderr, or valid file path

Provides a location for printing error logs as they occur. See the Error Log Management section for more details.

+
+
+

+23. Error Log Management +

+
+

Warning

+

This document has been replaced by a new CUDA Programming Guide. The information in this document should be considered legacy, and this document is no longer being updated as of CUDA 13.0. Please refer to the CUDA Programming Guide for up-to-date information on CUDA.

+
+

The Error Log Management mechanism allows for CUDA API errors to be reported +to developers in a plain-English format that describes the cause of the issue.

+
+

+23.1. Background +

+

Traditionally, the only indication of a failed CUDA API call is the return of a non-zero code. +As of CUDA Toolkit 12.9, the CUDA Runtime defines over 100 different return codes +for error conditions, but many of them are generic and give the developer no assistance with debugging the cause.

+
+
+

+23.2. Activation +

+

Set the CUDA_LOG_FILE environment variable. Acceptable values are stdout, stderr, or a valid path on the system to write a file. +The log buffer can be dumped via API even if CUDA_LOG_FILE was not set before program execution. +NOTE: An error-free execution may not print any logs.

+
+
+

+23.3. Output +

+

Logs are output in the following format:

+
+
[Time][TID][Source][Severity][API Entry Point] Message
+
+
+

The following line is an actual error message that is generated if the developer tries to dump the Error Log Management logs to an unallocated buffer:

+
+
[22:21:32.099][25642][CUDA][E][cuLogsDumpToMemory] buffer cannot be NULL
+
+
+

Where before, all the developer would have gotten is CUDA_ERROR_INVALID_VALUE in the return code and possibly “invalid argument” if cuGetErrorString is called.

+
+
+

+23.4. API Description +

+

The CUDA Driver provides APIs in two categories for interacting with the Error Log Management feature.

+

This feature allows developers to register callback functions to be used whenever an error log is generated, where the callback signature is:

+
+
void callbackFunc(void *data, CUlogLevel logLevel, char *message, size_t length)
+
+
+

Callbacks are registered with this API:

+
+
CUresult cuLogsRegisterCallback(CUlogsCallback callbackFunc, void *userData, CUlogsCallbackHandle *callback_out)
+
+
+

Where userData is passed to the callback function without modifications. callback_out should be stored by the caller for use in cuLogsUnregisterCallback.

+
+
CUresult cuLogsUnregisterCallback(CUlogsCallbackHandle callback)
+
+
+

The other set of API functions are for managing the output of logs. An important concept is the log iterator, which points to the current end of the buffer:

+
+
CUresult cuLogsCurrent(CUlogIterator *iterator_out, unsigned int flags)
+
+
+

The iterator position can be kept by the calling software in situations where a dump of the entire log buffer is not desired. Currently, the flags parameter must be 0, with additional options reserved for future CUDA releases.

+

At any time, the error log buffer can be dumped to either a file or memory with these functions:

+
+
CUresult cuLogsDumpToFile(CUlogIterator *iterator, const char *pathToFile, unsigned int flags)
+CUresult cuLogsDumpToMemory(CUlogIterator *iterator, char *buffer, size_t *size, unsigned int flags)
+
+
+

If iterator is NULL, the entire buffer will be dumped, up to the maximum of 100 entries. If iterator is not NULL, logs will be dumped starting from that entry and the value of iterator will be updated to the current end of the logs, as if cuLogsCurrent had been called. If there have been more than 100 log entries into the buffer, a note will be added at the start of the dump noting this.

+

The flags parameter must be 0, with additional options reserved for future CUDA releases.

+

The cuLogsDumpToMemory function has additional considerations:

+
    +
  1. The buffer itself will be null-terminated, but each individual log entry will only be separated by a newline (n) character.

  2. +
  3. The maximum size of the buffer is 25600 bytes.

  4. +
  5. If the value provided in size is not sufficient to store all desired logs, a note will be added as the first entry and the oldest entries that do not fit will not be dumped.

  6. +
  7. After returning, size will contain the actual number of bytes written to the provided buffer.

  8. +
+
+
+

+23.5. Limitations and Known Issues +

+
    +
  1. The log buffer is limited to 100 entries. After this limit is reached, the oldest entries will be replaced and log dumps will contain a line noting the rollover.

  2. +
  3. Not all CUDA APIs are covered yet. This is an ongoing project to provide better usage error reporting for all APIs.

  4. +
  5. The Error Log Management log location (if given) will not be tested for validity until/unless a log is generated.

  6. +
  7. The Error Log Management APIs are currently only available via the CUDA Driver. Equivalent APIs will be added to the CUDA Runtime in a future release.

  8. +
  9. The log messages are not localized to any language and all provided logs are in US English.

  10. +
+
+
+
+

+24. Unified Memory Programming +

+
+

Warning

+

This document has been replaced by a new CUDA Programming Guide. The information in this document should be considered legacy, and this document is no longer being updated as of CUDA 13.0. Please refer to the CUDA Programming Guide for up-to-date information on CUDA.

+
+
+

Note

+

This chapter applies to devices with compute capability 5.0 or higher unless stated otherwise. +For devices with compute capability lower than 5.0, refer to the CUDA toolkit documentation for CUDA 11.8.

+
+

This documentation on Unified Memory is divided into 3 parts:

+ +
+

+24.1. Unified Memory Introduction +

+

CUDA Unified Memory provides all processors with:

+
    +
  • A single unified memory pool, that is, +a single pointer value enables all processors in the system +(all CPUs, all GPUs, etc.) +to access this memory with all of their native memory operations +(pointer dereferences, atomics, etc.).

  • +
  • Concurrent access to the unified memory pool from all processors in the system.

  • +
+

Unified Memory improves GPU programming in several ways:

+
    +
  • Productivity: GPU programs may access Unified Memory from GPU and CPU threads +concurrently without needing to create separate allocations (cudaMalloc()) and +copy memory manually back and forth (cudaMemcpy*()).

  • +
  • +

    Performance:

    +
      +
    • Data access speed may be maximized by migrating data towards processors that access it most frequently. +Applications can trigger manual migration of data and may use hints to control migration heuristics.

    • +
    • Total system memory usage may be reduced by avoiding duplicating memory on both CPUs and GPUs.

    • +
    +
  • +
  • Functionality: It enables GPU programs to work on data that exceeds the GPU memory’s capacity.

  • +
+

With CUDA Unified Memory, data movement still takes place, and hints may improve performance. +These hints are not required for correctness or functionality, that is, programmers may focus on parallelizing +their applications across GPUs and CPUs first, and worry about data-movement later in the development cycle as a performance optimization. +Note that the physical location of data is invisible to a program and may be changed at any time, +but accesses to the data’s virtual address will remain valid and coherent from any processor regardless of locality.

+

There are two main ways to obtain CUDA Unified Memory:

+ +
+

+24.1.1. System Requirements for Unified Memory +

+

The following table shows the different levels of support for CUDA Unified Memory, +the device properties required to detect these levels of support +and links to the documentation specific to each level of support:

+ + +++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+Table 31 Overview of levels of unified memory support +

Unified Memory Support Level

System device properties

Further documentation

Full CUDA Unified Memory: all memory has full support. This includes System-Allocated and CUDA Managed Memory.

+
+
Set to 1: pageableMemoryAccess +
+
+Systems with hardware acceleration also have the following properties set to 1:
+
+hostNativeAtomicSupported, pageableMemoryAccessUsesHostPageTables, directManagedMemAccessFromHost +
+
+

Unified Memory on devices with full CUDA Unified Memory support

Only CUDA Managed Memory has full support.

+
+
Set to 1: concurrentManagedAccess +
+
Set to 0: pageableMemoryAccess +
+
+

Unified Memory on devices with only CUDA Managed Memory support

CUDA Managed Memory without full support: unified addressing but no concurrent access.

+
+
Set to 1: managedMemory +
+
Set to 0: concurrentManagedAccess +
+
+
+ +

No Unified Memory support.

Set to 0: managedMemory

CUDA for Tegra Memory Management

+

The behavior of an application that attempts to use Unified Memory on a system that does not support it is undefined. +The following properties enable CUDA applications to check the level of system support for Unified Memory, and +to be portable between systems with different levels of support:

+
    +
  • +

    pageableMemoryAccess: This property is set to 1 on systems with CUDA Unified Memory support where +all threads may access System-Allocated Memory and CUDA Managed Memory. +These systems include NVIDIA Grace Hopper, IBM Power9 + Volta, and modern Linux systems with HMM enabled (see next bullet), among others.

    +
      +
    • Linux HMM requires Linux kernel version 6.1.24+, 6.2.11+ or 6.3+, +devices with compute capability 7.5 or higher and +a CUDA driver version 535+ installed with +Open Kernel Modules.

    • +
    +
  • +
  • concurrentManagedAccess: This property is set to 1 on systems with full CUDA Managed Memory support. +When this property is set to 0, there is only partial support for Unified Memory in CUDA Managed Memory. +For Tegra support of Unified Memory, see +CUDA for Tegra Memory Management.

  • +
+

A program may query the level of GPU support for CUDA Unified Memory, by querying the attributes in +Table 31 using cudaGetDeviceProperties().

+
+
+

+24.1.2. Programming Model +

+

With CUDA Unified Memory, separate allocations between host and device, and explicit memory transfers between them, are no longer required. +Programs may allocate Unified Memory in the following ways:

+ +

Most examples in this chapter provide at least two versions, one using CUDA Managed Memory and one using System-Allocated Memory. +Tabs allow you to choose between them. The following samples illustrate how Unified Memory simplifies CUDA programs:

+
+ +
+ ++++ + + + + + + +
+
+
__global__ void write_value(int* ptr, int v) {
+  *ptr = v;
+}
+
+int main() {
+  int* d_ptr = nullptr;
+  // Does not require any unified memory support
+  cudaMalloc(&d_ptr, sizeof(int));
+  write_value<<<1, 1>>>(d_ptr, 1);
+  int h_value;
+  // Copy memory back to the host and synchronize
+  cudaMemcpy(&h_value, d_ptr, sizeof(int),
+             cudaMemcpyDefault);
+  printf("value = %d\n", h_value); 
+  cudaFree(d_ptr); 
+  return 0;
+}
+
+
+
+
+
__global__ void write_value(int* ptr, int v) {
+  *ptr = v;
+}
+
+int main() {
+  // Requires System-Allocated Memory support
+  int* ptr = (int*)malloc(sizeof(int));
+  write_value<<<1, 1>>>(ptr, 1);
+  // Synchronize required
+  // (before, cudaMemcpy was synchronizing)
+  cudaDeviceSynchronize();
+  printf("value = %d\n", *ptr); 
+  free(ptr); 
+  return 0;
+}
+
+
+
+
+ +
+ ++++ + + + + + + +
+
+
__global__ void write_value(int* ptr, int v) {
+  *ptr = v;
+}
+
+int main() {
+  int* d_ptr = nullptr;
+  // Does not require any unified memory support
+  cudaMalloc(&d_ptr, sizeof(int));
+  write_value<<<1, 1>>>(d_ptr, 1);
+  int h_value;
+  // Copy memory back to the host and synchronize
+  cudaMemcpy(&h_value, d_ptr, sizeof(int),
+             cudaMemcpyDefault);
+  printf("value = %d\n", h_value); 
+  cudaFree(d_ptr); 
+  return 0;
+}
+
+
+
+
+
__global__ void write_value(int* ptr, int v) {
+  *ptr = v;
+}
+
+int main() {
+  // Requires System-Allocated Memory support
+  int value;
+  write_value<<<1, 1>>>(&value, 1);
+  // Synchronize required
+  // (before, cudaMemcpy was synchronizing)
+  cudaDeviceSynchronize();
+  printf("value = %d\n", value);
+  return 0;
+}
+
+
+
+
+ +
+ ++++ + + + + + + +
+
+
__global__ void write_value(int* ptr, int v) {
+  *ptr = v;
+}
+
+int main() {
+  int* d_ptr = nullptr;
+  // Does not require any unified memory support
+  cudaMalloc(&d_ptr, sizeof(int));
+  write_value<<<1, 1>>>(d_ptr, 1);
+  int h_value;
+  // Copy memory back to the host and synchronize
+  cudaMemcpy(&h_value, d_ptr, sizeof(int),
+             cudaMemcpyDefault);
+  printf("value = %d\n", h_value); 
+  cudaFree(d_ptr); 
+  return 0;
+}
+
+
+
+
+
__global__ void write_value(int* ptr, int v) {
+  *ptr = v;
+}
+
+int main() {
+  int* ptr = nullptr;
+  // Requires CUDA Managed Memory support
+  cudaMallocManaged(&ptr, sizeof(int));
+  write_value<<<1, 1>>>(ptr, 1);
+  // Synchronize required
+  // (before, cudaMemcpy was synchronizing)
+  cudaDeviceSynchronize();
+  printf("value = %d\n", *ptr); 
+  cudaFree(ptr); 
+  return 0;
+}
+
+
+
+
+ +
+ ++++ + + + + + + +
+
+
__global__ void write_value(int* ptr, int v) {
+  *ptr = v;
+}
+
+int main() {
+  int* d_ptr = nullptr;
+  // Does not require any unified memory support
+  cudaMalloc(&d_ptr, sizeof(int));
+  write_value<<<1, 1>>>(d_ptr, 1);
+  int h_value;
+  // Copy memory back to the host and synchronize
+  cudaMemcpy(&h_value, d_ptr, sizeof(int),
+             cudaMemcpyDefault);
+  printf("value = %d\n", h_value); 
+  cudaFree(d_ptr); 
+  return 0;
+}
+
+
+
+
+
__global__ void write_value(int* ptr, int v) {
+  *ptr = v;
+}
+
+// Requires CUDA Managed Memory support
+__managed__ int value;
+
+int main() {
+  write_value<<<1, 1>>>(&value, 1);
+  // Synchronize required
+  // (before, cudaMemcpy was synchronizing)
+  cudaDeviceSynchronize();
+  printf("value = %d\n", value);
+  return 0;
+}
+
+
+
+
+
+

In the example above, the device writes a value which is then read by the host:

+
    +
  • Without Unified Memory: both host- and device-side storage for the written value is required (h_value and d_ptr in the example), +as is an explicit copy between the two using cudaMemcpy().

  • +
  • +

    With Unified Memory: device accesses data directly from the host. ptr / value may be used without a separate h_value / d_ptr +allocation and no copy routine is required, greatly simplifying and reducing the size of the program. With:

    +
      +
    • System Allocated: no other changes required.

    • +
    • Managed Memory: data allocation changed to use cudaMallocManaged(), which returns a pointer valid from both host and device code.

    • +
    +
  • +
+
+

+24.1.2.1. Allocation APIs for System-Allocated Memory +

+

On systems with full CUDA Unified Memory support, all memory is unified memory. +This includes memory allocated with system allocation APIs, such as malloc(), mmap(), C++ new() operator, +and also automatic variables on CPU thread stacks, thread locals, global variables, and so on.

+

System-Allocated Memory may be populated on first touch, depending on the API and system settings used. +First touch means that:

+
    +
  • The allocation APIs allocate virtual memory and return immediately, and

  • +
  • physical memory is populated when a thread accesses the memory for the first time.

  • +
+

Usually, the physical memory will be chosen “close” to the processor that thread is running on. For example,

+
    +
  • GPU thread accesses it first: physical GPU memory of GPU that thread runs on is chosen.

  • +
  • CPU thread accesses it first: physical CPU memory in the memory NUMA node of the CPU core that thread runs on is chosen.

  • +
+

CUDA Unified Memory Hint and Prefetch APIs, cudaMemAdvise and cudaMemPreftchAsync, may be used on System-Allocated Memory. +These APIs are covered below in the Data Usage Hints section.

+
+
__global__ void printme(char *str) {
+  printf(str);
+}
+
+int main() {
+  // Allocate 100 bytes of memory, accessible to both Host and Device code
+  char *s = (char*)malloc(100);
+  // Physical allocation placed in CPU memory because host accesses "s" first
+  strncpy(s, "Hello Unified Memory\n", 99);
+  // Here we pass "s" to a kernel without explicitly copying
+  printme<<< 1, 1 >>>(s);
+  cudaDeviceSynchronize();
+  // Free as for normal CUDA allocations
+  cudaFree(s); 
+  return  0;
+}
+
+
+
+
+

+24.1.2.2. Allocation API for CUDA Managed Memory: cudaMallocManaged() +

+

On systems with CUDA Managed Memory support, unified memory may be allocated using:

+
+
__host__ cudaError_t cudaMallocManaged(void **devPtr, size_t size);
+
+
+

This API is syntactically identical to cudaMalloc(): it allocates size bytes of managed memory and +sets devPtr to refer to the allocation. +CUDA Managed Memory is also deallocated with cudaFree().

+

On systems with full CUDA Managed Memory support, managed memory allocations +may be accessed concurrently by all CPUs and GPUs in the system. +Replacing host calls to cudaMalloc() with cudaMallocManaged() does not impact program semantics on these systems; +device code is not able to call cudaMallocManaged().

+

The following example shows the use of cudaMallocManaged():

+
+
__global__ void printme(char *str) {
+  printf(str);
+}
+
+int main() {
+  // Allocate 100 bytes of memory, accessible to both Host and Device code
+  char *s;
+  cudaMallocManaged(&s, 100);
+  // Note direct Host-code use of "s"
+  strncpy(s, "Hello Unified Memory\n", 99);
+  // Here we pass "s" to a kernel without explicitly copying
+  printme<<< 1, 1 >>>(s);
+  cudaDeviceSynchronize();
+  // Free as for normal CUDA allocations
+  cudaFree(s); 
+  return  0;
+}
+
+
+
+

Note

+

For systems that support CUDA Managed Memory allocations, but do not provide full support, +see Coherency and Concurrency. +Implementation details (may change any time):

+
    +
  • Devices of compute capability 5.x allocate CUDA Managed Memory on the GPU.

  • +
  • Devices of compute capability 6.x and greater populate the memory on first touch, just like System-Allocated Memory APIs.

  • +
+
+
+
+

+24.1.2.3. Global-Scope Managed Variables Using __managed__ +

+

CUDA __managed__ variables behave as if they were allocated via cudaMallocManaged() +(see Allocation API for CUDA Managed Memory: cudaMallocManaged()). +They simplify programs with global variables, making it particularly easy to exchange data between host and device +without manual allocations or copying.

+

On systems with full CUDA Unified Memory support, +file-scope or global-scope variables cannot be directly accessed by device code. +But a pointer to these variables may be passed to the kernel as an argument, +see System-Allocated Memory: in-depth examples for examples.

+
+ +
+
+
__global__ void write_value(int* ptr, int v) {
+  *ptr = v;
+}
+
+int main() {
+  // Requires System-Allocated Memory support
+  int value;
+  write_value<<<1, 1>>>(&value, 1);
+  // Synchronize required
+  // (before, cudaMemcpy was synchronizing)
+  cudaDeviceSynchronize();
+  printf("value = %d\n", value);
+  return 0;
+}
+
+
+
+ +
+
+
__global__ void write_value(int* ptr, int v) {
+  *ptr = v;
+}
+
+// Requires CUDA Managed Memory support
+__managed__ int value;
+
+int main() {
+  write_value<<<1, 1>>>(&value, 1);
+  // Synchronize required
+  // (before, cudaMemcpy was synchronizing)
+  cudaDeviceSynchronize();
+  printf("value = %d\n", value);
+  return 0;
+}
+
+
+
+
+

Note the absence of explicit cudaMemcpy() commands and the fact that the written value value is visible on both CPU and GPU.

+

CUDA __managed__ variable implies __device__ and is equivalent to __managed__ __device__, which is also allowed. +Variables marked __constant__ may not be marked as __managed__.

+

A valid CUDA context is necessary for the correct operation of __managed__ variables. +Accessing __managed__ variables can trigger CUDA context creation if a context for the current device hasn’t already been created. +In the example above, accessing value before the kernel launch triggers context creation on the default device. +In the absence of that access, the kernel launch would have triggered context creation.

+

C++ objects declared as __managed__ are subject to certain specific constraints, particularly where static initializers are concerned. +Please refer to C++ Language Support for a list of these constraints.

+
+

Note

+

For devices with CUDA Managed Memory without full support, +visibility of __managed__ variables for asynchronous operations executing in CUDA streams +is discussed in the section on +Managing Data Visibility and Concurrent CPU + GPU Access with Streams.

+
+
+
+

+24.1.2.4. Difference between Unified Memory and Mapped Memory +

+

The main difference between Unified Memory and Mapped Memory is that +CUDA Mapped Memory does not guarantee that all kinds of memory accesses (for example atomics) are supported on all systems, +while Unified Memory does. The limited set of memory operations that are guaranteed to be portably supported by CUDA Mapped Memory +is available on more systems than Unified Memory.

+
+
+

+24.1.2.5. Pointer Attributes +

+

CUDA Programs may check whether a pointer addresses a CUDA Managed Memory allocation by calling cudaPointerGetAttributes() and +testing whether the pointer attribute value is cudaMemoryTypeManaged.

+

This API returns cudaMemoryTypeHost for System-Allocated Memory that has been registered with cudaHostRegister() +and cudaMemoryTypeUnregistered for System-Allocated Memory that CUDA is unaware of.

+

Pointer attributes do not state where the memory resides, they state how the memory was allocated or registered.

+

The following example shows how to detect the type of pointer at runtime:

+
+
char const* kind(cudaPointerAttributes a, bool pma, bool cma) {
+    switch(a.type) {
+    case cudaMemoryTypeHost: return pma?
+      "Unified: CUDA Host or Registered Memory" :
+      "Not Unified: CUDA Host or Registered Memory";
+    case cudaMemoryTypeDevice: return "Not Unified: CUDA Device Memory";
+    case cudaMemoryTypeManaged: return cma?
+      "Unified: CUDA Managed Memory" : "Not Unified: CUDA Managed Memory";
+    case cudaMemoryTypeUnregistered: return pma?
+      "Unified: System-Allocated Memory" :
+      "Not Unified: System-Allocated Memory";
+    default: return "unknown";
+    }
+}
+
+void check_pointer(int i, void* ptr) {
+  cudaPointerAttributes attr;
+  cudaPointerGetAttributes(&attr, ptr);
+  int pma = 0, cma = 0, device = 0;
+  cudaGetDevice(&device);
+  cudaDeviceGetAttribute(&pma, cudaDevAttrPageableMemoryAccess, device);
+  cudaDeviceGetAttribute(&cma, cudaDevAttrConcurrentManagedAccess, device);
+  printf("Pointer %d: memory is %s\n", i, kind(attr, pma, cma));
+}
+
+__managed__ int managed_var = 5;
+
+int main() {
+  int* ptr[5];
+  ptr[0] = (int*)malloc(sizeof(int));
+  cudaMallocManaged(&ptr[1], sizeof(int));
+  cudaMallocHost(&ptr[2], sizeof(int));
+  cudaMalloc(&ptr[3], sizeof(int));
+  ptr[4] = &managed_var;
+
+  for (int i = 0; i < 5; ++i) check_pointer(i, ptr[i]);
+  
+  cudaFree(ptr[3]);
+  cudaFreeHost(ptr[2]);
+  cudaFree(ptr[1]);
+  free(ptr[0]);
+  return 0;
+}
+
+
+
+
+

+24.1.2.6. Runtime detection of Unified Memory Support Level +

+

The following example shows how to detect the Unified Memory support level at runtime:

+
+
int main() {
+  int d;
+  cudaGetDevice(&d);
+
+  int pma = 0;
+  cudaDeviceGetAttribute(&pma, cudaDevAttrPageableMemoryAccess, d);
+  printf("Full Unified Memory Support: %s\n", pma == 1? "YES" : "NO");
+  
+  int cma = 0;
+  cudaDeviceGetAttribute(&cma, cudaDevAttrConcurrentManagedAccess, d);
+  printf("CUDA Managed Memory with full support: %s\n", cma == 1? "YES" : "NO");
+
+  return 0;
+}
+
+
+
+
+

+24.1.2.7. GPU Memory Oversubscription +

+

Unified Memory enables applications to oversubscribe the memory of any individual processor: +in other words they can allocate and share arrays larger than +the memory capacity of any individual processor in the system, +enabling among others out-of-core processing of datasets that do not fit within +a single GPU, without adding significant complexity to the programming model.

+
+
+

+24.1.2.8. Performance Hints +

+

The following sections describes the available unified memory performance hints, +which may be used on all Unified Memory, for example, CUDA Managed memory or, +on systems with full CUDA Unified Memory support, +also all System-Allocated Memory. +These APIs are hints, that is, they do not impact the semantics of applications, only their peformance. +That is, they can be added or removed anywhere on any application without impacting its results.

+

CUDA Unified Memory may not always have all the information necessary to make +the best performance decisions related to unified memory. +These performance hints enable the application to provide CUDA with more information.

+

Note that applications should only use these hints if they improve their performance.

+
+
+24.1.2.8.1. Data Prefetching +
+

The cudaMemPrefetchAsync API is an asynchronous stream-ordered API that may migrate data to reside closer to the specified processor. +The data may be accessed while it is being prefetched. +The migration does not begin until all prior operations in the stream have completed, +and completes before any subsequent operation in the stream.

+
+
cudaError_t cudaMemPrefetchAsync(const void *devPtr,
+                                 size_t count,
+                                 struct cudaMemLocation location,
+                                 unsigned int flags,
+                                 cudaStream_t stream);
+
+
+

A memory region containing [devPtr, devPtr + count) may be migrated to +the destination device location.id if location.type is cudaMemLocationTypeDevice +- or CPU if location.type is cudaMemLocationTypeHost - +when the prefetch task is executed in the given stream. +For details on flags, see the current +CUDA Runtime API documentation.

+

Consider a simple code example below:

+
+ +
+
+
void test_prefetch_sam(cudaStream_t s) {
+  char *data = (char*)malloc(N);
+  init_data(data, N);                                         // execute on CPU
+  cudaMemLocation location = {.type = cudaMemLocationTypeDevice, .id = myGpuId};
+  cudaMemPrefetchAsync(data, N, location, s, 0 /* flags */);  // prefetch to GPU
+  mykernel<<<(N + TPB - 1) / TPB, TPB, 0, s>>>(data, N);      // execute on GPU
+  location = {.type = cudaMemLocationTypeHost};
+  cudaMemPrefetchAsync(data, N, location, s, 0 /* flags */);  // prefetch to CPU
+  cudaStreamSynchronize(s);
+  use_data(data, N);
+  free(data);
+}
+
+
+
+ +
+
+
void test_prefetch_managed(cudaStream_t s) {
+  char *data;
+  cudaMallocManaged(&data, N);
+  init_data(data, N);                                         // execute on CPU
+  cudaMemLocation location = {.type = cudaMemLocationTypeDevice, .id = myGpuId};
+  cudaMemPrefetchAsync(data, N, location, s, 0 /* flags */);  // prefetch to GPU
+  mykernel<<<(N + TPB - 1) / TPB, TPB, 0, s>>>(data, N);      // execute on GPU
+  location = {.type = cudaMemLocationTypeHost};
+  cudaMemPrefetchAsync(data, N, location, s, 0 /* flags */);  // prefetch to CPU
+  cudaStreamSynchronize(s);
+  use_data(data, N);
+  cudaFree(data);
+}
+
+
+
+
+
+
+
+24.1.2.8.2. Memory Discarding +
+

The cudaMemDiscardBatchAsync API allows applications to inform the CUDA runtime that the contents +of specified memory ranges are no longer useful. The Unified Memory driver performs automatic memory +transfers due to fault-based migration or memory evictions to support device memory oversubscription. +These automatic memory transfers can sometimes be redundant, which severely decreases performance. +Marking an address range as ‘discard’ will inform the Unified Memory driver that the application has +consumed the contents in the range and there is no need to migrate this data on prefetches or page evictions +in order to make room for other allocations. Reading a discarded page without a subsequent write access +or prefetch will yield an indeterminate value. Whereas any new writes after the discard operation is guaranteed +to be seen by a subsequent read access. Concurrent accesses or prefetches to address ranges being discarded +will result in undefined behavior.

+
+
cudaError_t cudaMemDiscardBatchAsync(void **dptrs,
+                                    size_t *sizes,
+                                    size_t count,
+                                    unsigned long long flags,
+                                    cudaStream_t stream);
+
+
+

The function performs a batch of memory discards on address ranges specified in dptrs and sizes arrays. +Both arrays must be of the same length as specified by count. Each memory range must refer to +managed memory allocated via cudaMallocManaged or declared via __managed__ variables.

+

The cudaMemDiscardAndPrefetchBatchAsync API combines both discard and prefetch operations. Calling +cudaMemDiscardAndPrefetchBatchAsync is semantically equivalent to calling cudaMemDiscardBatchAsync +followed by cudaMemPrefetchBatchAsync, but is more optimal. This is useful when the application needs +the memory to be on the target location but does not need the contents of the memory.

+
+
cudaError_t cudaMemDiscardAndPrefetchBatchAsync(void **dptrs,
+                                               size_t *sizes,
+                                               size_t count,
+                                               struct cudaMemLocation *prefetchLocs,
+                                               size_t *prefetchLocIdxs,
+                                               size_t numPrefetchLocs,
+                                               unsigned long long flags,
+                                               cudaStream_t stream);
+
+
+

The prefetchLocs array specifies the destinations for prefetching, while prefetchLocIdxs +indicates which operations each prefetch location applies to. For example, if a batch has 10 operations +and the first 6 should be prefetched to one location while the remaining 4 to another, then +numPrefetchLocs would be 2, prefetchLocIdxs would be {0, 6}, and prefetchLocs would +contain the two destination locations.

+

Important considerations:

+
    +
  • Reading from a discarded range without a subsequent write or prefetch will return an indeterminate value

  • +
  • The discard operation can be undone by writing to the range or prefetching it via cudaMemPrefetchAsync

  • +
  • Any reads, writes, or prefetches that occur simultaneously with the discard operation result in undefined behavior

  • +
  • All devices must have a non-zero value for cudaDevAttrConcurrentManagedAccess

  • +
+
+
+
+24.1.2.8.3. Data Usage Hints +
+

When multiple processors simultaneously access the same data, +cudaMemAdvise may be used to hint how the data at +[devPtr, devPtr + count) will be accessed:

+
+
cudaError_t cudaMemAdvise(const void *devPtr,
+                          size_t count,
+                          enum cudaMemoryAdvise advice,
+                          struct cudaMemLocation location);
+
+
+

Where advice may take the following values:

+
    +
  • cudaMemAdviseSetReadMostly: +This implies that the data is mostly going to be read from and only occasionally written to. +In general, it allows trading off read bandwidth for write bandwidth on this region. +Example:

  • +
+
+
void test_advise_managed(cudaStream_t stream) {
+  char *dataPtr;
+  size_t dataSize = 64 * TPB;  // 16 KiB
+  // Allocate memory using cudaMallocManaged
+  // (malloc may be used on systems with full CUDA Unified memory support)
+  cudaMallocManaged(&dataPtr, dataSize);
+  // Set the advice on the memory region
+  cudaMemLocation loc = {.type = cudaMemLocationTypeDevice, .id = myGpuId};
+  cudaMemAdvise(dataPtr, dataSize, cudaMemAdviseSetReadMostly, loc);
+  int outerLoopIter = 0;
+  while (outerLoopIter < maxOuterLoopIter) {
+    // The data is written to in the outer loop on the CPU
+    init_data(dataPtr, dataSize);
+    // The data is made available to all GPUs by prefetching.
+    // Prefetching here causes read duplication of data instead
+    // of data migration
+    cudaMemLocation location;
+    location.type = cudaMemLocationTypeDevice;
+    for (int device = 0; device < maxDevices; device++) {
+      location.id = device;
+      cudaMemPrefetchAsync(dataPtr, dataSize, location, 0 /* flags */, stream);
+    }
+    // The kernel only reads this data in the inner loop
+    int innerLoopIter = 0;
+    while (innerLoopIter < maxInnerLoopIter) {
+      mykernel<<<32, TPB, 0, stream>>>((const char *)dataPtr, dataSize);
+      innerLoopIter++;
+    }
+    outerLoopIter++;
+  }
+  cudaFree(dataPtr);
+}
+
+
+
    +
  • cudaMemAdviseSetPreferredLocation: +In general, any memory may be migrated at any time to any location, for example, +when a given processor is running out of physical memory. +This hint tells the system that migrating this memory region away from +its preferred location is undesired, by setting the preferred location for +the data to be the physical memory belonging to device. +Passing in a value of cudaMemLocationTypeHost for location.type sets the preferred location as CPU memory. +Other hints, like cudaMemPrefetchAsync, may override this hint, +leading the memory to be migrated away from its preferred location.

  • +
+
    +
  • cudaMemAdviseSetAccessedBy: +In some systems, it may be beneficial for performance to establish a +mapping into memory before accessing the data from a given processor. +This hint tells the system that the data will be frequently accessed by location.id +when location.type is cudaMemLocationTypeDevice, +enabling the system to assume that creating these mappings pays off. +This hint does not imply where the data should reside, +but it can be combined with cudaMemAdviseSetPreferredLocation to specify that.

  • +
+

Each advice can be also unset by using one of the following values: +cudaMemAdviseUnsetReadMostly, cudaMemAdviseUnsetPreferredLocation and +cudaMemAdviseUnsetAccessedBy.

+
+
+
+24.1.2.8.4. Querying Data Usage Attributes on Managed Memory +
+

A program can query memory range attributes assigned through cudaMemAdvise +or cudaMemPrefetchAsync on CUDA Managed Memory by using the following API:

+
+
cudaMemRangeGetAttribute(void *data,
+                         size_t dataSize,
+                         enum cudaMemRangeAttribute attribute,
+                         const void *devPtr,
+                         size_t count);
+
+
+

This function queries an attribute of the memory range starting at devPtr with a size of count bytes. +The memory range must refer to managed memory allocated via cudaMallocManaged or +declared via __managed__ variables. It is possible to query the following attributes:

+
    +
  • cudaMemRangeAttributeReadMostly: +the result returned will be 1 if the entire memory range has the cudaMemAdviseSetReadMostly attribute set, or 0 otherwise.

  • +
  • cudaMemRangeAttributePreferredLocation: +the result returned will be a GPU device id or cudaCpuDeviceId if the entire +memory range has the corresponding processor as preferred location, +otherwise cudaInvalidDeviceId will be returned. +An application can use this query API to make decision about staging data through +CPU or GPU depending on the preferred location attribute of the managed pointer. +Note that the actual location of the memory range at the time +of the query may be different from the preferred location.

  • +
  • cudaMemRangeAttributeAccessedBy: +will return the list of devices that have that advise set for that memory range.

  • +
  • cudaMemRangeAttributeLastPrefetchLocation: +will return the last location to which the memory range was prefetched +explicitly using cudaMemPrefetchAsync. +Note that this simply returns the last location that the application requested to prefetch the memory range to. +It gives no indication as to whether the prefetch operation to that location has completed or even begun.

  • +
  • +
    +
    +cudaMemRangeAttributePreferredLocationType:
    +
    +

    will return the location type of the preferred location which will be cudaMemLocationTypeDevice if all pages +in the memory range have the same GPU as their preferred location, or will be cudaMemLocationTypeHost if all +pages in the memory range have the CPU as their preferred location, or it will be cudaMemLocationTypeHostNuma +if all the pages in the memory range have the same host NUMA node ID as their preferred location or it will be +cudaMemLocationTypeInvalid if either all the pages don’t have the same preferred location or some of the pages +don’t have a preferred location at all.

    +
    +
    +
  • +
  • +
    +
    +cudaMemRangeAttributePreferredLocationId:
    +
    +

    If the cudaMemRangeAttributePreferredLocationType query for the same address range returns cudaMemLocationTypeDevice, +it will be a valid device ordinal or if it returns cudaMemLocationTypeHostNuma, it will be a valid host NUMA node ID +or if it returns any other location type, the id should be ignored.

    +
    +
    +
  • +
  • +
    +
    +cudaMemRangeAttributeLastPrefetchLocationType:
    +
    +

    will be the last location type to which all pages in the memory range were prefetched explicitly via cudaMemPrefetchAsync +which will be cudaMemLocationTypeDevice if all pages in the memory range were prefetched to the same GPU, +or will be cudaMemLocationTypeHost if all pages in the memory range were prefetched to the CPU or it will be +cudaMemLocationTypeHostNuma if all the pages in the memory range were prefetched to the same host NUMA node ID +or it will be cudaMemLocationTypeInvalid if either all the pages were not prefetched to the same location or some +of the pages were never prefetched at all.

    +
    +
    +
  • +
  • +
    +
    +cudaMemRangeAttributeLastPrefetchLocationId:
    +
    +

    If the cudaMemRangeAttributeLastPrefetchLocationType query for the same address range returns cudaMemLocationTypeDevice, +it will be a valid device ordinal or if it returns cudaMemLocationTypeHostNuma, it will be a valid host NUMA node ID +or if it returns any other location type, the id should be ignored.

    +
    +
    +
  • +
+

Additionally, multiple attributes can be queried by using corresponding cudaMemRangeGetAttributes function.

+
+
+
+
+
+

+24.2. Unified memory on devices with full CUDA Unified Memory support +

+
+

+24.2.1. System-Allocated Memory: in-depth examples +

+

Systems with full CUDA Unified Memory support +allow the device to access any memory owned by the host process interacting with the device. +This section shows a few advanced use-cases, using a kernel that simply prints +the first 8 characters of an input character array to the standard output stream:

+
+
__global__ void kernel(const char* type, const char* data) {
+  static const int n_char = 8;
+  printf("%s - first %d characters: '", type, n_char);
+  for (int i = 0; i < n_char; ++i) printf("%c", data[i]);
+  printf("'\n");
+}
+
+
+

The following tabs show various ways of how this kernel may be called:

+
+ +
+
+
void test_malloc() {
+  const char test_string[] = "Hello World";
+  char* heap_data = (char*)malloc(sizeof(test_string));
+  strncpy(heap_data, test_string, sizeof(test_string));
+  kernel<<<1, 1>>>("malloc", heap_data);
+  ASSERT(cudaDeviceSynchronize() == cudaSuccess,
+    "CUDA failed with '%s'", cudaGetErrorString(cudaGetLastError()));
+  free(heap_data);
+}
+
+
+
+ +
+
+
void test_managed() {
+  const char test_string[] = "Hello World";
+  char* data;
+  cudaMallocManaged(&data, sizeof(test_string));
+  strncpy(data, test_string, sizeof(test_string));
+  kernel<<<1, 1>>>("managed", data);
+  ASSERT(cudaDeviceSynchronize() == cudaSuccess,
+    "CUDA failed with '%s'", cudaGetErrorString(cudaGetLastError()));
+  cudaFree(data);
+}
+
+
+
+ +
+
+
void test_stack() {
+  const char test_string[] = "Hello World";
+  kernel<<<1, 1>>>("stack", test_string);
+  ASSERT(cudaDeviceSynchronize() == cudaSuccess,
+    "CUDA failed with '%s'", cudaGetErrorString(cudaGetLastError()));
+}
+
+
+
+ +
+
+
void test_static() {
+  static const char test_string[] = "Hello World";
+  kernel<<<1, 1>>>("static", test_string);
+  ASSERT(cudaDeviceSynchronize() == cudaSuccess,
+    "CUDA failed with '%s'", cudaGetErrorString(cudaGetLastError()));
+}
+
+
+
+ +
+
+
const char global_string[] = "Hello World";
+
+void test_global() {
+  kernel<<<1, 1>>>("global", global_string);
+  ASSERT(cudaDeviceSynchronize() == cudaSuccess,
+    "CUDA failed with '%s'", cudaGetErrorString(cudaGetLastError()));
+}
+
+
+
+ +
+
+
// declared in separate file, see below
+extern char* ext_data;
+
+void test_extern() {
+  kernel<<<1, 1>>>("extern", ext_data);
+  ASSERT(cudaDeviceSynchronize() == cudaSuccess,
+    "CUDA failed with '%s'", cudaGetErrorString(cudaGetLastError()));
+}
+
+
+
+
/** This may be a non-CUDA file */
+char* ext_data;
+static const char global_string[] = "Hello World";
+
+void __attribute__ ((constructor)) setup(void) {
+  ext_data = (char*)malloc(sizeof(global_string));
+  strncpy(ext_data, global_string, sizeof(global_string));
+}
+
+void __attribute__ ((destructor)) tear_down(void) {
+  free(ext_data);
+}
+
+
+
+
+

The first three tabs above show the example as already detailed in the +Programming Model section. +The next three tabs show various ways a file-scope or global-scope variable can +be accessed from the device.

+

Note that for the extern variable, it could be declared and its memory +owned and managed by a third-party library, which does not interact with CUDA at all.

+

Also note that stack variables as well as file-scope and global-scope variables can +only be accessed through a pointer by the GPU. In this specific example, this is +convenient because the character array is already declared as a pointer: const char*. +However, consider the following example with a global-scope integer:

+
+
// this variable is declared at global scope
+int global_variable;
+
+__global__ void kernel_uncompilable() {
+  // this causes a compilation error: global (__host__) variables must not
+  // be accessed from __device__ / __global__ code
+  printf("%d\n", global_variable);
+}
+
+// On systems with pageableMemoryAccess set to 1, we can access the address
+// of a global variable. The below kernel takes that address as an argument
+__global__ void kernel(int* global_variable_addr) {
+  printf("%d\n", *global_variable_addr);
+}
+int main() {
+  kernel<<<1, 1>>>(&global_variable);
+  ...
+  return 0;
+}
+
+
+

In the example above, we need to ensure to pass a pointer to the global variable +to the kernel instead of directly accessing the global variable in the kernel. +This is because global variables without the __managed__ specifier are declared +as __host__-only by default, thus most compilers won’t allow using these +variables directly in device code as of now.

+
+

+24.2.1.1. File-backed Unified Memory +

+

Since systems with full CUDA Unified Memory support +allow the device to access any memory owned by the host process, +they can directly access file-backed memory.

+

Here, we show a modified version of the initial example shown in the previous section to use +file-backed memory in order to print a string from the GPU, read directly from an input file. +In the following example, the memory is backed by a physical file, but the example +applies to memory-backed files, too, as detailed in the section on +Inter-Process Communication (IPC) with Unified Memory.

+
+
__global__ void kernel(const char* type, const char* data) {
+  static const int n_char = 8;
+  printf("%s - first %d characters: '", type, n_char);
+  for (int i = 0; i < n_char; ++i) printf("%c", data[i]);
+  printf("'\n");
+}
+
+
+
+
void test_file_backed() {
+  int fd = open(INPUT_FILE_NAME, O_RDONLY);
+  ASSERT(fd >= 0, "Invalid file handle");
+  struct stat file_stat;
+  int status = fstat(fd, &file_stat);
+  ASSERT(status >= 0, "Invalid file stats");
+  char* mapped = (char*)mmap(0, file_stat.st_size, PROT_READ, MAP_PRIVATE, fd, 0);
+  ASSERT(mapped != MAP_FAILED, "Cannot map file into memory");
+  kernel<<<1, 1>>>("file-backed", mapped);
+  ASSERT(cudaDeviceSynchronize() == cudaSuccess,
+    "CUDA failed with '%s'", cudaGetErrorString(cudaGetLastError()));
+  ASSERT(munmap(mapped, file_stat.st_size) == 0, "Cannot unmap file");
+  ASSERT(close(fd) == 0, "Cannot close file");
+}
+
+
+

Note that on systems without the hostNativeAtomicSupported property, including +systems with Linux HMM enabled, +atomic accesses to file-backed memory are not supported.

+
+
+

+24.2.1.2. Inter-Process Communication (IPC) with Unified Memory +

+
+

Note

+

As of now, using IPC with Unified Memory can have significant performance implications.

+
+

Many applications prefer to manage one GPU per process, but still need to use Unified Memory, +for example for over-subscription, and access it from multiple GPUs.

+

CUDA IPC (see Interprocess Communication) +does not support Managed Memory: handles to this type of memory may not be shared through +any of the mechanisms discussed in this section. +On systems with full CUDA Unified Memory support, +System-Allocated Memory is Inter-Process Communication (IPC) capable. +Once access to System-Allocated Memory has been shared with other processes, +the same Programming Model applies, +similar to File-backed Unified Memory.

+

See the following references for more information on various ways of creating +IPC-capable System-Allocated Memory under Linux:

+ +

Note that it is not possible to share memory between different hosts and their devices using this technique.

+
+
+
+

+24.2.2. Performance Tuning +

+

In order to achieve good performance with Unified Memory, it is important to:

+
    +
  • Understand how paging works on your system, and how to avoid unnecessary page faults.

  • +
  • Understand the various mechanisms allowing you to keep data local to the accessing processor.

  • +
  • Consider tuning your application for the granularity of memory transfers of your system.

  • +
+

As general advice, Performance Hints +might provide improved performance, but using them incorrectly might degrade performance +compared to the default behavior. +Also note that any hint has a performance cost associated with it on the host, +thus useful hints must at the very least improve performance enough to overcome this cost.

+
+

+24.2.2.1. Memory Paging and Page Sizes +

+

Many of the sections for unified memory performance tuning assume prior knowledge on virtual addressing, +memory pages and page sizes. +This section attempts to define all necessary terms and explain why paging matters for performance.

+

All currently supported systems for Unified Memory use a virtual address space: +this means that memory addresses used by an application represent a virtual location +which might be mapped to a physical location where the memory actually resides.

+

All currently supported processors, including both CPUs and GPUs, additionally use +memory paging. Because all systems use a virtual address space, there are two types +of memory pages:

+
    +
  • Virtual pages: this represents a fixed-size contiguous chunk of virtual memory +per process tracked by the operating system, which can be mapped into physical memory. +Note that the virtual page is linked to the mapping: for example, a single +virtual address might be mapped into physical memory using different page sizes.

  • +
  • Physical pages: this represents a fixed-size contiguous chunk of memory +the processor’s main Memory Management Unit (MMU) supports and into which +a virtual page can be mapped.

  • +
+

Currently, all x86_64 CPUs use 4KiB physical pages. +Arm CPUs support multiple physical page sizes - 4KiB, 16KiB, 32KiB and 64KiB - depending on the exact CPU. +Finally, NVIDIA GPUs support multiple physical page sizes, but prefer 2MiB physical pages or larger. +Note that these sizes are subject to change in future hardware.

+

The default page size of virtual pages usually corresponds to the physical page size, +but an application may use different page sizes as long as they are supported by the +operating system and the hardware. Typically, supported virtual page sizes must be +powers of 2 and multiples of the physical page size.

+

The logical entity tracking the mapping of virtual pages into physical pages will be referred to as a page table, +and each mapping of a given virtual page with a given virtual size to physical pages is called a page table entry (PTE). +All supported processors provide specific caches for the page table to speed up the translation of +virtual addresses to physical addresses. These caches are called translation lookaside buffers (TLBs).

+

There are two important aspects for performance tuning of applications:

+
    +
  • the choice of virtual page size,

  • +
  • whether the system offers a combined page table used by both CPUs and GPUs, +or separate page tables for each CPU and GPU individually.

  • +
+
+
+24.2.2.1.1. Choosing the right page size +
+

In general, small page sizes lead to less (virtual) memory fragmentation but more TLB misses, +whereas larger page sizes lead to more memory fragmentation but less TLB misses. +Additionally, memory migration is generally more expensive with larger page sizes compared to +smaller page sizes, because we typically migrate full memory pages. This can cause +larger latency spikes in an application using large page sizes. See also the next section +for more details on page faults.

+

One important aspect for performance tuning is that TLB misses are generally +significantly more expensive on the GPU compared to the CPU. This means that +if a GPU thread frequently accesses random locations of Unified Memory mapped +using a small enough page size, it might be significantly slower compared to +the same accesses to Unified Memory mapped using a large enough page size. +While a similar effect might occur for a CPU thread randomly accessing a large +area of memory mapped using a small page size, the slowdown is less pronounced, +meaning that the application might want to trade-off this slowdown with +having less memory fragmentation.

+

Note that in general, applications should not tune their performance to the +physical page size of a given processor, since physical page sizes are subject +to change depending on the hardware. The advice above only applies to virtual +page sizes.

+
+
+
+24.2.2.1.2. CPU and GPU page tables: hardware coherency vs. software coherency +
+
+

Note

+

In the remainder of the performance tuning documentation, we will refer +to systems with a combined page table for both CPUs and GPUs as hardware +coherent systems. Systems with separate page tables for CPUs and GPUs are +referred to as software coherent.

+
+

Hardware coherent systems such as NVIDIA Grace Hopper offer a logically combined +page table for both CPUs and GPUs. +This is important because in order to access +System-Allocated Memory from the GPU, +the GPU uses whichever page table entry was created by the CPU for the requested memory. +If that page table entry uses the default CPU page size of 4KiB or 64KiB, +accesses to large virtual memory areas will cause significant TLB misses, +thus significant slowdowns.

+

See the section on configuring huge pages for examples on how to ensure +System-Allocated Memory uses large enough page sizes to avoid this type of issue.

+

On the other hand, on systems where the CPUs and GPUs each have their own logical +page table, different performance tuning aspects should be considered: +in order to guarantee coherency, these systems +usually use page faults in case a processor accesses a memory address mapped +into the physical memory of a different processor. Such a page fault means that:

+
    +
  • it needs to be ensured that the currently owning processor (where the physical page currently resides) +cannot access this page anymore, either by deleting the page table entry or updating it.

  • +
  • it needs to be ensured that the processor requesting access can access this page, +either by creating a new page table entry or updating and existing entry, such that +it becomes valid/active.

  • +
  • the physical page backing this virtual page must be moved/migrated to the processor +requesting access: this can be an expensive operation, and the amount of work +is proportional to the page size.

  • +
+

Overall, hardware coherent systems provide significant performance benefits +compared to software coherent systems in cases where frequent concurrent accesses +to the same memory page are made by both CPU and GPU threads:

+
    +
  • less page-faults: these systems do not need to use page-faults for emulating coherency or migrating memory,

  • +
  • less contention: these systems are coherent at cache-line granularity instead of page-size granularity, that is, +when there is contention from multiple processors within a cache line, only the cache line is exchanged which is much smaller than the smallest page-size, +and when the different processors access different cache-lines within a page, then there is no contention.

  • +
+

This impacts the performance of the following scenarios:

+
    +
  • Atomic updates to the same address concurrently from both CPUs and GPUs.

  • +
  • Signaling a GPU thread from a CPU thread or vice-versa.

  • +
+
+
+
+

+24.2.2.2. Direct Unified Memory Access from host +

+

Some devices have hardware support for coherent reads, stores and atomic accesses +from the host on GPU-resident unified memory. +These devices have the attribute cudaDevAttrDirectManagedMemAccessFromHost set to 1. +Note that all hardware coherent systems have +this attribute set for NVLink-connected devices. +On these systems, the host has direct access to GPU-resident memory without page faults and +data migration (see Data Usage Hints +for more details on memory usage hints). Note that with CUDA Managed Memory, +the cudaMemAdviseSetAccessedBy hint with location type cudaMemLocationTypeHost is necessary +to enable this direct access without page faults.

+

Consider an example code below:

+
+ +
+
+
__global__ void write(int *ret, int a, int b) {
+  ret[threadIdx.x] = a + b + threadIdx.x;
+}
+
+__global__ void append(int *ret, int a, int b) {
+  ret[threadIdx.x] += a + b + threadIdx.x;
+}
+void test_malloc() {
+  int *ret = (int*)malloc(1000 * sizeof(int));
+  // for shared page table systems, the following hint is not necessary
+  cudaMemLocation location = {.type = cudaMemLocationTypeHost};
+  cudaMemAdvise(ret, 1000 * sizeof(int), cudaMemAdviseSetAccessedBy, location);
+
+  write<<< 1, 1000 >>>(ret, 10, 100);            // pages populated in GPU memory
+  cudaDeviceSynchronize();
+  for(int i = 0; i < 1000; i++)
+      printf("%d: A+B = %d\n", i, ret[i]);        // directManagedMemAccessFromHost=1: CPU accesses GPU memory directly without migrations
+                                                  // directManagedMemAccessFromHost=0: CPU faults and triggers device-to-host migrations
+  append<<< 1, 1000 >>>(ret, 10, 100);            // directManagedMemAccessFromHost=1: GPU accesses GPU memory without migrations
+  cudaDeviceSynchronize();                        // directManagedMemAccessFromHost=0: GPU faults and triggers host-to-device migrations
+  free(ret);
+}
+
+
+
+ +
+
+
__global__ void write(int *ret, int a, int b) {
+  ret[threadIdx.x] = a + b + threadIdx.x;
+}
+
+__global__ void append(int *ret, int a, int b) {
+  ret[threadIdx.x] += a + b + threadIdx.x;
+}
+
+void test_managed() {
+  int *ret;
+  cudaMallocManaged(&ret, 1000 * sizeof(int));
+  cudaMemLocation location = {.type = cudaMemLocationTypeHost};
+  cudaMemAdvise(ret, 1000 * sizeof(int), cudaMemAdviseSetAccessedBy, location);  // set direct access hint
+
+  write<<< 1, 1000 >>>(ret, 10, 100);            // pages populated in GPU memory
+  cudaDeviceSynchronize();
+  for(int i = 0; i < 1000; i++)
+      printf("%d: A+B = %d\n", i, ret[i]);        // directManagedMemAccessFromHost=1: CPU accesses GPU memory directly without migrations
+                                                  // directManagedMemAccessFromHost=0: CPU faults and triggers device-to-host migrations
+  append<<< 1, 1000 >>>(ret, 10, 100);            // directManagedMemAccessFromHost=1: GPU accesses GPU memory without migrations
+  cudaDeviceSynchronize();                        // directManagedMemAccessFromHost=0: GPU faults and triggers host-to-device migrations
+  cudaFree(ret); 
+}
+
+
+
+
+

After write kernel is completed, ret will be created and initialized in GPU memory. +Next, the CPU will access ret followed by append kernel using the same ret memory again. +This code will show different behavior depending on the system architecture and support of hardware coherency:

+
    +
  • On systems with directManagedMemAccessFromHost=1: +CPU accesses to the managed buffer will not trigger any migrations; +the data will remain resident in GPU memory and any subsequent GPU kernels +can continue to access it directly without inflicting faults or migrations.

  • +
  • On systems with directManagedMemAccessFromHost=0: +CPU accesses to the managed buffer will page fault and initiate data migration; +any GPU kernel trying to access the same data first time will page fault and +migrate pages back to GPU memory.

  • +
+
+
+

+24.2.2.3. Host Native Atomics +

+

Some devices, including NVLink-connected devices in +hardware coherent systems, support hardware-accelerated +atomic accesses to CPU-resident memory. This implies that atomic accesses to host memory +do not have to be emulated with a page fault. +For these devices, the attribute cudaDevAttrHostNativeAtomicSupported is set to 1.

+
+
+

+24.2.2.4. Atomic accesses & synchronization primitives +

+

CUDA Unified Memory supports all atomic operations available to host and device threads, +enabling all threads to cooperate by concurrently accessing the same shared memory location. +The CUDA C++ standard library +provides many heterogeneous synchronization primitives tuned for concurrent use between host and device threads, +including cuda::atomic, cuda::atomic_ref, cuda::barrier, cuda::semaphore, among many others.

+

On systems without CPU and GPU page tables: hardware coherency vs. software coherency, +atomic accesses from the device to file-backed host memory are not supported. +The following example code is valid on systems with CPU and GPU page tables: hardware coherency vs. software coherency +but exhibits undefined behavior on other systems:

+
+
#include <cuda/atomic>
+
+#include <cstdio>
+#include <fcntl.h>
+#include <sys/mman.h>
+
+#define ERR(msg, ...) { fprintf(stderr, msg, ##__VA_ARGS__); return EXIT_FAILURE; }
+
+__global__ void kernel(int* ptr) {
+  cuda::atomic_ref{*ptr}.store(2);
+}
+
+int main() {
+  // this will be closed/deleted by default on exit
+  FILE* tmp_file = tmpfile64();
+  // need to allocate space in the file, we do this with posix_fallocate here
+  int status = posix_fallocate(fileno(tmp_file), 0, 4096);
+  if (status != 0) ERR("Failed to allocate space in temp file\n");
+  int* ptr = (int*)mmap(NULL, 4096, PROT_READ | PROT_WRITE, MAP_PRIVATE, fileno(tmp_file), 0);
+  if (ptr == MAP_FAILED) ERR("Failed to map temp file\n");
+
+  // initialize the value in our file-backed memory
+  *ptr = 1;
+  printf("Atom value: %d\n", *ptr);
+
+  // device and host thread access ptr concurrently, using cuda::atomic_ref
+  kernel<<<1, 1>>>(ptr);
+  while (cuda::atomic_ref{*ptr}.load() != 2);
+  // this will always be 2
+  printf("Atom value: %d\n", *ptr);
+
+  return EXIT_SUCCESS;
+}
+
+
+

On systems without CPU and GPU page tables: hardware coherency vs. software coherency, +atomic accesses to unified memory may incur page faults which can lead to significant latencies. +Note that this is not the case for all GPU atomics to CPU memory on these systems: +operations listed by nvidia-smi -q | grep "Atomic Caps Outbound" may avoid page faults.

+

On systems with CPU and GPU page tables: hardware coherency vs. software coherency, +atomics between host and device do not require page faults, +but may still fault for other reasons that any memory access can fault for.

+
+
+

+24.2.2.5. Memcpy()/Memset() Behavior With Unified Memory +

+

cudaMemcpy*() and cudaMemset*() accept any unified memory pointer as arguments.

+

For cudaMemcpy*(), the direction specified as cudaMemcpyKind is a performance hint, +which can have a higher performance impact if any of the arguments is a unified memory pointer.

+

Thus, it is recommended to follow the following performance advice:

+
    +
  • When the physical location of unified memory is known, use an accurate cudaMemcpyKind hint.

  • +
  • Prefer cudaMemcpyDefault over an inaccurate cudaMemcpyKind hint.

  • +
  • Always use populated (initialized) buffers: avoid using these APIs to initialize memory.

  • +
  • Avoid using cudaMemcpy*() if both pointers point to System-Allocated Memory: +launch a kernel or use a CPU memory copy algorithm such as std::memcpy instead.

  • +
+
+
+
+
+

+24.3. Unified memory on devices without full CUDA Unified Memory support +

+
+

+24.3.1. Unified memory on devices with only CUDA Managed Memory support +

+

For devices with compute capability 6.x or higher but without pageable memory access, +CUDA Managed Memory is fully supported and coherent. +The programming model and performance tuning of unified memory is largely similar +to the model as described in +Unified memory on devices with full CUDA Unified Memory support, +with the notable exception that system allocators cannot be used to allocate memory. +Thus, the following list of sub-sections do not apply:

+ +
+
+

+24.3.2. Unified memory on Windows or devices with compute capability 5.x +

+

Devices with compute capability lower than 6.0 or Windows platforms support CUDA Managed Memory v1.0 with limited support for data migration and coherency as well as memory oversubscription. The following sub-sections describe in more detail how to use and optimize Managed Memory on these platforms.

+
+

+24.3.2.1. Data Migration and Coherency +

+

GPU architectures of compute capability lower than 6.0 do not support fine-grained movement of the managed data to GPU on-demand. Whenever a GPU kernel is launched all managed memory generally has to be transferred to GPU memory to avoid faulting on memory access. With compute capability 6.x a new GPU page faulting mechanism is introduced that provides more seamless Unified Memory functionality. Combined with the system-wide virtual address space, page faulting provides several benefits. First, page faulting means that the CUDA system software doesn’t need to synchronize all managed memory allocations to the GPU before each kernel launch. If a kernel running on the GPU accesses a page that is not resident in its memory, it faults, allowing the page to be automatically migrated to the GPU memory on-demand. Alternatively, the page may be mapped into the GPU address space for access over the PCIe or NVLink interconnects (mapping on access can sometimes be faster than migration). Note that Unified Memory is system-wide: GPUs (and CPUs) can fault on and migrate memory pages either from CPU memory or from the memory of other GPUs in the system.

+
+
+

+24.3.2.2. GPU Memory Oversubscription +

+

Devices of compute capability lower than 6.0 cannot allocate more managed memory than the physical size of GPU memory.

+
+
+

+24.3.2.3. Multi-GPU +

+

On systems with devices of compute capabilities lower than 6.0 managed allocations are automatically visible to all GPUs in a system via the peer-to-peer capabilities of the GPUs. Managed memory allocations behave similar to unmanaged memory allocated using cudaMalloc(): the current active device is the home for the physical allocation but other GPUs in the system will access the memory at reduced bandwidth over the PCIe bus.

+

On Linux the managed memory is allocated in GPU memory as long as all GPUs that are actively being used by a program have the peer-to-peer support. If at any time the application starts using a GPU that doesn’t have peer-to-peer support with any of the other GPUs that have managed allocations on them, then the driver will migrate all managed allocations to system memory. In this case, all GPUs experience PCIe bandwidth restrictions.

+

On Windows, if peer mappings are not available (for example, between GPUs of different architectures), then the system will automatically fall back to using zero-copy memory, regardless of whether both GPUs are actually used by a program. If only one GPU is actually going to be used, it is necessary to set the CUDA_VISIBLE_DEVICES environment variable before launching the program. This constrains which GPUs are visible and allows managed memory to be allocated in GPU memory.

+

Alternatively, on Windows users can also set CUDA_MANAGED_FORCE_DEVICE_ALLOC to a non-zero value to force the driver to always use device memory for physical storage. When this environment variable is set to a non-zero value, all devices used in that process that support managed memory have to be peer-to-peer compatible with each other. The error ::cudaErrorInvalidDevice will be returned if a device that supports managed memory is used and it is not peer-to-peer compatible with any of the other managed memory supporting devices that were previously used in that process, even if ::cudaDeviceReset has been called on those devices. These environment variables are described in CUDA Environment Variables. Note that starting from CUDA 8.0 CUDA_MANAGED_FORCE_DEVICE_ALLOC has no effect on Linux operating systems.

+
+
+

+24.3.2.4. Coherency and Concurrency +

+

Simultaneous access to managed memory on devices of compute capability lower than 6.0 is not possible, because coherence could not be guaranteed if the CPU accessed a Unified Memory allocation while a GPU kernel was active.

+
+
+24.3.2.4.1. GPU Exclusive Access To Managed Memory +
+

To ensure coherency on pre-6.x GPU architectures, the Unified Memory programming model puts constraints on data accesses while both the CPU and GPU are executing concurrently. In effect, the GPU has exclusive access to all managed data while any kernel operation is executing, regardless of whether the specific kernel is actively using the data. When managed data is used with cudaMemcpy*() or cudaMemset*(), the system may choose to access the source or destination from the host or the device, which will put constraints on concurrent CPU access to that data while the cudaMemcpy*() or cudaMemset*() is executing. See Memcpy()/Memset() Behavior With Unified Memory for further details.

+

It is not permitted for the CPU to access any managed allocations or variables while the GPU is active for devices with concurrentManagedAccess property set to 0. On these systems concurrent CPU/GPU accesses, even to different managed memory allocations, will cause a segmentation fault because the page is considered inaccessible to the CPU.

+
+
__device__ __managed__ int x, y=2;
+__global__  void  kernel() {
+    x = 10;
+}
+int main() {
+    kernel<<< 1, 1 >>>();
+    y = 20;            // Error on GPUs not supporting concurrent access
+
+    cudaDeviceSynchronize();
+    return  0;
+}
+
+
+

In example above, the GPU program kernel is still active when the CPU touches y. (Note how it occurs before cudaDeviceSynchronize().) The code runs successfully on devices of compute capability 6.x due to the GPU page faulting capability which lifts all restrictions on simultaneous access. However, such memory access is invalid on pre-6.x architectures even though the CPU is accessing different data than the GPU. The program must explicitly synchronize with the GPU before accessing y:

+
+
__device__ __managed__ int x, y=2;
+__global__  void  kernel() {
+    x = 10;
+}
+int main() {
+    kernel<<< 1, 1 >>>();
+    cudaDeviceSynchronize();
+    y = 20;            //  Success on GPUs not supporing concurrent access
+    return  0;
+}
+
+
+

As this example shows, on systems with pre-6.x GPU architectures, a CPU thread may not access any managed data in between performing a kernel launch and a subsequent synchronization call, regardless of whether the GPU kernel actually touches that same data (or any managed data at all). The mere potential for concurrent CPU and GPU access is sufficient for a process-level exception to be raised.

+

Note that if memory is dynamically allocated with cudaMallocManaged() or cuMemAllocManaged() while the GPU is active, the behavior of the memory is unspecified until additional work is launched or the GPU is synchronized. Attempting to access the memory on the CPU during this time may or may not cause a segmentation fault. This does not apply to memory allocated using the flag cudaMemAttachHost or CU_MEM_ATTACH_HOST.

+
+
+
+24.3.2.4.2. Explicit Synchronization and Logical GPU Activity +
+

Note that explicit synchronization is required even if kernel runs quickly and finishes before the CPU touches y in the above example. Unified Memory uses logical activity to determine whether the GPU is idle. This aligns with the CUDA programming model, which specifies that a kernel can run at any time following a launch and is not guaranteed to have finished until the host issues a synchronization call.

+

Any function call that logically guarantees the GPU completes its work is valid. This includes cudaDeviceSynchronize(); cudaStreamSynchronize() and cudaStreamQuery() (provided it returns cudaSuccess and not cudaErrorNotReady) where the specified stream is the only stream still executing on the GPU; cudaEventSynchronize() and cudaEventQuery() in cases where the specified event is not followed by any device work; as well as uses of cudaMemcpy() and cudaMemset() that are documented as being fully synchronous with respect to the host.

+

Dependencies created between streams will be followed to infer completion of other streams by synchronizing on a stream or event. Dependencies can be created via cudaStreamWaitEvent() or implicitly when using the default (NULL) stream.

+

It is legal for the CPU to access managed data from within a stream callback, provided no other stream that could potentially be accessing managed data is active on the GPU. In addition, a callback that is not followed by any device work can be used for synchronization: for example, by signaling a condition variable from inside the callback; otherwise, CPU access is valid only for the duration of the callback(s).

+

There are several important points of note:

+
    +
  • It is always permitted for the CPU to access non-managed zero-copy data while the GPU is active.

  • +
  • The GPU is considered active when it is running any kernel, even if that kernel does not make use of managed data. If a kernel might use data, then access is forbidden, unless device property concurrentManagedAccess is 1.

  • +
  • There are no constraints on concurrent inter-GPU access of managed memory, other than those that apply to multi-GPU access of non-managed memory.

  • +
  • There are no constraints on concurrent GPU kernels accessing managed data.

  • +
+

Note how the last point allows for races between GPU kernels, as is currently the case for non-managed GPU memory. As mentioned previously, managed memory functions identically to non-managed memory from the perspective of the GPU. The following code example illustrates these points:

+
+
int main() {
+    cudaStream_t stream1, stream2;
+    cudaStreamCreate(&stream1);
+    cudaStreamCreate(&stream2);
+    int *non_managed, *managed, *also_managed;
+    cudaMallocHost(&non_managed, 4);    // Non-managed, CPU-accessible memory
+    cudaMallocManaged(&managed, 4);
+    cudaMallocManaged(&also_managed, 4);
+    // Point 1: CPU can access non-managed data.
+    kernel<<< 1, 1, 0, stream1 >>>(managed);
+    *non_managed = 1;
+    // Point 2: CPU cannot access any managed data while GPU is busy,
+    //          unless concurrentManagedAccess = 1
+    // Note we have not yet synchronized, so "kernel" is still active.
+    *also_managed = 2;      // Will issue segmentation fault
+    // Point 3: Concurrent GPU kernels can access the same data.
+    kernel<<< 1, 1, 0, stream2 >>>(managed);
+    // Point 4: Multi-GPU concurrent access is also permitted.
+    cudaSetDevice(1);
+    kernel<<< 1, 1 >>>(managed);
+    return  0;
+}
+
+
+
+
+
+24.3.2.4.3. Managing Data Visibility and Concurrent CPU + GPU Access with Streams +
+

Until now it was assumed that for SM architectures before 6.x: 1) any active kernel may use any managed memory, and 2) it was invalid to use managed memory from the CPU while a kernel is active. Here we present a system for finer-grained control of managed memory designed to work on all devices supporting managed memory, including older architectures with concurrentManagedAccess equal to 0.

+

The CUDA programming model provides streams as a mechanism for programs to indicate dependence and independence among kernel launches. Kernels launched into the same stream are guaranteed to execute consecutively, while kernels launched into different streams are permitted to execute concurrently. Streams describe independence between work items and hence allow potentially greater efficiency through concurrency.

+

Unified Memory builds upon the stream-independence model by allowing a CUDA program to explicitly associate managed allocations with a CUDA stream. In this way, the programmer indicates the use of data by kernels based on whether they are launched into a specified stream or not. This enables opportunities for concurrency based on program-specific data access patterns. The function to control this behavior is:

+
+
cudaError_t cudaStreamAttachMemAsync(cudaStream_t stream,
+                                     void *ptr,
+                                     size_t length=0,
+                                     unsigned int flags=0);
+
+
+

The cudaStreamAttachMemAsync() function associates length bytes of memory starting from ptr with the specified stream. (Currently, length must always be 0 to indicate that the entire region should be attached.) Because of this association, the Unified Memory system allows CPU access to this memory region so long as all operations in stream have completed, regardless of whether other streams are active. In effect, this constrains exclusive ownership of the managed memory region by an active GPU to per-stream activity instead of whole-GPU activity.

+

Most importantly, if an allocation is not associated with a specific stream, it is visible to all running kernels regardless of their stream. This is the default visibility for a cudaMallocManaged() allocation or a __managed__ variable; hence, the simple-case rule that the CPU may not touch the data while any kernel is running.

+

By associating an allocation with a specific stream, the program makes a guarantee that only kernels launched into that stream will touch that data. No error checking is performed by the Unified Memory system: it is the programmer’s responsibility to ensure that guarantee is honored.

+

In addition to allowing greater concurrency, the use of cudaStreamAttachMemAsync() can (and typically does) enable data transfer optimizations within the Unified Memory system that may affect latencies and other overhead.

+
+
+
+24.3.2.4.4. Stream Association Examples +
+

Associating data with a stream allows fine-grained control over CPU + GPU concurrency, but what data is visible to which streams must be kept in mind when using devices of compute capability lower than 6.0. Looking at the earlier synchronization example:

+
+
__device__ __managed__ int x, y=2;
+__global__  void  kernel() {
+    x = 10;
+}
+int main() {
+    cudaStream_t stream1;
+    cudaStreamCreate(&stream1);
+    cudaStreamAttachMemAsync(stream1, &y, 0, cudaMemAttachHost);
+    cudaDeviceSynchronize();          // Wait for Host attachment to occur.
+    kernel<<< 1, 1, 0, stream1 >>>(); // Note: Launches into stream1.
+    y = 20;                           // Success – a kernel is running but “y”
+                                      // has been associated with no stream.
+    return  0;
+}
+
+
+

Here we explicitly associate y with host accessibility, thus enabling access at all times from the CPU. (As before, note the absence of cudaDeviceSynchronize() before the access.) Accesses to y by the GPU running kernel will now produce undefined results.

+

Note that associating a variable with a stream does not change the associating of any other variable. For example, associating x with stream1 does not ensure that only x is accessed by kernels launched in stream1, thus an error is caused by this code:

+
+
__device__ __managed__ int x, y=2;
+__global__  void  kernel() {
+    x = 10;
+}
+int main() {
+    cudaStream_t stream1;
+    cudaStreamCreate(&stream1);
+    cudaStreamAttachMemAsync(stream1, &x);// Associate “x” with stream1.
+    cudaDeviceSynchronize();              // Wait for “x” attachment to occur.
+    kernel<<< 1, 1, 0, stream1 >>>();     // Note: Launches into stream1.
+    y = 20;                               // ERROR: “y” is still associated globally
+                                          // with all streams by default
+    return  0;
+}
+
+
+

Note how the access to y will cause an error because, even though x has been associated with a stream, we have told the system nothing about who can see y. The system therefore conservatively assumes that kernel might access it and prevents the CPU from doing so.

+
+
+
+24.3.2.4.5. Stream Attach With Multithreaded Host Programs +
+

The primary use for cudaStreamAttachMemAsync() is to enable independent task parallelism using CPU threads. Typically in such a program, a CPU thread creates its own stream for all work that it generates because using CUDA’s NULL stream would cause dependencies between threads.

+

The default global visibility of managed data to any GPU stream can make it difficult to avoid interactions between CPU threads in a multi-threaded program. Function cudaStreamAttachMemAsync() is therefore used to associate a thread’s managed allocations with that thread’s own stream, and the association is typically not changed for the life of the thread.

+

Such a program would simply add a single call to cudaStreamAttachMemAsync() to use unified memory for its data accesses:

+
+
// This function performs some task, in its own private stream.
+void run_task(int *in, int *out, int length) {
+    // Create a stream for us to use.
+    cudaStream_t stream;
+    cudaStreamCreate(&stream);
+    // Allocate some managed data and associate with our stream.
+    // Note the use of the host-attach flag to cudaMallocManaged();
+    // we then associate the allocation with our stream so that
+    // our GPU kernel launches can access it.
+    int *data;
+    cudaMallocManaged((void **)&data, length, cudaMemAttachHost);
+    cudaStreamAttachMemAsync(stream, data);
+    cudaStreamSynchronize(stream);
+    // Iterate on the data in some way, using both Host & Device.
+    for(int i=0; i<N; i++) {
+        transform<<< 100, 256, 0, stream >>>(in, data, length);
+        cudaStreamSynchronize(stream);
+        host_process(data, length);    // CPU uses managed data.
+        convert<<< 100, 256, 0, stream >>>(out, data, length);
+    }
+    cudaStreamSynchronize(stream);
+    cudaStreamDestroy(stream);
+    cudaFree(data);
+}
+
+
+

In this example, the allocation-stream association is established just once, and then data is used repeatedly by both the host and device. The result is much simpler code than occurs with explicitly copying data between host and device, although the result is the same.

+
+
+
+24.3.2.4.6. Advanced Topic: Modular Programs and Data Access Constraints +
+

In the previous example cudaMallocManaged() specifies the cudaMemAttachHost flag, which creates an allocation that is initially invisible to device-side execution. (The default allocation would be visible to all GPU kernels on all streams.) This ensures that there is no accidental interaction with another thread’s execution in the interval between the data allocation and when the data is acquired for a specific stream.

+

Without this flag, a new allocation would be considered in-use on the GPU if a kernel launched by another thread happens to be running. This might impact the thread’s ability to access the newly allocated data from the CPU (for example, within a base-class constructor) before it is able to explicitly attach it to a private stream. To enable safe independence between threads, therefore, allocations should be made specifying this flag.

+
+

Note

+

An alternative would be to place a process-wide barrier across all threads after the allocation has been attached to the stream. This would ensure that all threads complete their data/stream associations before any kernels are launched, avoiding the hazard. A second barrier would be needed before the stream is destroyed because stream destruction causes allocations to revert to their default visibility. The cudaMemAttachHost flag exists both to simplify this process, and because it is not always possible to insert global barriers where required.

+
+
+
+
+24.3.2.4.7. Memcpy()/Memset() Behavior With Stream-associated Unified Memory +
+

See Memcpy()/Memset() Behavior With Unified Memory for a general overview of cudaMemcpy* / cudaMemset* behavior on devices with concurrentManagedAccess set. On devices where concurrentManagedAccess is not set, the following rules apply:

+

If cudaMemcpyHostTo* is specified and the source data is unified memory, then it will be accessed from the host if it is coherently accessible from the host in the copy stream (1); otherwise it will be accessed from the device. Similar rules apply to the destination when cudaMemcpy*ToHost is specified and the destination is unified memory.

+

If cudaMemcpyDeviceTo* is specified and the source data is unified memory, then it will be accessed from the device. The source must be coherently accessible from the device in the copy stream (2); otherwise, an error is returned. Similar rules apply to the destination when cudaMemcpy*ToDevice is specified and the destination is unified memory.

+

If cudaMemcpyDefault is specified, then unified memory will be accessed from the host either if it cannot be coherently accessed from the device in the copy stream (2) or if the preferred location for the data is cudaCpuDeviceId and it can be coherently accessed from the host in the copy stream (1); otherwise, it will be accessed from the device.

+

When using cudaMemset*() with unified memory, the data must be coherently accessible from the device in the stream being used for the cudaMemset() operation (2); otherwise, an error is returned.

+

When data is accessed from the device either by cudaMemcpy* or cudaMemset*, the stream of operation is considered to be active on the GPU. During this time, any CPU access of data that is associated with that stream or data that has global visibility, will result in a segmentation fault if the GPU has a zero value for the device attribute concurrentManagedAccess. The program must synchronize appropriately to ensure the operation has completed before accessing any associated data from the CPU.

+
+
+
    +
  1. Coherently accessible from the host in a given stream means that the memory neither has global visibility nor is it associated with the given stream.

  2. +
+
+
+
+
+
    +
  1. Coherently accessible from the device in a given stream means that the memory either has global visibility or is associated with the given stream.

  2. +
+
+
+
+
+
+
+
+
+

+25. Lazy Loading +

+
+

Warning

+

This document has been replaced by a new CUDA Programming Guide. The information in this document should be considered legacy, and this document is no longer being updated as of CUDA 13.0. Please refer to the CUDA Programming Guide for up-to-date information on CUDA.

+
+
+

+25.1. What is Lazy Loading? +

+

Lazy Loading delays loading of CUDA modules and kernels from program initialization closer to kernels execution. +If a program does not use every single kernel it has included, then some kernels will be loaded unneccessarily. +This is very common, especially if you include any libraries. +Most of the time, programs only use a small amount of kernels from libraries they include.

+

Thanks to Lazy Loading, programs are able to only load kernels they are actually going to use, saving time on initialization. +This reduces memory overhead, both on GPU memory and host memory.

+

Lazy Loading is enabled by setting the CUDA_MODULE_LOADING environment variable to LAZY.

+

Firstly, CUDA Runtime will no longer load all modules during program initialization, with the exception of modules containing managed variables. +Each module will be loaded on first usage of a variable or a kernel from that module. +This optimization is only relevant to CUDA Runtime users, CUDA Driver users who use cuModuleLoad are unaffected. This optimization shipped in CUDA 11.8. +The behavior for CUDA Driver users who use cuLibraryLoad to load module data into memory can be changed by +setting the CUDA_MODULE_DATA_LOADING environment variable.

+

Secondly, loading a module (cuModuleLoad*() family of functions) will not be loading kernels immediately, +instead it will delay loading of a kernel until cuModuleGetFunction() is called. +There are certain exceptions here, some kernels have to be loaded during cuModuleLoad*(), +such as kernels of which pointers are stored in global variables. +This optimization is relevant to both CUDA Runtime and CUDA Driver users. +CUDA Runtime will only call cuModuleGetFunction() when a kernel is used/referenced for the first time. +This optimization shipped in CUDA 11.7.

+

Both of these optimizations are designed to be invisible to the user, assuming CUDA Programming Model is followed.

+
+
+

+25.2. Lazy Loading version support +

+

Lazy Loading is a CUDA Runtime and CUDA Driver feature. Upgrades to both might be necessary to utilize the feature.

+
+

+25.2.1. Driver +

+

Lazy Loading requires R515+ user-mode library, but it supports Forward Compatibility, meaning it can run on top of older kernel mode drivers.

+

Without R515+ user-mode library, Lazy Loading is not available in any shape or form, even if toolkit version is 11.7+.

+
+
+

+25.2.2. Toolkit +

+

Lazy Loading was introduced in CUDA 11.7, and received a significant upgrade in CUDA 11.8.

+

If your application uses CUDA Runtime, then in order to see benefits from Lazy Loading your application must use 11.7+ CUDA Runtime.

+

As CUDA Runtime is usually linked statically into programs and libraries, +this means that you have to recompile your program with CUDA 11.7+ toolkit and use CUDA 11.7+ libraries.

+

Otherwise you will not see the benefits of Lazy Loading, even if your driver version supports it.

+

If only some of your libraries are 11.7+, you will only see benefits of Lazy Loading in those libraries. +Other libraries will still load everything eagerly.

+
+
+

+25.2.3. Compiler +

+

Lazy Loading does not require any compiler support. Both SASS and PTX compiled with pre-11.7 compilers can be loaded with Lazy Loading enabled, +and will see full benefits of the feature. However, 11.7+ CUDA Runtime is still required, as described above.

+
+
+
+

+25.3. Triggering loading of kernels in lazy mode +

+

Loading kernels and variables happens automatically, without any need for explicit loading. +Simply launching a kernel or referencing a variable or a kernel will automatically load relevant modules and kernels.

+

However, if for any reason you wish to load a kernel without executing it or modifying it in any way, we recommend the following.

+
+

+25.3.1. CUDA Driver API +

+

Loading of kernels happens during cuModuleGetFunction() call. +This call is necessary even without Lazy Loading, as it is the only way to obtain a kernel handle.

+

However, you can also use this API to control with finer granularity when kernels are loaded.

+
+
+

+25.3.2. CUDA Runtime API +

+

CUDA Runtime API manages module management automatically, +so we recommend simply using cudaFuncGetAttributes() to reference the kernel.

+

This will ensure that the kernel is loaded without changing the state.

+
+
+
+

+25.4. Querying whether Lazy Loading is Turned On +

+

In order to check whether user enabled Lazy Loading, CUresult cuModuleGetLoadingMode ( CUmoduleLoadingMode* mode ) can be used.

+

It’s important to note that CUDA must be initialized before running this function. Sample usage can be seen in the snippet below.

+
+
#include "cuda.h"
+#include "assert.h"
+#include "iostream"
+
+int main() {
+        CUmoduleLoadingMode mode;
+
+        assert(CUDA_SUCCESS == cuInit(0));
+        assert(CUDA_SUCCESS == cuModuleGetLoadingMode(&mode));
+
+        std::cout << "CUDA Module Loading Mode is " << ((mode == CU_MODULE_LAZY_LOADING) ? "lazy" : "eager") << std::endl;
+
+        return 0;
+}
+
+
+
+
+

+25.5. Possible Issues when Adopting Lazy Loading +

+

Lazy Loading is designed so that it should not require any modifications to applications to use it. +That said, there are some caveats, especially when applications are not fully compliant with CUDA Programming Model.

+
+

+25.5.1. Concurrent Execution +

+

Loading kernels might require context synchronization. +Some programs incorrectly treat the possibility of concurrent execution of kernels as a guarantee. +In such cases, if program assumes that two kernels will be able to execute concurrently, +and one of the kernels will not return without the other kernel executing, there is a possibility of a deadlock.

+

If kernel A will be spinning in an infinite loop until kernel B is executing. +In such case launching kernel B will trigger lazy loading of kernel B. If this loading will require context synchronization, +then we have a deadlock: kernel A is waiting for kernel B, but loading kernel B is stuck waiting for kernel A to finish to synchronize the context.

+

Such program is an anti-pattern, but if for any reason you want to keep it you can do the following:

+
    +
  • preload all kernels that you hope to execute concurrently prior to launching them

  • +
  • run application with CUDA_MODULE_DATA_LOADING=EAGER to force loading data eagerly without forcing each function to load eagerly

  • +
+
+
+

+25.5.2. Allocators +

+

Lazy Loading delays loading code from initialization phase of the program closer to execution phase. +Loading code onto the GPU requires memory allocation.

+

If your application tries to allocate the entire VRAM on startup, for example, to use it for its own allocator, +then it might turn out that there will be no more memory left to load the kernels. +This is despite the fact that overall Lazy Loading frees up more memory for the user. +CUDA will need to allocate some memory to load each kernel, which usually happens at first launch time of each kernel. +If your application allocator greedily allocated everything, CUDA will fail to allocate memory.

+

Possible solutions:

+
    +
  • use cudaMallocAsync() instead of an allocator that allocates the entire VRAM on startup

  • +
  • add some buffer to compensate for the delayed loading of kernels

  • +
  • preload all kernels that will be used in the program before trying to initialize your allocator

  • +
+
+
+

+25.5.3. Autotuning +

+

Some applications launch several kernels implementing the same functionality to determine which one is the fastest. +While it is overall advisable to run at least one warmup iteration, it becomes especially important with Lazy Loading. +After all, including time taken to load the kernel will skew your results.

+

Possible solutions:

+
    +
  • do at least one warmup interaction prior to measurement

  • +
  • preload the benchmarked kernel prior to launching it

  • +
+
+
+
+
+

+26. Extended GPU Memory +

+
+

Warning

+

This document has been replaced by a new CUDA Programming Guide. The information in this document should be considered legacy, and this document is no longer being updated as of CUDA 13.0. Please refer to the CUDA Programming Guide for up-to-date information on CUDA.

+
+

The Extended GPU Memory (EGM) feature, utilizing the high-bandwidth +NVLink-C2C, facilitates efficient access to all system memory by GPUs, +in a single-node system. +EGM applies to integrated CPU-GPU NVIDIA systems by allowing physical memory +allocation that can be accessed from any GPU +thread within the setup. EGM ensures that all GPUs can access +its resources at the speed of either GPU-GPU NVLink or NVLink-C2C.

+
+EGM +
+

In this setup, memory accesses occur via the local high-bandwidth +NVLink-C2C. For remote memory accesses, +GPU NVLink and, in some cases, NVLink-C2C are used. With EGM, GPU +threads gain the capability to access all available memory resources, +including CPU attached memory and HBM3, over the NVSwitch fabric.

+
+

+26.1. Preliminaries +

+

Before diving into API changes for EGM functionalities, we are going to +cover currently supported topologies, identifier assignment, +prerequisites for virtual memory management, and CUDA types for EGM.

+
+

+26.1.1. EGM Platforms: System topology +

+

Currently, EGM can be enabled in three platforms: (1) Single-Node, Single-GPU: +Consists of an Arm-based CPU, CPU attached memory, and a GPU. Between the CPU +and the GPU there is a high bandwidth C2C (Chip-to-Chip) interconnect. +(2) Single-Node, Multi-GPU: Consists of fully connected four +single-node, single-GPU platforms. (3) Multi-Node, Single-GPU: +Two or more single-node multi-socket systems.

+
+

Note

+

Using cgroups to limit available devices will block routing over EGM +and cause performance issues. Use CUDA_VISIBLE_DEVICES instead.

+
+
+
+

+26.1.2. Socket Identifiers: What are they? How to access them? +

+

NUMA (Non-Uniform Memory Access) is a memory architecture used in +multi-processor computer systems such that the memory is divided into +multiple nodes. Each node has its own processors and memory. In such a +system, NUMA divides the system into nodes and assigns a unique +identifier (numaID) to every node.

+

EGM uses the NUMA node identifier which is assigned by the operating +system. Note that, this identifier is different from the ordinal of a +device and it is associated with the closest host node. In addition to +the existing methods, the user can obtain the identifier of the host +node (numaID) by calling cuDeviceGetAttribute +with CU_DEVICE_ATTRIBUTE_HOST_NUMA_ID attribute type as follows:

+
+
int numaId;
+cuDeviceGetAttribute(&numaId, CU_DEVICE_ATTRIBUTE_HOST_NUMA_ID, deviceOrdinal);
+
+
+
+
+

+26.1.3. Allocators and EGM support +

+

Mapping system memory as EGM does not cause any performance issues. In +fact, accessing a remote socket’s system memory mapped as EGM is going +to be faster. Because, with EGM traffic is guaranteed to be routed over +NVLinks. Currently, cuMemCreate and cudaMemPoolCreate allocators are +supported with appropriate location type and NUMA identifiers.

+
+
+

+26.1.4. Memory management extensions to current APIs +

+

Currently, EGM memory can be mapped with Virtual Memory (cuMemCreate)  or +Stream Ordered Memory (cudaMemPoolCreate) allocators. The user is +responsible for allocating physical memory and mapping it to a virtual +memory address space on all sockets.

+
+

Note

+

Multi-node, single-GPU platforms require interprocess +communication. Therefore we encourage the reader to see Chapter 3

+
+
+

Note

+

We encourage readers to read CUDA Programming Guide’s Chapter 10 and Chapter 11 for a better understanding.

+
+

New CUDA property types have been added to APIs for allowing those +approaches to understand allocation locations using NUMA-like node +identifiers:

+ ++++ + + + + + + + + + + + + + + +

CUDA Type

Used with

CU_MEM_LOCATION_TYPE_HOST_NUMA

CUmemAllocationProp for +cuMemCreate

cudaMemLocationTypeHostNuma

cudaMemPoolProps for +cudaMemPoolCreate

+
+

Note

+

Please see CUDA Driver API +and CUDA Runtime Data Types +to find more about NUMA specific CUDA types.

+
+
+
+
+

+26.2. Using the EGM Interface +

+
+

+26.2.1. Single-Node, Single-GPU +

+

Any of the existing CUDA host allocators as well as system allocated +memory can be used to benefit from high-bandwidth C2C. To the user, +local access is what a host allocation is today.

+
+

Note

+

Refer to the tuning guide for more information about memory allocators and page sizes.

+
+
+
+

+26.2.2. Single-Node, Multi-GPU +

+

In a multi-GPU system, the user has to provide host information for +the placement. As we mentioned, a natural way to express that +information would be by using NUMA node IDs and EGM follows this +approach. Therefore, using the cuDeviceGetAttribute function the +user should be able to learn the closest NUMA node id. (See Socket Identifiers: What are they? How to access them?). +Then the user can allocate and manage EGM memory using VMM (Virtual +Memory Management) API or CUDA Memory Pool.

+
+

+26.2.2.1. Using VMM APIs +

+

The first step in memory allocation using Virtual Memory Management APIs +is to create a physical memory chunk that will provide a backing for the +allocation. See CUDA Programming Guide’s Virtual Memory Management section +for more details. In EGM allocations the user has to explicitly provide +CU_MEM_LOCATION_TYPE_HOST_NUMA  as the location type and +numaID as the location identifier. Also in EGM, allocations must +be aligned to appropriate granularity of the platform. The following +code snippet shows allocating physical memory with cuMemCreate:

+
+
CUmemAllocationProp prop{};
+prop.type = CU_MEM_ALLOCATION_TYPE_PINNED;
+prop.location.type = CU_MEM_LOCATION_TYPE_HOST_NUMA;
+prop.location.id = numaId;
+size_t granularity = 0;
+cuMemGetAllocationGranularity(&granularity, &prop, MEM_ALLOC_GRANULARITY_MINIMUM);
+size_t padded_size = ROUND_UP(size, granularity);
+CUmemGenericAllocationHandle allocHandle;
+cuMemCreate(&allocHandle, padded_size, &prop, 0);
+
+
+

After physical memory allocation, we have to reserve an address space +and map it to a pointer. These procedures do not have EGM-specific +changes:

+
+
CUdeviceptr dptr;
+cuMemAddressReserve(&dptr, padded_size, 0, 0, 0);
+cuMemMap(dptr, padded_size, 0, allocHandle, 0);
+
+
+

Finally, the user has to explicitly protect mapped virtual address +ranges. Otherwise access to the mapped space would result in a crash. +Similar to the memory allocation, the user has to provide +CU_MEM_LOCATION_TYPE_HOST_NUMA as the location type and +numaId as the location identifier. Following code snippet create +an access descriptors for the host node and the GPU to give read and +write access for the mapped memory to both of them:

+
+
CUmemAccessDesc accessDesc[2]{{}};
+accessDesc[0].location.type = CU_MEM_LOCATION_TYPE_HOST_NUMA;
+accessDesc[0].location.id = numaId;
+accessDesc[0].flags = CU_MEM_ACCESS_FLAGS_PROT_READWRITE;
+accessDesc[1].location.type = CU_MEM_LOCATION_TYPE_DEVICE;
+accessDesc[1].location.id = currentDev;
+accessDesc[1].flags = CU_MEM_ACCESS_FLAGS_PROT_READWRITE;
+cuMemSetAccess(dptr, size, accessDesc, 2);
+
+
+
+
+

+26.2.2.2. Using CUDA Memory Pool +

+

To define EGM, the user can create a memory pool on a node and give +access to peers. In this case, the user has to explicitly define +cudaMemLocationTypeHostNuma as the location type and numaId +as the location identifier. The following code snippet shows creating a +memory pool cudaMemPoolCreate:

+
+
cudaSetDevice(homeDevice);
+cudaMemPoolProps props{};
+props.allocType = cudaMemAllocationTypePinned;
+props.location.type = cudaMemLocationTypeHostNuma;
+props.location.id = numaId;
+cudaMemPoolCreate(&memPool, &props);
+
+
+

Additionally, for direct connect peer access, it is also possible to use +the existing peer access API, cudaMemPoolSetAccess. An example +for an accessingDevice is shown in the following code snippet:

+
+
cudaMemAccessDesc desc{};
+desc.flags = cudaMemAccessFlagsProtReadWrite;
+desc.location.type = cudaMemLocationTypeDevice;
+desc.location.id = accessingDevice;
+cudaMemPoolSetAccess(memPool, &desc, 1);
+
+
+

When the memory pool is created, and accesses are given, the user can +set created memory pool to the residentDevice and start allocating +memory using cudaMallocAsync:

+
+
cudaDeviceSetMemPool(residentDevice, memPool);
+cudaMallocAsync(&ptr, size, memPool, stream);
+
+
+
+

Note

+

EGM is mapped with 2MB pages. Therefore, users may encounter more TLB +misses when accessing very large allocations.

+
+
+
+
+

+26.2.3. Multi-Node, Single-GPU +

+

Beyond memory allocation, remote peer access does not have EGM-specific +modification and it follows CUDA inter process (IPC) protocol. See +CUDA Programming Guide +for more details in IPC.

+

The user should allocate memory using cuMemCreate and again the +user has to explicitly provide CU_MEM_LOCATION_TYPE_HOST_NUMA as +the location type and numaID as the location identifier. In +addition CU_MEM_HANDLE_TYPE_FABRIC should be defined as the +requested handle type. The following code snippet shows allocating +physical memory on Node A:

+
+
CUmemAllocationProp prop{};
+prop.type = CU_MEM_ALLOCATION_TYPE_PINNED;
+prop.requestedHandleTypes = CU_MEM_HANDLE_TYPE_FABRIC;
+prop.location.type = CU_MEM_LOCATION_TYPE_HOST_NUMA;
+prop.location.id = numaId;
+size_t granularity = 0;
+cuMemGetAllocationGranularity(&granularity, &prop,
+                              MEM_ALLOC_GRANULARITY_MINIMUM);
+size_t padded_size = ROUND_UP(size, granularity);
+size_t page_size = ...;
+assert(padded_size % page_size == 0);
+CUmemGenericAllocationHandle allocHandle;
+cuMemCreate(&allocHandle, padded_size, &prop, 0);
+
+
+

After creating allocation handle using cuMemCreate the user can +export that handle to the other node, Node B, calling +cuMemExportToShareableHandle:

+
+
cuMemExportToShareableHandle(&fabricHandle, allocHandle,
+                             CU_MEM_HANDLE_TYPE_FABRIC, 0);
+// At this point, fabricHandle should be sent to Node B via TCP/IP.
+
+
+

On Node B, the handle can be imported using +cuMemImportFromShareableHandle and treated as any other fabric +handle

+
+
// At this point, fabricHandle should be received from Node A via TCP/IP.
+CUmemGenericAllocationHandle allocHandle;
+cuMemImportFromShareableHandle(&allocHandle, &fabricHandle,
+                               CU_MEM_HANDLE_TYPE_FABRIC);
+
+
+

When handle is imported at Node B, then the user can reserve an address +space and map it locally in a regular fashion:

+
+
size_t granularity = 0;
+cuMemGetAllocationGranularity(&granularity, &prop,
+                              MEM_ALLOC_GRANULARITY_MINIMUM);
+size_t padded_size = ROUND_UP(size, granularity);
+size_t page_size = ...;
+assert(padded_size % page_size == 0);
+CUdeviceptr dptr;
+cuMemAddressReserve(&dptr, padded_size, 0, 0, 0);
+cuMemMap(dptr, padded_size, 0, allocHandle, 0);
+
+
+

As the final step, the user should give appropriate accesses to each of +the local GPUs at Node B. An example code snippet that gives read and +write access to eight local GPUs:

+
+
// Give all 8 local  GPUS access to exported EGM memory located on Node A.                                                               |
+CUmemAccessDesc accessDesc[8];
+for (int i = 0; i < 8; i++) {
+   accessDesc[i].location.type = CU_MEM_LOCATION_TYPE_DEVICE;
+   accessDesc[i].location.id = i;
+   accessDesc[i].flags = CU_MEM_ACCESS_FLAGS_PROT_READWRITE;
+}
+cuMemSetAccess(dptr, size, accessDesc, 8);
+
+
+
+
+
+
+

+27. Notices +

+
+

+27.1. Notice +

+

This document is provided for information purposes only and shall not be regarded as a warranty of a certain functionality, condition, or quality of a product. NVIDIA Corporation (“NVIDIA”) makes no representations or warranties, expressed or implied, as to the accuracy or completeness of the information contained in this document and assumes no responsibility for any errors contained herein. NVIDIA shall have no liability for the consequences or use of such information or for any infringement of patents or other rights of third parties that may result from its use. This document is not a commitment to develop, release, or deliver any Material (defined below), code, or functionality.

+

NVIDIA reserves the right to make corrections, modifications, enhancements, improvements, and any other changes to this document, at any time without notice.

+

Customer should obtain the latest relevant information before placing orders and should verify that such information is current and complete.

+

NVIDIA products are sold subject to the NVIDIA standard terms and conditions of sale supplied at the time of order acknowledgement, unless otherwise agreed in an individual sales agreement signed by authorized representatives of NVIDIA and customer (“Terms of Sale”). NVIDIA hereby expressly objects to applying any customer general terms and conditions with regards to the purchase of the NVIDIA product referenced in this document. No contractual obligations are formed either directly or indirectly by this document.

+

NVIDIA products are not designed, authorized, or warranted to be suitable for use in medical, military, aircraft, space, or life support equipment, nor in applications where failure or malfunction of the NVIDIA product can reasonably be expected to result in personal injury, death, or property or environmental damage. NVIDIA accepts no liability for inclusion and/or use of NVIDIA products in such equipment or applications and therefore such inclusion and/or use is at customer’s own risk.

+

NVIDIA makes no representation or warranty that products based on this document will be suitable for any specified use. Testing of all parameters of each product is not necessarily performed by NVIDIA. It is customer’s sole responsibility to evaluate and determine the applicability of any information contained in this document, ensure the product is suitable and fit for the application planned by customer, and perform the necessary testing for the application in order to avoid a default of the application or the product. Weaknesses in customer’s product designs may affect the quality and reliability of the NVIDIA product and may result in additional or different conditions and/or requirements beyond those contained in this document. NVIDIA accepts no liability related to any default, damage, costs, or problem which may be based on or attributable to: (i) the use of the NVIDIA product in any manner that is contrary to this document or (ii) customer product designs.

+

No license, either expressed or implied, is granted under any NVIDIA patent right, copyright, or other NVIDIA intellectual property right under this document. Information published by NVIDIA regarding third-party products or services does not constitute a license from NVIDIA to use such products or services or a warranty or endorsement thereof. Use of such information may require a license from a third party under the patents or other intellectual property rights of the third party, or a license from NVIDIA under the patents or other intellectual property rights of NVIDIA.

+

Reproduction of information in this document is permissible only if approved in advance by NVIDIA in writing, reproduced without alteration and in full compliance with all applicable export laws and regulations, and accompanied by all associated conditions, limitations, and notices.

+

THIS DOCUMENT AND ALL NVIDIA DESIGN SPECIFICATIONS, REFERENCE BOARDS, FILES, DRAWINGS, DIAGNOSTICS, LISTS, AND OTHER DOCUMENTS (TOGETHER AND SEPARATELY, “MATERIALS”) ARE BEING PROVIDED “AS IS.” NVIDIA MAKES NO WARRANTIES, EXPRESSED, IMPLIED, STATUTORY, OR OTHERWISE WITH RESPECT TO THE MATERIALS, AND EXPRESSLY DISCLAIMS ALL IMPLIED WARRANTIES OF NONINFRINGEMENT, MERCHANTABILITY, AND FITNESS FOR A PARTICULAR PURPOSE. TO THE EXTENT NOT PROHIBITED BY LAW, IN NO EVENT WILL NVIDIA BE LIABLE FOR ANY DAMAGES, INCLUDING WITHOUT LIMITATION ANY DIRECT, INDIRECT, SPECIAL, INCIDENTAL, PUNITIVE, OR CONSEQUENTIAL DAMAGES, HOWEVER CAUSED AND REGARDLESS OF THE THEORY OF LIABILITY, ARISING OUT OF ANY USE OF THIS DOCUMENT, EVEN IF NVIDIA HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. Notwithstanding any damages that customer might incur for any reason whatsoever, NVIDIA’s aggregate and cumulative liability towards customer for the products described herein shall be limited in accordance with the Terms of Sale for the product.

+
+
+

+27.2. OpenCL +

+

OpenCL is a trademark of Apple Inc. used under license to the Khronos Group Inc.

+
+
+

+27.3. Trademarks +

+

NVIDIA and the NVIDIA logo are trademarks or registered trademarks of NVIDIA Corporation in the U.S. and other countries. Other company and product names may be trademarks of the respective companies with which they are associated.

+
+
+ + +
+
+ +
+
+
+
+ + + + + + + diff --git a/research/gpu-fusion-splatting/mirrors/nvidia-cuda-programming-guide.pdf b/research/gpu-fusion-splatting/mirrors/nvidia-cuda-programming-guide.pdf new file mode 100644 index 0000000..e22b2b7 Binary files /dev/null and b/research/gpu-fusion-splatting/mirrors/nvidia-cuda-programming-guide.pdf differ diff --git a/research/gpu-fusion-splatting/mirrors/nvidia-optical-flow-programming-guide.pdf b/research/gpu-fusion-splatting/mirrors/nvidia-optical-flow-programming-guide.pdf new file mode 100644 index 0000000..fb3f34b Binary files /dev/null and b/research/gpu-fusion-splatting/mirrors/nvidia-optical-flow-programming-guide.pdf differ diff --git a/research/gpu-fusion-splatting/mirrors/nvidia-vk-gaussian-splatting-blog.html b/research/gpu-fusion-splatting/mirrors/nvidia-vk-gaussian-splatting-blog.html new file mode 100644 index 0000000..f382540 --- /dev/null +++ b/research/gpu-fusion-splatting/mirrors/nvidia-vk-gaussian-splatting-blog.html @@ -0,0 +1,1157 @@ + + + + + + + + + Real-Time GPU-Accelerated Gaussian Splatting with NVIDIA DesignWorks Sample vk_gaussian_splatting | NVIDIA Technical Blog + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+
+
+
+
+ + +
+
+
+
+ +
+
+
+ +
+
+
+
+
+ + +
+
+ +
+
+
+ + Content Creation / Rendering + + + + + +
+
+
+ +

Real-Time GPU-Accelerated Gaussian Splatting with NVIDIA DesignWorks Sample vk_gaussian_splatting

+ + + + + +
+
+
+ + + + + +
+

Gaussian splatting is a novel approach to rendering complex 3D scenes by representing them as a collection of anisotropic Gaussians in 3D space. This technique enables real-time rendering of photorealistic scenes learned from small sets of images, making it ideal for applications in gaming, virtual reality, and real-time professional visualization.

+ + + +

vk_gaussian_splatting is a new Vulkan-based sample that demonstrates real-time Gaussian splatting, a cutting-edge volume rendering technique that enables highly efficient representations of radiance fields. It is the latest addition to the NVIDIA DesignWorks Samples.

+ + + +

The NVIDIA DevTech team envisions this new sample project as a testbed to explore and compare various approaches to real-time visualization of 3D Gaussian splatting (3DGS). By evaluating various techniques and optimizations, the team aims to provide valuable insights into performance, quality, and implementation trade-offs when using the Vulkan API. 

+ + + +

The initial implementation is based on rasterization and demonstrates two approaches for rendering splats, one leveraging mesh shaders and another using vertex shaders. 

+ + +
+ +
+ + +

Because Gaussian splats require back-to-front sorting for correct alpha compositing, two alternative sorting methods are provided: 

+ + + +
    +
  • A GPU-based Radix Sort implemented in a compute pipeline
  • + + + +
  • A CPU-based asynchronous sorting strategy that uses the multithreaded sort function from the C++ STL
  • +
+ + +
+ +
+ + +

The sample allows you to explore and experiment with multiple aspects of this rendering technique, including:

+ + + +
    +
  • Several visualization modes to inspect the different aspects of Gaussian splats (spherical harmonics, splats, point density, and more)
  • + + + +
  • A complete benchmarking system is available and enables profiling in real time
  • + + + +
  • More details about both the RAM and VRAM memory consumption, to understand the stream of data to render
  • + + + +
  • GPU timings for each stage of the different techniques investigated, to have an understanding of the workload and potential bottlenecks
  • + + + +
  • Graphical reports made generated with all of these numbers
  • +
+ + +
+ +
+ + +

This sample provides a starting point for developers looking to experiment with Gaussian splatting rendering techniques and Vulkan-based optimizations.

+ + + +

To start exploring real-time rendering of neural radiance fields, check out the nvpro-samples/vk_gaussian_splatting GitHub repo. 

+
+ +
+
+ + Discuss (0) + +
+ +
+
+ + +
+
+ +
+
+

Tags

+
+ +
+ + + +
+
+
+ +
+
+
+
+
+
+
+
+
+
+ + + +
+
+
+ +
+ + + + + + + + +
+
+
+
+
+
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+
+ + diff --git a/research/gpu-fusion-splatting/mirrors/nvidia-vpi-main.html b/research/gpu-fusion-splatting/mirrors/nvidia-vpi-main.html new file mode 100644 index 0000000..7306368 --- /dev/null +++ b/research/gpu-fusion-splatting/mirrors/nvidia-vpi-main.html @@ -0,0 +1,228 @@ + + + + + + + + + + + + + +VPI - Vision Programming Interface: Main Page + + + + + + + + + + + + + + + + + + + + +
+
+ + + + + + +
+ + +

VPI - Vision Programming Interface

+

4.0 Release

+
+
+ + + + + + + +
+
+ +
+
+
+ +
+ +
+
+ + +
+ +
+ +
+
+
VPI - Vision Programming Interface Documentation
+
+
+

NVIDIA® Vision Programming Interface (VPI) is a software library that implements computer vision (CV) and image processing (IP) algorithms on several computing hardware platforms available in NVIDIA embedded and discrete devices. Previous releases are available: 3.2 Release, 3.1 Release, or the 3.0 Release.

+

VPI provides seamless access to computing hardware that must be accessed through different and sometimes incompatible APIs, such as OpenCV and NVIDIA® CUDA® SDK, or for which public APIs do not exist, such as PVA (Programmable Vision Accelerator), VIC (Video and Image Compositor) and OFA (Optical Flow Accelerator).

+

The API is designed to be easy to use, without sacrificing performance. This allows for rapid prototyping and fine tuning, significantly reducing time-to-market. It can used directly in systems based in C/C++ and Python.

+

Within VPI, the same algorithm is implemented in different backends, such as CPU, GPU, PVA, VIC and OFA. The processing pipeline can be set up to utilize the full installed computing capacity of the target device. For example, the GPU can perform inference on one frame while the PVA and VIC preprocess the image in a subsequent frame. The CPU can perform housekeeping tasks such as updating the GUI and postprocessing images without affecting the performance of other tasks.

+

VPI provides seamless zero-copy memory mapping among the supported device backends, depending on certain memory characteristics. For platforms that support it, zero-copy memory mapping yields a substantial increase in throughput.

+

VPI supports easy interoperation with existing projects that make use of OpenCV and NVIDIA® CUDA® SDK libraries, among others. This allows for gradual replacement of existing computing tasks with faster VPI equivalents.

+

Here are some examples of algorithms provided by VPI:

+ + + + + + + + +
+
+
+ + + + + + diff --git a/research/gpu-fusion-splatting/mirrors/opencv-cuda-module.html b/research/gpu-fusion-splatting/mirrors/opencv-cuda-module.html new file mode 100644 index 0000000..4b0cd59 --- /dev/null +++ b/research/gpu-fusion-splatting/mirrors/opencv-cuda-module.html @@ -0,0 +1,161 @@ + + + + + + + +OpenCV: CUDA-accelerated Computer Vision + + + + + + + + + + + + + +
+
+ + + + + + + + + + + +
+
OpenCV +  3.4.20-dev +
+
Open Source Computer Vision
+
+
+ + + + + + + +
+ +
+
+ + +
+ +
+ +
+ +
+
CUDA-accelerated Computer Vision
+
+ + + + + + + + diff --git a/research/gpu-fusion-splatting/mirrors/peng-2024-rtg-slam.pdf b/research/gpu-fusion-splatting/mirrors/peng-2024-rtg-slam.pdf new file mode 100644 index 0000000..1bed71b Binary files /dev/null and b/research/gpu-fusion-splatting/mirrors/peng-2024-rtg-slam.pdf differ diff --git a/research/gpu-fusion-splatting/mirrors/wu-2024-4d-gaussian-splatting.pdf b/research/gpu-fusion-splatting/mirrors/wu-2024-4d-gaussian-splatting.pdf new file mode 100644 index 0000000..a13c96d Binary files /dev/null and b/research/gpu-fusion-splatting/mirrors/wu-2024-4d-gaussian-splatting.pdf differ diff --git a/research/gpu-fusion-splatting/mirrors/ye-2025-gsplat.pdf b/research/gpu-fusion-splatting/mirrors/ye-2025-gsplat.pdf new file mode 100644 index 0000000..218bc62 Binary files /dev/null and b/research/gpu-fusion-splatting/mirrors/ye-2025-gsplat.pdf differ diff --git a/research/gpu-fusion-splatting/summary.md b/research/gpu-fusion-splatting/summary.md new file mode 100644 index 0000000..e5436af --- /dev/null +++ b/research/gpu-fusion-splatting/summary.md @@ -0,0 +1,165 @@ +# GPU Fusion And Live Splat Rendering + +## Objective + +Design the implementation path for fusing live camera/Leap/audio observations and rendering a Dreams-like volumetric brushstroke or Gaussian-splat field with the hot path on GPU compute. + +## Current Mechanism + +The visual-spatial map now says the world model owns truth, while brushstroke/splat packets are render lowerings. This note adds the GPU implementation constraint: + +```text +capture ingress +-> pinned host staging +-> async GPU upload / CUDA streams +-> GPU feature/marker/depth work +-> GPU-friendly track/claim buffers +-> splat/brush packet generation +-> CUDA/Vulkan/OpenGL renderer interop +``` + +The CPU remains the conductor: device enumeration, calibration files, graph scheduling, small control decisions, logging, and fallback. It should not be the pixel furnace. + +## Key Findings + +The original 3D Gaussian Splatting implementation is CUDA-backed and includes a real-time viewer requiring CUDA-capable hardware for its real-time path [GraphDeco]. Kerbl et al. use anisotropic Gaussians, adaptive density control, and a fast visibility-aware renderer to avoid wasting work on empty space [Kerbl 2023]. + +NVIDIA's `vk_gaussian_splatting` sample is useful because it treats 3DGS as a real-time visualization testbed and compares graphics-pipeline approaches for splat rendering, including mesh-shader and vertex-shader paths [NVIDIA vkGS]. That suggests we should not blindly force every render step through CUDA if Vulkan/D3D mesh/compute stages do the final raster work better. GPU compute is the goal; CUDA is one implementation authority, not a religion with a tiny hat. + +`gsplat` is the strongest practical CUDA library reference. Its documentation describes CUDA-accelerated Gaussian rasterization, and the JMLR paper frames it as a PyTorch front-end with highly optimized CUDA kernels, intended to be faster, more memory-efficient, and easier to modify than the original backend [gsplat Docs; Ye 2025]. Nerfstudio's Splatfacto uses gsplat as its rasterization backend [Nerfstudio]. + +For sensor processing, NVIDIA VPI matters because it exposes computer vision operations across hardware backends, including CUDA, and includes building blocks such as stereo disparity, KLT feature tracking, and perspective warp [NVIDIA VPI]. OpenCV's CUDA module provides GPU-accelerated computer vision primitives [OpenCV CUDA]. NVIDIA's Optical Flow SDK exposes hardware acceleration for optical flow and stereo disparity on supported GPUs [NVIDIA OFA]. + +CUDA streams and asynchronous execution are the runtime spine. NVIDIA's CUDA guide defines streams as ordered work queues for memory copies and kernels; separate streams can express overlap between capture upload, feature extraction, track update, and render prep when the hardware supports it [CUDA Async]. + +## Proposed GPU Ownership Model + +```mermaid +flowchart TD + A["Camera/Leap frame capture"] --> B["Pinned host frame rings"] + B --> C["Async H2D upload streams"] + C --> D["GPU image preprocessing"] + D --> E["GPU marker/features/stereo/depth"] + E --> F["GPU observation buffers"] + G["Audio estimators"] --> H["CPU/GPU world claims"] + F --> I["Track update / fusion buffers"] + H --> I + I --> J["Brush/splat packet builder"] + J --> K["GPU splat renderer"] + K --> L["OBS / preview / virtual camera output"] +``` + +## Data Model + +Prefer structure-of-arrays buffers for CUDA kernels: + +```text +SensorState: + intrinsics[] + extrinsics_world_from_sensor[] + latency_model[] + clock_model[] + +ObservationBuffer: + sensor_id[] + timestamp_ns[] + uv_or_xyz[] + confidence[] + marker_id[] + +TrackBuffer: + id[] + position[] + velocity[] + covariance[] + confidence[] + last_update_time[] + +BrushPacketBuffer: + stable_key[] + position[] + orientation_or_quat[] + scale[] + color[] + opacity[] + confidence[] + source_track_id[] +``` + +Do not start with a generic object graph on the GPU. That path becomes pointer soup with a profiler attached. + +## Pipeline Proposal + +1. Capture ingress. + Each camera writes into a small ring of pinned host buffers. Use one capture thread per physical device if the driver requires it, but keep frame metadata uniform: device id, sequence, host timestamp, exposure/gain, and intended world transform version. + +2. Async upload. + Copy frames into GPU memory on per-device or per-class CUDA streams. Use CUDA events to timestamp stage completion. CPU blocks only on bounded ring pressure, not on per-frame processing. + +3. GPU image work. + PS3 Eyes: threshold/blob/marker extraction. + LeapUVC: stereo rectification, close-range disparity or marker extraction. + Kiyos: downsampled feature/color projection path; keep full-res RGB for texture/projection only when budget allows. + +4. Fusion. + First implementation can run track updates on CPU if observation counts are tiny, but keep buffers GPU-shaped from day one. Move Kalman/EKF batch updates to CUDA once there are enough tracks or per-pixel/depth observations to justify it. + +5. Packet generation. + Convert tracks and local surfaces into brush/splat packets on GPU or with a CPU fallback writing the same SoA buffer. This packet buffer is the render ABI. + +6. Render. + Use an existing CUDA 3DGS/gsplat path for experiments, but the production live renderer may be Vulkan/D3D compute/mesh shader with CUDA interop depending on OBS/preview integration. The invariant is GPU-resident packet data, not a specific API logo. + +7. Readback discipline. + Read back metrics and small debug summaries only: counts, residuals, confidence, dropped frames, GPU timings. Do not read back full frames or point clouds during live operation unless recording/debugging. + +## Gaussian vs Brushstroke + +Classic 3DGS wants optimized radiance-field Gaussians from posed images. Our live scene wants expressive, sensor-driven volumetric strokes. Use 3DGS machinery where it helps: + +- anisotropic splat parameterization +- tile/bin/sort/raster kernels +- packed GPU buffers +- visibility-aware culling +- LOD/density control ideas + +But do not require full offline radiance-field training before the live rig can show anything. V1 brush packets can be directly emitted from tracks and RGB projections. Later, local regions can be optimized into Gaussian fields when the data is stable enough. + +## First GPU Slice + +1. GPU-upload one PS3 Eye frame and one Kiyo frame using pinned host rings. +2. Run a CUDA threshold/blob extraction kernel on the PS3 Eye frame. +3. Emit one `ObservationBuffer` entry for a bright marker. +4. Triangulate on CPU or GPU from two PS3 Eye observations. +5. Write one `BrushPacketBuffer` entry. +6. Render one translucent anisotropic splat/brush from that GPU buffer. +7. Expose timings: capture, upload, detect, fuse, packetize, render. + +That proves the GPU memory path before the larger system starts wearing a cape. + +## Cut Line + +Cut any plan that: + +- captures frames, processes pixels on CPU, then uploads final pretty geometry only +- requires CPU readback of dense point clouds every frame +- starts with full 4D Gaussian training before sparse live tracking works +- makes CUDA kernels own calibration semantics +- ties the world model to one renderer backend +- ignores USB/capture bandwidth and upload overlap + +## Citations + +- [CUDA Guide] NVIDIA CUDA C++ Programming Guide. Mirror: `mirrors/nvidia-cuda-programming-guide.pdf`. +- [CUDA Async] NVIDIA CUDA C++ Programming Guide, asynchronous/concurrent execution. Mirror: `mirrors/nvidia-cuda-async-concurrent-execution.html`. +- [NVIDIA VPI] NVIDIA Vision Programming Interface. Mirror: `mirrors/nvidia-vpi-main.html`. +- [NVIDIA OFA] NVIDIA Optical Flow SDK Programming Guide. Mirror: `mirrors/nvidia-optical-flow-programming-guide.pdf`. +- [OpenCV CUDA] OpenCV CUDA-accelerated Computer Vision module. Mirror: `mirrors/opencv-cuda-module.html`. +- [GraphDeco] GraphDeco/Inria `gaussian-splatting`. Mirrors: `mirrors/graphdeco-gaussian-splatting-github.html`, `mirrors/graphdeco-gaussian-splatting-readme.md`. +- [NVIDIA vkGS] NVIDIA Developer Blog, `vk_gaussian_splatting`. Mirror: `mirrors/nvidia-vk-gaussian-splatting-blog.html`. +- [gsplat Docs] gsplat documentation. Mirror: `mirrors/gsplat-docs.html`. +- [Ye 2025] M. Ye et al., "gsplat: An Open-Source Library for Gaussian Splatting." Mirror: `mirrors/ye-2025-gsplat.pdf`. +- [Nerfstudio] Nerfstudio, "Splatfacto." Mirror: `mirrors/nerfstudio-splatfacto.html`. +- [Kerbl 2023] Bernhard Kerbl et al., "3D Gaussian Splatting for Real-Time Radiance Field Rendering." Mirror: `mirrors/kerbl-2023-3d-gaussian-splatting.pdf`. +- [Wu 2024] Guanjun Wu et al., "4D Gaussian Splatting for Real-Time Dynamic Scene Rendering." Mirror: `mirrors/wu-2024-4d-gaussian-splatting.pdf`. +- [Peng 2024] Zhexi Peng et al., "RTG-SLAM." Mirror: `mirrors/peng-2024-rtg-slam.pdf`. diff --git a/research/live-adaptive-sync/bibliography.md b/research/live-adaptive-sync/bibliography.md new file mode 100644 index 0000000..d9201fa --- /dev/null +++ b/research/live-adaptive-sync/bibliography.md @@ -0,0 +1,36 @@ +# Bibliography: Live Adaptive Sync + +Mirrored copies live under `research/live-adaptive-sync/mirrors/`. + +## Sampling-Rate Offset / Clock Drift + +- L. Wang and S. Doclo, "Correlation Maximization-Based Sampling Rate Offset Estimation for Distributed Microphone Arrays," IEEE/ACM TASLP, 2016. + Mirror: `mirrors/wang-doclo-2016-correlation-maximization-sro.pdf` + +- J. Schmalenstroeer and R. Haeb-Umbach, "Sampling Rate Synchronisation in Acoustic Sensor Networks with a Pre-Trained Clock Skew Error Model," 2013. + Mirror: `mirrors/schmalenstroeer-haeb-umbach-2013-pretrained-clock-skew.pdf` + +- P. Didier, T. van Waterschoot, S. Doclo, and M. Moonen, "Sampling Rate Offset Estimation and Compensation for Distributed Adaptive Node-Specific Signal Estimation in Wireless Acoustic Sensor Networks," 2023. + Mirror: `mirrors/didier-2023-sro-estimation-compensation-wasn.pdf` + +- "On Synchronization of Wireless Acoustic Sensor Networks in the Presence of Time-varying Sampling Rate Offsets and Speaker Changes," arXiv:2110.12820. + Mirror: `mirrors/synchronization-time-varying-sro-speaker-changes.pdf` + +- "Online Estimation of Sampling Rate Offsets," EUSIPCO 2021. + Mirror: `mirrors/online-estimation-sampling-rate-offsets-2021.pdf` + +## Delay / Temporal Alignment / Localization + +- A. Shrestha et al., "Temporal Synchronization of Multiple Audio Signals," Google Research. + Mirror: `mirrors/google-temporal-synchronization-multiple-audio.html` + +- F. Grondin and J. Glass, "A Study of the Complexity and Accuracy of Direction of Arrival Estimation Methods Based on GCC-PHAT for a Pair of Close Microphones," arXiv:1811.11787, 2018. + Mirror: `mirrors/grondin-glass-2018-gcc-phat-close-mics.pdf` + +## Implementation References + +- FFmpeg Filters Documentation, especially `aresample` / timestamp-driven audio compensation. + Mirror: `mirrors/ffmpeg-filters.html` + +- SoX Resampler library / libsoxr documentation, especially variable-rate resampling (`SOXR_VR`). + Mirror: `mirrors/soxr-wiki-home.html` diff --git a/research/live-adaptive-sync/mirrors/didier-2023-sro-estimation-compensation-wasn.pdf b/research/live-adaptive-sync/mirrors/didier-2023-sro-estimation-compensation-wasn.pdf new file mode 100644 index 0000000..b51552e Binary files /dev/null and b/research/live-adaptive-sync/mirrors/didier-2023-sro-estimation-compensation-wasn.pdf differ diff --git a/research/live-adaptive-sync/mirrors/ffmpeg-filters.html b/research/live-adaptive-sync/mirrors/ffmpeg-filters.html new file mode 100644 index 0000000..bef0ce5 --- /dev/null +++ b/research/live-adaptive-sync/mirrors/ffmpeg-filters.html @@ -0,0 +1,40068 @@ + + + + + FFmpeg Filters Documentation + + + + + + + + + + + + + +
+ + + +
+
+

+ FFmpeg Filters Documentation +

+
+
+
+ + + + +

Table of Contents

+ +
+ + +
+ + +

1 Description

+ +

This document describes filters, sources, and sinks provided by the +libavfilter library. +

+ +

2 Filtering Introduction

+ +

Filtering in FFmpeg is enabled through the libavfilter library. +

+

In libavfilter, a filter can have multiple inputs and multiple +outputs. +To illustrate the sorts of things that are possible, we consider the +following filtergraph. +

+
                [main]
+input --> split ---------------------> overlay --> output
+            |                             ^
+            |[tmp]                  [flip]|
+            +-----> crop --> vflip -------+
+
+

This filtergraph splits the input stream in two streams, then sends one +stream through the crop filter and the vflip filter, before merging it +back with the other stream by overlaying it on top. You can use the +following command to achieve this: +

+
+
ffmpeg -i INPUT -vf "split [main][tmp]; [tmp] crop=iw:ih/2:0:0, vflip [flip]; [main][flip] overlay=0:H/2" OUTPUT
+
+ +

The result will be that the top half of the video is mirrored +onto the bottom half of the output video. +

+

Filters in the same linear chain are separated by commas, and distinct +linear chains of filters are separated by semicolons. In our example, +crop,vflip are in one linear chain, split and +overlay are separately in another. The points where the linear +chains join are labelled by names enclosed in square brackets. In the +example, the split filter generates two outputs that are associated to +the labels [main] and [tmp]. +

+

The stream sent to the second output of split, labelled as +[tmp], is processed through the crop filter, which crops +away the lower half part of the video, and then vertically flipped. The +overlay filter takes in input the first unchanged output of the +split filter (which was labelled as [main]), and overlay on its +lower half the output generated by the crop,vflip filterchain. +

+

Some filters take in input a list of parameters: they are specified +after the filter name and an equal sign, and are separated from each other +by a colon. +

+

There exist so-called source filters that do not have an +audio/video input, and sink filters that will not have audio/video +output. +

+ +

3 graph2dot

+ +

The graph2dot program included in the FFmpeg tools +directory can be used to parse a filtergraph description and issue a +corresponding textual representation in the dot language. +

+

Invoke the command: +

+
graph2dot -h
+
+ +

to see how to use graph2dot. +

+

You can then pass the dot description to the dot program (from +the graphviz suite of programs) and obtain a graphical representation +of the filtergraph. +

+

For example the sequence of commands: +

+
echo GRAPH_DESCRIPTION | \
+tools/graph2dot -o graph.tmp && \
+dot -Tpng graph.tmp -o graph.png && \
+display graph.png
+
+ +

can be used to create and display an image representing the graph +described by the GRAPH_DESCRIPTION string. Note that this string must be +a complete self-contained graph, with its inputs and outputs explicitly defined. +For example if your command line is of the form: +

+
ffmpeg -i infile -vf scale=640:360 outfile
+
+

your GRAPH_DESCRIPTION string will need to be of the form: +

+
nullsrc,scale=640:360,nullsink
+
+

you may also need to set the nullsrc parameters and add a format +filter in order to simulate a specific input file. +

+ +

4 Filtergraph description

+ +

A filtergraph is a directed graph of connected filters. It can contain +cycles, and there can be multiple links between a pair of +filters. Each link has one input pad on one side connecting it to one +filter from which it takes its input, and one output pad on the other +side connecting it to one filter accepting its output. +

+

Each filter in a filtergraph is an instance of a filter class +registered in the application, which defines the features and the +number of input and output pads of the filter. +

+

A filter with no input pads is called a "source", and a filter with no +output pads is called a "sink". +

+

4.1 Filtergraph syntax

+ +

A filtergraph has a textual representation, which is recognized by the +-filter/-vf/-af and +-filter_complex options in ffmpeg and +-vf/-af in ffplay, and by the +avfilter_graph_parse_ptr() function defined in +libavfilter/avfilter.h. +

+

A filterchain consists of a sequence of connected filters, each one +connected to the previous one in the sequence. A filterchain is +represented by a list of ","-separated filter descriptions. +

+

A filtergraph consists of a sequence of filterchains. A sequence of +filterchains is represented by a list of ";"-separated filterchain +descriptions. +

+

A filter is represented by a string of the form: +[in_link_1]...[in_link_N]filter_name@id=arguments[out_link_1]...[out_link_M] +

+

filter_name is the name of the filter class of which the +described filter is an instance of, and has to be the name of one of +the filter classes registered in the program optionally followed by "@id". +The name of the filter class is optionally followed by a string +"=arguments". +

+

arguments is a string which contains the parameters used to +initialize the filter instance. It may have one of two forms: +

    +
  • A ’:’-separated list of key=value pairs. + +
  • A ’:’-separated list of value. In this case, the keys are assumed to be +the option names in the order they are declared. E.g. the fade filter +declares three options in this order – type, start_frame and +nb_frames. Then the parameter list in:0:30 means that the value +in is assigned to the option type, 0 to +start_frame and 30 to nb_frames. + +
  • A ’:’-separated list of mixed direct value and long key=value +pairs. The direct value must precede the key=value pairs, and +follow the same constraints order of the previous point. The following +key=value pairs can be set in any preferred order. + +
+ +

If the option value itself is a list of items (e.g. the format filter +takes a list of pixel formats), the items in the list are usually separated by +‘|’. +

+

The list of arguments can be quoted using the character ‘'’ as initial +and ending mark, and the character ‘\’ for escaping the characters +within the quoted text; otherwise the argument string is considered +terminated when the next special character (belonging to the set +‘[]=;,’) is encountered. +

+

A special syntax implemented in the ffmpeg CLI tool allows loading +option values from files. This is done be prepending a slash ’/’ to the option +name, then the supplied value is interpreted as a path from which the actual +value is loaded. E.g. +

+
ffmpeg -i <INPUT> -vf drawtext=/text=/tmp/some_text <OUTPUT>
+
+

will load the text to be drawn from /tmp/some_text. API users wishing to +implement a similar feature should use the avfilter_graph_segment_*() +functions together with custom IO code. +

+

The name and arguments of the filter are optionally preceded and +followed by a list of link labels. +A link label allows one to name a link and associate it to a filter output +or input pad. The preceding labels in_link_1 +... in_link_N, are associated to the filter input pads, +the following labels out_link_1 ... out_link_M, are +associated to the output pads. +

+

When two link labels with the same name are found in the +filtergraph, a link between the corresponding input and output pad is +created. +

+

If an output pad is not labelled, it is linked by default to the first +unlabelled input pad of the next filter in the filterchain. +For example in the filterchain +

+
nullsrc, split[L1], [L2]overlay, nullsink
+
+

the split filter instance has two output pads, and the overlay filter +instance two input pads. The first output pad of split is labelled +"L1", the first input pad of overlay is labelled "L2", and the second +output pad of split is linked to the second input pad of overlay, +which are both unlabelled. +

+

In a filter description, if the input label of the first filter is not +specified, "in" is assumed; if the output label of the last filter is not +specified, "out" is assumed. +

+

In a complete filterchain all the unlabelled filter input and output +pads must be connected. A filtergraph is considered valid if all the +filter input and output pads of all the filterchains are connected. +

+

Leading and trailing whitespaces (space, tabs, or line feeds) separating tokens +in the filtergraph specification are ignored. This means that the filtergraph +can be expressed using empty lines and spaces to improve readability. +

+

For example, the filtergraph: +

+
testsrc,split[L1],hflip[L2];[L1][L2] hstack
+
+ +

can be represented as: +

+
testsrc,
+split [L1], hflip [L2];
+
+[L1][L2] hstack
+
+ +

Libavfilter will automatically insert scale filters where format +conversion is required. It is possible to specify swscale flags +for those automatically inserted scalers by prepending +sws_flags=flags; +to the filtergraph description. +

+

Here is a BNF description of the filtergraph syntax: +

+
NAME             ::= sequence of alphanumeric characters and '_'
+FILTER_NAME      ::= NAME["@"NAME]
+LINKLABEL        ::= "[" NAME "]"
+LINKLABELS       ::= LINKLABEL [LINKLABELS]
+FILTER_ARGUMENTS ::= sequence of chars (possibly quoted)
+FILTER           ::= [LINKLABELS] FILTER_NAME ["=" FILTER_ARGUMENTS] [LINKLABELS]
+FILTERCHAIN      ::= FILTER [,FILTERCHAIN]
+FILTERGRAPH      ::= [sws_flags=flags;] FILTERCHAIN [;FILTERGRAPH]
+
+ +

4.2 Notes on filtergraph escaping

+ +

Filtergraph description composition entails several levels of +escaping. See (ffmpeg-utils)the "Quoting and escaping" +section in the ffmpeg-utils(1) manual for more +information about the employed escaping procedure. +

+

A first level escaping affects the content of each filter option +value, which may contain the special character : used to +separate values, or one of the escaping characters \'. +

+

A second level escaping affects the whole filter description, which +may contain the escaping characters \' or the special +characters [],; used by the filtergraph description. +

+

Finally, when you specify a filtergraph on a shell commandline, you +need to perform a third level escaping for the shell special +characters contained within it. +

+

For example, consider the following string to be embedded in +the drawtext filter description text value: +

+
this is a 'string': may contain one, or more, special characters
+
+ +

This string contains the ' special escaping character, and the +: special character, so it needs to be escaped in this way: +

+
text=this is a \'string\'\: may contain one, or more, special characters
+
+ +

A second level of escaping is required when embedding the filter +description in a filtergraph description, in order to escape all the +filtergraph special characters. Thus the example above becomes: +

+
drawtext=text=this is a \\\'string\\\'\\: may contain one\, or more\, special characters
+
+

(note that in addition to the \' escaping special characters, +also , needs to be escaped). +

+

Finally an additional level of escaping is needed when writing the +filtergraph description in a shell command, which depends on the +escaping rules of the adopted shell. For example, assuming that +\ is special and needs to be escaped with another \, the +previous string will finally result in: +

+
-vf "drawtext=text=this is a \\\\\\'string\\\\\\'\\\\: may contain one\\, or more\\, special characters"
+
+ +

In order to avoid cumbersome escaping when using a commandline tool accepting a +filter specification as input, it is advisable to avoid direct inclusion of the +filter or options specification in the shell. +

+

For example, in case of the drawtext filter, you might prefer to +use the textfile option in place of text to specify the text +to render. +

+

5 Timeline editing

+ +

Some filters support a generic enable option. For the filters +supporting timeline editing, this option can be set to an expression which is +evaluated before sending a frame to the filter. If the evaluation is non-zero, +the filter will be enabled, otherwise the frame will be sent unchanged to the +next filter in the filtergraph. +

+

The expression accepts the following values: +

+
‘t’
+

timestamp expressed in seconds, NAN if the input timestamp is unknown +

+
+
‘n’
+

sequential number of the input frame, starting from 0 +

+
+
‘pos’
+

the position in the file of the input frame, NAN if unknown; deprecated, do +not use +

+
+
‘w’
+
‘h’
+

width and height of the input frame if video +

+
+ +

Additionally, these filters support an enable command that can be used +to re-define the expression. +

+

Like any other filtering option, the enable option follows the same +rules. +

+

For example, to enable a blur filter (smartblur) from 10 seconds to 3 +minutes, and a curves filter starting at 3 seconds: +

+
smartblur = enable='between(t,10,3*60)',
+curves    = enable='gte(t,3)' : preset=cross_process
+
+ +

See ffmpeg -filters to view which filters have timeline support. +

+ +

6 Changing options at runtime with a command

+ +

Some options can be changed during the operation of the filter using +a command. These options are marked ’T’ on the output of +ffmpeg -h filter=<name of filter>. +The name of the command is the name of the option and the argument is +the new value. +

+

7 Options for filters with several inputs (framesync)

+ +

Some filters with several inputs support a common set of options. +These options can only be set by name, not with the short notation. +

+
+
eof_action
+

The action to take when EOF is encountered on the secondary input; it accepts +one of the following values: +

+
+
repeat
+

Repeat the last frame (the default). +

+
endall
+

End both streams. +

+
pass
+

Pass the main input through. +

+
+ +
+
shortest
+

If set to 1, force the output to terminate when the shortest input +terminates. Default value is 0. +

+
+
repeatlast
+

If set to 1, force the filter to extend the last frame of secondary streams +until the end of the primary stream. A value of 0 disables this behavior. +Default value is 1. +

+
+
ts_sync_mode
+

How strictly to sync streams based on secondary input timestamps; it accepts +one of the following values: +

+
+
default
+

Frame from secondary input with the nearest lower or equal timestamp to the +primary input frame. +

+
nearest
+

Frame from secondary input with the absolute nearest timestamp to the primary +input frame. +

+
+
+
+ + +

8 Audio Filters

+ +

When you configure your FFmpeg build, you can disable any of the +existing filters using --disable-filters. +The configure output will show the audio filters included in your +build. +

+

Below is a description of the currently available audio filters. +

+

8.1 aap

+

Apply Affine Projection algorithm to the first audio stream using the second audio stream. +

+

This adaptive filter is used to estimate unknown audio based on multiple input audio samples. +Affine projection algorithm can make trade-offs between computation complexity with convergence speed. +

+

A description of the accepted options follows. +

+
+
order
+

Set the filter order. +

+
+
projection
+

Set the projection order. +

+
+
mu
+

Set the filter mu. +

+
+
delta
+

Set the coefficient to initialize internal covariance matrix. +

+
+
out_mode
+

Set the filter output samples. It accepts the following values: +

+
i
+

Pass the 1st input. +

+
+
d
+

Pass the 2nd input. +

+
+
o
+

Pass difference between desired, 2nd input and error signal estimate. +

+
+
n
+

Pass difference between input, 1st input and error signal estimate. +

+
+
e
+

Pass error signal estimated samples. +

+

Default value is o. +

+
+ +
+
precision
+

Set which precision to use when processing samples. +

+
+
auto
+

Auto pick internal sample format depending on other filters. +

+
+
float
+

Always use single-floating point precision sample format. +

+
+
double
+

Always use double-floating point precision sample format. +

+
+
+
+ +

8.2 acompressor

+ +

A compressor is mainly used to reduce the dynamic range of a signal. +Especially modern music is mostly compressed at a high ratio to +improve the overall loudness. It’s done to get the highest attention +of a listener, "fatten" the sound and bring more "power" to the track. +If a signal is compressed too much it may sound dull or "dead" +afterwards or it may start to "pump" (which could be a powerful effect +but can also destroy a track completely). +The right compression is the key to reach a professional sound and is +the high art of mixing and mastering. Because of its complex settings +it may take a long time to get the right feeling for this kind of effect. +

+

Compression is done by detecting the volume above a chosen level +threshold and dividing it by the factor set with ratio. +So if you set the threshold to -12dB and your signal reaches -6dB a ratio +of 2:1 will result in a signal at -9dB. Because an exact manipulation of +the signal would cause distortion of the waveform the reduction can be +levelled over the time. This is done by setting "Attack" and "Release". +attack determines how long the signal has to rise above the threshold +before any reduction will occur and release sets the time the signal +has to fall below the threshold to reduce the reduction again. Shorter signals +than the chosen attack time will be left untouched. +The overall reduction of the signal can be made up afterwards with the +makeup setting. So compressing the peaks of a signal about 6dB and +raising the makeup to this level results in a signal twice as loud than the +source. To gain a softer entry in the compression the knee flattens the +hard edge at the threshold in the range of the chosen decibels. +

+

The filter accepts the following options: +

+
+
level_in
+

Set input gain. Default is 1. Range is between 0.015625 and 64. +

+
+
mode
+

Set mode of compressor operation. Can be upward or downward. +Default is downward. +

+
+
threshold
+

If a signal of stream rises above this level it will affect the gain +reduction. +By default it is 0.125. Range is between 0.00097563 and 1. +

+
+
ratio
+

Set a ratio by which the signal is reduced. 1:2 means that if the level +rose 4dB above the threshold, it will be only 2dB above after the reduction. +Default is 2. Range is between 1 and 20. +

+
+
attack
+

Amount of milliseconds the signal has to rise above the threshold before gain +reduction starts. Default is 20. Range is between 0.01 and 2000. +

+
+
release
+

Amount of milliseconds the signal has to fall below the threshold before +reduction is decreased again. Default is 250. Range is between 0.01 and 9000. +

+
+
makeup
+

Set the amount by how much signal will be amplified after processing. +Default is 1. Range is from 1 to 64. +

+
+
knee
+

Curve the sharp knee around the threshold to enter gain reduction more softly. +Default is 2.82843. Range is between 1 and 8. +

+
+
link
+

Choose if the average level between all channels of input stream +or the louder(maximum) channel of input stream affects the +reduction. Default is average. +

+
+
detection
+

Should the exact signal be taken in case of peak or an RMS one in case +of rms. Default is rms which is mostly smoother. +

+
+
mix
+

How much to use compressed signal in output. Default is 1. +Range is between 0 and 1. +

+
+ +

8.2.1 Commands

+ +

This filter supports the all above options as commands. +

+

8.3 acontrast

+

Simple audio dynamic range compression/expansion filter. +

+

The filter accepts the following options: +

+
+
contrast
+

Set contrast. Default is 33. Allowed range is between 0 and 100. +

+
+ +

8.4 acopy

+ +

Copy the input audio source unchanged to the output. This is mainly useful for +testing purposes. +

+

8.5 acrossfade

+ +

Apply cross fade from one input audio stream to another input audio stream. +The cross fade is applied for specified duration near the end of first stream. +

+

The filter accepts the following options: +

+
+
inputs, n
+

Specify the number of inputs to crossfade. When crossfading multiple inputs, +each input will be concatenated and crossfaded in sequence, similar to the +concat filter. Default is 2. +

+
+
nb_samples, ns
+

Specify the number of samples for which the cross fade effect has to last. +At the end of the cross fade effect the first input audio will be completely +silent. Default is 44100. +

+
+
duration, d
+

Specify the duration of the cross fade effect. See +(ffmpeg-utils)the Time duration section in the ffmpeg-utils(1) manual +for the accepted syntax. +By default the duration is determined by nb_samples. +If set this option is used instead of nb_samples. +

+
+
overlap, o
+

Should first stream end overlap with second stream start. Default is enabled. +

+
+
curve1
+

Set curve for cross fade transition for first stream. +

+
+
curve2
+

Set curve for cross fade transition for second stream. +

+

For description of available curve types see afade filter description. +

+
+ +

8.5.1 Examples

+ +
    +
  • Cross fade from one input to another: +
    +
    ffmpeg -i first.flac -i second.flac -filter_complex acrossfade=d=10:c1=exp:c2=exp output.flac
    +
    + +
  • Cross fade from one input to another but without overlapping: +
    +
    ffmpeg -i first.flac -i second.flac -filter_complex acrossfade=d=10:o=0:c1=exp:c2=exp output.flac
    +
    + +

    Concatenate multiple inputs with cross fade between each: +

    +
    ffmpeg -i first.flac -i second.flac -i third.flac -filter_complex acrossfade=n=3 output.flac
    +
    +
+ +

8.6 acrossover

+

Split audio stream into several bands. +

+

This filter splits audio stream into two or more frequency ranges. +Summing all streams back will give flat output. +

+

The filter accepts the following options: +

+
+
split
+

Set split frequencies. Those must be positive and increasing. +

+
+
order
+

Set filter order for each band split. This controls filter roll-off or steepness +of filter transfer function. +Available values are: +

+
+
‘2nd’
+

12 dB per octave. +

+
‘4th’
+

24 dB per octave. +

+
‘6th’
+

36 dB per octave. +

+
‘8th’
+

48 dB per octave. +

+
‘10th’
+

60 dB per octave. +

+
‘12th’
+

72 dB per octave. +

+
‘14th’
+

84 dB per octave. +

+
‘16th’
+

96 dB per octave. +

+
‘18th’
+

108 dB per octave. +

+
‘20th’
+

120 dB per octave. +

+
+ +

Default is 4th. +

+
+
level
+

Set input gain level. Allowed range is from 0 to 1. Default value is 1. +

+
+
gains
+

Set output gain for each band. Default value is 1 for all bands. +

+
+
precision
+

Set which precision to use when processing samples. +

+
+
auto
+

Auto pick internal sample format depending on other filters. +

+
+
float
+

Always use single-floating point precision sample format. +

+
+
double
+

Always use double-floating point precision sample format. +

+
+ +

Default value is auto. +

+
+ +

8.6.1 Examples

+ +
    +
  • Split input audio stream into two bands (low and high) with split frequency of 1500 Hz, +each band will be in separate stream: +
    +
    ffmpeg -i in.flac -filter_complex 'acrossover=split=1500[LOW][HIGH]' -map '[LOW]' low.wav -map '[HIGH]' high.wav
    +
    + +
  • Same as above, but with higher filter order: +
    +
    ffmpeg -i in.flac -filter_complex 'acrossover=split=1500:order=8th[LOW][HIGH]' -map '[LOW]' low.wav -map '[HIGH]' high.wav
    +
    + +
  • Same as above, but also with additional middle band (frequencies between 1500 and 8000): +
    +
    ffmpeg -i in.flac -filter_complex 'acrossover=split=1500 8000:order=8th[LOW][MID][HIGH]' -map '[LOW]' low.wav -map '[MID]' mid.wav -map '[HIGH]' high.wav
    +
    +
+ +

8.7 acrusher

+ +

Reduce audio bit resolution. +

+

This filter is bit crusher with enhanced functionality. A bit crusher +is used to audibly reduce number of bits an audio signal is sampled +with. This doesn’t change the bit depth at all, it just produces the +effect. Material reduced in bit depth sounds more harsh and "digital". +This filter is able to even round to continuous values instead of discrete +bit depths. +Additionally it has a D/C offset which results in different crushing of +the lower and the upper half of the signal. +An Anti-Aliasing setting is able to produce "softer" crushing sounds. +

+

Another feature of this filter is the logarithmic mode. +This setting switches from linear distances between bits to logarithmic ones. +The result is a much more "natural" sounding crusher which doesn’t gate low +signals for example. The human ear has a logarithmic perception, +so this kind of crushing is much more pleasant. +Logarithmic crushing is also able to get anti-aliased. +

+

The filter accepts the following options: +

+
+
level_in
+

Set level in. +

+
+
level_out
+

Set level out. +

+
+
bits
+

Set bit reduction. +

+
+
mix
+

Set mixing amount. +

+
+
mode
+

Can be linear: lin or logarithmic: log. +

+
+
dc
+

Set DC. +

+
+
aa
+

Set anti-aliasing. +

+
+
samples
+

Set sample reduction. +

+
+
lfo
+

Enable LFO. By default disabled. +

+
+
lforange
+

Set LFO range. +

+
+
lforate
+

Set LFO rate. +

+
+ +

8.7.1 Commands

+ +

This filter supports the all above options as commands. +

+

8.8 acue

+ +

Delay audio filtering until a given wallclock timestamp. See the cue +filter. +

+

8.9 adeclick

+

Remove impulsive noise from input audio. +

+

Samples detected as impulsive noise are replaced by interpolated samples using +autoregressive modelling. +

+
+
window, w
+

Set window size, in milliseconds. Allowed range is from 10 to +100. Default value is 55 milliseconds. +This sets size of window which will be processed at once. +

+
+
overlap, o
+

Set window overlap, in percentage of window size. Allowed range is from +50 to 95. Default value is 75 percent. +Setting this to a very high value increases impulsive noise removal but makes +whole process much slower. +

+
+
arorder, a
+

Set autoregression order, in percentage of window size. Allowed range is from +0 to 25. Default value is 2 percent. This option also +controls quality of interpolated samples using neighbour good samples. +

+
+
threshold, t
+

Set threshold value. Allowed range is from 1 to 100. +Default value is 2. +This controls the strength of impulsive noise which is going to be removed. +The lower value, the more samples will be detected as impulsive noise. +

+
+
burst, b
+

Set burst fusion, in percentage of window size. Allowed range is 0 to +10. Default value is 2. +If any two samples detected as noise are spaced less than this value then any +sample between those two samples will be also detected as noise. +

+
+
method, m
+

Set overlap method. +

+

It accepts the following values: +

+
add, a
+

Select overlap-add method. Even not interpolated samples are slightly +changed with this method. +

+
+
save, s
+

Select overlap-save method. Not interpolated samples remain unchanged. +

+
+ +

Default value is a. +

+
+ +

8.10 adeclip

+

Remove clipped samples from input audio. +

+

Samples detected as clipped are replaced by interpolated samples using +autoregressive modelling. +

+
+
window, w
+

Set window size, in milliseconds. Allowed range is from 10 to 100. +Default value is 55 milliseconds. +This sets size of window which will be processed at once. +

+
+
overlap, o
+

Set window overlap, in percentage of window size. Allowed range is from 50 +to 95. Default value is 75 percent. +

+
+
arorder, a
+

Set autoregression order, in percentage of window size. Allowed range is from +0 to 25. Default value is 8 percent. This option also controls +quality of interpolated samples using neighbour good samples. +

+
+
threshold, t
+

Set threshold value. Allowed range is from 1 to 100. +Default value is 10. Higher values make clip detection less aggressive. +

+
+
hsize, n
+

Set size of histogram used to detect clips. Allowed range is from 100 to 9999. +Default value is 1000. Higher values make clip detection less aggressive. +

+
+
method, m
+

Set overlap method. +

+

It accepts the following values: +

+
add, a
+

Select overlap-add method. Even not interpolated samples are slightly changed +with this method. +

+
+
save, s
+

Select overlap-save method. Not interpolated samples remain unchanged. +

+
+ +

Default value is a. +

+
+ +

8.11 adecorrelate

+

Apply decorrelation to input audio stream. +

+

The filter accepts the following options: +

+
+
stages
+

Set decorrelation stages of filtering. Allowed +range is from 1 to 16. Default value is 6. +

+
+
seed
+

Set random seed used for setting delay in samples across channels. +

+
+ +

8.12 adelay

+ +

Delay one or more audio channels. +

+

Samples in delayed channel are filled with silence. +

+

The filter accepts the following option: +

+
+
delays
+

Set list of delays in milliseconds for each channel separated by ’|’. +Unused delays will be silently ignored. If number of given delays is +smaller than number of channels all remaining channels will not be delayed. +If you want to delay exact number of samples, append ’S’ to number. +If you want instead to delay in seconds, append ’s’ to number. +

+
+
all
+

Use last set delay for all remaining channels. By default is disabled. +This option if enabled changes how option delays is interpreted. +

+
+ +

8.12.1 Examples

+ +
    +
  • Delay first channel by 1.5 seconds, the third channel by 0.5 seconds and leave +the second channel (and any other channels that may be present) unchanged. +
    +
    adelay=1500|0|500
    +
    + +
  • Delay second channel by 500 samples, the third channel by 700 samples and leave +the first channel (and any other channels that may be present) unchanged. +
    +
    adelay=0|500S|700S
    +
    + +
  • Delay all channels by same number of samples: +
    +
    adelay=delays=64S:all=1
    +
    +
+ +

8.13 adenorm

+

Remedy denormals in audio by adding extremely low-level noise. +

+

This filter shall be placed before any filter that can produce denormals. +

+

A description of the accepted parameters follows. +

+
+
level
+

Set level of added noise in dB. Default is -351. +Allowed range is from -451 to -90. +

+
+
type
+

Set type of added noise. +

+
+
dc
+

Add DC signal. +

+
ac
+

Add AC signal. +

+
square
+

Add square signal. +

+
pulse
+

Add pulse signal. +

+
+ +

Default is dc. +

+
+ +

8.13.1 Commands

+ +

This filter supports the all above options as commands. +

+

8.14 aderivative, aintegral

+ +

Compute derivative/integral of audio stream. +

+

Applying both filters one after another produces original audio. +

+

8.15 adrc

+ +

Apply spectral dynamic range controller filter to input audio stream. +

+

A description of the accepted options follows. +

+
+
transfer
+

Set the transfer expression. +

+

The expression can contain the following constants: +

+
+
ch
+

current channel number +

+
+
sn
+

current sample number +

+
+
nb_channels
+

number of channels +

+
+
t
+

timestamp expressed in seconds +

+
+
sr
+

sample rate +

+
+
p
+

current frequency power value, in dB +

+
+
f
+

current frequency in Hz +

+
+ +

Default value is p. +

+
+
attack
+

Set the attack in milliseconds. Default is 50 milliseconds. +Allowed range is from 1 to 1000 milliseconds. +

+
release
+

Set the release in milliseconds. Default is 100 milliseconds. +Allowed range is from 5 to 2000 milliseconds. +

+
channels
+

Set which channels to filter, by default all channels in audio stream are filtered. +

+
+ +

8.15.1 Commands

+ +

This filter supports the all above options as commands. +

+

8.15.2 Examples

+ +
    +
  • Apply spectral compression to all frequencies with threshold of -50 dB and 1:6 ratio: +
    +
    adrc=transfer='if(gt(p,-50),-50+(p-(-50))/6,p)':attack=50:release=100
    +
    + +
  • Similar to above but with 1:2 ratio and filtering only front center channel: +
    +
    adrc=transfer='if(gt(p,-50),-50+(p-(-50))/2,p)':attack=50:release=100:channels=FC
    +
    + +
  • Apply spectral noise gate to all frequencies with threshold of -85 dB and with short attack time and short release time: +
    +
    adrc=transfer='if(lte(p,-85),p-800,p)':attack=1:release=5
    +
    + +
  • Apply spectral expansion to all frequencies with threshold of -10 dB and 1:2 ratio: +
    +
    adrc=transfer='if(lt(p,-10),-10+(p-(-10))*2,p)':attack=50:release=100
    +
    + +
  • Apply limiter to max -60 dB to all frequencies, with attack of 2 ms and release of 10 ms: +
    +
    adrc=transfer='min(p,-60)':attack=2:release=10
    +
    +
+ +

8.16 adynamicequalizer

+ +

Apply dynamic equalization to input audio stream. +

+

A description of the accepted options follows. +

+
+
threshold
+

Set the detection threshold used to trigger equalization. +Threshold detection is using detection filter. +Default value is 0. Allowed range is from 0 to 100. +

+
+
dfrequency
+

Set the detection frequency in Hz used for detection filter used to trigger equalization. +Default value is 1000 Hz. Allowed range is between 2 and 1000000 Hz. +

+
+
dqfactor
+

Set the detection resonance factor for detection filter used to trigger equalization. +Default value is 1. Allowed range is from 0.001 to 1000. +

+
+
tfrequency
+

Set the target frequency of equalization filter. +Default value is 1000 Hz. Allowed range is between 2 and 1000000 Hz. +

+
+
tqfactor
+

Set the target resonance factor for target equalization filter. +Default value is 1. Allowed range is from 0.001 to 1000. +

+
+
attack
+

Set the amount of milliseconds the signal from detection has to rise above +the detection threshold before equalization starts. +Default is 20. Allowed range is between 1 and 2000. +

+
+
release
+

Set the amount of milliseconds the signal from detection has to fall below the +detection threshold before equalization ends. +Default is 200. Allowed range is between 1 and 2000. +

+
+
ratio
+

Set the ratio by which the equalization gain is raised. +Default is 1. Allowed range is between 0 and 30. +

+
+
makeup
+

Set the makeup offset by which the equalization gain is raised. +Default is 0. Allowed range is between 0 and 100. +

+
+
range
+

Set the max allowed cut/boost amount. Default is 50. +Allowed range is from 1 to 200. +

+
+
mode
+

Set the mode of filter operation, can be one of the following: +

+
+
‘listen’
+

Output only isolated detection signal. +

+
‘cutbelow’
+

Cut frequencies below detection threshold. +

+
‘cutabove’
+

Cut frequencies above detection threshold. +

+
‘boostbelow’
+

Boost frequencies below detection threshold. +

+
‘boostabove’
+

Boost frequencies above detection threshold. +

+
+

Default mode is ‘cutbelow’. +

+
+
dftype
+

Set the type of detection filter, can be one of the following: +

+
+
‘bandpass’
+
‘lowpass’
+
‘highpass’
+
‘peak’
+
+

Default type is ‘bandpass’. +

+
+
tftype
+

Set the type of target filter, can be one of the following: +

+
+
‘bell’
+
‘lowshelf’
+
‘highshelf’
+
+

Default type is ‘bell’. +

+
+
auto
+

Automatically gather threshold from detection filter. By default +is ‘disabled’. +This option is useful to detect threshold in certain time frame of +input audio stream, in such case option value is changed at runtime. +

+

Available values are: +

+
‘disabled’
+

Disable using automatically gathered threshold value. +

+
‘off’
+

Stop picking threshold value. +

+
‘on’
+

Start picking threshold value. +

+
‘adaptive’
+

Adaptively pick threshold value, by calculating sliding window entropy. +

+
+ +
+
precision
+

Set which precision to use when processing samples. +

+
+
auto
+

Auto pick internal sample format depending on other filters. +

+
+
float
+

Always use single-floating point precision sample format. +

+
+
double
+

Always use double-floating point precision sample format. +

+
+
+
+ +

8.16.1 Commands

+ +

This filter supports the all above options as commands. +

+

8.17 adynamicsmooth

+ +

Apply dynamic smoothing to input audio stream. +

+

A description of the accepted options follows. +

+
+
sensitivity
+

Set an amount of sensitivity to frequency fluctations. Default is 2. +Allowed range is from 0 to 1e+06. +

+
+
basefreq
+

Set a base frequency for smoothing. Default value is 22050. +Allowed range is from 2 to 1e+06. +

+
+ +

8.17.1 Commands

+ +

This filter supports the all above options as commands. +

+

8.18 aecho

+ +

Apply echoing to the input audio. +

+

Echoes are reflected sound and can occur naturally amongst mountains +(and sometimes large buildings) when talking or shouting; digital echo +effects emulate this behaviour and are often used to help fill out the +sound of a single instrument or vocal. The time difference between the +original signal and the reflection is the delay, and the +loudness of the reflected signal is the decay. +Multiple echoes can have different delays and decays. +

+

A description of the accepted parameters follows. +

+
+
in_gain
+

Set input gain of reflected signal. Default is 0.6. +

+
+
out_gain
+

Set output gain of reflected signal. Default is 0.3. +

+
+
delays
+

Set list of time intervals in milliseconds between original signal and reflections +separated by ’|’. Allowed range for each delay is (0 - 90000.0]. +Default is 1000. +

+
+
decays
+

Set list of loudness of reflected signals separated by ’|’. +Allowed range for each decay is (0 - 1.0]. +Default is 0.5. +

+
+ +

8.18.1 Examples

+ +
    +
  • Make it sound as if there are twice as many instruments as are actually playing: +
    +
    aecho=0.8:0.88:60:0.4
    +
    + +
  • If delay is very short, then it sounds like a (metallic) robot playing music: +
    +
    aecho=0.8:0.88:6:0.4
    +
    + +
  • A longer delay will sound like an open air concert in the mountains: +
    +
    aecho=0.8:0.9:1000:0.3
    +
    + +
  • Same as above but with one more mountain: +
    +
    aecho=0.8:0.9:1000|1800:0.3|0.25
    +
    +
+ +

8.19 aemphasis

+

Audio emphasis filter creates or restores material directly taken from LPs or +emphased CDs with different filter curves. E.g. to store music on vinyl the +signal has to be altered by a filter first to even out the disadvantages of +this recording medium. +Once the material is played back the inverse filter has to be applied to +restore the distortion of the frequency response. +

+

The filter accepts the following options: +

+
+
level_in
+

Set input gain. +

+
+
level_out
+

Set output gain. +

+
+
mode
+

Set filter mode. For restoring material use reproduction mode, otherwise +use production mode. Default is reproduction mode. +

+
+
type
+

Set filter type. Selects medium. Can be one of the following: +

+
+
col
+

select Columbia. +

+
emi
+

select EMI. +

+
bsi
+

select BSI (78RPM). +

+
riaa
+

select RIAA. +

+
cd
+

select Compact Disc (CD). +

+
50fm
+

select 50µs (FM). +

+
75fm
+

select 75µs (FM). +

+
50kf
+

select 50µs (FM-KF). +

+
75kf
+

select 75µs (FM-KF). +

+
+
+
+ +

8.19.1 Commands

+ +

This filter supports the all above options as commands. +

+

8.20 aeval

+ +

Modify an audio signal according to the specified expressions. +

+

This filter accepts one or more expressions (one for each channel), +which are evaluated and used to modify a corresponding audio signal. +

+

It accepts the following parameters: +

+
+
exprs
+

Set the ’|’-separated expressions list for each separate channel. If +the number of input channels is greater than the number of +expressions, the last specified expression is used for the remaining +output channels. +

+
+
channel_layout, c
+

Set output channel layout. If not specified, the channel layout is +specified by the number of expressions. If set to ‘same’, it will +use by default the same input channel layout. +

+
+ +

Each expression in exprs can contain the following constants and functions: +

+
+
ch
+

channel number of the current expression +

+
+
n
+

number of the evaluated sample, starting from 0 +

+
+
s
+

sample rate +

+
+
t
+

time of the evaluated sample expressed in seconds +

+
+
nb_in_channels
+
nb_out_channels
+

input and output number of channels +

+
+
val(CH)
+

the value of input channel with number CH +

+
+ +

Note: this filter is slow. For faster processing you should use a +dedicated filter. +

+

8.20.1 Examples

+ +
    +
  • Half volume: +
    +
    aeval=val(ch)/2:c=same
    +
    + +
  • Invert phase of the second channel: +
    +
    aeval=val(0)|-val(1)
    +
    +
+ +

8.21 aexciter

+ +

An exciter is used to produce high sound that is not present in the +original signal. This is done by creating harmonic distortions of the +signal which are restricted in range and added to the original signal. +An Exciter raises the upper end of an audio signal without simply raising +the higher frequencies like an equalizer would do to create a more +"crisp" or "brilliant" sound. +

+

The filter accepts the following options: +

+
+
level_in
+

Set input level prior processing of signal. +Allowed range is from 0 to 64. +Default value is 1. +

+
+
level_out
+

Set output level after processing of signal. +Allowed range is from 0 to 64. +Default value is 1. +

+
+
amount
+

Set the amount of harmonics added to original signal. +Allowed range is from 0 to 64. +Default value is 1. +

+
+
drive
+

Set the amount of newly created harmonics. +Allowed range is from 0.1 to 10. +Default value is 8.5. +

+
+
blend
+

Set the octave of newly created harmonics. +Allowed range is from -10 to 10. +Default value is 0. +

+
+
freq
+

Set the lower frequency limit of producing harmonics in Hz. +Allowed range is from 2000 to 12000 Hz. +Default is 7500 Hz. +

+
+
ceil
+

Set the upper frequency limit of producing harmonics. +Allowed range is from 9999 to 20000 Hz. +If value is lower than 10000 Hz no limit is applied. +

+
+
listen
+

Mute the original signal and output only added harmonics. +By default is disabled. +

+
+ +

8.21.1 Commands

+ +

This filter supports the all above options as commands. +

+

8.22 afade

+ +

Apply fade-in/out effect to input audio. +

+

A description of the accepted parameters follows. +

+
+
type, t
+

Specify the effect type, can be either in for fade-in, or +out for a fade-out effect. Default is in. +

+
+
start_sample, ss
+

Specify the number of the start sample for starting to apply the fade +effect. Default is 0. +

+
+
nb_samples, ns
+

Specify the number of samples for which the fade effect has to last. At +the end of the fade-in effect the output audio will have the same +volume as the input audio, at the end of the fade-out transition +the output audio will be silence. Default is 44100. +

+
+
start_time, st
+

Specify the start time of the fade effect. Default is 0. +The value must be specified as a time duration; see +(ffmpeg-utils)the Time duration section in the ffmpeg-utils(1) manual +for the accepted syntax. +If set this option is used instead of start_sample. +

+
+
duration, d
+

Specify the duration of the fade effect. See +(ffmpeg-utils)the Time duration section in the ffmpeg-utils(1) manual +for the accepted syntax. +At the end of the fade-in effect the output audio will have the same +volume as the input audio, at the end of the fade-out transition +the output audio will be silence. +By default the duration is determined by nb_samples. +If set this option is used instead of nb_samples. +

+
+
curve
+

Set curve for fade transition. +

+

It accepts the following values: +

+
tri
+

select triangular, linear slope (default) +

+
qsin
+

select quarter of sine wave +

+
hsin
+

select half of sine wave +

+
esin
+

select exponential sine wave +

+
log
+

select logarithmic +

+
ipar
+

select inverted parabola +

+
qua
+

select quadratic +

+
cub
+

select cubic +

+
squ
+

select square root +

+
cbr
+

select cubic root +

+
par
+

select parabola +

+
exp
+

select exponential +

+
iqsin
+

select inverted quarter of sine wave +

+
ihsin
+

select inverted half of sine wave +

+
dese
+

select double-exponential seat +

+
desi
+

select double-exponential sigmoid +

+
losi
+

select logistic sigmoid +

+
sinc
+

select sine cardinal function +

+
isinc
+

select inverted sine cardinal function +

+
quat
+

select quartic +

+
quatr
+

select quartic root +

+
qsin2
+

select squared quarter of sine wave +

+
hsin2
+

select squared half of sine wave +

+
nofade
+

no fade applied +

+
+ +
+
silence
+

Set the initial gain for fade-in or final gain for fade-out. +Default value is 0.0. +

+
+
unity
+

Set the initial gain for fade-out or final gain for fade-in. +Default value is 1.0. +

+
+ +

8.22.1 Commands

+ +

This filter supports the all above options as commands. +

+

8.22.2 Examples

+ +
    +
  • Fade in first 15 seconds of audio: +
    +
    afade=t=in:ss=0:d=15
    +
    + +
  • Fade out last 25 seconds of a 900 seconds audio: +
    +
    afade=t=out:st=875:d=25
    +
    +
+ +

8.23 afftdn

+

Denoise audio samples with FFT. +

+

A description of the accepted parameters follows. +

+
+
noise_reduction, nr
+

Set the noise reduction in dB, allowed range is 0.01 to 97. +Default value is 12 dB. +

+
+
noise_floor, nf
+

Set the noise floor in dB, allowed range is -80 to -20. +Default value is -50 dB. +

+
+
noise_type, nt
+

Set the noise type. +

+

It accepts the following values: +

+
white, w
+

Select white noise. +

+
+
vinyl, v
+

Select vinyl noise. +

+
+
shellac, s
+

Select shellac noise. +

+
+
custom, c
+

Select custom noise, defined in bn option. +

+

Default value is white noise. +

+
+ +
+
band_noise, bn
+

Set custom band noise profile for every one of 15 bands. +Bands are separated by ’ ’ or ’|’. +

+
+
residual_floor, rf
+

Set the residual floor in dB, allowed range is -80 to -20. +Default value is -38 dB. +

+
+
track_noise, tn
+

Enable noise floor tracking. By default is disabled. +With this enabled, noise floor is automatically adjusted. +

+
+
track_residual, tr
+

Enable residual tracking. By default is disabled. +

+
+
output_mode, om
+

Set the output mode. +

+

It accepts the following values: +

+
input, i
+

Pass input unchanged. +

+
+
output, o
+

Pass noise filtered out. +

+
+
noise, n
+

Pass only noise. +

+

Default value is output. +

+
+ +
+
adaptivity, ad
+

Set the adaptivity factor, used how fast to adapt gains adjustments per +each frequency bin. Value 0 enables instant adaptation, while higher values +react much slower. +Allowed range is from 0 to 1. Default value is 0.5. +

+
+
floor_offset, fo
+

Set the noise floor offset factor. This option is used to adjust offset applied to measured +noise floor. It is only effective when noise floor tracking is enabled. +Allowed range is from -2.0 to 2.0. Default value is 1.0. +

+
+
noise_link, nl
+

Set the noise link used for multichannel audio. +

+

It accepts the following values: +

+
none
+

Use unchanged channel’s noise floor. +

+
+
min
+

Use measured min noise floor of all channels. +

+
+
max
+

Use measured max noise floor of all channels. +

+
+
average
+

Use measured average noise floor of all channels. +

+

Default value is min. +

+
+ +
+
band_multiplier, bm
+

Set the band multiplier factor, used how much to spread bands across frequency bins. +Allowed range is from 0.2 to 5. Default value is 1.25. +

+
+
sample_noise, sn
+

Toggle capturing and measurement of noise profile from input audio. +

+

It accepts the following values: +

+
start, begin
+

Start sample noise capture. +

+
+
stop, end
+

Stop sample noise capture and measure new noise band profile. +

+

Default value is none. +

+
+ +
+
gain_smooth, gs
+

Set gain smooth spatial radius, used to smooth gains applied to each frequency bin. +Useful to reduce random music noise artefacts. +Higher values increases smoothing of gains. +Allowed range is from 0 to 50. +Default value is 0. +

+
+ +

8.23.1 Commands

+ +

This filter supports the some above mentioned options as commands. +

+

8.23.2 Examples

+ +
    +
  • Reduce white noise by 10dB, and use previously measured noise floor of -40dB: +
    +
    afftdn=nr=10:nf=-40
    +
    + +
  • Reduce white noise by 10dB, also set initial noise floor to -80dB and enable automatic +tracking of noise floor so noise floor will gradually change during processing: +
    +
    afftdn=nr=10:nf=-80:tn=1
    +
    + +
  • Reduce noise by 20dB, using noise floor of -40dB and using commands to take noise profile +of first 0.4 seconds of input audio: +
    +
    asendcmd=0.0 afftdn sn start,asendcmd=0.4 afftdn sn stop,afftdn=nr=20:nf=-40
    +
    +
+ +

8.24 afftfilt

+

Apply arbitrary expressions to samples in frequency domain. +

+
+
real
+

Set frequency domain real expression for each separate channel separated +by ’|’. Default is "re". +If the number of input channels is greater than the number of +expressions, the last specified expression is used for the remaining +output channels. +

+
+
imag
+

Set frequency domain imaginary expression for each separate channel +separated by ’|’. Default is "im". +

+

Each expression in real and imag can contain the following +constants and functions: +

+
+
sr
+

sample rate +

+
+
b
+

current frequency bin number +

+
+
nb
+

number of available bins +

+
+
ch
+

channel number of the current expression +

+
+
chs
+

number of channels +

+
+
pts
+

current frame pts +

+
+
re
+

current real part of frequency bin of current channel +

+
+
im
+

current imaginary part of frequency bin of current channel +

+
+
real(b, ch)
+

Return the value of real part of frequency bin at location (bin,channel) +

+
+
imag(b, ch)
+

Return the value of imaginary part of frequency bin at location (bin,channel) +

+
+ +
+
win_size
+

Set window size. Allowed range is from 16 to 131072. +Default is 4096 +

+
+
win_func
+

Set window function. +

+

It accepts the following values: +

+
‘rect’
+
‘bartlett’
+
‘hann, hanning’
+
‘hamming’
+
‘blackman’
+
‘welch’
+
‘flattop’
+
‘bharris’
+
‘bnuttall’
+
‘bhann’
+
‘sine’
+
‘nuttall’
+
‘lanczos’
+
‘gauss’
+
‘tukey’
+
‘dolph’
+
‘cauchy’
+
‘parzen’
+
‘poisson’
+
‘bohman’
+
‘kaiser’
+
+

Default is hann. +

+
+
overlap
+

Set window overlap. If set to 1, the recommended overlap for selected +window function will be picked. Default is 0.75. +

+
+ +

8.24.1 Examples

+ +
    +
  • Leave almost only low frequencies in audio: +
    +
    afftfilt="'real=re * (1-clip((b/nb)*b,0,1))':imag='im * (1-clip((b/nb)*b,0,1))'"
    +
    + +
  • Apply robotize effect: +
    +
    afftfilt="real='hypot(re,im)*sin(0)':imag='hypot(re,im)*cos(0)':win_size=512:overlap=0.75"
    +
    + +
  • Apply whisper effect: +
    +
    afftfilt="real='hypot(re,im)*cos((random(0)*2-1)*2*3.14)':imag='hypot(re,im)*sin((random(1)*2-1)*2*3.14)':win_size=128:overlap=0.8"
    +
    + +
  • Apply phase shift: +
    +
    afftfilt="real=re*cos(1)-im*sin(1):imag=re*sin(1)+im*cos(1)"
    +
    +
+ +

8.25 afir

+ +

Apply an arbitrary Finite Impulse Response filter. +

+

This filter is designed for applying long FIR filters, +up to 60 seconds long. +

+

It can be used as component for digital crossover filters, +room equalization, cross talk cancellation, wavefield synthesis, +auralization, ambiophonics, ambisonics and spatialization. +

+

This filter uses the streams higher than first one as FIR coefficients. +If the non-first stream holds a single channel, it will be used +for all input channels in the first stream, otherwise +the number of channels in the non-first stream must be same as +the number of channels in the first stream. +

+

It accepts the following parameters: +

+
+
dry
+

Set dry gain. This sets input gain. +

+
+
wet
+

Set wet gain. This sets final output gain. +

+
+
length
+

Set Impulse Response filter length. Default is 1, which means whole IR is processed. +

+
+
gtype
+

This option is deprecated, and does nothing. +

+
+
irnorm
+

Set norm to be applied to IR coefficients before filtering. +Allowed range is from -1 to 2. +IR coefficients are normalized with calculated vector norm set by this option. +For negative values, no norm is calculated, and IR coefficients are not modified at all. +Default is 1. +

+
+
irlink
+

For multichannel IR if this option is set to true, all IR channels will be +normalized with maximal measured gain of all IR channels coefficients as set by irnorm option. +When disabled, all IR coefficients in each IR channel will be normalized independently. +Default is true. +

+
+
irgain
+

Set gain to be applied to IR coefficients before filtering. +Allowed range is 0 to 1. This gain is applied after any gain applied with irnorm option. +

+
+
irfmt
+

Set format of IR stream. Can be mono or input. +Default is input. +

+
+
maxir
+

Set max allowed Impulse Response filter duration in seconds. Default is 30 seconds. +Allowed range is 0.1 to 60 seconds. +

+
+
response
+

This option is deprecated, and does nothing. +

+
+
channel
+

This option is deprecated, and does nothing. +

+
+
size
+

This option is deprecated, and does nothing. +

+
+
rate
+

This option is deprecated, and does nothing. +

+
+
minp
+

Set minimal partition size used for convolution. Default is 8192. +Allowed range is from 1 to 65536. +Lower values decreases latency at cost of higher CPU usage. +

+
+
maxp
+

Set maximal partition size used for convolution. Default is 8192. +Allowed range is from 8 to 65536. +Lower values may increase CPU usage. +

+
+
nbirs
+

Set number of input impulse responses streams which will be switchable at runtime. +Allowed range is from 1 to 32. Default is 1. +

+
+
ir
+

Set IR stream which will be used for convolution, starting from 0, should always be +lower than supplied value by nbirs option. Default is 0. +This option can be changed at runtime via commands. +

+
+
precision
+

Set which precision to use when processing samples. +

+
+
auto
+

Auto pick internal sample format depending on other filters. +

+
+
float
+

Always use single-floating point precision sample format. +

+
+
double
+

Always use double-floating point precision sample format. +

+
+ +

Default value is auto. +

+
+
irload
+

Set when to load IR stream. Can be init or access. +First one load and prepares all IRs on initialization, second one +once on first access of specific IR. +Default is init. +

+
+ +

8.25.1 Examples

+ +
    +
  • Apply reverb to stream using mono IR file as second input, complete command using ffmpeg: +
    +
    ffmpeg -i input.wav -i middle_tunnel_1way_mono.wav -lavfi afir output.wav
    +
    + +
  • Apply true stereo processing given input stereo stream, and two stereo impulse responses for left and right channel, +the impulse response files are files with names l_ir.wav and r_ir.wav, and setting irnorm option value: +
    +
    "pan=4C|c0=FL|c1=FL|c2=FR|c3=FR[a];amovie=l_ir.wav[LIR];amovie=r_ir.wav[RIR];[LIR][RIR]amerge[ir];[a][ir]afir=irfmt=input:irnorm=1.2,pan=stereo|FL<c0+c2|FR<c1+c3"
    +
    + +
  • Similar to above example, but with irgain explicitly set to estimated value and with irnorm disabled: +
    +
    "pan=4C|c0=FL|c1=FL|c2=FR|c3=FR[a];amovie=l_ir.wav[LIR];amovie=r_ir.wav[RIR];[LIR][RIR]amerge[ir];[a][ir]afir=irfmt=input:irgain=-5dB:irnom=-1,pan=stereo|FL<c0+c2|FR<c1+c3"
    +
    +
+ +

8.26 aformat

+ +

Set output format constraints for the input audio. The framework will +negotiate the most appropriate format to minimize conversions. +

+

It accepts the following parameters: +

+
sample_fmts, f
+

A ’|’-separated list of requested sample formats. +

+
+
sample_rates, r
+

A ’|’-separated list of requested sample rates. +

+
+
channel_layouts, cl
+

A ’|’-separated list of requested channel layouts. +

+

See (ffmpeg-utils)the Channel Layout section in the ffmpeg-utils(1) manual +for the required syntax. +

+
+ +

If a parameter is omitted, all values are allowed. +

+

Force the output to either unsigned 8-bit or signed 16-bit stereo +

+
aformat=sample_fmts=u8|s16:channel_layouts=stereo
+
+ +

8.27 afreqshift

+

Apply frequency shift to input audio samples. +

+

The filter accepts the following options: +

+
+
shift
+

Specify frequency shift. Allowed range is -INT_MAX to INT_MAX. +Default value is 0.0. +

+
+
level
+

Set output gain applied to final output. Allowed range is from 0.0 to 1.0. +Default value is 1.0. +

+
+
order
+

Set filter order used for filtering. Allowed range is from 1 to 16. +Default value is 8. +

+
+ +

8.27.1 Commands

+ +

This filter supports the all above options as commands. +

+

8.28 afwtdn

+

Reduce broadband noise from input samples using Wavelets. +

+

A description of the accepted options follows. +

+
+
sigma
+

Set the noise sigma, allowed range is from 0 to 1. +Default value is 0. +This option controls strength of denoising applied to input samples. +Most useful way to set this option is via decibels, eg. -45dB. +

+
+
levels
+

Set the number of wavelet levels of decomposition. +Allowed range is from 1 to 12. +Default value is 10. +Setting this too low make denoising performance very poor. +

+
+
wavet
+

Set wavelet type for decomposition of input frame. +They are sorted by number of coefficients, from lowest to highest. +More coefficients means worse filtering speed, but overall better quality. +Available wavelets are: +

+
+
‘sym2’
+
‘sym4’
+
‘rbior68’
+
‘deb10’
+
‘sym10’
+
‘coif5’
+
‘bl3’
+
+ +
+
percent
+

Set percent of full denoising. Allowed range is from 0 to 100 percent. +Default value is 85 percent or partial denoising. +

+
+
profile
+

If enabled, first input frame will be used as noise profile. +If first frame samples contain non-noise performance will be very poor. +

+
+
adaptive
+

If enabled, input frames are analyzed for presence of noise. +If noise is detected with high possibility then input frame profile will be +used for processing following frames, until new noise frame is detected. +

+
+
samples
+

Set size of single frame in number of samples. Allowed range is from 512 to +65536. Default frame size is 8192 samples. +

+
+
softness
+

Set softness applied inside thresholding function. Allowed range is from 0 to +10. Default softness is 1. +

+
+ +

8.28.1 Commands

+ +

This filter supports the all above options as commands. +

+

8.29 agate

+ +

A gate is mainly used to reduce lower parts of a signal. This kind of signal +processing reduces disturbing noise between useful signals. +

+

Gating is done by detecting the volume below a chosen level threshold +and dividing it by the factor set with ratio. The bottom of the noise +floor is set via range. Because an exact manipulation of the signal +would cause distortion of the waveform the reduction can be levelled over +time. This is done by setting attack and release. +

+

attack determines how long the signal has to fall below the threshold +before any reduction will occur and release sets the time the signal +has to rise above the threshold to reduce the reduction again. +Shorter signals than the chosen attack time will be left untouched. +

+
+
level_in
+

Set input level before filtering. +Default is 1. Allowed range is from 0.015625 to 64. +

+
+
mode
+

Set the mode of operation. Can be upward or downward. +Default is downward. If set to upward mode, higher parts of signal +will be amplified, expanding dynamic range in upward direction. +Otherwise, in case of downward lower parts of signal will be reduced. +

+
+
range
+

Set the level of gain reduction when the signal is below the threshold. +Default is 0.06125. Allowed range is from 0 to 1. +Setting this to 0 disables reduction and then filter behaves like expander. +

+
+
threshold
+

If a signal rises above this level the gain reduction is released. +Default is 0.125. Allowed range is from 0 to 1. +

+
+
ratio
+

Set a ratio by which the signal is reduced. +Default is 2. Allowed range is from 1 to 9000. +

+
+
attack
+

Amount of milliseconds the signal has to rise above the threshold before gain +reduction stops. +Default is 20 milliseconds. Allowed range is from 0.01 to 9000. +

+
+
release
+

Amount of milliseconds the signal has to fall below the threshold before the +reduction is increased again. Default is 250 milliseconds. +Allowed range is from 0.01 to 9000. +

+
+
makeup
+

Set amount of amplification of signal after processing. +Default is 1. Allowed range is from 1 to 64. +

+
+
knee
+

Curve the sharp knee around the threshold to enter gain reduction more softly. +Default is 2.828427125. Allowed range is from 1 to 8. +

+
+
detection
+

Choose if exact signal should be taken for detection or an RMS like one. +Default is rms. Can be peak or rms. +

+
+
link
+

Choose if the average level between all channels or the louder channel affects +the reduction. +Default is average. Can be average or maximum. +

+
+ +

8.29.1 Commands

+ +

This filter supports the all above options as commands. +

+

8.30 aiir

+ +

Apply an arbitrary Infinite Impulse Response filter. +

+

It accepts the following parameters: +

+
+
zeros, z
+

Set B/numerator/zeros/reflection coefficients. +

+
+
poles, p
+

Set A/denominator/poles/ladder coefficients. +

+
+
gains, k
+

Set channels gains. +

+
+
dry_gain
+

Set input gain. +

+
+
wet_gain
+

Set output gain. +

+
+
format, f
+

Set coefficients format. +

+
+
‘ll’
+

lattice-ladder function +

+
‘sf’
+

analog transfer function +

+
‘tf’
+

digital transfer function +

+
‘zp’
+

Z-plane zeros/poles, cartesian (default) +

+
‘pr’
+

Z-plane zeros/poles, polar radians +

+
‘pd’
+

Z-plane zeros/poles, polar degrees +

+
‘sp’
+

S-plane zeros/poles +

+
+ +
+
process, r
+

Set type of processing. +

+
+
‘d’
+

direct processing +

+
‘s’
+

serial processing +

+
‘p’
+

parallel processing +

+
+ +
+
precision, e
+

Set filtering precision. +

+
+
‘dbl’
+

double-precision floating-point (default) +

+
‘flt’
+

single-precision floating-point +

+
‘i32’
+

32-bit integers +

+
‘i16’
+

16-bit integers +

+
+ +
+
normalize, n
+

Normalize filter coefficients, by default is enabled. +Enabling it will normalize magnitude response at DC to 0dB. +

+
+
mix
+

How much to use filtered signal in output. Default is 1. +Range is between 0 and 1. +

+
+
response
+

Show IR frequency response, magnitude(magenta), phase(green) and group delay(yellow) in additional video stream. +By default it is disabled. +

+
+
channel
+

Set for which IR channel to display frequency response. By default is first channel +displayed. This option is used only when response is enabled. +

+
+
size
+

Set video stream size. This option is used only when response is enabled. +

+
+ +

Coefficients in tf and sf format are separated by spaces and are in ascending +order. +

+

Coefficients in zp format are separated by spaces and order of coefficients +doesn’t matter. Coefficients in zp format are complex numbers with i +imaginary unit. +

+

Different coefficients and gains can be provided for every channel, in such case +use ’|’ to separate coefficients or gains. Last provided coefficients will be +used for all remaining channels. +

+

8.30.1 Examples

+ +
    +
  • Apply 2 pole elliptic notch at around 5000Hz for 48000 Hz sample rate: +
    +
    aiir=k=1:z=7.957584807809675810E-1 -2.575128568908332300 3.674839853930788710 -2.57512875289799137 7.957586296317130880E-1:p=1 -2.86950072432325953 3.63022088054647218 -2.28075678147272232 6.361362326477423500E-1:f=tf:r=d
    +
    + +
  • Same as above but in zp format: +
    +
    aiir=k=0.79575848078096756:z=0.80918701+0.58773007i 0.80918701-0.58773007i 0.80884700+0.58784055i 0.80884700-0.58784055i:p=0.63892345+0.59951235i 0.63892345-0.59951235i 0.79582691+0.44198673i 0.79582691-0.44198673i:f=zp:r=s
    +
    + +
  • Apply 3-rd order analog normalized Butterworth low-pass filter, using analog transfer function format: +
    +
    aiir=z=1.3057 0 0 0:p=1.3057 2.3892 2.1860 1:f=sf:r=d
    +
    +
+ +

8.31 alimiter

+ +

The limiter prevents an input signal from rising over a desired threshold. +This limiter uses lookahead technology to prevent your signal from distorting. +It means that there is a small delay after the signal is processed. Keep in mind +that the delay it produces is the attack time you set. +

+

The filter accepts the following options: +

+
+
level_in
+

Set input gain. Default is 1. +

+
+
level_out
+

Set output gain. Default is 1. +

+
+
limit
+

Don’t let signals above this level pass the limiter. Default is 1. +

+
+
attack
+

The limiter will reach its attenuation level in this amount of time in +milliseconds. Default is 5 milliseconds. +

+
+
release
+

Come back from limiting to attenuation 1.0 in this amount of milliseconds. +Default is 50 milliseconds. +

+
+
asc
+

When gain reduction is always needed ASC takes care of releasing to an +average reduction level rather than reaching a reduction of 0 in the release +time. +

+
+
asc_level
+

Select how much the release time is affected by ASC, 0 means nearly no changes +in release time while 1 produces higher release times. +

+
+
level
+

Auto level output signal. Default is enabled. +This normalizes audio back to 0dB if enabled. +

+
+
latency
+

Compensate the delay introduced by using the lookahead buffer set with attack +parameter. Also flush the valid audio data in the lookahead buffer when the +stream hits EOF. +

+
+ +

Depending on picked setting it is recommended to upsample input 2x or 4x times +with aresample before applying this filter. +

+

8.32 allpass

+ +

Apply a two-pole all-pass filter with central frequency (in Hz) +frequency, and filter-width width. +An all-pass filter changes the audio’s frequency to phase relationship +without changing its frequency to amplitude relationship. +

+

The filter accepts the following options: +

+
+
frequency, f
+

Set frequency in Hz. +

+
+
width_type, t
+

Set method to specify band-width of filter. +

+
h
+

Hz +

+
q
+

Q-Factor +

+
o
+

octave +

+
s
+

slope +

+
k
+

kHz +

+
+ +
+
width, w
+

Specify the band-width of a filter in width_type units. +

+
+
mix, m
+

How much to use filtered signal in output. Default is 1. +Range is between 0 and 1. +

+
+
channels, c
+

Specify which channels to filter, by default all available are filtered. +

+
+
normalize, n
+

Normalize biquad coefficients, by default is disabled. +Enabling it will normalize magnitude response at DC to 0dB. +

+
+
order, o
+

Set the filter order, can be 1 or 2. Default is 2. +

+
+
transform, a
+

Set transform type of IIR filter. +

+
di
+
dii
+
tdi
+
tdii
+
latt
+
svf
+
zdf
+
+ +
+
precision, r
+

Set precision of filtering. +

+
auto
+

Pick automatic sample format depending on surround filters. +

+
s16
+

Always use signed 16-bit. +

+
s32
+

Always use signed 32-bit. +

+
f32
+

Always use float 32-bit. +

+
f64
+

Always use float 64-bit. +

+
+
+
+ +

8.32.1 Commands

+ +

This filter supports the following commands: +

+
frequency, f
+

Change allpass frequency. +Syntax for the command is : "frequency" +

+
+
width_type, t
+

Change allpass width_type. +Syntax for the command is : "width_type" +

+
+
width, w
+

Change allpass width. +Syntax for the command is : "width" +

+
+
mix, m
+

Change allpass mix. +Syntax for the command is : "mix" +

+
+ +

8.33 aloop

+ +

Loop audio samples. +

+

The filter accepts the following options: +

+
+
loop
+

Set the number of loops. Setting this value to -1 will result in infinite loops. +Default is 0. +

+
+
size
+

Set maximal number of samples. Default is 0. +

+
+
start
+

Set first sample of loop. Default is 0. +

+
+
time
+

Set the time of loop start in seconds. +Only used if option named start is set to -1. +

+
+ +

8.34 amerge

+ +

Merge two or more audio streams into a single multi-channel stream. +

+

All inputs must have the same sample rate, and format. +

+

If inputs do not have the same duration, the output will stop with the +shortest. +

+

The filter accepts the following options: +

+
+
inputs
+

Set the number of inputs. Default is 2. +

+
+
layout_mode
+
+

This option controls how the output channel layout is determined and if the +audio channels are reordered during merge. +

+
+
legacy
+
+

This is the mode how the filter behaved historically so it is the default. +

+

If the channel layouts of the inputs are known and disjoint, and therefore +compatible, the channel layout of the output will be set accordingly and the +channels will be reordered as necessary. If the channel layouts of the inputs +are not disjoint, some of them are unknown, or they are using special channel +layouts, such as ambisonics, the output will have all the channels of the first +input then all the channels of the second input, in that order, and the channel +layout of the output will be the default value corresponding to the total +number of channels. +

+

For example, if the first input is in 2.1 (FL+FR+LF) and the second input +is FC+BL+BR, then the output will be in 5.1, with the channels in the +following order: a1, a2, b1, a3, b2, b3 (a1 is the first channel of the +first input, b1 is the first channel of the second input). +

+

On the other hand, if both input are in stereo, the output channels will be +in the default order: a1, a2, b1, b2, and the channel layout will be +arbitrarily set to 4.0, which may or may not be the expected value. +

+
+
reset
+

This mode ignores the input channel layouts and does no channel reordering. +The output will have all the channels of the first input, then all the channels +of the second input, in that order, and so on. +

+

The output channel layout will only specify the total channel count. +

+
+
normal
+

This mode keeps channel name and designation information from the input +channels and does no channel reordering. The output will have all the channels +of the first input, then all the channels of the second input, in that order, +and so on. +

+
+
+ +
+
+ +

8.34.1 Examples

+ +
    +
  • Merge two mono files into a stereo stream: +
    +
    amovie=left.wav [l] ; amovie=right.mp3 [r] ; [l] [r] amerge
    +
    + +
  • Multiple merges assuming 1 video stream and 6 audio streams in input.mkv: +
    +
    ffmpeg -i input.mkv -filter_complex "[0:1][0:2][0:3][0:4][0:5][0:6] amerge=inputs=6" -c:a pcm_s16le output.mkv
    +
    +
+ +

8.35 amix

+ +

Mixes multiple audio inputs into a single output. +

+

Note that this filter only supports float samples (the amerge +and pan audio filters support many formats). If the amix +input has integer samples then aresample will be automatically +inserted to perform the conversion to float samples. +

+

It accepts the following parameters: +

+
inputs
+

The number of inputs. If unspecified, it defaults to 2. +

+
+
duration
+

How to determine the end-of-stream. +

+
longest
+

The duration of the longest input. (default) +

+
+
shortest
+

The duration of the shortest input. +

+
+
first
+

The duration of the first input. +

+
+
+ +
+
dropout_transition
+

The transition time, in seconds, for volume renormalization when an input +stream ends. The default value is 2 seconds. +

+
+
weights
+

Specify weight of each input audio stream as a sequence of numbers separated +by a space. If fewer weights are specified compared to number of inputs, the +last weight is assigned to the remaining inputs. +Default weight for each input is 1. +

+
+
normalize
+

Always scale inputs instead of only doing summation of samples. +Beware of heavy clipping if inputs are not normalized prior or after filtering +by this filter if this option is disabled. By default is enabled. +

+
+ +

8.35.1 Examples

+ +
    +
  • This will mix 3 input audio streams to a single output with the same duration as the +first input and a dropout transition time of 3 seconds: +
    +
    ffmpeg -i INPUT1 -i INPUT2 -i INPUT3 -filter_complex amix=inputs=3:duration=first:dropout_transition=3 OUTPUT
    +
    + +
  • This will mix one vocal and one music input audio stream to a single output with the same duration as the +longest input. The music will have quarter the weight as the vocals, and the inputs are not normalized: +
    +
    ffmpeg -i VOCALS -i MUSIC -filter_complex amix=inputs=2:duration=longest:dropout_transition=0:weights="1 0.25":normalize=0 OUTPUT
    +
    + +
+ +

8.35.2 Commands

+ +

This filter supports the following commands: +

+
weights
+
normalize
+

Syntax is same as option with same name. +

+
+ +

8.36 amultiply

+ +

Multiply first audio stream with second audio stream and store result +in output audio stream. Multiplication is done by multiplying each +sample from first stream with sample at same position from second stream. +

+

With this element-wise multiplication one can create amplitude fades and +amplitude modulations. +

+

8.37 anequalizer

+ +

High-order parametric multiband equalizer for each channel. +

+

It accepts the following parameters: +

+
params
+
+

This option string is in format: +"cchn f=cf w=w g=g t=f | ..." +Each equalizer band is separated by ’|’. +

+
+
chn
+

Set channel number to which equalization will be applied. +If input doesn’t have that channel the entry is ignored. +

+
+
f
+

Set central frequency for band. +If input doesn’t have that frequency the entry is ignored. +

+
+
w
+

Set band width in Hertz. +

+
+
g
+

Set band gain in dB. +

+
+
t
+

Set filter type for band, optional, can be: +

+
+
‘0’
+

Butterworth, this is default. +

+
+
‘1’
+

Chebyshev type 1. +

+
+
‘2’
+

Chebyshev type 2. +

+
+
+
+ +
+
curves
+

With this option activated frequency response of anequalizer is displayed +in video stream. +

+
+
size
+

Set video stream size. Only useful if curves option is activated. +

+
+
mgain
+

Set max gain that will be displayed. Only useful if curves option is activated. +Setting this to a reasonable value makes it possible to display gain which is derived from +neighbour bands which are too close to each other and thus produce higher gain +when both are activated. +

+
+
fscale
+

Set frequency scale used to draw frequency response in video output. +Can be linear or logarithmic. Default is logarithmic. +

+
+
colors
+

Set color for each channel curve which is going to be displayed in video stream. +This is list of color names separated by space or by ’|’. +Unrecognised or missing colors will be replaced by white color. +

+
+ +

8.37.1 Examples

+ +
    +
  • Lower gain by 10 of central frequency 200Hz and width 100 Hz +for first 2 channels using Chebyshev type 1 filter: +
    +
    anequalizer=c0 f=200 w=100 g=-10 t=1|c1 f=200 w=100 g=-10 t=1
    +
    +
+ +

8.37.2 Commands

+ +

This filter supports the following commands: +

+
change
+

Alter existing filter parameters. +Syntax for the commands is : "fN|f=freq|w=width|g=gain" +

+

fN is existing filter number, starting from 0, if no such filter is available +error is returned. +freq set new frequency parameter. +width set new width parameter in Hertz. +gain set new gain parameter in dB. +

+

Full filter invocation with asendcmd may look like this: +asendcmd=c=’4.0 anequalizer change 0|f=200|w=50|g=1’,anequalizer=... +

+
+ +

8.38 anlmdn

+ +

Reduce broadband noise in audio samples using Non-Local Means algorithm. +

+

Each sample is adjusted by looking for other samples with similar contexts. This +context similarity is defined by comparing their surrounding patches of size +p. Patches are searched in an area of r around the sample. +

+

The filter accepts the following options: +

+
+
strength, s
+

Set denoising strength. Allowed range is from 0.00001 to 10000. Default value is 0.00001. +

+
+
patch, p
+

Set patch radius duration. Allowed range is from 1 to 100 milliseconds. +Default value is 2 milliseconds. +

+
+
research, r
+

Set research radius duration. Allowed range is from 2 to 300 milliseconds. +Default value is 6 milliseconds. +

+
+
output, o
+

Set the output mode. +

+

It accepts the following values: +

+
i
+

Pass input unchanged. +

+
+
o
+

Pass noise filtered out. +

+
+
n
+

Pass only noise. +

+

Default value is o. +

+
+ +
+
smooth, m
+

Set smooth factor. Default value is 11. Allowed range is from 1 to 1000. +

+
+ +

8.38.1 Commands

+ +

This filter supports the all above options as commands. +

+

8.39 anlmf, anlms

+

Apply Normalized Least-Mean-(Squares|Fourth) algorithm to the first audio stream using the second audio stream. +

+

This adaptive filter is used to mimic a desired filter by finding the filter coefficients that +relate to producing the least mean square of the error signal (difference between the desired, +2nd input audio stream and the actual signal, the 1st input audio stream). +

+

A description of the accepted options follows. +

+
+
order
+

Set filter order. +

+
+
mu
+

Set filter mu. +

+
+
eps
+

Set the filter eps. +

+
+
leakage
+

Set the filter leakage. +

+
+
out_mode
+

It accepts the following values: +

+
i
+

Pass the 1st input. +

+
+
d
+

Pass the 2nd input. +

+
+
o
+

Pass difference between desired, 2nd input and error signal estimate. +

+
+
n
+

Pass difference between input, 1st input and error signal estimate. +

+
+
e
+

Pass error signal estimated samples. +

+

Default value is o. +

+
+ +
+
precision
+

Set which precision to use when processing samples. +

+
+
auto
+

Auto pick internal sample format depending on other filters. +

+
+
float
+

Always use single-floating point precision sample format. +

+
+
double
+

Always use double-floating point precision sample format. +

+
+
+
+ +

8.39.1 Examples

+ +
    +
  • One of many usages of this filter is noise reduction, input audio is filtered +with same samples that are delayed by fixed amount, one such example for stereo audio is: +
    +
    asplit[a][b],[a]adelay=32S|32S[a],[b][a]anlms=order=128:leakage=0.0005:mu=.5:out_mode=o
    +
    +
+ +

8.39.2 Commands

+ +

This filter supports the same commands as options, excluding option order. +

+

8.40 anull

+ +

Pass the audio source unchanged to the output. +

+

8.41 apad

+ +

Pad the end of an audio stream with silence. +

+

This can be used together with ffmpeg -shortest to +extend audio streams to the same length as the video stream. +

+

A description of the accepted options follows. +

+
+
packet_size
+

Set silence packet size. Default value is 4096. +

+
+
pad_len
+

Set the number of samples of silence to add to the end. After the +value is reached, the stream is terminated. This option is mutually +exclusive with whole_len. +

+
+
whole_len
+

Set the minimum total number of samples in the output audio stream. If +the value is longer than the input audio length, silence is added to +the end, until the value is reached. This option is mutually exclusive +with pad_len. +

+
+
pad_dur
+

Specify the duration of samples of silence to add. See +(ffmpeg-utils)the Time duration section in the ffmpeg-utils(1) manual +for the accepted syntax. Used only if set to non-negative value. +

+
+
whole_dur
+

Specify the minimum total duration in the output audio stream. See +(ffmpeg-utils)the Time duration section in the ffmpeg-utils(1) manual +for the accepted syntax. Used only if set to non-negative value. If the value is longer than +the input audio length, silence is added to the end, until the value is reached. +This option is mutually exclusive with pad_dur +

+
+ +

If neither the pad_len nor the whole_len nor pad_dur +nor whole_dur option is set, the filter will add silence to the end of +the input stream indefinitely. +

+

Note that for ffmpeg 4.4 and earlier a zero pad_dur or +whole_dur also caused the filter to add silence indefinitely. +

+

8.41.1 Examples

+ +
    +
  • Add 1024 samples of silence to the end of the input: +
    +
    apad=pad_len=1024
    +
    + +
  • Make sure the audio output will contain at least 10000 samples, pad +the input with silence if required: +
    +
    apad=whole_len=10000
    +
    + +
  • Use ffmpeg to pad the audio input with silence, so that the +video stream will always result the shortest and will be converted +until the end in the output file when using the shortest +option: +
    +
    ffmpeg -i VIDEO -i AUDIO -filter_complex "[1:0]apad" -shortest OUTPUT
    +
    +
+ +

8.42 aphaser

+

Add a phasing effect to the input audio. +

+

A phaser filter creates series of peaks and troughs in the frequency spectrum. +The position of the peaks and troughs are modulated so that they vary over time, creating a sweeping effect. +

+

A description of the accepted parameters follows. +

+
+
in_gain
+

Set input gain. Default is 0.4. +

+
+
out_gain
+

Set output gain. Default is 0.74 +

+
+
delay
+

Set delay in milliseconds. Default is 3.0. +

+
+
decay
+

Set decay. Default is 0.4. +

+
+
speed
+

Set modulation speed in Hz. Default is 0.5. +

+
+
type
+

Set modulation type. Default is triangular. +

+

It accepts the following values: +

+
‘triangular, t’
+
‘sinusoidal, s’
+
+
+
+ +

8.43 aphaseshift

+

Apply phase shift to input audio samples. +

+

The filter accepts the following options: +

+
+
shift
+

Specify phase shift. Allowed range is from -1.0 to 1.0. +Default value is 0.0. +

+
+
level
+

Set output gain applied to final output. Allowed range is from 0.0 to 1.0. +Default value is 1.0. +

+
+
order
+

Set filter order used for filtering. Allowed range is from 1 to 16. +Default value is 8. +

+
+ +

8.43.1 Commands

+ +

This filter supports the all above options as commands. +

+

8.44 apsnr

+

Measure Audio Peak Signal-to-Noise Ratio. +

+

This filter takes two audio streams for input, and outputs first +audio stream. +Results are in dB per channel at end of either input. +

+

8.45 apsyclip

+

Apply Psychoacoustic clipper to input audio stream. +

+

The filter accepts the following options: +

+
+
level_in
+

Set input gain. By default it is 1. Range is [0.015625 - 64]. +

+
+
level_out
+

Set output gain. By default it is 1. Range is [0.015625 - 64]. +

+
+
clip
+

Set the clipping start value. Default value is 0dBFS or 1. +

+
+
diff
+

Output only difference samples, useful to hear introduced distortions. +By default is disabled. +

+
+
adaptive
+

Set strength of adaptive distortion applied. Default value is 0.5. +Allowed range is from 0 to 1. +

+
+
iterations
+

Set number of iterations of psychoacoustic clipper. +Allowed range is from 1 to 20. Default value is 10. +

+
+
level
+

Auto level output signal. Default is disabled. +This normalizes audio back to 0dBFS if enabled. +

+
+ +

8.45.1 Commands

+ +

This filter supports the all above options as commands. +

+

8.46 apulsator

+ +

Audio pulsator is something between an autopanner and a tremolo. +But it can produce funny stereo effects as well. Pulsator changes the volume +of the left and right channel based on a LFO (low frequency oscillator) with +different waveforms and shifted phases. +This filter have the ability to define an offset between left and right +channel. An offset of 0 means that both LFO shapes match each other. +The left and right channel are altered equally - a conventional tremolo. +An offset of 50% means that the shape of the right channel is exactly shifted +in phase (or moved backwards about half of the frequency) - pulsator acts as +an autopanner. At 1 both curves match again. Every setting in between moves the +phase shift gapless between all stages and produces some "bypassing" sounds with +sine and triangle waveforms. The more you set the offset near 1 (starting from +the 0.5) the faster the signal passes from the left to the right speaker. +

+

The filter accepts the following options: +

+
+
level_in
+

Set input gain. By default it is 1. Range is [0.015625 - 64]. +

+
+
level_out
+

Set output gain. By default it is 1. Range is [0.015625 - 64]. +

+
+
mode
+

Set waveform shape the LFO will use. Can be one of: sine, triangle, square, +sawup or sawdown. Default is sine. +

+
+
amount
+

Set modulation. Define how much of original signal is affected by the LFO. +

+
+
offset_l
+

Set left channel offset. Default is 0. Allowed range is [0 - 1]. +

+
+
offset_r
+

Set right channel offset. Default is 0.5. Allowed range is [0 - 1]. +

+
+
width
+

Set pulse width. Default is 1. Allowed range is [0 - 2]. +

+
+
timing
+

Set possible timing mode. Can be one of: bpm, ms or hz. Default is hz. +

+
+
bpm
+

Set bpm. Default is 120. Allowed range is [30 - 300]. Only used if timing +is set to bpm. +

+
+
ms
+

Set ms. Default is 500. Allowed range is [10 - 2000]. Only used if timing +is set to ms. +

+
+
hz
+

Set frequency in Hz. Default is 2. Allowed range is [0.01 - 100]. Only used +if timing is set to hz. +

+
+ +

8.47 aresample

+ +

Resample the input audio to the specified parameters, using the +libswresample library. If none are specified then the filter will +automatically convert between its input and output. +

+

This filter is also able to stretch/squeeze the audio data to make it match +the timestamps or to inject silence / cut out audio to make it match the +timestamps, do a combination of both or do neither. +

+

The filter accepts the syntax +[sample_rate:]resampler_options, where sample_rate +expresses a sample rate and resampler_options is a list of +key=value pairs, separated by ":". See the +(ffmpeg-resampler)"Resampler Options" section in the +ffmpeg-resampler(1) manual +for the complete list of supported options. +

+

8.47.1 Examples

+ +
    +
  • Resample the input audio to 44100Hz: +
    +
    aresample=44100
    +
    + +
  • Stretch/squeeze samples to the given timestamps, with a maximum of 1000 +samples per second compensation: +
    +
    aresample=async=1000
    +
    +
+ +

8.48 areverse

+ +

Reverse an audio clip. +

+

Warning: This filter requires memory to buffer the entire clip, so trimming +is suggested. +

+

8.48.1 Examples

+ +
    +
  • Take the first 5 seconds of a clip, and reverse it. +
    +
    atrim=end=5,areverse
    +
    +
+ +

8.49 arls

+

Apply Recursive Least Squares algorithm to the first audio stream using the second audio stream. +

+

This adaptive filter is used to mimic a desired filter by recursively finding the filter coefficients that +relate to producing the minimal weighted linear least squares cost function of the error signal (difference +between the desired, 2nd input audio stream and the actual signal, the 1st input audio stream). +

+

A description of the accepted options follows. +

+
+
order
+

Set the filter order. +

+
+
lambda
+

Set the forgetting factor. +

+
+
delta
+

Set the coefficient to initialize internal covariance matrix. +

+
+
out_mode
+

Set the filter output samples. It accepts the following values: +

+
i
+

Pass the 1st input. +

+
+
d
+

Pass the 2nd input. +

+
+
o
+

Pass difference between desired, 2nd input and error signal estimate. +

+
+
n
+

Pass difference between input, 1st input and error signal estimate. +

+
+
e
+

Pass error signal estimated samples. +

+

Default value is o. +

+
+ +
+
precision
+

Set which precision to use when processing samples. +

+
+
auto
+

Auto pick internal sample format depending on other filters. +

+
+
float
+

Always use single-floating point precision sample format. +

+
+
double
+

Always use double-floating point precision sample format. +

+
+
+
+ +

8.50 arnndn

+ +

Reduce noise from speech using Recurrent Neural Networks. +

+

This filter accepts the following options: +

+
+
model, m
+

Set train model file to load. This option is always required. +

+
+
mix
+

Set how much to mix filtered samples into final output. +Allowed range is from -1 to 1. Default value is 1. +Negative values are special, they set how much to keep filtered noise +in the final filter output. Set this option to -1 to hear actual +noise removed from input signal. +

+
+ +

8.50.1 Commands

+ +

This filter supports the all above options as commands. +

+

8.51 asdr

+

Measure Audio Signal-to-Distortion Ratio. +

+

This filter takes two audio streams for input, and outputs first +audio stream. +Results are in dB per channel at end of either input. +

+

8.52 asetnsamples

+ +

Set the number of samples per each output audio frame. +

+

The last output packet may contain a different number of samples, as +the filter will flush all the remaining samples when the input audio +signals its end. +

+

The filter accepts the following options: +

+
+
nb_out_samples, n
+

Set the number of frames per each output audio frame. The number is +intended as the number of samples per each channel. +Default value is 1024. +

+
+
pad, p
+

If set to 1, the filter will pad the last audio frame with zeroes, so +that the last frame will contain the same number of samples as the +previous ones. Default value is 1. +

+
+ +

For example, to set the number of per-frame samples to 1234 and +disable padding for the last frame, use: +

+
asetnsamples=n=1234:p=0
+
+ +

8.53 asetrate

+ +

Set the sample rate without altering the PCM data. +This will result in a change of speed and pitch. +

+

The filter accepts the following options: +

+
+
sample_rate, r
+

Set the output sample rate. Default is 44100 Hz. +

+
+ +

8.54 ashowinfo

+ +

Show a line containing various information for each input audio frame. +The input audio is not modified. +

+

The shown line contains a sequence of key/value pairs of the form +key:value. +

+

The following values are shown in the output: +

+
+
n
+

The (sequential) number of the input frame, starting from 0. +

+
+
pts
+

The presentation timestamp of the input frame, in time base units; the time base +depends on the filter input pad, and is usually 1/sample_rate. +

+
+
pts_time
+

The presentation timestamp of the input frame in seconds. +

+
+
fmt
+

The sample format. +

+
+
chlayout
+

The channel layout. +

+
+
rate
+

The sample rate for the audio frame. +

+
+
nb_samples
+

The number of samples (per channel) in the frame. +

+
+
checksum
+

The Adler-32 checksum (printed in hexadecimal) of the audio data. For planar +audio, the data is treated as if all the planes were concatenated. +

+
+
plane_checksums
+

A list of Adler-32 checksums for each data plane. +

+
+ +

8.55 asisdr

+

Measure Audio Scaled-Invariant Signal-to-Distortion Ratio. +

+

This filter takes two audio streams for input, and outputs first +audio stream. +Results are in dB per channel at end of either input. +

+

8.56 asoftclip

+

Apply audio soft clipping. +

+

Soft clipping is a type of distortion effect where the amplitude of a signal is saturated +along a smooth curve, rather than the abrupt shape of hard-clipping. +

+

This filter accepts the following options: +

+
+
type
+

Set type of soft-clipping. +

+

It accepts the following values: +

+
hard
+
tanh
+
atan
+
cubic
+
exp
+
alg
+
quintic
+
sin
+
erf
+
+ +
+
threshold
+

Set threshold from where to start clipping. Default value is 0dB or 1. +

+
+
output
+

Set gain applied to output. Default value is 0dB or 1. +

+
+
param
+

Set additional parameter which controls sigmoid function. +

+
+
oversample
+

Set oversampling factor. +

+
+ +

8.56.1 Commands

+ +

This filter supports the all above options as commands. +

+

8.57 aspectralstats

+ +

Display frequency domain statistical information about the audio channels. +Statistics are calculated and stored as metadata for each audio channel and for each audio frame. +

+

It accepts the following option: +

+
win_size
+

Set the window length in samples. Default value is 2048. +Allowed range is from 32 to 65536. +

+
+
win_func
+

Set window function. +

+

It accepts the following values: +

+
‘rect’
+
‘bartlett’
+
‘hann, hanning’
+
‘hamming’
+
‘blackman’
+
‘welch’
+
‘flattop’
+
‘bharris’
+
‘bnuttall’
+
‘bhann’
+
‘sine’
+
‘nuttall’
+
‘lanczos’
+
‘gauss’
+
‘tukey’
+
‘dolph’
+
‘cauchy’
+
‘parzen’
+
‘poisson’
+
‘bohman’
+
‘kaiser’
+
+

Default is hann. +

+
+
overlap
+

Set window overlap. Allowed range is from 0 +to 1. Default value is 0.5. +

+
+
measure
+

Select the parameters which are measured. The metadata keys can +be used as flags, default is all which measures everything. +none disables all measurement. +

+
+ +

A list of each metadata key follows: +

+
+
mean
+
variance
+
centroid
+
spread
+
skewness
+
kurtosis
+
entropy
+
flatness
+
crest
+
flux
+
slope
+
decrease
+
rolloff
+
+ +

8.58 asr

+

Automatic Speech Recognition +

+

This filter uses PocketSphinx for speech recognition. To enable +compilation of this filter, you need to configure FFmpeg with +--enable-pocketsphinx. +

+

It accepts the following options: +

+
+
rate
+

Set sampling rate of input audio. Defaults is 16000. +This need to match speech models, otherwise one will get poor results. +

+
+
hmm
+

Set dictionary containing acoustic model files. +

+
+
dict
+

Set pronunciation dictionary. +

+
+
lm
+

Set language model file. +

+
+
lmctl
+

Set language model set. +

+
+
lmname
+

Set which language model to use. +

+
+
logfn
+

Set output for log messages. +

+
+ +

The filter exports recognized speech as the frame metadata lavfi.asr.text. +

+

8.59 astats

+ +

Display time domain statistical information about the audio channels. +Statistics are calculated and displayed for each audio channel and, +where applicable, an overall figure is also given. +

+

It accepts the following option: +

+
length
+

Short window length in seconds, used for peak and through RMS measurement. +Default is 0.05 (50 milliseconds). Allowed range is [0 - 10]. +

+
+
metadata
+
+

Set metadata injection. All the metadata keys are prefixed with lavfi.astats.X, +where X is channel number starting from 1 or string Overall. Default is +disabled. +

+

Available keys for each channel are: +Bit_depth +Crest_factor +DC_offset +Dynamic_range +Entropy +Flat_factor +Max_difference +Max_level +Mean_difference +Min_difference +Min_level +Noise_floor +Noise_floor_count +Number_of_Infs +Number_of_NaNs +Number_of_denormals +Peak_count +Abs_Peak_count +Peak_level +RMS_difference +RMS_peak +RMS_trough +Zero_crossings +Zero_crossings_rate +

+

and for Overall: +Bit_depth +DC_offset +Entropy +Flat_factor +Max_difference +Max_level +Mean_difference +Min_difference +Min_level +Noise_floor +Noise_floor_count +Number_of_Infs +Number_of_NaNs +Number_of_denormals +Number_of_samples +Peak_count +Abs_Peak_count +Peak_level +RMS_difference +RMS_level +RMS_peak +RMS_trough +

+

For example, a full key looks like lavfi.astats.1.DC_offset or +lavfi.astats.Overall.Peak_count. +

+

Read below for the description of the keys. +

+
+
reset
+

Set the number of frames over which cumulative stats are calculated before +being reset. Default is disabled. +

+
+
measure_perchannel
+

Select the parameters which are measured per channel. The metadata keys can +be used as flags, default is all which measures everything. +none disables all per channel measurement. +

+
+
measure_overall
+

Select the parameters which are measured overall. The metadata keys can +be used as flags, default is all which measures everything. +none disables all overall measurement. +

+
+
+ +

A description of the measure keys follow: +

+
+
none
+

no measures +

+
+
all
+

all measures +

+
+
Bit_depth
+

overall bit depth of audio, i.e. number of bits used for each sample +

+
+
Crest_factor
+

standard ratio of peak to RMS level (note: not in dB) +

+
+
DC_offset
+

mean amplitude displacement from zero +

+
+
Dynamic_range
+

measured dynamic range of audio in dB +

+
+
Entropy
+

entropy measured across whole audio, entropy of value near 1.0 is typically measured for white noise +

+
+
Flat_factor
+

flatness (i.e. consecutive samples with the same value) of the signal at its peak levels +(i.e. either Min_level or Max_level) +

+
+
Max_difference
+

maximal difference between two consecutive samples +

+
+
Max_level
+

maximal sample level +

+
+
Mean_difference
+

mean difference between two consecutive samples, i.e. the average of each difference between two consecutive samples +

+
+
Min_difference
+

minimal difference between two consecutive samples +

+
+
Min_level
+

minimal sample level +

+
+
Noise_floor
+

minimum local peak measured in dBFS over a short window +

+
+
Noise_floor_count
+

number of occasions (not the number of samples) that the signal attained +Noise floor +

+
+
Number_of_Infs
+

number of samples with an infinite value +

+
+
Number_of_NaNs
+

number of samples with a NaN (not a number) value +

+
+
Number_of_denormals
+

number of samples with a subnormal value +

+
+
Number_of_samples
+

number of samples +

+
+
Peak_count
+

number of occasions (not the number of samples) that the signal attained either +Min_level or Max_level +

+
+
Abs_Peak_count
+

number of occasions that the absolute samples taken from the signal attained +max absolute value of Min_level and Max_level +

+
+
Peak_level
+

standard peak level measured in dBFS +

+
+
RMS_difference
+

Root Mean Square difference between two consecutive samples +

+
+
RMS_level
+

standard RMS level measured in dBFS +

+
+
RMS_peak
+
RMS_trough
+

peak and through values for RMS level measured over a short window, +measured in dBFS. +

+
+
Zero crossings
+

number of points where the waveform crosses the zero level axis +

+
+
Zero crossings rate
+

rate of Zero crossings and number of audio samples +

+
+ +

8.60 asubboost

+

Boost subwoofer frequencies. +

+

The filter accepts the following options: +

+
+
dry
+

Set dry gain, how much of original signal is kept. Allowed range is from 0 to 1. +Default value is 1.0. +

+
+
wet
+

Set wet gain, how much of filtered signal is kept. Allowed range is from 0 to 1. +Default value is 1.0. +

+
+
boost
+

Set max boost factor. Allowed range is from 1 to 12. Default value is 2. +

+
+
decay
+

Set delay line decay gain value. Allowed range is from 0 to 1. +Default value is 0.0. +

+
+
feedback
+

Set delay line feedback gain value. Allowed range is from 0 to 1. +Default value is 0.9. +

+
+
cutoff
+

Set cutoff frequency in Hertz. Allowed range is 50 to 900. +Default value is 100. +

+
+
slope
+

Set slope amount for cutoff frequency. Allowed range is 0.0001 to 1. +Default value is 0.5. +

+
+
delay
+

Set delay. Allowed range is from 1 to 100. +Default value is 20. +

+
+
channels
+

Set the channels to process. Default value is all available. +

+
+ +

8.60.1 Commands

+ +

This filter supports the all above options as commands. +

+

8.61 asubcut

+

Cut subwoofer frequencies. +

+

This filter allows to set custom, steeper +roll off than highpass filter, and thus is able to more attenuate +frequency content in stop-band. +

+

The filter accepts the following options: +

+
+
cutoff
+

Set cutoff frequency in Hertz. Allowed range is 2 to 200. +Default value is 20. +

+
+
order
+

Set filter order. Available values are from 3 to 20. +Default value is 10. +

+
+
level
+

Set input gain level. Allowed range is from 0 to 1. Default value is 1. +

+
+ +

8.61.1 Commands

+ +

This filter supports the all above options as commands. +

+

8.62 asupercut

+

Cut super frequencies. +

+

The filter accepts the following options: +

+
+
cutoff
+

Set cutoff frequency in Hertz. Allowed range is 20000 to 192000. +Default value is 20000. +

+
+
order
+

Set filter order. Available values are from 3 to 20. +Default value is 10. +

+
+
level
+

Set input gain level. Allowed range is from 0 to 1. Default value is 1. +

+
+ +

8.62.1 Commands

+ +

This filter supports the all above options as commands. +

+

8.63 asuperpass

+

Apply high order Butterworth band-pass filter. +

+

The filter accepts the following options: +

+
+
centerf
+

Set center frequency in Hertz. Allowed range is 2 to 999999. +Default value is 1000. +

+
+
order
+

Set filter order. Available values are from 4 to 20. +Default value is 4. +

+
+
qfactor
+

Set Q-factor. Allowed range is from 0.01 to 100. Default value is 1. +

+
+
level
+

Set input gain level. Allowed range is from 0 to 2. Default value is 1. +

+
+ +

8.63.1 Commands

+ +

This filter supports the all above options as commands. +

+

8.64 asuperstop

+

Apply high order Butterworth band-stop filter. +

+

The filter accepts the following options: +

+
+
centerf
+

Set center frequency in Hertz. Allowed range is 2 to 999999. +Default value is 1000. +

+
+
order
+

Set filter order. Available values are from 4 to 20. +Default value is 4. +

+
+
qfactor
+

Set Q-factor. Allowed range is from 0.01 to 100. Default value is 1. +

+
+
level
+

Set input gain level. Allowed range is from 0 to 2. Default value is 1. +

+
+ +

8.64.1 Commands

+ +

This filter supports the all above options as commands. +

+

8.65 atempo

+ +

Adjust audio tempo. +

+

The filter accepts exactly one parameter, the audio tempo. If not +specified then the filter will assume nominal 1.0 tempo. Tempo must +be in the [0.5, 100.0] range. +

+

Note that tempo greater than 2 will skip some samples rather than +blend them in. If for any reason this is a concern it is always +possible to daisy-chain several instances of atempo to achieve the +desired product tempo. +

+

8.65.1 Examples

+ +
    +
  • Slow down audio to 80% tempo: +
    +
    atempo=0.8
    +
    + +
  • To speed up audio to 300% tempo: +
    +
    atempo=3
    +
    + +
  • To speed up audio to 300% tempo by daisy-chaining two atempo instances: +
    +
    atempo=sqrt(3),atempo=sqrt(3)
    +
    +
+ +

8.65.2 Commands

+ +

This filter supports the following commands: +

+
tempo
+

Change filter tempo scale factor. +Syntax for the command is : "tempo" +

+
+ +

8.66 atilt

+

Apply spectral tilt filter to audio stream. +

+

This filter apply any spectral roll-off slope over any specified frequency band. +

+

The filter accepts the following options: +

+
+
freq
+

Set central frequency of tilt in Hz. Default is 10000 Hz. +

+
+
slope
+

Set slope direction of tilt. Default is 0. Allowed range is from -1 to 1. +

+
+
width
+

Set width of tilt. Default is 1000. Allowed range is from 100 to 10000. +

+
+
order
+

Set order of tilt filter. +

+
+
level
+

Set input volume level. Allowed range is from 0 to 4. +Default is 1. +

+
+ +

8.66.1 Commands

+ +

This filter supports the all above options as commands. +

+

8.67 atrim

+ +

Trim the input so that the output contains one continuous subpart of the input. +

+

It accepts the following parameters: +

+
start
+

Timestamp (in seconds) of the start of the section to keep. I.e. the audio +sample with the timestamp start will be the first sample in the output. +

+
+
end
+

Specify time of the first audio sample that will be dropped, i.e. the +audio sample immediately preceding the one with the timestamp end will be +the last sample in the output. +

+
+
start_pts
+

Same as start, except this option sets the start timestamp in samples +instead of seconds. +

+
+
end_pts
+

Same as end, except this option sets the end timestamp in samples instead +of seconds. +

+
+
duration
+

The maximum duration of the output in seconds. +

+
+
start_sample
+

The number of the first sample that should be output. +

+
+
end_sample
+

The number of the first sample that should be dropped. +

+
+ +

start, end, and duration are expressed as time +duration specifications; see +(ffmpeg-utils)the Time duration section in the ffmpeg-utils(1) manual. +

+

Note that the first two sets of the start/end options and the duration +option look at the frame timestamp, while the _sample options simply count the +samples that pass through the filter. So start/end_pts and start/end_sample will +give different results when the timestamps are wrong, inexact or do not start at +zero. Also note that this filter does not modify the timestamps. If you wish +to have the output timestamps start at zero, insert the asetpts filter after the +atrim filter. +

+

If multiple start or end options are set, this filter tries to be greedy and +keep all samples that match at least one of the specified constraints. To keep +only the part that matches all the constraints at once, chain multiple atrim +filters. +

+

The defaults are such that all the input is kept. So it is possible to set e.g. +just the end values to keep everything before the specified time. +

+

Examples: +

    +
  • Drop everything except the second minute of input: +
    +
    ffmpeg -i INPUT -af atrim=60:120
    +
    + +
  • Keep only the first 1000 samples: +
    +
    ffmpeg -i INPUT -af atrim=end_sample=1000
    +
    + +
+ +

8.68 axcorrelate

+

Calculate normalized windowed cross-correlation between two input audio streams. +

+

Resulted samples are always between -1 and 1 inclusive. +If result is 1 it means two input samples are highly correlated in that selected segment. +Result 0 means they are not correlated at all. +If result is -1 it means two input samples are out of phase, which means they cancel each +other. +

+

The filter accepts the following options: +

+
+
size
+

Set size of segment over which cross-correlation is calculated. +Default is 256. Allowed range is from 2 to 131072. +

+
+
algo
+

Set algorithm for cross-correlation. Can be slow or fast or best. +Default is best. Fast algorithm assumes mean values over any given segment +are always zero and thus need much less calculations to make. +This is generally not true, but is valid for typical audio streams. +

+
+ +

8.68.1 Examples

+ +
    +
  • Calculate correlation between channels in stereo audio stream: +
    +
    ffmpeg -i stereo.wav -af channelsplit,axcorrelate=size=1024:algo=fast correlation.wav
    +
    +
+ +

8.69 bandpass

+ +

Apply a two-pole Butterworth band-pass filter with central +frequency frequency, and (3dB-point) band-width width. +The csg option selects a constant skirt gain (peak gain = Q) +instead of the default: constant 0dB peak gain. +The filter roll off at 6dB per octave (20dB per decade). +

+

The filter accepts the following options: +

+
+
frequency, f
+

Set the filter’s central frequency. Default is 3000. +

+
+
csg
+

Constant skirt gain if set to 1. Defaults to 0. +

+
+
width_type, t
+

Set method to specify band-width of filter. +

+
h
+

Hz +

+
q
+

Q-Factor +

+
o
+

octave +

+
s
+

slope +

+
k
+

kHz +

+
+ +
+
width, w
+

Specify the band-width of a filter in width_type units. +

+
+
mix, m
+

How much to use filtered signal in output. Default is 1. +Range is between 0 and 1. +

+
+
channels, c
+

Specify which channels to filter, by default all available are filtered. +

+
+
normalize, n
+

Normalize biquad coefficients, by default is disabled. +Enabling it will normalize magnitude response at DC to 0dB. +

+
+
transform, a
+

Set transform type of IIR filter. +

+
di
+
dii
+
tdi
+
tdii
+
latt
+
svf
+
zdf
+
+ +
+
precision, r
+

Set precision of filtering. +

+
auto
+

Pick automatic sample format depending on surround filters. +

+
s16
+

Always use signed 16-bit. +

+
s32
+

Always use signed 32-bit. +

+
f32
+

Always use float 32-bit. +

+
f64
+

Always use float 64-bit. +

+
+ +
+
block_size, b
+

Set block size used for reverse IIR processing. If this value is set to high enough +value (higher than impulse response length truncated when reaches near zero values) filtering +will become linear phase otherwise if not big enough it will just produce nasty artifacts. +

+

Note that filter delay will be exactly this many samples when set to non-zero value. +

+
+ +

8.69.1 Commands

+ +

This filter supports the following commands: +

+
frequency, f
+

Change bandpass frequency. +Syntax for the command is : "frequency" +

+
+
width_type, t
+

Change bandpass width_type. +Syntax for the command is : "width_type" +

+
+
width, w
+

Change bandpass width. +Syntax for the command is : "width" +

+
+
mix, m
+

Change bandpass mix. +Syntax for the command is : "mix" +

+
+ +

8.70 bandreject

+ +

Apply a two-pole Butterworth band-reject filter with central +frequency frequency, and (3dB-point) band-width width. +The filter roll off at 6dB per octave (20dB per decade). +

+

The filter accepts the following options: +

+
+
frequency, f
+

Set the filter’s central frequency. Default is 3000. +

+
+
width_type, t
+

Set method to specify band-width of filter. +

+
h
+

Hz +

+
q
+

Q-Factor +

+
o
+

octave +

+
s
+

slope +

+
k
+

kHz +

+
+ +
+
width, w
+

Specify the band-width of a filter in width_type units. +

+
+
mix, m
+

How much to use filtered signal in output. Default is 1. +Range is between 0 and 1. +

+
+
channels, c
+

Specify which channels to filter, by default all available are filtered. +

+
+
normalize, n
+

Normalize biquad coefficients, by default is disabled. +Enabling it will normalize magnitude response at DC to 0dB. +

+
+
transform, a
+

Set transform type of IIR filter. +

+
di
+
dii
+
tdi
+
tdii
+
latt
+
svf
+
zdf
+
+ +
+
precision, r
+

Set precision of filtering. +

+
auto
+

Pick automatic sample format depending on surround filters. +

+
s16
+

Always use signed 16-bit. +

+
s32
+

Always use signed 32-bit. +

+
f32
+

Always use float 32-bit. +

+
f64
+

Always use float 64-bit. +

+
+ +
+
block_size, b
+

Set block size used for reverse IIR processing. If this value is set to high enough +value (higher than impulse response length truncated when reaches near zero values) filtering +will become linear phase otherwise if not big enough it will just produce nasty artifacts. +

+

Note that filter delay will be exactly this many samples when set to non-zero value. +

+
+ +

8.70.1 Commands

+ +

This filter supports the following commands: +

+
frequency, f
+

Change bandreject frequency. +Syntax for the command is : "frequency" +

+
+
width_type, t
+

Change bandreject width_type. +Syntax for the command is : "width_type" +

+
+
width, w
+

Change bandreject width. +Syntax for the command is : "width" +

+
+
mix, m
+

Change bandreject mix. +Syntax for the command is : "mix" +

+
+ +

8.71 bass, lowshelf

+ +

Boost or cut the bass (lower) frequencies of the audio using a two-pole +shelving filter with a response similar to that of a standard +hi-fi’s tone-controls. This is also known as shelving equalisation (EQ). +

+

The filter accepts the following options: +

+
+
gain, g
+

Give the gain at 0 Hz. Its useful range is about -20 +(for a large cut) to +20 (for a large boost). +Beware of clipping when using a positive gain. +

+
+
frequency, f
+

Set the filter’s central frequency and so can be used +to extend or reduce the frequency range to be boosted or cut. +The default value is 100 Hz. +

+
+
width_type, t
+

Set method to specify band-width of filter. +

+
h
+

Hz +

+
q
+

Q-Factor +

+
o
+

octave +

+
s
+

slope +

+
k
+

kHz +

+
+ +
+
width, w
+

Determine how steep is the filter’s shelf transition. +

+
+
poles, p
+

Set number of poles. Default is 2. +

+
+
mix, m
+

How much to use filtered signal in output. Default is 1. +Range is between 0 and 1. +

+
+
channels, c
+

Specify which channels to filter, by default all available are filtered. +

+
+
normalize, n
+

Normalize biquad coefficients, by default is disabled. +Enabling it will normalize magnitude response at DC to 0dB. +

+
+
transform, a
+

Set transform type of IIR filter. +

+
di
+
dii
+
tdi
+
tdii
+
latt
+
svf
+
zdf
+
+ +
+
precision, r
+

Set precision of filtering. +

+
auto
+

Pick automatic sample format depending on surround filters. +

+
s16
+

Always use signed 16-bit. +

+
s32
+

Always use signed 32-bit. +

+
f32
+

Always use float 32-bit. +

+
f64
+

Always use float 64-bit. +

+
+ +
+
block_size, b
+

Set block size used for reverse IIR processing. If this value is set to high enough +value (higher than impulse response length truncated when reaches near zero values) filtering +will become linear phase otherwise if not big enough it will just produce nasty artifacts. +

+

Note that filter delay will be exactly this many samples when set to non-zero value. +

+
+ +

8.71.1 Commands

+ +

This filter supports the following commands: +

+
frequency, f
+

Change bass frequency. +Syntax for the command is : "frequency" +

+
+
width_type, t
+

Change bass width_type. +Syntax for the command is : "width_type" +

+
+
width, w
+

Change bass width. +Syntax for the command is : "width" +

+
+
gain, g
+

Change bass gain. +Syntax for the command is : "gain" +

+
+
mix, m
+

Change bass mix. +Syntax for the command is : "mix" +

+
+ +

8.72 biquad

+ +

Apply a biquad IIR filter with the given coefficients. +Where b0, b1, b2 and a0, a1, a2 +are the numerator and denominator coefficients respectively. +and channels, c specify which channels to filter, by default all +available are filtered. +

+

8.72.1 Commands

+ +

This filter supports the following commands: +

+
a0
+
a1
+
a2
+
b0
+
b1
+
b2
+

Change biquad parameter. +Syntax for the command is : "value" +

+
+
mix, m
+

How much to use filtered signal in output. Default is 1. +Range is between 0 and 1. +

+
+
channels, c
+

Specify which channels to filter, by default all available are filtered. +

+
+
normalize, n
+

Normalize biquad coefficients, by default is disabled. +Enabling it will normalize magnitude response at DC to 0dB. +

+
+
transform, a
+

Set transform type of IIR filter. +

+
di
+
dii
+
tdi
+
tdii
+
latt
+
svf
+
zdf
+
+ +
+
precision, r
+

Set precision of filtering. +

+
auto
+

Pick automatic sample format depending on surround filters. +

+
s16
+

Always use signed 16-bit. +

+
s32
+

Always use signed 32-bit. +

+
f32
+

Always use float 32-bit. +

+
f64
+

Always use float 64-bit. +

+
+ +
+
block_size, b
+

Set block size used for reverse IIR processing. If this value is set to high enough +value (higher than impulse response length truncated when reaches near zero values) filtering +will become linear phase otherwise if not big enough it will just produce nasty artifacts. +

+

Note that filter delay will be exactly this many samples when set to non-zero value. +

+
+ +

8.73 bs2b

+

Bauer stereo to binaural transformation, which improves headphone listening of +stereo audio records. +

+

To enable compilation of this filter you need to configure FFmpeg with +--enable-libbs2b. +

+

It accepts the following parameters: +

+
profile
+

Pre-defined crossfeed level. +

+
default
+

Default level (fcut=700, feed=50). +

+
+
cmoy
+

Chu Moy circuit (fcut=700, feed=60). +

+
+
jmeier
+

Jan Meier circuit (fcut=650, feed=95). +

+
+
+ +
+
fcut
+

Cut frequency (in Hz). +

+
+
feed
+

Feed level (in Hz). +

+
+
+ +

8.74 channelmap

+ +

Remap input channels to new locations. +

+

It accepts the following parameters: +

+
map
+

Map channels from input to output. The argument is a ’|’-separated list of +mappings, each in the in_channel-out_channel or +in_channel form. in_channel can be either the name of the +input channel (e.g. FL for front left) or its index in the input channel +layout. out_channel is the name of the output channel or its index in the +output channel layout. If out_channel is not given then it is implicitly +an index, starting with zero and increasing by one for each mapping. Mixing +different types of mappings is not allowed and will result in a parse error. +

+
+
channel_layout
+

The channel layout of the output stream. If not specified, then filter will +guess it based on the out_channel names or the number of mappings. +Guessed layouts will not necessarily contain channels in the order of the +mappings. +

+
+ +

If no mapping is present, the filter will implicitly map input channels to +output channels, preserving indices. +

+

8.74.1 Examples

+ +
    +
  • For example, assuming a 5.1+downmix input MOV file, +
    +
    ffmpeg -i in.mov -filter 'channelmap=map=DL-FL|DR-FR' out.wav
    +
    +

    will create an output WAV file tagged as stereo from the downmix channels of +the input. +

    +
  • To fix a 5.1 WAV improperly encoded in AAC’s native channel order +
    +
    ffmpeg -i in.wav -filter 'channelmap=1|2|0|5|3|4:5.1' out.wav
    +
    +
+ +

8.75 channelsplit

+ +

Split each channel from an input audio stream into a separate output stream. +

+

It accepts the following parameters: +

+
channel_layout
+

The channel layout of the input stream. The default is "stereo". +

+
channels
+

A channel layout describing the channels to be extracted as separate output streams +or "all" to extract each input channel as a separate stream. The default is "all". +

+

Choosing channels not present in channel layout in the input will result in an error. +

+
+ +

8.75.1 Examples

+ +
    +
  • For example, assuming a stereo input MP3 file, +
    +
    ffmpeg -i in.mp3 -filter_complex channelsplit out.mkv
    +
    +

    will create an output Matroska file with two audio streams, one containing only +the left channel and the other the right channel. +

    +
  • Split a 5.1 WAV file into per-channel files: +
    +
    ffmpeg -i in.wav -filter_complex
    +'channelsplit=channel_layout=5.1[FL][FR][FC][LFE][SL][SR]'
    +-map '[FL]' front_left.wav -map '[FR]' front_right.wav -map '[FC]'
    +front_center.wav -map '[LFE]' lfe.wav -map '[SL]' side_left.wav -map '[SR]'
    +side_right.wav
    +
    + +
  • Extract only LFE from a 5.1 WAV file: +
    +
    ffmpeg -i in.wav -filter_complex 'channelsplit=channel_layout=5.1:channels=LFE[LFE]'
    +-map '[LFE]' lfe.wav
    +
    +
+ +

8.76 chorus

+

Add a chorus effect to the audio. +

+

Can make a single vocal sound like a chorus, but can also be applied to instrumentation. +

+

Chorus resembles an echo effect with a short delay, but whereas with echo the delay is +constant, with chorus, it is varied using sinusoidal or triangular modulation. +The modulation depth defines the range the modulated delay is played before or after +the delay. Hence the delayed sound will sound slower or faster, that is the delayed +sound tuned around the original one, like in a chorus where some vocals are slightly +off key. +

+

It accepts the following parameters: +

+
in_gain
+

Set input gain. Default is 0.4. +

+
+
out_gain
+

Set output gain. Default is 0.4. +

+
+
delays
+

Set delays. A typical delay is around 40ms to 60ms. +

+
+
decays
+

Set decays. +

+
+
speeds
+

Set speeds. +

+
+
depths
+

Set depths. +

+
+ +

8.76.1 Examples

+ +
    +
  • A single delay: +
    +
    chorus=0.7:0.9:55:0.4:0.25:2
    +
    + +
  • Two delays: +
    +
    chorus=0.6:0.9:50|60:0.4|0.32:0.25|0.4:2|1.3
    +
    + +
  • Fuller sounding chorus with three delays: +
    +
    chorus=0.5:0.9:50|60|40:0.4|0.32|0.3:0.25|0.4|0.3:2|2.3|1.3
    +
    +
+ +

8.77 compand

+

Compress or expand the audio’s dynamic range. +

+

It accepts the following parameters: +

+
+
attacks
+
decays
+

A list of times in seconds for each channel over which the instantaneous level +of the input signal is averaged to determine its volume. attacks refers to +increase of volume and decays refers to decrease of volume. For most +situations, the attack time (response to the audio getting louder) should be +shorter than the decay time, because the human ear is more sensitive to sudden +loud audio than sudden soft audio. A typical value for attack is 0.3 seconds and +a typical value for decay is 0.8 seconds. +If specified number of attacks & decays is lower than number of channels, the last +set attack/decay will be used for all remaining channels. +

+
+
points
+

A list of points for the transfer function, specified in dB relative to the +maximum possible signal amplitude. Each key points list must be defined using +the following syntax: x0/y0|x1/y1|x2/y2|.... or +x0/y0 x1/y1 x2/y2 .... +

+

The input values must be in strictly increasing order but the transfer function +does not have to be monotonically rising. The point 0/0 is assumed but +may be overridden (by 0/out-dBn). Typical values for the transfer +function are -70/-70|-60/-20|1/0. +

+
+
soft-knee
+

Set the curve radius in dB for all joints. It defaults to 0.01. +

+
+
gain
+

Set the additional gain in dB to be applied at all points on the transfer +function. This allows for easy adjustment of the overall gain. +It defaults to 0. +

+
+
volume
+

Set an initial volume, in dB, to be assumed for each channel when filtering +starts. This permits the user to supply a nominal level initially, so that, for +example, a very large gain is not applied to initial signal levels before the +companding has begun to operate. A typical value for audio which is initially +quiet is -90 dB. It defaults to 0. +

+
+
delay
+

Set a delay, in seconds. The input audio is analyzed immediately, but audio is +delayed before being fed to the volume adjuster. Specifying a delay +approximately equal to the attack/decay times allows the filter to effectively +operate in predictive rather than reactive mode. It defaults to 0. +

+
+
+ +

8.77.1 Examples

+ +
    +
  • Make music with both quiet and loud passages suitable for listening to in a +noisy environment: +
    +
    compand=.3|.3:1|1:-90/-60|-60/-40|-40/-30|-20/-20:6:0:-90:0.2
    +
    + +

    Another example for audio with whisper and explosion parts: +

    +
    compand=0|0:1|1:-90/-900|-70/-70|-30/-9|0/-3:6:0:0:0
    +
    + +
  • A noise gate for when the noise is at a lower level than the signal: +
    +
    compand=.1|.1:.2|.2:-900/-900|-50.1/-900|-50/-50:.01:0:-90:.1
    +
    + +
  • Here is another noise gate, this time for when the noise is at a higher level +than the signal (making it, in some ways, similar to squelch): +
    +
    compand=.1|.1:.1|.1:-45.1/-45.1|-45/-900|0/-900:.01:45:-90:.1
    +
    + +
  • 2:1 compression starting at -6dB: +
    +
    compand=points=-80/-80|-6/-6|0/-3.8|20/3.5
    +
    + +
  • 2:1 compression starting at -9dB: +
    +
    compand=points=-80/-80|-9/-9|0/-5.3|20/2.9
    +
    + +
  • 2:1 compression starting at -12dB: +
    +
    compand=points=-80/-80|-12/-12|0/-6.8|20/1.9
    +
    + +
  • 2:1 compression starting at -18dB: +
    +
    compand=points=-80/-80|-18/-18|0/-9.8|20/0.7
    +
    + +
  • 3:1 compression starting at -15dB: +
    +
    compand=points=-80/-80|-15/-15|0/-10.8|20/-5.2
    +
    + +
  • Compressor/Gate: +
    +
    compand=points=-80/-105|-62/-80|-15.4/-15.4|0/-12|20/-7.6
    +
    + +
  • Expander: +
    +
    compand=attacks=0:points=-80/-169|-54/-80|-49.5/-64.6|-41.1/-41.1|-25.8/-15|-10.8/-4.5|0/0|20/8.3
    +
    + +
  • Hard limiter at -6dB: +
    +
    compand=attacks=0:points=-80/-80|-6/-6|20/-6
    +
    + +
  • Hard limiter at -12dB: +
    +
    compand=attacks=0:points=-80/-80|-12/-12|20/-12
    +
    + +
  • Hard noise gate at -35 dB: +
    +
    compand=attacks=0:points=-80/-115|-35.1/-80|-35/-35|20/20
    +
    + +
  • Soft limiter: +
    +
    compand=attacks=0:points=-80/-80|-12.4/-12.4|-6/-8|0/-6.8|20/-2.8
    +
    +
+ +

8.78 compensationdelay

+ +

Compensation Delay Line is a metric based delay to compensate differing +positions of microphones or speakers. +

+

For example, you have recorded guitar with two microphones placed in +different locations. Because the front of sound wave has fixed speed in +normal conditions, the phasing of microphones can vary and depends on +their location and interposition. The best sound mix can be achieved when +these microphones are in phase (synchronized). Note that a distance of +~30 cm between microphones makes one microphone capture the signal in +antiphase to the other microphone. That makes the final mix sound moody. +This filter helps to solve phasing problems by adding different delays +to each microphone track and make them synchronized. +

+

The best result can be reached when you take one track as base and +synchronize other tracks one by one with it. +Remember that synchronization/delay tolerance depends on sample rate, too. +Higher sample rates will give more tolerance. +

+

The filter accepts the following parameters: +

+
+
mm
+

Set millimeters distance. This is compensation distance for fine tuning. +Default is 0. +

+
+
cm
+

Set cm distance. This is compensation distance for tightening distance setup. +Default is 0. +

+
+
m
+

Set meters distance. This is compensation distance for hard distance setup. +Default is 0. +

+
+
dry
+

Set dry amount. Amount of unprocessed (dry) signal. +Default is 0. +

+
+
wet
+

Set wet amount. Amount of processed (wet) signal. +Default is 1. +

+
+
temp
+

Set temperature in degrees Celsius. This is the temperature of the environment. +Default is 20. +

+
+ +

8.78.1 Commands

+ +

This filter supports the all above options as commands. +

+

8.79 crossfeed

+

Apply headphone crossfeed filter. +

+

Crossfeed is the process of blending the left and right channels of stereo +audio recording. +It is mainly used to reduce extreme stereo separation of low frequencies. +

+

The intent is to produce more speaker like sound to the listener. +

+

The filter accepts the following options: +

+
+
strength
+

Set strength of crossfeed. Default is 0.2. Allowed range is from 0 to 1. +This sets gain of low shelf filter for side part of stereo image. +Default is -6dB. Max allowed is -30db when strength is set to 1. +

+
+
range
+

Set soundstage wideness. Default is 0.5. Allowed range is from 0 to 1. +This sets cut off frequency of low shelf filter. Default is cut off near +1550 Hz. With range set to 1 cut off frequency is set to 2100 Hz. +

+
+
slope
+

Set curve slope of low shelf filter. Default is 0.5. +Allowed range is from 0.01 to 1. +

+
+
level_in
+

Set input gain. Default is 0.9. +

+
+
level_out
+

Set output gain. Default is 1. +

+
+
block_size
+

Set block size used for reverse IIR processing. If this value is set to high enough +value (higher than impulse response length truncated when reaches near zero values) filtering +will become linear phase otherwise if not big enough it will just produce nasty artifacts. +

+

Note that filter delay will be exactly this many samples when set to non-zero value. +

+
+ +

8.79.1 Commands

+ +

This filter supports the all above options as commands. +

+

8.80 crystalizer

+

Simple algorithm for audio noise sharpening. +

+

This filter linearly increases differences between each audio sample. +

+

The filter accepts the following options: +

+
+
i
+

Sets the intensity of effect (default: 2.0). Must be in range between -10.0 to 0 +(unchanged sound) to 10.0 (maximum effect). +To inverse filtering use negative value. +

+
+
c
+

Enable clipping. By default is enabled. +

+
+ +

8.80.1 Commands

+ +

This filter supports the all above options as commands. +

+

8.81 dcshift

+

Apply a DC shift to the audio. +

+

This can be useful to remove a DC offset (caused perhaps by a hardware problem +in the recording chain) from the audio. The effect of a DC offset is reduced +headroom and hence volume. The astats filter can be used to determine if +a signal has a DC offset. +

+
+
shift
+

Set the DC shift, allowed range is [-1, 1]. It indicates the amount to shift +the audio. +

+
+
limitergain
+

Optional. It should have a value much less than 1 (e.g. 0.05 or 0.02) and is +used to prevent clipping. +

+
+ +

8.82 deesser

+ +

Apply de-essing to the audio samples. +

+
+
i
+

Set intensity for triggering de-essing. Allowed range is from 0 to 1. +Default is 0. +

+
+
m
+

Set amount of ducking on treble part of sound. Allowed range is from 0 to 1. +Default is 0.5. +

+
+
f
+

How much of original frequency content to keep when de-essing. Allowed range is from 0 to 1. +Default is 0.5. +

+
+
s
+

Set the output mode. +

+

It accepts the following values: +

+
i
+

Pass input unchanged. +

+
+
o
+

Pass ess filtered out. +

+
+
e
+

Pass only ess. +

+

Default value is o. +

+
+ +
+
+ +

8.83 dialoguenhance

+

Enhance dialogue in stereo audio. +

+

This filter accepts stereo input and produce surround (3.0) channels output. +The newly produced front center channel have enhanced speech dialogue originally +available in both stereo channels. +This filter outputs front left and front right channels same as available in stereo input. +

+

The filter accepts the following options: +

+
+
original
+

Set the original center factor to keep in front center channel output. +Allowed range is from 0 to 1. Default value is 1. +

+
+
enhance
+

Set the dialogue enhance factor to put in front center channel output. +Allowed range is from 0 to 3. Default value is 1. +

+
+
voice
+

Set the voice detection factor. +Allowed range is from 2 to 32. Default value is 2. +

+
+ +

8.83.1 Commands

+ +

This filter supports the all above options as commands. +

+

8.84 drmeter

+

Measure audio dynamic range. +

+

DR values of 14 and higher is found in very dynamic material. DR of 8 to 13 +is found in transition material. And anything less that 8 have very poor dynamics +and is very compressed. +

+

The filter accepts the following options: +

+
+
length
+

Set window length in seconds used to split audio into segments of equal length. +Default is 3 seconds. +

+
+ +

8.85 dynaudnorm

+

Dynamic Audio Normalizer. +

+

This filter applies a certain amount of gain to the input audio in order +to bring its peak magnitude to a target level (e.g. 0 dBFS). However, in +contrast to more "simple" normalization algorithms, the Dynamic Audio +Normalizer *dynamically* re-adjusts the gain factor to the input audio. +This allows for applying extra gain to the "quiet" sections of the audio +while avoiding distortions or clipping the "loud" sections. In other words: +The Dynamic Audio Normalizer will "even out" the volume of quiet and loud +sections, in the sense that the volume of each section is brought to the +same target level. Note, however, that the Dynamic Audio Normalizer achieves +this goal *without* applying "dynamic range compressing". It will retain 100% +of the dynamic range *within* each section of the audio file. +

+
+
framelen, f
+

Set the frame length in milliseconds. In range from 10 to 8000 milliseconds. +Default is 500 milliseconds. +The Dynamic Audio Normalizer processes the input audio in small chunks, +referred to as frames. This is required, because a peak magnitude has no +meaning for just a single sample value. Instead, we need to determine the +peak magnitude for a contiguous sequence of sample values. While a "standard" +normalizer would simply use the peak magnitude of the complete file, the +Dynamic Audio Normalizer determines the peak magnitude individually for each +frame. The length of a frame is specified in milliseconds. By default, the +Dynamic Audio Normalizer uses a frame length of 500 milliseconds, which has +been found to give good results with most files. +Note that the exact frame length, in number of samples, will be determined +automatically, based on the sampling rate of the individual input audio file. +

+
+
gausssize, g
+

Set the Gaussian filter window size. In range from 3 to 301, must be odd +number. Default is 31. +Probably the most important parameter of the Dynamic Audio Normalizer is the +window size of the Gaussian smoothing filter. The filter’s window size +is specified in frames, centered around the current frame. For the sake of +simplicity, this must be an odd number. Consequently, the default value of 31 +takes into account the current frame, as well as the 15 preceding frames and +the 15 subsequent frames. Using a larger window results in a stronger +smoothing effect and thus in less gain variation, i.e. slower gain +adaptation. Conversely, using a smaller window results in a weaker smoothing +effect and thus in more gain variation, i.e. faster gain adaptation. +In other words, the more you increase this value, the more the Dynamic Audio +Normalizer will behave like a "traditional" normalization filter. On the +contrary, the more you decrease this value, the more the Dynamic Audio +Normalizer will behave like a dynamic range compressor. +

+
+
peak, p
+

Set the target peak value. This specifies the highest permissible magnitude +level for the normalized audio input. This filter will try to approach the +target peak magnitude as closely as possible, but at the same time it also +makes sure that the normalized signal will never exceed the peak magnitude. +A frame’s maximum local gain factor is imposed directly by the target peak +magnitude. The default value is 0.95 and thus leaves a headroom of 5%*. +It is not recommended to go above this value. +

+
+
maxgain, m
+

Set the maximum gain factor. In range from 1.0 to 100.0. Default is 10.0. +The Dynamic Audio Normalizer determines the maximum possible (local) gain +factor for each input frame, i.e. the maximum gain factor that does not +result in clipping or distortion. The maximum gain factor is determined by +the frame’s highest magnitude sample. However, the Dynamic Audio Normalizer +additionally bounds the frame’s maximum gain factor by a predetermined +(global) maximum gain factor. This is done in order to avoid excessive gain +factors in "silent" or almost silent frames. By default, the maximum gain +factor is 10.0, For most inputs the default value should be sufficient and +it usually is not recommended to increase this value. Though, for input +with an extremely low overall volume level, it may be necessary to allow even +higher gain factors. Note, however, that the Dynamic Audio Normalizer does +not simply apply a "hard" threshold (i.e. cut off values above the threshold). +Instead, a "sigmoid" threshold function will be applied. This way, the +gain factors will smoothly approach the threshold value, but never exceed that +value. +

+
+
targetrms, r
+

Set the target RMS. In range from 0.0 to 1.0. Default is 0.0 - disabled. +By default, the Dynamic Audio Normalizer performs "peak" normalization. +This means that the maximum local gain factor for each frame is defined +(only) by the frame’s highest magnitude sample. This way, the samples can +be amplified as much as possible without exceeding the maximum signal +level, i.e. without clipping. Optionally, however, the Dynamic Audio +Normalizer can also take into account the frame’s root mean square, +abbreviated RMS. In electrical engineering, the RMS is commonly used to +determine the power of a time-varying signal. It is therefore considered +that the RMS is a better approximation of the "perceived loudness" than +just looking at the signal’s peak magnitude. Consequently, by adjusting all +frames to a constant RMS value, a uniform "perceived loudness" can be +established. If a target RMS value has been specified, a frame’s local gain +factor is defined as the factor that would result in exactly that RMS value. +Note, however, that the maximum local gain factor is still restricted by the +frame’s highest magnitude sample, in order to prevent clipping. +

+
+
coupling, n
+

Enable channels coupling. By default is enabled. +By default, the Dynamic Audio Normalizer will amplify all channels by the same +amount. This means the same gain factor will be applied to all channels, i.e. +the maximum possible gain factor is determined by the "loudest" channel. +However, in some recordings, it may happen that the volume of the different +channels is uneven, e.g. one channel may be "quieter" than the other one(s). +In this case, this option can be used to disable the channel coupling. This way, +the gain factor will be determined independently for each channel, depending +only on the individual channel’s highest magnitude sample. This allows for +harmonizing the volume of the different channels. +

+
+
correctdc, c
+

Enable DC bias correction. By default is disabled. +An audio signal (in the time domain) is a sequence of sample values. +In the Dynamic Audio Normalizer these sample values are represented in the +-1.0 to 1.0 range, regardless of the original input format. Normally, the +audio signal, or "waveform", should be centered around the zero point. +That means if we calculate the mean value of all samples in a file, or in a +single frame, then the result should be 0.0 or at least very close to that +value. If, however, there is a significant deviation of the mean value from +0.0, in either positive or negative direction, this is referred to as a +DC bias or DC offset. Since a DC bias is clearly undesirable, the Dynamic +Audio Normalizer provides optional DC bias correction. +With DC bias correction enabled, the Dynamic Audio Normalizer will determine +the mean value, or "DC correction" offset, of each input frame and subtract +that value from all of the frame’s sample values which ensures those samples +are centered around 0.0 again. Also, in order to avoid "gaps" at the frame +boundaries, the DC correction offset values will be interpolated smoothly +between neighbouring frames. +

+
+
altboundary, b
+

Enable alternative boundary mode. By default is disabled. +The Dynamic Audio Normalizer takes into account a certain neighbourhood +around each frame. This includes the preceding frames as well as the +subsequent frames. However, for the "boundary" frames, located at the very +beginning and at the very end of the audio file, not all neighbouring +frames are available. In particular, for the first few frames in the audio +file, the preceding frames are not known. And, similarly, for the last few +frames in the audio file, the subsequent frames are not known. Thus, the +question arises which gain factors should be assumed for the missing frames +in the "boundary" region. The Dynamic Audio Normalizer implements two modes +to deal with this situation. The default boundary mode assumes a gain factor +of exactly 1.0 for the missing frames, resulting in a smooth "fade in" and +"fade out" at the beginning and at the end of the input, respectively. +

+
+
compress, s
+

Set the compress factor. In range from 0.0 to 30.0. Default is 0.0. +By default, the Dynamic Audio Normalizer does not apply "traditional" +compression. This means that signal peaks will not be pruned and thus the +full dynamic range will be retained within each local neighbourhood. However, +in some cases it may be desirable to combine the Dynamic Audio Normalizer’s +normalization algorithm with a more "traditional" compression. +For this purpose, the Dynamic Audio Normalizer provides an optional compression +(thresholding) function. If (and only if) the compression feature is enabled, +all input frames will be processed by a soft knee thresholding function prior +to the actual normalization process. Put simply, the thresholding function is +going to prune all samples whose magnitude exceeds a certain threshold value. +However, the Dynamic Audio Normalizer does not simply apply a fixed threshold +value. Instead, the threshold value will be adjusted for each individual +frame. +In general, smaller parameters result in stronger compression, and vice versa. +Values below 3.0 are not recommended, because audible distortion may appear. +

+
+
threshold, t
+

Set the target threshold value. This specifies the lowest permissible +magnitude level for the audio input which will be normalized. +If input frame volume is above this value frame will be normalized. +Otherwise frame may not be normalized at all. The default value is set +to 0, which means all input frames will be normalized. +This option is mostly useful if digital noise is not wanted to be amplified. +

+
+
channels, h
+

Specify which channels to filter, by default all available channels are filtered. +

+
+
overlap, o
+

Specify overlap for frames. If set to 0 (default) no frame overlapping is done. +Using >0 and <1 values will make less conservative gain adjustments, like +when framelen option is set to smaller value, if framelen option value is +compensated for non-zero overlap then gain adjustments will be smoother across time +compared to zero overlap case. +

+
+
curve, v
+

Specify the peak mapping curve expression which is going to be used when calculating +gain applied to frames. The max output frame gain will still be limited by other +options mentioned previously for this filter. +

+

The expression can contain the following constants: +

+
+
ch
+

current channel number +

+
+
sn
+

current sample number +

+
+
nb_channels
+

number of channels +

+
+
t
+

timestamp expressed in seconds +

+
+
sr
+

sample rate +

+
+
p
+

current frame peak value +

+
+
+
+ +

8.85.1 Commands

+ +

This filter supports the all above options as commands. +

+

8.86 earwax

+ +

Make audio easier to listen to on headphones. +

+

This filter adds ‘cues’ to 44.1kHz stereo (i.e. audio CD format) audio +so that when listened to on headphones the stereo image is moved from +inside your head (standard for headphones) to outside and in front of +the listener (standard for speakers). +

+

Ported from SoX. +

+

8.87 equalizer

+ +

Apply a two-pole peaking equalisation (EQ) filter. With this +filter, the signal-level at and around a selected frequency can +be increased or decreased, whilst (unlike bandpass and bandreject +filters) that at all other frequencies is unchanged. +

+

In order to produce complex equalisation curves, this filter can +be given several times, each with a different central frequency. +

+

The filter accepts the following options: +

+
+
frequency, f
+

Set the filter’s central frequency in Hz. +

+
+
width_type, t
+

Set method to specify band-width of filter. +

+
h
+

Hz +

+
q
+

Q-Factor +

+
o
+

octave +

+
s
+

slope +

+
k
+

kHz +

+
+ +
+
width, w
+

Specify the band-width of a filter in width_type units. +

+
+
gain, g
+

Set the required gain or attenuation in dB. +Beware of clipping when using a positive gain. +

+
+
mix, m
+

How much to use filtered signal in output. Default is 1. +Range is between 0 and 1. +

+
+
channels, c
+

Specify which channels to filter, by default all available are filtered. +

+
+
normalize, n
+

Normalize biquad coefficients, by default is disabled. +Enabling it will normalize magnitude response at DC to 0dB. +

+
+
transform, a
+

Set transform type of IIR filter. +

+
di
+
dii
+
tdi
+
tdii
+
latt
+
svf
+
zdf
+
+ +
+
precision, r
+

Set precision of filtering. +

+
auto
+

Pick automatic sample format depending on surround filters. +

+
s16
+

Always use signed 16-bit. +

+
s32
+

Always use signed 32-bit. +

+
f32
+

Always use float 32-bit. +

+
f64
+

Always use float 64-bit. +

+
+ +
+
block_size, b
+

Set block size used for reverse IIR processing. If this value is set to high enough +value (higher than impulse response length truncated when reaches near zero values) filtering +will become linear phase otherwise if not big enough it will just produce nasty artifacts. +

+

Note that filter delay will be exactly this many samples when set to non-zero value. +

+
+ +

8.87.1 Examples

+
    +
  • Attenuate 10 dB at 1000 Hz, with a bandwidth of 200 Hz: +
    +
    equalizer=f=1000:t=h:width=200:g=-10
    +
    + +
  • Apply 2 dB gain at 1000 Hz with Q 1 and attenuate 5 dB at 100 Hz with Q 2: +
    +
    equalizer=f=1000:t=q:w=1:g=2,equalizer=f=100:t=q:w=2:g=-5
    +
    +
+ +

8.87.2 Commands

+ +

This filter supports the following commands: +

+
frequency, f
+

Change equalizer frequency. +Syntax for the command is : "frequency" +

+
+
width_type, t
+

Change equalizer width_type. +Syntax for the command is : "width_type" +

+
+
width, w
+

Change equalizer width. +Syntax for the command is : "width" +

+
+
gain, g
+

Change equalizer gain. +Syntax for the command is : "gain" +

+
+
mix, m
+

Change equalizer mix. +Syntax for the command is : "mix" +

+
+ +

8.88 extrastereo

+ +

Linearly increases the difference between left and right channels which +adds some sort of "live" effect to playback. +

+

The filter accepts the following options: +

+
+
m
+

Sets the difference coefficient (default: 2.5). 0.0 means mono sound +(average of both channels), with 1.0 sound will be unchanged, with +-1.0 left and right channels will be swapped. +

+
+
c
+

Enable clipping. By default is enabled. +

+
+ +

8.88.1 Commands

+ +

This filter supports the all above options as commands. +

+

8.89 firequalizer

+

Apply FIR Equalization using arbitrary frequency response. +

+

The filter accepts the following option: +

+
+
gain
+

Set gain curve equation (in dB). The expression can contain variables: +

+
f
+

the evaluated frequency +

+
sr
+

sample rate +

+
ch
+

channel number, set to 0 when multichannels evaluation is disabled +

+
chid
+

channel id, see libavutil/channel_layout.h, set to the first channel id when +multichannels evaluation is disabled +

+
chs
+

number of channels +

+
chlayout
+

channel_layout, see libavutil/channel_layout.h +

+
+
+

and functions: +

+
gain_interpolate(f)
+

interpolate gain on frequency f based on gain_entry +

+
cubic_interpolate(f)
+

same as gain_interpolate, but smoother +

+
+

This option is also available as command. Default is gain_interpolate(f). +

+
+
gain_entry
+

Set gain entry for gain_interpolate function. The expression can +contain functions: +

+
entry(f, g)
+

store gain entry at frequency f with value g +

+
+

This option is also available as command. +

+
+
delay
+

Set filter delay in seconds. Higher value means more accurate. +Default is 0.01. +

+
+
accuracy
+

Set filter accuracy in Hz. Lower value means more accurate. +Default is 5. +

+
+
wfunc
+

Set window function. Acceptable values are: +

+
rectangular
+

rectangular window, useful when gain curve is already smooth +

+
hann
+

hann window (default) +

+
hamming
+

hamming window +

+
blackman
+

blackman window +

+
nuttall3
+

3-terms continuous 1st derivative nuttall window +

+
mnuttall3
+

minimum 3-terms discontinuous nuttall window +

+
nuttall
+

4-terms continuous 1st derivative nuttall window +

+
bnuttall
+

minimum 4-terms discontinuous nuttall (blackman-nuttall) window +

+
bharris
+

blackman-harris window +

+
tukey
+

tukey window +

+
+ +
+
fixed
+

If enabled, use fixed number of audio samples. This improves speed when +filtering with large delay. Default is disabled. +

+
+
multi
+

Enable multichannels evaluation on gain. Default is disabled. +

+
+
zero_phase
+

Enable zero phase mode by subtracting timestamp to compensate delay. +Default is disabled. +

+
+
scale
+

Set scale used by gain. Acceptable values are: +

+
linlin
+

linear frequency, linear gain +

+
linlog
+

linear frequency, logarithmic (in dB) gain (default) +

+
loglin
+

logarithmic (in octave scale where 20 Hz is 0) frequency, linear gain +

+
loglog
+

logarithmic frequency, logarithmic gain +

+
+ +
+
dumpfile
+

Set file for dumping, suitable for gnuplot. +

+
+
dumpscale
+

Set scale for dumpfile. Acceptable values are same with scale option. +Default is linlog. +

+
+
fft2
+

Enable 2-channel convolution using complex FFT. This improves speed significantly. +Default is disabled. +

+
+
min_phase
+

Enable minimum phase impulse response. Default is disabled. +

+
+ +

8.89.1 Examples

+
    +
  • lowpass at 1000 Hz: +
    +
    firequalizer=gain='if(lt(f,1000), 0, -INF)'
    +
    +
  • lowpass at 1000 Hz with gain_entry: +
    +
    firequalizer=gain_entry='entry(1000,0); entry(1001, -INF)'
    +
    +
  • custom equalization: +
    +
    firequalizer=gain_entry='entry(100,0); entry(400, -4); entry(1000, -6); entry(2000, 0)'
    +
    +
  • higher delay with zero phase to compensate delay: +
    +
    firequalizer=delay=0.1:fixed=on:zero_phase=on
    +
    +
  • lowpass on left channel, highpass on right channel: +
    +
    firequalizer=gain='if(eq(chid,1), gain_interpolate(f), if(eq(chid,2), gain_interpolate(1e6+f), 0))'
    +:gain_entry='entry(1000, 0); entry(1001,-INF); entry(1e6+1000,0)':multi=on
    +
    +
+ +

8.90 flanger

+

Apply a flanging effect to the audio. +

+

The filter accepts the following options: +

+
+
delay
+

Set base delay in milliseconds. Range from 0 to 30. Default value is 0. +

+
+
depth
+

Set added sweep delay in milliseconds. Range from 0 to 10. Default value is 2. +

+
+
regen
+

Set percentage regeneration (delayed signal feedback). Range from -95 to 95. +Default value is 0. +

+
+
width
+

Set percentage of delayed signal mixed with original. Range from 0 to 100. +Default value is 71. +

+
+
speed
+

Set sweeps per second (Hz). Range from 0.1 to 10. Default value is 0.5. +

+
+
shape
+

Set swept wave shape, can be triangular or sinusoidal. +Default value is sinusoidal. +

+
+
phase
+

Set swept wave percentage-shift for multi channel. Range from 0 to 100. +Default value is 25. +

+
+
interp
+

Set delay-line interpolation, linear or quadratic. +Default is linear. +

+
+ +

8.91 haas

+

Apply Haas effect to audio. +

+

Note that this makes most sense to apply on mono signals. +With this filter applied to mono signals it give some directionality and +stretches its stereo image. +

+

The filter accepts the following options: +

+
+
level_in
+

Set input level. By default is 1, or 0dB +

+
+
level_out
+

Set output level. By default is 1, or 0dB. +

+
+
side_gain
+

Set gain applied to side part of signal. By default is 1. +

+
+
middle_source
+

Set kind of middle source. Can be one of the following: +

+
+
‘left’
+

Pick left channel. +

+
+
‘right’
+

Pick right channel. +

+
+
‘mid’
+

Pick middle part signal of stereo image. +

+
+
‘side’
+

Pick side part signal of stereo image. +

+
+ +
+
middle_phase
+

Change middle phase. By default is disabled. +

+
+
left_delay
+

Set left channel delay. By default is 2.05 milliseconds. +

+
+
left_balance
+

Set left channel balance. By default is -1. +

+
+
left_gain
+

Set left channel gain. By default is 1. +

+
+
left_phase
+

Change left phase. By default is disabled. +

+
+
right_delay
+

Set right channel delay. By defaults is 2.12 milliseconds. +

+
+
right_balance
+

Set right channel balance. By default is 1. +

+
+
right_gain
+

Set right channel gain. By default is 1. +

+
+
right_phase
+

Change right phase. By default is enabled. +

+
+ +

8.92 hdcd

+ +

Decodes High Definition Compatible Digital (HDCD) data. A 16-bit PCM stream with +embedded HDCD codes is expanded into a 20-bit PCM stream. +

+

The filter supports the Peak Extend and Low-level Gain Adjustment features +of HDCD, and detects the Transient Filter flag. +

+
+
ffmpeg -i HDCD16.flac -af hdcd OUT24.flac
+
+ +

When using the filter with wav, note the default encoding for wav is 16-bit, +so the resulting 20-bit stream will be truncated back to 16-bit. Use something +like -acodec pcm_s24le after the filter to get 24-bit PCM output. +

+
ffmpeg -i HDCD16.wav -af hdcd OUT16.wav
+ffmpeg -i HDCD16.wav -af hdcd -c:a pcm_s24le OUT24.wav
+
+ +

The filter accepts the following options: +

+
+
disable_autoconvert
+

Disable any automatic format conversion or resampling in the filter graph. +

+
+
process_stereo
+

Process the stereo channels together. If target_gain does not match between +channels, consider it invalid and use the last valid target_gain. +

+
+
cdt_ms
+

Set the code detect timer period in ms. +

+
+
force_pe
+

Always extend peaks above -3dBFS even if PE isn’t signaled. +

+
+
analyze_mode
+

Replace audio with a solid tone and adjust the amplitude to signal some +specific aspect of the decoding process. The output file can be loaded in +an audio editor alongside the original to aid analysis. +

+

analyze_mode=pe:force_pe=true can be used to see all samples above the PE level. +

+

Modes are: +

+
‘0, off’
+

Disabled +

+
‘1, lle’
+

Gain adjustment level at each sample +

+
‘2, pe’
+

Samples where peak extend occurs +

+
‘3, cdt’
+

Samples where the code detect timer is active +

+
‘4, tgm’
+

Samples where the target gain does not match between channels +

+
+
+
+ +

8.93 headphone

+ +

Apply head-related transfer functions (HRTFs) to create virtual +loudspeakers around the user for binaural listening via headphones. +The HRIRs are provided via additional streams, for each channel +one stereo input stream is needed. +

+

The filter accepts the following options: +

+
+
map
+

Set mapping of input streams for convolution. +The argument is a ’|’-separated list of channel names in order as they +are given as additional stream inputs for filter. +This also specify number of input streams. Number of input streams +must be not less than number of channels in first stream plus one. +

+
+
gain
+

Set gain applied to audio. Value is in dB. Default is 0. +

+
+
type
+

Set processing type. Can be time or freq. time is +processing audio in time domain which is slow. +freq is processing audio in frequency domain which is fast. +Default is freq. +

+
+
lfe
+

Set custom gain for LFE channels. Value is in dB. Default is 0. +

+
+
size
+

Set size of frame in number of samples which will be processed at once. +Default value is 1024. Allowed range is from 1024 to 96000. +

+
+
hrir
+

Set format of hrir stream. +Default value is stereo. Alternative value is multich. +If value is set to stereo, number of additional streams should +be greater or equal to number of input channels in first input stream. +Also each additional stream should have stereo number of channels. +If value is set to multich, number of additional streams should +be exactly one. Also number of input channels of additional stream +should be equal or greater than twice number of channels of first input +stream. +

+
+ +

8.93.1 Examples

+ +
    +
  • Full example using wav files as coefficients with amovie filters for 7.1 downmix, +each amovie filter use stereo file with IR coefficients as input. +The files give coefficients for each position of virtual loudspeaker: +
    +
    ffmpeg -i input.wav
    +-filter_complex "amovie=azi_270_ele_0_DFC.wav[sr];amovie=azi_90_ele_0_DFC.wav[sl];amovie=azi_225_ele_0_DFC.wav[br];amovie=azi_135_ele_0_DFC.wav[bl];amovie=azi_0_ele_0_DFC.wav,asplit[fc][lfe];amovie=azi_35_ele_0_DFC.wav[fl];amovie=azi_325_ele_0_DFC.wav[fr];[0:a][fl][fr][fc][lfe][bl][br][sl][sr]headphone=FL|FR|FC|LFE|BL|BR|SL|SR"
    +output.wav
    +
    + +
  • Full example using wav files as coefficients with amovie filters for 7.1 downmix, +but now in multich hrir format. +
    +
    ffmpeg -i input.wav -filter_complex "amovie=minp.wav[hrirs];[0:a][hrirs]headphone=map=FL|FR|FC|LFE|BL|BR|SL|SR:hrir=multich"
    +output.wav
    +
    +
+ +

8.94 highpass

+ +

Apply a high-pass filter with 3dB point frequency. +The filter can be either single-pole, or double-pole (the default). +The filter roll off at 6dB per pole per octave (20dB per pole per decade). +

+

The filter accepts the following options: +

+
+
frequency, f
+

Set frequency in Hz. Default is 3000. +

+
+
poles, p
+

Set number of poles. Default is 2. +

+
+
width_type, t
+

Set method to specify band-width of filter. +

+
h
+

Hz +

+
q
+

Q-Factor +

+
o
+

octave +

+
s
+

slope +

+
k
+

kHz +

+
+ +
+
width, w
+

Specify the band-width of a filter in width_type units. +Applies only to double-pole filter. +The default is 0.707q and gives a Butterworth response. +

+
+
mix, m
+

How much to use filtered signal in output. Default is 1. +Range is between 0 and 1. +

+
+
channels, c
+

Specify which channels to filter, by default all available are filtered. +

+
+
normalize, n
+

Normalize biquad coefficients, by default is disabled. +Enabling it will normalize magnitude response at DC to 0dB. +

+
+
transform, a
+

Set transform type of IIR filter. +

+
di
+
dii
+
tdi
+
tdii
+
latt
+
svf
+
zdf
+
+ +
+
precision, r
+

Set precision of filtering. +

+
auto
+

Pick automatic sample format depending on surround filters. +

+
s16
+

Always use signed 16-bit. +

+
s32
+

Always use signed 32-bit. +

+
f32
+

Always use float 32-bit. +

+
f64
+

Always use float 64-bit. +

+
+ +
+
block_size, b
+

Set block size used for reverse IIR processing. If this value is set to high enough +value (higher than impulse response length truncated when reaches near zero values) filtering +will become linear phase otherwise if not big enough it will just produce nasty artifacts. +

+

Note that filter delay will be exactly this many samples when set to non-zero value. +

+
+ +

8.94.1 Commands

+ +

This filter supports the following commands: +

+
frequency, f
+

Change highpass frequency. +Syntax for the command is : "frequency" +

+
+
width_type, t
+

Change highpass width_type. +Syntax for the command is : "width_type" +

+
+
width, w
+

Change highpass width. +Syntax for the command is : "width" +

+
+
mix, m
+

Change highpass mix. +Syntax for the command is : "mix" +

+
+ +

8.95 join

+ +

Join multiple input streams into one multi-channel stream. +

+

It accepts the following parameters: +

+
inputs
+

The number of input streams. It defaults to 2. +

+
+
channel_layout
+

The desired output channel layout. It defaults to stereo. +

+
+
map
+

Map channels from inputs to output. The argument is a ’|’-separated list of +mappings, each in the input_idx.in_channel-out_channel +form. input_idx is the 0-based index of the input stream. in_channel +can be either the name of the input channel (e.g. FL for front left) or its +index in the specified input stream. out_channel is the name of the output +channel. +

+
+ +

The filter will attempt to guess the mappings when they are not specified +explicitly. It does so by first trying to find an unused matching input channel +and if that fails it picks the first unused input channel. +

+

Join 3 inputs (with properly set channel layouts): +

+
ffmpeg -i INPUT1 -i INPUT2 -i INPUT3 -filter_complex join=inputs=3 OUTPUT
+
+ +

Build a 5.1 output from 6 single-channel streams: +

+
ffmpeg -i fl -i fr -i fc -i sl -i sr -i lfe -filter_complex
+'join=inputs=6:channel_layout=5.1:map=0.0-FL|1.0-FR|2.0-FC|3.0-SL|4.0-SR|5.0-LFE'
+out
+
+ +

8.96 ladspa

+ +

Load a LADSPA (Linux Audio Developer’s Simple Plugin API) plugin. +

+

To enable compilation of this filter you need to configure FFmpeg with +--enable-ladspa. +

+
+
file, f
+

Specifies the name of LADSPA plugin library to load. If the environment +variable LADSPA_PATH is defined, the LADSPA plugin is searched in +each one of the directories specified by the colon separated list in +LADSPA_PATH, otherwise in the standard LADSPA paths, which are in +this order: HOME/.ladspa/lib/, /usr/local/lib/ladspa/, +/usr/lib/ladspa/. +

+
+
plugin, p
+

Specifies the plugin within the library. Some libraries contain only +one plugin, but others contain many of them. If this is not set filter +will list all available plugins within the specified library. +

+
+
controls, c
+

Set the ’|’ separated list of controls which are zero or more floating point +values that determine the behavior of the loaded plugin (for example delay, +threshold or gain). +Controls need to be defined using the following syntax: +c0=value0|c1=value1|c2=value2|..., where +valuei is the value set on the i-th control. +Alternatively they can be also defined using the following syntax: +value0|value1|value2|..., where +valuei is the value set on the i-th control. +If controls is set to help, all available controls and +their valid ranges are printed. +

+
+
sample_rate, s
+

Specify the sample rate, default to 44100. Only used if plugin have +zero inputs. +

+
+
nb_samples, n
+

Set the number of samples per channel per each output frame, default +is 1024. Only used if plugin have zero inputs. +

+
+
duration, d
+

Set the minimum duration of the sourced audio. See +(ffmpeg-utils)the Time duration section in the ffmpeg-utils(1) manual +for the accepted syntax. +Note that the resulting duration may be greater than the specified duration, +as the generated audio is always cut at the end of a complete frame. +If not specified, or the expressed duration is negative, the audio is +supposed to be generated forever. +Only used if plugin have zero inputs. +

+
+
latency, l
+

Enable latency compensation, by default is disabled. +Only used if plugin have inputs. +

+
+ +

8.96.1 Examples

+ +
    +
  • List all available plugins within amp (LADSPA example plugin) library: +
    +
    ladspa=file=amp
    +
    + +
  • List all available controls and their valid ranges for vcf_notch +plugin from VCF library: +
    +
    ladspa=f=vcf:p=vcf_notch:c=help
    +
    + +
  • Simulate low quality audio equipment using Computer Music Toolkit (CMT) +plugin library: +
    +
    ladspa=file=cmt:plugin=lofi:controls=c0=22|c1=12|c2=12
    +
    + +
  • Add reverberation to the audio using TAP-plugins +(Tom’s Audio Processing plugins): +
    +
    ladspa=file=tap_reverb:tap_reverb
    +
    + +
  • Generate white noise, with 0.2 amplitude: +
    +
    ladspa=file=cmt:noise_source_white:c=c0=.2
    +
    + +
  • Generate 20 bpm clicks using plugin C* Click - Metronome from the +C* Audio Plugin Suite (CAPS) library: +
    +
    ladspa=file=caps:Click:c=c1=20'
    +
    + +
  • Apply C* Eq10X2 - Stereo 10-band equaliser effect: +
    +
    ladspa=caps:Eq10X2:c=c0=-48|c9=-24|c3=12|c4=2
    +
    + +
  • Increase volume by 20dB using fast lookahead limiter from Steve Harris +SWH Plugins collection: +
    +
    ladspa=fast_lookahead_limiter_1913:fastLookaheadLimiter:20|0|2
    +
    + +
  • Attenuate low frequencies using Multiband EQ from Steve Harris +SWH Plugins collection: +
    +
    ladspa=mbeq_1197:mbeq:-24|-24|-24|0|0|0|0|0|0|0|0|0|0|0|0
    +
    + +
  • Reduce stereo image using Narrower from the C* Audio Plugin Suite +(CAPS) library: +
    +
    ladspa=caps:Narrower
    +
    + +
  • Another white noise, now using C* Audio Plugin Suite (CAPS) library: +
    +
    ladspa=caps:White:.2
    +
    + +
  • Some fractal noise, using C* Audio Plugin Suite (CAPS) library: +
    +
    ladspa=caps:Fractal:c=c1=1
    +
    + +
  • Dynamic volume normalization using VLevel plugin: +
    +
    ladspa=vlevel-ladspa:vlevel_mono
    +
    +
+ +

8.96.2 Commands

+ +

This filter supports the following commands: +

+
cN
+

Modify the N-th control value. +

+

If the specified value is not valid, it is ignored and prior one is kept. +

+
+ +

8.97 loudnorm

+ +

EBU R128 loudness normalization. Includes both dynamic and linear normalization modes. +Support for both single pass (livestreams, files) and double pass (files) modes. +This algorithm can target IL, LRA, and maximum true peak. In dynamic mode, to accurately +detect true peaks, the audio stream will be upsampled to 192 kHz. +Use the -ar option or aresample filter to explicitly set an output sample rate. +

+

The filter accepts the following options: +

+
+
I, i
+

Set integrated loudness target. +Range is -70.0 - -5.0. Default value is -24.0. +

+
+
LRA, lra
+

Set loudness range target. +Range is 1.0 - 50.0. Default value is 7.0. +

+
+
TP, tp
+

Set maximum true peak. +Range is -9.0 - +0.0. Default value is -2.0. +

+
+
measured_I, measured_i
+

Measured IL of input file. +Range is -99.0 - +0.0. +

+
+
measured_LRA, measured_lra
+

Measured LRA of input file. +Range is 0.0 - 99.0. +

+
+
measured_TP, measured_tp
+

Measured true peak of input file. +Range is -99.0 - +99.0. +

+
+
measured_thresh
+

Measured threshold of input file. +Range is -99.0 - +0.0. +

+
+
offset
+

Set offset gain. Gain is applied before the true-peak limiter. +Range is -99.0 - +99.0. Default is +0.0. +

+
+
linear
+

Normalize by linearly scaling the source audio. +measured_I, measured_LRA, measured_TP, +and measured_thresh must all be specified. Target LRA shouldn’t +be lower than source LRA and the change in integrated loudness shouldn’t +result in a true peak which exceeds the target TP. If any of these +conditions aren’t met, normalization mode will revert to dynamic. +Options are true or false. Default is true. +

+
+
dual_mono
+

Treat mono input files as "dual-mono". If a mono file is intended for playback +on a stereo system, its EBU R128 measurement will be perceptually incorrect. +If set to true, this option will compensate for this effect. +Multi-channel input files are not affected by this option. +Options are true or false. Default is false. +

+
+
print_format
+

Set print format for stats. Options are summary, json, or none. +Default value is none. +

+
+
stats_file
+

Write stats to specified file. Format is controlled by print_format, +which must be set. Specify - to write to standard output. +Default is unset. +

+
+ +

8.98 lowpass

+ +

Apply a low-pass filter with 3dB point frequency. +The filter can be either single-pole or double-pole (the default). +The filter roll off at 6dB per pole per octave (20dB per pole per decade). +

+

The filter accepts the following options: +

+
+
frequency, f
+

Set frequency in Hz. Default is 500. +

+
+
poles, p
+

Set number of poles. Default is 2. +

+
+
width_type, t
+

Set method to specify band-width of filter. +

+
h
+

Hz +

+
q
+

Q-Factor +

+
o
+

octave +

+
s
+

slope +

+
k
+

kHz +

+
+ +
+
width, w
+

Specify the band-width of a filter in width_type units. +Applies only to double-pole filter. +The default is 0.707q and gives a Butterworth response. +

+
+
mix, m
+

How much to use filtered signal in output. Default is 1. +Range is between 0 and 1. +

+
+
channels, c
+

Specify which channels to filter, by default all available are filtered. +

+
+
normalize, n
+

Normalize biquad coefficients, by default is disabled. +Enabling it will normalize magnitude response at DC to 0dB. +

+
+
transform, a
+

Set transform type of IIR filter. +

+
di
+
dii
+
tdi
+
tdii
+
latt
+
svf
+
zdf
+
+ +
+
precision, r
+

Set precision of filtering. +

+
auto
+

Pick automatic sample format depending on surround filters. +

+
s16
+

Always use signed 16-bit. +

+
s32
+

Always use signed 32-bit. +

+
f32
+

Always use float 32-bit. +

+
f64
+

Always use float 64-bit. +

+
+ +
+
block_size, b
+

Set block size used for reverse IIR processing. If this value is set to high enough +value (higher than impulse response length truncated when reaches near zero values) filtering +will become linear phase otherwise if not big enough it will just produce nasty artifacts. +

+

Note that filter delay will be exactly this many samples when set to non-zero value. +

+
+ +

8.98.1 Examples

+
    +
  • Lowpass only LFE channel, it LFE is not present it does nothing: +
    +
    lowpass=c=LFE
    +
    +
+ +

8.98.2 Commands

+ +

This filter supports the following commands: +

+
frequency, f
+

Change lowpass frequency. +Syntax for the command is : "frequency" +

+
+
width_type, t
+

Change lowpass width_type. +Syntax for the command is : "width_type" +

+
+
width, w
+

Change lowpass width. +Syntax for the command is : "width" +

+
+
mix, m
+

Change lowpass mix. +Syntax for the command is : "mix" +

+
+ +

8.99 lv2

+ +

Load a LV2 (LADSPA Version 2) plugin. +

+

To enable compilation of this filter you need to configure FFmpeg with +--enable-lv2. +

+
+
plugin, p
+

Specifies the plugin URI. You may need to escape ’:’. +

+
+
controls, c
+

Set the ’|’ separated list of controls which are zero or more floating point +values that determine the behavior of the loaded plugin (for example delay, +threshold or gain). +If controls is set to help, all available controls and +their valid ranges are printed. +

+
+
sample_rate, s
+

Specify the sample rate, default to 44100. Only used if plugin have +zero inputs. +

+
+
nb_samples, n
+

Set the number of samples per channel per each output frame, default +is 1024. Only used if plugin have zero inputs. +

+
+
duration, d
+

Set the minimum duration of the sourced audio. See +(ffmpeg-utils)the Time duration section in the ffmpeg-utils(1) manual +for the accepted syntax. +Note that the resulting duration may be greater than the specified duration, +as the generated audio is always cut at the end of a complete frame. +If not specified, or the expressed duration is negative, the audio is +supposed to be generated forever. +Only used if plugin have zero inputs. +

+
+ +

8.99.1 Examples

+ +
    +
  • Apply bass enhancer plugin from Calf: +
    +
    lv2=p=http\\\\://calf.sourceforge.net/plugins/BassEnhancer:c=amount=2
    +
    + +
  • Apply vinyl plugin from Calf: +
    +
    lv2=p=http\\\\://calf.sourceforge.net/plugins/Vinyl:c=drone=0.2|aging=0.5
    +
    + +
  • Apply bit crusher plugin from ArtyFX: +
    +
    lv2=p=http\\\\://www.openavproductions.com/artyfx#bitta:c=crush=0.3
    +
    +
+ +

8.99.2 Commands

+ +

This filter supports all options that are exported by plugin as commands. +

+

8.100 mcompand

+

Multiband Compress or expand the audio’s dynamic range. +

+

The input audio is divided into bands using 4th order Linkwitz-Riley IIRs. +This is akin to the crossover of a loudspeaker, and results in flat frequency +response when absent compander action. +

+

It accepts the following parameters: +

+
+
args
+

This option syntax is: +attack,decay,[attack,decay..] soft-knee points crossover_frequency [delay [initial_volume [gain]]] | attack,decay ... +For explanation of each item refer to compand filter documentation. +

+
+ +

8.101 pan

+ +

Mix channels with specific gain levels. The filter accepts the output +channel layout followed by a set of channels definitions. +

+

This filter is also designed to efficiently remap the channels of an audio +stream. +

+

The filter accepts parameters of the form: +"l|outdef|outdef|..." +

+
+
l
+

output channel layout or number of channels +

+
+
outdef
+

output channel specification, of the form: +"out_name=[gain*]in_name[(+-)[gain*]in_name...]" +

+
+
out_name
+

output channel to define, either a channel name (FL, FR, etc.) or a channel +number (c0, c1, etc.) +

+
+
gain
+

multiplicative coefficient for the channel, 1 leaving the volume unchanged +

+
+
in_name
+

input channel to use, see out_name for details; it is not possible to mix +named and numbered input channels +

+
+ +

If the ‘=’ in a channel specification is replaced by ‘<’, then the gains for +that specification will be renormalized so that the total is 1, thus +avoiding clipping noise. +

+

8.101.1 Mixing examples

+ +

For example, if you want to down-mix from stereo to mono, but with a bigger +factor for the left channel: +

+
pan=1c|c0=0.9*c0+0.1*c1
+
+ +

A customized down-mix to stereo that works automatically for 3-, 4-, 5- and +7-channels surround: +

+
pan=stereo| FL < FL + 0.5*FC + 0.6*BL + 0.6*SL | FR < FR + 0.5*FC + 0.6*BR + 0.6*SR
+
+ +

Note that ffmpeg integrates a default down-mix (and up-mix) system +that should be preferred (see "-ac" option) unless you have very specific +needs. +

+

8.101.2 Remapping examples

+ +

The channel remapping will be effective if, and only if: +

+
    +
  • gain coefficients are zeroes or ones, +
  • only one input per channel output, +
+ +

If all these conditions are satisfied, the filter will notify the user ("Pure +channel mapping detected"), and use an optimized and lossless method to do the +remapping. +

+

For example, if you have a 5.1 source and want a stereo audio stream by +dropping the extra channels: +

+
pan="stereo| c0=FL | c1=FR"
+
+ +

Given the same source, you can also switch front left and front right channels +and keep the input channel layout: +

+
pan="5.1| c0=c1 | c1=c0 | c2=c2 | c3=c3 | c4=c4 | c5=c5"
+
+ +

If the input is a stereo audio stream, you can mute the front left channel (and +still keep the stereo channel layout) with: +

+
pan="stereo|c1=c1"
+
+ +

Still with a stereo audio stream input, you can copy the right channel in both +front left and right: +

+
pan="stereo| c0=FR | c1=FR"
+
+ +

8.102 replaygain

+ +

ReplayGain scanner filter. This filter takes an audio stream as an input and +outputs it unchanged. +At end of filtering it displays track_gain and track_peak. +

+

The filter accepts the following exported read-only options: +

+
+
track_gain
+

Exported track gain in dB at end of stream. +

+
+
track_peak
+

Exported track peak at end of stream. +

+
+ +

8.103 resample

+ +

Convert the audio sample format, sample rate and channel layout. It is +not meant to be used directly. +

+

8.104 rubberband

+

Apply time-stretching and pitch-shifting with librubberband. +

+

To enable compilation of this filter, you need to configure FFmpeg with +--enable-librubberband. +

+

The filter accepts the following options: +

+
+
tempo
+

Set tempo scale factor. +

+
+
pitch
+

Set pitch scale factor. +

+
+
transients
+

Set transients detector. +Possible values are: +

+
crisp
+
mixed
+
smooth
+
+ +
+
detector
+

Set detector. +Possible values are: +

+
compound
+
percussive
+
soft
+
+ +
+
phase
+

Set phase. +Possible values are: +

+
laminar
+
independent
+
+ +
+
window
+

Set processing window size. +Possible values are: +

+
standard
+
short
+
long
+
+ +
+
smoothing
+

Set smoothing. +Possible values are: +

+
off
+
on
+
+ +
+
formant
+

Enable formant preservation when shift pitching. +Possible values are: +

+
shifted
+
preserved
+
+ +
+
pitchq
+

Set pitch quality. +Possible values are: +

+
quality
+
speed
+
consistency
+
+ +
+
channels
+

Set channels. +Possible values are: +

+
apart
+
together
+
+
+
+ +

8.104.1 Commands

+ +

This filter supports the following commands: +

+
tempo
+

Change filter tempo scale factor. +Syntax for the command is : "tempo" +

+
+
pitch
+

Change filter pitch scale factor. +Syntax for the command is : "pitch" +

+
+ +

8.105 sidechaincompress

+ +

This filter acts like normal compressor but has the ability to compress +detected signal using second input signal. +It needs two input streams and returns one output stream. +First input stream will be processed depending on second stream signal. +The filtered signal then can be filtered with other filters in later stages of +processing. See pan and amerge filter. +

+

The filter accepts the following options: +

+
+
level_in
+

Set input gain. Default is 1. Range is between 0.015625 and 64. +

+
+
mode
+

Set mode of compressor operation. Can be upward or downward. +Default is downward. +

+
+
threshold
+

If a signal of second stream raises above this level it will affect the gain +reduction of first stream. +By default is 0.125. Range is between 0.00097563 and 1. +

+
+
ratio
+

Set a ratio about which the signal is reduced. 1:2 means that if the level +raised 4dB above the threshold, it will be only 2dB above after the reduction. +Default is 2. Range is between 1 and 20. +

+
+
attack
+

Amount of milliseconds the signal has to rise above the threshold before gain +reduction starts. Default is 20. Range is between 0.01 and 2000. +

+
+
release
+

Amount of milliseconds the signal has to fall below the threshold before +reduction is decreased again. Default is 250. Range is between 0.01 and 9000. +

+
+
makeup
+

Set the amount by how much signal will be amplified after processing. +Default is 1. Range is from 1 to 64. +

+
+
knee
+

Curve the sharp knee around the threshold to enter gain reduction more softly. +Default is 2.82843. Range is between 1 and 8. +

+
+
link
+

Choose if the average level between all channels of side-chain stream +or the louder(maximum) channel of side-chain stream affects the +reduction. Default is average. +

+
+
detection
+

Should the exact signal be taken in case of peak or an RMS one in case +of rms. Default is rms which is mainly smoother. +

+
+
level_sc
+

Set sidechain gain. Default is 1. Range is between 0.015625 and 64. +

+
+
mix
+

How much to use compressed signal in output. Default is 1. +Range is between 0 and 1. +

+
+ +

8.105.1 Commands

+ +

This filter supports the all above options as commands. +

+

8.105.2 Examples

+ +
    +
  • Full ffmpeg example taking 2 audio inputs, 1st input to be compressed +depending on the signal of 2nd input and later compressed signal to be +merged with 2nd input: +
    +
    ffmpeg -i main.flac -i sidechain.flac -filter_complex "[1:a]asplit=2[sc][mix];[0:a][sc]sidechaincompress[compr];[compr][mix]amerge"
    +
    +
+ +

8.106 sidechaingate

+ +

A sidechain gate acts like a normal (wideband) gate but has the ability to +filter the detected signal before sending it to the gain reduction stage. +Normally a gate uses the full range signal to detect a level above the +threshold. +For example: If you cut all lower frequencies from your sidechain signal +the gate will decrease the volume of your track only if not enough highs +appear. With this technique you are able to reduce the resonation of a +natural drum or remove "rumbling" of muted strokes from a heavily distorted +guitar. +It needs two input streams and returns one output stream. +First input stream will be processed depending on second stream signal. +

+

The filter accepts the following options: +

+
+
level_in
+

Set input level before filtering. +Default is 1. Allowed range is from 0.015625 to 64. +

+
+
mode
+

Set the mode of operation. Can be upward or downward. +Default is downward. If set to upward mode, higher parts of signal +will be amplified, expanding dynamic range in upward direction. +Otherwise, in case of downward lower parts of signal will be reduced. +

+
+
range
+

Set the level of gain reduction when the signal is below the threshold. +Default is 0.06125. Allowed range is from 0 to 1. +Setting this to 0 disables reduction and then filter behaves like expander. +

+
+
threshold
+

If a signal rises above this level the gain reduction is released. +Default is 0.125. Allowed range is from 0 to 1. +

+
+
ratio
+

Set a ratio about which the signal is reduced. +Default is 2. Allowed range is from 1 to 9000. +

+
+
attack
+

Amount of milliseconds the signal has to rise above the threshold before gain +reduction stops. +Default is 20 milliseconds. Allowed range is from 0.01 to 9000. +

+
+
release
+

Amount of milliseconds the signal has to fall below the threshold before the +reduction is increased again. Default is 250 milliseconds. +Allowed range is from 0.01 to 9000. +

+
+
makeup
+

Set amount of amplification of signal after processing. +Default is 1. Allowed range is from 1 to 64. +

+
+
knee
+

Curve the sharp knee around the threshold to enter gain reduction more softly. +Default is 2.828427125. Allowed range is from 1 to 8. +

+
+
detection
+

Choose if exact signal should be taken for detection or an RMS like one. +Default is rms. Can be peak or rms. +

+
+
link
+

Choose if the average level between all channels or the louder channel affects +the reduction. +Default is average. Can be average or maximum. +

+
+
level_sc
+

Set sidechain gain. Default is 1. Range is from 0.015625 to 64. +

+
+ +

8.106.1 Commands

+ +

This filter supports the all above options as commands. +

+

8.107 silencedetect

+ +

Detect silence in an audio stream. +

+

This filter logs a message when it detects that the input audio volume is less +or equal to a noise tolerance value for a duration greater or equal to the +minimum detected noise duration. +

+

The printed times and duration are expressed in seconds. The +lavfi.silence_start or lavfi.silence_start.X metadata key +is set on the first frame whose timestamp equals or exceeds the detection +duration and it contains the timestamp of the first frame of the silence. +

+

The lavfi.silence_duration or lavfi.silence_duration.X +and lavfi.silence_end or lavfi.silence_end.X metadata +keys are set on the first frame after the silence. If mono is +enabled, and each channel is evaluated separately, the .X +suffixed keys are used, and X corresponds to the channel number. +

+

The filter accepts the following options: +

+
+
noise, n
+

Set noise tolerance. Can be specified in dB (in case "dB" is appended to the +specified value) or amplitude ratio. Default is -60dB, or 0.001. +

+
+
duration, d
+

Set silence duration until notification (default is 2 seconds). See +(ffmpeg-utils)the Time duration section in the ffmpeg-utils(1) manual +for the accepted syntax. +

+
+
mono, m
+

Process each channel separately, instead of combined. By default is disabled. +

+
+ +

8.107.1 Examples

+ +
    +
  • Detect 5 seconds of silence with -50dB noise tolerance: +
    +
    silencedetect=n=-50dB:d=5
    +
    + +
  • Complete example with ffmpeg to detect silence with 0.0001 noise +tolerance in silence.mp3: +
    +
    ffmpeg -i silence.mp3 -af silencedetect=noise=0.0001 -f null -
    +
    +
+ +

8.108 silenceremove

+ +

Remove silence from the beginning, middle or end of the audio. +

+

The filter accepts the following options: +

+
+
start_periods
+

This value is used to indicate if audio should be trimmed at beginning of +the audio. A value of zero indicates no silence should be trimmed from the +beginning. When specifying a non-zero value, it trims audio up until it +finds non-silence. Normally, when trimming silence from beginning of audio +the start_periods will be 1 but it can be increased to higher +values to trim all audio up to specific count of non-silence periods. +Default value is 0. +

+
+
start_duration
+

Specify the amount of time that non-silence must be detected before it stops +trimming audio. By increasing the duration, bursts of noises can be treated +as silence and trimmed off. Default value is 0. +

+
+
start_threshold
+

This indicates what sample value should be treated as silence. For digital +audio, a value of 0 may be fine but for audio recorded from analog, +you may wish to increase the value to account for background noise. +Can be specified in dB (in case "dB" is appended to the specified value) +or amplitude ratio. Default value is 0. +

+
+
start_silence
+

Specify max duration of silence at beginning that will be kept after +trimming. Default is 0, which is equal to trimming all samples detected +as silence. +

+
+
start_mode
+

Specify mode of detection of silence end at start of multi-channel audio. +Can be any or all. Default is any. +With any, any sample from any channel that is detected as non-silence +will trigger end of silence trimming at start of audio stream. +With all, only if every sample from every channel is detected as non-silence +will trigger end of silence trimming at start of audio stream, limited usage. +

+
+
stop_periods
+

Set the count for trimming silence from the end of audio. When specifying a +positive value, it trims audio after it finds specified silence period. +To remove silence from the middle of a file, specify a stop_periods +that is negative. This value is then treated as a positive value and is +used to indicate the effect should restart processing as specified by +stop_periods, making it suitable for removing periods of silence +in the middle of the audio. +Default value is 0. +

+
+
stop_duration
+

Specify a duration of silence that must exist before audio is not copied any +more. By specifying a higher duration, silence that is wanted can be left in +the audio. +Default value is 0. +

+
+
stop_threshold
+

This is the same as start_threshold but for trimming silence from +the end of audio. +Can be specified in dB (in case "dB" is appended to the specified value) +or amplitude ratio. Default value is 0. +

+
+
stop_silence
+

Specify max duration of silence at end that will be kept after +trimming. Default is 0, which is equal to trimming all samples detected +as silence. +

+
+
stop_mode
+

Specify mode of detection of silence start after start of multi-channel audio. +Can be any or all. Default is all. +With any, any sample from any channel that is detected as silence +will trigger start of silence trimming after start of audio stream, limited usage. +With all, only if every sample from every channel is detected as silence +will trigger start of silence trimming after start of audio stream. +

+
+
detection
+

Set how is silence detected. +

+
avg
+

Mean of absolute values of samples in moving window. +

+
rms
+

Root squared mean of absolute values of samples in moving window. +

+
peak
+

Maximum of absolute values of samples in moving window. +

+
median
+

Median of absolute values of samples in moving window. +

+
ptp
+

Absolute of max peak to min peak difference of samples in moving window. +

+
dev
+

Standard deviation of values of samples in moving window. +

+
+

Default value is rms. +

+
+
window
+

Set duration in number of seconds used to calculate size of window in number +of samples for detecting silence. Using 0 will effectively disable +any windowing and use only single sample per channel for silence detection. +In that case it may be needed to also set start_silence and/or +stop_silence to nonzero values with also start_duration and/or +stop_duration to nonzero values. +Default value is 0.02. Allowed range is from 0 to 10. +

+
+
timestamp
+

Set processing mode of every audio frame output timestamp. +

+
write
+

Full timestamps rewrite, keep only the start time for the first output frame. +

+
copy
+

Non-dropped frames are left with same timestamp as input audio frame. +

+
+

Defaults value is write. +

+
+ +

8.108.1 Examples

+ +
    +
  • The following example shows how this filter can be used to start a recording +that does not contain the delay at the start which usually occurs between +pressing the record button and the start of the performance: +
    +
    silenceremove=start_periods=1:start_duration=5:start_threshold=0.02
    +
    + +
  • Trim all silence encountered from beginning to end where there is more than 1 +second of silence in audio: +
    +
    silenceremove=stop_periods=-1:stop_duration=1:stop_threshold=-90dB
    +
    + +
  • Trim all digital silence samples, using peak detection, from beginning to end +where there is more than 0 samples of digital silence in audio and digital +silence is detected in all channels at same positions in stream: +
    +
    silenceremove=window=0:detection=peak:stop_mode=all:start_mode=all:stop_periods=-1:stop_threshold=0
    +
    + +
  • Trim every 2nd encountered silence period from beginning to end where there is +more than 1 second of silence per silence period in audio: +
    +
    silenceremove=stop_periods=-2:stop_duration=1:stop_threshold=-90dB
    +
    + +
  • Similar as above, but keep maximum of 0.5 seconds of silence from each trimmed period: +
    +
    silenceremove=stop_periods=-2:stop_duration=1:stop_threshold=-90dB:stop_silence=0.5
    +
    + +
  • Similar as above, but keep maximum of 1.5 seconds of silence from start of audio: +
    +
    silenceremove=stop_periods=-2:stop_duration=1:stop_threshold=-90dB:stop_silence=0.5:start_periods=1:start_duration=1:start_silence=1.5:stop_threshold=-90dB
    +
    +
+ +

8.108.2 Commands

+ +

This filter supports some above options as commands. +

+

8.109 sofalizer

+ +

SOFAlizer uses head-related transfer functions (HRTFs) to create virtual +loudspeakers around the user for binaural listening via headphones (audio +formats up to 9 channels supported). +The HRTFs are stored in SOFA files (see http://www.sofacoustics.org/ for a database). +SOFAlizer is developed at the Acoustics Research Institute (ARI) of the +Austrian Academy of Sciences. +

+

To enable compilation of this filter you need to configure FFmpeg with +--enable-libmysofa. +

+

The filter accepts the following options: +

+
+
sofa
+

Set the SOFA file used for rendering. +

+
+
gain
+

Set gain applied to audio. Value is in dB. Default is 0. +

+
+
rotation
+

Set rotation of virtual loudspeakers in deg. Default is 0. +

+
+
elevation
+

Set elevation of virtual speakers in deg. Default is 0. +

+
+
radius
+

Set distance in meters between loudspeakers and the listener with near-field +HRTFs. Default is 1. +

+
+
type
+

Set processing type. Can be time or freq. time is +processing audio in time domain which is slow. +freq is processing audio in frequency domain which is fast. +Default is freq. +

+
+
speakers
+

Set custom positions of virtual loudspeakers. Syntax for this option is: +<CH> <AZIM> <ELEV>[|<CH> <AZIM> <ELEV>|...]. +Each virtual loudspeaker is described with short channel name following with +azimuth and elevation in degrees. +Each virtual loudspeaker description is separated by ’|’. +For example to override front left and front right channel positions use: +’speakers=FL 45 15|FR 345 15’. +Descriptions with unrecognised channel names are ignored. +

+
+
lfegain
+

Set custom gain for LFE channels. Value is in dB. Default is 0. +

+
+
framesize
+

Set custom frame size in number of samples. Default is 1024. +Allowed range is from 1024 to 96000. Only used if option ‘type’ +is set to freq. +

+
+
normalize
+

Should all IRs be normalized upon importing SOFA file. +By default is enabled. +

+
+
interpolate
+

Should nearest IRs be interpolated with neighbor IRs if exact position +does not match. By default is disabled. +

+
+
minphase
+

Minphase all IRs upon loading of SOFA file. By default is disabled. +

+
+
anglestep
+

Set neighbor search angle step. Only used if option interpolate is enabled. +

+
+
radstep
+

Set neighbor search radius step. Only used if option interpolate is enabled. +

+
+ +

8.109.1 Examples

+ +
    +
  • Using ClubFritz6 sofa file: +
    +
    sofalizer=sofa=/path/to/ClubFritz6.sofa:type=freq:radius=1
    +
    + +
  • Using ClubFritz12 sofa file and bigger radius with small rotation: +
    +
    sofalizer=sofa=/path/to/ClubFritz12.sofa:type=freq:radius=2:rotation=5
    +
    + +
  • Similar as above but with custom speaker positions for front left, front right, back left and back right +and also with custom gain: +
    +
    "sofalizer=sofa=/path/to/ClubFritz6.sofa:type=freq:radius=2:speakers=FL 45|FR 315|BL 135|BR 225:gain=28"
    +
    +
+ +

8.110 speechnorm

+

Speech Normalizer. +

+

This filter expands or compresses each half-cycle of audio samples +(local set of samples all above or all below zero and between two nearest zero crossings) depending +on threshold value, so audio reaches target peak value under conditions controlled by below options. +

+

The filter accepts the following options: +

+
+
peak, p
+

Set the expansion target peak value. This specifies the highest allowed absolute amplitude +level for the normalized audio input. Default value is 0.95. Allowed range is from 0.0 to 1.0. +

+
+
expansion, e
+

Set the maximum expansion factor. Allowed range is from 1.0 to 50.0. Default value is 2.0. +This option controls maximum local half-cycle of samples expansion. The maximum expansion +would be such that local peak value reaches target peak value but never to surpass it and that +ratio between new and previous peak value does not surpass this option value. +

+
+
compression, c
+

Set the maximum compression factor. Allowed range is from 1.0 to 50.0. Default value is 2.0. +This option controls maximum local half-cycle of samples compression. This option is used +only if threshold option is set to value greater than 0.0, then in such cases +when local peak is lower or same as value set by threshold all samples belonging to +that peak’s half-cycle will be compressed by current compression factor. +

+
+
threshold, t
+

Set the threshold value. Default value is 0.0. Allowed range is from 0.0 to 1.0. +This option specifies which half-cycles of samples will be compressed and which will be expanded. +Any half-cycle samples with their local peak value below or same as this option value will be +compressed by current compression factor, otherwise, if greater than threshold value they will be +expanded with expansion factor so that it could reach peak target value but never surpass it. +

+
+
raise, r
+

Set the expansion raising amount per each half-cycle of samples. Default value is 0.001. +Allowed range is from 0.0 to 1.0. This controls how fast expansion factor is raised per +each new half-cycle until it reaches expansion value. +Setting this options too high may lead to distortions. +

+
+
fall, f
+

Set the compression raising amount per each half-cycle of samples. Default value is 0.001. +Allowed range is from 0.0 to 1.0. This controls how fast compression factor is raised per +each new half-cycle until it reaches compression value. +

+
+
channels, h
+

Specify which channels to filter, by default all available channels are filtered. +

+
+
invert, i
+

Enable inverted filtering, by default is disabled. This inverts interpretation of threshold +option. When enabled any half-cycle of samples with their local peak value below or same as +threshold option will be expanded otherwise it will be compressed. +

+
+
link, l
+

Link channels when calculating gain applied to each filtered channel sample, by default is disabled. +When disabled each filtered channel gain calculation is independent, otherwise when this option +is enabled the minimum of all possible gains for each filtered channel is used. +

+
+
rms, m
+

Set the expansion target RMS value. This specifies the highest allowed RMS level for the normalized +audio input. Default value is 0.0, thus disabled. Allowed range is from 0.0 to 1.0. +

+
+ +

8.110.1 Commands

+ +

This filter supports the all above options as commands. +

+

8.110.2 Examples

+ +
    +
  • Weak and slow amplification: +
    +
    speechnorm=e=3:r=0.00001:l=1
    +
    + +
  • Moderate and slow amplification: +
    +
    speechnorm=e=6.25:r=0.00001:l=1
    +
    + +
  • Strong and fast amplification: +
    +
    speechnorm=e=12.5:r=0.0001:l=1
    +
    + +
  • Very strong and fast amplification: +
    +
    speechnorm=e=25:r=0.0001:l=1
    +
    + +
  • Extreme and fast amplification: +
    +
    speechnorm=e=50:r=0.0001:l=1
    +
    +
+ +

8.111 stereotools

+ +

This filter has some handy utilities to manage stereo signals, for converting +M/S stereo recordings to L/R signal while having control over the parameters +or spreading the stereo image of master track. +

+

The filter accepts the following options: +

+
+
level_in
+

Set input level before filtering for both channels. Defaults is 1. +Allowed range is from 0.015625 to 64. +

+
+
level_out
+

Set output level after filtering for both channels. Defaults is 1. +Allowed range is from 0.015625 to 64. +

+
+
balance_in
+

Set input balance between both channels. Default is 0. +Allowed range is from -1 to 1. +

+
+
balance_out
+

Set output balance between both channels. Default is 0. +Allowed range is from -1 to 1. +

+
+
softclip
+

Enable softclipping. Results in analog distortion instead of harsh digital 0dB +clipping. Disabled by default. +

+
+
mutel
+

Mute the left channel. Disabled by default. +

+
+
muter
+

Mute the right channel. Disabled by default. +

+
+
phasel
+

Change the phase of the left channel. Disabled by default. +

+
+
phaser
+

Change the phase of the right channel. Disabled by default. +

+
+
mode
+

Set stereo mode. Available values are: +

+
+
‘lr>lr’
+

Left/Right to Left/Right, this is default. +

+
+
‘lr>ms’
+

Left/Right to Mid/Side. +

+
+
‘ms>lr’
+

Mid/Side to Left/Right. +

+
+
‘lr>ll’
+

Left/Right to Left/Left. +

+
+
‘lr>rr’
+

Left/Right to Right/Right. +

+
+
‘lr>l+r’
+

Left/Right to Left + Right. +

+
+
‘lr>rl’
+

Left/Right to Right/Left. +

+
+
‘ms>ll’
+

Mid/Side to Left/Left. +

+
+
‘ms>rr’
+

Mid/Side to Right/Right. +

+
+
‘ms>rl’
+

Mid/Side to Right/Left. +

+
+
‘lr>l-r’
+

Left/Right to Left - Right. +

+
+ +
+
slev
+

Set level of side signal. Default is 1. +Allowed range is from 0.015625 to 64. +

+
+
sbal
+

Set balance of side signal. Default is 0. +Allowed range is from -1 to 1. +

+
+
mlev
+

Set level of the middle signal. Default is 1. +Allowed range is from 0.015625 to 64. +

+
+
mpan
+

Set middle signal pan. Default is 0. Allowed range is from -1 to 1. +

+
+
base
+

Set stereo base between mono and inversed channels. Default is 0. +Allowed range is from -1 to 1. +

+
+
delay
+

Set delay in milliseconds how much to delay left from right channel and +vice versa. Default is 0. Allowed range is from -20 to 20. +

+
+
sclevel
+

Set S/C level. Default is 1. Allowed range is from 1 to 100. +

+
+
phase
+

Set the stereo phase in degrees. Default is 0. Allowed range is from 0 to 360. +

+
+
bmode_in, bmode_out
+

Set balance mode for balance_in/balance_out option. +

+

Can be one of the following: +

+
+
‘balance’
+

Classic balance mode. Attenuate one channel at time. +Gain is raised up to 1. +

+
+
‘amplitude’
+

Similar as classic mode above but gain is raised up to 2. +

+
+
‘power’
+

Equal power distribution, from -6dB to +6dB range. +

+
+
+
+ +

8.111.1 Commands

+ +

This filter supports the all above options as commands. +

+

8.111.2 Examples

+ +
    +
  • Apply karaoke like effect: +
    +
    stereotools=mlev=0.015625
    +
    + +
  • Convert M/S signal to L/R: +
    +
    "stereotools=mode=ms>lr"
    +
    +
+ +

8.112 stereowiden

+ +

This filter enhance the stereo effect by suppressing signal common to both +channels and by delaying the signal of left into right and vice versa, +thereby widening the stereo effect. +

+

The filter accepts the following options: +

+
+
delay
+

Time in milliseconds of the delay of left signal into right and vice versa. +Default is 20 milliseconds. +

+
+
feedback
+

Amount of gain in delayed signal into right and vice versa. Gives a delay +effect of left signal in right output and vice versa which gives widening +effect. Default is 0.3. +

+
+
crossfeed
+

Cross feed of left into right with inverted phase. This helps in suppressing +the mono. If the value is 1 it will cancel all the signal common to both +channels. Default is 0.3. +

+
+
drymix
+

Set level of input signal of original channel. Default is 0.8. +

+
+ +

8.112.1 Commands

+ +

This filter supports the all above options except delay as commands. +

+

8.113 superequalizer

+

Apply 18 band equalizer. +

+

The filter accepts the following options: +

+
1b
+

Set 65Hz band gain. +

+
2b
+

Set 92Hz band gain. +

+
3b
+

Set 131Hz band gain. +

+
4b
+

Set 185Hz band gain. +

+
5b
+

Set 262Hz band gain. +

+
6b
+

Set 370Hz band gain. +

+
7b
+

Set 523Hz band gain. +

+
8b
+

Set 740Hz band gain. +

+
9b
+

Set 1047Hz band gain. +

+
10b
+

Set 1480Hz band gain. +

+
11b
+

Set 2093Hz band gain. +

+
12b
+

Set 2960Hz band gain. +

+
13b
+

Set 4186Hz band gain. +

+
14b
+

Set 5920Hz band gain. +

+
15b
+

Set 8372Hz band gain. +

+
16b
+

Set 11840Hz band gain. +

+
17b
+

Set 16744Hz band gain. +

+
18b
+

Set 20000Hz band gain. +

+
+ +

8.114 surround

+

Apply audio surround upmix filter. +

+

This filter allows to produce multichannel output from audio stream. +

+

The filter accepts the following options: +

+
+
chl_out
+

Set output channel layout. By default, this is 5.1. +

+

See (ffmpeg-utils)the Channel Layout section in the ffmpeg-utils(1) manual +for the required syntax. +

+
+
chl_in
+

Set input channel layout. By default, this is stereo. +

+

See (ffmpeg-utils)the Channel Layout section in the ffmpeg-utils(1) manual +for the required syntax. +

+
+
level_in
+

Set input volume level. By default, this is 1. +

+
+
level_out
+

Set output volume level. By default, this is 1. +

+
+
lfe
+

Enable LFE channel output if output channel layout has it. By default, this is enabled. +

+
+
lfe_low
+

Set LFE low cut off frequency. By default, this is 128 Hz. +

+
+
lfe_high
+

Set LFE high cut off frequency. By default, this is 256 Hz. +

+
+
lfe_mode
+

Set LFE mode, can be add or sub. Default is add. +In add mode, LFE channel is created from input audio and added to output. +In sub mode, LFE channel is created from input audio and added to output but +also all non-LFE output channels are subtracted with output LFE channel. +

+
+
smooth
+

Set temporal smoothness strength, used to gradually change factors when transforming +stereo sound in time. Allowed range is from 0.0 to 1.0. +Useful to improve output quality with focus option values greater than 0.0. +Default is 0.0. Only values inside this range and without edges are effective. +

+
+
angle
+

Set angle of stereo surround transform, Allowed range is from 0 to 360. +Default is 90. +

+
+
focus
+

Set focus of stereo surround transform, Allowed range is from -1 to 1. +Default is 0. +

+
+
fc_in
+

Set front center input volume. By default, this is 1. +

+
+
fc_out
+

Set front center output volume. By default, this is 1. +

+
+
fl_in
+

Set front left input volume. By default, this is 1. +

+
+
fl_out
+

Set front left output volume. By default, this is 1. +

+
+
fr_in
+

Set front right input volume. By default, this is 1. +

+
+
fr_out
+

Set front right output volume. By default, this is 1. +

+
+
sl_in
+

Set side left input volume. By default, this is 1. +

+
+
sl_out
+

Set side left output volume. By default, this is 1. +

+
+
sr_in
+

Set side right input volume. By default, this is 1. +

+
+
sr_out
+

Set side right output volume. By default, this is 1. +

+
+
bl_in
+

Set back left input volume. By default, this is 1. +

+
+
bl_out
+

Set back left output volume. By default, this is 1. +

+
+
br_in
+

Set back right input volume. By default, this is 1. +

+
+
br_out
+

Set back right output volume. By default, this is 1. +

+
+
bc_in
+

Set back center input volume. By default, this is 1. +

+
+
bc_out
+

Set back center output volume. By default, this is 1. +

+
+
lfe_in
+

Set LFE input volume. By default, this is 1. +

+
+
lfe_out
+

Set LFE output volume. By default, this is 1. +

+
+
allx
+

Set spread usage of stereo image across X axis for all channels. +Allowed range is from -1 to 15. +By default this value is negative -1, and thus unused. +

+
+
ally
+

Set spread usage of stereo image across Y axis for all channels. +Allowed range is from -1 to 15. +By default this value is negative -1, and thus unused. +

+
+
fcx, flx, frx, blx, brx, slx, srx, bcx
+

Set spread usage of stereo image across X axis for each channel. +Allowed range is from 0.06 to 15. +By default this value is 0.5. +

+
+
fcy, fly, fry, bly, bry, sly, sry, bcy
+

Set spread usage of stereo image across Y axis for each channel. +Allowed range is from 0.06 to 15. +By default this value is 0.5. +

+
+
win_size
+

Set window size. Allowed range is from 1024 to 65536. Default size is 4096. +

+
+
win_func
+

Set window function. +

+

It accepts the following values: +

+
‘rect’
+
‘bartlett’
+
‘hann, hanning’
+
‘hamming’
+
‘blackman’
+
‘welch’
+
‘flattop’
+
‘bharris’
+
‘bnuttall’
+
‘bhann’
+
‘sine’
+
‘nuttall’
+
‘lanczos’
+
‘gauss’
+
‘tukey’
+
‘dolph’
+
‘cauchy’
+
‘parzen’
+
‘poisson’
+
‘bohman’
+
‘kaiser’
+
+

Default is hann. +

+
+
overlap
+

Set window overlap. If set to 1, the recommended overlap for selected +window function will be picked. Default is 0.5. +

+
+ +

8.115 tiltshelf

+ +

Boost or cut the lower frequencies and cut or boost higher frequencies +of the audio using a two-pole shelving filter with a response similar to +that of a standard hi-fi’s tone-controls. +This is also known as shelving equalisation (EQ). +

+

The filter accepts the following options: +

+
+
gain, g
+

Give the gain at 0 Hz. Its useful range is about -20 +(for a large cut) to +20 (for a large boost). +Beware of clipping when using a positive gain. +

+
+
frequency, f
+

Set the filter’s central frequency and so can be used +to extend or reduce the frequency range to be boosted or cut. +The default value is 3000 Hz. +

+
+
width_type, t
+

Set method to specify band-width of filter. +

+
h
+

Hz +

+
q
+

Q-Factor +

+
o
+

octave +

+
s
+

slope +

+
k
+

kHz +

+
+ +
+
width, w
+

Determine how steep is the filter’s shelf transition. +

+
+
poles, p
+

Set number of poles. Default is 2. +

+
+
mix, m
+

How much to use filtered signal in output. Default is 1. +Range is between 0 and 1. +

+
+
channels, c
+

Specify which channels to filter, by default all available are filtered. +

+
+
normalize, n
+

Normalize biquad coefficients, by default is disabled. +Enabling it will normalize magnitude response at DC to 0dB. +

+
+
transform, a
+

Set transform type of IIR filter. +

+
di
+
dii
+
tdi
+
tdii
+
latt
+
svf
+
zdf
+
+ +
+
precision, r
+

Set precision of filtering. +

+
auto
+

Pick automatic sample format depending on surround filters. +

+
s16
+

Always use signed 16-bit. +

+
s32
+

Always use signed 32-bit. +

+
f32
+

Always use float 32-bit. +

+
f64
+

Always use float 64-bit. +

+
+ +
+
block_size, b
+

Set block size used for reverse IIR processing. If this value is set to high enough +value (higher than impulse response length truncated when reaches near zero values) filtering +will become linear phase otherwise if not big enough it will just produce nasty artifacts. +

+

Note that filter delay will be exactly this many samples when set to non-zero value. +

+
+ +

8.115.1 Commands

+ +

This filter supports some options as commands. +

+

8.116 treble, highshelf

+ +

Boost or cut treble (upper) frequencies of the audio using a two-pole +shelving filter with a response similar to that of a standard +hi-fi’s tone-controls. This is also known as shelving equalisation (EQ). +

+

The filter accepts the following options: +

+
+
gain, g
+

Give the gain at whichever is the lower of ~22 kHz and the +Nyquist frequency. Its useful range is about -20 (for a large cut) +to +20 (for a large boost). Beware of clipping when using a positive gain. +

+
+
frequency, f
+

Set the filter’s central frequency and so can be used +to extend or reduce the frequency range to be boosted or cut. +The default value is 3000 Hz. +

+
+
width_type, t
+

Set method to specify band-width of filter. +

+
h
+

Hz +

+
q
+

Q-Factor +

+
o
+

octave +

+
s
+

slope +

+
k
+

kHz +

+
+ +
+
width, w
+

Determine how steep is the filter’s shelf transition. +

+
+
poles, p
+

Set number of poles. Default is 2. +

+
+
mix, m
+

How much to use filtered signal in output. Default is 1. +Range is between 0 and 1. +

+
+
channels, c
+

Specify which channels to filter, by default all available are filtered. +

+
+
normalize, n
+

Normalize biquad coefficients, by default is disabled. +Enabling it will normalize magnitude response at DC to 0dB. +

+
+
transform, a
+

Set transform type of IIR filter. +

+
di
+
dii
+
tdi
+
tdii
+
latt
+
svf
+
zdf
+
+ +
+
precision, r
+

Set precision of filtering. +

+
auto
+

Pick automatic sample format depending on surround filters. +

+
s16
+

Always use signed 16-bit. +

+
s32
+

Always use signed 32-bit. +

+
f32
+

Always use float 32-bit. +

+
f64
+

Always use float 64-bit. +

+
+ +
+
block_size, b
+

Set block size used for reverse IIR processing. If this value is set to high enough +value (higher than impulse response length truncated when reaches near zero values) filtering +will become linear phase otherwise if not big enough it will just produce nasty artifacts. +

+

Note that filter delay will be exactly this many samples when set to non-zero value. +

+
+ +

8.116.1 Commands

+ +

This filter supports the following commands: +

+
frequency, f
+

Change treble frequency. +Syntax for the command is : "frequency" +

+
+
width_type, t
+

Change treble width_type. +Syntax for the command is : "width_type" +

+
+
width, w
+

Change treble width. +Syntax for the command is : "width" +

+
+
gain, g
+

Change treble gain. +Syntax for the command is : "gain" +

+
+
mix, m
+

Change treble mix. +Syntax for the command is : "mix" +

+
+ +

8.117 tremolo

+ +

Sinusoidal amplitude modulation. +

+

The filter accepts the following options: +

+
+
f
+

Modulation frequency in Hertz. Modulation frequencies in the subharmonic range +(20 Hz or lower) will result in a tremolo effect. +This filter may also be used as a ring modulator by specifying +a modulation frequency higher than 20 Hz. +Range is 0.1 - 20000.0. Default value is 5.0 Hz. +

+
+
d
+

Depth of modulation as a percentage. Range is 0.0 - 1.0. +Default value is 0.5. +

+
+ +

8.118 vibrato

+ +

Sinusoidal phase modulation. +

+

The filter accepts the following options: +

+
+
f
+

Modulation frequency in Hertz. +Range is 0.1 - 20000.0. Default value is 5.0 Hz. +

+
+
d
+

Depth of modulation as a percentage. Range is 0.0 - 1.0. +Default value is 0.5. +

+
+ +

8.119 virtualbass

+ +

Apply audio Virtual Bass filter. +

+

This filter accepts stereo input and produce stereo with LFE (2.1) channels output. +The newly produced LFE channel have enhanced virtual bass originally obtained from both stereo channels. +This filter outputs front left and front right channels unchanged as available in stereo input. +

+

The filter accepts the following options: +

+
+
cutoff
+

Set the virtual bass cutoff frequency. Default value is 250 Hz. +Allowed range is from 100 to 500 Hz. +

+
+
strength
+

Set the virtual bass strength. Allowed range is from 0.5 to 3. +Default value is 3. +

+
+ +

8.120 volume

+ +

Adjust the input audio volume. +

+

It accepts the following parameters: +

+
volume
+

Set audio volume expression. +

+

Output values are clipped to the maximum value. +

+

The output audio volume is given by the relation: +

+
output_volume = volume * input_volume
+
+ +

The default value for volume is "1.0". +

+
+
precision
+

This parameter represents the mathematical precision. +

+

It determines which input sample formats will be allowed, which affects the +precision of the volume scaling. +

+
+
fixed
+

8-bit fixed-point; this limits input sample format to U8, S16, and S32. +

+
float
+

32-bit floating-point; this limits input sample format to FLT. (default) +

+
double
+

64-bit floating-point; this limits input sample format to DBL. +

+
+ +
+
replaygain
+

Choose the behaviour on encountering ReplayGain side data in input frames. +

+
+
drop
+

Remove ReplayGain side data, ignoring its contents (the default). +

+
+
ignore
+

Ignore ReplayGain side data, but leave it in the frame. +

+
+
track
+

Prefer the track gain, if present. +

+
+
album
+

Prefer the album gain, if present. +

+
+ +
+
replaygain_preamp
+

Pre-amplification gain in dB to apply to the selected replaygain gain. +

+

Default value for replaygain_preamp is 0.0. +

+
+
replaygain_noclip
+

Prevent clipping by limiting the gain applied. +

+

Default value for replaygain_noclip is 1. +

+
+
eval
+

Set when the volume expression is evaluated. +

+

It accepts the following values: +

+
‘once’
+

only evaluate expression once during the filter initialization, or +when the ‘volume’ command is sent +

+
+
‘frame’
+

evaluate expression for each incoming frame +

+
+ +

Default value is ‘once’. +

+
+ +

The volume expression can contain the following parameters. +

+
+
n
+

frame number (starting at zero) +

+
nb_channels
+

number of channels +

+
nb_consumed_samples
+

number of samples consumed by the filter +

+
nb_samples
+

number of samples in the current frame +

+
pos
+

original frame position in the file; deprecated, do not use +

+
pts
+

frame PTS +

+
sample_rate
+

sample rate +

+
startpts
+

PTS at start of stream +

+
startt
+

time at start of stream +

+
t
+

frame time +

+
tb
+

timestamp timebase +

+
volume
+

last set volume value +

+
+ +

Note that when eval is set to ‘once’ only the +sample_rate and tb variables are available, all other +variables will evaluate to NAN. +

+

8.120.1 Commands

+ +

This filter supports the following commands: +

+
volume
+

Modify the volume expression. +The command accepts the same syntax of the corresponding option. +

+

If the specified expression is not valid, it is kept at its current +value. +

+
+ +

8.120.2 Examples

+ +
    +
  • Halve the input audio volume: +
    +
    volume=volume=0.5
    +volume=volume=1/2
    +volume=volume=-6.0206dB
    +
    + +

    In all the above example the named key for volume can be +omitted, for example like in: +

    +
    volume=0.5
    +
    + +
  • Increase input audio power by 6 decibels using fixed-point precision: +
    +
    volume=volume=6dB:precision=fixed
    +
    + +
  • Fade volume after time 10 with an annihilation period of 5 seconds: +
    +
    volume='if(lt(t,10),1,max(1-(t-10)/5,0))':eval=frame
    +
    +
+ +

8.121 volumedetect

+ +

Detect the volume of the input video. +

+

The filter has no parameters. It supports only 16-bit signed integer samples, +so the input will be converted when needed. Statistics about the volume will +be printed in the log when the input stream end is reached. +

+

In particular it will show the mean volume (root mean square), maximum +volume (on a per-sample basis), and the beginning of a histogram of the +registered volume values (from the maximum value to a cumulated 1/1000 of +the samples). +

+

All volumes are in decibels relative to the maximum PCM value. +

+

8.121.1 Examples

+ +

Here is an excerpt of the output: +

+
[Parsed_volumedetect_0  0xa23120] mean_volume: -27 dB
+[Parsed_volumedetect_0  0xa23120] max_volume: -4 dB
+[Parsed_volumedetect_0  0xa23120] histogram_4db: 6
+[Parsed_volumedetect_0  0xa23120] histogram_5db: 62
+[Parsed_volumedetect_0  0xa23120] histogram_6db: 286
+[Parsed_volumedetect_0  0xa23120] histogram_7db: 1042
+[Parsed_volumedetect_0  0xa23120] histogram_8db: 2551
+[Parsed_volumedetect_0  0xa23120] histogram_9db: 4609
+[Parsed_volumedetect_0  0xa23120] histogram_10db: 8409
+
+ +

It means that: +

    +
  • The mean square energy is approximately -27 dB, or 10^-2.7. +
  • The largest sample is at -4 dB, or more precisely between -4 dB and -5 dB. +
  • There are 6 samples at -4 dB, 62 at -5 dB, 286 at -6 dB, etc. +
+ +

In other words, raising the volume by +4 dB does not cause any clipping, +raising it by +5 dB causes clipping for 6 samples, etc. +

+

8.122 whisper

+ +

It runs automatic speech recognition using the OpenAI’s Whisper model. +

+

It requires the whisper.cpp library (https://github.com/ggml-org/whisper.cpp) +as a prerequisite. After installing the library it can be enabled using: +./configure --enable-whisper. +

+

The filter has following options: +

+
+
model
+

The file path of the downloaded whisper.cpp model (mandatory). +

+
+
language
+

The language to use for transcription (’auto’ for auto-detect). +Default value: "auto" +

+
+
translate
+

If enabled, translate the transcription from the source language to English. A +multilingual model is required to enable this option. +Default value: "false" +

+
+
queue
+

The maximum size that will be queued into the filter before processing the audio +with whisper. Using a small value the audio stream will be processed more often, +but the transcription quality will be lower and the required processing power +will be higher. Using a large value (e.g. 10-20s) will produce more accurate +results using less CPU (as using the whisper-cli tool), but the transcription +latency will be higher, thus not useful to process real-time streams. +Consider using the vad_model option associated with a large queue value. +Default value: "3" +

+
+
use_gpu
+

If the GPU support should be enabled. +Default value: "true" +

+
+
gpu_device
+

The GPU device index to use. +Default value: "0" +

+
+
destination
+

If set, the transcription output will be sent to the specified file or URL +(use one of the FFmpeg AVIO protocols); otherwise, the output will be logged as +info messages. +The output will also be set in the "lavfi.whisper.text" frame metadata. +If the destination is a file and it already exists, it will be overwritten. +

+
+
format
+

The destination format string; it could be "text" (only the transcribed text +will be sent to the destination), "srt" (subtitle format) or "json". +Default value: "text" +

+
+
max_len
+

Maximum segment length in characters. When set to a value greater than 0, +transcription segments will be split by word to not exceed this length. This is +useful for generating subtitles with shorter lines. +Default value: "0" +

+
+
vad_model
+

Path to the VAD model file. If set, the filter will load an additional voice +activity detection module (https://github.com/snakers4/silero-vad) that will be +used to fragment the audio queue; use this option setting a valid path obtained +from the whisper.cpp repository (e.g. "../whisper.cpp/models/ggml-silero-v5.1.2.bin") +and increase the queue parameter to a higher value (e.g. 20). +

+
+
vad_threshold
+

The VAD threshold to use. +Default value: "0.5" +

+
+
vad_min_speech_duration
+

The minimum VAD speaking duration. +Default value: "0.1" +

+
+
vad_min_silence_duration
+

The minimum VAD silence duration. +Default value: "0.5" +

+
+
+ +

8.122.1 Examples

+
    +
  • Run a transcription with srt file generation: +
    +
    ffmpeg -i input.mp4 -vn -af "whisper=model=../whisper.cpp/models/ggml-base.en.bin\
    +:language=en\
    +:queue=3\
    +:destination=output.srt\
    +:format=srt" -f null -
    +
    + +
  • Run a transcription and send the output in JSON format to an HTTP service: +
    +
    ffmpeg -i input.mp4 -vn -af "whisper=model=../whisper.cpp/models/ggml-base.en.bin\
    +:language=en\
    +:queue=3\
    +:destination=http\\://localhost\\:3000\
    +:format=json' -f null -
    +
    + +
  • Transcribe the microphone input using the VAD option: +
    +
    ffmpeg -loglevel warning -f pulse -i default \
    +-af 'highpass=f=200,lowpass=f=3000,whisper=model=../whisper.cpp/models/ggml-medium.bin\
    +:language=en\
    +:queue=10\
    +:destination=-\
    +:format=json\
    +:vad_model=../whisper.cpp/models/ggml-silero-v5.1.2.bin' -f null -
    +
    + +
+ + +

9 Audio Sources

+ +

Below is a description of the currently available audio sources. +

+

9.1 abuffer

+ +

Buffer audio frames, and make them available to the filter chain. +

+

This source is mainly intended for a programmatic use, in particular +through the interface defined in libavfilter/buffersrc.h. +

+

It accepts the following parameters: +

+
time_base
+

The timebase which will be used for timestamps of submitted frames. It must be +either a floating-point number or in numerator/denominator form. +

+
+
sample_rate
+

The sample rate of the incoming audio buffers. +

+
+
sample_fmt
+

The sample format of the incoming audio buffers. +Either a sample format name or its corresponding integer representation from +the enum AVSampleFormat in libavutil/samplefmt.h +

+
+
channel_layout
+

The channel layout of the incoming audio buffers. +Either a channel layout name from channel_layout_map in +libavutil/channel_layout.c or its corresponding integer representation +from the AV_CH_LAYOUT_* macros in libavutil/channel_layout.h +

+
+
channels
+

The number of channels of the incoming audio buffers. +If both channels and channel_layout are specified, then they +must be consistent. +

+
+
+ +

9.1.1 Examples

+ +
+
abuffer=sample_rate=44100:sample_fmt=s16p:channel_layout=stereo
+
+ +

will instruct the source to accept planar 16bit signed stereo at 44100Hz. +Since the sample format with name "s16p" corresponds to the number +6 and the "stereo" channel layout corresponds to the value 0x3, this is +equivalent to: +

+
abuffer=sample_rate=44100:sample_fmt=6:channel_layout=0x3
+
+ +

9.2 aevalsrc

+ +

Generate an audio signal specified by an expression. +

+

This source accepts in input one or more expressions (one for each +channel), which are evaluated and used to generate a corresponding +audio signal. +

+

This source accepts the following options: +

+
+
exprs
+

Set the ’|’-separated expressions list for each separate channel. In case the +channel_layout option is not specified, the selected channel layout +depends on the number of provided expressions. Otherwise the last +specified expression is applied to the remaining output channels. +

+
+
channel_layout, c
+

Set the channel layout. The number of channels in the specified layout +must be equal to the number of specified expressions. +

+
+
duration, d
+

Set the minimum duration of the sourced audio. See +(ffmpeg-utils)the Time duration section in the ffmpeg-utils(1) manual +for the accepted syntax. +Note that the resulting duration may be greater than the specified +duration, as the generated audio is always cut at the end of a +complete frame. +

+

If not specified, or the expressed duration is negative, the audio is +supposed to be generated forever. +

+
+
nb_samples, n
+

Set the number of samples per channel per each output frame, +default to 1024. +

+
+
sample_rate, s
+

Specify the sample rate, default to 44100. +

+
+ +

Each expression in exprs can contain the following constants: +

+
+
n
+

number of the evaluated sample, starting from 0 +

+
+
t
+

time of the evaluated sample expressed in seconds, starting from 0 +

+
+
s
+

sample rate +

+
+
+ +

9.2.1 Examples

+ +
    +
  • Generate silence: +
    +
    aevalsrc=0
    +
    + +
  • Generate a sin signal with frequency of 440 Hz, set sample rate to +8000 Hz: +
    +
    aevalsrc="sin(440*2*PI*t):s=8000"
    +
    + +
  • Generate a two channels signal, specify the channel layout (Front +Center + Back Center) explicitly: +
    +
    aevalsrc="sin(420*2*PI*t)|cos(430*2*PI*t):c=FC|BC"
    +
    + +
  • Generate white noise: +
    +
    aevalsrc="-2+random(0)"
    +
    + +
  • Generate an amplitude modulated signal: +
    +
    aevalsrc="sin(10*2*PI*t)*sin(880*2*PI*t)"
    +
    + +
  • Generate 2.5 Hz binaural beats on a 360 Hz carrier: +
    +
    aevalsrc="0.1*sin(2*PI*(360-2.5/2)*t) | 0.1*sin(2*PI*(360+2.5/2)*t)"
    +
    + +
+ +

9.3 afdelaysrc

+ +

Generate a fractional delay FIR coefficients. +

+

The resulting stream can be used with afir filter for filtering the audio signal. +

+

The filter accepts the following options: +

+
+
delay, d
+

Set the fractional delay. Default is 0. +

+
+
sample_rate, r
+

Set the sample rate, default is 44100. +

+
+
nb_samples, n
+

Set the number of samples per each frame. Default is 1024. +

+
+
taps, t
+

Set the number of filter coefficients in output audio stream. +Default value is 0. +

+
+
channel_layout, c
+

Specifies the channel layout, and can be a string representing a channel layout. +The default value of channel_layout is "stereo". +

+
+ +

9.4 afireqsrc

+ +

Generate a FIR equalizer coefficients. +

+

The resulting stream can be used with afir filter for filtering the audio signal. +

+

The filter accepts the following options: +

+
+
preset, p
+

Set equalizer preset. +Default preset is flat. +

+

Available presets are: +

+
‘custom’
+
‘flat’
+
‘acoustic’
+
‘bass’
+
‘beats’
+
‘classic’
+
‘clear’
+
‘deep bass’
+
‘dubstep’
+
‘electronic’
+
‘hard-style’
+
‘hip-hop’
+
‘jazz’
+
‘metal’
+
‘movie’
+
‘pop’
+
‘r&b’
+
‘rock’
+
‘vocal booster’
+
+ +
+
gains, g
+

Set custom gains for each band. Only used if the preset option is set to custom. +Gains are separated by white spaces and each gain is set in dBFS. +Default is 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0. +

+
+
bands, b
+

Set the custom bands from where custom equalizer gains are set. +This must be in strictly increasing order. Only used if the preset option is set to custom. +Bands are separated by white spaces and each band represent frequency in Hz. +Default is 25 40 63 100 160 250 400 630 1000 1600 2500 4000 6300 10000 16000 24000. +

+
+
taps, t
+

Set number of filter coefficients in output audio stream. +Default value is 4096. +

+
+
sample_rate, r
+

Set sample rate of output audio stream, default is 44100. +

+
+
nb_samples, n
+

Set number of samples per each frame in output audio stream. Default is 1024. +

+
+
interp, i
+

Set interpolation method for FIR equalizer coefficients. Can be linear or cubic. +

+
+
phase, h
+

Set phase type of FIR filter. Can be linear or min: minimum-phase. +Default is minimum-phase filter. +

+
+ +

9.5 afirsrc

+ +

Generate a FIR coefficients using frequency sampling method. +

+

The resulting stream can be used with afir filter for filtering the audio signal. +

+

The filter accepts the following options: +

+
+
taps, t
+

Set number of filter coefficients in output audio stream. +Default value is 1025. +

+
+
frequency, f
+

Set frequency points from where magnitude and phase are set. +This must be in non decreasing order, and first element must be 0, while last element +must be 1. Elements are separated by white spaces. +

+
+
magnitude, m
+

Set magnitude value for every frequency point set by frequency. +Number of values must be same as number of frequency points. +Values are separated by white spaces. +

+
+
phase, p
+

Set phase value for every frequency point set by frequency. +Number of values must be same as number of frequency points. +Values are separated by white spaces. +

+
+
sample_rate, r
+

Set sample rate, default is 44100. +

+
+
nb_samples, n
+

Set number of samples per each frame. Default is 1024. +

+
+
win_func, w
+

Set window function. Default is blackman. +

+
+ +

9.6 anullsrc

+ +

The null audio source, return unprocessed audio frames. It is mainly useful +as a template and to be employed in analysis / debugging tools, or as +the source for filters which ignore the input data (for example the sox +synth filter). +

+

This source accepts the following options: +

+
+
channel_layout, cl
+
+

Specifies the channel layout, and can be either an integer or a string +representing a channel layout. The default value of channel_layout +is "stereo". +

+

Check the channel_layout_map definition in +libavutil/channel_layout.c for the mapping between strings and +channel layout values. +

+
+
sample_rate, r
+

Specifies the sample rate, and defaults to 44100. +

+
+
nb_samples, n
+

Set the number of samples per requested frames. +

+
+
duration, d
+

Set the duration of the sourced audio. See +(ffmpeg-utils)the Time duration section in the ffmpeg-utils(1) manual +for the accepted syntax. +

+

If not specified, or the expressed duration is negative, the audio is +supposed to be generated forever. +

+
+ +

9.6.1 Examples

+ +
    +
  • Set the sample rate to 48000 Hz and the channel layout to AV_CH_LAYOUT_MONO. +
    +
    anullsrc=r=48000:cl=4
    +
    + +
  • Do the same operation with a more obvious syntax: +
    +
    anullsrc=r=48000:cl=mono
    +
    +
+ +

All the parameters need to be explicitly defined. +

+

9.7 flite

+ +

Synthesize a voice utterance using the libflite library. +

+

To enable compilation of this filter you need to configure FFmpeg with +--enable-libflite. +

+

Note that versions of the flite library prior to 2.0 are not thread-safe. +

+

The filter accepts the following options: +

+
+
list_voices
+

If set to 1, list the names of the available voices and exit +immediately. Default value is 0. +

+
+
nb_samples, n
+

Set the maximum number of samples per frame. Default value is 512. +

+
+
textfile
+

Set the filename containing the text to speak. +

+
+
text
+

Set the text to speak. +

+
+
voice, v
+

Set the voice to use for the speech synthesis. Default value is +kal. See also the list_voices option. +

+
+ +

9.7.1 Examples

+ +
    +
  • Read from file speech.txt, and synthesize the text using the +standard flite voice: +
    +
    flite=textfile=speech.txt
    +
    + +
  • Read the specified text selecting the slt voice: +
    +
    flite=text='So fare thee well, poor devil of a Sub-Sub, whose commentator I am':voice=slt
    +
    + +
  • Input text to ffmpeg: +
    +
    ffmpeg -f lavfi -i flite=text='So fare thee well, poor devil of a Sub-Sub, whose commentator I am':voice=slt
    +
    + +
  • Make ffplay speak the specified text, using flite and +the lavfi device: +
    +
    ffplay -f lavfi flite=text='No more be grieved for which that thou hast done.'
    +
    +
+ +

For more information about libflite, check: +http://www.festvox.org/flite/ +

+

9.8 anoisesrc

+ +

Generate a noise audio signal. +

+

The filter accepts the following options: +

+
+
sample_rate, r
+

Specify the sample rate. Default value is 48000 Hz. +

+
+
amplitude, a
+

Specify the amplitude (0.0 - 1.0) of the generated audio stream. Default value +is 1.0. +

+
+
duration, d
+

Specify the duration of the generated audio stream. Not specifying this option +results in noise with an infinite length. +

+
+
color, colour, c
+

Specify the color of noise. Available noise colors are white, pink, brown, +blue, violet and velvet. Default color is white. +

+
+
seed, s
+

Specify a value used to seed the PRNG. +

+
+
nb_samples, n
+

Set the number of samples per each output frame, default is 1024. +

+
+
density
+

Set the density (0.0 - 1.0) for the velvet noise generator, default is 0.05. +

+
+ +

9.8.1 Examples

+ +
    +
  • Generate 60 seconds of pink noise, with a 44.1 kHz sampling rate and an amplitude of 0.5: +
    +
    anoisesrc=d=60:c=pink:r=44100:a=0.5
    +
    +
+ +

9.9 hilbert

+ +

Generate odd-tap Hilbert transform FIR coefficients. +

+

The resulting stream can be used with afir filter for phase-shifting +the signal by 90 degrees. +

+

This is used in many matrix coding schemes and for analytic signal generation. +The process is often written as a multiplication by i (or j), the imaginary unit. +

+

The filter accepts the following options: +

+
+
sample_rate, s
+

Set sample rate, default is 44100. +

+
+
taps, t
+

Set length of FIR filter, default is 22051. +

+
+
nb_samples, n
+

Set number of samples per each frame. +

+
+
win_func, w
+

Set window function to be used when generating FIR coefficients. +

+
+ +

9.10 sinc

+ +

Generate a sinc kaiser-windowed low-pass, high-pass, band-pass, or band-reject FIR coefficients. +

+

The resulting stream can be used with afir filter for filtering the audio signal. +

+

The filter accepts the following options: +

+
+
sample_rate, r
+

Set sample rate, default is 44100. +

+
+
nb_samples, n
+

Set number of samples per each frame. Default is 1024. +

+
+
hp
+

Set high-pass frequency. Default is 0. +

+
+
lp
+

Set low-pass frequency. Default is 0. +If high-pass frequency is lower than low-pass frequency and low-pass frequency +is higher than 0 then filter will create band-pass filter coefficients, +otherwise band-reject filter coefficients. +

+
+
phase
+

Set filter phase response. Default is 50. Allowed range is from 0 to 100. +

+
+
beta
+

Set Kaiser window beta. +

+
+
att
+

Set stop-band attenuation. Default is 120dB, allowed range is from 40 to 180 dB. +

+
+
round
+

Enable rounding, by default is disabled. +

+
+
hptaps
+

Set number of taps for high-pass filter. +

+
+
lptaps
+

Set number of taps for low-pass filter. +

+
+ +

9.11 sine

+ +

Generate an audio signal made of a sine wave with amplitude 1/8. +

+

The audio signal is bit-exact. +

+

The filter accepts the following options: +

+
+
frequency, f
+

Set the carrier frequency. Default is 440 Hz. +

+
+
beep_factor, b
+

Enable a periodic beep every second with frequency beep_factor times +the carrier frequency. Default is 0, meaning the beep is disabled. +

+
+
sample_rate, r
+

Specify the sample rate, default is 44100. +

+
+
duration, d
+

Specify the duration of the generated audio stream. +

+
+
samples_per_frame
+

Set the number of samples per output frame. +

+

The expression can contain the following constants: +

+
+
n
+

The (sequential) number of the output audio frame, starting from 0. +

+
+
pts
+

The PTS (Presentation TimeStamp) of the output audio frame, +expressed in TB units. +

+
+
t
+

The PTS of the output audio frame, expressed in seconds. +

+
+
TB
+

The timebase of the output audio frames. +

+
+ +

Default is 1024. +

+
+ +

9.11.1 Examples

+ +
    +
  • Generate a simple 440 Hz sine wave: +
    +
    sine
    +
    + +
  • Generate a 220 Hz sine wave with a 880 Hz beep each second, for 5 seconds: +
    +
    sine=220:4:d=5
    +sine=f=220:b=4:d=5
    +sine=frequency=220:beep_factor=4:duration=5
    +
    + +
  • Generate a 1 kHz sine wave following 1602,1601,1602,1601,1602 NTSC +pattern: +
    +
    sine=1000:samples_per_frame='st(0,mod(n,5)); 1602-not(not(eq(ld(0),1)+eq(ld(0),3)))'
    +
    +
+ + +

10 Audio Sinks

+ +

Below is a description of the currently available audio sinks. +

+

10.1 abuffersink

+ +

Buffer audio frames, and make them available to the end of filter chain. +

+

This sink is mainly intended for programmatic use, in particular +through the interface defined in libavfilter/buffersink.h +or the options system. +

+

It accepts a pointer to an AVABufferSinkContext structure, which +defines the incoming buffers’ formats, to be passed as the opaque +parameter to avfilter_init_filter for initialization. +

10.2 anullsink

+ +

Null audio sink; do absolutely nothing with the input audio. It is +mainly useful as a template and for use in analysis / debugging +tools. +

+ +

11 Video Filters

+ +

When you configure your FFmpeg build, you can disable any of the +existing filters using --disable-filters. +The configure output will show the video filters included in your +build. +

+

Below is a description of the currently available video filters. +

+

11.1 addroi

+ +

Mark a region of interest in a video frame. +

+

The frame data is passed through unchanged, but metadata is attached +to the frame indicating regions of interest which can affect the +behaviour of later encoding. Multiple regions can be marked by +applying the filter multiple times. +

+
+
x
+

Region distance in pixels from the left edge of the frame. +

+
y
+

Region distance in pixels from the top edge of the frame. +

+
w
+

Region width in pixels. +

+
h
+

Region height in pixels. +

+

The parameters x, y, w and h are expressions, +and may contain the following variables: +

+
iw
+

Width of the input frame. +

+
ih
+

Height of the input frame. +

+
+ +
+
qoffset
+

Quantisation offset to apply within the region. +

+

This must be a real value in the range -1 to +1. A value of zero +indicates no quality change. A negative value asks for better quality +(less quantisation), while a positive value asks for worse quality +(greater quantisation). +

+

The range is calibrated so that the extreme values indicate the +largest possible offset - if the rest of the frame is encoded with the +worst possible quality, an offset of -1 indicates that this region +should be encoded with the best possible quality anyway. Intermediate +values are then interpolated in some codec-dependent way. +

+

For example, in 10-bit H.264 the quantisation parameter varies between +-12 and 51. A typical qoffset value of -1/10 therefore indicates that +this region should be encoded with a QP around one-tenth of the full +range better than the rest of the frame. So, if most of the frame +were to be encoded with a QP of around 30, this region would get a QP +of around 24 (an offset of approximately -1/10 * (51 - -12) = -6.3). +An extreme value of -1 would indicate that this region should be +encoded with the best possible quality regardless of the treatment of +the rest of the frame - that is, should be encoded at a QP of -12. +

+
clear
+

If set to true, remove any existing regions of interest marked on the +frame before adding the new one. +

+
+ +

11.1.1 Examples

+ +
    +
  • Mark the centre quarter of the frame as interesting. +
    +
    addroi=iw/4:ih/4:iw/2:ih/2:-1/10
    +
    +
  • Mark the 100-pixel-wide region on the left edge of the frame as very +uninteresting (to be encoded at much lower quality than the rest of +the frame). +
    +
    addroi=0:0:100:ih:+1/5
    +
    +
+ +

11.2 alphaextract

+ +

Extract the alpha component from the input as a grayscale video. This +is especially useful with the alphamerge filter. +

+

11.3 alphamerge

+ +

Add or replace the alpha component of the primary input with the +grayscale value of a second input. This is intended for use with +alphaextract to allow the transmission or storage of frame +sequences that have alpha in a format that doesn’t support an alpha +channel. +

+

For example, to reconstruct full frames from a normal YUV-encoded video +and a separate video created with alphaextract, you might use: +

+
movie=in_alpha.mkv [alpha]; [in][alpha] alphamerge [out]
+
+ +

11.4 amplify

+ +

Amplify differences between current pixel and pixels of adjacent frames in +same pixel location. +

+

This filter accepts the following options: +

+
+
radius
+

Set frame radius. Default is 2. Allowed range is from 1 to 63. +For example radius of 3 will instruct filter to calculate average of 7 frames. +

+
+
factor
+

Set factor to amplify difference. Default is 2. Allowed range is from 0 to 65535. +

+
+
threshold
+

Set threshold for difference amplification. Any difference greater or equal to +this value will not alter source pixel. Default is 10. +Allowed range is from 0 to 65535. +

+
+
tolerance
+

Set tolerance for difference amplification. Any difference lower to +this value will not alter source pixel. Default is 0. +Allowed range is from 0 to 65535. +

+
+
low
+

Set lower limit for changing source pixel. Default is 65535. Allowed range is from 0 to 65535. +This option controls maximum possible value that will decrease source pixel value. +

+
+
high
+

Set high limit for changing source pixel. Default is 65535. Allowed range is from 0 to 65535. +This option controls maximum possible value that will increase source pixel value. +

+
+
planes
+

Set which planes to filter. Default is all. Allowed range is from 0 to 15. +

+
+ +

11.4.1 Commands

+ +

This filter supports the following commands that corresponds to option of same name: +

+
factor
+
threshold
+
tolerance
+
low
+
high
+
planes
+
+ +

11.5 ass

+ +

Same as the subtitles filter, except that it doesn’t require libavcodec +and libavformat to work. On the other hand, it is limited to ASS (Advanced +Substation Alpha) subtitles files. +

+

This filter accepts the following option in addition to the common options from +the subtitles filter: +

+
+
shaping
+

Set the shaping engine +

+

Available values are: +

+
‘auto’
+

The default libass shaping engine, which is the best available. +

+
‘simple’
+

Fast, font-agnostic shaper that can do only substitutions +

+
‘complex’
+

Slower shaper using OpenType for substitutions and positioning +

+
+ +

The default is auto. +

+
+ +

11.6 atadenoise

+

Apply an Adaptive Temporal Averaging Denoiser to the video input. +

+

The filter accepts the following options: +

+
+
0a
+

Set threshold A for 1st plane. Default is 0.02. +Valid range is 0 to 0.3. +

+
+
0b
+

Set threshold B for 1st plane. Default is 0.04. +Valid range is 0 to 5. +

+
+
1a
+

Set threshold A for 2nd plane. Default is 0.02. +Valid range is 0 to 0.3. +

+
+
1b
+

Set threshold B for 2nd plane. Default is 0.04. +Valid range is 0 to 5. +

+
+
2a
+

Set threshold A for 3rd plane. Default is 0.02. +Valid range is 0 to 0.3. +

+
+
2b
+

Set threshold B for 3rd plane. Default is 0.04. +Valid range is 0 to 5. +

+

Threshold A is designed to react on abrupt changes in the input signal and +threshold B is designed to react on continuous changes in the input signal. +

+
+
s
+

Set number of frames filter will use for averaging. Default is 9. Must be odd +number in range [5, 129]. +

+
+
p
+

Set what planes of frame filter will use for averaging. Default is all. +

+
+
a
+

Set what variant of algorithm filter will use for averaging. Default is p parallel. +Alternatively can be set to s serial. +

+

Parallel can be faster then serial, while other way around is never true. +Parallel will abort early on first change being greater then thresholds, while serial +will continue processing other side of frames if they are equal or below thresholds. +

+
+
0s
+
1s
+
2s
+

Set sigma for 1st plane, 2nd plane or 3rd plane. Default is 32767. +Valid range is from 0 to 32767. +This options controls weight for each pixel in radius defined by size. +Default value means every pixel have same weight. +Setting this option to 0 effectively disables filtering. +

+
+ +

11.6.1 Commands

+

This filter supports same commands as options except option s. +The command accepts the same syntax of the corresponding option. +

+

11.7 avgblur

+ +

Apply average blur filter. +

+

The filter accepts the following options: +

+
+
sizeX
+

Set horizontal radius size. +

+
+
planes
+

Set which planes to filter. By default all planes are filtered. +

+
+
sizeY
+

Set vertical radius size, if zero it will be same as sizeX. +Default is 0. +

+
+ +

11.7.1 Commands

+

This filter supports same commands as options. +The command accepts the same syntax of the corresponding option. +

+

If the specified expression is not valid, it is kept at its current +value. +

+

11.8 backgroundkey

+ +

Turns a static background into transparency. +

+

The filter accepts the following option: +

+
+
threshold
+

Threshold for scene change detection. +

+
similarity
+

Similarity percentage with the background. +

+
blend
+

Set the blend amount for pixels that are not similar. +

+
+ +

11.8.1 Commands

+ +

This filter supports the all above options as commands. +

+

11.9 bbox

+ +

Compute the bounding box for the non-black pixels in the input frame +luma plane. +

+

This filter computes the bounding box containing all the pixels with a +luma value greater than the minimum allowed value. +The parameters describing the bounding box are printed on the filter +log. +

+

The filter accepts the following option: +

+
+
min_val
+

Set the minimal luma value. Default is 16. +

+
+ +

11.9.1 Commands

+ +

This filter supports the all above options as commands. +

+

11.10 bilateral

+

Apply bilateral filter, spatial smoothing while preserving edges. +

+

The filter accepts the following options: +

+
sigmaS
+

Set sigma of gaussian function to calculate spatial weight. +Allowed range is 0 to 512. Default is 0.1. +

+
+
sigmaR
+

Set sigma of gaussian function to calculate range weight. +Allowed range is 0 to 1. Default is 0.1. +

+
+
planes
+

Set planes to filter. Default is first only. +

+
+ +

11.10.1 Commands

+ +

This filter supports the all above options as commands. +

+

11.11 bitplanenoise

+ +

Show and measure bit plane noise. +

+

The filter accepts the following options: +

+
+
bitplane
+

Set which plane to analyze. Default is 1. +

+
+
filter
+

Filter out noisy pixels from bitplane set above. +Default is disabled. +

+
+ +

11.12 blackdetect, blackdetect_vulkan

+ +

Detect video intervals that are (almost) completely black. Can be +useful to detect chapter transitions, commercials, or invalid +recordings. +

+

The filter outputs its detection analysis to both the log as well as +frame metadata. If a black segment of at least the specified minimum +duration is found, a line with the start and end timestamps as well +as duration is printed to the log with level info. In addition, +a log line with level debug is printed per frame showing the +black amount detected for that frame. +

+

The filter also attaches metadata to the first frame of a black +segment with key lavfi.black_start and to the first frame +after the black segment ends with key lavfi.black_end. The +value is the frame’s timestamp. This metadata is added regardless +of the minimum duration specified. +

+

The filter accepts the following options: +

+
+
black_min_duration, d
+

Set the minimum detected black duration expressed in seconds. It must +be a non-negative floating point number. +

+

Default value is 2.0. +

+
+
picture_black_ratio_th, pic_th
+

Set the threshold for considering a picture "black". +Express the minimum value for the ratio: +

+
nb_black_pixels / nb_pixels
+
+ +

for which a picture is considered black. +Default value is 0.98. +

+
+
pixel_black_th, pix_th
+

Set the threshold for considering a pixel "black". +

+

The threshold expresses the maximum pixel luma value for which a +pixel is considered "black". The provided value is scaled according to +the following equation: +

+
absolute_threshold = luma_minimum_value + pixel_black_th * luma_range_size
+
+ +

luma_range_size and luma_minimum_value depend on +the input video format, the range is [0-255] for YUV full-range +formats and [16-235] for YUV non full-range formats. +

+

Default value is 0.10. +

+
+
alpha
+

If true, check the alpha channel instead of the luma channel. Detects frames +which are (almost) transparent, instead of frames which are almost black. +

+

Default value is disabled. +

+
+ +

The following example sets the maximum pixel threshold to the minimum +value, and detects only black intervals of 2 or more seconds: +

+
blackdetect=d=2:pix_th=0.00
+
+ +

11.13 blackframe

+ +

Detect frames that are (almost) completely black. Can be useful to +detect chapter transitions or commercials. Output lines consist of +the frame number of the detected frame, the percentage of blackness, +the position in the file if known or -1 and the timestamp in seconds. +

+

In order to display the output lines, you need to set the loglevel at +least to the AV_LOG_INFO value. +

+

This filter exports frame metadata lavfi.blackframe.pblack. +The value represents the percentage of pixels in the picture that +are below the threshold value. +

+

It accepts the following parameters: +

+
+
amount
+

The percentage of the pixels that have to be below the threshold; it defaults to +98. +

+
+
threshold, thresh
+

The threshold below which a pixel value is considered black; it defaults to +32. +

+
+
+ +

11.14 blend

+ +

Blend two video frames into each other. +

+

The blend filter takes two input streams and outputs one +stream, the first input is the "top" layer and second input is +"bottom" layer. By default, the output terminates when the longest input terminates. +

+

The tblend (time blend) filter takes two consecutive frames +from one single stream, and outputs the result obtained by blending +the new frame on top of the old frame. +

+

A description of the accepted options follows. +

+
+
c0_mode
+
c1_mode
+
c2_mode
+
c3_mode
+
all_mode
+

Set blend mode for specific pixel component or all pixel components in case +of all_mode. Default value is normal. +

+

Available values for component modes are: +

+
‘addition’
+
‘and’
+
‘average’
+
‘bleach’
+
‘burn’
+
‘darken’
+
‘difference’
+
‘divide’
+
‘dodge’
+
‘exclusion’
+
‘extremity’
+
‘freeze’
+
‘geometric’
+
‘glow’
+
‘grainextract’
+
‘grainmerge’
+
‘hardlight’
+
‘hardmix’
+
‘hardoverlay’
+
‘harmonic’
+
‘heat’
+
‘interpolate’
+
‘lighten’
+
‘linearlight’
+
‘multiply’
+
‘multiply128’
+
‘negation’
+
‘normal’
+
‘or’
+
‘overlay’
+
‘phoenix’
+
‘pinlight’
+
‘reflect’
+
‘screen’
+
‘softdifference’
+
‘softlight’
+
‘stain’
+
‘subtract’
+
‘vividlight’
+
‘xor’
+
+ +
+
c0_opacity
+
c1_opacity
+
c2_opacity
+
c3_opacity
+
all_opacity
+

Set blend opacity for specific pixel component or all pixel components in case +of all_opacity. Only used in combination with pixel component blend modes. +

+
+
c0_expr
+
c1_expr
+
c2_expr
+
c3_expr
+
all_expr
+

Set blend expression for specific pixel component or all pixel components in case +of all_expr. Note that related mode options will be ignored if those are set. +

+

The expressions can use the following variables: +

+
+
N
+

The sequential number of the filtered frame, starting from 0. +

+
+
X
+
Y
+

the coordinates of the current sample +

+
+
W
+
H
+

the width and height of currently filtered plane +

+
+
SW
+
SH
+

Width and height scale for the plane being filtered. It is the +ratio between the dimensions of the current plane to the luma plane, +e.g. for a yuv420p frame, the values are 1,1 for +the luma plane and 0.5,0.5 for the chroma planes. +

+
+
T
+

Time of the current frame, expressed in seconds. +

+
+
TOP, A
+

Value of pixel component at current location for first video frame (top layer). +

+
+
BOTTOM, B
+

Value of pixel component at current location for second video frame (bottom layer). +

+
+
+
+ +

The blend filter also supports the framesync options. +

+

11.14.1 Examples

+ +
    +
  • Apply transition from bottom layer to top layer in first 10 seconds: +
    +
    blend=all_expr='A*(if(gte(T,10),1,T/10))+B*(1-(if(gte(T,10),1,T/10)))'
    +
    + +
  • Apply linear horizontal transition from top layer to bottom layer: +
    +
    blend=all_expr='A*(X/W)+B*(1-X/W)'
    +
    + +
  • Apply 1x1 checkerboard effect: +
    +
    blend=all_expr='if(eq(mod(X,2),mod(Y,2)),A,B)'
    +
    + +
  • Apply uncover left effect: +
    +
    blend=all_expr='if(gte(N*SW+X,W),A,B)'
    +
    + +
  • Apply uncover down effect: +
    +
    blend=all_expr='if(gte(Y-N*SH,0),A,B)'
    +
    + +
  • Apply uncover up-left effect: +
    +
    blend=all_expr='if(gte(T*SH*40+Y,H)*gte((T*40*SW+X)*W/H,W),A,B)'
    +
    + +
  • Split diagonally video and shows top and bottom layer on each side: +
    +
    blend=all_expr='if(gt(X,Y*(W/H)),A,B)'
    +
    + +
  • Display differences between the current and the previous frame: +
    +
    tblend=all_mode=grainextract
    +
    +
+ +

11.14.2 Commands

+

This filter supports same commands as options. +

+

11.15 blockdetect

+ +

Determines blockiness of frames without altering the input frames. +

+

Based on Remco Muijs and Ihor Kirenko: "A no-reference blocking artifact measure for adaptive video processing." 2005 13th European signal processing conference. +

+

The filter accepts the following options: +

+
+
period_min
+
period_max
+

Set minimum and maximum values for determining pixel grids (periods). +Default values are [3,24]. +

+
+
planes
+

Set planes to filter. Default is first only. +

+
+ +

11.15.1 Examples

+ +
    +
  • Determine blockiness for the first plane and search for periods within [8,32]: +
    +
    blockdetect=period_min=8:period_max=32:planes=1
    +
    +
+ +

11.16 blurdetect

+ +

Determines blurriness of frames without altering the input frames. +

+

Based on Marziliano, Pina, et al. "A no-reference perceptual blur metric." +Allows for a block-based abbreviation. +

+

The filter accepts the following options: +

+
+
low
+
high
+

Set low and high threshold values used by the Canny thresholding +algorithm. +

+

The high threshold selects the "strong" edge pixels, which are then +connected through 8-connectivity with the "weak" edge pixels selected +by the low threshold. +

+

low and high threshold values must be chosen in the range +[0,1], and low should be lesser or equal to high. +

+

Default value for low is 20/255, and default value for high +is 50/255. +

+
+
radius
+

Define the radius to search around an edge pixel for local maxima. +

+
+
block_pct
+

Determine blurriness only for the most significant blocks, given in percentage. +

+
+
block_width
+

Determine blurriness for blocks of width block_width. If set to any value smaller 1, no blocks are used and the whole image is processed as one no matter of block_height. +

+
+
block_height
+

Determine blurriness for blocks of height block_height. If set to any value smaller 1, no blocks are used and the whole image is processed as one no matter of block_width. +

+
+
planes
+

Set planes to filter. Default is first only. +

+
+ +

11.16.1 Examples

+ +
    +
  • Determine blur for 80% of most significant 32x32 blocks: +
    +
    blurdetect=block_width=32:block_height=32:block_pct=80
    +
    +
+ +

11.17 bm3d

+ +

Denoise frames using Block-Matching 3D algorithm. +

+

The filter accepts the following options. +

+
+
sigma
+

Set denoising strength. Default value is 1. +Allowed range is from 0 to 999.9. +The denoising algorithm is very sensitive to sigma, so adjust it +according to the source. +

+
+
block
+

Set local patch size. This sets dimensions in 2D. +

+
+
bstep
+

Set sliding step for processing blocks. Default value is 4. +Allowed range is from 1 to 64. +Smaller values allows processing more reference blocks and is slower. +

+
+
group
+

Set maximal number of similar blocks for 3rd dimension. Default value is 1. +When set to 1, no block matching is done. Larger values allows more blocks +in single group. +Allowed range is from 1 to 256. +

+
+
range
+

Set radius for search block matching. Default is 9. +Allowed range is from 1 to INT32_MAX. +

+
+
mstep
+

Set step between two search locations for block matching. Default is 1. +Allowed range is from 1 to 64. Smaller is slower. +

+
+
thmse
+

Set threshold of mean square error for block matching. Valid range is 0 to +INT32_MAX. +

+
+
hdthr
+

Set thresholding parameter for hard thresholding in 3D transformed domain. +Larger values results in stronger hard-thresholding filtering in frequency +domain. +

+
+
estim
+

Set filtering estimation mode. Can be basic or final. +Default is basic. +

+
+
ref
+

If enabled, filter will use 2nd stream for block matching. +Default is disabled for basic value of estim option, +and always enabled if value of estim is final. +

+
+
planes
+

Set planes to filter. Default is all available except alpha. +

+
+ +

11.17.1 Examples

+ +
    +
  • Basic filtering with bm3d: +
    +
    bm3d=sigma=3:block=4:bstep=2:group=1:estim=basic
    +
    + +
  • Same as above, but filtering only luma: +
    +
    bm3d=sigma=3:block=4:bstep=2:group=1:estim=basic:planes=1
    +
    + +
  • Same as above, but with both estimation modes: +
    +
    split[a][b],[a]bm3d=sigma=3:block=4:bstep=2:group=1:estim=basic[a],[b][a]bm3d=sigma=3:block=4:bstep=2:group=16:estim=final:ref=1
    +
    + +
  • Same as above, but prefilter with nlmeans filter instead: +
    +
    split[a][b],[a]nlmeans=s=3:r=7:p=3[a],[b][a]bm3d=sigma=3:block=4:bstep=2:group=16:estim=final:ref=1
    +
    +
+ +

11.18 boxblur

+ +

Apply a boxblur algorithm to the input video. +

+

It accepts the following parameters: +

+
+
luma_radius, lr
+
luma_power, lp
+
chroma_radius, cr
+
chroma_power, cp
+
alpha_radius, ar
+
alpha_power, ap
+
+ +

A description of the accepted options follows. +

+
+
luma_radius, lr
+
chroma_radius, cr
+
alpha_radius, ar
+

Set an expression for the box radius in pixels used for blurring the +corresponding input plane. +

+

The radius value must be a non-negative number, and must not be +greater than the value of the expression min(w,h)/2 for the +luma and alpha planes, and of min(cw,ch)/2 for the chroma +planes. +

+

Default value for luma_radius is "2". If not specified, +chroma_radius and alpha_radius default to the +corresponding value set for luma_radius. +

+

The expressions can contain the following constants: +

+
w
+
h
+

The input width and height in pixels. +

+
+
cw
+
ch
+

The input chroma image width and height in pixels. +

+
+
hsub
+
vsub
+

The horizontal and vertical chroma subsample values. For example, for the +pixel format "yuv422p", hsub is 2 and vsub is 1. +

+
+ +
+
luma_power, lp
+
chroma_power, cp
+
alpha_power, ap
+

Specify how many times the boxblur filter is applied to the +corresponding plane. +

+

Default value for luma_power is 2. If not specified, +chroma_power and alpha_power default to the +corresponding value set for luma_power. +

+

A value of 0 will disable the effect. +

+
+ +

11.18.1 Examples

+ +
    +
  • Apply a boxblur filter with the luma, chroma, and alpha radii +set to 2: +
    +
    boxblur=luma_radius=2:luma_power=1
    +boxblur=2:1
    +
    + +
  • Set the luma radius to 2, and alpha and chroma radius to 0: +
    +
    boxblur=2:1:cr=0:ar=0
    +
    + +
  • Set the luma and chroma radii to a fraction of the video dimension: +
    +
    boxblur=luma_radius=min(h\,w)/10:luma_power=1:chroma_radius=min(cw\,ch)/10:chroma_power=1
    +
    +
+ +

11.19 bwdif

+ +

Deinterlace the input video ("bwdif" stands for "Bob Weaver +Deinterlacing Filter"). +

+

Motion adaptive deinterlacing based on yadif with the use of w3fdif and cubic +interpolation algorithms. +It accepts the following parameters: +

+
+
mode
+

The interlacing mode to adopt. It accepts one of the following values: +

+
+
0, send_frame
+

Output one frame for each frame. +

+
1, send_field
+

Output one frame for each field. +

+
+ +

The default value is send_field. +

+
+
parity
+

The picture field parity assumed for the input interlaced video. It accepts one +of the following values: +

+
+
0, tff
+

Assume the top field is first. +

+
1, bff
+

Assume the bottom field is first. +

+
-1, auto
+

Enable automatic detection of field parity. +

+
+ +

The default value is auto. +If the interlacing is unknown or the decoder does not export this information, +top field first will be assumed. +

+
+
deint
+

Specify which frames to deinterlace. Accepts one of the following +values: +

+
+
0, all
+

Deinterlace all frames. +

+
1, interlaced
+

Only deinterlace frames marked as interlaced. +

+
+ +

The default value is all. +

+
+ +

11.20 ccrepack

+ +

Repack CEA-708 closed captioning side data +

+

This filter fixes various issues seen with commercial encoders +related to upstream malformed CEA-708 payloads, specifically +incorrect number of tuples (wrong cc_count for the target FPS), +and incorrect ordering of tuples (i.e. the CEA-608 tuples are not at +the first entries in the payload). +

+

11.21 cas

+ +

Apply Contrast Adaptive Sharpen filter to video stream. +

+

The filter accepts the following options: +

+
+
strength
+

Set the sharpening strength. Default value is 0. +

+
+
planes
+

Set planes to filter. Default value is to filter all +planes except alpha plane. +

+
+ +

11.21.1 Commands

+

This filter supports same commands as options. +

+

11.22 chromahold

+

Remove all color information for all colors except for certain one. +

+

The filter accepts the following options: +

+
+
color
+

The color which will not be replaced with neutral chroma. +

+
+
similarity
+

Similarity percentage with the above color. +0.01 matches only the exact key color, while 1.0 matches everything. +

+
+
blend
+

Blend percentage. +0.0 makes pixels either fully gray, or not gray at all. +Higher values result in more preserved color. +

+
+
yuv
+

Signals that the color passed is already in YUV instead of RGB. +

+

Literal colors like "green" or "red" don’t make sense with this enabled anymore. +This can be used to pass exact YUV values as hexadecimal numbers. +

+
+ +

11.22.1 Commands

+

This filter supports same commands as options. +The command accepts the same syntax of the corresponding option. +

+

If the specified expression is not valid, it is kept at its current +value. +

+

11.23 chromakey

+

YUV colorspace color/chroma keying. +

+

The filter accepts the following options: +

+
+
color
+

The color which will be replaced with transparency. +

+
+
similarity
+

Similarity percentage with the key color. +

+

0.01 matches only the exact key color, while 1.0 matches everything. +

+
+
blend
+

Blend percentage. +

+

0.0 makes pixels either fully transparent, or not transparent at all. +

+

Higher values result in semi-transparent pixels, with a higher transparency +the more similar the pixels color is to the key color. +

+
+
yuv
+

Signals that the color passed is already in YUV instead of RGB. +

+

Literal colors like "green" or "red" don’t make sense with this enabled anymore. +This can be used to pass exact YUV values as hexadecimal numbers. +

+
+ +

11.23.1 Commands

+

This filter supports same commands as options. +The command accepts the same syntax of the corresponding option. +

+

If the specified expression is not valid, it is kept at its current +value. +

+

11.23.2 Examples

+ +
    +
  • Make every green pixel in the input image transparent: +
    +
    ffmpeg -i input.png -vf chromakey=green out.png
    +
    + +
  • Overlay a greenscreen-video on top of a static black background. +
    +
    ffmpeg -f lavfi -i color=c=black:s=1280x720 -i video.mp4 -shortest -filter_complex "[1:v]chromakey=0x70de77:0.1:0.2[ckout];[0:v][ckout]overlay[out]" -map "[out]" output.mkv
    +
    +
+ +

11.24 chromanr

+

Reduce chrominance noise. +

+

The filter accepts the following options: +

+
+
thres
+

Set threshold for averaging chrominance values. +Sum of absolute difference of Y, U and V pixel components of current +pixel and neighbour pixels lower than this threshold will be used in +averaging. Luma component is left unchanged and is copied to output. +Default value is 30. Allowed range is from 1 to 200. +

+
+
sizew
+

Set horizontal radius of rectangle used for averaging. +Allowed range is from 1 to 100. Default value is 5. +

+
+
sizeh
+

Set vertical radius of rectangle used for averaging. +Allowed range is from 1 to 100. Default value is 5. +

+
+
stepw
+

Set horizontal step when averaging. Default value is 1. +Allowed range is from 1 to 50. +Mostly useful to speed-up filtering. +

+
+
steph
+

Set vertical step when averaging. Default value is 1. +Allowed range is from 1 to 50. +Mostly useful to speed-up filtering. +

+
+
threy
+

Set Y threshold for averaging chrominance values. +Set finer control for max allowed difference between Y components +of current pixel and neighbour pixels. +Default value is 200. Allowed range is from 1 to 200. +

+
+
threu
+

Set U threshold for averaging chrominance values. +Set finer control for max allowed difference between U components +of current pixel and neighbour pixels. +Default value is 200. Allowed range is from 1 to 200. +

+
+
threv
+

Set V threshold for averaging chrominance values. +Set finer control for max allowed difference between V components +of current pixel and neighbour pixels. +Default value is 200. Allowed range is from 1 to 200. +

+
+
distance
+

Set distance type used in calculations. +

+
+
‘manhattan’
+

Absolute difference. +

+
‘euclidean’
+

Difference squared. +

+
+ +

Default distance type is manhattan. +

+
+ +

11.24.1 Commands

+

This filter supports same commands as options. +The command accepts the same syntax of the corresponding option. +

+

11.25 chromashift

+

Shift chroma pixels horizontally and/or vertically. +

+

The filter accepts the following options: +

+
cbh
+

Set amount to shift chroma-blue horizontally. +

+
cbv
+

Set amount to shift chroma-blue vertically. +

+
crh
+

Set amount to shift chroma-red horizontally. +

+
crv
+

Set amount to shift chroma-red vertically. +

+
edge
+

Set edge mode, can be smear, default, or warp. +

+
+ +

11.25.1 Commands

+ +

This filter supports the all above options as commands. +

+

11.26 ciescope

+ +

Display CIE color diagram with pixels overlaid onto it. +

+

The filter accepts the following options: +

+
+
system
+

Set color system. +

+
+
‘ntsc, 470m’
+
‘ebu, 470bg’
+
‘smpte’
+
‘240m’
+
‘apple’
+
‘widergb’
+
‘cie1931’
+
‘rec709, hdtv’
+
‘uhdtv, rec2020’
+
‘dcip3’
+
+ +
+
cie
+

Set CIE system. +

+
+
‘xyy’
+
‘ucs’
+
‘luv’
+
+ +
+
gamuts
+

Set what gamuts to draw. +

+

See system option for available values. +

+
+
size, s
+

Set ciescope size, by default set to 512. +

+
+
intensity, i
+

Set intensity used to map input pixel values to CIE diagram. +

+
+
contrast
+

Set contrast used to draw tongue colors that are out of active color system gamut. +

+
+
corrgamma
+

Correct gamma displayed on scope, by default enabled. +

+
+
showwhite
+

Show white point on CIE diagram, by default disabled. +

+
+
gamma
+

Set input gamma. Used only with XYZ input color space. +

+
+
fill
+

Fill with CIE colors. By default is enabled. +

+
+ +

11.27 codecview

+ +

Visualize information exported by some codecs. +

+

Some codecs can export information through frames using side-data or other +means. For example, some MPEG based codecs export motion vectors through the +export_mvs flag in the codec flags2 option. +

+

The filter accepts the following option: +

+
+
block
+

Display block partition structure using the luma plane. +

+
+
mv
+

Set motion vectors to visualize. +

+

Available flags for mv are: +

+
+
‘pf’
+

forward predicted MVs of P-frames +

+
‘bf’
+

forward predicted MVs of B-frames +

+
‘bb’
+

backward predicted MVs of B-frames +

+
+ +
+
qp
+

Display quantization parameters using the chroma planes. +

+
+
mv_type, mvt
+

Set motion vectors type to visualize. Includes MVs from all frames unless specified by frame_type option. +

+

Available flags for mv_type are: +

+
+
‘fp’
+

forward predicted MVs +

+
‘bp’
+

backward predicted MVs +

+
+ +
+
frame_type, ft
+

Set frame type to visualize motion vectors of. +

+

Available flags for frame_type are: +

+
+
‘if’
+

intra-coded frames (I-frames) +

+
‘pf’
+

predicted frames (P-frames) +

+
‘bf’
+

bi-directionally predicted frames (B-frames) +

+
+
+
+ +

11.27.1 Examples

+ +
    +
  • Visualize forward predicted MVs of all frames using ffplay: +
    +
    ffplay -flags2 +export_mvs input.mp4 -vf codecview=mv_type=fp
    +
    + +
  • Visualize multi-directionals MVs of P and B-Frames using ffplay: +
    +
    ffplay -flags2 +export_mvs input.mp4 -vf codecview=mv=pf+bf+bb
    +
    +
+ +

11.28 colorbalance

+

Modify intensity of primary colors (red, green and blue) of input frames. +

+

The filter allows an input frame to be adjusted in the shadows, midtones or highlights +regions for the red-cyan, green-magenta or blue-yellow balance. +

+

A positive adjustment value shifts the balance towards the primary color, a negative +value towards the complementary color. +

+

The filter accepts the following options: +

+
+
rs
+
gs
+
bs
+

Adjust red, green and blue shadows (darkest pixels). +

+
+
rm
+
gm
+
bm
+

Adjust red, green and blue midtones (medium pixels). +

+
+
rh
+
gh
+
bh
+

Adjust red, green and blue highlights (brightest pixels). +

+

Allowed ranges for options are [-1.0, 1.0]. Defaults are 0. +

+
+
pl
+

Preserve lightness when changing color balance. Default is disabled. +

+
+ +

11.28.1 Examples

+ +
    +
  • Add red color cast to shadows: +
    +
    colorbalance=rs=.3
    +
    +
+ +

11.28.2 Commands

+ +

This filter supports the all above options as commands. +

+

11.29 colorcontrast

+ +

Adjust color contrast between RGB components. +

+

The filter accepts the following options: +

+
+
rc
+

Set the red-cyan contrast. Defaults is 0.0. Allowed range is from -1.0 to 1.0. +

+
+
gm
+

Set the green-magenta contrast. Defaults is 0.0. Allowed range is from -1.0 to 1.0. +

+
+
by
+

Set the blue-yellow contrast. Defaults is 0.0. Allowed range is from -1.0 to 1.0. +

+
+
rcw
+
gmw
+
byw
+

Set the weight of each rc, gm, by option value. Default value is 0.0. +Allowed range is from 0.0 to 1.0. If all weights are 0.0 filtering is disabled. +

+
+
pl
+

Set the amount of preserving lightness. Default value is 0.0. Allowed range is from 0.0 to 1.0. +

+
+ +

11.29.1 Commands

+ +

This filter supports the all above options as commands. +

+

11.30 colorcorrect

+ +

Adjust color white balance selectively for blacks and whites. +This filter operates in YUV colorspace. +

+

The filter accepts the following options: +

+
+
rl
+

Set the red shadow spot. Allowed range is from -1.0 to 1.0. +Default value is 0. +

+
+
bl
+

Set the blue shadow spot. Allowed range is from -1.0 to 1.0. +Default value is 0. +

+
+
rh
+

Set the red highlight spot. Allowed range is from -1.0 to 1.0. +Default value is 0. +

+
+
bh
+

Set the blue highlight spot. Allowed range is from -1.0 to 1.0. +Default value is 0. +

+
+
saturation
+

Set the amount of saturation. Allowed range is from -3.0 to 3.0. +Default value is 1. +

+
+
analyze
+

If set to anything other than manual it will analyze every frame and use derived +parameters for filtering output frame. +

+

Possible values are: +

+
‘manual’
+
‘average’
+
‘minmax’
+
‘median’
+
+

Default value is manual. +

+
+ +

11.30.1 Commands

+ +

This filter supports the all above options as commands. +

+

11.31 colorchannelmixer

+ +

Adjust video input frames by re-mixing color channels. +

+

This filter modifies a color channel by adding the values associated to +the other channels of the same pixels. For example if the value to +modify is red, the output value will be: +

+
red=red*rr + blue*rb + green*rg + alpha*ra
+
+ +

The filter accepts the following options: +

+
+
rr
+
rg
+
rb
+
ra
+

Adjust contribution of input red, green, blue and alpha channels for output red channel. +Default is 1 for rr, and 0 for rg, rb and ra. +

+
+
gr
+
gg
+
gb
+
ga
+

Adjust contribution of input red, green, blue and alpha channels for output green channel. +Default is 1 for gg, and 0 for gr, gb and ga. +

+
+
br
+
bg
+
bb
+
ba
+

Adjust contribution of input red, green, blue and alpha channels for output blue channel. +Default is 1 for bb, and 0 for br, bg and ba. +

+
+
ar
+
ag
+
ab
+
aa
+

Adjust contribution of input red, green, blue and alpha channels for output alpha channel. +Default is 1 for aa, and 0 for ar, ag and ab. +

+

Allowed ranges for options are [-2.0, 2.0]. +

+
+
pc
+

Set preserve color mode. The accepted values are: +

+
‘none’
+

Disable color preserving, this is default. +

+
‘lum’
+

Preserve luminance. +

+
‘max’
+

Preserve max value of RGB triplet. +

+
‘avg’
+

Preserve average value of RGB triplet. +

+
‘sum’
+

Preserve sum value of RGB triplet. +

+
‘nrm’
+

Preserve normalized value of RGB triplet. +

+
‘pwr’
+

Preserve power value of RGB triplet. +

+
+ +
+
pa
+

Set the preserve color amount when changing colors. Allowed range is from [0.0, 1.0]. +Default is 0.0, thus disabled. +

+
+ +

11.31.1 Examples

+ +
    +
  • Convert source to grayscale: +
    +
    colorchannelmixer=.3:.4:.3:0:.3:.4:.3:0:.3:.4:.3
    +
    +
  • Simulate sepia tones: +
    +
    colorchannelmixer=.393:.769:.189:0:.349:.686:.168:0:.272:.534:.131
    +
    +
+ +

11.31.2 Commands

+ +

This filter supports the all above options as commands. +

+

11.32 colordetect

+

Analyze the video frames to determine the effective value range and alpha +mode. +

+

The filter accepts the following options: +

+
+
mode
+

Set of properties to detect. Unavailable properties, such as alpha mode for +an input image without an alpha channel, will be ignored automatically. +

+

Accepts a combination of the following flags: +

+
+
‘color_range’
+

Detect if the source contains luma pixels outside the limited (MPEG) range, +which indicates that this is a full range YUV source. +

+
‘alpha_mode’
+

Detect if the source contains color values above the alpha channel, which +indicates that the alpha channel is independent (straight), rather than +premultiplied. Also detects if the alpha plane is fully opaque or not. +

+
‘all’
+

Enable detection of all of the above properties. This is the default. +

+
+ +
+
+ +

11.33 colorize

+

Overlay a solid color on the video stream. +

+

The filter accepts the following options: +

+
+
hue
+

Set the color hue. Allowed range is from 0 to 360. +Default value is 0. +

+
+
saturation
+

Set the color saturation. Allowed range is from 0 to 1. +Default value is 0.5. +

+
+
lightness
+

Set the color lightness. Allowed range is from 0 to 1. +Default value is 0.5. +

+
+
mix
+

Set the mix of source lightness. By default is set to 1.0. +Allowed range is from 0.0 to 1.0. +

+
+ +

11.33.1 Commands

+ +

This filter supports the all above options as commands. +

+

11.34 colorkey

+

RGB colorspace color keying. +This filter operates on 8-bit RGB format frames by setting the alpha component of each pixel +which falls within the similarity radius of the key color to 0. The alpha value for pixels outside +the similarity radius depends on the value of the blend option. +

+

The filter accepts the following options: +

+
+
color
+

Set the color for which alpha will be set to 0 (full transparency). +See (ffmpeg-utils)"Color" section in the ffmpeg-utils manual. +Default is black. +

+
+
similarity
+

Set the radius from the key color within which other colors also have full transparency. +The computed distance is related to the unit fractional distance in 3D space between the RGB values +of the key color and the pixel’s color. Range is 0.01 to 1.0. 0.01 matches within a very small radius +around the exact key color, while 1.0 matches everything. +Default is 0.01. +

+
+
blend
+

Set how the alpha value for pixels that fall outside the similarity radius is computed. +0.0 makes pixels either fully transparent or fully opaque. +Higher values result in semi-transparent pixels, with greater transparency +the more similar the pixel color is to the key color. +Range is 0.0 to 1.0. Default is 0.0. +

+
+ +

11.34.1 Examples

+ +
    +
  • Make every green pixel in the input image transparent: +
    +
    ffmpeg -i input.png -vf colorkey=green out.png
    +
    + +
  • Overlay a greenscreen-video on top of a static background image. +
    +
    ffmpeg -i background.png -i video.mp4 -filter_complex "[1:v]colorkey=0x3BBD1E:0.3:0.2[ckout];[0:v][ckout]overlay[out]" -map "[out]" output.flv
    +
    +
+ +

11.34.2 Commands

+

This filter supports same commands as options. +The command accepts the same syntax of the corresponding option. +

+

If the specified expression is not valid, it is kept at its current +value. +

+

11.35 colorhold

+

Remove all color information for all RGB colors except for certain one. +

+

The filter accepts the following options: +

+
+
color
+

The color which will not be replaced with neutral gray. +

+
+
similarity
+

Similarity percentage with the above color. +0.01 matches only the exact key color, while 1.0 matches everything. +

+
+
blend
+

Blend percentage. 0.0 makes pixels fully gray. +Higher values result in more preserved color. +

+
+ +

11.35.1 Commands

+

This filter supports same commands as options. +The command accepts the same syntax of the corresponding option. +

+

If the specified expression is not valid, it is kept at its current +value. +

+

11.36 colorlevels

+ +

Adjust video input frames using levels. +

+

The filter accepts the following options: +

+
+
rimin
+
gimin
+
bimin
+
aimin
+

Adjust red, green, blue and alpha input black point. +Allowed ranges for options are [-1.0, 1.0]. Defaults are 0. +

+
+
rimax
+
gimax
+
bimax
+
aimax
+

Adjust red, green, blue and alpha input white point. +Allowed ranges for options are [-1.0, 1.0]. Defaults are 1. +

+

Input levels are used to lighten highlights (bright tones), darken shadows +(dark tones), change the balance of bright and dark tones. +

+
+
romin
+
gomin
+
bomin
+
aomin
+

Adjust red, green, blue and alpha output black point. +Allowed ranges for options are [0, 1.0]. Defaults are 0. +

+
+
romax
+
gomax
+
bomax
+
aomax
+

Adjust red, green, blue and alpha output white point. +Allowed ranges for options are [0, 1.0]. Defaults are 1. +

+

Output levels allows manual selection of a constrained output level range. +

+
+
preserve
+

Set preserve color mode. The accepted values are: +

+
‘none’
+

Disable color preserving, this is default. +

+
‘lum’
+

Preserve luminance. +

+
‘max’
+

Preserve max value of RGB triplet. +

+
‘avg’
+

Preserve average value of RGB triplet. +

+
‘sum’
+

Preserve sum value of RGB triplet. +

+
‘nrm’
+

Preserve normalized value of RGB triplet. +

+
‘pwr’
+

Preserve power value of RGB triplet. +

+
+
+
+ +

11.36.1 Examples

+ +
    +
  • Make video output darker: +
    +
    colorlevels=rimin=0.058:gimin=0.058:bimin=0.058
    +
    + +
  • Increase contrast: +
    +
    colorlevels=rimin=0.039:gimin=0.039:bimin=0.039:rimax=0.96:gimax=0.96:bimax=0.96
    +
    + +
  • Make video output lighter: +
    +
    colorlevels=rimax=0.902:gimax=0.902:bimax=0.902
    +
    + +
  • Increase brightness: +
    +
    colorlevels=romin=0.5:gomin=0.5:bomin=0.5
    +
    +
+ +

11.36.2 Commands

+ +

This filter supports the all above options as commands. +

+

11.37 colormap

+

Apply custom color maps to video stream. +

+

This filter needs three input video streams. +First stream is video stream that is going to be filtered out. +Second and third video stream specify color patches for source +color to target color mapping. +

+

The filter accepts the following options: +

+
+
patch_size
+

Set the source and target video stream patch size in pixels. +

+
+
nb_patches
+

Set the max number of used patches from source and target video stream. +Default value is number of patches available in additional video streams. +Max allowed number of patches is 64. +

+
+
type
+

Set the adjustments used for target colors. Can be relative or absolute. +Defaults is absolute. +

+
+
kernel
+

Set the kernel used to measure color differences between mapped colors. +

+

The accepted values are: +

+
‘euclidean’
+
‘weuclidean’
+
+ +

Default is euclidean. +

+
+ +

11.38 colormatrix

+ +

Convert color matrix. +

+

The filter accepts the following options: +

+
+
src
+
dst
+

Specify the source and destination color matrix. Both values must be +specified. +

+

The accepted values are: +

+
‘bt709’
+

BT.709 +

+
+
‘fcc’
+

FCC +

+
+
‘bt601’
+

BT.601 +

+
+
‘bt470’
+

BT.470 +

+
+
‘bt470bg’
+

BT.470BG +

+
+
‘smpte170m’
+

SMPTE-170M +

+
+
‘smpte240m’
+

SMPTE-240M +

+
+
‘bt2020’
+

BT.2020 +

+
+
+
+ +

For example to convert from BT.601 to SMPTE-240M, use the command: +

+
colormatrix=bt601:smpte240m
+
+ +

11.39 colorspace

+ +

Convert colorspace, transfer characteristics or color primaries. +Input video needs to have an even size. +

+

The filter accepts the following options: +

+
+
+
all
+

Specify all color properties at once. +

+

The accepted values are: +

+
‘bt470m’
+

BT.470M +

+
+
‘bt470bg’
+

BT.470BG +

+
+
‘bt601-6-525’
+

BT.601-6 525 +

+
+
‘bt601-6-625’
+

BT.601-6 625 +

+
+
‘bt709’
+

BT.709 +

+
+
‘smpte170m’
+

SMPTE-170M +

+
+
‘smpte240m’
+

SMPTE-240M +

+
+
‘bt2020’
+

BT.2020 +

+
+
+ +
+
space
+

Specify output colorspace. +

+

The accepted values are: +

+
‘bt709’
+

BT.709 +

+
+
‘fcc’
+

FCC +

+
+
‘bt470bg’
+

BT.470BG or BT.601-6 625 +

+
+
‘smpte170m’
+

SMPTE-170M or BT.601-6 525 +

+
+
‘smpte240m’
+

SMPTE-240M +

+
+
‘ycgco’
+

YCgCo +

+
+
‘bt2020ncl’
+

BT.2020 with non-constant luminance +

+
+
+ +
+
trc
+

Specify output transfer characteristics. +

+

The accepted values are: +

+
‘bt709’
+

BT.709 +

+
+
‘bt470m’
+

BT.470M +

+
+
‘bt470bg’
+

BT.470BG +

+
+
‘gamma22’
+

Constant gamma of 2.2 +

+
+
‘gamma28’
+

Constant gamma of 2.8 +

+
+
‘smpte170m’
+

SMPTE-170M, BT.601-6 625 or BT.601-6 525 +

+
+
‘smpte240m’
+

SMPTE-240M +

+
+
‘srgb’
+

SRGB +

+
+
‘iec61966-2-1’
+

iec61966-2-1 +

+
+
‘iec61966-2-4’
+

iec61966-2-4 +

+
+
‘xvycc’
+

xvycc +

+
+
‘bt2020-10’
+

BT.2020 for 10-bits content +

+
+
‘bt2020-12’
+

BT.2020 for 12-bits content +

+
+
+ +
+
primaries
+

Specify output color primaries. +

+

The accepted values are: +

+
‘bt709’
+

BT.709 +

+
+
‘bt470m’
+

BT.470M +

+
+
‘bt470bg’
+

BT.470BG or BT.601-6 625 +

+
+
‘smpte170m’
+

SMPTE-170M or BT.601-6 525 +

+
+
‘smpte240m’
+

SMPTE-240M +

+
+
‘film’
+

film +

+
+
‘smpte431’
+

SMPTE-431 +

+
+
‘smpte432’
+

SMPTE-432 +

+
+
‘bt2020’
+

BT.2020 +

+
+
‘jedec-p22’
+

JEDEC P22 phosphors +

+
+
+ +
+
range
+

Specify output color range. +

+

The accepted values are: +

+
‘tv’
+

TV (restricted) range +

+
+
‘mpeg’
+

MPEG (restricted) range +

+
+
‘pc’
+

PC (full) range +

+
+
‘jpeg’
+

JPEG (full) range +

+
+
+ +
+
format
+

Specify output color format. +

+

The accepted values are: +

+
‘yuv420p’
+

YUV 4:2:0 planar 8-bits +

+
+
‘yuv420p10’
+

YUV 4:2:0 planar 10-bits +

+
+
‘yuv420p12’
+

YUV 4:2:0 planar 12-bits +

+
+
‘yuv422p’
+

YUV 4:2:2 planar 8-bits +

+
+
‘yuv422p10’
+

YUV 4:2:2 planar 10-bits +

+
+
‘yuv422p12’
+

YUV 4:2:2 planar 12-bits +

+
+
‘yuv444p’
+

YUV 4:4:4 planar 8-bits +

+
+
‘yuv444p10’
+

YUV 4:4:4 planar 10-bits +

+
+
‘yuv444p12’
+

YUV 4:4:4 planar 12-bits +

+
+
+ +
+
fast
+

Do a fast conversion, which skips gamma/primary correction. This will take +significantly less CPU, but will be mathematically incorrect. To get output +compatible with that produced by the colormatrix filter, use fast=1. +

+
+
dither
+

Specify dithering mode. +

+

The accepted values are: +

+
‘none’
+

No dithering +

+
+
‘fsb’
+

Floyd-Steinberg dithering +

+
+ +
+
wpadapt
+

Whitepoint adaptation mode. +

+

The accepted values are: +

+
‘bradford’
+

Bradford whitepoint adaptation +

+
+
‘vonkries’
+

von Kries whitepoint adaptation +

+
+
‘identity’
+

identity whitepoint adaptation (i.e. no whitepoint adaptation) +

+
+ +
+
clipgamut
+

Controls how to clip out-of-gamut colors that arise as a result of colorspace conversion. +

+

The accepted values are: +

+
‘none’
+

No clipping of out of gamut colors. +

+
+
‘rgb’
+

Clips the RGB values to the [0, 1] range when building the gamma transfer LUTs. +

+
+ +
+
iall
+

Override all input properties at once. Same accepted values as all. +

+
+
ispace
+

Override input colorspace. Same accepted values as space. +

+
+
iprimaries
+

Override input color primaries. Same accepted values as primaries. +

+
+
itrc
+

Override input transfer characteristics. Same accepted values as trc. +

+
+
irange
+

Override input color range. Same accepted values as range. +

+
+
+ +

The filter converts the transfer characteristics, color space and color +primaries to the specified user values. The output value, if not specified, +is set to a default value based on the "all" property. If that property is +also not specified, the filter will log an error. The output color range and +format default to the same value as the input color range and format. The +input transfer characteristics, color space, color primaries and color range +should be set on the input data. If any of these are missing, the filter will +log an error and no conversion will take place. +

+

For example to convert the input to SMPTE-240M, use the command: +

+
colorspace=smpte240m
+
+ +

11.40 colortemperature

+

Adjust color temperature in video to simulate variations in ambient color temperature. +

+

The filter accepts the following options: +

+
+
temperature
+

Set the temperature in Kelvin. Allowed range is from 1000 to 40000. +Default value is 6500 K. +

+
+
mix
+

Set mixing with filtered output. Allowed range is from 0 to 1. +Default value is 1. +

+
+
pl
+

Set the amount of preserving lightness. Allowed range is from 0 to 1. +Default value is 0. +

+
+ +

11.40.1 Commands

+

This filter supports same commands as options. +

+

11.41 convolution

+ +

Apply convolution of 3x3, 5x5, 7x7 or horizontal/vertical up to 49 elements. +

+

The filter accepts the following options: +

+
+
0m
+
1m
+
2m
+
3m
+

Set matrix for each plane. +Matrix is sequence of 9, 25 or 49 signed integers in square mode, +and from 1 to 49 odd number of signed integers in row mode. +

+
+
0rdiv
+
1rdiv
+
2rdiv
+
3rdiv
+

Set multiplier for calculated value for each plane. +If unset or 0, it will be 1/sum of all matrix elements. +

+
+
0bias
+
1bias
+
2bias
+
3bias
+

Set bias for each plane. This value is added to the result of the multiplication. +Useful for making the overall image brighter or darker. Default is 0.0. +

+
+
0mode
+
1mode
+
2mode
+
3mode
+

Set matrix mode for each plane. Can be square, row or column. +Default is square. +

+
+ +

11.41.1 Commands

+ +

This filter supports the all above options as commands. +

+

11.41.2 Examples

+ +
    +
  • Apply sharpen: +
    +
    convolution="0 -1 0 -1 5 -1 0 -1 0:0 -1 0 -1 5 -1 0 -1 0:0 -1 0 -1 5 -1 0 -1 0:0 -1 0 -1 5 -1 0 -1 0"
    +
    + +
  • Apply blur: +
    +
    convolution="1 1 1 1 1 1 1 1 1:1 1 1 1 1 1 1 1 1:1 1 1 1 1 1 1 1 1:1 1 1 1 1 1 1 1 1:1/9:1/9:1/9:1/9"
    +
    + +
  • Apply edge enhance: +
    +
    convolution="0 0 0 -1 1 0 0 0 0:0 0 0 -1 1 0 0 0 0:0 0 0 -1 1 0 0 0 0:0 0 0 -1 1 0 0 0 0:5:1:1:1:0:128:128:128"
    +
    + +
  • Apply edge detect: +
    +
    convolution="0 1 0 1 -4 1 0 1 0:0 1 0 1 -4 1 0 1 0:0 1 0 1 -4 1 0 1 0:0 1 0 1 -4 1 0 1 0:5:5:5:1:0:128:128:128"
    +
    + +
  • Apply laplacian edge detector which includes diagonals: +
    +
    convolution="1 1 1 1 -8 1 1 1 1:1 1 1 1 -8 1 1 1 1:1 1 1 1 -8 1 1 1 1:1 1 1 1 -8 1 1 1 1:5:5:5:1:0:128:128:0"
    +
    + +
  • Apply emboss: +
    +
    convolution="-2 -1 0 -1 1 1 0 1 2:-2 -1 0 -1 1 1 0 1 2:-2 -1 0 -1 1 1 0 1 2:-2 -1 0 -1 1 1 0 1 2"
    +
    +
+ +

11.42 convolve

+ +

Apply 2D convolution of video stream in frequency domain using second stream +as impulse. +

+

The filter accepts the following options: +

+
+
planes
+

Set which planes to process. +

+
+
impulse
+

Set which impulse video frames will be processed, can be first +or all. Default is all. +

+
+ +

The convolve filter also supports the framesync options. +

+

11.43 copy

+ +

Copy the input video source unchanged to the output. This is mainly useful for +testing purposes. +

+

11.44 coreimage

+

Video filtering on GPU using Apple’s CoreImage API on OSX. +

+

Hardware acceleration is based on an OpenGL context. Usually, this means it is +processed by video hardware. However, software-based OpenGL implementations +exist which means there is no guarantee for hardware processing. It depends on +the respective OSX. +

+

There are many filters and image generators provided by Apple that come with a +large variety of options. The filter has to be referenced by its name along +with its options. +

+

The coreimage filter accepts the following options: +

+
list_filters
+

List all available filters and generators along with all their respective +options as well as possible minimum and maximum values along with the default +values. +

+
list_filters=true
+
+ +
+
filter
+

Specify all filters by their respective name and options. +Use list_filters to determine all valid filter names and options. +Numerical options are specified by a float value and are automatically clamped +to their respective value range. Vector and color options have to be specified +by a list of space separated float values. Character escaping has to be done. +A special option name default is available to use default options for a +filter. +

+

It is required to specify either default or at least one of the filter options. +All omitted options are used with their default values. +The syntax of the filter string is as follows: +

+
filter=<NAME>@<OPTION>=<VALUE>[@<OPTION>=<VALUE>][@...][#<NAME>@<OPTION>=<VALUE>[@<OPTION>=<VALUE>][@...]][#...]
+
+ +
+
output_rect
+

Specify a rectangle where the output of the filter chain is copied into the +input image. It is given by a list of space separated float values: +

+
output_rect=x\ y\ width\ height
+
+

If not given, the output rectangle equals the dimensions of the input image. +The output rectangle is automatically cropped at the borders of the input +image. Negative values are valid for each component. +

+
output_rect=25\ 25\ 100\ 100
+
+
+
+ +

Several filters can be chained for successive processing without GPU-HOST +transfers allowing for fast processing of complex filter chains. +Currently, only filters with zero (generators) or exactly one (filters) input +image and one output image are supported. Also, transition filters are not yet +usable as intended. +

+

Some filters generate output images with additional padding depending on the +respective filter kernel. The padding is automatically removed to ensure the +filter output has the same size as the input image. +

+

For image generators, the size of the output image is determined by the +previous output image of the filter chain or the input image of the whole +filterchain, respectively. The generators do not use the pixel information of +this image to generate their output. However, the generated output is +blended onto this image, resulting in partial or complete coverage of the +output image. +

+

The coreimagesrc video source can be used for generating input images +which are directly fed into the filter chain. By using it, providing input +images by another video source or an input video is not required. +

+

11.44.1 Examples

+ +
    +
  • List all filters available: +
    +
    coreimage=list_filters=true
    +
    + +
  • Use the CIBoxBlur filter with default options to blur an image: +
    +
    coreimage=filter=CIBoxBlur@default
    +
    + +
  • Use a filter chain with CISepiaTone at default values and CIVignetteEffect with +its center at 100x100 and a radius of 50 pixels: +
    +
    coreimage=filter=CIBoxBlur@default#CIVignetteEffect@inputCenter=100\ 100@inputRadius=50
    +
    + +
  • Use nullsrc and CIQRCodeGenerator to create a QR code for the FFmpeg homepage, +given as complete and escaped command-line for Apple’s standard bash shell: +
    +
    ffmpeg -f lavfi -i nullsrc=s=100x100,coreimage=filter=CIQRCodeGenerator@inputMessage=https\\\\\://FFmpeg.org/@inputCorrectionLevel=H -frames:v 1 QRCode.png
    +
    +
+ +

11.45 corr

+ +

Obtain the correlation between two input videos. +

+

This filter takes two input videos. +

+

Both input videos must have the same resolution and pixel format for +this filter to work correctly. Also it assumes that both inputs +have the same number of frames, which are compared one by one. +

+

The obtained per component, average, min and max correlation is printed through +the logging system. +

+

The filter stores the calculated correlation of each frame in frame metadata. +

+

This filter also supports the framesync options. +

+

In the below example the input file main.mpg being processed is compared +with the reference file ref.mpg. +

+
+
ffmpeg -i main.mpg -i ref.mpg -lavfi corr -f null -
+
+ +

11.46 cover_rect

+ +

Cover a rectangular object +

+

It accepts the following options: +

+
+
cover
+

Filepath of the optional cover image, needs to be in yuv420. +

+
+
mode
+

Set covering mode. +

+

It accepts the following values: +

+
‘cover’
+

cover it by the supplied image +

+
‘blur’
+

cover it by interpolating the surrounding pixels +

+
+ +

Default value is blur. +

+
+ +

11.46.1 Examples

+ +
    +
  • Cover a rectangular object by the supplied image of a given video using ffmpeg: +
    +
    ffmpeg -i file.ts -vf find_rect=newref.pgm,cover_rect=cover.jpg:mode=cover new.mkv
    +
    +
+ +

11.47 crop

+ +

Crop the input video to given dimensions. +

+

It accepts the following parameters: +

+
+
w, out_w
+

The width of the output video. It defaults to iw. +This expression is evaluated only once during the filter +configuration, or when the ‘w’ or ‘out_w’ command is sent. +

+
+
h, out_h
+

The height of the output video. It defaults to ih. +This expression is evaluated only once during the filter +configuration, or when the ‘h’ or ‘out_h’ command is sent. +

+
+
x
+

The horizontal position, in the input video, of the left edge of the output +video. It defaults to (in_w-out_w)/2. +This expression is evaluated per-frame. +

+
+
y
+

The vertical position, in the input video, of the top edge of the output video. +It defaults to (in_h-out_h)/2. +This expression is evaluated per-frame. +

+
+
keep_aspect
+

If set to 1 will force the output display aspect ratio +to be the same of the input, by changing the output sample aspect +ratio. It defaults to 0. +

+
+
exact
+

Enable exact cropping. If enabled, subsampled videos will be cropped at exact +width/height/x/y as specified and will not be rounded to nearest smaller value. +It defaults to 0. +

+
+ +

The out_w, out_h, x, y parameters are +expressions containing the following constants: +

+
+
x
+
y
+

The computed values for x and y. They are evaluated for +each new frame. +

+
+
in_w
+
in_h
+

The input width and height. +

+
+
iw
+
ih
+

These are the same as in_w and in_h. +

+
+
out_w
+
out_h
+

The output (cropped) width and height. +

+
+
ow
+
oh
+

These are the same as out_w and out_h. +

+
+
a
+

same as iw / ih +

+
+
sar
+

input sample aspect ratio +

+
+
dar
+

input display aspect ratio, it is the same as (iw / ih) * sar +

+
+
hsub
+
vsub
+

horizontal and vertical chroma subsample values. For example for the +pixel format "yuv422p" hsub is 2 and vsub is 1. +

+
+
n
+

The number of the input frame, starting from 0. +

+
+
pos
+

the position in the file of the input frame, NAN if unknown; deprecated, +do not use +

+
+
t
+

The timestamp expressed in seconds. It’s NAN if the input timestamp is unknown. +

+
+
+ +

The expression for out_w may depend on the value of out_h, +and the expression for out_h may depend on out_w, but they +cannot depend on x and y, as x and y are +evaluated after out_w and out_h. +

+

The x and y parameters specify the expressions for the +position of the top-left corner of the output (non-cropped) area. They +are evaluated for each frame. If the evaluated value is not valid, it +is approximated to the nearest valid value. +

+

The expression for x may depend on y, and the expression +for y may depend on x. +

+

11.47.1 Examples

+ +
    +
  • Crop area with size 100x100 at position (12,34). +
    +
    crop=100:100:12:34
    +
    + +

    Using named options, the example above becomes: +

    +
    crop=w=100:h=100:x=12:y=34
    +
    + +
  • Crop the central input area with size 100x100: +
    +
    crop=100:100
    +
    + +
  • Crop the central input area with size 2/3 of the input video: +
    +
    crop=2/3*in_w:2/3*in_h
    +
    + +
  • Crop the input video central square: +
    +
    crop=out_w=in_h
    +crop=in_h
    +
    + +
  • Delimit the rectangle with the top-left corner placed at position +100:100 and the right-bottom corner corresponding to the right-bottom +corner of the input image. +
    +
    crop=in_w-100:in_h-100:100:100
    +
    + +
  • Crop 10 pixels from the left and right borders, and 20 pixels from +the top and bottom borders +
    +
    crop=in_w-2*10:in_h-2*20
    +
    + +
  • Keep only the bottom right quarter of the input image: +
    +
    crop=in_w/2:in_h/2:in_w/2:in_h/2
    +
    + +
  • Crop height for getting Greek harmony: +
    +
    crop=in_w:1/PHI*in_w
    +
    + +
  • Apply trembling effect: +
    +
    crop=in_w/2:in_h/2:(in_w-out_w)/2+((in_w-out_w)/2)*sin(n/10):(in_h-out_h)/2 +((in_h-out_h)/2)*sin(n/7)
    +
    + +
  • Apply erratic camera effect depending on timestamp: +
    +
    crop=in_w/2:in_h/2:(in_w-out_w)/2+((in_w-out_w)/2)*sin(t*10):(in_h-out_h)/2 +((in_h-out_h)/2)*sin(t*13)
    +
    + +
  • Set x depending on the value of y: +
    +
    crop=in_w/2:in_h/2:y:10+10*sin(n/10)
    +
    +
+ +

11.47.2 Commands

+ +

This filter supports the following commands: +

+
w, out_w
+
h, out_h
+
x
+
y
+

Set width/height of the output video and the horizontal/vertical position +in the input video. +The command accepts the same syntax of the corresponding option. +

+

If the specified expression is not valid, it is kept at its current +value. +

+
+ +

11.48 cropdetect

+ +

Auto-detect the crop size. +

+

It calculates the necessary cropping parameters and prints the +recommended parameters via the logging system. The detected dimensions +correspond to the non-black or video area of the input video according to mode. +

+

It accepts the following parameters: +

+
+
mode
+

Depending on mode crop detection is based on either the mere black value of surrounding pixels or a combination of motion vectors and edge pixels. +

+
+
‘black’
+

Detect black pixels surrounding the playing video. For fine control use option limit. +

+
+
‘mvedges’
+

Detect the playing video by the motion vectors inside the video and scanning for edge pixels typically forming the border of a playing video. +

+
+ +
+
limit
+

Set higher black value threshold, which can be optionally specified +from nothing (0) to everything (255 for 8-bit based formats). An intensity +value greater to the set value is considered non-black. It defaults to 24. +You can also specify a value between 0.0 and 1.0 which will be scaled depending +on the bitdepth of the pixel format. +

+
+
round
+

The value which the width/height should be divisible by. It defaults to +16. The offset is automatically adjusted to center the video. Use 2 to +get only even dimensions (needed for 4:2:2 video). 16 is best when +encoding to most video codecs. +

+
+
skip
+

Set the number of initial frames for which evaluation is skipped. +Default is 2. Range is 0 to INT_MAX. +

+
+
reset_count, reset
+

Set the counter that determines after how many frames cropdetect will +reset the previously detected largest video area and start over to +detect the current optimal crop area. Default value is 0. +

+

This can be useful when channel logos distort the video area. 0 +indicates ’never reset’, and returns the largest area encountered during +playback. +

+
+
mv_threshold
+

Set motion in pixel units as threshold for motion detection. It defaults to 8. +

+
+
low
+
high
+

Set low and high threshold values used by the Canny thresholding +algorithm. +

+

The high threshold selects the "strong" edge pixels, which are then +connected through 8-connectivity with the "weak" edge pixels selected +by the low threshold. +

+

low and high threshold values must be chosen in the range +[0,1], and low should be lesser or equal to high. +

+

Default value for low is 5/255, and default value for high +is 15/255. +

+
+ +

11.48.1 Examples

+ +
    +
  • Find video area surrounded by black borders: +
    +
    ffmpeg -i file.mp4 -vf cropdetect,metadata=mode=print -f null -
    +
    + +
  • Find an embedded video area, generate motion vectors beforehand: +
    +
    ffmpeg -i file.mp4 -vf mestimate,cropdetect=mode=mvedges,metadata=mode=print -f null -
    +
    + +
  • Find an embedded video area, use motion vectors from decoder: +
    +
    ffmpeg -flags2 +export_mvs -i file.mp4 -vf cropdetect=mode=mvedges,metadata=mode=print -f null -
    +
    +
+ +

11.48.2 Commands

+ +

This filter supports the following commands: +

+
+
limit
+

The command accepts the same syntax of the corresponding option. +If the specified expression is not valid, it is kept at its current value. +

+
+ +

11.49 cue

+ +

Delay video filtering until a given wallclock timestamp. The filter first +passes on preroll amount of frames, then it buffers at most +buffer amount of frames and waits for the cue. After reaching the cue +it forwards the buffered frames and also any subsequent frames coming in its +input. +

+

The filter can be used synchronize the output of multiple ffmpeg processes for +realtime output devices like decklink. By putting the delay in the filtering +chain and pre-buffering frames the process can pass on data to output almost +immediately after the target wallclock timestamp is reached. +

+

Perfect frame accuracy cannot be guaranteed, but the result is good enough for +some use cases. +

+
+
cue
+

The cue timestamp expressed in a UNIX timestamp in microseconds. Default is 0. +

+
+
preroll
+

The duration of content to pass on as preroll expressed in seconds. Default is 0. +

+
+
buffer
+

The maximum duration of content to buffer before waiting for the cue expressed +in seconds. Default is 0. +

+
+
+ +

11.50 curves

+ +

Apply color adjustments using curves. +

+

This filter is similar to the Adobe Photoshop and GIMP curves tools. Each +component (red, green and blue) has its values defined by N key points +tied from each other using a smooth curve. The x-axis represents the pixel +values from the input frame, and the y-axis the new pixel values to be set for +the output frame. +

+

By default, a component curve is defined by the two points (0;0) and +(1;1). This creates a straight line where each original pixel value is +"adjusted" to its own value, which means no change to the image. +

+

The filter allows you to redefine these two points and add some more. A new +curve will be defined to pass smoothly through all these new coordinates. The +new defined points need to be strictly increasing over the x-axis, and their +x and y values must be in the [0;1] interval. The curve is +formed by using a natural or monotonic cubic spline interpolation, depending +on the interp option (default: natural). The natural +spline produces a smoother curve in general while the monotonic (pchip) +spline guarantees the transitions between the specified points to be +monotonic. If the computed curves happened to go outside the vector spaces, +the values will be clipped accordingly. +

+

The filter accepts the following options: +

+
+
preset
+

Select one of the available color presets. This option can be used in addition +to the r, g, b parameters; in this case, the later +options takes priority on the preset values. +Available presets are: +

+
‘none’
+
‘color_negative’
+
‘cross_process’
+
‘darker’
+
‘increase_contrast’
+
‘lighter’
+
‘linear_contrast’
+
‘medium_contrast’
+
‘negative’
+
‘strong_contrast’
+
‘vintage’
+
+

Default is none. +

+
master, m
+

Set the master key points. These points will define a second pass mapping. It +is sometimes called a "luminance" or "value" mapping. It can be used with +r, g, b or all since it acts like a +post-processing LUT. +

+
red, r
+

Set the key points for the red component. +

+
green, g
+

Set the key points for the green component. +

+
blue, b
+

Set the key points for the blue component. +

+
all
+

Set the key points for all components (not including master). +Can be used in addition to the other key points component +options. In this case, the unset component(s) will fallback on this +all setting. +

+
psfile
+

Specify a Photoshop curves file (.acv) to import the settings from. +

+
plot
+

Save Gnuplot script of the curves in specified file. +

+
interp
+

Specify the kind of interpolation. Available algorithms are: +

+
‘natural’
+

Natural cubic spline using a piece-wise cubic polynomial that is twice continuously differentiable. +

+
‘pchip’
+

Monotonic cubic spline using a piecewise cubic Hermite interpolating polynomial (PCHIP). +

+
+ +
+
+ +

To avoid some filtergraph syntax conflicts, each key points list need to be +defined using the following syntax: x0/y0 x1/y1 x2/y2 .... +

+

11.50.1 Commands

+ +

This filter supports same commands as options. +

+

11.50.2 Examples

+ +
    +
  • Increase slightly the middle level of blue: +
    +
    curves=blue='0/0 0.5/0.58 1/1'
    +
    + +
  • Vintage effect: +
    +
    curves=r='0/0.11 .42/.51 1/0.95':g='0/0 0.50/0.48 1/1':b='0/0.22 .49/.44 1/0.8'
    +
    +

    Here we obtain the following coordinates for each components: +

    +
    red
    +

    (0;0.11) (0.42;0.51) (1;0.95) +

    +
    green
    +

    (0;0) (0.50;0.48) (1;1) +

    +
    blue
    +

    (0;0.22) (0.49;0.44) (1;0.80) +

    +
    + +
  • The previous example can also be achieved with the associated built-in preset: +
    +
    curves=preset=vintage
    +
    + +
  • Or simply: +
    +
    curves=vintage
    +
    + +
  • Use a Photoshop preset and redefine the points of the green component: +
    +
    curves=psfile='MyCurvesPresets/purple.acv':green='0/0 0.45/0.53 1/1'
    +
    + +
  • Check out the curves of the cross_process profile using ffmpeg +and gnuplot: +
    +
    ffmpeg -f lavfi -i color -vf curves=cross_process:plot=/tmp/curves.plt -frames:v 1 -f null -
    +gnuplot -p /tmp/curves.plt
    +
    +
+ +

11.51 datascope

+ +

Video data analysis filter. +

+

This filter shows hexadecimal pixel values of part of video. +

+

The filter accepts the following options: +

+
+
size, s
+

Set output video size. +

+
+
x
+

Set x offset from where to pick pixels. +

+
+
y
+

Set y offset from where to pick pixels. +

+
+
mode
+

Set scope mode, can be one of the following: +

+
‘mono’
+

Draw hexadecimal pixel values with white color on black background. +

+
+
‘color’
+

Draw hexadecimal pixel values with input video pixel color on black +background. +

+
+
‘color2’
+

Draw hexadecimal pixel values on color background picked from input video, +the text color is picked in such way so its always visible. +

+
+ +
+
axis
+

Draw rows and columns numbers on left and top of video. +

+
+
opacity
+

Set background opacity. +

+
+
format
+

Set display number format. Can be hex, or dec. Default is hex. +

+
+
components
+

Set pixel components to display. By default all pixel components are displayed. +

+
+ +

11.51.1 Commands

+ +

This filter supports same commands as options excluding size option. +

+

11.52 dblur

+

Apply Directional blur filter. +

+

The filter accepts the following options: +

+
+
angle
+

Set angle of directional blur. Default is 45. +

+
+
radius
+

Set radius of directional blur. Default is 5. +

+
+
planes
+

Set which planes to filter. By default all planes are filtered. +

+
+ +

11.52.1 Commands

+

This filter supports same commands as options. +The command accepts the same syntax of the corresponding option. +

+

If the specified expression is not valid, it is kept at its current +value. +

+

11.53 dctdnoiz

+ +

Denoise frames using 2D DCT (frequency domain filtering). +

+

This filter is not designed for real time. +

+

The filter accepts the following options: +

+
+
sigma, s
+

Set the noise sigma constant. +

+

This sigma defines a hard threshold of 3 * sigma; every DCT +coefficient (absolute value) below this threshold with be dropped. +

+

If you need a more advanced filtering, see expr. +

+

Default is 0. +

+
+
overlap
+

Set number overlapping pixels for each block. Since the filter can be slow, you +may want to reduce this value, at the cost of a less effective filter and the +risk of various artefacts. +

+

If the overlapping value doesn’t permit processing the whole input width or +height, a warning will be displayed and according borders won’t be denoised. +

+

Default value is blocksize-1, which is the best possible setting. +

+
+
expr, e
+

Set the coefficient factor expression. +

+

For each coefficient of a DCT block, this expression will be evaluated as a +multiplier value for the coefficient. +

+

If this is option is set, the sigma option will be ignored. +

+

The absolute value of the coefficient can be accessed through the c +variable. +

+
+
n
+

Set the blocksize using the number of bits. 1<<n defines the +blocksize, which is the width and height of the processed blocks. +

+

The default value is 3 (8x8) and can be raised to 4 for a +blocksize of 16x16. Note that changing this setting has huge consequences +on the speed processing. Also, a larger block size does not necessarily means a +better de-noising. +

+
+ +

11.53.1 Examples

+ +

Apply a denoise with a sigma of 4.5: +

+
dctdnoiz=4.5
+
+ +

The same operation can be achieved using the expression system: +

+
dctdnoiz=e='gte(c, 4.5*3)'
+
+ +

Violent denoise using a block size of 16x16: +

+
dctdnoiz=15:n=4
+
+ +

11.54 deband

+ +

Remove banding artifacts from input video. +It works by replacing banded pixels with average value of referenced pixels. +

+

The filter accepts the following options: +

+
+
1thr
+
2thr
+
3thr
+
4thr
+

Set banding detection threshold for each plane. Default is 0.02. +Valid range is 0.00003 to 0.5. +If difference between current pixel and reference pixel is less than threshold, +it will be considered as banded. +

+
+
range, r
+

Banding detection range in pixels. Default is 16. If positive, random number +in range 0 to set value will be used. If negative, exact absolute value +will be used. +The range defines square of four pixels around current pixel. +

+
+
direction, d
+

Set direction in radians from which four pixel will be compared. If positive, +random direction from 0 to set direction will be picked. If negative, exact of +absolute value will be picked. For example direction 0, -PI or -2*PI radians +will pick only pixels on same row and -PI/2 will pick only pixels on same +column. +

+
+
blur, b
+

If enabled, current pixel is compared with average value of all four +surrounding pixels. The default is enabled. If disabled current pixel is +compared with all four surrounding pixels. The pixel is considered banded +if only all four differences with surrounding pixels are less than threshold. +

+
+
coupling, c
+

If enabled, current pixel is changed if and only if all pixel components are banded, +e.g. banding detection threshold is triggered for all color components. +The default is disabled. +

+
+ +

11.54.1 Commands

+ +

This filter supports the all above options as commands. +

+

11.55 deblock

+ +

Remove blocking artifacts from input video. +

+

The filter accepts the following options: +

+
+
filter
+

Set filter type, can be weak or strong. Default is strong. +This controls what kind of deblocking is applied. +

+
+
block
+

Set size of block, allowed range is from 4 to 512. Default is 8. +

+
+
alpha
+
beta
+
gamma
+
delta
+

Set blocking detection thresholds. Allowed range is 0 to 1. +Defaults are: 0.098 for alpha and 0.05 for the rest. +Using higher threshold gives more deblocking strength. +Setting alpha controls threshold detection at exact edge of block. +Remaining options controls threshold detection near the edge. Each one for +below/above or left/right. Setting any of those to 0 disables +deblocking. +

+
+
planes
+

Set planes to filter. Default is to filter all available planes. +

+
+ +

11.55.1 Examples

+ +
    +
  • Deblock using weak filter and block size of 4 pixels. +
    +
    deblock=filter=weak:block=4
    +
    + +
  • Deblock using strong filter, block size of 4 pixels and custom thresholds for +deblocking more edges. +
    +
    deblock=filter=strong:block=4:alpha=0.12:beta=0.07:gamma=0.06:delta=0.05
    +
    + +
  • Similar as above, but filter only first plane. +
    +
    deblock=filter=strong:block=4:alpha=0.12:beta=0.07:gamma=0.06:delta=0.05:planes=1
    +
    + +
  • Similar as above, but filter only second and third plane. +
    +
    deblock=filter=strong:block=4:alpha=0.12:beta=0.07:gamma=0.06:delta=0.05:planes=6
    +
    +
+ +

11.55.2 Commands

+ +

This filter supports the all above options as commands. +

+

11.56 decimate

+ +

Drop duplicated frames at regular intervals. +

+

The filter accepts the following options: +

+
+
cycle
+

Set the number of frames from which one will be dropped. Setting this to +N means one frame in every batch of N frames will be dropped. +Default is 5. +

+
+
dupthresh
+

Set the threshold for duplicate detection. If the difference metric for a frame +is less than or equal to this value, then it is declared as duplicate. Default +is 1.1 +

+
+
scthresh
+

Set scene change threshold. Default is 15. +

+
+
blockx
+
blocky
+

Set the size of the x and y-axis blocks used during metric calculations. +Larger blocks give better noise suppression, but also give worse detection of +small movements. Must be a power of two. Default is 32. +

+
+
ppsrc
+

Mark main input as a pre-processed input and activate clean source input +stream. This allows the input to be pre-processed with various filters to help +the metrics calculation while keeping the frame selection lossless. When set to +1, the first stream is for the pre-processed input, and the second +stream is the clean source from where the kept frames are chosen. Default is +0. +

+
+
chroma
+

Set whether or not chroma is considered in the metric calculations. Default is +1. +

+
+
mixed
+

Set whether or not the input only partially contains content to be decimated. +Default is false. +If enabled video output stream will be in variable frame rate. +

+
+ +

11.57 deconvolve

+ +

Apply 2D deconvolution of video stream in frequency domain using second stream +as impulse. +

+

The filter accepts the following options: +

+
+
planes
+

Set which planes to process. +

+
+
impulse
+

Set which impulse video frames will be processed, can be first +or all. Default is all. +

+
+
noise
+

Set noise when doing divisions. Default is 0.0000001. Useful when width +and height are not same and not power of 2 or if stream prior to convolving +had noise. +

+
+ +

The deconvolve filter also supports the framesync options. +

+

11.58 dedot

+ +

Reduce cross-luminance (dot-crawl) and cross-color (rainbows) from video. +

+

It accepts the following options: +

+
+
m
+

Set mode of operation. Can be combination of dotcrawl for cross-luminance reduction and/or +rainbows for cross-color reduction. +

+
+
lt
+

Set spatial luma threshold. Lower values increases reduction of cross-luminance. +

+
+
tl
+

Set tolerance for temporal luma. Higher values increases reduction of cross-luminance. +

+
+
tc
+

Set tolerance for chroma temporal variation. Higher values increases reduction of cross-color. +

+
+
ct
+

Set temporal chroma threshold. Lower values increases reduction of cross-color. +

+
+ +

11.59 deflate

+ +

Apply deflate effect to the video. +

+

This filter replaces the pixel by the local(3x3) average by taking into account +only values lower than the pixel. +

+

It accepts the following options: +

+
+
threshold0
+
threshold1
+
threshold2
+
threshold3
+

Limit the maximum change for each plane, default is 65535. +If 0, plane will remain unchanged. +

+
+ +

11.59.1 Commands

+ +

This filter supports the all above options as commands. +

+

11.60 deflicker

+ +

Remove temporal frame luminance variations. +

+

It accepts the following options: +

+
+
size, s
+

Set moving-average filter size in frames. Default is 5. Allowed range is 2 - 129. +

+
+
mode, m
+

Set averaging mode to smooth temporal luminance variations. +

+

Available values are: +

+
‘am’
+

Arithmetic mean +

+
+
‘gm’
+

Geometric mean +

+
+
‘hm’
+

Harmonic mean +

+
+
‘qm’
+

Quadratic mean +

+
+
‘cm’
+

Cubic mean +

+
+
‘pm’
+

Power mean +

+
+
‘median’
+

Median +

+
+ +
+
bypass
+

Do not actually modify frame. Useful when one only wants metadata. +

+
+ +

11.61 deinterlace_d3d12

+ +

Deinterlace the input video using D3D12 hardware-accelerated video processing. +

+

This filter uses the DirectX 12 Video Processor to perform deinterlacing on +GPU, supporting both simple bob and driver-defined advanced (motion-adaptive) +algorithms. It requires the input to be in d3d12 hardware pixel format. +

+

The filter automatically queries the hardware for the required number of +temporal reference frames (if needed) and manages a frame queue to supply them. +

+

It accepts the following options: +

+
+
method
+

The deinterlacing method. It accepts one of the following values: +

+
+
‘default’
+

Select the best deinterlacing method available on the hardware. +If the driver supports custom, that method is used; otherwise +falls back to bob. This is the default. +

+
+
‘bob’
+

Bob deinterlacing. Each field is independently scaled to full +frame height. Simple and fast, but may produce visible bob artifacts +on moving content. +

+
+
‘custom’
+

Driver-defined advanced deinterlacing. The exact algorithm is +hardware-specific and typically employs motion-adaptive techniques +with temporal reference frames for higher quality. +

+
+ +

The default value is default. +

+
+
mode
+

Specify the interlacing mode. It accepts one of the following values: +

+
+
‘frame’
+

Send one frame for each frame. The output frame rate equals the input +frame rate. This is the default. +

+
+
‘field’
+

Send one frame for each field. The output frame rate is double the +input frame rate. +

+
+ +

The default value is frame. +

+
+
deint
+

Specify which frames to deinterlace. It accepts one of the following values: +

+
+
‘all’
+

Deinterlace all frames. This is the default. +

+
+
‘interlaced’
+

Only deinterlace frames marked as interlaced; progressive frames are +passed through unchanged. +

+
+ +

The default value is all. +

+
+ +

11.61.1 Examples

+ +
    +
  • Deinterlace an interlaced video using software decoding and the best +available hardware deinterlacing method: +
    +
    ffmpeg -init_hw_device d3d12va=d3d12 -filter_hw_device d3d12 -i input.ts \
    +       -vf "format=nv12,hwupload,deinterlace_d3d12=method=default,hwdownload,format=nv12" \
    +       -c:v libx264 -crf 18 output.mp4
    +
    + +
  • Deinterlace at field rate (double the frame rate) using bob method with +hardware decoding (full d3d12 pipeline): +
    +
    ffmpeg -hwaccel d3d12va -hwaccel_output_format d3d12 -i input.ts \
    +       -vf "deinterlace_d3d12=method=bob:mode=field" \
    +       -c:v h264_d3d12va output.mp4
    +
    + +
  • Deinterlace only interlaced frames, passing progressive frames through: +
    +
    ffmpeg -init_hw_device d3d12va=d3d12 -filter_hw_device d3d12 -i input.ts \
    +       -vf "format=nv12,hwupload,deinterlace_d3d12=deint=interlaced,hwdownload,format=nv12" \
    +       -c:v libx264 -crf 18 output.mp4
    +
    + +
+ +

11.62 dejudder

+ +

Remove judder produced by partially interlaced telecined content. +

+

Judder can be introduced, for instance, by pullup filter. If the original +source was partially telecined content then the output of pullup,dejudder +will have a variable frame rate. May change the recorded frame rate of the +container. Aside from that change, this filter will not affect constant frame +rate video. +

+

The option available in this filter is: +

+
cycle
+

Specify the length of the window over which the judder repeats. +

+

Accepts any integer greater than 1. Useful values are: +

+
‘4’
+

If the original was telecined from 24 to 30 fps (Film to NTSC). +

+
+
‘5’
+

If the original was telecined from 25 to 30 fps (PAL to NTSC). +

+
+
‘20’
+

If a mixture of the two. +

+
+ +

The default is ‘4’. +

+
+ +

11.63 delogo

+ +

Suppress a TV station logo by a simple interpolation of the surrounding +pixels. Just set a rectangle covering the logo and watch it disappear +(and sometimes something even uglier appear - your mileage may vary). +

+

It accepts the following parameters: +

+
x
+
y
+

Specify the top left corner coordinates of the logo. They must be +specified. +

+
+
w
+
h
+

Specify the width and height of the logo to clear. They must be +specified. +

+
+
show
+

When set to 1, a green rectangle is drawn on the screen to simplify +finding the right x, y, w, and h parameters. +The default value is 0. +

+

The rectangle is drawn on the outermost pixels which will be (partly) +replaced with interpolated values. The values of the next pixels +immediately outside this rectangle in each direction will be used to +compute the interpolated pixel values inside the rectangle. +

+
+
+ +

11.63.1 Examples

+ +
    +
  • Set a rectangle covering the area with top left corner coordinates 0,0 +and size 100x77: +
    +
    delogo=x=0:y=0:w=100:h=77
    +
    + +
+ +

11.64 derain

+ +

Remove the rain in the input image/video by applying the derain methods based on +convolutional neural networks. Supported models: +

+ + +

Training as well as model generation scripts are provided in +the repository at https://github.com/XueweiMeng/derain_filter.git. +

+

The filter accepts the following options: +

+
+
filter_type
+

Specify which filter to use. This option accepts the following values: +

+
+
‘derain’
+

Derain filter. To conduct derain filter, you need to use a derain model. +

+
+
‘dehaze’
+

Dehaze filter. To conduct dehaze filter, you need to use a dehaze model. +

+
+

Default value is ‘derain’. +

+
+
dnn_backend
+

Specify which DNN backend to use for model loading and execution. This option accepts +the following values: +

+
+
‘tensorflow’
+

TensorFlow backend. To enable this backend you +need to install the TensorFlow for C library (see +https://www.tensorflow.org/install/lang_c) and configure FFmpeg with +--enable-libtensorflow +

+
+ +
+
model
+

Set path to model file specifying network architecture and its parameters. +Note that different backends use different file formats. TensorFlow can load files for only its format. +

+
+ +

To get full functionality (such as async execution), please use the dnn_processing filter. +

+

11.65 deshake

+ +

Attempt to fix small changes in horizontal and/or vertical shift. This +filter helps remove camera shake from hand-holding a camera, bumping a +tripod, moving on a vehicle, etc. +

+

The filter accepts the following options: +

+
+
x
+
y
+
w
+
h
+

Specify a rectangular area where to limit the search for motion +vectors. +If desired the search for motion vectors can be limited to a +rectangular area of the frame defined by its top left corner, width +and height. These parameters have the same meaning as the drawbox +filter which can be used to visualise the position of the bounding +box. +

+

This is useful when simultaneous movement of subjects within the frame +might be confused for camera motion by the motion vector search. +

+

If any or all of x, y, w and h are set to -1 +then the full frame is used. This allows later options to be set +without specifying the bounding box for the motion vector search. +

+

Default - search the whole frame. +

+
+
rx
+
ry
+

Specify the maximum extent of movement in x and y directions in the +range 0-64 pixels. Default 16. +

+
+
edge
+

Specify how to generate pixels to fill blanks at the edge of the +frame. Available values are: +

+
‘blank, 0’
+

Fill zeroes at blank locations +

+
‘original, 1’
+

Original image at blank locations +

+
‘clamp, 2’
+

Extruded edge value at blank locations +

+
‘mirror, 3’
+

Mirrored edge at blank locations +

+
+

Default value is ‘mirror’. +

+
+
blocksize
+

Specify the blocksize to use for motion search. Range 4-128 pixels, +default 8. +

+
+
contrast
+

Specify the contrast threshold for blocks. Only blocks with more than +the specified contrast (difference between darkest and lightest +pixels) will be considered. Range 1-255, default 125. +

+
+
search
+

Specify the search strategy. Available values are: +

+
‘exhaustive, 0’
+

Set exhaustive search +

+
‘less, 1’
+

Set less exhaustive search. +

+
+

Default value is ‘exhaustive’. +

+
+
filename
+

If set then a detailed log of the motion search is written to the +specified file. +

+
+
+ +

11.66 despill

+ +

Remove unwanted contamination of foreground colors, caused by reflected color of +greenscreen or bluescreen. +

+

This filter accepts the following options: +

+
+
type
+

Set what type of despill to use. +

+
+
mix
+

Set how spillmap will be generated. +

+
+
expand
+

Set how much to get rid of still remaining spill. +

+
+
red
+

Controls amount of red in spill area. +

+
+
green
+

Controls amount of green in spill area. +Should be -1 for greenscreen. +

+
+
blue
+

Controls amount of blue in spill area. +Should be -1 for bluescreen. +

+
+
brightness
+

Controls brightness of spill area, preserving colors. +

+
+
alpha
+

Modify alpha from generated spillmap. +

+
+ +

11.66.1 Commands

+ +

This filter supports the all above options as commands. +

+

11.67 detelecine

+ +

Apply an exact inverse of the telecine operation. It requires a predefined +pattern specified using the pattern option which must be the same as that passed +to the telecine filter. +

+

This filter accepts the following options: +

+
+
first_field
+
+
‘top, t’
+

top field first +

+
‘bottom, b’
+

bottom field first +The default value is top. +

+
+ +
+
pattern
+

A string of numbers representing the pulldown pattern you wish to apply. +The default value is 23. +

+
+
start_frame
+

A number representing position of the first frame with respect to the telecine +pattern. This is to be used if the stream is cut. The default value is 0. +

+
+ +

11.68 dilation

+ +

Apply dilation effect to the video. +

+

This filter replaces the pixel by the local(3x3) maximum. +

+

It accepts the following options: +

+
+
threshold0
+
threshold1
+
threshold2
+
threshold3
+

Limit the maximum change for each plane, default is 65535. +If 0, plane will remain unchanged. +

+
+
coordinates
+

Flag which specifies the pixel to refer to. Default is 255 i.e. all eight +pixels are used. +

+

Flags to local 3x3 coordinates maps like this: +

+

1 2 3 + 4 5 + 6 7 8 +

+
+ +

11.68.1 Commands

+ +

This filter supports the all above options as commands. +

+

11.69 displace

+ +

Displace pixels as indicated by second and third input stream. +

+

It takes three input streams and outputs one stream, the first input is the +source, and second and third input are displacement maps. +

+

The second input specifies how much to displace pixels along the +x-axis, while the third input specifies how much to displace pixels +along the y-axis. +If one of displacement map streams terminates, last frame from that +displacement map will be used. +

+

Note that once generated, displacements maps can be reused over and over again. +

+

A description of the accepted options follows. +

+
+
edge
+

Set displace behavior for pixels that are out of range. +

+

Available values are: +

+
‘blank’
+

Missing pixels are replaced by black pixels. +

+
+
‘smear’
+

Adjacent pixels will spread out to replace missing pixels. +

+
+
‘wrap’
+

Out of range pixels are wrapped so they point to pixels of other side. +

+
+
‘mirror’
+

Out of range pixels will be replaced with mirrored pixels. +

+
+

Default is ‘smear’. +

+
+
+ +

11.69.1 Examples

+ +
    +
  • Add ripple effect to rgb input of video size hd720: +
    +
    ffmpeg -i INPUT -f lavfi -i nullsrc=s=hd720,lutrgb=128:128:128 -f lavfi -i nullsrc=s=hd720,geq='r=128+30*sin(2*PI*X/400+T):g=128+30*sin(2*PI*X/400+T):b=128+30*sin(2*PI*X/400+T)' -lavfi '[0][1][2]displace' OUTPUT
    +
    + +
  • Add wave effect to rgb input of video size hd720: +
    +
    ffmpeg -i INPUT -f lavfi -i nullsrc=hd720,geq='r=128+80*(sin(sqrt((X-W/2)*(X-W/2)+(Y-H/2)*(Y-H/2))/220*2*PI+T)):g=128+80*(sin(sqrt((X-W/2)*(X-W/2)+(Y-H/2)*(Y-H/2))/220*2*PI+T)):b=128+80*(sin(sqrt((X-W/2)*(X-W/2)+(Y-H/2)*(Y-H/2))/220*2*PI+T))' -lavfi '[1]split[x][y],[0][x][y]displace' OUTPUT
    +
    +
+ +

11.70 dnn_classify

+ +

Do classification with deep neural networks based on bounding boxes. +

+

The filter accepts the following options: +

+
+
dnn_backend
+

Specify which DNN backend to use for model loading and execution. This option accepts +only openvino now, tensorflow backends will be added. +

+
+
model
+

Set path to model file specifying network architecture and its parameters. +Note that different backends use different file formats. +

+
+
input
+

Set the input name of the dnn network. +

+
+
output
+

Set the output name of the dnn network. +

+
+
confidence
+

Set the confidence threshold (default: 0.5). +

+
+
labels
+

Set path to label file specifying the mapping between label id and name. +Each label name is written in one line, tailing spaces and empty lines are skipped. +The first line is the name of label id 0, +and the second line is the name of label id 1, etc. +The label id is considered as name if the label file is not provided. +

+
+
backend_configs
+

Set the configs to be passed into backend +

+

For tensorflow backend, you can set its configs with sess_config options, +please use tools/python/tf_sess_config.py to get the configs for your system. +

+
+
+ +

11.71 dnn_detect

+ +

Do object detection with deep neural networks. +

+

The filter accepts the following options: +

+
+
dnn_backend
+

Specify which DNN backend to use for model loading and execution. This option accepts +only openvino now, tensorflow backends will be added. +

+
+
model
+

Set path to model file specifying network architecture and its parameters. +Note that different backends use different file formats. +

+
+
input
+

Set the input name of the dnn network. +

+
+
output
+

Set the output name of the dnn network. +

+
+
confidence
+

Set the confidence threshold (default: 0.5). +

+
+
labels
+

Set path to label file specifying the mapping between label id and name. +Each label name is written in one line, tailing spaces and empty lines are skipped. +The first line is the name of label id 0 (usually it is ’background’), +and the second line is the name of label id 1, etc. +The label id is considered as name if the label file is not provided. +

+
+
backend_configs
+

Set the configs to be passed into backend. To use async execution, set async (default: set). +Roll back to sync execution if the backend does not support async. +

+
+
+ +

11.72 dnn_processing

+ +

Do image processing with deep neural networks. It works together with another filter +which converts the pixel format of the Frame to what the dnn network requires. +

+

The filter accepts the following options: +

+
+
dnn_backend
+

Specify which DNN backend to use for model loading and execution. This option accepts +the following values: +

+
+
‘tensorflow’
+

TensorFlow backend. To enable this backend you +need to install the TensorFlow for C library (see +https://www.tensorflow.org/install/lang_c) and configure FFmpeg with +--enable-libtensorflow +

+
+
‘openvino’
+

OpenVINO backend. To enable this backend you +need to build and install the OpenVINO for C library (see +https://github.com/openvinotoolkit/openvino/blob/master/build-instruction.md) and configure FFmpeg with +--enable-libopenvino (–extra-cflags=-I... –extra-ldflags=-L... might +be needed if the header files and libraries are not installed into system path) +

+
+
‘torch’
+

Libtorch backend. To enable this backend you need to build and install Libtroch +for C++ library. Please download cxx11 ABI version (see +https://pytorch.org/get-started/locally) +and configure FFmpeg with --enable-libtorch +--extra-cflags=-I/libtorch_root/libtorch/include +--extra-cflags=-I/libtorch_root/libtorch/include/torch/csrc/api/include +--extra-ldflags=-L/libtorch_root/libtorch/lib/ +

+
+
+ +
+
model
+

Set path to model file specifying network architecture and its parameters. +Note that different backends use different file formats. TensorFlow, OpenVINO +and Libtorch backend can load files for only its format. +

+
+
input
+

Set the input name of the dnn network. +

+
+
output
+

Set the output name of the dnn network. +

+
+
backend_configs
+

Set the configs to be passed into backend. To use async execution, set async (default: set). +Roll back to sync execution if the backend does not support async. +

+

For tensorflow backend, you can set its configs with sess_config options, +please use tools/python/tf_sess_config.py to get the configs of TensorFlow backend for your system. +

+
+
+ +

11.72.1 Examples

+ +
    +
  • Remove rain in rgb24 frame with can.pb (see derain filter): +
    +
    ./ffmpeg -i rain.jpg -vf format=rgb24,dnn_processing=dnn_backend=tensorflow:model=can.pb:input=x:output=y derain.jpg
    +
    + +
  • Handle the Y channel with srcnn.pb (see sr filter) for frame with yuv420p (planar YUV formats supported): +
    +
    ./ffmpeg -i 480p.jpg -vf format=yuv420p,scale=w=iw*2:h=ih*2,dnn_processing=dnn_backend=tensorflow:model=srcnn.pb:input=x:output=y -y srcnn.jpg
    +
    + +
  • Handle the Y channel with espcn.pb (see sr filter), which changes frame size, for format yuv420p (planar YUV formats supported), +please use tools/python/tf_sess_config.py to get the configs of TensorFlow backend for your system. +
    +
    ./ffmpeg -i 480p.jpg -vf format=yuv420p,dnn_processing=dnn_backend=tensorflow:model=espcn.pb:input=x:output=y:backend_configs=sess_config=0x10022805320e09cdccccccccccec3f20012a01303801 -y tmp.espcn.jpg
    +
    + +
+ +

11.73 drawbox

+ +

Draw a colored box on the input image. +

+

It accepts the following parameters: +

+
+
x
+
y
+

The expressions which specify the top left corner coordinates of the box. It defaults to 0. +

+
+
width, w
+
height, h
+

The expressions which specify the width and height of the box; if 0 they are interpreted as +the input width and height. It defaults to 0. +

+
+
color, c
+

Specify the color of the box to write. For the general syntax of this option, +check the (ffmpeg-utils)"Color" section in the ffmpeg-utils manual. If the special +value invert is used, the box edge color is the same as the +video with inverted luma. +

+
+
thickness, t
+

The expression which sets the thickness of the box edge. +A value of fill will create a filled box. Default value is 3. +

+

See below for the list of accepted constants. +

+
+
replace
+

Applicable if the input has alpha. With value 1, the pixels of the painted box +will overwrite the video’s color and alpha pixels. +Default is 0, which composites the box onto the input, leaving the video’s alpha intact. +

+
+ +

The parameters for x, y, w and h and t are expressions containing the +following constants: +

+
+
dar
+

The input display aspect ratio, it is the same as (w / h) * sar. +

+
+
hsub
+
vsub
+

horizontal and vertical chroma subsample values. For example for the +pixel format "yuv422p" hsub is 2 and vsub is 1. +

+
+
in_h, ih
+
in_w, iw
+

The input width and height. +

+
+
sar
+

The input sample aspect ratio. +

+
+
x
+
y
+

The x and y offset coordinates where the box is drawn. +

+
+
w
+
h
+

The width and height of the drawn box. +

+
+
box_source
+

Box source can be set as side_data_detection_bboxes if you want to use box data in +detection bboxes of side data. +

+

If box_source is set, the x, y, width and height will be ignored and +still use box data in detection bboxes of side data. So please do not use this parameter if you were +not sure about the box source. +

+
+
t
+

The thickness of the drawn box. +

+

These constants allow the x, y, w, h and t expressions to refer to +each other, so you may for example specify y=x/dar or h=w/dar. +

+
+
+ +

11.73.1 Examples

+ +
    +
  • Draw a black box around the edge of the input image: +
    +
    drawbox
    +
    + +
  • Draw a box with color red and an opacity of 50%: +
    +
    drawbox=10:20:200:60:red@0.5
    +
    + +

    The previous example can be specified as: +

    +
    drawbox=x=10:y=20:w=200:h=60:color=red@0.5
    +
    + +
  • Fill the box with pink color: +
    +
    drawbox=x=10:y=10:w=100:h=100:color=pink@0.5:t=fill
    +
    + +
  • Draw a 2-pixel red 2.40:1 mask: +
    +
    drawbox=x=-t:y=0.5*(ih-iw/2.4)-t:w=iw+t*2:h=iw/2.4+t*2:t=2:c=red
    +
    +
+ +

11.73.2 Commands

+

This filter supports same commands as options. +The command accepts the same syntax of the corresponding option. +

+

If the specified expression is not valid, it is kept at its current +value. +

+

11.74 drawgraph

+

Draw a graph using input video metadata. +

+

It accepts the following parameters: +

+
+
m1
+

Set 1st frame metadata key from which metadata values will be used to draw a graph. +

+
+
fg1
+

Set 1st foreground color expression. +

+
+
m2
+

Set 2nd frame metadata key from which metadata values will be used to draw a graph. +

+
+
fg2
+

Set 2nd foreground color expression. +

+
+
m3
+

Set 3rd frame metadata key from which metadata values will be used to draw a graph. +

+
+
fg3
+

Set 3rd foreground color expression. +

+
+
m4
+

Set 4th frame metadata key from which metadata values will be used to draw a graph. +

+
+
fg4
+

Set 4th foreground color expression. +

+
+
min
+

Set minimal value of metadata value. +

+
+
max
+

Set maximal value of metadata value. +

+
+
bg
+

Set graph background color. Default is white. +

+
+
mode
+

Set graph mode. +

+

Available values for mode is: +

+
‘bar’
+
‘dot’
+
‘line’
+
+ +

Default is line. +

+
+
slide
+

Set slide mode. +

+

Available values for slide is: +

+
‘frame’
+

Draw new frame when right border is reached. +

+
+
‘replace’
+

Replace old columns with new ones. +

+
+
‘scroll’
+

Scroll from right to left. +

+
+
‘rscroll’
+

Scroll from left to right. +

+
+
‘picture’
+

Draw single picture. +

+
+ +

Default is frame. +

+
+
size
+

Set size of graph video. For the syntax of this option, check the +(ffmpeg-utils)"Video size" section in the ffmpeg-utils manual. +The default value is 900x256. +

+
+
rate, r
+

Set the output frame rate. Default value is 25. +

+

The foreground color expressions can use the following variables: +

+
MIN
+

Minimal value of metadata value. +

+
+
MAX
+

Maximal value of metadata value. +

+
+
VAL
+

Current metadata key value. +

+
+ +

The color is defined as 0xAABBGGRR. +

+
+ +

Example using metadata from signalstats filter: +

+
signalstats,drawgraph=lavfi.signalstats.YAVG:min=0:max=255
+
+ +

Example using metadata from ebur128 filter: +

+
ebur128=metadata=1,adrawgraph=lavfi.r128.M:min=-120:max=5
+
+ +

11.75 drawgrid

+ +

Draw a grid on the input image. +

+

It accepts the following parameters: +

+
+
x
+
y
+

The expressions which specify the coordinates of some point of grid intersection (meant to configure offset). Both default to 0. +

+
+
width, w
+
height, h
+

The expressions which specify the width and height of the grid cell, if 0 they are interpreted as the +input width and height, respectively, minus thickness, so image gets +framed. Default to 0. +

+
+
color, c
+

Specify the color of the grid. For the general syntax of this option, +check the (ffmpeg-utils)"Color" section in the ffmpeg-utils manual. If the special +value invert is used, the grid color is the same as the +video with inverted luma. +

+
+
thickness, t
+

The expression which sets the thickness of the grid line. Default value is 1. +

+

See below for the list of accepted constants. +

+
+
replace
+

Applicable if the input has alpha. With 1 the pixels of the painted grid +will overwrite the video’s color and alpha pixels. +Default is 0, which composites the grid onto the input, leaving the video’s alpha intact. +

+
+ +

The parameters for x, y, w and h and t are expressions containing the +following constants: +

+
+
dar
+

The input display aspect ratio, it is the same as (w / h) * sar. +

+
+
hsub
+
vsub
+

horizontal and vertical chroma subsample values. For example for the +pixel format "yuv422p" hsub is 2 and vsub is 1. +

+
+
in_h, ih
+
in_w, iw
+

The input grid cell width and height. +

+
+
sar
+

The input sample aspect ratio. +

+
+
x
+
y
+

The x and y coordinates of some point of grid intersection (meant to configure offset). +

+
+
w
+
h
+

The width and height of the drawn cell. +

+
+
t
+

The thickness of the drawn cell. +

+

These constants allow the x, y, w, h and t expressions to refer to +each other, so you may for example specify y=x/dar or h=w/dar. +

+
+
+ +

11.75.1 Examples

+ +
    +
  • Draw a grid with cell 100x100 pixels, thickness 2 pixels, with color red and an opacity of 50%: +
    +
    drawgrid=width=100:height=100:thickness=2:color=red@0.5
    +
    + +
  • Draw a white 3x3 grid with an opacity of 50%: +
    +
    drawgrid=w=iw/3:h=ih/3:t=2:c=white@0.5
    +
    +
+ +

11.75.2 Commands

+

This filter supports same commands as options. +The command accepts the same syntax of the corresponding option. +

+

If the specified expression is not valid, it is kept at its current +value. +

+

11.76 drawtext

+ +

Draw a text string or text from a specified file on top of a video, using the +libfreetype library. +

+

To enable compilation of this filter, you need to configure FFmpeg with +--enable-libfreetype and --enable-libharfbuzz. +To enable default font fallback and the font option you need to +configure FFmpeg with --enable-libfontconfig. +To enable the text_shaping option, you need to configure FFmpeg with +--enable-libfribidi. +

+

11.76.1 Syntax

+ +

It accepts the following parameters: +

+
+
box
+

Used to draw a box around text using the background color. +The value must be either 1 (enable) or 0 (disable). +The default value of box is 0. +

+
+
boxborderw
+

Set the width of the border to be drawn around the box using boxcolor. +The value must be specified using one of the following formats: +

    +
  • boxborderw=10 set the width of all the borders to 10 +
  • boxborderw=10|20 set the width of the top and bottom borders to 10 + and the width of the left and right borders to 20 +
  • boxborderw=10|20|30 set the width of the top border to 10, the width + of the bottom border to 30 and the width of the left and right borders to 20 +
  • boxborderw=10|20|30|40 set the borders width to 10 (top), 20 (right), + 30 (bottom), 40 (left) +
+

The default value of boxborderw is "0". +

+
+
boxcolor
+

The color to be used for drawing box around text. For the syntax of this +option, check the (ffmpeg-utils)"Color" section in the ffmpeg-utils manual. +

+

The default value of boxcolor is "white". +

+
+
line_spacing
+

Set the line spacing in pixels. The default value of line_spacing is 0. +

+
+
text_align
+

Set the vertical and horizontal alignment of the text with respect to the box boundaries. +The value is combination of flags, one for the vertical alignment (T=top, +M=middle, B=bottom) and one for the horizontal alignment (L=left, C=center, R=right). +Please note that tab characters are only supported with the left horizontal alignment. +

+
+
y_align
+

Specify what the y value is referred to. Possible values are: +

    +
  • text the top of the highest glyph of the first text line is placed at y +
  • baseline the baseline of the first text line is placed at y +
  • font the baseline of the first text line is placed at y plus the + ascent (in pixels) defined in the font metrics +
+

The default value of y_align is "text" for backward compatibility. +

+
+
borderw
+

Set the width of the border to be drawn around the text using bordercolor. +The default value of borderw is 0. +

+
+
bordercolor
+

Set the color to be used for drawing border around text. For the syntax of this +option, check the (ffmpeg-utils)"Color" section in the ffmpeg-utils manual. +

+

The default value of bordercolor is "black". +

+
+
expansion
+

Select how the text is expanded. Can be either none, +strftime (deprecated) or normal (default). See the +Text expansion section below for details. +

+
+
basetime
+

Set a start time for the count. Value is in microseconds. Only applied +in the deprecated strftime expansion mode. To emulate in normal expansion +mode use the pts function, supplying the start time (in seconds) +as the second argument. +

+
+
fix_bounds
+

If true, check and fix text coords to avoid clipping. +

+
+
fontcolor
+

The color to be used for drawing fonts. For the syntax of this option, check +the (ffmpeg-utils)"Color" section in the ffmpeg-utils manual. +

+

The default value of fontcolor is "black". +

+
+
fontcolor_expr
+

String which is expanded the same way as text to obtain dynamic +fontcolor value. By default this option has empty value and is not +processed. When this option is set, it overrides fontcolor option. +

+
+
font
+

The font family to be used for drawing text. By default Sans. +

+
+
fontfile
+

The font file to be used for drawing text. The path must be included. +This parameter is mandatory if the fontconfig support is disabled. +

+
+
alpha
+

Draw the text applying alpha blending. The value can +be a number between 0.0 and 1.0. +The expression accepts the same variables x, y as well. +The default value is 1. +Please see fontcolor_expr. +

+
+
fontsize
+

The font size to be used for drawing text. +The default value of fontsize is 16. +

+
+
text_shaping
+

If set to 1, attempt to shape the text (for example, reverse the order of +right-to-left text and join Arabic characters) before drawing it. +Otherwise, just draw the text exactly as given. +By default 1 (if supported). +

+
+
ft_load_flags
+

The flags to be used for loading the fonts. +

+

The flags map the corresponding flags supported by libfreetype, and are +a combination of the following values: +

+
default
+
no_scale
+
no_hinting
+
render
+
no_bitmap
+
vertical_layout
+
force_autohint
+
crop_bitmap
+
pedantic
+
ignore_global_advance_width
+
no_recurse
+
ignore_transform
+
monochrome
+
linear_design
+
no_autohint
+
+ +

Default value is "default". +

+

For more information consult the documentation for the FT_LOAD_* +libfreetype flags. +

+
+
shadowcolor
+

The color to be used for drawing a shadow behind the drawn text. For the +syntax of this option, check the (ffmpeg-utils)"Color" section in the +ffmpeg-utils manual. +

+

The default value of shadowcolor is "black". +

+
+
boxw
+

Set the width of the box to be drawn around text. +The default value of boxw is computed automatically to match the text width +

+
+
boxh
+

Set the height of the box to be drawn around text. +The default value of boxh is computed automatically to match the text height +

+
+
shadowx
+
shadowy
+

The x and y offsets for the text shadow position with respect to the +position of the text. They can be either positive or negative +values. The default value for both is "0". +

+
+
start_number
+

The starting frame number for the n/frame_num variable. The default value +is "0". +

+
+
tabsize
+

The size in number of spaces to use for rendering the tab. +Default value is 4. +

+
+
timecode
+

Set the initial timecode representation in "hh:mm:ss[:;.]ff" +format. It can be used with or without text parameter. timecode_rate +option must be specified. +

+
+
timecode_rate, rate, r
+

Set the timecode frame rate (timecode only). Value will be rounded to nearest +integer. Minimum value is "1". +Drop-frame timecode is supported for frame rates 30 & 60. +

+
+
tc24hmax
+

If set to 1, the output of the timecode option will wrap around at 24 hours. +Default is 0 (disabled). +

+
+
text
+

The text string to be drawn. The text must be a sequence of UTF-8 +encoded characters. +This parameter is mandatory if no file is specified with the parameter +textfile. +

+
+
textfile
+

A text file containing text to be drawn. The text must be a sequence +of UTF-8 encoded characters. +

+

This parameter is mandatory if no text string is specified with the +parameter text. +

+

If both text and textfile are specified, an error is thrown. +

+
+
text_source
+

Text source should be set as side_data_detection_bboxes if you want to use text data in +detection bboxes of side data. +

+

If text source is set, text and textfile will be ignored and still use +text data in detection bboxes of side data. So please do not use this parameter +if you are not sure about the text source. +

+
+
reload
+

The textfile will be reloaded at specified frame interval. +Be sure to update textfile atomically, or it may be read partially, +or even fail. +Range is 0 to INT_MAX. Default is 0. +

+
+
x
+
y
+

The expressions which specify the offsets where text will be drawn +within the video frame. They are relative to the top/left border of the +output image. +

+

The default value of x and y is "0". +

+

See below for the list of accepted constants and functions. +

+
+ +

The parameters for x and y are expressions containing the +following constants and functions: +

+
+
dar
+

input display aspect ratio, it is the same as (w / h) * sar +

+
+
hsub
+
vsub
+

horizontal and vertical chroma subsample values. For example for the +pixel format "yuv422p" hsub is 2 and vsub is 1. +

+
+
line_h, lh
+

the height of each text line +

+
+
main_h, h, H
+

the input height +

+
+
main_w, w, W
+

the input width +

+
+
max_glyph_a, ascent
+

the maximum distance from the baseline to the highest/upper grid +coordinate used to place a glyph outline point, for all the rendered +glyphs. +It is a positive value, due to the grid’s orientation with the Y axis +upwards. +

+
+
max_glyph_d, descent
+

the maximum distance from the baseline to the lowest grid coordinate +used to place a glyph outline point, for all the rendered glyphs. +This is a negative value, due to the grid’s orientation, with the Y axis +upwards. +

+
+
max_glyph_h
+

maximum glyph height, that is the maximum height for all the glyphs +contained in the rendered text, it is equivalent to ascent - +descent. +

+
+
max_glyph_w
+

maximum glyph width, that is the maximum width for all the glyphs +contained in the rendered text +

+
+
font_a
+

the ascent size defined in the font metrics +

+
+
font_d
+

the descent size defined in the font metrics +

+
+
top_a
+

the maximum ascender of the glyphs of the first text line +

+
+
bottom_d
+

the maximum descender of the glyphs of the last text line +

+
+
n
+

the number of input frame, starting from 0 +

+
+
rand(min, max)
+

return a random number included between min and max +

+
+
sar
+

The input sample aspect ratio. +

+
+
t
+

timestamp expressed in seconds, NAN if the input timestamp is unknown +

+
+
text_h, th
+

the height of the rendered text +

+
+
text_w, tw
+

the width of the rendered text +

+
+
x
+
y
+

the x and y offset coordinates where the text is drawn. +

+

These parameters allow the x and y expressions to refer +to each other, so you can for example specify y=x/dar. +

+
+
pict_type
+

A one character description of the current frame’s picture type. +

+
+
pkt_pos
+

The current packet’s position in the input file or stream +(in bytes, from the start of the input). A value of -1 indicates +this info is not available. +

+
+
duration
+

The current packet’s duration, in seconds. +

+
+
pkt_size
+

The current packet’s size (in bytes). +

+
+ +

11.76.2 Text expansion

+ +

If expansion is set to strftime, the filter recognizes +sequences accepted by the strftime C function in the provided +text and expands them accordingly. Check the documentation of +strftime. This feature is deprecated in favor of normal +expansion with the gmtime or localtime expansion +functions. +

+

If expansion is set to none, the text is printed verbatim. +

+

If expansion is set to normal (which is the default), +the following expansion mechanism is used. +

+

The backslash character ‘\’, followed by any character, always expands to +the second character. +

+

Sequences of the form %{...} are expanded. The text between the +braces is a function name, possibly followed by arguments separated by ’:’. +If the arguments contain special characters or delimiters (’:’ or ’}’), +they should be escaped. +

+

Note that they probably must also be escaped as the value for the text +option in the filter argument string and as the filter argument in the +filtergraph description, and possibly also for the shell, that makes up to four +levels of escaping; using a text file with the textfile option avoids +these problems. +

+

The following functions are available: +

+
+
expr, e
+

The expression evaluation result. +

+

It must take one argument specifying the expression to be evaluated, +which accepts the same constants and functions as the x and +y values. Note that not all constants should be used, for +example the text size is not known when evaluating the expression, so +the constants text_w and text_h will have an undefined +value. +

+
+
expr_int_format, eif
+

Evaluate the expression’s value and output as formatted integer. +

+

The first argument is the expression to be evaluated, just as for the expr function. +The second argument specifies the output format. Allowed values are ‘x’, +‘X’, ‘d’ and ‘u’. They are treated exactly as in the +printf function. +The third parameter is optional and sets the number of positions taken by the output. +It can be used to add padding with zeros from the left. +

+
+
gmtime
+

The time at which the filter is running, expressed in UTC. +It can accept an argument: a strftime C function format string. +The format string is extended to support the variable %[1-6]N +which prints fractions of the second with optionally specified number of digits. +

+
+
localtime
+

The time at which the filter is running, expressed in the local time zone. +It can accept an argument: a strftime C function format string. +The format string is extended to support the variable %[1-6]N +which prints fractions of the second with optionally specified number of digits. +

+
+
metadata
+

Frame metadata. Takes one or two arguments. +

+

The first argument is mandatory and specifies the metadata key. +

+

The second argument is optional and specifies a default value, used when the +metadata key is not found or empty. +

+

Available metadata can be identified by inspecting entries +starting with TAG included within each frame section +printed by running ffprobe -show_frames. +

+

String metadata generated in filters leading to +the drawtext filter are also available. +

+
+
n, frame_num
+

The frame number, starting from 0. +

+
+
pict_type
+

A one character description of the current picture type. +

+
+
pts
+

The timestamp of the current frame. +It can take up to three arguments. +

+

The first argument is the format of the timestamp; it defaults to flt +for seconds as a decimal number with microsecond accuracy; hms stands +for a formatted [-]HH:MM:SS.mmm timestamp with millisecond accuracy. +gmtime stands for the timestamp of the frame formatted as UTC time; +localtime stands for the timestamp of the frame formatted as +local time zone time. +

+

The second argument is an offset added to the timestamp. +

+

If the format is set to hms, a third argument 24HH may be +supplied to present the hour part of the formatted timestamp in 24h format +(00-23). +

+

If the format is set to localtime or gmtime, a third +argument may be supplied: a strftime C function format string. +By default, YYYY-MM-DD HH:MM:SS format will be used. +

+
+ +

11.76.3 Commands

+ +

This filter supports altering parameters via commands: +

+
reinit
+

Alter existing filter parameters. +

+

Syntax for the argument is the same as for filter invocation, e.g. +

+
+
fontsize=56:fontcolor=green:text='Hello World'
+
+ +

Full filter invocation with sendcmd would look like this: +

+
+
sendcmd=c='56.0 drawtext reinit fontsize=56\:fontcolor=green\:text=Hello\\ World'
+
+ +

If the entire argument can’t be parsed or applied as valid values then the filter will +continue with its existing parameters. +

+
+
+ +

The following options are also supported as commands: +

+
    +
  • x +
  • y +
  • alpha +
  • fontsize +
  • fontcolor +
  • boxcolor +
  • bordercolor +
  • shadowcolor +
  • box +
  • boxw +
  • boxh +
  • boxborderw +
  • line_spacing +
  • text_align +
  • shadowx +
  • shadowy +
  • borderw +
+ +

11.76.4 Examples

+ +
    +
  • Draw "Test Text" with font FreeSerif, using the default values for the +optional parameters. + +
    +
    drawtext="fontfile=/usr/share/fonts/truetype/freefont/FreeSerif.ttf: text='Test Text'"
    +
    + +
  • Draw ’Test Text’ with font FreeSerif of size 24 at position x=100 +and y=50 (counting from the top-left corner of the screen), text is +yellow with a red box around it. Both the text and the box have an +opacity of 20%. + +
    +
    drawtext="fontfile=/usr/share/fonts/truetype/freefont/FreeSerif.ttf: text='Test Text':\
    +          x=100: y=50: fontsize=24: fontcolor=yellow@0.2: box=1: boxcolor=red@0.2"
    +
    + +

    Note that the double quotes are not necessary if spaces are not used +within the parameter list. +

    +
  • Show the text at the center of the video frame: +
    +
    drawtext="fontsize=30:fontfile=FreeSerif.ttf:text='hello world':x=(w-text_w)/2:y=(h-text_h)/2"
    +
    + +
  • Show the text at a random position, switching to a new position every 30 seconds: +
    +
    drawtext="fontsize=30:fontfile=FreeSerif.ttf:text='hello world':x=if(eq(mod(t\,30)\,0)\,rand(0\,(w-text_w))\,x):y=if(eq(mod(t\,30)\,0)\,rand(0\,(h-text_h))\,y)"
    +
    + +
  • Show a text line sliding from right to left in the last row of the video +frame. The file LONG_LINE is assumed to contain a single line +with no newlines. +
    +
    drawtext="fontsize=15:fontfile=FreeSerif.ttf:text=LONG_LINE:y=h-line_h:x=-50*t"
    +
    + +
  • Show the content of file CREDITS off the bottom of the frame and scroll up. +
    +
    drawtext="fontsize=20:fontfile=FreeSerif.ttf:textfile=CREDITS:y=h-20*t"
    +
    + +
  • Draw a single green letter "g", at the center of the input video. +The glyph baseline is placed at half screen height. +
    +
    drawtext="fontsize=60:fontfile=FreeSerif.ttf:fontcolor=green:text=g:x=(w-max_glyph_w)/2:y=h/2-ascent"
    +
    + +
  • Show text for 1 second every 3 seconds: +
    +
    drawtext="fontfile=FreeSerif.ttf:fontcolor=white:x=100:y=x/dar:enable=lt(mod(t\,3)\,1):text='blink'"
    +
    + +
  • Use fontconfig to set the font. Note that the colons need to be escaped. +
    +
    drawtext='fontfile=Linux Libertine O-40\\:style=Semibold:text=FFmpeg'
    +
    + +
  • Draw "Test Text" with font size dependent on height of the video. +
    +
    drawtext="text='Test Text': fontsize=h/30: x=(w-text_w)/2: y=(h-text_h*2)"
    +
    + +
  • Print the date of a real-time encoding (see documentation for the +strftime C function): +
    +
    drawtext='fontfile=FreeSans.ttf:text=%{localtime\:%a %b %d %Y}'
    +
    + +
  • Show text fading in and out (appearing/disappearing): +
    +
    #!/bin/sh
    +DS=1.0 # display start
    +DE=10.0 # display end
    +FID=1.5 # fade in duration
    +FOD=5 # fade out duration
    +ffplay -f lavfi "color,drawtext=text=TEST:fontsize=50:fontfile=FreeSerif.ttf:fontcolor_expr=ff0000%{eif\\\\: clip(255*(1*between(t\\, $DS + $FID\\, $DE - $FOD) + ((t - $DS)/$FID)*between(t\\, $DS\\, $DS + $FID) + (-(t - $DE)/$FOD)*between(t\\, $DE - $FOD\\, $DE) )\\, 0\\, 255) \\\\: x\\\\: 2 }"
    +
    + +
  • Horizontally align multiple separate texts. Note that max_glyph_a +and the fontsize value are included in the y offset. +
    +
    drawtext=fontfile=FreeSans.ttf:text=DOG:fontsize=24:x=10:y=20+24-max_glyph_a,
    +drawtext=fontfile=FreeSans.ttf:text=cow:fontsize=24:x=80:y=20+24-max_glyph_a
    +
    + +
  • Plot special lavf.image2dec.source_basename metadata onto each frame if +such metadata exists. Otherwise, plot the string "NA". Note that image2 demuxer +must have option -export_path_metadata 1 for the special metadata fields +to be available for filters. +
    +
    drawtext="fontsize=20:fontcolor=white:fontfile=FreeSans.ttf:text='%{metadata\:lavf.image2dec.source_basename\:NA}':x=10:y=10"
    +
    + +
+ +

For more information about libfreetype, check: +http://www.freetype.org/. +

+

For more information about fontconfig, check: +http://freedesktop.org/software/fontconfig/fontconfig-user.html. +

+

For more information about libfribidi, check: +http://fribidi.org/. +

+

For more information about libharfbuzz, check: +https://github.com/harfbuzz/harfbuzz. +

+

11.77 drawvg

+ +

Draw vector graphics on top of video frames, by executing a script written in +a custom language called VGS (Vector Graphics Script). +

+

The documentation for the language can be found in +(drawvg-reference)drawvg - Language Reference. A version of this reference +with rendered examples can be found at the +author’s site. +

+

Graphics are rendered using the cario 2D +graphics library. +

+

To enable compilation of this filter, you need to configure FFmpeg with +--enable-cairo. +

+

11.77.1 Parameters

+ +

Either script or file must be set. +

+
+
s, script
+

Script source to draw the graphics. +

+
+
file
+

Path of the file to load the script source. +

+
+
+ +

11.77.2 Pixel Formats

+ +

Since Cairo only supports RGB images, if the input video is something else (like +YUV 4:2:0), before executing the script the video is converted to a format +compatible with Cairo. Then, you have to use either the format filter, +or the -pix_fmt option, to convert it to the expected format in the +output. +

+

11.77.3 Examples

+ +
    +
  • Draw the outline of an ellipse. + +
    +
    ffmpeg -i input.webm \
    +    -vf 'drawvg=ellipse (w/2) (h/2) (w/3) (h/3) stroke' \
    +    -pix_fmt yuv420p \
    +    output.webm
    +
    + +
  • +Draw a square rotating in the middle of the frame. + +

    The script for drawvg is in a file draw.vgs: +

    +
    +
    translate (w/2) (h/2)
    +rotate t
    +rect -100 -100 200 200
    +setcolor red@0.5
    +fill
    +
    + +

    Then: +

    +
    +
    ffmpeg -i input.webm -vf 'drawvg=file=draw.vgs,format=yuv420p' output.webm
    +
    + +
+ +

11.78 edgedetect

+ +

Detect and draw edges. The filter uses the Canny Edge Detection algorithm. +

+

The filter accepts the following options: +

+
+
low
+
high
+

Set low and high threshold values used by the Canny thresholding +algorithm. +

+

The high threshold selects the "strong" edge pixels, which are then +connected through 8-connectivity with the "weak" edge pixels selected +by the low threshold. +

+

low and high threshold values must be chosen in the range +[0,1], and low should be lesser or equal to high. +

+

Default value for low is 20/255, and default value for high +is 50/255. +

+
+
mode
+

Define the drawing mode. +

+
+
‘wires’
+

Draw white/gray wires on black background. +

+
+
‘colormix’
+

Mix the colors to create a paint/cartoon effect. +

+
+
‘canny’
+

Apply Canny edge detector on all selected planes. +

+
+

Default value is wires. +

+
+
planes
+

Select planes for filtering. By default all available planes are filtered. +

+
+ +

11.78.1 Examples

+ +
    +
  • Standard edge detection with custom values for the hysteresis thresholding: +
    +
    edgedetect=low=0.1:high=0.4
    +
    + +
  • Painting effect without thresholding: +
    +
    edgedetect=mode=colormix:high=0
    +
    +
+ +

11.79 elbg

+ +

Apply a posterize effect using the ELBG (Enhanced LBG) algorithm. +

+

For each input image, the filter will compute the optimal mapping from +the input to the output given the codebook length, that is the number +of distinct output colors. +

+

This filter accepts the following options. +

+
+
codebook_length, l
+

Set codebook length. The value must be a positive integer, and +represents the number of distinct output colors. Default value is 256. +

+
+
nb_steps, n
+

Set the maximum number of iterations to apply for computing the optimal +mapping. The higher the value the better the result and the higher the +computation time. Default value is 1. +

+
+
seed, s
+

Set a random seed, must be an integer included between 0 and +UINT32_MAX. If not specified, or if explicitly set to -1, the filter +will try to use a good random seed on a best effort basis. +

+
+
pal8
+

Set pal8 output pixel format. This option does not work with codebook +length greater than 256. Default is disabled. +

+
+
use_alpha
+

Include alpha values in the quantization calculation. Allows creating +palettized output images (e.g. PNG8) with multiple alpha smooth blending. +

+
+ +

11.80 entropy

+ +

Measure graylevel entropy in histogram of color channels of video frames. +

+

It accepts the following parameters: +

+
+
mode
+

Can be either normal or diff. Default is normal. +

+

diff mode measures entropy of histogram delta values, absolute differences +between neighbour histogram values. +

+
+ +

11.81 epx

+

Apply the EPX magnification filter which is designed for pixel art. +

+

It accepts the following option: +

+
+
n
+

Set the scaling dimension: 2 for 2xEPX, 3 for +3xEPX. +Default is 3. +

+
+ +

11.82 eq

+

Set brightness, contrast, saturation and approximate gamma adjustment. +

+

The filter accepts the following options: +

+
+
contrast
+

Set the contrast expression. The value must be a float value in range +-1000.0 to 1000.0. The default value is "1". +

+
+
brightness
+

Set the brightness expression. The value must be a float value in +range -1.0 to 1.0. The default value is "0". +

+
+
saturation
+

Set the saturation expression. The value must be a float in +range 0.0 to 3.0. The default value is "1". +

+
+
gamma
+

Set the gamma expression. The value must be a float in range +0.1 to 10.0. The default value is "1". +

+
+
gamma_r
+

Set the gamma expression for red. The value must be a float in +range 0.1 to 10.0. The default value is "1". +

+
+
gamma_g
+

Set the gamma expression for green. The value must be a float in range +0.1 to 10.0. The default value is "1". +

+
+
gamma_b
+

Set the gamma expression for blue. The value must be a float in range +0.1 to 10.0. The default value is "1". +

+
+
gamma_weight
+

Set the gamma weight expression. It can be used to reduce the effect +of a high gamma value on bright image areas, e.g. keep them from +getting overamplified and just plain white. The value must be a float +in range 0.0 to 1.0. A value of 0.0 turns the +gamma correction all the way down while 1.0 leaves it at its +full strength. Default is "1". +

+
+
eval
+

Set when the expressions for brightness, contrast, saturation and +gamma expressions are evaluated. +

+

It accepts the following values: +

+
‘init’
+

only evaluate expressions once during the filter initialization or +when a command is processed +

+
+
‘frame’
+

evaluate expressions for each incoming frame +

+
+ +

Default value is ‘init’. +

+
+ +

The expressions accept the following parameters: +

+
n
+

frame count of the input frame starting from 0 +

+
+
pos
+

byte position of the corresponding packet in the input file, NAN if +unspecified; deprecated, do not use +

+
+
r
+

frame rate of the input video, NAN if the input frame rate is unknown +

+
+
t
+

timestamp expressed in seconds, NAN if the input timestamp is unknown +

+
+ +

11.82.1 Commands

+

The filter supports the following commands: +

+
+
contrast
+

Set the contrast expression. +

+
+
brightness
+

Set the brightness expression. +

+
+
saturation
+

Set the saturation expression. +

+
+
gamma
+

Set the gamma expression. +

+
+
gamma_r
+

Set the gamma_r expression. +

+
+
gamma_g
+

Set gamma_g expression. +

+
+
gamma_b
+

Set gamma_b expression. +

+
+
gamma_weight
+

Set gamma_weight expression. +

+

The command accepts the same syntax of the corresponding option. +

+

If the specified expression is not valid, it is kept at its current +value. +

+
+
+ +

11.83 erosion

+ +

Apply erosion effect to the video. +

+

This filter replaces the pixel by the local(3x3) minimum. +

+

It accepts the following options: +

+
+
threshold0
+
threshold1
+
threshold2
+
threshold3
+

Limit the maximum change for each plane, default is 65535. +If 0, plane will remain unchanged. +

+
+
coordinates
+

Flag which specifies the pixel to refer to. Default is 255 i.e. all eight +pixels are used. +

+

Flags to local 3x3 coordinates maps like this: +

+

1 2 3 + 4 5 + 6 7 8 +

+
+ +

11.83.1 Commands

+ +

This filter supports the all above options as commands. +

+

11.84 estdif

+ +

Deinterlace the input video ("estdif" stands for "Edge Slope +Tracing Deinterlacing Filter"). +

+

Spatial only filter that uses edge slope tracing algorithm +to interpolate missing lines. +It accepts the following parameters: +

+
+
mode
+

The interlacing mode to adopt. It accepts one of the following values: +

+
+
frame
+

Output one frame for each frame. +

+
field
+

Output one frame for each field. +

+
+ +

The default value is field. +

+
+
parity
+

The picture field parity assumed for the input interlaced video. It accepts one +of the following values: +

+
+
tff
+

Assume the top field is first. +

+
bff
+

Assume the bottom field is first. +

+
auto
+

Enable automatic detection of field parity. +

+
+ +

The default value is auto. +If the interlacing is unknown or the decoder does not export this information, +top field first will be assumed. +

+
+
deint
+

Specify which frames to deinterlace. Accepts one of the following +values: +

+
+
all
+

Deinterlace all frames. +

+
interlaced
+

Only deinterlace frames marked as interlaced. +

+
+ +

The default value is all. +

+
+
rslope
+

Specify the search radius for edge slope tracing. Default value is 1. +Allowed range is from 1 to 15. +

+
+
redge
+

Specify the search radius for best edge matching. Default value is 2. +Allowed range is from 0 to 15. +

+
+
ecost
+

Specify the edge cost for edge matching. Default value is 2. +Allowed range is from 0 to 50. +

+
+
mcost
+

Specify the middle cost for edge matching. Default value is 1. +Allowed range is from 0 to 50. +

+
+
dcost
+

Specify the distance cost for edge matching. Default value is 1. +Allowed range is from 0 to 50. +

+
+
interp
+

Specify the interpolation used. Default is 4-point interpolation. It accepts one +of the following values: +

+
+
2p
+

Two-point interpolation. +

+
4p
+

Four-point interpolation. +

+
6p
+

Six-point interpolation. +

+
+
+
+ +

11.84.1 Commands

+

This filter supports same commands as options. +

+

11.85 exposure

+

Adjust exposure of the video stream. +

+

The filter accepts the following options: +

+
+
exposure
+

Set the exposure correction in EV. Allowed range is from -3.0 to 3.0 EV +Default value is 0 EV. +

+
+
black
+

Set the black level correction. Allowed range is from -1.0 to 1.0. +Default value is 0. +

+
+ +

11.85.1 Commands

+ +

This filter supports same commands as options. +

+

11.86 extractplanes

+ +

Extract color channel components from input video stream into +separate grayscale video streams. +

+

The filter accepts the following option: +

+
+
planes
+

Set plane(s) to extract. +

+

Available values for planes are: +

+
‘y’
+
‘u’
+
‘v’
+
‘a’
+
‘r’
+
‘g’
+
‘b’
+
+ +

Choosing planes not available in the input will result in an error. +That means you cannot select r, g, b planes +with y, u, v planes at same time. +

+
+ +

11.86.1 Examples

+ +
    +
  • Extract luma, u and v color channel component from input video frame +into 3 grayscale outputs: +
    +
    ffmpeg -i video.avi -filter_complex 'extractplanes=y+u+v[y][u][v]' -map '[y]' y.avi -map '[u]' u.avi -map '[v]' v.avi
    +
    +
+ +

11.87 fade

+ +

Apply a fade-in/out effect to the input video. +

+

It accepts the following parameters: +

+
+
type, t
+

The effect type can be either "in" for a fade-in, or "out" for a fade-out +effect. +Default is in. +

+
+
start_frame, s
+

Specify the number of the frame to start applying the fade +effect at. Default is 0. +

+
+
nb_frames, n
+

The number of frames that the fade effect lasts. At the end of the +fade-in effect, the output video will have the same intensity as the input video. +At the end of the fade-out transition, the output video will be filled with the +selected color. +Default is 25. +

+
+
alpha
+

If set to 1, fade only alpha channel, if one exists on the input. +Default value is 0. +

+
+
start_time, st
+

Specify the timestamp (in seconds) of the frame to start to apply the fade +effect. If both start_frame and start_time are specified, the fade will start at +whichever comes last. Default is 0. +

+
+
duration, d
+

The number of seconds for which the fade effect has to last. At the end of the +fade-in effect the output video will have the same intensity as the input video, +at the end of the fade-out transition the output video will be filled with the +selected color. +If both duration and nb_frames are specified, duration is used. Default is 0 +(nb_frames is used by default). +

+
+
color, c
+

Specify the color of the fade. Default is "black". +

+
+ +

11.87.1 Examples

+ +
    +
  • Fade in the first 30 frames of video: +
    +
    fade=in:0:30
    +
    + +

    The command above is equivalent to: +

    +
    fade=t=in:s=0:n=30
    +
    + +
  • Fade out the last 45 frames of a 200-frame video: +
    +
    fade=out:155:45
    +fade=type=out:start_frame=155:nb_frames=45
    +
    + +
  • Fade in the first 25 frames and fade out the last 25 frames of a 1000-frame video: +
    +
    fade=in:0:25, fade=out:975:25
    +
    + +
  • Make the first 5 frames yellow, then fade in from frame 5-24: +
    +
    fade=in:5:20:color=yellow
    +
    + +
  • Fade in alpha over first 25 frames of video: +
    +
    fade=in:0:25:alpha=1
    +
    + +
  • Make the first 5.5 seconds black, then fade in for 0.5 seconds: +
    +
    fade=t=in:st=5.5:d=0.5
    +
    + +
+ +

11.88 feedback

+

Apply feedback video filter. +

+

This filter pass cropped input frames to 2nd output. +From there it can be filtered with other video filters. +After filter receives frame from 2nd input, that frame +is combined on top of original frame from 1st input and passed +to 1st output. +

+

The typical usage is filter only part of frame. +

+

The filter accepts the following options: +

+
x
+
y
+

Set the top left crop position. +

+
+
w
+
h
+

Set the crop size. +

+
+ +

11.88.1 Examples

+ +
    +
  • Blur only top left rectangular part of video frame size 100x100 with gblur filter. +
    +
    [in][blurin]feedback=x=0:y=0:w=100:h=100[out][blurout];[blurout]gblur=8[blurin]
    +
    + +
  • Draw black box on top left part of video frame of size 100x100 with drawbox filter. +
    +
    [in][blurin]feedback=x=0:y=0:w=100:h=100[out][blurout];[blurout]drawbox=x=0:y=0:w=100:h=100:t=100[blurin]
    +
    + +
  • Pixelize rectangular part of video frame of size 100x100 with pixelize filter. +
    +
    [in][blurin]feedback=x=320:y=240:w=100:h=100[out][blurout];[blurout]pixelize[blurin]
    +
    +
+ +

11.89 fftdnoiz

+

Denoise frames using 3D FFT (frequency domain filtering). +

+

The filter accepts the following options: +

+
+
sigma
+

Set the noise sigma constant. This sets denoising strength. +Default value is 1. Allowed range is from 0 to 30. +Using very high sigma with low overlap may give blocking artifacts. +

+
+
amount
+

Set amount of denoising. By default all detected noise is reduced. +Default value is 1. Allowed range is from 0 to 1. +

+
+
block
+

Set size of block in pixels, Default is 32, can be 8 to 256. +

+
+
overlap
+

Set block overlap. Default is 0.5. Allowed range is from 0.2 to 0.8. +

+
+
method
+

Set denoising method. Default is wiener, can also be hard. +

+
+
prev
+

Set number of previous frames to use for denoising. By default is set to 0. +

+
+
next
+

Set number of next frames to to use for denoising. By default is set to 0. +

+
+
planes
+

Set planes which will be filtered, by default are all available filtered +except alpha. +

+
+ +

11.90 fftfilt

+

Apply arbitrary expressions to samples in frequency domain +

+
+
dc_Y
+

Adjust the dc value (gain) of the luma plane of the image. The filter +accepts an integer value in range 0 to 1000. The default +value is set to 0. +

+
+
dc_U
+

Adjust the dc value (gain) of the 1st chroma plane of the image. The +filter accepts an integer value in range 0 to 1000. The +default value is set to 0. +

+
+
dc_V
+

Adjust the dc value (gain) of the 2nd chroma plane of the image. The +filter accepts an integer value in range 0 to 1000. The +default value is set to 0. +

+
+
weight_Y
+

Set the frequency domain weight expression for the luma plane. +

+
+
weight_U
+

Set the frequency domain weight expression for the 1st chroma plane. +

+
+
weight_V
+

Set the frequency domain weight expression for the 2nd chroma plane. +

+
+
eval
+

Set when the expressions are evaluated. +

+

It accepts the following values: +

+
‘init’
+

Only evaluate expressions once during the filter initialization. +

+
+
‘frame’
+

Evaluate expressions for each incoming frame. +

+
+ +

Default value is ‘init’. +

+

The filter accepts the following variables: +

+
X
+
Y
+

The coordinates of the current sample. +

+
+
W
+
H
+

The width and height of the image. +

+
+
N
+

The number of input frame, starting from 0. +

+
+
WS
+
HS
+

The size of FFT array for horizontal and vertical processing. +

+
+ +

11.90.1 Examples

+ +
    +
  • High-pass: +
    +
    fftfilt=dc_Y=128:weight_Y='squish(1-(Y+X)/100)'
    +
    + +
  • Low-pass: +
    +
    fftfilt=dc_Y=0:weight_Y='squish((Y+X)/100-1)'
    +
    + +
  • Sharpen: +
    +
    fftfilt=dc_Y=0:weight_Y='1+squish(1-(Y+X)/100)'
    +
    + +
  • Blur: +
    +
    fftfilt=dc_Y=0:weight_Y='exp(-4 * ((Y+X)/(W+H)))'
    +
    + +
+ +

11.91 field

+ +

Extract a single field from an interlaced image using stride +arithmetic to avoid wasting CPU time. The output frames are marked as +non-interlaced. +

+

The filter accepts the following options: +

+
+
type
+

Specify whether to extract the top (if the value is 0 or +top) or the bottom field (if the value is 1 or +bottom). +

+
+ +

11.92 fieldhint

+ +

Create new frames by copying the top and bottom fields from surrounding frames +supplied as numbers by the hint file. +

+
+
hint
+

Set file containing hints: absolute/relative frame numbers. +

+

There must be one line for each frame in a clip. Each line must contain two +numbers separated by the comma, optionally followed by - or +. +Numbers supplied on each line of file can not be out of [N-1,N+1] where N +is current frame number for absolute mode or out of [-1, 1] range +for relative mode. First number tells from which frame to pick up top +field and second number tells from which frame to pick up bottom field. +

+

If optionally followed by + output frame will be marked as interlaced, +else if followed by - output frame will be marked as progressive, else +it will be marked same as input frame. +If optionally followed by t output frame will use only top field, or in +case of b it will use only bottom field. +If line starts with # or ; that line is skipped. +

+
+
mode
+

Can be item absolute or relative or pattern. Default is absolute. +The pattern mode is same as relative mode, except at last entry of file if there +are more frames to process than hint file is seek back to start. +

+
+ +

Example of first several lines of hint file for relative mode: +

+
0,0 - # first frame
+1,0 - # second frame, use third's frame top field and second's frame bottom field
+1,0 - # third frame, use fourth's frame top field and third's frame bottom field
+1,0 -
+0,0 -
+0,0 -
+1,0 -
+1,0 -
+1,0 -
+0,0 -
+0,0 -
+1,0 -
+1,0 -
+1,0 -
+0,0 -
+
+ +

11.93 fieldmatch

+ +

Field matching filter for inverse telecine. It is meant to reconstruct the +progressive frames from a telecined stream. The filter does not drop duplicated +frames, so to achieve a complete inverse telecine fieldmatch needs to be +followed by a decimation filter such as decimate in the filtergraph. +

+

The separation of the field matching and the decimation is notably motivated by +the possibility of inserting a de-interlacing filter fallback between the two. +If the source has mixed telecined and real interlaced content, +fieldmatch will not be able to match fields for the interlaced parts. +But these remaining combed frames will be marked as interlaced, and thus can be +de-interlaced by a later filter such as yadif before decimation. +

+

In addition to the various configuration options, fieldmatch can take an +optional second stream, activated through the ppsrc option. If +enabled, the frames reconstruction will be based on the fields and frames from +this second stream. This allows the first input to be pre-processed in order to +help the various algorithms of the filter, while keeping the output lossless +(assuming the fields are matched properly). Typically, a field-aware denoiser, +or brightness/contrast adjustments can help. +

+

Note that this filter uses the same algorithms as TIVTC/TFM (AviSynth project) +and VIVTC/VFM (VapourSynth project). The later is a light clone of TFM from +which fieldmatch is based on. While the semantic and usage are very +close, some behaviour and options names can differ. +

+

The decimate filter currently only works for constant frame rate input. +If your input has mixed telecined (30fps) and progressive content with a lower +framerate like 24fps use the following filterchain to produce the necessary cfr +stream: dejudder,fps=30000/1001,fieldmatch,decimate. +

+

The filter accepts the following options: +

+
+
order
+

Specify the assumed field order of the input stream. Available values are: +

+
+
‘auto’
+

Auto detect parity (use FFmpeg’s internal parity value). +

+
‘bff’
+

Assume bottom field first. +

+
‘tff’
+

Assume top field first. +

+
+ +

Note that it is sometimes recommended not to trust the parity announced by the +stream. +

+

Default value is auto. +

+
+
mode
+

Set the matching mode or strategy to use. pc mode is the safest in the +sense that it won’t risk creating jerkiness due to duplicate frames when +possible, but if there are bad edits or blended fields it will end up +outputting combed frames when a good match might actually exist. On the other +hand, pcn_ub mode is the most risky in terms of creating jerkiness, +but will almost always find a good frame if there is one. The other values are +all somewhere in between pc and pcn_ub in terms of risking +jerkiness and creating duplicate frames versus finding good matches in sections +with bad edits, orphaned fields, blended fields, etc. +

+

More details about p/c/n/u/b are available in p/c/n/u/b meaning section. +

+

Available values are: +

+
+
‘pc’
+

2-way matching (p/c) +

+
‘pc_n’
+

2-way matching, and trying 3rd match if still combed (p/c + n) +

+
‘pc_u’
+

2-way matching, and trying 3rd match (same order) if still combed (p/c + u) +

+
‘pc_n_ub’
+

2-way matching, trying 3rd match if still combed, and trying 4th/5th matches if +still combed (p/c + n + u/b) +

+
‘pcn’
+

3-way matching (p/c/n) +

+
‘pcn_ub’
+

3-way matching, and trying 4th/5th matches if all 3 of the original matches are +detected as combed (p/c/n + u/b) +

+
+ +

The parenthesis at the end indicate the matches that would be used for that +mode assuming order=tff (and field on auto or +top). +

+

In terms of speed pc mode is by far the fastest and pcn_ub is +the slowest. +

+

Default value is pc_n. +

+
+
ppsrc
+

Mark the main input stream as a pre-processed input, and enable the secondary +input stream as the clean source to pick the fields from. See the filter +introduction for more details. It is similar to the clip2 feature from +VFM/TFM. +

+

Default value is 0 (disabled). +

+
+
field
+

Set the field to match from. It is recommended to set this to the same value as +order unless you experience matching failures with that setting. In +certain circumstances changing the field that is used to match from can have a +large impact on matching performance. Available values are: +

+
+
‘auto’
+

Automatic (same value as order). +

+
‘bottom’
+

Match from the bottom field. +

+
‘top’
+

Match from the top field. +

+
+ +

Default value is auto. +

+
+
mchroma
+

Set whether or not chroma is included during the match comparisons. In most +cases it is recommended to leave this enabled. You should set this to 0 +only if your clip has bad chroma problems such as heavy rainbowing or other +artifacts. Setting this to 0 could also be used to speed things up at +the cost of some accuracy. +

+

Default value is 1. +

+
+
y0
+
y1
+

These define an exclusion band which excludes the lines between y0 and +y1 from being included in the field matching decision. An exclusion +band can be used to ignore subtitles, a logo, or other things that may +interfere with the matching. y0 sets the starting scan line and +y1 sets the ending line; all lines in between y0 and +y1 (including y0 and y1) will be ignored. Setting +y0 and y1 to the same value will disable the feature. +y0 and y1 defaults to 0. +

+
+
scthresh
+

Set the scene change detection threshold as a percentage of maximum change on +the luma plane. Good values are in the [8.0, 14.0] range. Scene change +detection is only relevant in case combmatch=sc. The range for +scthresh is [0.0, 100.0]. +

+

Default value is 12.0. +

+
+
combmatch
+

When combatch is not none, fieldmatch will take into +account the combed scores of matches when deciding what match to use as the +final match. Available values are: +

+
+
‘none’
+

No final matching based on combed scores. +

+
‘sc’
+

Combed scores are only used when a scene change is detected. +

+
‘full’
+

Use combed scores all the time. +

+
+ +

Default is sc. +

+
+
combdbg
+

Force fieldmatch to calculate the combed metrics for certain matches and +print them. This setting is known as micout in TFM/VFM vocabulary. +Available values are: +

+
+
‘none’
+

No forced calculation. +

+
‘pcn’
+

Force p/c/n calculations. +

+
‘pcnub’
+

Force p/c/n/u/b calculations. +

+
+ +

Default value is none. +

+
+
cthresh
+

This is the area combing threshold used for combed frame detection. This +essentially controls how "strong" or "visible" combing must be to be detected. +Larger values mean combing must be more visible and smaller values mean combing +can be less visible or strong and still be detected. Valid settings are from +-1 (every pixel will be detected as combed) to 255 (no pixel will +be detected as combed). This is basically a pixel difference value. A good +range is [8, 12]. +

+

Default value is 9. +

+
+
chroma
+

Sets whether or not chroma is considered in the combed frame decision. Only +disable this if your source has chroma problems (rainbowing, etc.) that are +causing problems for the combed frame detection with chroma enabled. Actually, +using chroma=0 is usually more reliable, except for the case +where there is chroma only combing in the source. +

+

Default value is 0. +

+
+
blockx
+
blocky
+

Respectively set the x-axis and y-axis size of the window used during combed +frame detection. This has to do with the size of the area in which +combpel pixels are required to be detected as combed for a frame to be +declared combed. See the combpel parameter description for more info. +Possible values are any number that is a power of 2 starting at 4 and going up +to 512. +

+

Default value is 16. +

+
+
combpel
+

The number of combed pixels inside any of the blocky by +blockx size blocks on the frame for the frame to be detected as +combed. While cthresh controls how "visible" the combing must be, this +setting controls "how much" combing there must be in any localized area (a +window defined by the blockx and blocky settings) on the +frame. Minimum value is 0 and maximum is blocky x blockx (at +which point no frames will ever be detected as combed). This setting is known +as MI in TFM/VFM vocabulary. +

+

Default value is 80. +

+
+ +

11.93.1 p/c/n/u/b meaning

+ +

11.93.1.1 p/c/n

+ +

We assume the following telecined stream: +

+
+
Top fields:     1 2 2 3 4
+Bottom fields:  1 2 3 4 4
+
+ +

The numbers correspond to the progressive frame the fields relate to. Here, the +first two frames are progressive, the 3rd and 4th are combed, and so on. +

+

When fieldmatch is configured to run a matching from bottom +(field=bottom) this is how this input stream get transformed: +

+
+
Input stream:
+                T     1 2 2 3 4
+                B     1 2 3 4 4   <-- matching reference
+
+Matches:              c c n n c
+
+Output stream:
+                T     1 2 3 4 4
+                B     1 2 3 4 4
+
+ +

As a result of the field matching, we can see that some frames get duplicated. +To perform a complete inverse telecine, you need to rely on a decimation filter +after this operation. See for instance the decimate filter. +

+

The same operation now matching from top fields (field=top) +looks like this: +

+
+
Input stream:
+                T     1 2 2 3 4   <-- matching reference
+                B     1 2 3 4 4
+
+Matches:              c c p p c
+
+Output stream:
+                T     1 2 2 3 4
+                B     1 2 2 3 4
+
+ +

In these examples, we can see what p, c and n mean; +basically, they refer to the frame and field of the opposite parity: +

+
    +
  • p matches the field of the opposite parity in the previous frame +
  • c matches the field of the opposite parity in the current frame +
  • n matches the field of the opposite parity in the next frame +
+ +

11.93.1.2 u/b

+ +

The u and b matching are a bit special in the sense that they match +from the opposite parity flag. In the following examples, we assume that we are +currently matching the 2nd frame (Top:2, bottom:2). According to the match, a +’x’ is placed above and below each matched fields. +

+

With bottom matching (field=bottom): +

+
Match:           c         p           n          b          u
+
+                 x       x               x        x          x
+  Top          1 2 2     1 2 2       1 2 2      1 2 2      1 2 2
+  Bottom       1 2 3     1 2 3       1 2 3      1 2 3      1 2 3
+                 x         x           x        x              x
+
+Output frames:
+                 2          1          2          2          2
+                 2          2          2          1          3
+
+ +

With top matching (field=top): +

+
Match:           c         p           n          b          u
+
+                 x         x           x        x              x
+  Top          1 2 2     1 2 2       1 2 2      1 2 2      1 2 2
+  Bottom       1 2 3     1 2 3       1 2 3      1 2 3      1 2 3
+                 x       x               x        x          x
+
+Output frames:
+                 2          2          2          1          2
+                 2          1          3          2          2
+
+ +

11.93.2 Examples

+ +

Simple IVTC of a top field first telecined stream: +

+
fieldmatch=order=tff:combmatch=none, decimate
+
+ +

Advanced IVTC, with fallback on yadif for still combed frames: +

+
fieldmatch=order=tff:combmatch=full, yadif=deint=interlaced, decimate
+
+ +

11.94 fieldorder

+ +

Transform the field order of the input video. +

+

It accepts the following parameters: +

+
+
order
+

The output field order. Valid values are tff for top field first or bff +for bottom field first. +

+
+ +

The default value is ‘tff’. +

+

The transformation is done by shifting the picture content up or down +by one line, and filling the remaining line with appropriate picture content. +This method is consistent with most broadcast field order converters. +

+

If the input video is not flagged as being interlaced, or it is already +flagged as being of the required output field order, then this filter does +not alter the incoming video. +

+

It is very useful when converting to or from PAL DV material, +which is bottom field first. +

+

For example: +

+
ffmpeg -i in.vob -vf "fieldorder=bff" out.dv
+
+ +

11.95 fillborders

+ +

Fill borders of the input video, without changing video stream dimensions. +Sometimes video can have garbage at the four edges and you may not want to +crop video input to keep size multiple of some number. +

+

This filter accepts the following options: +

+
+
left
+

Number of pixels to fill from left border. +

+
+
right
+

Number of pixels to fill from right border. +

+
+
top
+

Number of pixels to fill from top border. +

+
+
bottom
+

Number of pixels to fill from bottom border. +

+
+
mode
+

Set fill mode. +

+

It accepts the following values: +

+
‘smear’
+

fill pixels using outermost pixels +

+
+
‘mirror’
+

fill pixels using mirroring (half sample symmetric) +

+
+
‘fixed’
+

fill pixels with constant value +

+
+
‘reflect’
+

fill pixels using reflecting (whole sample symmetric) +

+
+
‘wrap’
+

fill pixels using wrapping +

+
+
‘fade’
+

fade pixels to constant value +

+
+
‘margins’
+

fill pixels at top and bottom with weighted averages pixels near borders +

+
+ +

Default is smear. +

+
+
color
+

Set color for pixels in fixed or fade mode. Default is black. +

+
+ +

11.95.1 Commands

+

This filter supports same commands as options. +The command accepts the same syntax of the corresponding option. +

+

If the specified expression is not valid, it is kept at its current +value. +

+

11.96 find_rect

+ +

Find a rectangular object in the input video. +

+

The object to search for must be specified as a gray8 image specified with the +object option. +

+

For each possible match, a score is computed. If the score reaches the specified +threshold, the object is considered found. +

+

If the input video contains multiple instances of the object, the filter will +find only one of them. +

+

When an object is found, the following metadata entries are set in the matching +frame: +

+
lavfi.rect.w
+

width of object +

+
+
lavfi.rect.h
+

height of object +

+
+
lavfi.rect.x
+

x position of object +

+
+
lavfi.rect.y
+

y position of object +

+
+
lavfi.rect.score
+

match score of the found object +

+
+ +

It accepts the following options: +

+
+
object
+

Filepath of the object image, needs to be in gray8. +

+
+
threshold
+

Detection threshold, expressed as a decimal number in the range 0-1. +

+

A threshold value of 0.01 means only exact matches, a threshold of 0.99 means +almost everything matches. +

+

Default value is 0.5. +

+
+
mipmaps
+

Number of mipmaps, default is 3. +

+
+
xmin, ymin, xmax, ymax
+

Specifies the rectangle in which to search. +

+
+
discard
+

Discard frames where object is not detected. Default is disabled. +

+
+ +

11.96.1 Examples

+ +
    +
  • Cover a rectangular object by the supplied image of a given video using ffmpeg: +
    +
    ffmpeg -i file.ts -vf find_rect=newref.pgm,cover_rect=cover.jpg:mode=cover new.mkv
    +
    + +
  • Find the position of an object in each frame using ffprobe and write +it to a log file: +
    +
    ffprobe -f lavfi movie=test.mp4,find_rect=object=object.pgm:threshold=0.3 \
    +  -show_entries frame=pkt_pts_time:frame_tags=lavfi.rect.x,lavfi.rect.y \
    +  -of csv -o find_rect.csv
    +
    + +
+ +

11.97 floodfill

+ +

Flood area with values of same pixel components with another values. +

+

It accepts the following options: +

+
x
+

Set pixel x coordinate. +

+
+
y
+

Set pixel y coordinate. +

+
+
s0
+

Set source #0 component value. +

+
+
s1
+

Set source #1 component value. +

+
+
s2
+

Set source #2 component value. +

+
+
s3
+

Set source #3 component value. +

+
+
d0
+

Set destination #0 component value. +

+
+
d1
+

Set destination #1 component value. +

+
+
d2
+

Set destination #2 component value. +

+
+
d3
+

Set destination #3 component value. +

+
+ +

11.98 format

+ +

Convert the input video to one of the specified pixel formats. +Libavfilter will try to pick one that is suitable as input to +the next filter. +

+

It accepts the following parameters: +

+
pix_fmts
+

A ’|’-separated list of pixel format names, such as +"pix_fmts=yuv420p|monow|rgb24". +

+
+
color_spaces
+

A ’|’-separated list of color space names, such as +"color_spaces=bt709|bt470bg|bt2020nc". +

+
+
color_ranges
+

A ’|’-separated list of color range names, such as +"color_ranges=tv|pc". +

+
+
alpha_modes
+

A ’|’-separated list of color range names, such as +"alpha_modes=straight|premultiplied". +

+
+
+ +

11.98.1 Examples

+ +
    +
  • Convert the input video to the yuv420p format +
    +
    format=pix_fmts=yuv420p
    +
    + +

    Convert the input video to any of the formats in the list +

    +
    format=pix_fmts=yuv420p|yuv444p|yuv410p
    +
    +
+ +

11.99 fps

+ +

Convert the video to specified constant frame rate by duplicating or dropping +frames as necessary. +

+

It accepts the following parameters: +

+
fps
+

The desired output frame rate. It accepts expressions containing the following +constants: +

+
+
‘source_fps’
+

The input’s frame rate +

+
+
‘ntsc’
+

NTSC frame rate of 30000/1001 +

+
+
‘pal’
+

PAL frame rate of 25.0 +

+
+
‘film’
+

Film frame rate of 24.0 +

+
+
‘ntsc_film’
+

NTSC-film frame rate of 24000/1001 +

+
+ +

The default is 25. +

+
+
start_time
+

Assume the first PTS should be the given value, in seconds. This allows for +padding/trimming at the start of stream. By default, no assumption is made +about the first frame’s expected PTS, so no padding or trimming is done. +For example, this could be set to 0 to pad the beginning with duplicates of +the first frame if a video stream starts after the audio stream or to trim any +frames with a negative PTS. +

+
+
round
+

Timestamp (PTS) rounding method. +

+

Possible values are: +

+
zero
+

round towards 0 +

+
inf
+

round away from 0 +

+
down
+

round towards -infinity +

+
up
+

round towards +infinity +

+
near
+

round to nearest +

+
+

The default is near. +

+
+
eof_action
+

Action performed when reading the last frame. +

+

Possible values are: +

+
round
+

Use same timestamp rounding method as used for other frames. +

+
pass
+

Pass through last frame if input duration has not been reached yet. +

+
+

The default is round. +

+
+
+ +

Alternatively, the options can be specified as a flat string: +fps[:start_time[:round]]. +

+

See also the setpts filter. +

+

11.99.1 Examples

+ +
    +
  • A typical usage in order to set the fps to 25: +
    +
    fps=fps=25
    +
    + +
  • Sets the fps to 24, using abbreviation and rounding method to round to nearest: +
    +
    fps=fps=film:round=near
    +
    +
+ +

11.100 framepack

+ +

Pack two different video streams into a stereoscopic video, setting proper +metadata on supported codecs. The two views should have the same size and +framerate and processing will stop when the shorter video ends. Please note +that you may conveniently adjust view properties with the scale and +fps filters. +

+

It accepts the following parameters: +

+
format
+

The desired packing format. Supported values are: +

+
+
sbs
+

The views are next to each other (default). +

+
+
tab
+

The views are on top of each other. +

+
+
lines
+

The views are packed by line. +

+
+
columns
+

The views are packed by column. +

+
+
frameseq
+

The views are temporally interleaved. +

+
+
+ +
+
+ +

Some examples: +

+
+
# Convert left and right views into a frame-sequential video
+ffmpeg -i LEFT -i RIGHT -filter_complex framepack=frameseq OUTPUT
+
+# Convert views into a side-by-side video with the same output resolution as the input
+ffmpeg -i LEFT -i RIGHT -filter_complex [0:v]scale=w=iw/2[left],[1:v]scale=w=iw/2[right],[left][right]framepack=sbs OUTPUT
+
+ +

11.101 framerate

+ +

Change the frame rate by interpolating new video output frames from the source +frames. +

+

This filter is not designed to function correctly with interlaced media. If +you wish to change the frame rate of interlaced media then you are required +to deinterlace before this filter and re-interlace after this filter. +

+

A description of the accepted options follows. +

+
+
fps
+

Specify the output frames per second. This option can also be specified +as a value alone. The default is 50. +

+
+
interp_start
+

Specify the start of a range where the output frame will be created as a +linear interpolation of two frames. The range is [0-255], +the default is 15. +

+
+
interp_end
+

Specify the end of a range where the output frame will be created as a +linear interpolation of two frames. The range is [0-255], +the default is 240. +

+
+
scene
+

Specify the level at which a scene change is detected as a value between +0 and 100 to indicate a new scene; a low value reflects a low +probability for the current frame to introduce a new scene, while a higher +value means the current frame is more likely to be one. +The default is 8.2. +

+
+
flags
+

Specify flags influencing the filter process. +

+

Available value for flags is: +

+
+
scene_change_detect, scd
+

Enable scene change detection using the value of the option scene. +This flag is enabled by default. +

+
+
+
+ +

11.102 frc_amf

+ +

Double the frame rate, using Frame Rate Converter (FRC) provided by +AMD Advanced Media Framework library for hardware acceleration. +

+

The filter accepts the following options: +

+
+
engine_type
+

Specify the engine used to run shaders. +

+
‘dx11’
+

DirectX 11. +

+
+
‘dx12’
+

DirectX 12 (Default value). +

+
+ +
+
enable
+

Boolean value: enable/disable FRC. Dynamic value, can be altered at a runtime +without re-initializing the filter. (Default value: enabled). +

+
+
fallback_mode
+

Fallback behavior in case of low interpolation confidence. +

+
‘duplicate’
+

Duplicate frame. +

+
+
‘blend’
+

Blend two frames together (Default value). +

+
+ +
+
indicator
+

Boolean value: show FRC indicator square in the top left corner of the video +(Default value: disabled). +

+
+
profile
+

Level of hierarchical motion search. +

+
‘low’
+

Less levels of hierarchical motion search. +Only recommended for extremely low resolutions. +

+
+
‘high’
+

Recommended for any resolution up to 1440p. (Default value) +

+
+
‘super’
+

More levels of hierarchical motion search. Recommended for resolutions 1440p +or higher. +

+
+ +
+
mv_search_mode
+

Performance mode of the motion search. +

+
‘native’
+

Conduct motion search on the full resolution of source images. +

+
+
‘performance’
+

Conduct motion search on the down scaled source images. +Recommended for APU or low end GPU for better performance. +

+
+ +
+
use_future_frame
+

Boolean value: enable dependency on future frame, improves quality for the cost +of latency (Default value: enabled). +

+
+ +

11.103 framestep

+ +

Select one frame every N-th frame. +

+

This filter accepts the following option: +

+
step
+

Select frame after every step frames. +Allowed values are positive integers higher than 0. Default value is 1. +

+
+ +

11.104 freezedetect

+ +

Detect frozen video. +

+

This filter logs a message and sets frame metadata when it detects that the +input video has no significant change in content during a specified duration. +Video freeze detection calculates the mean average absolute difference of all +the components of video frames and compares it to a noise floor. +

+

The printed times and duration are expressed in seconds. The +lavfi.freezedetect.freeze_start metadata key is set on the first frame +whose timestamp equals or exceeds the detection duration and it contains the +timestamp of the first frame of the freeze. The +lavfi.freezedetect.freeze_duration and +lavfi.freezedetect.freeze_end metadata keys are set on the first frame +after the freeze. +

+

The filter accepts the following options: +

+
+
noise, n
+

Set noise tolerance. Can be specified in dB (in case "dB" is appended to the +specified value) or as a difference ratio between 0 and 1. Default is -60dB, or +0.001. +

+
+
duration, d
+

Set freeze duration until notification (default is 2 seconds). +

+
+ +

11.105 freezeframes

+ +

Freeze video frames. +

+

This filter freezes video frames using frame from 2nd input. +

+

The filter accepts the following options: +

+
+
first
+

Set number of first frame from which to start freeze. +

+
+
last
+

Set number of last frame from which to end freeze. +

+
+
replace
+

Set number of frame from 2nd input which will be used instead of replaced frames. +

+
+ +

11.106 frei0r

+ +

Apply a frei0r effect to the input video. +

+

To enable the compilation of this filter, you need to install the frei0r +header and configure FFmpeg with --enable-frei0r. +

+

It accepts the following parameters: +

+
+
filter_name
+

The name of the frei0r effect to load. If the environment variable +FREI0R_PATH is defined, the frei0r effect is searched for in each of the +directories specified by the colon-separated list in FREI0R_PATH. +Otherwise, the standard frei0r paths are searched, in this order: +HOME/.frei0r-1/lib/, /usr/local/lib/frei0r-1/, +/usr/lib/frei0r-1/. +

+
+
filter_params
+

A ’|’-separated list of parameters to pass to the frei0r effect. +

+
+
+ +

A frei0r effect parameter can be a boolean (its value is either +"y" or "n"), a double, a color (specified as +R/G/B, where R, G, and B are floating point +numbers between 0.0 and 1.0, inclusive) or a color description as specified in the +(ffmpeg-utils)"Color" section in the ffmpeg-utils manual, +a position (specified as X/Y, where +X and Y are floating point numbers) and/or a string. +

+

The number and types of parameters depend on the loaded effect. If an +effect parameter is not specified, the default value is set. +

+

11.106.1 Examples

+ +
    +
  • Apply the distort0r effect, setting the first two double parameters: +
    +
    frei0r=filter_name=distort0r:filter_params=0.5|0.01
    +
    + +
  • Apply the colordistance effect, taking a color as the first parameter: +
    +
    frei0r=colordistance:0.2/0.3/0.4
    +frei0r=colordistance:violet
    +frei0r=colordistance:0x112233
    +
    + +
  • Apply the perspective effect, specifying the top left and top right image +positions: +
    +
    frei0r=perspective:0.2/0.2|0.8/0.2
    +
    +
+ +

For more information, see +http://frei0r.dyne.org +

+

11.106.2 Commands

+ +

This filter supports the filter_params option as commands. +

+

11.107 fspp

+ +

Apply fast and simple postprocessing. It is a faster version of spp. +

+

It splits (I)DCT into horizontal/vertical passes. Unlike the simple post- +processing filter, one of them is performed once per block, not per pixel. +This allows for much higher speed. +

+

The filter accepts the following options: +

+
+
quality
+

Set quality. This option defines the number of levels for averaging. It accepts +an integer in the range 4-5. Default value is 4. +

+
+
qp
+

Force a constant quantization parameter. It accepts an integer in range 0-63. +If not set, the filter will use the QP from the video stream (if available). +

+
+
strength
+

Set filter strength. It accepts an integer in range -15 to 32. Lower values mean +more details but also more artifacts, while higher values make the image smoother +but also blurrier. Default value is 0 − PSNR optimal. +

+
+
use_bframe_qp
+

Enable the use of the QP from the B-Frames if set to 1. Using this +option may cause flicker since the B-Frames have often larger QP. Default is +0 (not enabled). +

+
+
+ +

11.108 fsync

+ +

Synchronize video frames with an external mapping from a file. +

+

For each input PTS given in the map file it either drops or creates as many +frames as necessary to recreate the sequence of output frames given in the +map file. +

+

This filter is useful to recreate the output frames of a framerate conversion +by the fps filter, recorded into a map file using the ffmpeg option +-stats_mux_pre, and do further processing to the corresponding frames +e.g. quality comparison. +

+

Each line of the map file must contain three items per input frame, the input +PTS (decimal), the output PTS (decimal) and the +output TIMEBASE (decimal/decimal), separated by a space. +This file format corresponds to the output +of -stats_mux_pre_fmt="{ptsi} {pts} {tb}". +

+

The filter assumes the map file is sorted by increasing input PTS. +

+

The filter accepts the following options: +

+
file, f
+

The filename of the map file to be used. +

+
+ +

Example: +

+
# Convert a video to 25 fps and record a MAP_FILE file with the default format of this filter
+ffmpeg -i INPUT -vf fps=fps=25 -stats_mux_pre MAP_FILE -stats_mux_pre_fmt "{ptsi} {pts} {tb}" OUTPUT
+
+# Sort MAP_FILE by increasing input PTS
+sort -n MAP_FILE
+
+# Use INPUT, OUTPUT and the MAP_FILE from above to compare the corresponding frames in INPUT and OUTPUT via SSIM
+ffmpeg -i INPUT -i OUTPUT -filter_complex '[0:v]fsync=file=MAP_FILE[ref];[1:v][ref]ssim' -f null -
+
+ +

11.109 gblur

+ +

Apply Gaussian blur filter. +

+

The filter accepts the following options: +

+
+
sigma
+

Set horizontal sigma, standard deviation of Gaussian blur. Default is 0.5. +

+
+
steps
+

Set number of steps for Gaussian approximation. Default is 1. +

+
+
planes
+

Set which planes to filter. By default all planes are filtered. +

+
+
sigmaV
+

Set vertical sigma, if negative it will be same as sigma. +Default is -1. +

+
+ +

11.109.1 Commands

+

This filter supports same commands as options. +The command accepts the same syntax of the corresponding option. +

+

If the specified expression is not valid, it is kept at its current +value. +

+

11.110 geq

+ +

Apply generic equation to each pixel. +

+

The filter accepts the following options: +

+
+
lum_expr, lum
+

Set the luma expression. +

+
cb_expr, cb
+

Set the chrominance blue expression. +

+
cr_expr, cr
+

Set the chrominance red expression. +

+
alpha_expr, a
+

Set the alpha expression. +

+
red_expr, r
+

Set the red expression. +

+
green_expr, g
+

Set the green expression. +

+
blue_expr, b
+

Set the blue expression. +

+
+ +

The colorspace is selected according to the specified options. If one +of the lum_expr, cb_expr, or cr_expr +options is specified, the filter will automatically select a YCbCr +colorspace. If one of the red_expr, green_expr, or +blue_expr options is specified, it will select an RGB +colorspace. +

+

If one of the chrominance expression is not defined, it falls back on the other +one. If no alpha expression is specified it will evaluate to opaque value. +If none of chrominance expressions are specified, they will evaluate +to the luma expression. +

+

The expressions can use the following variables and functions: +

+
+
N
+

The sequential number of the filtered frame, starting from 0. +

+
+
X
+
Y
+

The coordinates of the current sample. +

+
+
W
+
H
+

The width and height of the image. +

+
+
SW
+
SH
+

Width and height scale depending on the currently filtered plane. It is the +ratio between the corresponding luma plane number of pixels and the current +plane ones. E.g. for YUV4:2:0 the values are 1,1 for the luma plane, and +0.5,0.5 for chroma planes. +

+
+
T
+

Time of the current frame, expressed in seconds. +

+
+
p(x, y)
+

Return the value of the pixel at location (x,y) of the current +plane. +

+
+
lum(x, y)
+

Return the value of the pixel at location (x,y) of the luma +plane. +

+
+
cb(x, y)
+

Return the value of the pixel at location (x,y) of the +blue-difference chroma plane. Return 0 if there is no such plane. +

+
+
cr(x, y)
+

Return the value of the pixel at location (x,y) of the +red-difference chroma plane. Return 0 if there is no such plane. +

+
+
r(x, y)
+
g(x, y)
+
b(x, y)
+

Return the value of the pixel at location (x,y) of the +red/green/blue component. Return 0 if there is no such component. +

+
+
alpha(x, y)
+

Return the value of the pixel at location (x,y) of the alpha +plane. Return 0 if there is no such plane. +

+
+
psum(x,y), lumsum(x, y), cbsum(x,y), crsum(x,y), rsum(x,y), gsum(x,y), bsum(x,y), alphasum(x,y)
+

Sum of sample values in the rectangle from (0,0) to (x,y), this allows obtaining +sums of samples within a rectangle. See the functions without the sum postfix. +

+
+
interpolation
+

Set one of interpolation methods: +

+
nearest, n
+
bilinear, b
+
+

Default is bilinear. +

+
+ +

For functions, if x and y are outside the area, the value will be +automatically clipped to the closer edge. +

+

Please note that this filter can use multiple threads in which case each slice +will have its own expression state. If you want to use only a single expression +state because your expressions depend on previous state then you should limit +the number of filter threads to 1. +

+

11.110.1 Examples

+ +
    +
  • Flip the image horizontally: +
    +
    geq=p(W-X\,Y)
    +
    + +
  • Generate a bidimensional sine wave, with angle PI/3 and a +wavelength of 100 pixels: +
    +
    geq=128 + 100*sin(2*(PI/100)*(cos(PI/3)*(X-50*T) + sin(PI/3)*Y)):128:128
    +
    + +
  • Generate a fancy enigmatic moving light: +
    +
    nullsrc=s=256x256,geq=random(1)/hypot(X-cos(N*0.07)*W/2-W/2\,Y-sin(N*0.09)*H/2-H/2)^2*1000000*sin(N*0.02):128:128
    +
    + +
  • Generate a quick emboss effect: +
    +
    format=gray,geq=lum_expr='(p(X,Y)+(256-p(X-4,Y-4)))/2'
    +
    + +
  • Modify RGB components depending on pixel position: +
    +
    geq=r='X/W*r(X,Y)':g='(1-X/W)*g(X,Y)':b='(H-Y)/H*b(X,Y)'
    +
    + +
  • Create a radial gradient that is the same size as the input (also see +the vignette filter): +
    +
    geq=lum=255*gauss((X/W-0.5)*3)*gauss((Y/H-0.5)*3)/gauss(0)/gauss(0),format=gray
    +
    +
+ +

11.111 gradfun

+ +

Fix the banding artifacts that are sometimes introduced into nearly flat +regions by truncation to 8-bit color depth. +Interpolate the gradients that should go where the bands are, and +dither them. +

+

It is designed for playback only. Do not use it prior to +lossy compression, because compression tends to lose the dither and +bring back the bands. +

+

It accepts the following parameters: +

+
+
strength
+

The maximum amount by which the filter will change any one pixel. This is also +the threshold for detecting nearly flat regions. Acceptable values range from +.51 to 64; the default value is 1.2. Out-of-range values will be clipped to the +valid range. +

+
+
radius
+

The neighborhood to fit the gradient to. A larger radius makes for smoother +gradients, but also prevents the filter from modifying the pixels near detailed +regions. Acceptable values are 8-32; the default value is 16. Out-of-range +values will be clipped to the valid range. +

+
+
+ +

Alternatively, the options can be specified as a flat string: +strength[:radius] +

+

11.111.1 Examples

+ +
    +
  • Apply the filter with a 3.5 strength and radius of 8: +
    +
    gradfun=3.5:8
    +
    + +
  • Specify radius, omitting the strength (which will fall-back to the default +value): +
    +
    gradfun=radius=8
    +
    + +
+ +

11.112 graphmonitor

+

Show various filtergraph stats. +

+

With this filter one can debug complete filtergraph. +Especially issues with links filling with queued frames. +

+

The filter accepts the following options: +

+
+
size, s
+

Set video output size. Default is hd720. +

+
+
opacity, o
+

Set video opacity. Default is 0.9. Allowed range is from 0 to 1. +

+
+
mode, m
+

Set output mode flags. +

+

Available values for flags are: +

+
‘full’
+

No any filtering. Default. +

+
‘compact’
+

Show only filters with queued frames. +

+
‘nozero’
+

Show only filters with non-zero stats. +

+
‘noeof’
+

Show only filters with non-eof stat. +

+
‘nodisabled’
+

Show only filters that are enabled in timeline. +

+
+ +
+
flags, f
+

Set flags which enable which stats are shown in video. +

+

Available values for flags are: +

+
‘none’
+

All flags turned off. +

+
+
‘all’
+

All flags turned on. +

+
+
‘queue’
+

Display number of queued frames in each link. +

+
+
‘frame_count_in’
+

Display number of frames taken from filter. +

+
+
‘frame_count_out’
+

Display number of frames given out from filter. +

+
+
‘frame_count_delta’
+

Display delta number of frames between above two values. +

+
+
‘pts’
+

Display current filtered frame pts. +

+
+
‘pts_delta’
+

Display pts delta between current and previous frame. +

+
+
‘time’
+

Display current filtered frame time. +

+
+
‘time_delta’
+

Display time delta between current and previous frame. +

+
+
‘timebase’
+

Display time base for filter link. +

+
+
‘format’
+

Display used format for filter link. +

+
+
‘size’
+

Display video size or number of audio channels in case of audio used by filter link. +

+
+
‘rate’
+

Display video frame rate or sample rate in case of audio used by filter link. +

+
+
‘eof’
+

Display link output status. +

+
+
‘sample_count_in’
+

Display number of samples taken from filter. +

+
+
‘sample_count_out’
+

Display number of samples given out from filter. +

+
+
‘sample_count_delta’
+

Display delta number of samples between above two values. +

+
+
‘disabled’
+

Show the timeline filter status. +

+
+ +
+
rate, r
+

Set upper limit for video rate of output stream, Default value is 25. +This guarantee that output video frame rate will not be higher than this value. +

+
+ +

11.113 grayworld

+

A color constancy filter that applies color correction based on the grayworld assumption +

+

See: https://www.researchgate.net/publication/275213614_A_New_Color_Correction_Method_for_Underwater_Imaging +

+

The algorithm uses linear light, so input +data should be linearized beforehand (and possibly correctly tagged). +

+
+
ffmpeg -i INPUT -vf zscale=transfer=linear,grayworld,zscale=transfer=bt709,format=yuv420p OUTPUT
+
+ +

11.114 greyedge

+

A color constancy variation filter which estimates scene illumination via grey edge algorithm +and corrects the scene colors accordingly. +

+

See: https://staff.science.uva.nl/th.gevers/pub/GeversTIP07.pdf +

+

The filter accepts the following options: +

+
+
difford
+

The order of differentiation to be applied on the scene. Must be chosen in the range +[0,2] and default value is 1. +

+
+
minknorm
+

The Minkowski parameter to be used for calculating the Minkowski distance. Must +be chosen in the range [0,20] and default value is 1. Set to 0 for getting +max value instead of calculating Minkowski distance. +

+
+
sigma
+

The standard deviation of Gaussian blur to be applied on the scene. Must be +chosen in the range [0,1024.0] and default value = 1. floor( sigma * break_off_sigma(3) ) +can’t be equal to 0 if difford is greater than 0. +

+
+ +

11.114.1 Examples

+
    +
  • Grey Edge: +
    +
    greyedge=difford=1:minknorm=5:sigma=2
    +
    + +
  • Max Edge: +
    +
    greyedge=difford=1:minknorm=0:sigma=2
    +
    + +
+ +

11.115 guided

+

Apply guided filter for edge-preserving smoothing, dehazing and so on. +

+

The filter accepts the following options: +

+
radius
+

Set the box radius in pixels. +Allowed range is 1 to 20. Default is 3. +

+
+
eps
+

Set regularization parameter (with square). +Allowed range is 0 to 1. Default is 0.01. +

+
+
mode
+

Set filter mode. Can be basic or fast. +Default is basic. +

+
+
sub
+

Set subsampling ratio for fast mode. +Range is 2 to 64. Default is 4. +No subsampling occurs in basic mode. +

+
+
guidance
+

Set guidance mode. Can be off or on. Default is off. +If off, single input is required. +If on, two inputs of the same resolution and pixel format are required. +The second input serves as the guidance. +

+
+
planes
+

Set planes to filter. Default is first only. +

+
+ +

11.115.1 Commands

+

This filter supports the all above options as commands. +

+

11.115.2 Examples

+
    +
  • Edge-preserving smoothing with guided filter: +
    +
    ffmpeg -i in.png -vf guided out.png
    +
    + +
  • Dehazing, structure-transferring filtering, detail enhancement with guided filter. +For the generation of guidance image, refer to paper "Guided Image Filtering". +See: http://kaiminghe.com/publications/pami12guidedfilter.pdf. +
    +
    ffmpeg -i in.png -i guidance.png -filter_complex guided=guidance=on out.png
    +
    + +
+ +

11.116 haldclut

+ +

Apply a Hald CLUT to a video stream. +

+

First input is the video stream to process, and second one is the Hald CLUT. +The Hald CLUT input can be a simple picture or a complete video stream. +

+

The filter accepts the following options: +

+
+
clut
+

Set which CLUT video frames will be processed from second input stream, +can be first or all. Default is all. +

+
+
shortest
+

Force termination when the shortest input terminates. Default is 0. +

+
repeatlast
+

Continue applying the last CLUT after the end of the stream. A value of +0 disable the filter after the last frame of the CLUT is reached. +Default is 1. +

+
+ +

haldclut also has the same interpolation options as lut3d (both +filters share the same internals). +

+

This filter also supports the framesync options. +

+

More information about the Hald CLUT can be found on Eskil Steenberg’s website +(Hald CLUT author) at http://www.quelsolaar.com/technology/clut.html. +

+

11.116.1 Commands

+ +

This filter supports the interp option as commands. +

+

11.116.2 Workflow examples

+ +

11.116.2.1 Hald CLUT video stream

+ +

Generate an identity Hald CLUT stream altered with various effects: +

+
ffmpeg -f lavfi -i haldclutsrc=8 -vf "hue=H=2*PI*t:s=sin(2*PI*t)+1, curves=cross_process" -t 10 -c:v ffv1 clut.nut
+
+ +

Note: make sure you use a lossless codec. +

+

Then use it with haldclut to apply it on some random stream: +

+
ffmpeg -f lavfi -i mandelbrot -i clut.nut -filter_complex '[0][1] haldclut' -t 20 mandelclut.mkv
+
+ +

The Hald CLUT will be applied to the 10 first seconds (duration of +clut.nut), then the latest picture of that CLUT stream will be applied +to the remaining frames of the mandelbrot stream. +

+

11.116.2.2 Hald CLUT with preview

+ +

A Hald CLUT is supposed to be a squared image of Level*Level*Level by +Level*Level*Level pixels. For a given Hald CLUT, FFmpeg will select the +biggest possible square starting at the top left of the picture. The remaining +padding pixels (bottom or right) will be ignored. This area can be used to add +a preview of the Hald CLUT. +

+

Typically, the following generated Hald CLUT will be supported by the +haldclut filter: +

+
+
ffmpeg -f lavfi -i haldclutsrc=8 -vf "
+   pad=iw+320 [padded_clut];
+   smptebars=s=320x256, split [a][b];
+   [padded_clut][a] overlay=W-320:h, curves=color_negative [main];
+   [main][b] overlay=W-320" -frames:v 1 clut.png
+
+ +

It contains the original and a preview of the effect of the CLUT: SMPTE color +bars are displayed on the right-top, and below the same color bars processed by +the color changes. +

+

Then, the effect of this Hald CLUT can be visualized with: +

+
ffplay input.mkv -vf "movie=clut.png, [in] haldclut"
+
+ +

11.117 hflip

+ +

Flip the input video horizontally. +

+

For example, to horizontally flip the input video with ffmpeg: +

+
ffmpeg -i in.avi -vf "hflip" out.avi
+
+ +

11.118 histeq

+

This filter applies a global color histogram equalization on a +per-frame basis. +

+

It can be used to correct video that has a compressed range of pixel +intensities. The filter redistributes the pixel intensities to +equalize their distribution across the intensity range. It may be +viewed as an "automatically adjusting contrast filter". This filter is +useful only for correcting degraded or poorly captured source +video. +

+

The filter accepts the following options: +

+
+
strength
+

Determine the amount of equalization to be applied. As the strength +is reduced, the distribution of pixel intensities more-and-more +approaches that of the input frame. The value must be a float number +in the range [0,1] and defaults to 0.200. +

+
+
intensity
+

Set the maximum intensity that can generated and scale the output +values appropriately. The strength should be set as desired and then +the intensity can be limited if needed to avoid washing-out. The value +must be a float number in the range [0,1] and defaults to 0.210. +

+
+
antibanding
+

Set the antibanding level. If enabled the filter will randomly vary +the luminance of output pixels by a small amount to avoid banding of +the histogram. Possible values are none, weak or +strong. It defaults to none. +

+
+ +

11.119 histogram

+ +

Compute and draw a color distribution histogram for the input video. +

+

The computed histogram is a representation of the color component +distribution in an image. +

+

Standard histogram displays the color components distribution in an image. +Displays color graph for each color component. Shows distribution of +the Y, U, V, A or R, G, B components, depending on input format, in the +current frame. Below each graph a color component scale meter is shown. +

+

The filter accepts the following options: +

+
+
level_height
+

Set height of level. Default value is 200. +Allowed range is [50, 2048]. +

+
+
scale_height
+

Set height of color scale. Default value is 12. +Allowed range is [0, 40]. +

+
+
display_mode
+

Set display mode. +It accepts the following values: +

+
‘stack’
+

Per color component graphs are placed below each other. +

+
+
‘parade’
+

Per color component graphs are placed side by side. +

+
+
‘overlay’
+

Presents information identical to that in the parade, except +that the graphs representing color components are superimposed directly +over one another. +

+
+

Default is stack. +

+
+
levels_mode
+

Set mode. Can be either linear, or logarithmic. +Default is linear. +

+
+
components
+

Set what color components to display. +Default is 7. +

+
+
fgopacity
+

Set foreground opacity. Default is 0.7. +

+
+
bgopacity
+

Set background opacity. Default is 0.5. +

+
+
colors_mode
+

Set colors mode. +It accepts the following values: +

+
‘whiteonblack’
+
‘blackonwhite’
+
‘whiteongray’
+
‘blackongray’
+
‘coloronblack’
+
‘coloronwhite’
+
‘colorongray’
+
‘blackoncolor’
+
‘whiteoncolor’
+
‘grayoncolor’
+
+

Default is whiteonblack. +

+
+ +

11.119.1 Examples

+ +
    +
  • Calculate and draw histogram: +
    +
    ffplay -i input -vf histogram
    +
    + +
+ +

11.120 hqdn3d

+ +

This is a high precision/quality 3d denoise filter. It aims to reduce +image noise, producing smooth images and making still images really +still. It should enhance compressibility. +

+

It accepts the following optional parameters: +

+
+
luma_spatial
+

A non-negative floating point number which specifies spatial luma strength. +It defaults to 4.0. +

+
+
chroma_spatial
+

A non-negative floating point number which specifies spatial chroma strength. +It defaults to 3.0*luma_spatial/4.0. +

+
+
luma_tmp
+

A floating point number which specifies luma temporal strength. It defaults to +6.0*luma_spatial/4.0. +

+
+
chroma_tmp
+

A floating point number which specifies chroma temporal strength. It defaults to +luma_tmp*chroma_spatial/luma_spatial. +

+
+ +

11.120.1 Commands

+

This filter supports same commands as options. +The command accepts the same syntax of the corresponding option. +

+

If the specified expression is not valid, it is kept at its current +value. +

+

11.121 hwdownload

+ +

Download hardware frames to system memory. +

+

The input must be in hardware frames, and the output a non-hardware format. +Not all formats will be supported on the output - it may be necessary to insert +an additional format filter immediately following in the graph to get +the output in a supported format. +

+

11.122 hwmap

+ +

Map hardware frames to system memory or to another device. +

+

This filter has several different modes of operation; which one is used depends +on the input and output formats: +

    +
  • Hardware frame input, normal frame output + +

    Map the input frames to system memory and pass them to the output. If the +original hardware frame is later required (for example, after overlaying +something else on part of it), the hwmap filter can be used again +in the next mode to retrieve it. +

  • Normal frame input, hardware frame output + +

    If the input is actually a software-mapped hardware frame, then unmap it - +that is, return the original hardware frame. +

    +

    Otherwise, a device must be provided. Create new hardware surfaces on that +device for the output, then map them back to the software format at the input +and give those frames to the preceding filter. This will then act like the +hwupload filter, but may be able to avoid an additional copy when +the input is already in a compatible format. +

  • Hardware frame input and output + +

    A device must be supplied for the output, either directly or with the +derive_device option. The input and output devices must be of +different types and compatible - the exact meaning of this is +system-dependent, but typically it means that they must refer to the same +underlying hardware context (for example, refer to the same graphics card). +

    +

    If the input frames were originally created on the output device, then unmap +to retrieve the original frames. +

    +

    Otherwise, map the frames to the output device - create new hardware frames +on the output corresponding to the frames on the input. +

+ +

The following additional parameters are accepted: +

+
+
mode
+

Set the frame mapping mode. Some combination of: +

+
read
+

The mapped frame should be readable. +

+
write
+

The mapped frame should be writeable. +

+
overwrite
+

The mapping will always overwrite the entire frame. +

+

This may improve performance in some cases, as the original contents of the +frame need not be loaded. +

+
direct
+

The mapping must not involve any copying. +

+

Indirect mappings to copies of frames are created in some cases where either +direct mapping is not possible or it would have unexpected properties. +Setting this flag ensures that the mapping is direct and will fail if that is +not possible. +

+
+

Defaults to read+write if not specified. +

+
+
derive_device type
+

Rather than using the device supplied at initialisation, instead derive a new +device of type type from the device the input frames exist on. +

+
+
reverse
+

In a hardware to hardware mapping, map in reverse - create frames in the sink +and map them back to the source. This may be necessary in some cases where +a mapping in one direction is required but only the opposite direction is +supported by the devices being used. +

+

This option is dangerous - it may break the preceding filter in undefined +ways if there are any additional constraints on that filter’s output. +Do not use it without fully understanding the implications of its use. +

+
+ +

11.123 hwupload

+ +

Upload system memory frames to hardware surfaces. +

+

The device to upload to must be supplied when the filter is initialised. If +using ffmpeg, select the appropriate device with the -filter_hw_device +option or with the derive_device option. The input and output devices +must be of different types and compatible - the exact meaning of this is +system-dependent, but typically it means that they must refer to the same +underlying hardware context (for example, refer to the same graphics card). +

+

The following additional parameters are accepted: +

+
+
derive_device type
+

Rather than using the device supplied at initialisation, instead derive a new +device of type type from the device the input frames exist on. +

+
+ +

11.124 hwupload_cuda

+ +

Upload system memory frames to a CUDA device. +

+

It accepts the following optional parameters: +

+
+
device
+

The number of the CUDA device to use +

+
+ +

11.125 hqx

+ +

Apply a high-quality magnification filter designed for pixel art. This filter +was originally created by Maxim Stepin. +

+

It accepts the following option: +

+
+
n
+

Set the scaling dimension: 2 for hq2x, 3 for +hq3x and 4 for hq4x. +Default is 3. +

+
+ +

11.126 hstack

+

Stack input videos horizontally. +

+

All streams must be of same pixel format and of same height. +

+

Note that this filter is faster than using overlay and pad filter +to create same output. +

+

The filter accepts the following option: +

+
+
inputs
+

Set number of input streams. Default is 2. +

+
+
shortest
+

If set to 1, force the output to terminate when the shortest input +terminates. Default value is 0. +

+
+ +

11.127 hsvhold

+

Turns a certain HSV range into gray values. +

+

This filter measures color difference between set HSV color in options +and ones measured in video stream. Depending on options, output +colors can be changed to be gray or not. +

+

The filter accepts the following options: +

+
+
hue
+

Set the hue value which will be used in color difference calculation. +Allowed range is from -360 to 360. Default value is 0. +

+
+
sat
+

Set the saturation value which will be used in color difference calculation. +Allowed range is from -1 to 1. Default value is 0. +

+
+
val
+

Set the value which will be used in color difference calculation. +Allowed range is from -1 to 1. Default value is 0. +

+
+
similarity
+

Set similarity percentage with the key color. +Allowed range is from 0 to 1. Default value is 0.01. +

+

0.00001 matches only the exact key color, while 1.0 matches everything. +

+
+
blend
+

Blend percentage. +Allowed range is from 0 to 1. Default value is 0. +

+

0.0 makes pixels either fully gray, or not gray at all. +

+

Higher values result in more gray pixels, with a higher gray pixel +the more similar the pixels color is to the key color. +

+
+ +

11.128 hsvkey

+

Turns a certain HSV range into transparency. +

+

This filter measures color difference between set HSV color in options +and ones measured in video stream. Depending on options, output +colors can be changed to transparent by adding alpha channel. +

+

The filter accepts the following options: +

+
+
hue
+

Set the hue value which will be used in color difference calculation. +Allowed range is from -360 to 360. Default value is 0. +

+
+
sat
+

Set the saturation value which will be used in color difference calculation. +Allowed range is from -1 to 1. Default value is 0. +

+
+
val
+

Set the value which will be used in color difference calculation. +Allowed range is from -1 to 1. Default value is 0. +

+
+
similarity
+

Set similarity percentage with the key color. +Allowed range is from 0 to 1. Default value is 0.01. +

+

0.00001 matches only the exact key color, while 1.0 matches everything. +

+
+
blend
+

Blend percentage. +Allowed range is from 0 to 1. Default value is 0. +

+

0.0 makes pixels either fully transparent, or not transparent at all. +

+

Higher values result in semi-transparent pixels, with a higher transparency +the more similar the pixels color is to the key color. +

+
+ +

11.129 hue

+ +

Modify the hue and/or the saturation of the input. +

+

It accepts the following parameters: +

+
+
h
+

Specify the hue angle as a number of degrees. It accepts an expression, +and defaults to "0". +

+
+
s
+

Specify the saturation in the [-10,10] range. It accepts an expression and +defaults to "1". +

+
+
H
+

Specify the hue angle as a number of radians. It accepts an +expression, and defaults to "0". +

+
+
b
+

Specify the brightness in the [-10,10] range. It accepts an expression and +defaults to "0". +

+
+ +

h and H are mutually exclusive, and can’t be +specified at the same time. +

+

The b, h, H and s option values are +expressions containing the following constants: +

+
+
n
+

frame count of the input frame starting from 0 +

+
+
pts
+

presentation timestamp of the input frame expressed in time base units +

+
+
r
+

frame rate of the input video, NAN if the input frame rate is unknown +

+
+
t
+

timestamp expressed in seconds, NAN if the input timestamp is unknown +

+
+
tb
+

time base of the input video +

+
+ +

11.129.1 Examples

+ +
    +
  • Set the hue to 90 degrees and the saturation to 1.0: +
    +
    hue=h=90:s=1
    +
    + +
  • Same command but expressing the hue in radians: +
    +
    hue=H=PI/2:s=1
    +
    + +
  • Rotate hue and make the saturation swing between 0 +and 2 over a period of 1 second: +
    +
    hue="H=2*PI*t: s=sin(2*PI*t)+1"
    +
    + +
  • Apply a 3 seconds saturation fade-in effect starting at 0: +
    +
    hue="s=min(t/3\,1)"
    +
    + +

    The general fade-in expression can be written as: +

    +
    hue="s=min(0\, max((t-START)/DURATION\, 1))"
    +
    + +
  • Apply a 3 seconds saturation fade-out effect starting at 5 seconds: +
    +
    hue="s=max(0\, min(1\, (8-t)/3))"
    +
    + +

    The general fade-out expression can be written as: +

    +
    hue="s=max(0\, min(1\, (START+DURATION-t)/DURATION))"
    +
    + +
+ +

11.129.2 Commands

+ +

This filter supports the following commands: +

+
b
+
s
+
h
+
H
+

Modify the hue and/or the saturation and/or brightness of the input video. +The command accepts the same syntax of the corresponding option. +

+

If the specified expression is not valid, it is kept at its current +value. +

+
+ +

11.130 huesaturation

+

Apply hue-saturation-intensity adjustments to input video stream. +

+

This filter operates in RGB colorspace. +

+

This filter accepts the following options: +

+
+
hue
+

Set the hue shift in degrees to apply. Default is 0. +Allowed range is from -180 to 180. +

+
+
saturation
+

Set the saturation shift. Default is 0. +Allowed range is from -1 to 1. +

+
+
intensity
+

Set the intensity shift. Default is 0. +Allowed range is from -1 to 1. +

+
+
colors
+

Set which primary and complementary colors are going to be adjusted. +This options is set by providing one or multiple values. +This can select multiple colors at once. By default all colors are selected. +

+
‘r’
+

Adjust reds. +

+
‘y’
+

Adjust yellows. +

+
‘g’
+

Adjust greens. +

+
‘c’
+

Adjust cyans. +

+
‘b’
+

Adjust blues. +

+
‘m’
+

Adjust magentas. +

+
‘a’
+

Adjust all colors. +

+
+ +
+
strength
+

Set strength of filtering. Allowed range is from 0 to 100. +Default value is 1. +

+
+
rw, gw, bw
+

Set weight for each RGB component. Allowed range is from 0 to 1. +By default is set to 0.333, 0.334, 0.333. +Those options are used in saturation and lightess processing. +

+
+
lightness
+

Set preserving lightness, by default is disabled. +Adjusting hues can change lightness from original RGB triplet, +with this option enabled lightness is kept at same value. +

+
+ +

11.131 hysteresis

+ +

Grow first stream into second stream by connecting components. +This makes it possible to build more robust edge masks. +

+

This filter accepts the following options: +

+
+
planes
+

Set which planes will be processed as bitmap, unprocessed planes will be +copied from first stream. +By default value 0xf, all planes will be processed. +

+
+
threshold
+

Set threshold which is used in filtering. If pixel component value is higher than +this value filter algorithm for connecting components is activated. +By default value is 0. +

+
+ +

The hysteresis filter also supports the framesync options. +

+

11.132 iccdetect

+ +

Detect the colorspace from an embedded ICC profile (if present), and update +the frame’s tags accordingly. +

+

This filter accepts the following options: +

+
+
force
+

If true, the frame’s existing colorspace tags will always be overridden by +values detected from an ICC profile. Otherwise, they will only be assigned if +they contain unknown. Enabled by default. +

+
+ +

11.133 iccgen

+ +

Generate ICC profiles and attach them to frames. +

+

This filter accepts the following options: +

+
+
color_primaries
+
color_trc
+

Configure the colorspace that the ICC profile will be generated for. The +default value of auto infers the value from the input frame’s metadata, +defaulting to BT.709/sRGB as appropriate. +

+

See the setparams filter for a list of possible values, but note that +unknown are not valid values for this filter. +

+
+
force
+

If true, an ICC profile will be generated even if it would overwrite an +already existing ICC profile. Disabled by default. +

+
+ +

11.134 identity

+ +

Obtain the identity score between two input videos. +

+

This filter takes two input videos. +

+

Both input videos must have the same resolution and pixel format for +this filter to work correctly. Also it assumes that both inputs +have the same number of frames, which are compared one by one. +

+

The obtained per component, average, min and max identity score is printed through +the logging system. +

+

The filter stores the calculated identity scores of each frame in frame metadata. +

+

This filter also supports the framesync options. +

+

In the below example the input file main.mpg being processed is compared +with the reference file ref.mpg. +

+
+
ffmpeg -i main.mpg -i ref.mpg -lavfi identity -f null -
+
+ +

11.135 idet

+ +

Detect video interlacing type. +

+

This filter tries to detect if the input frames are interlaced, progressive, +top or bottom field first. It will also try to detect fields that are +repeated between adjacent frames (a sign of telecine). +

+

Single frame detection considers only immediately adjacent frames when classifying each frame. +Multiple frame detection incorporates the classification history of previous frames. +

+

The filter will log these metadata values: +

+
+
single.current_frame
+

Detected type of current frame using single-frame detection. One of: +“tff” (top field first), “bff” (bottom field first), +“progressive”, or “undetermined” +

+
+
single.tff
+

Cumulative number of frames detected as top field first using single-frame detection. +

+
+
multiple.tff
+

Cumulative number of frames detected as top field first using multiple-frame detection. +

+
+
single.bff
+

Cumulative number of frames detected as bottom field first using single-frame detection. +

+
+
multiple.current_frame
+

Detected type of current frame using multiple-frame detection. One of: +“tff” (top field first), “bff” (bottom field first), +“progressive”, or “undetermined” +

+
+
multiple.bff
+

Cumulative number of frames detected as bottom field first using multiple-frame detection. +

+
+
single.progressive
+

Cumulative number of frames detected as progressive using single-frame detection. +

+
+
multiple.progressive
+

Cumulative number of frames detected as progressive using multiple-frame detection. +

+
+
single.undetermined
+

Cumulative number of frames that could not be classified using single-frame detection. +

+
+
multiple.undetermined
+

Cumulative number of frames that could not be classified using multiple-frame detection. +

+
+
repeated.current_frame
+

Which field in the current frame is repeated from the last. One of “neither”, “top”, or “bottom”. +

+
+
repeated.neither
+

Cumulative number of frames with no repeated field. +

+
+
repeated.top
+

Cumulative number of frames with the top field repeated from the previous frame’s top field. +

+
+
repeated.bottom
+

Cumulative number of frames with the bottom field repeated from the previous frame’s bottom field. +

+
+ +

The filter accepts the following options: +

+
+
intl_thres
+

Set interlacing threshold. +

+
prog_thres
+

Set progressive threshold. +

+
rep_thres
+

Threshold for repeated field detection. +

+
half_life
+

Number of frames after which a given frame’s contribution to the +statistics is halved (i.e., it contributes only 0.5 to its +classification). The default of 0 means that all frames seen are given +full weight of 1.0 forever. +

+
analyze_interlaced_flag
+

When this is not 0 then idet will use the specified number of frames to determine +if the interlaced flag is accurate, it will not count undetermined frames. +If the flag is found to be accurate it will be used without any further +computations, if it is found to be inaccurate it will be cleared without any +further computations. This allows inserting the idet filter as a low computational +method to clean up the interlaced flag +

+
+ +

11.135.1 Examples

+ +

Inspect the field order of the first 360 frames in a video, in verbose detail: +

+
ffmpeg -i INPUT -filter:v idet,metadata=mode=print -frames:v 360 -an -f null -
+
+

The idet filter will add analysis metadata to each frame, which will then be +discarded. At the end, the filter will also print a final report with statistics. +

+

11.136 il

+ +

Deinterleave or interleave fields. +

+

This filter allows one to process interlaced images fields without +deinterlacing them. Deinterleaving splits the input frame into 2 +fields (so called half pictures). Odd lines are moved to the top +half of the output image, even lines to the bottom half. +You can process (filter) them independently and then re-interleave them. +

+

The filter accepts the following options: +

+
+
luma_mode, l
+
chroma_mode, c
+
alpha_mode, a
+

Available values for luma_mode, chroma_mode and +alpha_mode are: +

+
+
‘none’
+

Do nothing. +

+
+
‘deinterleave, d’
+

Deinterleave fields, placing one above the other. +

+
+
‘interleave, i’
+

Interleave fields. Reverse the effect of deinterleaving. +

+
+

Default value is none. +

+
+
luma_swap, ls
+
chroma_swap, cs
+
alpha_swap, as
+

Swap luma/chroma/alpha fields. Exchange even & odd lines. Default value is 0. +

+
+ +

11.136.1 Commands

+ +

This filter supports the all above options as commands. +

+

11.137 inflate

+ +

Apply inflate effect to the video. +

+

This filter replaces the pixel by the local(3x3) average by taking into account +only values higher than the pixel. +

+

It accepts the following options: +

+
+
threshold0
+
threshold1
+
threshold2
+
threshold3
+

Limit the maximum change for each plane, default is 65535. +If 0, plane will remain unchanged. +

+
+ +

11.137.1 Commands

+ +

This filter supports the all above options as commands. +

+

11.138 interlace, interlace_vulkan

+ +

Simple interlacing filter from progressive contents. This interleaves upper (or +lower) lines from odd frames with lower (or upper) lines from even frames, +halving the frame rate and preserving image height. +

+
+
   Original        Original             New Frame
+   Frame 'j'      Frame 'j+1'             (tff)
+  ==========      ===========       ==================
+    Line 0  -------------------->    Frame 'j' Line 0
+    Line 1          Line 1  ---->   Frame 'j+1' Line 1
+    Line 2 --------------------->    Frame 'j' Line 2
+    Line 3          Line 3  ---->   Frame 'j+1' Line 3
+     ...             ...                   ...
+New Frame + 1 will be generated by Frame 'j+2' and Frame 'j+3' and so on
+
+ +

It accepts the following optional parameters: +

+
+
scan
+

This determines whether the interlaced frame is taken from the even +(tff - default) or odd (bff) lines of the progressive frame. +

+
+
lowpass
+

Vertical lowpass filter to avoid twitter interlacing and +reduce moire patterns. +

+
+
‘0, off’
+

Disable vertical lowpass filter +

+
+
‘1, linear’
+

Enable linear filter (default) +

+
+
‘2, complex’
+

Enable complex filter. This will slightly less reduce twitter and moire +but better retain detail and subjective sharpness impression. +

+
+
+
+
+ +

11.139 kerndeint

+ +

Deinterlace input video by applying Donald Graft’s adaptive kernel +deinterling. Work on interlaced parts of a video to produce +progressive frames. +

+

The description of the accepted parameters follows. +

+
+
thresh
+

Set the threshold which affects the filter’s tolerance when +determining if a pixel line must be processed. It must be an integer +in the range [0,255] and defaults to 10. A value of 0 will result in +applying the process on every pixels. +

+
+
map
+

Paint pixels exceeding the threshold value to white if set to 1. +Default is 0. +

+
+
order
+

Set the fields order. Swap fields if set to 1, leave fields alone if +0. Default is 0. +

+
+
sharp
+

Enable additional sharpening if set to 1. Default is 0. +

+
+
twoway
+

Enable twoway sharpening if set to 1. Default is 0. +

+
+ +

11.139.1 Examples

+ +
    +
  • Apply default values: +
    +
    kerndeint=thresh=10:map=0:order=0:sharp=0:twoway=0
    +
    + +
  • Enable additional sharpening: +
    +
    kerndeint=sharp=1
    +
    + +
  • Paint processed pixels in white: +
    +
    kerndeint=map=1
    +
    +
+ +

11.140 kirsch

+

Apply kirsch operator to input video stream. +

+

The filter accepts the following option: +

+
+
planes
+

Set which planes will be processed, unprocessed planes will be copied. +By default value 0xf, all planes will be processed. +

+
+
scale
+

Set value which will be multiplied with filtered result. +

+
+
delta
+

Set value which will be added to filtered result. +

+
+ +

11.140.1 Commands

+ +

This filter supports the all above options as commands. +

+

11.141 lagfun

+ +

Slowly update darker pixels. +

+

This filter makes short flashes of light appear longer. +This filter accepts the following options: +

+
+
decay
+

Set factor for decaying. Default is .95. Allowed range is from 0 to 1. +

+
+
planes
+

Set which planes to filter. Default is all. Allowed range is from 0 to 15. +

+
+ +

11.141.1 Commands

+ +

This filter supports the all above options as commands. +

+

11.142 lenscorrection

+ +

Correct radial lens distortion +

+

This filter can be used to correct for radial distortion as can result from the use +of wide angle lenses, and thereby re-rectify the image. To find the right parameters +one can use tools available for example as part of opencv or simply trial-and-error. +To use opencv use the calibration sample (under samples/cpp) from the opencv sources +and extract the k1 and k2 coefficients from the resulting matrix. +

+

Note that effectively the same filter is available in the open-source tools Krita and +Digikam from the KDE project. +

+

In contrast to the vignette filter, which can also be used to compensate lens errors, +this filter corrects the distortion of the image, whereas vignette corrects the +brightness distribution, so you may want to use both filters together in certain +cases, though you will have to take care of ordering, i.e. whether vignetting should +be applied before or after lens correction. +

+

11.142.1 Options

+ +

The filter accepts the following options: +

+
+
cx
+

Relative x-coordinate of the focal point of the image, and thereby the center of the +distortion. This value has a range [0,1] and is expressed as fractions of the image +width. Default is 0.5. +

+
cy
+

Relative y-coordinate of the focal point of the image, and thereby the center of the +distortion. This value has a range [0,1] and is expressed as fractions of the image +height. Default is 0.5. +

+
k1
+

Coefficient of the quadratic correction term. This value has a range [-1,1]. 0 means +no correction. Default is 0. +

+
k2
+

Coefficient of the double quadratic correction term. This value has a range [-1,1]. +0 means no correction. Default is 0. +

+
i
+

Set interpolation type. Can be nearest or bilinear. +Default is nearest. +

+
fc
+

Specify the color of the unmapped pixels. For the syntax of this option, +check the (ffmpeg-utils)"Color" section in the ffmpeg-utils +manual. Default color is black@0. +

+
+ +

The formula that generates the correction is: +

+

r_src = r_tgt * (1 + k1 * (r_tgt / r_0)^2 + k2 * (r_tgt / r_0)^4) +

+

where r_0 is halve of the image diagonal and r_src and r_tgt are the +distances from the focal point in the source and target images, respectively. +

+

11.142.2 Commands

+ +

This filter supports the all above options as commands. +

+

11.143 lensfun

+ +

Apply lens correction via the lensfun library (http://lensfun.sourceforge.net/). +

+

The lensfun filter requires the camera make, camera model, and lens model +to apply the lens correction. The filter will load the lensfun database and +query it to find the corresponding camera and lens entries in the database. As +long as these entries can be found with the given options, the filter can +perform corrections on frames. Note that incomplete strings will result in the +filter choosing the best match with the given options, and the filter will +output the chosen camera and lens models (logged with level "info"). You must +provide the make, camera model, and lens model as they are required. +

+

To obtain a list of available makes and models, leave out one or both of make and +model options. The filter will send the full list to the log with level INFO. +The first column is the make and the second column is the model. +To obtain a list of available lenses, set any values for make and model and leave out the +lens_model option. The filter will send the full list of lenses in the log with level +INFO. The ffmpeg tool will exit after the list is printed. +

+

The filter accepts the following options: +

+
+
make
+

The make of the camera (for example, "Canon"). This option is required. +

+
+
model
+

The model of the camera (for example, "Canon EOS 100D"). This option is +required. +

+
+
lens_model
+

The model of the lens (for example, "Canon EF-S 18-55mm f/3.5-5.6 IS STM"). This +option is required. +

+
+
db_path
+

The full path to the lens database folder. If not set, the filter will attempt to +load the database from the install path when the library was built. Default is unset. +

+
+
mode
+

The type of correction to apply. The following values are valid options: +

+
+
‘vignetting’
+

Enables fixing lens vignetting. +

+
+
‘geometry’
+

Enables fixing lens geometry. This is the default. +

+
+
‘subpixel’
+

Enables fixing chromatic aberrations. +

+
+
‘vig_geo’
+

Enables fixing lens vignetting and lens geometry. +

+
+
‘vig_subpixel’
+

Enables fixing lens vignetting and chromatic aberrations. +

+
+
‘distortion’
+

Enables fixing both lens geometry and chromatic aberrations. +

+
+
‘all’
+

Enables all possible corrections. +

+
+
+
+
focal_length
+

The focal length of the image/video (zoom; expected constant for video). For +example, a 18–55mm lens has focal length range of [18–55], so a value in that +range should be chosen when using that lens. Default 18. +

+
+
aperture
+

The aperture of the image/video (expected constant for video). Note that +aperture is only used for vignetting correction. Default 3.5. +

+
+
focus_distance
+

The focus distance of the image/video (expected constant for video). Note that +focus distance is only used for vignetting and only slightly affects the +vignetting correction process. If unknown, leave it at the default value (which +is 1000). +

+
+
scale
+

The scale factor which is applied after transformation. After correction the +video is no longer necessarily rectangular. This parameter controls how much of +the resulting image is visible. The value 0 means that a value will be chosen +automatically such that there is little or no unmapped area in the output +image. 1.0 means that no additional scaling is done. Lower values may result +in more of the corrected image being visible, while higher values may avoid +unmapped areas in the output. +

+
+
target_geometry
+

The target geometry of the output image/video. The following values are valid +options: +

+
+
‘rectilinear (default)’
+
‘fisheye’
+
‘panoramic’
+
‘equirectangular’
+
‘fisheye_orthographic’
+
‘fisheye_stereographic’
+
‘fisheye_equisolid’
+
‘fisheye_thoby’
+
+
+
reverse
+

Apply the reverse of image correction (instead of correcting distortion, apply +it). +

+
+
interpolation
+

The type of interpolation used when correcting distortion. The following values +are valid options: +

+
+
‘nearest’
+
‘linear (default)’
+
‘lanczos’
+
+
+
+ +

11.143.1 Examples

+ +
    +
  • Apply lens correction with make "Canon", camera model "Canon EOS 100D", and lens +model "Canon EF-S 18-55mm f/3.5-5.6 IS STM" with focal length of "18" and +aperture of "8.0". + +
    +
    ffmpeg -i input.mov -vf lensfun=make=Canon:model="Canon EOS 100D":lens_model="Canon EF-S 18-55mm f/3.5-5.6 IS STM":focal_length=18:aperture=8 -c:v h264 -b:v 8000k output.mov
    +
    + +
  • Apply the same as before, but only for the first 5 seconds of video. + +
    +
    ffmpeg -i input.mov -vf lensfun=make=Canon:model="Canon EOS 100D":lens_model="Canon EF-S 18-55mm f/3.5-5.6 IS STM":focal_length=18:aperture=8:enable='lte(t\,5)' -c:v h264 -b:v 8000k output.mov
    +
    + +
+ +

11.144 lcevc

+ +

Low Complexity Enhancement Video Codec filter based on liblcevc_dec +(https://github.com/v-novaltd/LCEVCdec). +

+

11.145 libplacebo

+ +

Flexible GPU-accelerated processing filter based on libplacebo +(https://code.videolan.org/videolan/libplacebo). +

+

11.145.1 Options

+ +

The options for this filter are divided into the following sections: +

+

11.145.1.1 Output mode

+

These options control the overall output mode. By default, libplacebo will try +to preserve the source colorimetry and size as best as it can, but it will +apply any embedded film grain, dolby vision metadata or anamorphic SAR present +in source frames. +

+
inputs
+

Set the number of inputs. This can be used, alongside the idx variable, +to allow placing/blending multiple inputs inside the output frame. This +effectively enables functionality similar to hstack, overlay, etc. +

+
+
w
+
h
+

Set the output video dimension expression. Default values are iw and +ih. +

+

Allows for the same expressions as the scale filter. +

+
+
crop_x
+
crop_y
+

Set the input crop x/y expressions, default values are (iw-cw)/2 and +(ih-ch)/2. +

+
+
crop_w
+
crop_h
+

Set the input crop width/height expressions, default values are iw and +ih. +

+
+
pos_x
+
pos_y
+

Set the output placement x/y expressions, default values are (ow-pw)/2 +and (oh-ph)/2. +

+
+
pos_w
+
pos_h
+

Set the output placement width/height expressions, default values are ow +and oh. +

+
+
rotate
+

Rotate the input frame clockwise by the specified angle. +

+
+
‘0, 360’
+
‘90’
+
‘180’
+
‘270’
+
+ +
+
fps
+

Set the output frame rate. This can be rational, e.g. 60000/1001. If +set to the special string none (the default), input timestamps will +instead be passed through to the output unmodified. Otherwise, the input video +frames will be interpolated as necessary to rescale the video to the specified +target framerate, in a manner as determined by the frame_mixer option. +

+
+
format
+

Set the output format override. If unset (the default), frames will be output +in the same format as the respective input frames. Otherwise, format conversion +will be performed. +

+
+
force_original_aspect_ratio
+
force_divisible_by
+

Work the same as the identical scale filter options. Note that +force_divisible_by also works with fit_sense=constraint. +

+
+
reset_sar
+

If enabled, output frames will always have a pixel aspect ratio of 1:1. If +disabled (the default), any aspect ratio mismatches, including those from +e.g. anamorphic video sources, are forwarded to the output pixel aspect ratio. +

+
+
normalize_sar
+

Like reset_sar, but instead of stretching the video content to fill +the new output aspect ratio, the content is instead padded or cropped as +necessary. Mutually exclusive with fit_mode. Disabled by default. +

+
+
pad_crop_ratio
+

Specifies a ratio (between 0.0 and 1.0) between padding and +cropping when the input aspect ratio does not match the output aspect ratio and +normalize_sar is in effect. The default of 0.0 always pads the +content with black borders, while a value of 1.0 always crops off parts +of the content. Intermediate values are possible, leading to a mix of the two +approaches. +

+
+
fit_mode
+

Specify the content fit strategy according to a list of predefined modes. +Determines how the input image is to be placed inside the destination crop +rectangle (as defined by pos_x/y and pos_w/h). The names and +their implementations are taken from the CSS ’object-fit’ property. Note that +this option is mutually exclusive with normalize_sar. Defaults to +fill. Valid values are: +

+
+
‘fill’
+

Stretch the input to the output rectangle, ignoring aspect ratio mismatches. +Note that unless reset_sar is also enabled, the output will still +have the correct pixel aspect ratio tagged. +

+
+
‘contain’
+

Scale the input to fit inside the output, preserving aspect ratio by padding. +Equivalent to normalize_sar with pad_crop_ratio set to +0.0. +

+
+
‘cover’
+

Scale the input to fill the output, preserving aspect ratio by cropping. +Equivalent to normalize_sar with pad_crop_ratio set to +1.0. +

+
+
‘none, place’
+

Don’t scale the input. The input will be placed inside the output rectangle at +its natural size; which may result in additional padding or cropping. +

+
+
‘scale_down’
+

Scale the input down as much as needed to fit inside the output. Equivalent +to either contain or none, depending on whether the input is +larger than the output or not. +

+
+ +
+
fit_sense
+

When fit_mode is in use, this option controls how the fit strategy +is applied against the specified output resolution. Mutually exclusive with +force_original_aspect_ratio. Valid values are: +

+
+
‘target’
+

The computed output resolution is taken as the exact size of the output frame. +This is the default behavior. +

+
+
‘constraint’
+

The computed output resolution is a size reference against which the fit mode +is applied, enlarging or decreasing the true frame size as needed to fit the +content. +

+
+ +
+
fillcolor
+

Set the color used to fill the output area not covered by the output image, for +example as a result of normalize_sar. For the general syntax of this +option, check the (ffmpeg-utils)"Color" section in the ffmpeg-utils +manual. Defaults to black@0. +

+
+
corner_rounding
+

Render frames with rounded corners. The value, given as a float ranging from +0.0 to 1.0, indicates the relative degree of rounding, from fully +square to fully circular. In other words, it gives the radius divided by half +the smaller side length. Defaults to 0.0. +

+
+
lut
+

Specifies a custom LUT (in Adobe .cube format) to apply to the colors +as part of color conversion. The exact interpretation depends on the value +of lut_type. +

+
+
lut_type
+

Controls the interpretation of color values fed to and from the LUT +specified as lut. Valid values are: +

+
+
‘auto’
+

Chooses the interpretation of the LUT automatically from tagged +metadata, and otherwise falls back to ‘native’. (Default) +

+
+
‘native’
+

Applied to raw image contents in its native RGB colorspace (non-linear +light), before conversion to the output color space. +

+
+
‘normalized’
+

Applied to the normalized RGB image contents, in linear light, before +conversion to the output color space. +

+
+
‘conversion’
+

Fully replaces the conversion from the image color space to the output +color space. If such a LUT is present, it has the highest priority, and +overrides any ICC profiles, as well as options related to tone mapping +and output colorimetry (color_primaries, color_trc). +

+
+
+ +
+
extra_opts
+

Pass extra libplacebo internal configuration options. These can be specified +as a list of key=value pairs separated by ’:’. The following example +shows how to configure a custom filter kernel ("EWA LanczosSharp") and use it +to double the input image resolution: +

+
+
-vf "libplacebo=w=iw*2:h=ih*2:extra_opts='upscaler=custom\:upscaler_preset=ewa_lanczos\:upscaler_blur=0.9812505644269356'"
+
+ + +
+
shader_cache
+

File path of a cache directory that libplacebo will use to store and load +cached shader objects. This cache is not cleaned up automatically. If the +path does not end in a directory separator, the generated filenames will be +effectively prefixed by the last path component. All directories must already +exist. +

+
+
-vf "libplacebo=shader_cache=/tmp/pl-shader-"
+
+ +
+
colorspace
+
color_primaries
+
color_trc
+
range
+
chroma_location
+

Configure the colorspace that output frames will be delivered in. The default +value of auto outputs frames in the same format as the input frames, +leading to no change. For any other value, conversion will be performed. +

+

See the setparams filter for a list of possible values. +

+
+
alpha_mode
+

Choose the desired output alpha mode, when the output format has an alpha +channel. See the setparams filter for a list of possible values. +

+
+
apply_filmgrain
+

Apply film grain (e.g. AV1 or H.274) if present in source frames, and strip +it from the output. Enabled by default. +

+
+
apply_dolbyvision
+

Apply Dolby Vision RPU metadata if present in source frames, and strip it from +the output. Enabled by default. Note that Dolby Vision will always output +BT.2020+PQ, overriding the usual input frame metadata. These will also be +picked as the values of auto for the respective frame output options. +

+
+ +

In addition to the expression constants documented for the scale filter, +the crop_w, crop_h, crop_x, crop_y, +pos_w, pos_h, pos_x and pos_y options can +also contain the following constants: +

+
+
in_idx, idx
+

The (0-based) numeric index of the currently active input stream. +

+
crop_w, cw
+
crop_h, ch
+

The computed values of crop_w and crop_h. +

+
+
pos_w, pw
+
pos_h, ph
+

The computed values of pos_w and pos_h. +

+
+
in_t, t
+

The input frame timestamp, in seconds. NAN if input timestamp is unknown. +

+
+
out_t, ot
+

The input frame timestamp, in seconds. NAN if input timestamp is unknown. +

+
+
n
+

The input frame number, starting with 0. +

+
+ +

11.145.1.2 Scaling

+

The options in this section control how libplacebo performs upscaling and (if +necessary) downscaling. Note that libplacebo will always internally operate on +4:4:4 content, so any sub-sampled chroma formats such as yuv420p will +necessarily be upsampled and downsampled as part of the rendering process. That +means scaling might be in effect even if the source and destination resolution +are the same. +

+
upscaler
+
downscaler
+

Configure the filter kernel used for upscaling and downscaling. The respective +defaults are spline36 and mitchell. For a full list of possible +values, pass help to these options. The most important values are: +

+
‘none’
+

Forces the use of built-in GPU texture sampling (typically bilinear). Extremely +fast but poor quality, especially when downscaling. +

+
+
‘bilinear’
+

Bilinear interpolation. Can generally be done for free on GPUs, except when +doing so would lead to aliasing. Fast and low quality. +

+
+
‘nearest’
+

Nearest-neighbour interpolation. Sharp but highly aliasing. +

+
+
‘oversample’
+

Algorithm that looks visually similar to nearest-neighbour interpolation but +tries to preserve pixel aspect ratio. Good for pixel art, since it results in +minimal distortion of the artistic appearance. +

+
+
‘lanczos’
+

Standard sinc-sinc interpolation kernel. +

+
+
‘spline36’
+

Cubic spline approximation of lanczos. No difference in performance, but has +very slightly less ringing. +

+
+
‘ewa_lanczos’
+

Elliptically weighted average version of lanczos, based on a jinc-jinc kernel. +This is also popularly referred to as just "Jinc scaling". Slow but very high +quality. +

+
+
‘gaussian’
+

Gaussian kernel. Has certain ideal mathematical properties, but subjectively +very blurry. +

+
+
‘mitchell’
+

Cubic BC spline with parameters recommended by Mitchell and Netravali. Very +little ringing. +

+
+ +
+
frame_mixer
+

Controls the kernel used for mixing frames temporally. The default value is +none, which disables frame mixing. For a full list of possible values, +pass help to this option. The most important values are: +

+
‘none’
+

Disables frame mixing, giving a result equivalent to "nearest neighbour" +semantics. +

+
+
‘oversample’
+

Oversamples the input video to create a "Smooth Motion"-type effect: if an +output frame would exactly fall on the transition between two video frames, it +is blended according to the relative overlap. This is the recommended option +whenever preserving the original subjective appearance is desired. +

+
+
‘mitchell_clamp’
+

Larger filter kernel that smoothly interpolates multiple frames in a manner +designed to eliminate ringing and other artefacts as much as possible. This is +the recommended option wherever maximum visual smoothness is desired. +

+
+
‘linear’
+

Linear blend/fade between frames. Especially useful for constructing e.g. +slideshows. +

+
+ +
+
antiringing
+

Enables anti-ringing (for non-EWA filters). The value (between 0.0 and +1.0) configures the strength of the anti-ringing algorithm. May increase +aliasing if set too high. Disabled by default. +

+
+
sigmoid
+

Enable sigmoidal compression during upscaling. Reduces ringing slightly. +Enabled by default. +

+
+ +

11.145.1.3 Deinterlacing

+

Deinterlacing is automatically supported when frames are tagged as interlaced, +however frames are not deinterlaced unless a deinterlacing algorithm is chosen. +

+
deinterlace
+

The the deinterlacing algorithm to use. +

+
‘weave’
+

No deinterlacing, weave fields together into a single frame. This is the default. +

+
‘bob’
+

Naive bob deinterlacing, simply repeat each field line twice. +

+
‘yadif’
+

Yet another deinterlacing filter. See the yadif filter for more details. +

+
‘bwdif’
+

Bob weaver deinterlacing filter. See the bwdif filter for more details. +

+
+ +
+
skip_spatial_check
+

Skip the spatial deinterlacing check when using yadif deinterlacing. +

+
+
send_fields
+

Output a frame for each field, rather than for each frame. Note that this will +always double the tagged output frame rate, even if the input does not contain +any interlaced frames. Disabled by default. +

+
+ +

11.145.1.4 Debanding

+

Libplacebo comes with a built-in debanding filter that is good at counteracting +many common sources of banding and blocking. Turning this on is highly +recommended whenever quality is desired. +

+
deband
+

Enable (fast) debanding algorithm. Disabled by default. +

+
+
deband_iterations
+

Number of deband iterations of the debanding algorithm. Each iteration is +performed with progressively increased radius (and diminished threshold). +Recommended values are in the range 1 to 4. Defaults to 1. +

+
+
deband_threshold
+

Debanding filter strength. Higher numbers lead to more aggressive debanding. +Defaults to 4.0. +

+
+
deband_radius
+

Debanding filter radius. A higher radius is better for slow gradients, while +a lower radius is better for steep gradients. Defaults to 16.0. +

+
+
deband_grain
+

Amount of extra output grain to add. Helps hide imperfections. Defaults to +6.0. +

+
+ +

11.145.1.5 Color adjustment

+

A collection of subjective color controls. Not very rigorous, so the exact +effect will vary somewhat depending on the input primaries and colorspace. +

+
brightness
+

Brightness boost, between -1.0 and 1.0. Defaults to 0.0. +

+
+
contrast
+

Contrast gain, between 0.0 and 16.0. Defaults to 1.0. +

+
+
saturation
+

Saturation gain, between 0.0 and 16.0. Defaults to 1.0. +

+
+
hue
+

Hue shift in radians, between -3.14 and 3.14. Defaults to +0.0. This will rotate the UV subvector, defaulting to BT.709 +coefficients for RGB inputs. +

+
+
gamma
+

Gamma adjustment, between 0.0 and 16.0. Defaults to 1.0. +

+
+
temperature
+

Color temperature adjustment. Lower values make the output warmer/redder, +down to 1667, while higher values make the output cooler/bluer, up to +25000. Defaults to 6500 (neutral white). +

+
+
cones
+

Cone model to use for color blindness simulation. Accepts any combination of +l, m and s. Here are some examples: +

+
‘m’
+

Deuteranomaly / deuteranopia (affecting 3%-4% of the population) +

+
‘l’
+

Protanomaly / protanopia (affecting 1%-2% of the population) +

+
‘l+m’
+

Monochromacy (very rare) +

+
‘l+m+s’
+

Achromatopsy (complete loss of daytime vision, extremely rare) +

+
+ +
+
cone-strength
+

Gain factor for the cones specified by cones, between 0.0 and +10.0. A value of 1.0 results in no change to color vision. A +value of 0.0 (the default) simulates complete loss of those cones. Values +above 1.0 result in exaggerating the differences between cones, which +may help compensate for reduced color vision. +

+
+ +

11.145.1.6 Peak detection

+

To help deal with sources that only have static HDR10 metadata (or no tagging +whatsoever), libplacebo uses its own internal frame analysis compute shader to +analyze source frames and adapt the tone mapping function in realtime. If this +is too slow, or if exactly reproducible frame-perfect results are needed, it’s +recommended to turn this feature off. +

+
peak_detect
+

Enable HDR peak detection. Ignores static MaxCLL/MaxFALL values in favor of +dynamic detection from the input. Note that the detected values do not get +written back to the output frames, they merely guide the internal tone mapping +process. Enabled by default. +

+
+
smoothing_period
+

Peak detection smoothing period, between 0.0 and 1000.0. Higher +values result in peak detection becoming less responsive to changes in the +input. Defaults to 20.0. +

+
+
scene_threshold_low
+
scene_threshold_high
+

Lower and upper thresholds for scene change detection. Expressed in a +logarithmic scale between 0.0 and 100.0. Default to 1.0 +and 3.0, respectively. Setting either to a negative value disables +this functionality. +

+
+
percentile
+

Which percentile of the frame brightness histogram to use as the source peak +for tone-mapping. Defaults to 99.995, a fairly conservative value. +Setting this to 100.0 disables frame histogram measurement and instead +uses the true peak brightness for tone-mapping. +

+
+ +

11.145.1.7 Tone mapping

+

The options in this section control how libplacebo performs tone-mapping and +gamut-mapping when dealing with mismatches between wide-gamut or HDR content. +In general, libplacebo relies on accurate source tagging and mastering display +gamut information to produce the best results. +

+
gamut_mode
+

How to handle out-of-gamut colors that can occur as a result of colorimetric +gamut mapping. +

+
‘clip’
+

Do nothing, simply clip out-of-range colors to the RGB volume. Low quality but +extremely fast. +

+
‘perceptual’
+

Perceptually soft-clip colors to the gamut volume. This is the default. +

+
‘relative’
+

Relative colorimetric hard-clip. Similar to perceptual but without +the soft knee. +

+
‘saturation’
+

Saturation mapping, maps primaries directly to primaries in RGB space. +Not recommended except for artificial computer graphics for which a bright, +saturated display is desired. +

+
‘absolute’
+

Absolute colorimetric hard-clip. Performs no adjustment of the white point. +

+
‘desaturate’
+

Hard-desaturates out-of-gamut colors towards white, while preserving the +luminance. Has a tendency to distort the visual appearance of bright objects. +

+
‘darken’
+

Linearly reduces content brightness to preserves saturated details, followed by +clipping the remaining out-of-gamut colors. +

+
‘warn’
+

Highlight out-of-gamut pixels (by inverting/marking them). +

+
‘linear’
+

Linearly reduces chromaticity of the entire image to make it fit within the +target color volume. Be careful when using this on BT.2020 sources without +proper mastering metadata, as doing so will lead to excessive desaturation. +

+
+ +
+
tonemapping
+

Tone-mapping algorithm to use. Available values are: +

+
‘auto’
+

Automatic selection based on internal heuristics. This is the default. +

+
‘clip’
+

Performs no tone-mapping, just clips out-of-range colors. Retains perfect color +accuracy for in-range colors but completely destroys out-of-range information. +Does not perform any black point adaptation. Not configurable. +

+
‘st2094-40’
+

EETF from SMPTE ST 2094-40 Annex B, which applies the Bezier curves from HDR10+ +dynamic metadata based on Bezier curves to perform tone-mapping. The OOTF used +is adjusted based on the ratio between the targeted and actual display peak +luminances. +

+
‘st2094-10’
+

EETF from SMPTE ST 2094-10 Annex B.2, which takes into account the input signal +average luminance in addition to the maximum/minimum. The configurable contrast +parameter influences the slope of the linear output segment, defaulting to +1.0 for no increase/decrease in contrast. Note that this does not +currently include the subjective gain/offset/gamma controls defined in Annex +B.3. +

+
‘bt.2390’
+

EETF from the ITU-R Report BT.2390, a hermite spline roll-off with linear +segment. The knee point offset is configurable. Note that this parameter +defaults to 1.0, rather than the value of 0.5 from the ITU-R +spec. +

+
‘bt.2446a’
+

EETF from ITU-R Report BT.2446, method A. Designed for well-mastered HDR +sources. Can be used for both forward and inverse tone mapping. Not +configurable. +

+
‘spline’
+

Simple spline consisting of two polynomials, joined by a single pivot point. +The parameter gives the pivot point (in PQ space), defaulting to 0.30. +Can be used for both forward and inverse tone mapping. +

+
‘reinhard’
+

Simple non-linear, global tone mapping algorithm. The parameter specifies the +local contrast coefficient at the display peak. Essentially, a parameter of +0.5 implies that the reference white will be about half as bright as +when clipping. Defaults to 0.5, which results in the simplest +formulation of this function. +

+
‘mobius’
+

Generalization of the reinhard tone mapping algorithm to support an additional +linear slope near black. The tone mapping parameter indicates the trade-off +between the linear section and the non-linear section. Essentially, for a given +parameter x, every color value below x will be mapped linearly, +while higher values get non-linearly tone-mapped. Values near 1.0 make +this curve behave like clip, while values near 0.0 make this +curve behave like reinhard. The default value is 0.3, which +provides a good balance between colorimetric accuracy and preserving +out-of-gamut details. +

+
‘hable’
+

Piece-wise, filmic tone-mapping algorithm developed by John Hable for use in +Uncharted 2, inspired by a similar tone-mapping algorithm used by Kodak. +Popularized by its use in video games with HDR rendering. Preserves both dark +and bright details very well, but comes with the drawback of changing the +average brightness quite significantly. This is sort of similar to +reinhard with parameter 0.24. +

+
‘gamma’
+

Fits a gamma (power) function to transfer between the source and target color +spaces, effectively resulting in a perceptual hard-knee joining two roughly +linear sections. This preserves details at all scales fairly accurately, but +can result in an image with a muted or dull appearance. The parameter is used +as the cutoff point, defaulting to 0.5. +

+
‘linear’
+

Linearly stretches the input range to the output range, in PQ space. This will +preserve all details accurately, but results in a significantly different +average brightness. Can be used for inverse tone-mapping in addition to regular +tone-mapping. The parameter can be used as an additional linear gain +coefficient (defaulting to 1.0). +

+
+ +
+
tonemapping_param
+

For tunable tone mapping functions, this parameter can be used to fine-tune the +curve behavior. Refer to the documentation of tonemapping. The default +value of 0.0 is replaced by the curve’s preferred default setting. +

+
+
inverse_tonemapping
+

If enabled, this filter will also attempt stretching SDR signals to fill HDR +output color volumes. Disabled by default. +

+
+
tonemapping_lut_size
+

Size of the tone-mapping LUT, between 2 and 1024. Defaults to +256. Note that this figure is squared when combined with +peak_detect. +

+
+
contrast_recovery
+

Contrast recovery strength. If set to a value above 0.0, the source +image will be divided into high-frequency and low-frequency components, and a +portion of the high-frequency image is added back onto the tone-mapped output. +May cause excessive ringing artifacts for some HDR sources, but can improve the +subjective sharpness and detail left over in the image after tone-mapping. +Defaults to 0.30. +

+
+
contrast_smoothness
+

Contrast recovery lowpass kernel size. Defaults to 3.5. Increasing or +decreasing this will affect the visual appearance substantially. Has no effect +when contrast_recovery is disabled. +

+
+ +

11.145.1.8 Dithering

+

By default, libplacebo will dither whenever necessary, which includes rendering +to any integer format below 16-bit precision. It’s recommended to always leave +this on, since not doing so may result in visible banding in the output, even +if the debanding filter is enabled. If maximum performance is needed, +use ordered_fixed instead of disabling dithering. +

+
dithering
+

Dithering method to use. Accepts the following values: +

+
‘none’
+

Disables dithering completely. May result in visible banding. +

+
‘blue’
+

Dither with pseudo-blue noise. This is the default. +

+
‘ordered’
+

Tunable ordered dither pattern. +

+
‘ordered_fixed’
+

Faster ordered dither with a fixed size of 6. Texture-less. +

+
‘white’
+

Dither with white noise. Texture-less. +

+
+ +
+
dither_lut_size
+

Dither LUT size, as log base2 between 1 and 8. Defaults to +6, corresponding to a LUT size of 64x64. +

+
+
dither_temporal
+

Enables temporal dithering. Disabled by default. +

+
+ +

11.145.1.9 Custom shaders

+

libplacebo supports a number of custom shaders based on the mpv .hook GLSL +syntax. A collection of such shaders can be found here: +https://github.com/mpv-player/mpv/wiki/User-Scripts#user-shaders +

+

A full description of the mpv shader format is beyond the scope of this +section, but a summary can be found here: +https://mpv.io/manual/master/#options-glsl-shader +

+
custom_shader_path
+

Specifies a path to a custom shader file to load at runtime. +

+
+
custom_shader_bin
+

Specifies a complete custom shader as a raw string. +

+
+ +

11.145.1.10 Debugging / performance

+

All of the options in this section default off. They may be of assistance when +attempting to squeeze the maximum performance at the cost of quality. +

+
skip_aa
+

Disable anti-aliasing when downscaling. +

+
+
disable_linear
+

Disable linear light scaling. +

+
+
disable_builtin
+

Disable built-in GPU sampling (forces LUT). +

+
+
disable_fbos
+

Forcibly disable FBOs, resulting in loss of almost all functionality, but +offering the maximum possible speed. +

+
+ +

11.145.2 Commands

+

This filter supports almost all of the above options as commands. +

+

11.145.3 Examples

+
    +
  • Tone-map input to standard gamut BT.709 output: +
    +
    libplacebo=colorspace=bt709:color_primaries=bt709:color_trc=bt709:range=tv
    +
    + +
  • Rescale input to fit into standard 1080p, with high quality scaling: +
    +
    libplacebo=w=1920:h=1080:force_original_aspect_ratio=decrease:normalize_sar=true:upscaler=ewa_lanczos:downscaler=ewa_lanczos
    +
    + +
  • Interpolate low FPS / VFR input to smoothed constant 60 fps output: +
    +
    libplacebo=fps=60:frame_mixer=mitchell_clamp
    +
    + +
  • Convert input to standard sRGB JPEG: +
    +
    libplacebo=format=yuv420p:colorspace=bt470bg:color_primaries=bt709:color_trc=iec61966-2-1:range=pc
    +
    + +
  • Use higher quality debanding settings: +
    +
    libplacebo=deband=true:deband_iterations=3:deband_radius=8:deband_threshold=6
    +
    + +
  • Run this filter on the CPU, on systems with Mesa installed (and with the most +expensive options disabled): +
    +
    ffmpeg ... -init_hw_device vulkan:llvmpipe ... -vf libplacebo=upscaler=none:downscaler=none:peak_detect=false
    +
    + +
  • Suppress CPU-based AV1/H.274 film grain application in the decoder, in favor of +doing it with this filter. Note that this is only a gain if the frames are +either already on the GPU, or if you’re using libplacebo for other purposes, +since otherwise the VRAM roundtrip will more than offset any expected speedup. +
    +
    ffmpeg -export_side_data +film_grain ... -vf libplacebo=apply_filmgrain=true
    +
    + +
  • Interop with VAAPI hwdec to avoid round-tripping through RAM: +
    +
    ffmpeg -init_hw_device vulkan -hwaccel vaapi -hwaccel_output_format vaapi ... -vf libplacebo
    +
    +
+ +

11.146 libvmaf

+ +

Calculate the VMAF (Video Multi-Method Assessment Fusion) score for a +reference/distorted pair of input videos. +

+

The first input is the distorted video, and the second input is the reference video. +

+

The obtained VMAF score is printed through the logging system. +

+

It requires Netflix’s vmaf library (libvmaf) as a pre-requisite. +After installing the library it can be enabled using: +./configure --enable-libvmaf. +

+

The filter has following options: +

+
+
model
+

A ‘|‘ delimited list of vmaf models. Each model can be configured with a number of parameters. +Default value: "version=vmaf_v0.6.1" +

+
+
feature
+

A ‘|‘ delimited list of features. Each feature can be configured with a number of parameters. +

+
+
log_path
+

Set the file path to be used to store log files. +

+
+
log_fmt
+

Set the format of the log file (xml, json, csv, or sub). +

+
+
pool
+

Set the pool method to be used for computing vmaf. +Options are min, harmonic_mean or mean (default). +

+
+
n_threads
+

Set number of threads to be used when initializing libvmaf. +Default value: 0, no threads. +

+
+
n_subsample
+

Set frame subsampling interval to be used. +

+
+ +

This filter also supports the framesync options. +

+

11.146.1 Examples

+
    +
  • In the examples below, a distorted video distorted.mpg is +compared with a reference file reference.mpg. + +
  • Basic usage: +
    +
    ffmpeg -i distorted.mpg -i reference.mpg -lavfi libvmaf=log_path=output.xml -f null -
    +
    + +
  • Example with multiple models: +
    +
    ffmpeg -i distorted.mpg -i reference.mpg -lavfi libvmaf='model=version=vmaf_v0.6.1\\:name=vmaf|version=vmaf_v0.6.1neg\\:name=vmaf_neg' -f null -
    +
    + +
  • Example with multiple additional features: +
    +
    ffmpeg -i distorted.mpg -i reference.mpg -lavfi libvmaf='feature=name=psnr|name=ciede' -f null -
    +
    + +
  • Example with options and different containers: +
    +
    ffmpeg -i distorted.mpg -i reference.mkv -lavfi "[0:v]settb=AVTB,setpts=PTS-STARTPTS[main];[1:v]settb=AVTB,setpts=PTS-STARTPTS[ref];[main][ref]libvmaf=log_fmt=json:log_path=output.json" -f null -
    +
    +
+ +

11.147 libvmaf_cuda

+ +

This is the CUDA variant of the libvmaf filter. It only accepts CUDA frames. +

+

It requires Netflix’s vmaf library (libvmaf) as a pre-requisite. +After installing the library it can be enabled using: +./configure --enable-nonfree --enable-ffnvcodec --enable-libvmaf. +

+

11.147.1 Examples

+
    +
  • Basic usage showing CUVID hardware decoding and CUDA scaling with scale_cuda: +
    +
    ffmpeg \
    +    -hwaccel cuda -hwaccel_output_format cuda -codec:v av1_cuvid -i dis.obu \
    +    -hwaccel cuda -hwaccel_output_format cuda -codec:v av1_cuvid -i ref.obu \
    +    -filter_complex "
    +        [0:v]scale_cuda=format=yuv420p[dis]; \
    +        [1:v]scale_cuda=format=yuv420p[ref]; \
    +        [dis][ref]libvmaf_cuda=log_fmt=json:log_path=output.json
    +    " \
    +    -f null -
    +
    +
+ +

11.148 limitdiff

+

Apply limited difference filter using second and optionally third video stream. +

+

The filter accepts the following options: +

+
+
threshold
+

Set the threshold to use when allowing certain differences between video streams. +Any absolute difference value lower or exact than this threshold will pick pixel components from +first video stream. +

+
+
elasticity
+

Set the elasticity of soft thresholding when processing video streams. +This value multiplied with first one sets second threshold. +Any absolute difference value greater or exact than second threshold will pick pixel components +from second video stream. For values between those two threshold +linear interpolation between first and second video stream will be used. +

+
+
reference
+

Enable the reference (third) video stream processing. By default is disabled. +If set, this video stream will be used for calculating absolute difference with first video +stream. +

+
+
planes
+

Specify which planes will be processed. Defaults to all available. +

+
+ +

11.148.1 Commands

+ +

This filter supports the all above options as commands except option ‘reference’. +

+

11.149 limiter

+ +

Limits the pixel components values to the specified range [min, max]. +

+

The filter accepts the following options: +

+
+
min
+

Lower bound. Defaults to the lowest allowed value for the input. +

+
+
max
+

Upper bound. Defaults to the highest allowed value for the input. +

+
+
planes
+

Specify which planes will be processed. Defaults to all available. +

+
+ +

11.149.1 Commands

+ +

This filter supports the all above options as commands. +

+

11.150 loop

+ +

Loop video frames. +

+

The filter accepts the following options: +

+
+
loop
+

Set the number of loops. Setting this value to -1 will result in infinite loops. +Default is 0. +

+
+
size
+

Set maximal size in number of frames. Default is 0. +

+
+
start
+

Set first frame of loop. Default is 0. +

+
+
time
+

Set the time of loop start in seconds. +Only used if option named start is set to -1. +

+
+ +

11.150.1 Examples

+ +
    +
  • Loop single first frame infinitely: +
    +
    loop=loop=-1:size=1:start=0
    +
    + +
  • Loop single first frame 10 times: +
    +
    loop=loop=10:size=1:start=0
    +
    + +
  • Loop 10 first frames 5 times: +
    +
    loop=loop=5:size=10:start=0
    +
    +
+ +

11.151 lut1d

+ +

Apply a 1D LUT to an input video. +

+

The filter accepts the following options: +

+
+
file
+

Set the 1D LUT file name. +

+

Currently supported formats: +

+
‘cube’
+

Iridas +

+
‘csp’
+

cineSpace +

+
+ +
+
interp
+

Select interpolation mode. +

+

Available values are: +

+
+
‘nearest’
+

Use values from the nearest defined point. +

+
‘linear’
+

Interpolate values using the linear interpolation. +

+
‘cosine’
+

Interpolate values using the cosine interpolation. +

+
‘cubic’
+

Interpolate values using the cubic interpolation. +

+
‘spline’
+

Interpolate values using the spline interpolation. +

+
+
+
+ +

11.151.1 Commands

+ +

This filter supports the all above options as commands. +

+

11.152 lut3d

+ +

Apply a 3D LUT to an input video. +

+

The filter accepts the following options: +

+
+
file
+

Set the 3D LUT file name. +

+

Currently supported formats: +

+
‘3dl’
+

AfterEffects +

+
‘cube’
+

Iridas +

+
‘dat’
+

DaVinci +

+
‘m3d’
+

Pandora +

+
‘csp’
+

cineSpace +

+
+
+
interp
+

Select interpolation mode. +

+

Available values are: +

+
+
‘nearest’
+

Use values from the nearest defined point. +

+
‘trilinear’
+

Interpolate values using the 8 points defining a cube. +

+
‘tetrahedral’
+

Interpolate values using a tetrahedron. +

+
‘pyramid’
+

Interpolate values using a pyramid. +

+
‘prism’
+

Interpolate values using a prism. +

+
+
+
+ +

11.152.1 Commands

+ +

This filter supports the interp option as commands. +

+

11.153 lumakey

+ +

Turn certain luma values into transparency. +

+

The filter accepts the following options: +

+
+
threshold
+

Set the luma which will be used as base for transparency. +Default value is 0. +

+
+
tolerance
+

Set the range of luma values to be keyed out. +Default value is 0.01. +

+
+
softness
+

Set the range of softness. Default value is 0. +Use this to control gradual transition from zero to full transparency. +

+
+ +

11.153.1 Commands

+

This filter supports same commands as options. +The command accepts the same syntax of the corresponding option. +

+

If the specified expression is not valid, it is kept at its current +value. +

+

11.154 lut, lutrgb, lutyuv

+ +

Compute a look-up table for binding each pixel component input value +to an output value, and apply it to the input video. +

+

lutyuv applies a lookup table to a YUV input video, lutrgb +to an RGB input video. +

+

These filters accept the following parameters: +

+
c0
+

set first pixel component expression +

+
c1
+

set second pixel component expression +

+
c2
+

set third pixel component expression +

+
c3
+

set fourth pixel component expression, corresponds to the alpha component +

+
+
r
+

set red component expression +

+
g
+

set green component expression +

+
b
+

set blue component expression +

+
a
+

alpha component expression +

+
+
y
+

set Y/luma component expression +

+
u
+

set U/Cb component expression +

+
v
+

set V/Cr component expression +

+
+ +

Each of them specifies the expression to use for computing the lookup table for +the corresponding pixel component values. +

+

The exact component associated to each of the c* options depends on the +format in input. +

+

The lut filter requires either YUV or RGB pixel formats in input, +lutrgb requires RGB pixel formats in input, and lutyuv requires YUV. +

+

The expressions can contain the following constants and functions: +

+
+
w
+
h
+

The input width and height. +

+
+
val
+

The input value for the pixel component. +

+
+
clipval
+

The input value, clipped to the minval-maxval range. +

+
+
maxval
+

The maximum value for the pixel component. +

+
+
minval
+

The minimum value for the pixel component. +

+
+
negval
+

The negated value for the pixel component value, clipped to the +minval-maxval range; it corresponds to the expression +"maxval-clipval+minval". +

+
+
clip(val)
+

The computed value in val, clipped to the +minval-maxval range. +

+
+
gammaval(gamma)
+

The computed gamma correction value of the pixel component value, +clipped to the minval-maxval range. It corresponds to the +expression +"pow((clipval-minval)/(maxval-minval)\,gamma)*(maxval-minval)+minval" +

+
+
+ +

All expressions default to "clipval". +

+

11.154.1 Commands

+ +

This filter supports same commands as options. +

+

11.154.2 Examples

+ +
    +
  • Negate input video: +
    +
    lutrgb="r=maxval+minval-val:g=maxval+minval-val:b=maxval+minval-val"
    +lutyuv="y=maxval+minval-val:u=maxval+minval-val:v=maxval+minval-val"
    +
    + +

    The above is the same as: +

    +
    lutrgb="r=negval:g=negval:b=negval"
    +lutyuv="y=negval:u=negval:v=negval"
    +
    + +
  • Negate luma: +
    +
    lutyuv=y=negval
    +
    + +
  • Remove chroma components, turning the video into a graytone image: +
    +
    lutyuv="u=128:v=128"
    +
    + +
  • Apply a luma burning effect: +
    +
    lutyuv="y=2*val"
    +
    + +
  • Remove green and blue components: +
    +
    lutrgb="g=0:b=0"
    +
    + +
  • Set a constant alpha channel value on input: +
    +
    format=rgba,lutrgb=a="maxval-minval/2"
    +
    + +
  • Correct luma gamma by a factor of 0.5: +
    +
    lutyuv=y=gammaval(0.5)
    +
    + +
  • Discard least significant bits of luma: +
    +
    lutyuv=y='bitand(val, 128+64+32)'
    +
    + +
  • Technicolor like effect: +
    +
    lutyuv=u='(val-maxval/2)*2+maxval/2':v='(val-maxval/2)*2+maxval/2'
    +
    +
+ +

11.155 lut2, tlut2

+ +

The lut2 filter takes two input streams and outputs one +stream. +

+

The tlut2 (time lut2) filter takes two consecutive frames +from one single stream. +

+

This filter accepts the following parameters: +

+
c0
+

set first pixel component expression +

+
c1
+

set second pixel component expression +

+
c2
+

set third pixel component expression +

+
c3
+

set fourth pixel component expression, corresponds to the alpha component +

+
+
d
+

set output bit depth, only available for lut2 filter. By default is 0, +which means bit depth is automatically picked from first input format. +

+
+ +

The lut2 filter also supports the framesync options. +

+

Each of them specifies the expression to use for computing the lookup table for +the corresponding pixel component values. +

+

The exact component associated to each of the c* options depends on the +format in inputs. +

+

The expressions can contain the following constants: +

+
+
w
+
h
+

The input width and height. +

+
+
x
+

The first input value for the pixel component. +

+
+
y
+

The second input value for the pixel component. +

+
+
bdx
+

The first input video bit depth. +

+
+
bdy
+

The second input video bit depth. +

+
+ +

All expressions default to "x". +

+

11.155.1 Commands

+ +

This filter supports the all above options as commands except option d. +

+

11.155.2 Examples

+ +
    +
  • Highlight differences between two RGB video streams: +
    +
    lut2='ifnot(x-y,0,pow(2,bdx)-1):ifnot(x-y,0,pow(2,bdx)-1):ifnot(x-y,0,pow(2,bdx)-1)'
    +
    + +
  • Highlight differences between two YUV video streams: +
    +
    lut2='ifnot(x-y,0,pow(2,bdx)-1):ifnot(x-y,pow(2,bdx-1),pow(2,bdx)-1):ifnot(x-y,pow(2,bdx-1),pow(2,bdx)-1)'
    +
    + +
  • Show max difference between two video streams: +
    +
    lut2='if(lt(x,y),0,if(gt(x,y),pow(2,bdx)-1,pow(2,bdx-1))):if(lt(x,y),0,if(gt(x,y),pow(2,bdx)-1,pow(2,bdx-1))):if(lt(x,y),0,if(gt(x,y),pow(2,bdx)-1,pow(2,bdx-1)))'
    +
    +
+ +

11.156 maskedclamp

+ +

Clamp the first input stream with the second input and third input stream. +

+

Returns the value of first stream to be between second input +stream - undershoot and third input stream + overshoot. +

+

This filter accepts the following options: +

+
undershoot
+

Default value is 0. +

+
+
overshoot
+

Default value is 0. +

+
+
planes
+

Set which planes will be processed as bitmap, unprocessed planes will be +copied from first stream. +By default value 0xf, all planes will be processed. +

+
+ +

11.156.1 Commands

+ +

This filter supports the all above options as commands. +

+

11.157 maskedmax

+ +

Merge the second and third input stream into output stream using absolute differences +between second input stream and first input stream and absolute difference between +third input stream and first input stream. The picked value will be from second input +stream if second absolute difference is greater than first one or from third input stream +otherwise. +

+

This filter accepts the following options: +

+
planes
+

Set which planes will be processed as bitmap, unprocessed planes will be +copied from first stream. +By default value 0xf, all planes will be processed. +

+
+ +

11.157.1 Commands

+ +

This filter supports the all above options as commands. +

+

11.158 maskedmerge

+ +

Merge the first input stream with the second input stream using per pixel +weights in the third input stream. +

+

A value of 0 in the third stream pixel component means that pixel component +from first stream is returned unchanged, while maximum value (eg. 255 for +8-bit videos) means that pixel component from second stream is returned +unchanged. Intermediate values define the amount of merging between both +input stream’s pixel components. +

+

This filter accepts the following options: +

+
planes
+

Set which planes will be processed as bitmap, unprocessed planes will be +copied from first stream. +By default value 0xf, all planes will be processed. +

+
+ +

11.158.1 Commands

+ +

This filter supports the all above options as commands. +

+

11.159 maskedmin

+ +

Merge the second and third input stream into output stream using absolute differences +between second input stream and first input stream and absolute difference between +third input stream and first input stream. The picked value will be from second input +stream if second absolute difference is less than first one or from third input stream +otherwise. +

+

This filter accepts the following options: +

+
planes
+

Set which planes will be processed as bitmap, unprocessed planes will be +copied from first stream. +By default value 0xf, all planes will be processed. +

+
+ +

11.159.1 Commands

+ +

This filter supports the all above options as commands. +

+

11.160 maskedthreshold

+

Pick pixels comparing absolute difference of two video streams with fixed +threshold. +

+

If absolute difference between pixel component of first and second video +stream is equal or lower than user supplied threshold than pixel component +from first video stream is picked, otherwise pixel component from second +video stream is picked. +

+

This filter accepts the following options: +

+
threshold
+

Set threshold used when picking pixels from absolute difference from two input +video streams. +

+
+
planes
+

Set which planes will be processed as bitmap, unprocessed planes will be +copied from second stream. +By default value 0xf, all planes will be processed. +

+
+
mode
+

Set mode of filter operation. Can be abs or diff. +Default is abs. +

+
+ +

11.160.1 Commands

+ +

This filter supports the all above options as commands. +

+

11.161 maskfun

+

Create mask from input video. +

+

For example it is useful to create motion masks after tblend filter. +

+

This filter accepts the following options: +

+
+
low
+

Set low threshold. Any pixel component lower or exact than this value will be set to 0. +

+
+
high
+

Set high threshold. Any pixel component higher than this value will be set to max value +allowed for current pixel format. +

+
+
planes
+

Set planes to filter, by default all available planes are filtered. +

+
+
fill
+

Fill all frame pixels with this value. +

+
+
sum
+

Set max average pixel value for frame. If sum of all pixel components is higher that this +average, output frame will be completely filled with value set by fill option. +Typically useful for scene changes when used in combination with tblend filter. +

+
+ +

11.161.1 Commands

+ +

This filter supports the all above options as commands. +

+

11.162 mcdeint

+ +

Apply motion-compensation deinterlacing. +

+

It needs one field per frame as input and must thus be used together +with yadif=1/3 or equivalent. +

+

This filter accepts the following options: +

+
mode
+

Set the deinterlacing mode. +

+

It accepts one of the following values: +

+
‘fast’
+
‘medium’
+
‘slow’
+

use iterative motion estimation +

+
‘extra_slow’
+

like ‘slow’, but use multiple reference frames. +

+
+

Default value is ‘fast’. +

+
+
parity
+

Set the picture field parity assumed for the input video. It must be +one of the following values: +

+
+
‘0, tff’
+

assume top field first +

+
‘1, bff’
+

assume bottom field first +

+
+ +

Default value is ‘bff’. +

+
+
qp
+

Set per-block quantization parameter (QP) used by the internal +encoder. +

+

Higher values should result in a smoother motion vector field but less +optimal individual vectors. Default value is 1. +

+
+ +

11.163 median

+ +

Pick median pixel from certain rectangle defined by radius. +

+

This filter accepts the following options: +

+
+
radius
+

Set horizontal radius size. Default value is 1. +Allowed range is integer from 1 to 127. +

+
+
planes
+

Set which planes to process. Default is 15, which is all available planes. +

+
+
radiusV
+

Set vertical radius size. Default value is 0. +Allowed range is integer from 0 to 127. +If it is 0, value will be picked from horizontal radius option. +

+
+
percentile
+

Set median percentile. Default value is 0.5. +Default value of 0.5 will pick always median values, while 0 will pick +minimum values, and 1 maximum values. +

+
+ +

11.163.1 Commands

+

This filter supports same commands as options. +The command accepts the same syntax of the corresponding option. +

+

If the specified expression is not valid, it is kept at its current +value. +

+

11.164 mergeplanes

+ +

Merge color channel components from several video streams. +

+

The filter accepts up to 4 input streams, and merge selected input +planes to the output video. +

+

This filter accepts the following options: +

+
mapping
+

Set input to output plane mapping. Default is 0. +

+

The mappings is specified as a bitmap. It should be specified as a +hexadecimal number in the form 0xAa[Bb[Cc[Dd]]]. ’Aa’ describes the +mapping for the first plane of the output stream. ’A’ sets the number of +the input stream to use (from 0 to 3), and ’a’ the plane number of the +corresponding input to use (from 0 to 3). The rest of the mappings is +similar, ’Bb’ describes the mapping for the output stream second +plane, ’Cc’ describes the mapping for the output stream third plane and +’Dd’ describes the mapping for the output stream fourth plane. +

+
+
format
+

Set output pixel format. Default is yuva444p. +

+
+
map0s
+
map1s
+
map2s
+
map3s
+

Set input to output stream mapping for output Nth plane. Default is 0. +

+
+
map0p
+
map1p
+
map2p
+
map3p
+

Set input to output plane mapping for output Nth plane. Default is 0. +

+
+ +

11.164.1 Examples

+ +
    +
  • Merge three gray video streams of same width and height into single video stream: +
    +
    [a0][a1][a2]mergeplanes=0x001020:yuv444p
    +
    + +
  • Merge 1st yuv444p stream and 2nd gray video stream into yuva444p video stream: +
    +
    [a0][a1]mergeplanes=0x00010210:yuva444p
    +
    + +
  • Swap Y and A plane in yuva444p stream: +
    +
    format=yuva444p,mergeplanes=0x03010200:yuva444p
    +
    + +
  • Swap U and V plane in yuv420p stream: +
    +
    format=yuv420p,mergeplanes=0x000201:yuv420p
    +
    + +
  • Cast a rgb24 clip to yuv444p: +
    +
    format=rgb24,mergeplanes=0x000102:yuv444p
    +
    +
+ +

11.165 mestimate

+ +

Estimate and export motion vectors using block matching algorithms. +Motion vectors are stored in frame side data to be used by other filters. +

+

This filter accepts the following options: +

+
method
+

Specify the motion estimation method. Accepts one of the following values: +

+
+
‘esa’
+

Exhaustive search algorithm. +

+
‘tss’
+

Three step search algorithm. +

+
‘tdls’
+

Two dimensional logarithmic search algorithm. +

+
‘ntss’
+

New three step search algorithm. +

+
‘fss’
+

Four step search algorithm. +

+
‘ds’
+

Diamond search algorithm. +

+
‘hexbs’
+

Hexagon-based search algorithm. +

+
‘epzs’
+

Enhanced predictive zonal search algorithm. +

+
‘umh’
+

Uneven multi-hexagon search algorithm. +

+
+

Default value is ‘esa’. +

+
+
mb_size
+

Macroblock size. Default 16. +

+
+
search_param
+

Search parameter. Default 7. +

+
+ +

11.166 mestimate_d3d12

+ +

Estimate and export motion vectors using D3D12 hardware-accelerated motion estimation. +This filter uses GPU hardware motion estimation capabilities available in +DirectX 12 Video APIs to achieve significant performance improvements +over the software-based mestimate filter. +

+

Motion vectors are stored in frame side data to be used by other filters. +

+

This filter requires the input to be in d3d12 hardware pixel format. +The filter uses quarter-pixel precision for motion vector estimation. +

+

This filter accepts the following options: +

+
mb_size
+

Macroblock size. Only 8 and 16 are supported. +Default 16. +

+
+ +

11.166.1 Examples

+ +

Estimate motion vectors using D3D12 hardware acceleration with 16x16 blocks, +and visualize them: +

+
ffmpeg -hwaccel d3d12va -hwaccel_output_format d3d12 -i input.mp4 \
+        -vf mestimate_d3d12=mb_size=16,hwdownload,format=nv12,codecview=mv=pf \
+        -c:v libx264 -preset fast -b:v 5M output.mp4
+
+ +

11.167 midequalizer

+ +

Apply Midway Image Equalization effect using two video streams. +

+

Midway Image Equalization adjusts a pair of images to have the same +histogram, while maintaining their dynamics as much as possible. It’s +useful for e.g. matching exposures from a pair of stereo cameras. +

+

This filter has two inputs and one output, which must be of same pixel format, but +may be of different sizes. The output of filter is first input adjusted with +midway histogram of both inputs. +

+

This filter accepts the following option: +

+
+
planes
+

Set which planes to process. Default is 15, which is all available planes. +

+
+ +

11.168 minterpolate

+ +

Convert the video to specified frame rate using motion interpolation. +

+

This filter accepts the following options: +

+
fps
+

Specify the output frame rate. This can be rational e.g. 60000/1001. Frames are dropped if fps is lower than source fps. Default 60. +

+
+
mi_mode
+

Motion interpolation mode. Following values are accepted: +

+
‘dup’
+

Duplicate previous or next frame for interpolating new ones. +

+
‘blend’
+

Blend source frames. Interpolated frame is mean of previous and next frames. +

+
‘mci’
+

Motion compensated interpolation. Following options are effective when this mode is selected: +

+
+
‘mc_mode’
+

Motion compensation mode. Following values are accepted: +

+
‘obmc’
+

Overlapped block motion compensation. +

+
‘aobmc’
+

Adaptive overlapped block motion compensation. Window weighting coefficients are controlled adaptively according to the reliabilities of the neighboring motion vectors to reduce oversmoothing. +

+
+

Default mode is ‘obmc’. +

+
+
‘me_mode’
+

Motion estimation mode. Following values are accepted: +

+
‘bidir’
+

Bidirectional motion estimation. Motion vectors are estimated for each source frame in both forward and backward directions. +

+
‘bilat’
+

Bilateral motion estimation. Motion vectors are estimated directly for interpolated frame. +

+
+

Default mode is ‘bilat’. +

+
+
‘me’
+

The algorithm to be used for motion estimation. Following values are accepted: +

+
‘esa’
+

Exhaustive search algorithm. +

+
‘tss’
+

Three step search algorithm. +

+
‘tdls’
+

Two dimensional logarithmic search algorithm. +

+
‘ntss’
+

New three step search algorithm. +

+
‘fss’
+

Four step search algorithm. +

+
‘ds’
+

Diamond search algorithm. +

+
‘hexbs’
+

Hexagon-based search algorithm. +

+
‘epzs’
+

Enhanced predictive zonal search algorithm. +

+
‘umh’
+

Uneven multi-hexagon search algorithm. +

+
+

Default algorithm is ‘epzs’. +

+
+
‘mb_size’
+

Macroblock size. Default 16. +

+
+
‘search_param’
+

Motion estimation search parameter. Default 32. +

+
+
‘vsbmc’
+

Enable variable-size block motion compensation. Motion estimation is applied with smaller block sizes at object boundaries in order to make them less blurry. Default is 0 (disabled). +

+
+
+
+ +
+
scd
+

Scene change detection method. Scene change leads motion vectors to be in random direction. Scene change detection replace interpolated frames by duplicate ones. May not be needed for other modes. Following values are accepted: +

+
‘none’
+

Disable scene change detection. +

+
‘fdiff’
+

Frame difference. Corresponding pixel values are compared and if it satisfies scd_threshold scene change is detected. +

+
+

Default method is ‘fdiff’. +

+
+
scd_threshold
+

Scene change detection threshold. Default is 10.. +

+
+ +

11.169 mix

+ +

Mix several video input streams into one video stream. +

+

A description of the accepted options follows. +

+
+
inputs
+

The number of inputs. If unspecified, it defaults to 2. +

+
+
weights
+

Specify weight of each input video stream as sequence. +Each weight is separated by space. If number of weights +is smaller than number of frames last specified +weight will be used for all remaining unset weights. +

+
+
scale
+

Specify scale, if it is set it will be multiplied with sum +of each weight multiplied with pixel values to give final destination +pixel value. By default scale is auto scaled to sum of weights. +

+
+
planes
+

Set which planes to filter. Default is all. Allowed range is from 0 to 15. +

+
+
duration
+

Specify how end of stream is determined. +

+
‘longest’
+

The duration of the longest input. (default) +

+
+
‘shortest’
+

The duration of the shortest input. +

+
+
‘first’
+

The duration of the first input. +

+
+
+
+ +

11.169.1 Commands

+ +

This filter supports the following commands: +

+
weights
+
scale
+
planes
+

Syntax is same as option with same name. +

+
+ +

11.170 monochrome

+

Convert video to gray using custom color filter. +

+

A description of the accepted options follows. +

+
+
cb
+

Set the chroma blue spot. Allowed range is from -1 to 1. +Default value is 0. +

+
+
cr
+

Set the chroma red spot. Allowed range is from -1 to 1. +Default value is 0. +

+
+
size
+

Set the color filter size. Allowed range is from .1 to 10. +Default value is 1. +

+
+
high
+

Set the highlights strength. Allowed range is from 0 to 1. +Default value is 0. +

+
+ +

11.170.1 Commands

+ +

This filter supports the all above options as commands. +

+

11.171 morpho

+ +

This filter allows to apply main morphological grayscale transforms, +erode and dilate with arbitrary structures set in second input stream. +

+

Unlike naive implementation and much slower performance in erosion +and dilation filters, when speed is critical morpho filter +should be used instead. +

+

A description of accepted options follows, +

+
+
mode
+

Set morphological transform to apply, can be: +

+
+
‘erode’
+
‘dilate’
+
‘open’
+
‘close’
+
‘gradient’
+
‘tophat’
+
‘blackhat’
+
+

Default is erode. +

+
+
planes
+

Set planes to filter, by default all planes except alpha are filtered. +

+
+
structure
+

Set which structure video frames will be processed from second input stream, +can be first or all. Default is all. +

+
+ +

The morpho filter also supports the framesync options. +

+

11.171.1 Commands

+ +

This filter supports same commands as options. +

+

11.172 mpdecimate

+ +

Drop frames that do not differ greatly from the previous frame in +order to reduce frame rate. +

+

The main use of this filter is for very-low-bitrate encoding +(e.g. streaming over dialup modem), but it could in theory be used for +fixing movies that were inverse-telecined incorrectly. +

+

A description of the accepted options follows. +

+
+
max
+

Set the maximum number of consecutive frames which can be dropped (if +positive), or the minimum interval between dropped frames (if +negative). If the value is 0, the frame is dropped disregarding the +number of previous sequentially dropped frames. +

+

Default value is 0. +

+
+
keep
+

Set the maximum number of consecutive similar frames to ignore before to start dropping them. +If the value is 0, the frame is dropped disregarding the +number of previous sequentially similar frames. +

+

Default value is 0. +

+
+
hi
+
lo
+
frac
+

Set the dropping threshold values. +

+

Values for hi and lo are for 8x8 pixel blocks and +represent actual pixel value differences, so a threshold of 64 +corresponds to 1 unit of difference for each pixel, or the same spread +out differently over the block. +

+

A frame is a candidate for dropping if no 8x8 blocks differ by more +than a threshold of hi, and if no more than frac blocks (1 +meaning the whole image) differ by more than a threshold of lo. +

+

Default value for hi is 64*12, default value for lo is +64*5, and default value for frac is 0.33. +

+
+ +

11.173 msad

+ +

Obtain the MSAD (Mean Sum of Absolute Differences) between two input videos. +

+

This filter takes two input videos. +

+

Both input videos must have the same resolution and pixel format for +this filter to work correctly. Also it assumes that both inputs +have the same number of frames, which are compared one by one. +

+

The obtained per component, average, min and max MSAD is printed through +the logging system. +

+

The filter stores the calculated MSAD of each frame in frame metadata. +

+

This filter also supports the framesync options. +

+

In the below example the input file main.mpg being processed is compared +with the reference file ref.mpg. +

+
+
ffmpeg -i main.mpg -i ref.mpg -lavfi msad -f null -
+
+ +

11.174 multiply

+

Multiply first video stream pixels values with second video stream pixels values. +

+

The filter accepts the following options: +

+
+
scale
+

Set the scale applied to second video stream. By default is 1. +Allowed range is from 0 to 9. +

+
+
offset
+

Set the offset applied to second video stream. By default is 0.5. +Allowed range is from -1 to 1. +

+
+
planes
+

Specify planes from input video stream that will be processed. +By default all planes are processed. +

+
+ +

11.174.1 Commands

+ +

This filter supports same commands as options. +

+

11.175 negate

+ +

Negate (invert) the input video. +

+

It accepts the following option: +

+
+
components
+

Set components to negate. +

+

Available values for components are: +

+
‘y’
+
‘u’
+
‘v’
+
‘a’
+
‘r’
+
‘g’
+
‘b’
+
+ +
+
negate_alpha
+

With value 1, it negates the alpha component, if present. Default value is 0. +

+
+ +

11.175.1 Commands

+ +

This filter supports same commands as options. +

+

11.176 nlmeans

+ +

Denoise frames using Non-Local Means algorithm. +

+

Each pixel is adjusted by looking for other pixels with similar contexts. This +context similarity is defined by comparing their surrounding patches of size +pxp. Patches are searched in an area of rxr +around the pixel. +

+

Note that the research area defines centers for patches, which means some +patches will be made of pixels outside that research area. +

+

The filter accepts the following options. +

+
+
s
+

Set denoising strength. Default is 1.0. Must be in range [1.0, 30.0]. +

+
+
p
+

Set patch size. Default is 7. Must be odd number in range [0, 99]. +

+
+
pc
+

Same as p but for chroma planes. +

+

The default value is 0 and means automatic. +

+
+
r
+

Set research size. Default is 15. Must be odd number in range [0, 99]. +

+
+
rc
+

Same as r but for chroma planes. +

+

The default value is 0 and means automatic. +

+
+ +

11.177 nnedi

+ +

Deinterlace video using neural network edge directed interpolation. +

+

This filter accepts the following options: +

+
+
weights
+

Mandatory option, without binary file filter can not work. +Currently file can be found here: +https://github.com/dubhater/vapoursynth-nnedi3/blob/master/src/nnedi3_weights.bin +

+
+
deint
+

Set which frames to deinterlace, by default it is all. +Can be all or interlaced. +

+
+
field
+

Set mode of operation. +

+

Can be one of the following: +

+
+
‘af’
+

Use frame flags, both fields. +

+
‘a’
+

Use frame flags, single field. +

+
‘t’
+

Use top field only. +

+
‘b’
+

Use bottom field only. +

+
‘tf’
+

Use both fields, top first. +

+
‘bf’
+

Use both fields, bottom first. +

+
+ +
+
planes
+

Set which planes to process, by default filter process all frames. +

+
+
nsize
+

Set size of local neighborhood around each pixel, used by the predictor neural +network. +

+

Can be one of the following: +

+
+
‘s8x6’
+
‘s16x6’
+
‘s32x6’
+
‘s48x6’
+
‘s8x4’
+
‘s16x4’
+
‘s32x4’
+
+ +
+
nns
+

Set the number of neurons in predictor neural network. +Can be one of the following: +

+
+
‘n16’
+
‘n32’
+
‘n64’
+
‘n128’
+
‘n256’
+
+ +
+
qual
+

Controls the number of different neural network predictions that are blended +together to compute the final output value. Can be fast, default or +slow. +

+
+
etype
+

Set which set of weights to use in the predictor. +Can be one of the following: +

+
+
‘a, abs’
+

weights trained to minimize absolute error +

+
‘s, mse’
+

weights trained to minimize squared error +

+
+ +
+
pscrn
+

Controls whether or not the prescreener neural network is used to decide +which pixels should be processed by the predictor neural network and which +can be handled by simple cubic interpolation. +The prescreener is trained to know whether cubic interpolation will be +sufficient for a pixel or whether it should be predicted by the predictor nn. +The computational complexity of the prescreener nn is much less than that of +the predictor nn. Since most pixels can be handled by cubic interpolation, +using the prescreener generally results in much faster processing. +The prescreener is pretty accurate, so the difference between using it and not +using it is almost always unnoticeable. +

+

Can be one of the following: +

+
+
‘none’
+
‘original’
+
‘new’
+
‘new2’
+
‘new3’
+
+ +

Default is new. +

+
+ +

11.177.1 Commands

+

This filter supports same commands as options, excluding weights option. +

+

11.178 noformat

+ +

Force libavfilter not to use any of the specified pixel formats for the +input to the next filter. +

+

It accepts the following parameters: +

+
pix_fmts
+

A ’|’-separated list of pixel format names, such as +pix_fmts=yuv420p|monow|rgb24". +

+
+
+ +

11.178.1 Examples

+ +
    +
  • Force libavfilter to use a format different from yuv420p for the +input to the vflip filter: +
    +
    noformat=pix_fmts=yuv420p,vflip
    +
    + +
  • Convert the input video to any of the formats not contained in the list: +
    +
    noformat=yuv420p|yuv444p|yuv410p
    +
    +
+ +

11.179 noise

+ +

Add noise on video input frame. +

+

The filter accepts the following options: +

+
+
all_seed
+
c0_seed
+
c1_seed
+
c2_seed
+
c3_seed
+

Set noise seed for specific pixel component or all pixel components in case +of all_seed. Default value is 123457. +

+
+
all_strength, alls
+
c0_strength, c0s
+
c1_strength, c1s
+
c2_strength, c2s
+
c3_strength, c3s
+

Set noise strength for specific pixel component or all pixel components in case +all_strength. Default value is 0. Allowed range is [0, 100]. +

+
+
all_flags, allf
+
c0_flags, c0f
+
c1_flags, c1f
+
c2_flags, c2f
+
c3_flags, c3f
+

Set pixel component flags or set flags for all components if all_flags. +Available values for component flags are: +

+
‘a’
+

averaged temporal noise (smoother) +

+
‘p’
+

mix random noise with a (semi)regular pattern +

+
‘t’
+

temporal noise (noise pattern changes between frames) +

+
‘u’
+

uniform noise (gaussian otherwise) +

+
+
+
+ +

11.179.1 Examples

+ +

Add temporal and uniform noise to input video: +

+
noise=alls=20:allf=t+u
+
+ +

11.180 normalize

+ +

Normalize RGB video (aka histogram stretching, contrast stretching). +See: https://en.wikipedia.org/wiki/Normalization_(image_processing) +

+

For each channel of each frame, the filter computes the input range and maps +it linearly to the user-specified output range. The output range defaults +to the full dynamic range from pure black to pure white. +

+

Temporal smoothing can be used on the input range to reduce flickering (rapid +changes in brightness) caused when small dark or bright objects enter or leave +the scene. This is similar to the auto-exposure (automatic gain control) on a +video camera, and, like a video camera, it may cause a period of over- or +under-exposure of the video. +

+

The R,G,B channels can be normalized independently, which may cause some +color shifting, or linked together as a single channel, which prevents +color shifting. Linked normalization preserves hue. Independent normalization +does not, so it can be used to remove some color casts. Independent and linked +normalization can be combined in any ratio. +

+

The normalize filter accepts the following options: +

+
+
blackpt
+
whitept
+

Colors which define the output range. The minimum input value is mapped to +the blackpt. The maximum input value is mapped to the whitept. +The defaults are black and white respectively. Specifying white for +blackpt and black for whitept will give color-inverted, +normalized video. Shades of grey can be used to reduce the dynamic range +(contrast). Specifying saturated colors here can create some interesting +effects. +

+
+
smoothing
+

The number of previous frames to use for temporal smoothing. The input range +of each channel is smoothed using a rolling average over the current frame +and the smoothing previous frames. The default is 0 (no temporal +smoothing). +

+
+
independence
+

Controls the ratio of independent (color shifting) channel normalization to +linked (color preserving) normalization. 0.0 is fully linked, 1.0 is fully +independent. Defaults to 1.0 (fully independent). +

+
+
strength
+

Overall strength of the filter. 1.0 is full strength. 0.0 is a rather +expensive no-op. Defaults to 1.0 (full strength). +

+
+
+ +

11.180.1 Commands

+

This filter supports same commands as options, excluding smoothing option. +The command accepts the same syntax of the corresponding option. +

+

If the specified expression is not valid, it is kept at its current +value. +

+

11.180.2 Examples

+ +

Stretch video contrast to use the full dynamic range, with no temporal +smoothing; may flicker depending on the source content: +

+
normalize=blackpt=black:whitept=white:smoothing=0
+
+ +

As above, but with 50 frames of temporal smoothing; flicker should be +reduced, depending on the source content: +

+
normalize=blackpt=black:whitept=white:smoothing=50
+
+ +

As above, but with hue-preserving linked channel normalization: +

+
normalize=blackpt=black:whitept=white:smoothing=50:independence=0
+
+ +

As above, but with half strength: +

+
normalize=blackpt=black:whitept=white:smoothing=50:independence=0:strength=0.5
+
+ +

Map the darkest input color to red, the brightest input color to cyan: +

+
normalize=blackpt=red:whitept=cyan
+
+ +

11.181 null

+ +

Pass the video source unchanged to the output. +

+

11.182 ocio

+

OpenColorIO library filter +

+

This filter allows you to do color management using the OpenColorIO library. +See https://opencolorio.org/ for more details. To Enable +compilation of this filter, you need to configure FFmpeg with --enable-libopencolorio. +

+

It accepts the following options: +

+
+
config
+

By default the filter will use the OCIO config defined by the OCIO environment variable, but this parameter allows you to explicitly specify its location. +If you are getting started, you can use config=ocio://studio-config-v1.0.0_aces-v1.3_ocio-v2.1 which specifies one of the built in defaults. +

+
+
input
+

Set the input colorspace. +

+
+
output
+

Set the output colorspace. +

+
+
display
+

Set the display colorspace, used in combination with view. +

+
+
view
+

Set the view colorspace, used in combination with display. +

+
+
inverse
+

When used in combination with display and view, this inverts the transform, so going from a display/view to the "input colorspace". +

+
+
filetransform
+

Allows you to specify an external file-transform to use instead of the OCIO config file. This is useful for applying a single transform without needing a full OCIO config file. +

+
+
format
+

Allow you to specify the output pix_fmt of the OCIO filter. This *has* to be a RGB colorspace, so you really are limited to rgb24, rgba, rgb48, rgba48, gbrp10, gbrp12, gbrpf32le, gbrapf32le, for most encoding we would recommend rgb48 +

+
+
context_params
+

Allow you to specify additional context parameters for the OCIO filter. This is a list of key=value pairs, separated by colons. +

+
+
+ +

11.182.1 Examples

+ +

Map from ACEScg to ACEScct, this assumes the OCIO file is defined with the OCIO environment variable. +

+
input=ACEScg:output=ACEScct:format=rgb48
+
+ +

Map from ACEScg to a sRGB display using the "ACES 1.0 - SDR Video" view transform, this assumes the OCIO file is defined with the OCIO environment variable. Note you will need to wrap the argument in quotes to ensure that the spaces are interpreted correctly. +

+
input=ACEScg:display=sRGB - Display:view=ACES 1.0 - SDR Video:format=rgb48
+
+ + +

As above but using the OCIO file studio-config-v1.0.0_aces-v1.3_ocio-v2.1_ns.ocio rather than the OCIO environment variable. +

+
config=studio-config-v1.0.0_aces-v1.3_ocio-v2.1_ns.ocio:input=ACEScg:display=sRGB - Display:view=ACES 1.0 - SDR Video:format=rgb48
+
+ +

If you are converting to YCrCb you still will want to set the color matrix for the conversion. This is a good example of combining the two. +

+
ffmpeg -y -i SOURCEFRAMES.%05d.exr -c:v libx265 -vf "ocio=input=ACEScg:output=ACEScct:format=rgb48,scale=in_color_matrix=bt709:out_color_matrix=bt709,format=yuv444p10"  OUTPUTFILE.mov
+
+ +

11.183 ocr

+

Optical Character Recognition +

+

This filter uses Tesseract for optical character recognition. To enable +compilation of this filter, you need to configure FFmpeg with +--enable-libtesseract. +

+

It accepts the following options: +

+
+
datapath
+

Set datapath to tesseract data. Default is to use whatever was +set at installation. +

+
+
language
+

Set language, default is "eng". +

+
+
whitelist
+

Set character whitelist. +

+
+
blacklist
+

Set character blacklist. +

+
+ +

The filter exports recognized text as the frame metadata lavfi.ocr.text. +The filter exports confidence of recognized words as the frame metadata lavfi.ocr.confidence. +

+

11.184 ocv

+ +

Apply a video transform using libopencv. +

+

To enable this filter, install the libopencv library and headers and +configure FFmpeg with --enable-libopencv. +

+

It accepts the following parameters: +

+
+
filter_name
+

The name of the libopencv filter to apply. +

+
+
filter_params
+

The parameters to pass to the libopencv filter. If not specified, the default +values are assumed. +

+
+
+ +

Refer to the official libopencv documentation for more precise +information: +http://docs.opencv.org/master/modules/imgproc/doc/filtering.html +

+

Several libopencv filters are supported; see the following subsections. +

+

11.184.1 dilate

+ +

Dilate an image by using a specific structuring element. +It corresponds to the libopencv function cvDilate. +

+

It accepts the parameters: struct_el|nb_iterations. +

+

struct_el represents a structuring element, and has the syntax: +colsxrows+anchor_xxanchor_y/shape +

+

cols and rows represent the number of columns and rows of +the structuring element, anchor_x and anchor_y the anchor +point, and shape the shape for the structuring element. shape +must be "rect", "cross", "ellipse", or "custom". +

+

If the value for shape is "custom", it must be followed by a +string of the form "=filename". The file with name +filename is assumed to represent a binary image, with each +printable character corresponding to a bright pixel. When a custom +shape is used, cols and rows are ignored, the number +or columns and rows of the read file are assumed instead. +

+

The default value for struct_el is "3x3+0x0/rect". +

+

nb_iterations specifies the number of times the transform is +applied to the image, and defaults to 1. +

+

Some examples: +

+
# Use the default values
+ocv=dilate
+
+# Dilate using a structuring element with a 5x5 cross, iterating two times
+ocv=filter_name=dilate:filter_params=5x5+2x2/cross|2
+
+# Read the shape from the file diamond.shape, iterating two times.
+# The file diamond.shape may contain a pattern of characters like this
+#   *
+#  ***
+# *****
+#  ***
+#   *
+# The specified columns and rows are ignored
+# but the anchor point coordinates are not
+ocv=dilate:0x0+2x2/custom=diamond.shape|2
+
+ +

11.184.2 erode

+ +

Erode an image by using a specific structuring element. +It corresponds to the libopencv function cvErode. +

+

It accepts the parameters: struct_el:nb_iterations, +with the same syntax and semantics as the dilate filter. +

+

11.184.3 smooth

+ +

Smooth the input video. +

+

The filter takes the following parameters: +type|param1|param2|param3|param4. +

+

type is the type of smooth filter to apply, and must be one of +the following values: "blur", "blur_no_scale", "median", "gaussian", +or "bilateral". The default value is "gaussian". +

+

The meaning of param1, param2, param3, and param4 +depends on the smooth type. param1 and +param2 accept integer positive values or 0. param3 and +param4 accept floating point values. +

+

The default value for param1 is 3. The default value for the +other parameters is 0. +

+

These parameters correspond to the parameters assigned to the +libopencv function cvSmooth. +

+

11.185 oscilloscope

+ +

2D Video Oscilloscope. +

+

Useful to measure spatial impulse, step responses, chroma delays, etc. +

+

It accepts the following parameters: +

+
+
x
+

Set scope center x position. +

+
+
y
+

Set scope center y position. +

+
+
s
+

Set scope size, relative to frame diagonal. +

+
+
t
+

Set scope tilt/rotation. +

+
+
o
+

Set trace opacity. +

+
+
tx
+

Set trace center x position. +

+
+
ty
+

Set trace center y position. +

+
+
tw
+

Set trace width, relative to width of frame. +

+
+
th
+

Set trace height, relative to height of frame. +

+
+
c
+

Set which components to trace. By default it traces first three components. +

+
+
g
+

Draw trace grid. By default is enabled. +

+
+
st
+

Draw some statistics. By default is enabled. +

+
+
sc
+

Draw scope. By default is enabled. +

+
+ +

11.185.1 Commands

+

This filter supports same commands as options. +The command accepts the same syntax of the corresponding option. +

+

If the specified expression is not valid, it is kept at its current +value. +

+

11.185.2 Examples

+ +
    +
  • Inspect full first row of video frame. +
    +
    oscilloscope=x=0.5:y=0:s=1
    +
    + +
  • Inspect full last row of video frame. +
    +
    oscilloscope=x=0.5:y=1:s=1
    +
    + +
  • Inspect full 5th line of video frame of height 1080. +
    +
    oscilloscope=x=0.5:y=5/1080:s=1
    +
    + +
  • Inspect full last column of video frame. +
    +
    oscilloscope=x=1:y=0.5:s=1:t=1
    +
    + +
+ +

11.186 overlay

+ +

Overlay one video on top of another. +

+

It takes two inputs and has one output. The first input is the "main" +video on which the second input is overlaid. +

+

It accepts the following parameters: +

+

A description of the accepted options follows. +

+
+
x
+
y
+

Set the expression for the x and y coordinates of the overlaid video +on the main video. Default value is "0" for both expressions. In case +the expression is invalid, it is set to a huge value (meaning that the +overlay will not be displayed within the output visible area). +

+
+
eof_action
+

See framesync. +

+
+
eval
+

Set when the expressions for x, and y are evaluated. +

+

It accepts the following values: +

+
‘init’
+

only evaluate expressions once during the filter initialization or +when a command is processed +

+
+
‘frame’
+

evaluate expressions for each incoming frame +

+
+ +

Default value is ‘frame’. +

+
+
shortest
+

See framesync. +

+
+
format
+

Set the format for the output video. +

+

It accepts the following values: +

+
‘yuv420’
+

force YUV 4:2:0 8-bit planar output +

+
+
‘yuv420p10’
+

force YUV 4:2:0 10-bit planar output +

+
+
‘yuv422’
+

force YUV 4:2:2 8-bit planar output +

+
+
‘yuv422p10’
+

force YUV 4:2:2 10-bit planar output +

+
+
‘yuv444’
+

force YUV 4:4:4 8-bit planar output +

+
+
‘yuv444p10’
+

force YUV 4:4:4 10-bit planar output +

+
+
‘rgb’
+

force RGB 8-bit packed output +

+
+
‘gbrp’
+

force RGB 8-bit planar output +

+
+
‘auto’
+

automatically pick format +

+
+ +

Default value is ‘yuv420’. +

+
+
repeatlast
+

See framesync. +

+
+
alpha
+

Set format of alpha of the overlaid video, it can be straight or +premultiplied, or auto to choose the alpha mode automatically. +Default is auto. +

+
+ +

The x, and y expressions can contain the following +parameters. +

+
+
main_w, W
+
main_h, H
+

The main input width and height. +

+
+
overlay_w, w
+
overlay_h, h
+

The overlay input width and height. +

+
+
x
+
y
+

The computed values for x and y. They are evaluated for +each new frame. +

+
+
hsub
+
vsub
+

horizontal and vertical chroma subsample values of the output +format. For example for the pixel format "yuv422p" hsub is 2 and +vsub is 1. +

+
+
n
+

the number of input frame, starting from 0 +

+
+
pos
+

the position in the file of the input frame, NAN if unknown; deprecated, +do not use +

+
+
t
+

The timestamp, expressed in seconds. It’s NAN if the input timestamp is unknown. +

+
+
+ +

This filter also supports the framesync options. +

+

Note that the n, t variables are available only +when evaluation is done per frame, and will evaluate to NAN +when eval is set to ‘init’. +

+

Be aware that frames are taken from each input video in timestamp +order, hence, if their initial timestamps differ, it is a good idea +to pass the two inputs through a setpts=PTS-STARTPTS filter to +have them begin in the same zero timestamp, as the example for +the movie filter does. +

+

You can chain together more overlays but you should test the +efficiency of such approach. +

+

11.186.1 Commands

+ +

This filter supports the following commands: +

+
x
+
y
+

Modify the x and y of the overlay input. +The command accepts the same syntax of the corresponding option. +

+

If the specified expression is not valid, it is kept at its current +value. +

+
+ +

11.186.2 Examples

+ +
    +
  • Draw the overlay at 10 pixels from the bottom right corner of the main +video: +
    +
    overlay=main_w-overlay_w-10:main_h-overlay_h-10
    +
    + +

    Using named options the example above becomes: +

    +
    overlay=x=main_w-overlay_w-10:y=main_h-overlay_h-10
    +
    + +
  • Insert a transparent PNG logo in the bottom left corner of the input, +using the ffmpeg tool with the -filter_complex option: +
    +
    ffmpeg -i input -i logo -filter_complex 'overlay=10:main_h-overlay_h-10' output
    +
    + +
  • Insert 2 different transparent PNG logos (second logo on bottom +right corner) using the ffmpeg tool: +
    +
    ffmpeg -i input -i logo1 -i logo2 -filter_complex 'overlay=x=10:y=H-h-10,overlay=x=W-w-10:y=H-h-10' output
    +
    + +
  • Add a transparent color layer on top of the main video; WxH +must specify the size of the main input to the overlay filter: +
    +
    color=color=red@.3:size=WxH [over]; [in][over] overlay [out]
    +
    + +
  • Play an original video and a filtered version (here with the deshake +filter) side by side using the ffplay tool: +
    +
    ffplay input.avi -vf 'split[a][b]; [a]pad=iw*2:ih[src]; [b]deshake[filt]; [src][filt]overlay=w'
    +
    + +

    The above command is the same as: +

    +
    ffplay input.avi -vf 'split[b], pad=iw*2[src], [b]deshake, [src]overlay=w'
    +
    + +
  • Make a sliding overlay appearing from the left to the right top part of the +screen starting since time 2: +
    +
    overlay=x='if(gte(t,2), -w+(t-2)*20, NAN)':y=0
    +
    + +
  • Compose output by putting two input videos side to side: +
    +
    ffmpeg -i left.avi -i right.avi -filter_complex "
    +nullsrc=size=200x100 [background];
    +[0:v] setpts=PTS-STARTPTS, scale=100x100 [left];
    +[1:v] setpts=PTS-STARTPTS, scale=100x100 [right];
    +[background][left]       overlay=shortest=1       [background+left];
    +[background+left][right] overlay=shortest=1:x=100 [left+right]
    +"
    +
    + +
  • Mask 10-20 seconds of a video by applying the delogo filter to a section +
    +
    ffmpeg -i test.avi -codec:v:0 wmv2 -ar 11025 -b:v 9000k
    +-vf '[in]split[split_main][split_delogo];[split_delogo]trim=start=360:end=371,delogo=0:0:640:480[delogoed];[split_main][delogoed]overlay=eof_action=pass[out]'
    +masked.avi
    +
    + +
  • Chain several overlays in cascade: +
    +
    nullsrc=s=200x200 [bg];
    +testsrc=s=100x100, split=4 [in0][in1][in2][in3];
    +[in0] lutrgb=r=0, [bg]   overlay=0:0     [mid0];
    +[in1] lutrgb=g=0, [mid0] overlay=100:0   [mid1];
    +[in2] lutrgb=b=0, [mid1] overlay=0:100   [mid2];
    +[in3] null,       [mid2] overlay=100:100 [out0]
    +
    + +
+ +

11.187 owdenoise

+ +

Apply Overcomplete Wavelet denoiser. +

+

The filter accepts the following options: +

+
+
depth
+

Set depth. +

+

Larger depth values will denoise lower frequency components more, but +slow down filtering. +

+

Must be an int in the range 8-16, default is 8. +

+
+
luma_strength, ls
+

Set luma strength. +

+

Must be a double value in the range 0-1000, default is 1.0. +

+
+
chroma_strength, cs
+

Set chroma strength. +

+

Must be a double value in the range 0-1000, default is 1.0. +

+
+ +

11.188 pad

+ +

Add paddings to the input image, and place the original input at the +provided x, y coordinates. +

+

It accepts the following parameters: +

+
+
width, w
+
height, h
+

Specify an expression for the size of the output image with the +paddings added. If the value for width or height is 0, the +corresponding input size is used for the output. +

+

The width expression can reference the value set by the +height expression, and vice versa. +

+

The default value of width and height is 0. +

+
+
x
+
y
+

Specify the offsets to place the input image at within the padded area, +with respect to the top/left border of the output image. +

+

The x expression can reference the value set by the y +expression, and vice versa. +

+

The default value of x and y is 0. +

+

If x or y evaluate to a negative number, they’ll be changed +so the input image is centered on the padded area. +

+
+
color
+

Specify the color of the padded area. For the syntax of this option, +check the (ffmpeg-utils)"Color" section in the ffmpeg-utils +manual. +

+

The default value of color is "black". +

+
+
eval
+

Specify when to evaluate width, height, x and y expression. +

+

It accepts the following values: +

+
+
‘init’
+

Only evaluate expressions once during the filter initialization or when +a command is processed. +

+
+
‘frame’
+

Evaluate expressions for each incoming frame. +

+
+
+ +

Default value is ‘init’. +

+
+
aspect
+

Pad to aspect instead to a resolution. +

+
+
+ +

The value for the width, height, x, and y +options are expressions containing the following constants: +

+
+
in_w
+
in_h
+

The input video width and height. +

+
+
iw
+
ih
+

These are the same as in_w and in_h. +

+
+
out_w
+
out_h
+

The output width and height (the size of the padded area), as +specified by the width and height expressions. +

+
+
ow
+
oh
+

These are the same as out_w and out_h. +

+
+
x
+
y
+

The x and y offsets as specified by the x and y +expressions, or NAN if not yet specified. +

+
+
a
+

same as iw / ih +

+
+
sar
+

input sample aspect ratio +

+
+
dar
+

input display aspect ratio, it is the same as (iw / ih) * sar +

+
+
hsub
+
vsub
+

The horizontal and vertical chroma subsample values. For example for the +pixel format "yuv422p" hsub is 2 and vsub is 1. +

+
+ +

11.188.1 Examples

+ +
    +
  • Add paddings with the color "violet" to the input video. The output video +size is 640x480, and the top-left corner of the input video is placed at +column 0, row 40 +
    +
    pad=640:480:0:40:violet
    +
    + +

    The example above is equivalent to the following command: +

    +
    pad=width=640:height=480:x=0:y=40:color=violet
    +
    + +
  • Pad the input to get an output with dimensions increased by 3/2, +and put the input video at the center of the padded area: +
    +
    pad="3/2*iw:3/2*ih:(ow-iw)/2:(oh-ih)/2"
    +
    + +
  • Pad the input to get a squared output with size equal to the maximum +value between the input width and height, and put the input video at +the center of the padded area: +
    +
    pad="max(iw\,ih):ow:(ow-iw)/2:(oh-ih)/2"
    +
    + +
  • Pad the input to get a final w/h ratio of 16:9: +
    +
    pad="ih*16/9:ih:(ow-iw)/2:(oh-ih)/2"
    +
    + +
  • In case of anamorphic video, in order to set the output display aspect +correctly, it is necessary to use sar in the expression, +according to the relation: +
    +
    (ih * X / ih) * sar = output_dar
    +X = output_dar / sar
    +
    + +

    Thus the previous example needs to be modified to: +

    +
    pad="ih*16/9/sar:ih:(ow-iw)/2:(oh-ih)/2"
    +
    + +
  • Double the output size and put the input video in the bottom-right +corner of the output padded area: +
    +
    pad="2*iw:2*ih:ow-iw:oh-ih"
    +
    +
+ +

11.189 palettegen

+ +

Generate one palette for a whole video stream. +

+

It accepts the following options: +

+
+
max_colors
+

Set the maximum number of colors to quantize in the palette. +Note: the palette will still contain 256 colors; the unused palette entries +will be black. +

+
+
reserve_transparent
+

Create a palette of 255 colors maximum and reserve the last one for +transparency. Reserving the transparency color is useful for GIF optimization. +If not set, the maximum of colors in the palette will be 256. You probably want +to disable this option for a standalone image. +Set by default. +

+
+
transparency_color
+

Set the color that will be used as background for transparency. +

+
+
stats_mode
+

Set statistics mode. +

+

It accepts the following values: +

+
‘full’
+

Compute full frame histograms. +

+
‘diff’
+

Compute histograms only for the part that differs from previous frame. This +might be relevant to give more importance to the moving part of your input if +the background is static. +

+
‘single’
+

Compute new histogram for each frame. +

+
+ +

Default value is full. +

+
+ +

The filter also exports the frame metadata lavfi.color_quant_ratio +(nb_color_in / nb_color_out) which you can use to evaluate the degree of +color quantization of the palette. This information is also visible at +info logging level. +

+

11.189.1 Examples

+ +
    +
  • Generate a representative palette of a given video using ffmpeg: +
    +
    ffmpeg -i input.mkv -vf palettegen palette.png
    +
    +
+ +

11.190 paletteuse

+ +

Use a palette to downsample an input video stream. +

+

The filter takes two inputs: one video stream and a palette. The palette must +be a 256 pixels image. +

+

It accepts the following options: +

+
+
dither
+

Select dithering mode. Available algorithms are: +

+
‘bayer’
+

Ordered 8x8 bayer dithering (deterministic) +

+
‘heckbert’
+

Dithering as defined by Paul Heckbert in 1982 (simple error diffusion). +Note: this dithering is sometimes considered "wrong" and is included as a +reference. +

+
‘floyd_steinberg’
+

Floyd and Steingberg dithering (error diffusion) +

+
‘sierra2’
+

Frankie Sierra dithering v2 (error diffusion) +

+
‘sierra2_4a’
+

Frankie Sierra dithering v2 "Lite" (error diffusion) +

+
‘sierra3’
+

Frankie Sierra dithering v3 (error diffusion) +

+
‘burkes’
+

Burkes dithering (error diffusion) +

+
‘atkinson’
+

Atkinson dithering by Bill Atkinson at Apple Computer (error diffusion) +

+
‘none’
+

Disable dithering. +

+
+ +

Default is sierra2_4a. +

+
+
bayer_scale
+

When bayer dithering is selected, this option defines the scale of the +pattern (how much the crosshatch pattern is visible). A low value means more +visible pattern for less banding, and higher value means less visible pattern +at the cost of more banding. +

+

The option must be an integer value in the range [0,5]. Default is 2. +

+
+
diff_mode
+

If set, define the zone to process +

+
+
‘rectangle’
+

Only the changing rectangle will be reprocessed. This is similar to GIF +cropping/offsetting compression mechanism. This option can be useful for speed +if only a part of the image is changing, and has use cases such as limiting the +scope of the error diffusal dither to the rectangle that bounds the +moving scene (it leads to more deterministic output if the scene doesn’t change +much, and as a result less moving noise and better GIF compression). +

+
+ +

Default is none. +

+
+
new
+

Take new palette for each output frame. +

+
+
alpha_threshold
+

Sets the alpha threshold for transparency. Alpha values above this threshold +will be treated as completely opaque, and values below this threshold will be +treated as completely transparent. +

+

The option must be an integer value in the range [0,255]. Default is 128. +

+
+ +

11.190.1 Examples

+ +
    +
  • Use a palette (generated for example with palettegen) to encode a GIF +using ffmpeg: +
    +
    ffmpeg -i input.mkv -i palette.png -lavfi paletteuse output.gif
    +
    +
+ +

11.191 perspective

+ +

Correct perspective of video not recorded perpendicular to the screen. +

+

A description of the accepted parameters follows. +

+
+
x0
+
y0
+
x1
+
y1
+
x2
+
y2
+
x3
+
y3
+

Set coordinates expression for top left, top right, bottom left and bottom right corners. +Default values are 0:0:W:0:0:H:W:H with which perspective will remain unchanged. +If the sense option is set to source, then the specified points will be sent +to the corners of the destination. If the sense option is set to destination, +then the corners of the source will be sent to the specified coordinates. +

+

The expressions can use the following variables: +

+
+
W
+
H
+

the width and height of video frame. +

+
in
+

Input frame count. +

+
on
+

Output frame count. +

+
+ +
+
interpolation
+

Set interpolation for perspective correction. +

+

It accepts the following values: +

+
‘linear’
+
‘cubic’
+
+ +

Default value is ‘linear’. +

+
+
sense
+

Set interpretation of coordinate options. +

+

It accepts the following values: +

+
‘0, source’
+
+

Send point in the source specified by the given coordinates to +the corners of the destination. +

+
+
‘1, destination’
+
+

Send the corners of the source to the point in the destination specified +by the given coordinates. +

+

Default value is ‘source’. +

+
+ +
+
eval
+

Set when the expressions for coordinates x0,y0,...x3,y3 are evaluated. +

+

It accepts the following values: +

+
‘init’
+

only evaluate expressions once during the filter initialization or +when a command is processed +

+
+
‘frame’
+

evaluate expressions for each incoming frame +

+
+ +

Default value is ‘init’. +

+
+ +

11.192 phase

+ +

Delay interlaced video by one field time so that the field order changes. +

+

The intended use is to fix PAL movies that have been captured with the +opposite field order to the film-to-video transfer. +

+

A description of the accepted parameters follows. +

+
+
mode
+

Set phase mode. +

+

It accepts the following values: +

+
‘t’
+

Capture field order top-first, transfer bottom-first. +Filter will delay the bottom field. +

+
+
‘b’
+

Capture field order bottom-first, transfer top-first. +Filter will delay the top field. +

+
+
‘p’
+

Capture and transfer with the same field order. This mode only exists +for the documentation of the other options to refer to, but if you +actually select it, the filter will faithfully do nothing. +

+
+
‘a’
+

Capture field order determined automatically by field flags, transfer +opposite. +Filter selects among ‘t’ and ‘b’ modes on a frame by frame +basis using field flags. If no field information is available, +then this works just like ‘u’. +

+
+
‘u’
+

Capture unknown or varying, transfer opposite. +Filter selects among ‘t’ and ‘b’ on a frame by frame basis by +analyzing the images and selecting the alternative that produces best +match between the fields. +

+
+
‘T’
+

Capture top-first, transfer unknown or varying. +Filter selects among ‘t’ and ‘p’ using image analysis. +

+
+
‘B’
+

Capture bottom-first, transfer unknown or varying. +Filter selects among ‘b’ and ‘p’ using image analysis. +

+
+
‘A’
+

Capture determined by field flags, transfer unknown or varying. +Filter selects among ‘t’, ‘b’ and ‘p’ using field flags and +image analysis. If no field information is available, then this works just +like ‘U’. This is the default mode. +

+
+
‘U’
+

Both capture and transfer unknown or varying. +Filter selects among ‘t’, ‘b’ and ‘p’ using image analysis only. +

+
+
+
+ +

11.192.1 Commands

+ +

This filter supports the all above options as commands. +

+

11.193 photosensitivity

+

Reduce various flashes in video, so to help users with epilepsy. +

+

It accepts the following options: +

+
frames, f
+

Set how many frames to use when filtering. Default is 30. +

+
+
threshold, t
+

Set detection threshold factor. Default is 1. +Lower is stricter. +

+
+
skip
+

Set how many pixels to skip when sampling frames. Default is 1. +Allowed range is from 1 to 1024. +

+
+
bypass
+

Leave frames unchanged. Default is disabled. +

+
+ +

11.194 pixdesctest

+ +

Pixel format descriptor test filter, mainly useful for internal +testing. The output video should be equal to the input video. +

+

For example: +

+
format=monow, pixdesctest
+
+ +

can be used to test the monowhite pixel format descriptor definition. +

+

11.195 pixelize

+

Apply pixelization to video stream. +

+

The filter accepts the following options: +

+
width, w
+
height, h
+

Set block dimensions that will be used for pixelization. +Default value is 16. +

+
+
mode, m
+

Set the mode of pixelization used. +

+

Possible values are: +

+
‘avg’
+
‘min’
+
‘max’
+
+

Default value is avg. +

+
+
planes, p
+

Set what planes to filter. Default is to filter all planes. +

+
+ +

11.195.1 Commands

+ +

This filter supports all options as commands. +

+

11.196 pixscope

+ +

Display sample values of color channels. Mainly useful for checking color +and levels. Minimum supported resolution is 640x480. +

+

The filters accept the following options: +

+
+
x
+

Set scope X position, relative offset on X axis. +

+
+
y
+

Set scope Y position, relative offset on Y axis. +

+
+
w
+

Set scope width. +

+
+
h
+

Set scope height. +

+
+
o
+

Set window opacity. This window also holds statistics about pixel area. +

+
+
wx
+

Set window X position, relative offset on X axis. +

+
+
wy
+

Set window Y position, relative offset on Y axis. +

+
+ +

11.196.1 Commands

+ +

This filter supports same commands as options. +

+

11.197 pp7

+

Apply Postprocessing filter 7. It is variant of the spp filter, +similar to spp = 6 with 7 point DCT, where only the center sample is +used after IDCT. +

+

The filter accepts the following options: +

+
+
qp
+

Force a constant quantization parameter. It accepts an integer in range +0 to 63. If not set, the filter will use the QP from the video stream +(if available). +

+
+
mode
+

Set thresholding mode. Available modes are: +

+
+
‘hard’
+

Set hard thresholding. +

+
‘soft’
+

Set soft thresholding (better de-ringing effect, but likely blurrier). +

+
‘medium’
+

Set medium thresholding (good results, default). +

+
+
+
+ +

11.198 premultiply

+

Apply alpha premultiply effect to input video stream using first plane +of second stream as alpha. +

+

Both streams must have same dimensions and same pixel format. +

+

The filter accepts the following option: +

+
+
planes
+

Set which planes will be processed, unprocessed planes will be copied. +By default value 0xf, all planes will be processed. +

+
+
inplace
+

Do not require 2nd input for processing, instead use alpha plane from input stream. +

+
+ +

11.199 premultiply_dynamic

+

Dynamically premultiply or unpremultiply the input video stream as needed, +to match the requirements of the downstream filter graph. This is roughly +equivalent to either premultiply:inplace=yes or +unpremultiply:inplace=yes, or otherwise a noop. +

+

The filter accepts the following option: +

+
+
planes
+

Set which planes will be processed, unprocessed planes will be copied. +By default value 0xf, all planes will be processed. +

+
+ +

11.200 prewitt

+

Apply prewitt operator to input video stream. +

+

The filter accepts the following option: +

+
+
planes
+

Set which planes will be processed, unprocessed planes will be copied. +By default value 0xf, all planes will be processed. +

+
+
scale
+

Set value which will be multiplied with filtered result. +

+
+
delta
+

Set value which will be added to filtered result. +

+
+ +

11.200.1 Commands

+ +

This filter supports the all above options as commands. +

+

11.201 pseudocolor

+ +

Alter frame colors in video with pseudocolors. +

+

This filter accepts the following options: +

+
+
c0
+

set pixel first component expression +

+
+
c1
+

set pixel second component expression +

+
+
c2
+

set pixel third component expression +

+
+
c3
+

set pixel fourth component expression, corresponds to the alpha component +

+
+
index, i
+

set component to use as base for altering colors +

+
+
preset, p
+

Pick one of built-in LUTs. By default is set to none. +

+

Available LUTs: +

+
‘magma’
+
‘inferno’
+
‘plasma’
+
‘viridis’
+
‘turbo’
+
‘cividis’
+
‘range1’
+
‘range2’
+
‘shadows’
+
‘highlights’
+
‘solar’
+
‘nominal’
+
‘preferred’
+
‘total’
+
‘spectral’
+
‘cool’
+
‘heat’
+
‘fiery’
+
‘blues’
+
‘green’
+
‘helix’
+
+ +
+
opacity
+

Set opacity of output colors. Allowed range is from 0 to 1. +Default value is set to 1. +

+
+ +

Each of the expression options specifies the expression to use for computing +the lookup table for the corresponding pixel component values. +

+

The expressions can contain the following constants and functions: +

+
+
w
+
h
+

The input width and height. +

+
+
val
+

The input value for the pixel component. +

+
+
ymin, umin, vmin, amin
+

The minimum allowed component value. +

+
+
ymax, umax, vmax, amax
+

The maximum allowed component value. +

+
+ +

All expressions default to "val". +

+

11.201.1 Commands

+ +

This filter supports the all above options as commands. +

+

11.201.2 Examples

+ +
    +
  • Change too high luma values to gradient: +
    +
    pseudocolor="'if(between(val,ymax,amax),lerp(ymin,ymax,(val-ymax)/(amax-ymax)),-1):if(between(val,ymax,amax),lerp(umax,umin,(val-ymax)/(amax-ymax)),-1):if(between(val,ymax,amax),lerp(vmin,vmax,(val-ymax)/(amax-ymax)),-1):-1'"
    +
    +
+ +

11.202 psnr

+ +

Obtain the average, maximum and minimum PSNR (Peak Signal to Noise +Ratio) between two input videos. +

+

This filter takes in input two input videos, the first input is +considered the "main" source and is passed unchanged to the +output. The second input is used as a "reference" video for computing +the PSNR. +

+

Both video inputs must have the same resolution and pixel format for +this filter to work correctly. Also it assumes that both inputs +have the same number of frames, which are compared one by one. +

+

The obtained average PSNR is printed through the logging system. +

+

The filter stores the accumulated MSE (mean squared error) of each +frame, and at the end of the processing it is averaged across all frames +equally, and the following formula is applied to obtain the PSNR: +

+
+
PSNR = 10*log10(MAX^2/MSE)
+
+ +

Where MAX is the average of the maximum values of each component of the +image. +

+

The description of the accepted parameters follows. +

+
+
stats_file, f
+

If specified the filter will use the named file to save the PSNR of +each individual frame. When filename equals "-" the data is sent to +standard output. +

+
+
stats_version
+

Specifies which version of the stats file format to use. Details of +each format are written below. +Default value is 1. +

+
+
output_max
+

Determines whether the max value is output to the stats log. +Default value is 0. +Requires stats_version >= 2. If this is set and stats_version < 2, +the filter will return an error. +

+
+ +

This filter also supports the framesync options. +

+

The file printed if stats_file is selected, contains a sequence of +key/value pairs of the form key:value for each compared +couple of frames. +

+

If a stats_version greater than 1 is specified, a header line precedes +the list of per-frame-pair stats, with key value pairs following the frame +format with the following parameters: +

+
+
psnr_log_version
+

The version of the log file format. Will match stats_version. +

+
+
fields
+

A comma separated list of the per-frame-pair parameters included in +the log. +

+
+ +

A description of each shown per-frame-pair parameter follows: +

+
+
n
+

sequential number of the input frame, starting from 1 +

+
+
mse_avg
+

Mean Square Error pixel-by-pixel average difference of the compared +frames, averaged over all the image components. +

+
+
mse_y, mse_u, mse_v, mse_r, mse_g, mse_b, mse_a
+

Mean Square Error pixel-by-pixel average difference of the compared +frames for the component specified by the suffix. +

+
+
psnr_y, psnr_u, psnr_v, psnr_r, psnr_g, psnr_b, psnr_a
+

Peak Signal to Noise ratio of the compared frames for the component +specified by the suffix. +

+
+
max_avg, max_y, max_u, max_v
+

Maximum allowed value for each channel, and average over all +channels. +

+
+ +

11.202.1 Examples

+
    +
  • For example: +
    +
    movie=ref_movie.mpg, setpts=PTS-STARTPTS [main];
    +[main][ref] psnr="stats_file=stats.log" [out]
    +
    + +

    On this example the input file being processed is compared with the +reference file ref_movie.mpg. The PSNR of each individual frame +is stored in stats.log. +

    +
  • Another example with different containers: +
    +
    ffmpeg -i main.mpg -i ref.mkv -lavfi  "[0:v]settb=AVTB,setpts=PTS-STARTPTS[main];[1:v]settb=AVTB,setpts=PTS-STARTPTS[ref];[main][ref]psnr" -f null -
    +
    +
+ +

11.203 pullup

+ +

Pulldown reversal (inverse telecine) filter, capable of handling mixed +hard-telecine, 24000/1001 fps progressive, and 30000/1001 fps progressive +content. +

+

The pullup filter is designed to take advantage of future context in making +its decisions. This filter is stateless in the sense that it does not lock +onto a pattern to follow, but it instead looks forward to the following +fields in order to identify matches and rebuild progressive frames. +

+

To produce content with an even framerate, insert the fps filter after +pullup, use fps=24000/1001 if the input frame rate is 29.97fps, +fps=24 for 30fps and the (rare) telecined 25fps input. +

+

The filter accepts the following options: +

+
+
jl
+
jr
+
jt
+
jb
+

These options set the amount of "junk" to ignore at the left, right, top, and +bottom of the image, respectively. Left and right are in units of 8 pixels, +while top and bottom are in units of 2 lines. +The default is 8 pixels on each side. +

+
+
sb
+

Set the strict breaks. Setting this option to 1 will reduce the chances of +filter generating an occasional mismatched frame, but it may also cause an +excessive number of frames to be dropped during high motion sequences. +Conversely, setting it to -1 will make filter match fields more easily. +This may help processing of video where there is slight blurring between +the fields, but may also cause there to be interlaced frames in the output. +Default value is 0. +

+
+
mp
+

Set the metric plane to use. It accepts the following values: +

+
‘l’
+

Use luma plane. +

+
+
‘u’
+

Use chroma blue plane. +

+
+
‘v’
+

Use chroma red plane. +

+
+ +

This option may be set to use chroma plane instead of the default luma plane +for doing filter’s computations. This may improve accuracy on very clean +source material, but more likely will decrease accuracy, especially if there +is chroma noise (rainbow effect) or any grayscale video. +The main purpose of setting mp to a chroma plane is to reduce CPU +load and make pullup usable in realtime on slow machines. +

+
+ +

For best results (without duplicated frames in the output file) it is +necessary to change the output frame rate. For example, to inverse +telecine NTSC input: +

+
ffmpeg -i input -vf pullup -r 24000/1001 ...
+
+ +

11.204 qp

+ +

Change video quantization parameters (QP). +

+

The filter accepts the following option: +

+
+
qp
+

Set expression for quantization parameter. +

+
+ +

The expression is evaluated through the eval API and can contain, among others, +the following constants: +

+
+
known
+

1 if index is not 129, 0 otherwise. +

+
+
qp
+

Sequential index starting from -129 to 128. +

+
+ +

11.204.1 Examples

+ +
    +
  • Some equation like: +
    +
    qp=2+2*sin(PI*qp)
    +
    +
+ +

11.205 qrencode

+

Generate a QR code using the libqrencode library (see +https://fukuchi.org/works/qrencode/), and overlay it on top of the current +frame. +

+

To enable the compilation of this filter, you need to configure FFmpeg with +--enable-libqrencode. +

+

The QR code is generated from the provided text or text pattern. The +corresponding QR code is scaled and overlaid into the video output according to +the specified options. +

+

In case no text is specified, no QR code is overlaied. +

+

This filter accepts the following options: +

+
+
qrcode_width, q
+
padded_qrcode_width, Q
+

Specify an expression for the width of the rendered QR code, with and without +padding. The qrcode_width expression can reference the value set by the +padded_qrcode_width expression, and vice versa. +By default padded_qrcode_width is set to qrcode_width, meaning that +there is no padding. +

+

These expressions are evaluated for each new frame. +

+

See the qrencode Expressions section for details. +

+
+
x
+
y
+

Specify an expression for positioning the padded QR code top-left corner. The +x expression can reference the value set by the y expression, and +vice. +

+

By default x and y are set set to 0, meaning that the QR code +is placed in the top left corner of the input. +

+

These expressions are evaluated for each new frame. +

+

See the qrencode Expressions section for details. +

+
+
case_sensitive, cs
+

Instruct libqrencode to use case sensitive encoding. This is enabled by +default. This can be disabled to reduce the QR encoding size. +

+
+
level, l
+

Specify the QR encoding error correction level. With an higher correction level, +the encoding size will increase but the code will be more robust to corruption. +Lower level is L. +

+

It accepts the following values: +

+
‘L’
+
‘M’
+
‘Q’
+
‘H’
+
+ +
+
expansion
+

Select how the input text is expanded. Can be either none, or +normal (default). See the qrencode Text expansion +section below for details. +

+
+
text
+
textfile
+

Define the text to be rendered. In case neither is specified, no QR is encoded +(just an empty colored frame). +

+

In case expansion is enabled, the text is treated as a text template, using the +qrencode expansion mechanism. See the qrencode +Text expansion section below for details. +

+
+
background_color, bc
+
foreground_color, fc
+

Set the QR code and background color. The default value of +foreground_color is "black", the default value of background_color +is "white". +

+

For the syntax of the color options, check the (ffmpeg-utils)"Color" +section in the ffmpeg-utils manual. +

+
+ +

11.205.1 qrencode Expressions

+ +

The expressions set by the options contain the following constants and functions. +

+
+
dar
+

input display aspect ratio, it is the same as (w / h) * sar +

+
+
duration
+

the current frame’s duration, in seconds +

+
+
hsub
+
vsub
+

horizontal and vertical chroma subsample values. For example for the +pixel format "yuv422p" hsub is 2 and vsub is 1. +

+
+
main_h, H
+

the input height +

+
+
main_w, W
+

the input width +

+
+
n
+

the number of input frame, starting from 0 +

+
+
pict_type
+

a number representing the picture type +

+
+
qr_w, w
+

the width of the encoded QR code +

+
+
rendered_qr_w, q
+
rendered_padded_qr_w, Q
+

the width of the rendered QR code, without and without padding. +

+

These parameters allow the q and Q expressions to refer to each +other, so you can for example specify q=3/4*Q. +

+
+
rand(min, max)
+

return a random number included between min and max +

+
+
sar
+

the input sample aspect ratio +

+
+
t
+

timestamp expressed in seconds, NAN if the input timestamp is unknown +

+
+
x
+
y
+

the x and y offset coordinates where the text is drawn. +

+

These parameters allow the x and y expressions to refer to each +other, so you can for example specify y=x/dar. +

+
+ +

11.205.2 qrencode Text expansion

+ +

If expansion is set to none, the text is printed verbatim. +

+

If expansion is set to normal (which is the default), +the following expansion mechanism is used. +

+

The backslash character ‘\’, followed by any character, always expands to +the second character. +

+

Sequences of the form %{...} are expanded. The text between the +braces is a function name, possibly followed by arguments separated by ’:’. +If the arguments contain special characters or delimiters (’:’ or ’}’), +they should be escaped. +

+

Note that they probably must also be escaped as the value for the text +option in the filter argument string and as the filter argument in the +filtergraph description, and possibly also for the shell, that makes up to four +levels of escaping; using a text file with the textfile option avoids +these problems. +

+

The following functions are available: +

+
+
n, frame_num
+

return the frame number +

+
+
pts
+

Return the presentation timestamp of the current frame. +

+

It can take up to two arguments. +

+

The first argument is the format of the timestamp; it defaults to flt for +seconds as a decimal number with microsecond accuracy; hms stands for a +formatted [-]HH:MM:SS.mmm timestamp with millisecond accuracy. +gmtime stands for the timestamp of the frame formatted as UTC time; +localtime stands for the timestamp of the frame formatted as local time +zone time. If the format is set to hms24hh, the time is formatted in 24h +format (00-23). +

+

The second argument is an offset added to the timestamp. +

+

If the format is set to localtime or gmtime, a third argument may +be supplied: a strftime C function format string. By default, +YYYY-MM-DD HH:MM:SS format will be used. +

+
+
expr, e
+

Evaluate the expression’s value and output as a double. +

+

It must take one argument specifying the expression to be evaluated, accepting +the constants and functions defined in qrencode_expressions. +

+
+
expr_formatted, ef
+

Evaluate the expression’s value and output as a formatted string. +

+

The first argument is the expression to be evaluated, just as for the expr function. +The second argument specifies the output format. Allowed values are ‘x’, +‘X’, ‘d’ and ‘u’. They are treated exactly as in the +printf function. +The third parameter is optional and sets the number of positions taken by the output. +It can be used to add padding with zeros from the left. +

+
+
gmtime
+

The time at which the filter is running, expressed in UTC. +It can accept an argument: a strftime C function format string. +The format string is extended to support the variable %[1-6]N +which prints fractions of the second with optionally specified number of digits. +

+
+
localtime
+

The time at which the filter is running, expressed in the local time zone. +It can accept an argument: a strftime C function format string. +The format string is extended to support the variable %[1-6]N +which prints fractions of the second with optionally specified number of digits. +

+
+
metadata
+

Frame metadata. Takes one or two arguments. +

+

The first argument is mandatory and specifies the metadata key. +

+

The second argument is optional and specifies a default value, used when the +metadata key is not found or empty. +

+

Available metadata can be identified by inspecting entries starting with TAG +included within each frame section printed by running ffprobe +-show_frames. +

+

String metadata generated in filters leading to the qrencode filter are also +available. +

+
+
rand(min, max)
+

return a random number included between min and max +

+
+ +

11.205.3 Examples

+ +
    +
  • Generate a QR code encoding the specified text with the default size, overalaid +in the top left corner of the input video, with the default size: +
    +
    qrencode=text=www.ffmpeg.org
    +
    + +
  • Same as below, but select blue on pink colors: +
    +
    qrencode=text=www.ffmpeg.org:bc=pink@0.5:fc=blue
    +
    + +
  • Place the QR code in the bottom right corner of the input video: +
    +
    qrencode=text=www.ffmpeg.org:x=W-Q:y=H-Q
    +
    + +
  • Generate a QR code with width of 200 pixels and padding, making the padded width +4/3 of the QR code width: +
    +
    qrencode=text=www.ffmpeg.org:q=200:Q=4/3*q
    +
    + +
  • Generate a QR code with padded width of 200 pixels and padding, making the QR +code width 3/4 of the padded width: +
    +
    qrencode=text=www.ffmpeg.org:Q=200:q=3/4*Q
    +
    + +
  • Make the QR code a fraction of the input video width: +
    +
    qrencode=text=www.ffmpeg.org:q=W/5
    +
    + +
  • Generate a QR code encoding the frame number: +
    +
    qrencode=text=%{n}
    +
    + +
  • Generate a QR code encoding the GMT timestamp: +
    +
    qrencode=text=%{gmtime}
    +
    + +
  • Generate a QR code encoding the timestamp expressed as a float: +
    +
    qrencode=text=%{pts}
    +
    + +
+ +

11.206 quirc

+ +

Identify and decode a QR code using the libquirc library (see +https://github.com/dlbeer/quirc/), and print the identified QR codes +positions and payload as metadata. +

+

To enable the compilation of this filter, you need to configure FFmpeg with +--enable-libquirc. +

+

For each found QR code in the input video, some metadata entries are added with +the prefix lavfi.quirc.N, where N is the index, starting from 0, +associated to the QR code. +

+

A description of each metadata value follows: +

+
+
lavfi.quirc.count
+

the number of found QR codes, it is not set in case none was found +

+
+
lavfi.quirc.N.corner.M.x
+
lavfi.quirc.N.coreer.M.y
+

the x/y positions of the four corners of the square containing the QR code, +where M is the index of the corner starting from 0 +

+
+
lavfi.quirc.N.payload
+

the payload of the QR code +

+
+ +

11.207 random

+ +

Flush video frames from internal cache of frames into a random order. +No frame is discarded. +Inspired by frei0r nervous filter. +

+
+
frames
+

Set size in number of frames of internal cache, in range from 2 to +512. Default is 30. +

+
+
seed
+

Set seed for random number generator, must be an integer included between +0 and UINT32_MAX. If not specified, or if explicitly set to +less than 0, the filter will try to use a good random seed on a +best effort basis. +

+
+ +

11.208 readeia608

+ +

Read closed captioning (EIA-608) information from the top lines of a video frame. +

+

This filter adds frame metadata for lavfi.readeia608.X.cc and +lavfi.readeia608.X.line, where X is the number of the identified line +with EIA-608 data (starting from 0). A description of each metadata value follows: +

+
+
lavfi.readeia608.X.cc
+

The two bytes stored as EIA-608 data (printed in hexadecimal). +

+
+
lavfi.readeia608.X.line
+

The number of the line on which the EIA-608 data was identified and read. +

+
+ +

This filter accepts the following options: +

+
+
scan_min
+

Set the line to start scanning for EIA-608 data. Default is 0. +

+
+
scan_max
+

Set the line to end scanning for EIA-608 data. Default is 29. +

+
+
spw
+

Set the ratio of width reserved for sync code detection. +Default is 0.27. Allowed range is [0.1 - 0.7]. +

+
+
chp
+

Enable checking the parity bit. In the event of a parity error, the filter will output +0x00 for that character. Default is false. +

+
+
lp
+

Lowpass lines prior to further processing. Default is enabled. +

+
+ +

11.208.1 Commands

+ +

This filter supports the all above options as commands. +

+

11.208.2 Examples

+ +
    +
  • Output a csv with presentation time and the first two lines of identified EIA-608 captioning data. +
    +
    ffprobe -f lavfi -i movie=captioned_video.mov,readeia608 -show_entries frame=pts_time:frame_tags=lavfi.readeia608.0.cc,lavfi.readeia608.1.cc -of csv
    +
    +
+ +

11.209 readvitc

+ +

Read vertical interval timecode (VITC) information from the top lines of a +video frame. +

+

The filter adds frame metadata key lavfi.readvitc.tc_str with the +timecode value, if a valid timecode has been detected. Further metadata key +lavfi.readvitc.found is set to 0/1 depending on whether +timecode data has been found or not. +

+

This filter accepts the following options: +

+
+
scan_max
+

Set the maximum number of lines to scan for VITC data. If the value is set to +-1 the full video frame is scanned. Default is 45. +

+
+
thr_b
+

Set the luma threshold for black. Accepts float numbers in the range [0.0,1.0], +default value is 0.2. The value must be equal or less than thr_w. +

+
+
thr_w
+

Set the luma threshold for white. Accepts float numbers in the range [0.0,1.0], +default value is 0.6. The value must be equal or greater than thr_b. +

+
+ +

11.209.1 Examples

+ +
    +
  • Detect and draw VITC data onto the video frame; if no valid VITC is detected, +draw --:--:--:-- as a placeholder: +
    +
    ffmpeg -i input.avi -filter:v 'readvitc,drawtext=fontfile=FreeMono.ttf:text=%{metadata\\:lavfi.readvitc.tc_str\\:--\\\\\\:--\\\\\\:--\\\\\\:--}:x=(w-tw)/2:y=400-ascent'
    +
    +
+ +

11.210 remap

+ +

Remap pixels using 2nd: Xmap and 3rd: Ymap input video stream. +

+

Destination pixel at position (X, Y) will be picked from source (x, y) position +where x = Xmap(X, Y) and y = Ymap(X, Y). If mapping values are out of range, zero +value for pixel will be used for destination pixel. +

+

Xmap and Ymap input video streams must be of same dimensions. Output video stream +will have Xmap/Ymap video stream dimensions. +Xmap and Ymap input video streams are 16bit depth, single channel. +

+
+
format
+

Specify pixel format of output from this filter. Can be color or gray. +Default is color. +

+
+
fill
+

Specify the color of the unmapped pixels. For the syntax of this option, +check the (ffmpeg-utils)"Color" section in the ffmpeg-utils +manual. Default color is black. +

+
+ +

11.211 removegrain

+ +

The removegrain filter is a spatial denoiser for progressive video. +

+
+
m0
+

Set mode for the first plane. +

+
+
m1
+

Set mode for the second plane. +

+
+
m2
+

Set mode for the third plane. +

+
+
m3
+

Set mode for the fourth plane. +

+
+ +

Range of mode is from 0 to 24. Description of each mode follows: +

+
+
0
+

Leave input plane unchanged. Default. +

+
+
1
+

Clips the pixel with the minimum and maximum of the 8 neighbour pixels. +

+
+
2
+

Clips the pixel with the second minimum and maximum of the 8 neighbour pixels. +

+
+
3
+

Clips the pixel with the third minimum and maximum of the 8 neighbour pixels. +

+
+
4
+

Clips the pixel with the fourth minimum and maximum of the 8 neighbour pixels. +This is equivalent to a median filter. +

+
+
5
+

Line-sensitive clipping giving the minimal change. +

+
+
6
+

Line-sensitive clipping, intermediate. +

+
+
7
+

Line-sensitive clipping, intermediate. +

+
+
8
+

Line-sensitive clipping, intermediate. +

+
+
9
+

Line-sensitive clipping on a line where the neighbours pixels are the closest. +

+
+
10
+

Replaces the target pixel with the closest neighbour. +

+
+
11
+

[1 2 1] horizontal and vertical kernel blur. +

+
+
12
+

Same as mode 11. +

+
+
13
+

Bob mode, interpolates top field from the line where the neighbours +pixels are the closest. +

+
+
14
+

Bob mode, interpolates bottom field from the line where the neighbours +pixels are the closest. +

+
+
15
+

Bob mode, interpolates top field. Same as 13 but with a more complicated +interpolation formula. +

+
+
16
+

Bob mode, interpolates bottom field. Same as 14 but with a more complicated +interpolation formula. +

+
+
17
+

Clips the pixel with the minimum and maximum of respectively the maximum and +minimum of each pair of opposite neighbour pixels. +

+
+
18
+

Line-sensitive clipping using opposite neighbours whose greatest distance from +the current pixel is minimal. +

+
+
19
+

Replaces the pixel with the average of its 8 neighbours. +

+
+
20
+

Averages the 9 pixels ([1 1 1] horizontal and vertical blur). +

+
+
21
+

Clips pixels using the averages of opposite neighbour. +

+
+
22
+

Same as mode 21 but simpler and faster. +

+
+
23
+

Small edge and halo removal, but reputed useless. +

+
+
24
+

Similar as 23. +

+
+ +

11.212 removelogo

+ +

Suppress a TV station logo, using an image file to determine which +pixels comprise the logo. It works by filling in the pixels that +comprise the logo with neighboring pixels. +

+

The filter accepts the following options: +

+
+
filename, f
+

Set the filter bitmap file, which can be any image format supported by +libavformat. The width and height of the image file must match those of the +video stream being processed. +

+
+ +

Pixels in the provided bitmap image with a value of zero are not +considered part of the logo, non-zero pixels are considered part of +the logo. If you use white (255) for the logo and black (0) for the +rest, you will be safe. For making the filter bitmap, it is +recommended to take a screen capture of a black frame with the logo +visible, and then using a threshold filter followed by the erode +filter once or twice. +

+

If needed, little splotches can be fixed manually. Remember that if +logo pixels are not covered, the filter quality will be much +reduced. Marking too many pixels as part of the logo does not hurt as +much, but it will increase the amount of blurring needed to cover over +the image and will destroy more information than necessary, and extra +pixels will slow things down on a large logo. +

+

11.213 repeatfields

+ +

This filter uses the repeat_field flag from the Video ES headers and hard repeats +fields based on its value. +

+

11.214 reverse

+ +

Reverse a video clip. +

+

Warning: This filter requires memory to buffer the entire clip, so trimming +is suggested. +

+

11.214.1 Examples

+ +
    +
  • Take the first 5 seconds of a clip, and reverse it. +
    +
    trim=end=5,reverse
    +
    +
+ +

11.215 rgbashift

+

Shift R/G/B/A pixels horizontally and/or vertically. +

+

The filter accepts the following options: +

+
rh
+

Set amount to shift red horizontally. +

+
rv
+

Set amount to shift red vertically. +

+
gh
+

Set amount to shift green horizontally. +

+
gv
+

Set amount to shift green vertically. +

+
bh
+

Set amount to shift blue horizontally. +

+
bv
+

Set amount to shift blue vertically. +

+
ah
+

Set amount to shift alpha horizontally. +

+
av
+

Set amount to shift alpha vertically. +

+
edge
+

Set edge mode, can be smear, default, or warp. +

+
+ +

11.215.1 Commands

+ +

This filter supports the all above options as commands. +

+

11.216 roberts

+

Apply roberts cross operator to input video stream. +

+

The filter accepts the following option: +

+
+
planes
+

Set which planes will be processed, unprocessed planes will be copied. +By default value 0xf, all planes will be processed. +

+
+
scale
+

Set value which will be multiplied with filtered result. +

+
+
delta
+

Set value which will be added to filtered result. +

+
+ +

11.216.1 Commands

+ +

This filter supports the all above options as commands. +

+

11.217 rotate

+ +

Rotate video by an arbitrary angle expressed in radians. +

+

The filter accepts the following options: +

+

A description of the optional parameters follows. +

+
angle, a
+

Set an expression for the angle by which to rotate the input video +clockwise, expressed as a number of radians. A negative value will +result in a counter-clockwise rotation. By default it is set to "0". +

+

This expression is evaluated for each frame. +

+
+
out_w, ow
+

Set the output width expression, default value is "iw". +This expression is evaluated just once during configuration. +

+
+
out_h, oh
+

Set the output height expression, default value is "ih". +This expression is evaluated just once during configuration. +

+
+
bilinear
+

Enable bilinear interpolation if set to 1, a value of 0 disables +it. Default value is 1. +

+
+
fillcolor, c
+

Set the color used to fill the output area not covered by the rotated +image. For the general syntax of this option, check the +(ffmpeg-utils)"Color" section in the ffmpeg-utils manual. +If the special value "none" is selected then no +background is printed (useful for example if the background is never shown). +

+

Default value is "black". +

+
+ +

The expressions for the angle and the output size can contain the +following constants and functions: +

+
+
n
+

sequential number of the input frame, starting from 0. It is always NAN +before the first frame is filtered. +

+
+
t
+

time in seconds of the input frame, it is set to 0 when the filter is +configured. It is always NAN before the first frame is filtered. +

+
+
hsub
+
vsub
+

horizontal and vertical chroma subsample values. For example for the +pixel format "yuv422p" hsub is 2 and vsub is 1. +

+
+
in_w, iw
+
in_h, ih
+

the input video width and height +

+
+
out_w, ow
+
out_h, oh
+

the output width and height, that is the size of the padded area as +specified by the width and height expressions +

+
+
rotw(a)
+
roth(a)
+

the minimal width/height required for completely containing the input +video rotated by a radians. +

+

These are only available when computing the out_w and +out_h expressions. +

+
+ +

11.217.1 Examples

+ +
    +
  • Rotate the input by PI/6 radians clockwise: +
    +
    rotate=PI/6
    +
    + +
  • Rotate the input by PI/6 radians counter-clockwise: +
    +
    rotate=-PI/6
    +
    + +
  • Rotate the input by 45 degrees clockwise: +
    +
    rotate=45*PI/180
    +
    + +
  • Apply a constant rotation with period T, starting from an angle of PI/3: +
    +
    rotate=PI/3+2*PI*t/T
    +
    + +
  • Make the input video rotation oscillating with a period of T +seconds and an amplitude of A radians: +
    +
    rotate=A*sin(2*PI/T*t)
    +
    + +
  • Rotate the video, output size is chosen so that the whole rotating +input video is always completely contained in the output: +
    +
    rotate='2*PI*t:ow=hypot(iw,ih):oh=ow'
    +
    + +
  • Rotate the video, reduce the output size so that no background is ever +shown: +
    +
    rotate=2*PI*t:ow='min(iw,ih)/sqrt(2)':oh=ow:c=none
    +
    +
+ +

11.217.2 Commands

+ +

The filter supports the following commands: +

+
+
a, angle
+

Set the angle expression. +The command accepts the same syntax of the corresponding option. +

+

If the specified expression is not valid, it is kept at its current +value. +

+
+ +

11.218 sab

+ +

Apply Shape Adaptive Blur. +

+

The filter accepts the following options: +

+
+
luma_radius, lr
+

Set luma blur filter strength, must be a value in range 0.1-4.0, default +value is 1.0. A greater value will result in a more blurred image, and +in slower processing. +

+
+
luma_pre_filter_radius, lpfr
+

Set luma pre-filter radius, must be a value in the 0.1-2.0 range, default +value is 1.0. +

+
+
luma_strength, ls
+

Set luma maximum difference between pixels to still be considered, must +be a value in the 0.1-100.0 range, default value is 1.0. +

+
+
chroma_radius, cr
+

Set chroma blur filter strength, must be a value in range -0.9-4.0. A +greater value will result in a more blurred image, and in slower +processing. +

+
+
chroma_pre_filter_radius, cpfr
+

Set chroma pre-filter radius, must be a value in the -0.9-2.0 range. +

+
+
chroma_strength, cs
+

Set chroma maximum difference between pixels to still be considered, +must be a value in the -0.9-100.0 range. +

+
+ +

Each chroma option value, if not explicitly specified, is set to the +corresponding luma option value. +

+

11.219 scale

+ +

Scale (resize) the input video, using the libswscale library. +

+

The scale filter forces the output display aspect ratio to be the same +of the input, by changing the output sample aspect ratio. +

+

If the input image format is different from the format requested by +the next filter, the scale filter will convert the input to the +requested format. +

+

11.219.1 Options

+

The filter accepts the following options, any of the options supported +by the libswscale scaler, as well as any of the framesync options. +

+

See (ffmpeg-scaler)the ffmpeg-scaler manual for +the complete list of scaler options. +

+
+
width, w
+
height, h
+

Set the output video dimension expression. Default value is the input +dimension. +

+

If the width or w value is 0, the input width is used for +the output. If the height or h value is 0, the input height +is used for the output. +

+

If one and only one of the values is -n with n >= 1, the scale filter +will use a value that maintains the aspect ratio of the input image, +calculated from the other specified dimension. After that it will, +however, make sure that the calculated dimension is divisible by n and +adjust the value if necessary. +

+

If both values are -n with n >= 1, the behavior will be identical to +both values being set to 0 as previously detailed. +

+

See below for the list of accepted constants for use in the dimension +expression. +

+
+
eval
+

Specify when to evaluate width and height expression. It accepts the following values: +

+
+
‘init’
+

Only evaluate expressions once during the filter initialization or when a command is processed. +

+
+
‘frame’
+

Evaluate expressions for each incoming frame. +

+
+
+ +

Default value is ‘init’. +

+ +
+
interl
+

Set the interlacing mode. It accepts the following values: +

+
+
‘1’
+

Force interlaced aware scaling. +

+
+
‘0’
+

Do not apply interlaced scaling. +

+
+
‘-1’
+

Select interlaced aware scaling depending on whether the source frames +are flagged as interlaced or not. +

+
+ +

Default value is ‘0’. +

+
+
flags
+

Set libswscale scaling flags. See +(ffmpeg-scaler)the ffmpeg-scaler manual for the +complete list of values. If not explicitly specified the filter applies +the default flags. +

+ +
+
param0, param1
+

Set libswscale input parameters for scaling algorithms that need them. See +(ffmpeg-scaler)the ffmpeg-scaler manual for the +complete documentation. If not explicitly specified the filter applies +empty parameters. +

+
+
intent
+

Set the ICC rendering intent to use when transforming between different color +spaces. It accepts the following values: +

+
+
‘perceptual’
+

Use a perceptually guided tone and gamut mapping curve. The exact details of +the mapping used may change at any time and should not be relied on as stable. +This intent is recommended for final viewing of image/video content in typical +viewing settings. +

+
+
‘relative_colorimetric’
+

Statically clip out-of-gamut colors using a colorimetric clipping curve which +attempts to find the colorimetrically least dissimilar in-gamut color. This +intent performs white point adaptation and black point adaptation. This is +the default. This intent is recommended wherever faithful color reproduction +is of the utmost importance, even at the cost of clipping. +

+
+
‘absolute_colorimetric’
+

Hard clip out-of-gamut colors with no attempt at white or black point +reproduction. This intent will reproduce in-gamut colors 1:1 on the output +display as they would appear on the reference display, assuming the output +display is appropriately calibrated. +

+
+
‘saturation’
+

Performs saturation mapping - that is, stretches the input color volume +directly onto the output color volume, in non-linear fashion that preserves the +original signal appearance as much as possible. This intent is recommended for +signal content evaluation, as it will not lead to any clipping. It is roughly +analogous to not performing any color mapping, although it still takes into +account the mastering display primaries and any differences in encoding TRC. +

+
+ +
+
size, s
+

Set the video size. For the syntax of this option, check the +(ffmpeg-utils)"Video size" section in the ffmpeg-utils manual. +

+
+
in_color_matrix
+
out_color_matrix
+

Set in/output YCbCr color space type. +

+

This allows the autodetected value to be overridden as well as allows forcing +a specific value used for the output and encoder. +

+

If not specified, the color space type depends on the pixel format. +

+

Possible values: +

+
+
‘auto’
+

Choose automatically. +

+
+
‘bt709’
+

Format conforming to International Telecommunication Union (ITU) +Recommendation BT.709. +

+
+
‘fcc’
+

Set color space conforming to the United States Federal Communications +Commission (FCC) Code of Federal Regulations (CFR) Title 47 (2003) 73.682 (a). +

+
+
‘bt601’
+
‘bt470’
+
‘smpte170m’
+

Set color space conforming to: +

+
    +
  • ITU Radiocommunication Sector (ITU-R) Recommendation BT.601 + +
  • ITU-R Rec. BT.470-6 (1998) Systems B, B1, and G + +
  • Society of Motion Picture and Television Engineers (SMPTE) ST 170:2004 + +
+ +
+
‘smpte240m’
+

Set color space conforming to SMPTE ST 240:1999. +

+
+
‘bt2020’
+

Set color space conforming to ITU-R BT.2020 non-constant luminance system. +

+
+ +
+
in_range
+
out_range
+

Set in/output YCbCr sample range. +

+

This allows the autodetected value to be overridden as well as allows forcing +a specific value used for the output and encoder. If not specified, the +range depends on the pixel format. Possible values: +

+
+
‘auto/unknown’
+

Choose automatically. +

+
+
‘jpeg/full/pc’
+

Set full range (0-255 in case of 8-bit luma). +

+
+
‘mpeg/limited/tv’
+

Set "MPEG" range (16-235 in case of 8-bit luma). +

+
+ +
+
in_chroma_loc
+
out_chroma_loc
+

Set in/output chroma sample location. If not specified, center-sited chroma +is used by default. Possible values: +

+
+
‘auto, unknown’
+
‘left’
+
‘center’
+
‘topleft’
+
‘top’
+
‘bottomleft’
+
‘bottom’
+
+ +
+
in_primaries
+
out_primaries
+

Set in/output RGB primaries. +

+

This allows the autodetected value to be overridden as well as allows forcing +a specific value used for the output and encoder. Possible values: +

+
+
‘auto’
+

Choose automatically. This is the default. +

+
+
‘bt709’
+
‘bt470m’
+
‘bt470bg’
+
‘smpte170m’
+
‘smpte240m’
+
‘film’
+
‘bt2020’
+
‘smpte428’
+
‘smpte431’
+
‘smpte432’
+
‘jedec-p22’
+
‘ebu3213’
+
+ +
+
in_transfer
+
out_transfer
+

Set in/output transfer response curve (TRC). +

+

This allows the autodetected value to be overridden as well as allows forcing +a specific value used for the output and encoder. Possible values: +

+
+
‘auto’
+

Choose automatically. This is the default. +

+
+
‘bt709’
+
‘bt470m’
+
‘gamma22’
+
‘bt470bg’
+
‘gamma28’
+
‘smpte170m’
+
‘smpte240m’
+
‘linear’
+
‘iec61966-2-1’
+
‘srgb’
+
‘iec61966-2-4’
+
‘xvycc’
+
‘bt1361e’
+
‘bt2020-10’
+
‘bt2020-12’
+
‘smpte2084’
+
‘smpte428’
+
‘arib-std-b67’
+
+ +
+
force_original_aspect_ratio
+

Enable decreasing or increasing output video width or height if necessary to +keep the original aspect ratio. Possible values: +

+
+
‘disable’
+

Scale the video as specified and disable this feature. +

+
+
‘decrease’
+

The output video dimensions will automatically be decreased if needed. +

+
+
‘increase’
+

The output video dimensions will automatically be increased if needed. +

+
+
+ +

One useful instance of this option is that when you know a specific device’s +maximum allowed resolution, you can use this to limit the output video to +that, while retaining the aspect ratio. For example, device A allows +1280x720 playback, and your video is 1920x800. Using this option (set it to +decrease) and specifying 1280x720 to the command line makes the output +1280x533. +

+

Please note that this is a different thing than specifying -1 for w +or h, you still need to specify the output resolution for this option +to work. +

+
+
force_divisible_by
+

Ensures that both the output dimensions, width and height, are divisible by the +given integer when used together with force_original_aspect_ratio. This +works similar to using -n in the w and h options. +

+

This option respects the value set for force_original_aspect_ratio, +increasing or decreasing the resolution accordingly. The video’s aspect ratio +may be slightly modified. +

+

This option can be handy if you need to have a video fit within or exceed +a defined resolution using force_original_aspect_ratio but also have +encoder restrictions on width or height divisibility. +

+
+
reset_sar
+

Enabling this option leads to the output SAR being reset to 1. +Additionally, if the user requests proportional scaling either +through the width or height expressions, e.g. w=-4:h=360 or w=iw/2:h=-1 +or by enabling force_original_aspect_ratio, then the input DAR is taken into +account and the output is scaled to produce square pixels. +Default is false. +

+
+
+ +

The values of the w and h options are expressions +containing the following constants: +

+
+
in_w
+
in_h
+

The input width and height +

+
+
iw
+
ih
+

These are the same as in_w and in_h. +

+
+
out_w
+
out_h
+

The output (scaled) width and height +

+
+
ow
+
oh
+

These are the same as out_w and out_h +

+
+
a
+

The same as iw / ih +

+
+
sar
+

input sample aspect ratio +

+
+
dar
+

The input display aspect ratio. Calculated from (iw / ih) * sar. +

+
+
hsub
+
vsub
+

horizontal and vertical input chroma subsample values. For example for the +pixel format "yuv422p" hsub is 2 and vsub is 1. +

+
+
ohsub
+
ovsub
+

horizontal and vertical output chroma subsample values. For example for the +pixel format "yuv422p" hsub is 2 and vsub is 1. +

+
+
n
+

The (sequential) number of the input frame, starting from 0. +Only available with eval=frame. +

+
+
t
+

The presentation timestamp of the input frame, expressed as a number of +seconds. Only available with eval=frame. +

+
+
pos
+

The position (byte offset) of the frame in the input stream, or NaN if +this information is unavailable and/or meaningless (for example in case of synthetic video). +Only available with eval=frame. +Deprecated, do not use. +

+
+
ref_w, rw
+
ref_h, rh
+
ref_a
+
ref_dar, rdar
+
ref_n
+
ref_t
+
ref_pos
+

Eqvuialent to the above, but for a second reference input. If any of these +variables are present, this filter accepts two inputs. +

+
+ +

11.219.2 Examples

+ +
    +
  • Scale the input video to a size of 200x100 +
    +
    scale=w=200:h=100
    +
    + +

    This is equivalent to: +

    +
    scale=200:100
    +
    + +

    or: +

    +
    scale=200x100
    +
    + +
  • Specify a size abbreviation for the output size: +
    +
    scale=qcif
    +
    + +

    which can also be written as: +

    +
    scale=size=qcif
    +
    + +
  • Scale the input to 2x: +
    +
    scale=w=2*iw:h=2*ih
    +
    + +
  • The above is the same as: +
    +
    scale=2*in_w:2*in_h
    +
    + +
  • Scale the input to 2x with forced interlaced scaling: +
    +
    scale=2*iw:2*ih:interl=1
    +
    + +
  • Scale the input to half size: +
    +
    scale=w=iw/2:h=ih/2
    +
    + +
  • Increase the width, and set the height to the same size: +
    +
    scale=3/2*iw:ow
    +
    + +
  • Seek Greek harmony: +
    +
    scale=iw:1/PHI*iw
    +scale=ih*PHI:ih
    +
    + +
  • Increase the height, and set the width to 3/2 of the height: +
    +
    scale=w=3/2*oh:h=3/5*ih
    +
    + +
  • Increase the size, making the size a multiple of the chroma +subsample values: +
    +
    scale="trunc(3/2*iw/hsub)*hsub:trunc(3/2*ih/vsub)*vsub"
    +
    + +
  • Increase the width to a maximum of 500 pixels, +keeping the same aspect ratio as the input: +
    +
    scale=w='min(500\, iw*3/2):h=-1'
    +
    + +
  • Make pixels square by combining scale and setsar: +
    +
    scale='trunc(ih*dar):ih',setsar=1/1
    +
    + +
  • Make pixels square using reset_sar, +making sure the resulting resolution is even (required by some codecs): +
    +
    scale='-2:ih-mod(ih,2):reset_sar=1'
    +
    + +
  • Scale to target exactly, however reset SAR to 1: +
    +
    scale='400:300:reset_sar=1'
    +
    + +
  • Scale to even dimensions that fit within 400x300, preserving input SAR: +
    +
    scale='400:300:force_original_aspect_ratio=decrease:force_divisible_by=2'
    +
    + +
  • Scale to produce square pixels with even dimensions that fit within 400x300: +
    +
    scale='400:300:force_original_aspect_ratio=decrease:force_divisible_by=2:reset_sar=1'
    +
    + +
  • Scale a subtitle stream (sub) to match the main video (main) in size before +overlaying. ("scale2ref") +
    +
    '[main]split[a][b]; [ref][a]scale=rw:rh[c]; [b][c]overlay'
    +
    + +
  • Scale a logo to 1/10th the height of a video, while preserving its display +aspect ratio. +
    +
    [logo-in][video-in]scale=w=oh*dar:h=rh/10[logo-out]
    +
    +
+ +

11.219.3 Commands

+ +

This filter supports the following commands: +

+
width, w
+
height, h
+

Set the output video dimension expression. +The command accepts the same syntax of the corresponding option. +

+

If the specified expression is not valid, it is kept at its current +value. +

+
+ +

11.220 scharr

+

Apply scharr operator to input video stream. +

+

The filter accepts the following option: +

+
+
planes
+

Set which planes will be processed, unprocessed planes will be copied. +By default value 0xf, all planes will be processed. +

+
+
scale
+

Set value which will be multiplied with filtered result. +

+
+
delta
+

Set value which will be added to filtered result. +

+
+ +

11.220.1 Commands

+ +

This filter supports the all above options as commands. +

+

11.221 scroll

+

Scroll input video horizontally and/or vertically by constant speed. +

+

The filter accepts the following options: +

+
horizontal, h
+

Set the horizontal scrolling speed. Default is 0. Allowed range is from -1 to 1. +Negative values changes scrolling direction. +

+
+
vertical, v
+

Set the vertical scrolling speed. Default is 0. Allowed range is from -1 to 1. +Negative values changes scrolling direction. +

+
+
hpos
+

Set the initial horizontal scrolling position. Default is 0. Allowed range is from 0 to 1. +

+
+
vpos
+

Set the initial vertical scrolling position. Default is 0. Allowed range is from 0 to 1. +

+
+ +

11.221.1 Commands

+ +

This filter supports the following commands: +

+
horizontal, h
+

Set the horizontal scrolling speed. +

+
vertical, v
+

Set the vertical scrolling speed. +

+
+ +

11.222 scdet

+ +

Detect video scene change. +

+

This filter sets frame metadata with mafd between frame, the scene score, and +forward the frame to the next filter, so they can use these metadata to detect +scene change or others. +

+

In addition, this filter logs a message and sets frame metadata when it detects +a scene change by threshold. +

+

lavfi.scd.mafd metadata keys are set with mafd for every frame. +

+

lavfi.scd.score metadata keys are set with scene change score for every frame +to detect scene change. +

+

lavfi.scd.time metadata keys are set with current filtered frame time which +detect scene change with threshold. +

+

The filter accepts the following options: +

+
+
threshold, t
+

Set the scene change detection threshold as a percentage of maximum change. Good +values are in the [8.0, 14.0] range. The range for threshold is +[0., 100.]. +

+

Default value is 10.. +

+
+
sc_pass, s
+

Set the flag to pass scene change frames to the next filter. Default value is 0 +You can enable it if you want to get snapshot of scene change frames only. +

+
+ +

11.223 selectivecolor

+ +

Adjust cyan, magenta, yellow and black (CMYK) to certain ranges of colors (such +as "reds", "yellows", "greens", "cyans", ...). The adjustment range is defined +by the "purity" of the color (that is, how saturated it already is). +

+

This filter is similar to the Adobe Photoshop Selective Color tool. +

+

The filter accepts the following options: +

+
+
correction_method
+

Select color correction method. +

+

Available values are: +

+
‘absolute’
+

Specified adjustments are applied "as-is" (added/subtracted to original pixel +component value). +

+
‘relative’
+

Specified adjustments are relative to the original component value. +

+
+

Default is absolute. +

+
reds
+

Adjustments for red pixels (pixels where the red component is the maximum) +

+
yellows
+

Adjustments for yellow pixels (pixels where the blue component is the minimum) +

+
greens
+

Adjustments for green pixels (pixels where the green component is the maximum) +

+
cyans
+

Adjustments for cyan pixels (pixels where the red component is the minimum) +

+
blues
+

Adjustments for blue pixels (pixels where the blue component is the maximum) +

+
magentas
+

Adjustments for magenta pixels (pixels where the green component is the minimum) +

+
whites
+

Adjustments for white pixels (pixels where all components are greater than 128) +

+
neutrals
+

Adjustments for all pixels except pure black and pure white +

+
blacks
+

Adjustments for black pixels (pixels where all components are lesser than 128) +

+
psfile
+

Specify a Photoshop selective color file (.asv) to import the settings from. +

+
+ +

All the adjustment settings (reds, yellows, ...) accept up to +4 space separated floating point adjustment values in the [-1,1] range, +respectively to adjust the amount of cyan, magenta, yellow and black for the +pixels of its range. +

+

11.223.1 Examples

+ +
    +
  • Increase cyan by 50% and reduce yellow by 33% in every green areas, and +increase magenta by 27% in blue areas: +
    +
    selectivecolor=greens=.5 0 -.33 0:blues=0 .27
    +
    + +
  • Use a Photoshop selective color preset: +
    +
    selectivecolor=psfile=MySelectiveColorPresets/Misty.asv
    +
    +
+ +

11.224 separatefields

+ +

The separatefields takes a frame-based video input and splits +each frame into its components fields, producing a new half height clip +with twice the frame rate and twice the frame count. +

+

This filter use field-dominance information in frame to decide which +of each pair of fields to place first in the output. +If it gets it wrong use setfield filter before separatefields filter. +

+

11.225 setdar, setsar

+ +

The setdar filter sets the Display Aspect Ratio for the filter +output video. +

+

This is done by changing the specified Sample (aka Pixel) Aspect +Ratio, according to the following equation: +

+
DAR = HORIZONTAL_RESOLUTION / VERTICAL_RESOLUTION * SAR
+
+ +

Keep in mind that the setdar filter does not modify the pixel +dimensions of the video frame. Also, the display aspect ratio set by +this filter may be changed by later filters in the filterchain, +e.g. in case of scaling or if another "setdar" or a "setsar" filter is +applied. +

+

The setsar filter sets the Sample (aka Pixel) Aspect Ratio for +the filter output video. +

+

Note that as a consequence of the application of this filter, the +output display aspect ratio will change according to the equation +above. +

+

Keep in mind that the sample aspect ratio set by the setsar +filter may be changed by later filters in the filterchain, e.g. if +another "setsar" or a "setdar" filter is applied. +

+

It accepts the following parameters: +

+
+
r, ratio, dar (setdar only), sar (setsar only)
+

Set the aspect ratio used by the filter. +

+

The parameter can be a floating point number string, or an expression. If the +parameter is not specified, the value "0" is assumed, meaning that the same +input value is used. +

+
+
max
+

Set the maximum integer value to use for expressing numerator and +denominator when reducing the expressed aspect ratio to a rational. +Default value is 100. +

+
+
+ +

The parameter sar is an expression containing the following constants: +

+
+
w, h
+

The input width and height. +

+
+
a
+

Same as w / h. +

+
+
sar
+

The input sample aspect ratio. +

+
+
dar
+

The input display aspect ratio. It is the same as +(w / h) * sar. +

+
+
hsub, vsub
+

Horizontal and vertical chroma subsample values. For example, for the +pixel format "yuv422p" hsub is 2 and vsub is 1. +

+
+ +

11.225.1 Examples

+ +
    +
  • To change the display aspect ratio to 16:9, specify one of the following: +
    +
    setdar=dar=1.77777
    +setdar=dar=16/9
    +
    + +
  • To change the sample aspect ratio to 10:11, specify: +
    +
    setsar=sar=10/11
    +
    + +
  • To set a display aspect ratio of 16:9, and specify a maximum integer value of +1000 in the aspect ratio reduction, use the command: +
    +
    setdar=ratio=16/9:max=1000
    +
    + +
+ +

11.226 setfield

+ +

Force field for the output video frame. +

+

The setfield filter marks the interlace type field for the +output frames. It does not change the input frame, but only sets the +corresponding property, which affects how the frame is treated by +following filters (e.g. fieldorder or yadif). +

+

The filter accepts the following options: +

+
+
mode
+

Available values are: +

+
+
‘auto’
+

Keep the same field property. +

+
+
‘bff’
+

Mark the frame as bottom-field-first. +

+
+
‘tff’
+

Mark the frame as top-field-first. +

+
+
‘prog’
+

Mark the frame as progressive. +

+
+
+
+ +

11.227 setparams

+ +

Force frame parameter for the output video frame. +

+

The setparams filter marks interlace and color range for the +output frames. It does not change the input frame, but only sets the +corresponding property, which affects how the frame is treated by +filters/encoders. +

+
+
field_mode
+

Available values are: +

+
+
‘auto’
+

Keep the same field property (default). +

+
+
‘bff’
+

Mark the frame as bottom-field-first. +

+
+
‘tff’
+

Mark the frame as top-field-first. +

+
+
‘prog’
+

Mark the frame as progressive. +

+
+ +
+
range
+

Available values are: +

+
+
‘auto’
+

Keep the same color range property (default). +

+
+
‘unspecified, unknown’
+

Mark the frame as unspecified color range. +

+
+
‘limited, tv, mpeg’
+

Mark the frame as limited range. +

+
+
‘full, pc, jpeg’
+

Mark the frame as full range. +

+
+ +
+
color_primaries
+

Set the color primaries. +Available values are: +

+
+
‘auto’
+

Keep the same color primaries property (default). +

+
+
‘bt709’
+
‘unknown’
+
‘bt470m’
+
‘bt470bg’
+
‘smpte170m’
+
‘smpte240m’
+
‘film’
+
‘bt2020’
+
‘smpte428’
+
‘smpte431’
+
‘smpte432’
+
‘jedec-p22’
+
+ +
+
color_trc
+

Set the color transfer. +Available values are: +

+
+
‘auto’
+

Keep the same color trc property (default). +

+
+
‘bt709’
+
‘unknown’
+
‘bt470m’
+
‘bt470bg’
+
‘smpte170m’
+
‘smpte240m’
+
‘linear’
+
‘log100’
+
‘log316’
+
‘iec61966-2-4’
+
‘bt1361e’
+
‘iec61966-2-1’
+
‘bt2020-10’
+
‘bt2020-12’
+
‘smpte2084’
+
‘smpte428’
+
‘arib-std-b67’
+
+ +
+
colorspace
+

Set the colorspace. +Available values are: +

+
+
‘auto’
+

Keep the same colorspace property (default). +

+
+
‘gbr’
+
‘bt709’
+
‘unknown’
+
‘fcc’
+
‘bt470bg’
+
‘smpte170m’
+
‘smpte240m’
+
‘ycgco’
+
‘bt2020nc’
+
‘bt2020c’
+
‘smpte2085’
+
‘chroma-derived-nc’
+
‘chroma-derived-c’
+
‘ictcp’
+
+ +
+
chroma_location
+

Set the chroma sample location. +Available values are: +

+
+
‘auto’
+

Keep the same chroma location (default). +

+
+
‘unspecified, unknown’
+
‘left’
+
‘center’
+
‘topleft’
+
‘top’
+
‘bottomleft’
+
‘bottom’
+
+ +
+
alpha_mode
+

Set the alpha moda. +Available values are: +

+
+
‘auto’
+

Keep the same alpha mode (default). +

+
+
‘unspecified, unknown’
+
‘premultiplied’
+
‘straight’
+
+
+
+ +

11.228 shear

+

Apply shear transform to input video. +

+

This filter supports the following options: +

+
+
shx
+

Shear factor in X-direction. Default value is 0. +Allowed range is from -2 to 2. +

+
+
shy
+

Shear factor in Y-direction. Default value is 0. +Allowed range is from -2 to 2. +

+
+
fillcolor, c
+

Set the color used to fill the output area not covered by the transformed +video. For the general syntax of this option, check the +(ffmpeg-utils)"Color" section in the ffmpeg-utils manual. +If the special value "none" is selected then no +background is printed (useful for example if the background is never shown). +

+

Default value is "black". +

+
+
interp
+

Set interpolation type. Can be bilinear or nearest. Default is bilinear. +

+
+ +

11.228.1 Commands

+ +

This filter supports the all above options as commands. +

+

11.229 showinfo

+ +

Show a line containing various information for each input video frame. +The input video is not modified. +

+

This filter supports the following options: +

+
+
checksum
+

Calculate checksums of each plane. By default enabled. +

+
+
udu_sei_as_ascii
+

Try to print user data unregistered SEI as ascii character when possible, +in hex format otherwise. +

+
+ +

The shown line contains a sequence of key/value pairs of the form +key:value. +

+

The following values are shown in the output: +

+
+
n
+

The (sequential) number of the input frame, starting from 0. +

+
+
pts
+

The Presentation TimeStamp of the input frame, expressed as a number of +time base units. The time base unit depends on the filter input pad. +

+
+
pts_time
+

The Presentation TimeStamp of the input frame, expressed as a number of +seconds. +

+
+
fmt
+

The pixel format name. +

+
+
sar
+

The sample aspect ratio of the input frame, expressed in the form +num/den. +

+
+
s
+

The size of the input frame. For the syntax of this option, check the +(ffmpeg-utils)"Video size" section in the ffmpeg-utils manual. +

+
+
i
+

The type of interlaced mode ("P" for "progressive", "T" for top field first, "B" +for bottom field first). +

+
+
iskey
+

This is 1 if the frame is a key frame, 0 otherwise. +

+
+
type
+

The picture type of the input frame ("I" for an I-frame, "P" for a +P-frame, "B" for a B-frame, or "?" for an unknown type). +Also refer to the documentation of the AVPictureType enum and of +the av_get_picture_type_char function defined in +libavutil/avutil.h. +

+
+
checksum
+

The Adler-32 checksum (printed in hexadecimal) of all the planes of the input frame. +

+
+
plane_checksum
+

The Adler-32 checksum (printed in hexadecimal) of each plane of the input frame, +expressed in the form "[c0 c1 c2 c3]". +

+
+
mean
+

The mean value of pixels in each plane of the input frame, expressed in the form +"[mean0 mean1 mean2 mean3]". +

+
+
stdev
+

The standard deviation of pixel values in each plane of the input frame, expressed +in the form "[stdev0 stdev1 stdev2 stdev3]". +

+
+
+ +

11.230 showpalette

+ +

Displays the 256 colors palette of each frame. This filter is only relevant for +pal8 pixel format frames. +

+

It accepts the following option: +

+
+
s
+

Set the size of the box used to represent one palette color entry. Default is +30 (for a 30x30 pixel box). +

+
+ +

11.231 shuffleframes

+ +

Reorder and/or duplicate and/or drop video frames. +

+

It accepts the following parameters: +

+
+
mapping
+

Set the destination indexes of input frames. +This is space or ’|’ separated list of indexes that maps input frames to output +frames. Number of indexes also sets maximal value that each index may have. +’-1’ index have special meaning and that is to drop frame. +

+
+ +

The first frame has the index 0. The default is to keep the input unchanged. +

+

11.231.1 Examples

+ +
    +
  • Swap second and third frame of every three frames of the input: +
    +
    ffmpeg -i INPUT -vf "shuffleframes=0 2 1" OUTPUT
    +
    + +
  • Swap 10th and 1st frame of every ten frames of the input: +
    +
    ffmpeg -i INPUT -vf "shuffleframes=9 1 2 3 4 5 6 7 8 0" OUTPUT
    +
    +
+ +

11.232 shufflepixels

+ +

Reorder pixels in video frames. +

+

This filter accepts the following options: +

+
+
direction, d
+

Set shuffle direction. Can be forward or inverse direction. +Default direction is forward. +

+
+
mode, m
+

Set shuffle mode. Can be horizontal, vertical or block mode. +

+
+
width, w
+
height, h
+

Set shuffle block_size. In case of horizontal shuffle mode only width +part of size is used, and in case of vertical shuffle mode only height +part of size is used. +

+
+
seed, s
+

Set random seed used with shuffling pixels. Mainly useful to set to be able +to reverse filtering process to get original input. +For example, to reverse forward shuffle you need to use same parameters +and exact same seed and to set direction to inverse. +

+
+ +

11.233 shuffleplanes

+ +

Reorder and/or duplicate video planes. +

+

It accepts the following parameters: +

+
+
map0
+

The index of the input plane to be used as the first output plane. +

+
+
map1
+

The index of the input plane to be used as the second output plane. +

+
+
map2
+

The index of the input plane to be used as the third output plane. +

+
+
map3
+

The index of the input plane to be used as the fourth output plane. +

+
+
+ +

The first plane has the index 0. The default is to keep the input unchanged. +

+

11.233.1 Examples

+ +
    +
  • Swap the second and third planes of the input: +
    +
    ffmpeg -i INPUT -vf shuffleplanes=0:2:1:3 OUTPUT
    +
    +
+ +

11.234 signalstats

+

Evaluate various visual metrics that assist in determining issues associated +with the digitization of analog video media. +

+

By default the filter will log these metadata values: +

+
+
YMIN
+

Display the minimal Y value contained within the input frame. Expressed in +range of [0-255]. +

+
+
YLOW
+

Display the Y value at the 10% percentile within the input frame. Expressed in +range of [0-255]. +

+
+
YAVG
+

Display the average Y value within the input frame. Expressed in range of +[0-255]. +

+
+
YHIGH
+

Display the Y value at the 90% percentile within the input frame. Expressed in +range of [0-255]. +

+
+
YMAX
+

Display the maximum Y value contained within the input frame. Expressed in +range of [0-255]. +

+
+
UMIN
+

Display the minimal U value contained within the input frame. Expressed in +range of [0-255]. +

+
+
ULOW
+

Display the U value at the 10% percentile within the input frame. Expressed in +range of [0-255]. +

+
+
UAVG
+

Display the average U value within the input frame. Expressed in range of +[0-255]. +

+
+
UHIGH
+

Display the U value at the 90% percentile within the input frame. Expressed in +range of [0-255]. +

+
+
UMAX
+

Display the maximum U value contained within the input frame. Expressed in +range of [0-255]. +

+
+
VMIN
+

Display the minimal V value contained within the input frame. Expressed in +range of [0-255]. +

+
+
VLOW
+

Display the V value at the 10% percentile within the input frame. Expressed in +range of [0-255]. +

+
+
VAVG
+

Display the average V value within the input frame. Expressed in range of +[0-255]. +

+
+
VHIGH
+

Display the V value at the 90% percentile within the input frame. Expressed in +range of [0-255]. +

+
+
VMAX
+

Display the maximum V value contained within the input frame. Expressed in +range of [0-255]. +

+
+
SATMIN
+

Display the minimal saturation value contained within the input frame. +Expressed in range of [0-~181.02]. +

+
+
SATLOW
+

Display the saturation value at the 10% percentile within the input frame. +Expressed in range of [0-~181.02]. +

+
+
SATAVG
+

Display the average saturation value within the input frame. Expressed in range +of [0-~181.02]. +

+
+
SATHIGH
+

Display the saturation value at the 90% percentile within the input frame. +Expressed in range of [0-~181.02]. +

+
+
SATMAX
+

Display the maximum saturation value contained within the input frame. +Expressed in range of [0-~181.02]. +

+
+
HUEMED
+

Display the median value for hue within the input frame. Expressed in range of +[0-360]. +

+
+
HUEAVG
+

Display the average value for hue within the input frame. Expressed in range of +[0-360]. +

+
+
YDIF
+

Display the average of sample value difference between all values of the Y +plane in the current frame and corresponding values of the previous input frame. +Expressed in range of [0-255]. +

+
+
UDIF
+

Display the average of sample value difference between all values of the U +plane in the current frame and corresponding values of the previous input frame. +Expressed in range of [0-255]. +

+
+
VDIF
+

Display the average of sample value difference between all values of the V +plane in the current frame and corresponding values of the previous input frame. +Expressed in range of [0-255]. +

+
+
YBITDEPTH
+

Display bit depth of Y plane in current frame. +Expressed in range of [0-16]. +

+
+
UBITDEPTH
+

Display bit depth of U plane in current frame. +Expressed in range of [0-16]. +

+
+
VBITDEPTH
+

Display bit depth of V plane in current frame. +Expressed in range of [0-16]. +

+
+ +

The filter accepts the following options: +

+
+
stat
+
out
+
+

stat specify an additional form of image analysis. +out output video with the specified type of pixel highlighted. +

+

Both options accept the following values: +

+
+
‘tout’
+

Identify temporal outliers pixels. A temporal outlier is a pixel +unlike the neighboring pixels of the same field. Examples of temporal outliers +include the results of video dropouts, head clogs, or tape tracking issues. +

+
+
‘vrep’
+

Identify vertical line repetition. Vertical line repetition includes +similar rows of pixels within a frame. In born-digital video vertical line +repetition is common, but this pattern is uncommon in video digitized from an +analog source. When it occurs in video that results from the digitization of an +analog source it can indicate concealment from a dropout compensator. +

+
+
‘brng’
+

Identify pixels that fall outside of legal broadcast range. +

+
+ +
+
color, c
+

Set the highlight color for the out option. The default color is +yellow. +

+
+ +

11.234.1 Examples

+ +
    +
  • Output data of various video metrics: +
    +
    ffprobe -f lavfi movie=example.mov,signalstats="stat=tout+vrep+brng" -show_frames
    +
    + +
  • Output specific data about the minimum and maximum values of the Y plane per frame: +
    +
    ffprobe -f lavfi movie=example.mov,signalstats -show_entries frame_tags=lavfi.signalstats.YMAX,lavfi.signalstats.YMIN
    +
    + +
  • Playback video while highlighting pixels that are outside of broadcast range in red. +
    +
    ffplay example.mov -vf signalstats="out=brng:color=red"
    +
    + +
  • Playback video with signalstats metadata drawn over the frame. +
    +
    ffplay example.mov -vf signalstats=stat=brng+vrep+tout,drawtext=fontfile=FreeSerif.ttf:textfile=signalstat_drawtext.txt
    +
    + +

    The contents of signalstat_drawtext.txt used in the command are: +

    +
    time %{pts:hms}
    +Y (%{metadata:lavfi.signalstats.YMIN}-%{metadata:lavfi.signalstats.YMAX})
    +U (%{metadata:lavfi.signalstats.UMIN}-%{metadata:lavfi.signalstats.UMAX})
    +V (%{metadata:lavfi.signalstats.VMIN}-%{metadata:lavfi.signalstats.VMAX})
    +saturation maximum: %{metadata:lavfi.signalstats.SATMAX}
    +
    +
    +
+ +

11.235 signature

+ +

Calculates the MPEG-7 Video Signature. The filter can handle more than one +input. In this case the matching between the inputs can be calculated additionally. +The filter always passes through the first input. The signature of each stream can +be written into a file. +

+

It accepts the following options: +

+
+
detectmode
+

Enable or disable the matching process. +

+

Available values are: +

+
+
‘off’
+

Disable the calculation of a matching (default). +

+
‘full’
+

Calculate the matching for the whole video and output whether the whole video +matches or only parts. +

+
‘fast’
+

Calculate only until a matching is found or the video ends. Should be faster in +some cases. +

+
+ +
+
nb_inputs
+

Set the number of inputs. The option value must be a non negative integer. +Default value is 1. +

+
+
filename
+

Set the path to which the output is written. If there is more than one input, +the path must be a prototype, i.e. must contain %d or %0nd (where n is a positive +integer), that will be replaced with the input number. If no filename is +specified, no output will be written. This is the default. +

+
+
format
+

Choose the output format. +

+

Available values are: +

+
+
‘binary’
+

Use the specified binary representation (default). +

+
‘xml’
+

Use the specified xml representation. +

+
+ +
+
th_d
+

Set threshold to detect one word as similar. The option value must be an integer +greater than zero. The default value is 9000. +

+
+
th_dc
+

Set threshold to detect all words as similar. The option value must be an integer +greater than zero. The default value is 60000. +

+
+
th_xh
+

Set threshold to detect frames as similar. The option value must be an integer +greater than zero. The default value is 116. +

+
+
th_di
+

Set the minimum length of a sequence in frames to recognize it as matching +sequence. The option value must be a non negative integer value. +The default value is 0. +

+
+
th_it
+

Set the minimum relation, that matching frames to all frames must have. +The option value must be a double value between 0 and 1. The default value is 0.5. +

+
+ +

11.235.1 Examples

+ +
    +
  • To calculate the signature of an input video and store it in signature.bin: +
    +
    ffmpeg -i input.mkv -vf signature=filename=signature.bin -map 0:v -f null -
    +
    + +
  • To detect whether two videos match and store the signatures in XML format in +signature0.xml and signature1.xml: +
    +
    ffmpeg -i input1.mkv -i input2.mkv -filter_complex "[0:v][1:v] signature=nb_inputs=2:detectmode=full:format=xml:filename=signature%d.xml" -map :v -f null -
    +
    + +
+ +

11.236 siti

+ +

Calculate Spatial Information (SI) and Temporal Information (TI) scores for a video, +as defined in ITU-T Rec. P.910 (11/21): Subjective video quality assessment methods +for multimedia applications. Available PDF at https://www.itu.int/rec/T-REC-P.910-202111-S/en. +Note that this is a legacy implementation that corresponds to a superseded recommendation. +Refer to ITU-T Rec. P.910 (07/22) for the latest version: https://www.itu.int/rec/T-REC-P.910-202207-I/en +

+

It accepts the following option: +

+
+
print_summary
+

If set to 1, Summary statistics will be printed to the console. Default 0. +

+
+ +

11.236.1 Examples

+
    +
  • To calculate SI/TI metrics and print summary: +
    +
    ffmpeg -i input.mp4 -vf siti=print_summary=1 -f null -
    +
    +
+ +

11.237 smartblur

+ +

Blur the input video without impacting the outlines. +

+

It accepts the following options: +

+
+
luma_radius, lr
+

Set the luma radius. The option value must be a float number in +the range [0.1,5.0] that specifies the variance of the gaussian filter +used to blur the image (slower if larger). Default value is 1.0. +

+
+
luma_strength, ls
+

Set the luma strength. The option value must be a float number +in the range [-1.0,1.0] that configures the blurring. A value included +in [0.0,1.0] will blur the image whereas a value included in +[-1.0,0.0] will sharpen the image. Default value is 1.0. +

+
+
luma_threshold, lt
+

Set the luma threshold used as a coefficient to determine +whether a pixel should be blurred or not. The option value must be an +integer in the range [-30,30]. A value of 0 will filter all the image, +a value included in [0,30] will filter flat areas and a value included +in [-30,0] will filter edges. Default value is 0. +

+
+
chroma_radius, cr
+

Set the chroma radius. The option value must be a float number in +the range [0.1,5.0] that specifies the variance of the gaussian filter +used to blur the image (slower if larger). Default value is luma_radius. +

+
+
chroma_strength, cs
+

Set the chroma strength. The option value must be a float number +in the range [-1.0,1.0] that configures the blurring. A value included +in [0.0,1.0] will blur the image whereas a value included in +[-1.0,0.0] will sharpen the image. Default value is luma_strength. +

+
+
chroma_threshold, ct
+

Set the chroma threshold used as a coefficient to determine +whether a pixel should be blurred or not. The option value must be an +integer in the range [-30,30]. A value of 0 will filter all the image, +a value included in [0,30] will filter flat areas and a value included +in [-30,0] will filter edges. Default value is luma_threshold. +

+
+
alpha_radius, ar
+

Set the alpha radius. The option value must be a float number in +the range [0.1,5.0] that specifies the variance of the gaussian filter +used to blur the image (slower if larger). Default value is luma_radius. +

+
+
alpha_strength, as
+

Set the alpha strength. The option value must be a float number +in the range [-1.0,1.0] that configures the blurring. A value included +in [0.0,1.0] will blur the image whereas a value included in +[-1.0,0.0] will sharpen the image. Default value is luma_strength. +

+
+
alpha_threshold, at
+

Set the alpha threshold used as a coefficient to determine +whether a pixel should be blurred or not. The option value must be an +integer in the range [-30,30]. A value of 0 will filter all the image, +a value included in [0,30] will filter flat areas and a value included +in [-30,0] will filter edges. Default value is luma_threshold. +

+
+ +

If a chroma or alpha option is not explicitly set, the corresponding luma value +is set. +

+

11.238 sobel

+

Apply sobel operator to input video stream. +

+

The filter accepts the following option: +

+
+
planes
+

Set which planes will be processed, unprocessed planes will be copied. +By default value 0xf, all planes will be processed. +

+
+
scale
+

Set value which will be multiplied with filtered result. +

+
+
delta
+

Set value which will be added to filtered result. +

+
+ +

11.238.1 Commands

+ +

This filter supports the all above options as commands. +

+

11.239 spp

+ +

Apply a simple postprocessing filter that compresses and decompresses the image +at several (or - in the case of quality level 6 - all) shifts +and average the results. +

+

The filter accepts the following options: +

+
+
quality
+

Set quality. This option defines the number of levels for averaging. It accepts +an integer in the range 0-6. If set to 0, the filter will have no +effect. A value of 6 means the higher quality. For each increment of +that value the speed drops by a factor of approximately 2. Default value is +3. +

+
+
qp
+

Force a constant quantization parameter. If not set, the filter will use the QP +from the video stream (if available). +

+
+
mode
+

Set thresholding mode. Available modes are: +

+
+
‘hard’
+

Set hard thresholding (default). +

+
‘soft’
+

Set soft thresholding (better de-ringing effect, but likely blurrier). +

+
+ +
+
use_bframe_qp
+

Enable the use of the QP from the B-Frames if set to 1. Using this +option may cause flicker since the B-Frames have often larger QP. Default is +0 (not enabled). +

+
+ +

11.239.1 Commands

+ +

This filter supports the following commands: +

+
quality, level
+

Set quality level. The value max can be used to set the maximum level, +currently 6. +

+
+ +

11.240 sr

+ +

Scale the input by applying one of the super-resolution methods based on +convolutional neural networks. Supported models: +

+ + +

Training scripts as well as scripts for model file (.pb) saving can be found at +https://github.com/XueweiMeng/sr/tree/sr_dnn_native. Original repository +is at https://github.com/HighVoltageRocknRoll/sr.git. +

+

The filter accepts the following options: +

+
+
dnn_backend
+

Specify which DNN backend to use for model loading and execution. This option accepts +the following values: +

+
+
‘tensorflow’
+

TensorFlow backend. To enable this backend you +need to install the TensorFlow for C library (see +https://www.tensorflow.org/install/lang_c) and configure FFmpeg with +--enable-libtensorflow +

+
+ +
+
model
+

Set path to model file specifying network architecture and its parameters. +Note that different backends use different file formats. TensorFlow, OpenVINO backend +can load files for only its format. +

+
+
scale_factor
+

Set scale factor for SRCNN model. Allowed values are 2, 3 and 4. +Default value is 2. Scale factor is necessary for SRCNN model, because it accepts +input upscaled using bicubic upscaling with proper scale factor. +

+
+ +

To get full functionality (such as async execution), please use the dnn_processing filter. +

+

11.241 sr_amf

+ +

Upscale (size increasing) for the input video using AMD Advanced Media Framework library for hardware acceleration. +Use advanced algorithms for upscaling with higher output quality. +Setting the output width and height works in the same way as for the scale filter. +

+

The filter accepts the following options: +

+
w
+
h
+

Set the output video dimension expression. Default value is the input dimension. +

+

Allows for the same expressions as the scale filter. +

+
+
algorithm
+

Sets the algorithm used for scaling: +

+
+
bilinear
+

Bilinear +

+
+
bicubic
+

Bicubic +

+
+
sr1-0
+

Video SR1.0 +This is a default value +

+
+
point
+

Point +

+
+
sr1-1
+

Video SR1.1 +

+
+
+ +
+
sharpness
+

Control hq scaler sharpening. The value is a float in the range of [0.0, 2.0] +

+
+
format
+

Controls the output pixel format. By default, or if none is specified, the input +pixel format is used. +

+
+
keep-ratio
+

Force the scaler to keep the aspect ratio of the input image when the output size has a different aspect ratio. +Default value is false. +

+
+
fill
+

Specifies whether the output image outside the region of interest, +which does not fill the entire output surface should be filled with a solid color. +

+
+
+ +

11.241.1 Examples

+ +
    +
  • Scale input to 720p, keeping aspect ratio and ensuring the output is yuv420p. +
    +
    sr_amf=-2:720:format=yuv420p
    +
    + +
  • Upscale to 4K with algorithm video SR1.1. +
    +
    sr_amf=4096:2160:algorithm=sr1-1
    +
    +
+ +

11.242 ssim

+ +

Obtain the SSIM (Structural SImilarity Metric) between two input videos. +

+

This filter takes in input two input videos, the first input is +considered the "main" source and is passed unchanged to the +output. The second input is used as a "reference" video for computing +the SSIM. +

+

Both video inputs must have the same resolution and pixel format for +this filter to work correctly. Also it assumes that both inputs +have the same number of frames, which are compared one by one. +

+

The filter stores the calculated SSIM of each frame. +

+

The description of the accepted parameters follows. +

+
+
stats_file, f
+

If specified the filter will use the named file to save the SSIM of +each individual frame. When filename equals "-" the data is sent to +standard output. +

+
+ +

The file printed if stats_file is selected, contains a sequence of +key/value pairs of the form key:value for each compared +couple of frames. +

+

A description of each shown parameter follows: +

+
+
n
+

sequential number of the input frame, starting from 1 +

+
+
Y, U, V, R, G, B
+

SSIM of the compared frames for the component specified by the suffix. +

+
+
All
+

SSIM of the compared frames for the whole frame. +

+
+
dB
+

Same as above but in dB representation. +

+
+ +

This filter also supports the framesync options. +

+

11.242.1 Examples

+
    +
  • For example: +
    +
    movie=ref_movie.mpg, setpts=PTS-STARTPTS [main];
    +[main][ref] ssim="stats_file=stats.log" [out]
    +
    + +

    On this example the input file being processed is compared with the +reference file ref_movie.mpg. The SSIM of each individual frame +is stored in stats.log. +

    +
  • Another example with both psnr and ssim at same time: +
    +
    ffmpeg -i main.mpg -i ref.mpg -lavfi  "ssim;[0:v][1:v]psnr" -f null -
    +
    + +
  • Another example with different containers: +
    +
    ffmpeg -i main.mpg -i ref.mkv -lavfi  "[0:v]settb=AVTB,setpts=PTS-STARTPTS[main];[1:v]settb=AVTB,setpts=PTS-STARTPTS[ref];[main][ref]ssim" -f null -
    +
    +
+ +

11.243 stereo3d

+ +

Convert between different stereoscopic image formats. +

+

The filters accept the following options: +

+
+
in
+

Set stereoscopic image format of input. +

+

Available values for input image formats are: +

+
‘sbsl’
+

side by side parallel (left eye left, right eye right) +

+
+
‘sbsr’
+

side by side crosseye (right eye left, left eye right) +

+
+
‘sbs2l’
+

side by side parallel with half width resolution +(left eye left, right eye right) +

+
+
‘sbs2r’
+

side by side crosseye with half width resolution +(right eye left, left eye right) +

+
+
‘abl’
+
‘tbl’
+

above-below (left eye above, right eye below) +

+
+
‘abr’
+
‘tbr’
+

above-below (right eye above, left eye below) +

+
+
‘ab2l’
+
‘tb2l’
+

above-below with half height resolution +(left eye above, right eye below) +

+
+
‘ab2r’
+
‘tb2r’
+

above-below with half height resolution +(right eye above, left eye below) +

+
+
‘al’
+

alternating frames (left eye first, right eye second) +

+
+
‘ar’
+

alternating frames (right eye first, left eye second) +

+
+
‘irl’
+

interleaved rows (left eye has top row, right eye starts on next row) +

+
+
‘irr’
+

interleaved rows (right eye has top row, left eye starts on next row) +

+
+
‘icl’
+

interleaved columns, left eye first +

+
+
‘icr’
+

interleaved columns, right eye first +

+

Default value is ‘sbsl’. +

+
+ +
+
out
+

Set stereoscopic image format of output. +

+
+
‘sbsl’
+

side by side parallel (left eye left, right eye right) +

+
+
‘sbsr’
+

side by side crosseye (right eye left, left eye right) +

+
+
‘sbs2l’
+

side by side parallel with half width resolution +(left eye left, right eye right) +

+
+
‘sbs2r’
+

side by side crosseye with half width resolution +(right eye left, left eye right) +

+
+
‘abl’
+
‘tbl’
+

above-below (left eye above, right eye below) +

+
+
‘abr’
+
‘tbr’
+

above-below (right eye above, left eye below) +

+
+
‘ab2l’
+
‘tb2l’
+

above-below with half height resolution +(left eye above, right eye below) +

+
+
‘ab2r’
+
‘tb2r’
+

above-below with half height resolution +(right eye above, left eye below) +

+
+
‘al’
+

alternating frames (left eye first, right eye second) +

+
+
‘ar’
+

alternating frames (right eye first, left eye second) +

+
+
‘irl’
+

interleaved rows (left eye has top row, right eye starts on next row) +

+
+
‘irr’
+

interleaved rows (right eye has top row, left eye starts on next row) +

+
+
‘arbg’
+

anaglyph red/blue gray +(red filter on left eye, blue filter on right eye) +

+
+
‘argg’
+

anaglyph red/green gray +(red filter on left eye, green filter on right eye) +

+
+
‘arcg’
+

anaglyph red/cyan gray +(red filter on left eye, cyan filter on right eye) +

+
+
‘arch’
+

anaglyph red/cyan half colored +(red filter on left eye, cyan filter on right eye) +

+
+
‘arcc’
+

anaglyph red/cyan color +(red filter on left eye, cyan filter on right eye) +

+
+
‘arcd’
+

anaglyph red/cyan color optimized with the least squares projection of dubois +(red filter on left eye, cyan filter on right eye) +

+
+
‘agmg’
+

anaglyph green/magenta gray +(green filter on left eye, magenta filter on right eye) +

+
+
‘agmh’
+

anaglyph green/magenta half colored +(green filter on left eye, magenta filter on right eye) +

+
+
‘agmc’
+

anaglyph green/magenta colored +(green filter on left eye, magenta filter on right eye) +

+
+
‘agmd’
+

anaglyph green/magenta color optimized with the least squares projection of dubois +(green filter on left eye, magenta filter on right eye) +

+
+
‘aybg’
+

anaglyph yellow/blue gray +(yellow filter on left eye, blue filter on right eye) +

+
+
‘aybh’
+

anaglyph yellow/blue half colored +(yellow filter on left eye, blue filter on right eye) +

+
+
‘aybc’
+

anaglyph yellow/blue colored +(yellow filter on left eye, blue filter on right eye) +

+
+
‘aybd’
+

anaglyph yellow/blue color optimized with the least squares projection of dubois +(yellow filter on left eye, blue filter on right eye) +

+
+
‘ml’
+

mono output (left eye only) +

+
+
‘mr’
+

mono output (right eye only) +

+
+
‘chl’
+

checkerboard, left eye first +

+
+
‘chr’
+

checkerboard, right eye first +

+
+
‘icl’
+

interleaved columns, left eye first +

+
+
‘icr’
+

interleaved columns, right eye first +

+
+
‘hdmi’
+

HDMI frame pack +

+
+ +

Default value is ‘arcd’. +

+
+ +

11.243.1 Examples

+ +
    +
  • Convert input video from side by side parallel to anaglyph yellow/blue dubois: +
    +
    stereo3d=sbsl:aybd
    +
    + +
  • Convert input video from above below (left eye above, right eye below) to side by side crosseye. +
    +
    stereo3d=abl:sbsr
    +
    +
+ +

11.244 streamselect, astreamselect

+

Select video or audio streams. +

+

The filter accepts the following options: +

+
+
inputs
+

Set number of inputs. Default is 2. +

+
+
map
+

Set input indexes to remap to outputs. +

+
+ +

11.244.1 Commands

+ +

The streamselect and astreamselect filter supports the following +commands: +

+
+
map
+

Set input indexes to remap to outputs. +

+
+ +

11.244.2 Examples

+ +
    +
  • Select first 5 seconds 1st stream and rest of time 2nd stream: +
    +
    sendcmd='5.0 streamselect map 1',streamselect=inputs=2:map=0
    +
    + +
  • Same as above, but for audio: +
    +
    asendcmd='5.0 astreamselect map 1',astreamselect=inputs=2:map=0
    +
    +
+ +

11.245 subtitles

+ +

Draw subtitles on top of input video using the libass library. +

+

To enable compilation of this filter you need to configure FFmpeg with +--enable-libass. This filter also requires a build with libavcodec and +libavformat to convert the passed subtitles file to ASS (Advanced Substation +Alpha) subtitles format. +

+

The filter accepts the following options: +

+
+
filename, f
+

Set the filename of the subtitle file to read. It must be specified. +

+
+
original_size
+

Specify the size of the original video, the video for which the ASS file +was composed. For the syntax of this option, check the +(ffmpeg-utils)"Video size" section in the ffmpeg-utils manual. +Due to a misdesign in ASS aspect ratio arithmetic, this is necessary to +correctly scale the fonts if the aspect ratio has been changed. +

+
+
fontsdir
+

Set a directory path containing fonts that can be used by the filter. +These fonts will be used in addition to whatever the font provider uses. +

+
+
alpha
+

Process alpha channel, by default alpha channel is untouched. +

+
+
charenc
+

Set subtitles input character encoding. subtitles filter only. Only +useful if not UTF-8. +

+
+
stream_index, si
+

Set subtitles stream index. subtitles filter only. +

+
+
force_style
+

Override default style or script info parameters of the subtitles. It accepts a +string containing ASS style format KEY=VALUE couples separated by ",". +

+
+
wrap_unicode
+

Break lines according to the Unicode Line Breaking Algorithm. Availability requires +at least libass release 0.17.0 (or LIBASS_VERSION 0x01600010), and libass must +have been built with libunibreak. +

+

The option is enabled by default except for native ASS. +

+
+ +

If the first key is not specified, it is assumed that the first value +specifies the filename. +

+

For example, to render the file sub.srt on top of the input +video, use the command: +

+
subtitles=sub.srt
+
+ +

which is equivalent to: +

+
subtitles=filename=sub.srt
+
+ +

To render the default subtitles stream from file video.mkv, use: +

+
subtitles=video.mkv
+
+ +

To render the second subtitles stream from that file, use: +

+
subtitles=video.mkv:si=1
+
+ +

To make the subtitles stream from sub.srt appear in 80% transparent blue +DejaVu Serif, use: +

+
subtitles=sub.srt:force_style='Fontname=DejaVu Serif,PrimaryColour=&HCCFF0000'
+
+ +

11.246 super2xsai

+ +

Scale the input by 2x and smooth using the Super2xSaI (Scale and +Interpolate) pixel art scaling algorithm. +

+

Useful for enlarging pixel art images without reducing sharpness. +

+

11.247 swaprect

+ +

Swap two rectangular objects in video. +

+

This filter accepts the following options: +

+
+
w
+

Set object width. +

+
+
h
+

Set object height. +

+
+
x1
+

Set 1st rect x coordinate. +

+
+
y1
+

Set 1st rect y coordinate. +

+
+
x2
+

Set 2nd rect x coordinate. +

+
+
y2
+

Set 2nd rect y coordinate. +

+

All expressions are evaluated once for each frame. +

+
+ +

The all options are expressions containing the following constants: +

+
+
w
+
h
+

The input width and height. +

+
+
a
+

same as w / h +

+
+
sar
+

input sample aspect ratio +

+
+
dar
+

input display aspect ratio, it is the same as (w / h) * sar +

+
+
n
+

The number of the input frame, starting from 0. +

+
+
t
+

The timestamp expressed in seconds. It’s NAN if the input timestamp is unknown. +

+
+
pos
+

the position in the file of the input frame, NAN if unknown; deprecated, +do not use +

+
+ +

11.247.1 Commands

+ +

This filter supports the all above options as commands. +

+

11.248 swapuv

+

Swap U & V plane. +

+

11.249 tblend

+

Blend successive video frames. +

+

See blend +

+

11.250 telecine

+ +

Apply telecine process to the video. +

+

This filter accepts the following options: +

+
+
first_field
+
+
‘top, t’
+

top field first +

+
‘bottom, b’
+

bottom field first +The default value is top. +

+
+ +
+
pattern
+

A string of numbers representing the pulldown pattern you wish to apply. +The default value is 23. +

+
+ +
+
Some typical patterns:
+
+NTSC output (30i):
+27.5p: 32222
+24p: 23 (classic)
+24p: 2332 (preferred)
+20p: 33
+18p: 334
+16p: 3444
+
+PAL output (25i):
+27.5p: 12222
+24p: 222222222223 ("Euro pulldown")
+16.67p: 33
+16p: 33333334
+
+ +

11.251 thistogram

+ +

Compute and draw a color distribution histogram for the input video across time. +

+

Unlike histogram video filter which only shows histogram of single input frame +at certain time, this filter shows also past histograms of number of frames defined +by width option. +

+

The computed histogram is a representation of the color component +distribution in an image. +

+

The filter accepts the following options: +

+
+
width, w
+

Set width of single color component output. Default value is 0. +Value of 0 means width will be picked from input video. +This also set number of passed histograms to keep. +Allowed range is [0, 8192]. +

+
+
display_mode, d
+

Set display mode. +It accepts the following values: +

+
‘stack’
+

Per color component graphs are placed below each other. +

+
+
‘parade’
+

Per color component graphs are placed side by side. +

+
+
‘overlay’
+

Presents information identical to that in the parade, except +that the graphs representing color components are superimposed directly +over one another. +

+
+

Default is stack. +

+
+
levels_mode, m
+

Set mode. Can be either linear, or logarithmic. +Default is linear. +

+
+
components, c
+

Set what color components to display. +Default is 7. +

+
+
bgopacity, b
+

Set background opacity. Default is 0.9. +

+
+
envelope, e
+

Show envelope. Default is disabled. +

+
+
ecolor, ec
+

Set envelope color. Default is gold. +

+
+
slide
+

Set slide mode. +

+

Available values for slide is: +

+
‘frame’
+

Draw new frame when right border is reached. +

+
+
‘replace’
+

Replace old columns with new ones. +

+
+
‘scroll’
+

Scroll from right to left. +

+
+
‘rscroll’
+

Scroll from left to right. +

+
+
‘picture’
+

Draw single picture. +

+
+ +

Default is replace. +

+
+ +

11.252 threshold

+ +

Apply threshold effect to video stream. +

+

This filter needs four video streams to perform thresholding. +First stream is stream we are filtering. +Second stream is holding threshold values, third stream is holding min values, +and last, fourth stream is holding max values. +

+

The filter accepts the following option: +

+
+
planes
+

Set which planes will be processed, unprocessed planes will be copied. +By default value 0xf, all planes will be processed. +

+
+ +

For example if first stream pixel’s component value is less then threshold value +of pixel component from 2nd threshold stream, third stream value will picked, +otherwise fourth stream pixel component value will be picked. +

+

Using color source filter one can perform various types of thresholding: +

+

11.252.1 Commands

+ +

This filter supports the all options as commands. +

+

11.252.2 Examples

+ +
    +
  • Binary threshold, using gray color as threshold: +
    +
    ffmpeg -i 320x240.avi -f lavfi -i color=gray -f lavfi -i color=black -f lavfi -i color=white -lavfi threshold output.avi
    +
    + +
  • Inverted binary threshold, using gray color as threshold: +
    +
    ffmpeg -i 320x240.avi -f lavfi -i color=gray -f lavfi -i color=white -f lavfi -i color=black -lavfi threshold output.avi
    +
    + +
  • Truncate binary threshold, using gray color as threshold: +
    +
    ffmpeg -i 320x240.avi -f lavfi -i color=gray -i 320x240.avi -f lavfi -i color=gray -lavfi threshold output.avi
    +
    + +
  • Threshold to zero, using gray color as threshold: +
    +
    ffmpeg -i 320x240.avi -f lavfi -i color=gray -f lavfi -i color=white -i 320x240.avi -lavfi threshold output.avi
    +
    + +
  • Inverted threshold to zero, using gray color as threshold: +
    +
    ffmpeg -i 320x240.avi -f lavfi -i color=gray -i 320x240.avi -f lavfi -i color=white -lavfi threshold output.avi
    +
    +
+ +

11.253 thumbnail

+

Select the most representative frame in a given sequence of consecutive frames. +

+

The filter accepts the following options: +

+
+
n
+

Set the frames batch size to analyze; in a set of n frames, the filter +will pick one of them, and then handle the next batch of n frames until +the end. Default is 100. +

+
+
log
+

Set the log level to display picked frame stats. +Default is info. +

+
+ +

Since the filter keeps track of the whole frames sequence, a bigger n +value will result in a higher memory usage, so a high value is not recommended. +

+

11.253.1 Examples

+ +
    +
  • Extract one picture each 50 frames: +
    +
    thumbnail=50
    +
    + +
  • Complete example of a thumbnail creation with ffmpeg: +
    +
    ffmpeg -i in.avi -vf thumbnail,scale=300:200 -frames:v 1 out.png
    +
    +
+ +

11.254 tile

+ +

Tile several successive frames together. +

+

The untile filter can do the reverse. +

+

The filter accepts the following options: +

+
+
layout
+

Set the grid size in the form COLUMNSxROWS. Range is up to UINT_MAX cells. +Default is 6x5. +

+
+
nb_frames
+

Set the maximum number of frames to render in the given area. It must be less +than or equal to wxh. The default value is 0, meaning all +the area will be used. +

+
+
margin
+

Set the outer border margin in pixels. Range is 0 to 1024. Default is 0. +

+
+
padding
+

Set the inner border thickness (i.e. the number of pixels between frames). For +more advanced padding options (such as having different values for the edges), +refer to the pad video filter. Range is 0 to 1024. Default is 0. +

+
+
color
+

Specify the color of the unused area. For the syntax of this option, check the +(ffmpeg-utils)"Color" section in the ffmpeg-utils manual. +The default value of color is "black". +

+
+
overlap
+

Set the number of frames to overlap when tiling several successive frames together. +The value must be between 0 and nb_frames - 1. Default is 0. +

+
+
init_padding
+

Set the number of frames to initially be empty before displaying first output frame. +This controls how soon will one get first output frame. +The value must be between 0 and nb_frames - 1. Default is 0. +

+
+ +

11.254.1 Examples

+ +
    +
  • Produce 8x8 PNG tiles of all keyframes (-skip_frame nokey) in a movie: +
    +
    ffmpeg -skip_frame nokey -i file.avi -vf 'scale=128:72,tile=8x8' -an -vsync 0 keyframes%03d.png
    +
    +

    The -vsync 0 is necessary to prevent ffmpeg from +duplicating each output frame to accommodate the originally detected frame +rate. +

    +
  • Display 5 pictures in an area of 3x2 frames, +with 7 pixels between them, and 2 pixels of initial margin, using +mixed flat and named options: +
    +
    tile=3x2:nb_frames=5:padding=7:margin=2
    +
    +
+ +

11.255 tiltandshift

+

Apply tilt-and-shift effect. +

+

What happens when you invert time and space? +

+

Normally a video is composed of several frames that represent a different +instant of time and shows a scene that evolves in the space captured by the +frame. This filter is the antipode of that concept, taking inspiration from +tilt and shift photography. +

+

A filtered frame contains the whole timeline of events composing the sequence, +and this is obtained by placing a slice of pixels from each frame into a single +one. However, since there are no infinite-width frames, this is done up the +width of the input frame, and a video is recomposed by shifting away one +column for each subsequent frame. In order to map space to time, the filter +tilts each input frame as well, so that motion is preserved. This is accomplished +by progressively selecting a different column from each input frame. +

+

The end result is a sort of inverted parallax, so that far away objects move +much faster that the ones in the front. The ideal conditions for this video +effect are when there is either very little motion and the background is static, +or when there is a lot of motion and a very wide depth of field (e.g. wide +panorama, while moving on a train). +

+

The filter accepts the following parameters: +

+
+
tilt
+

Tilt video while shifting (default). When unset, video will be sliding a +static image, composed of the first column of each frame. +

+
+
start
+

What to do at the start of filtering (see below). +

+
+
end
+

What to do at the end of filtering (see below). +

+
+
hold
+

How many columns should pass through before start of filtering. +

+
+
pad
+

How many columns should be inserted before end of filtering. +

+
+
+ +

Normally the filter shifts and tilts from the very first frame, and stops when +the last one is received. However, before filtering starts, normal video may +be preserved, so that the effect is slowly shifted in its place. Similarly, +the last video frame may be reconstructed at the end. Alternatively it is +possible to just start and end with black. +

+
+
‘none’
+

Filtering starts immediately and ends when the last frame is received. +

+
+
‘frame’
+

The first frames or the very last frame are kept intact during processing. +

+
+
‘black’
+

Black is padded at the beginning or at the end of filtering. +

+
+
+ +

11.256 tinterlace

+ +

Perform various types of temporal field interlacing. +

+

Frames are counted starting from 1, so the first input frame is +considered odd. +

+

The filter accepts the following options: +

+
+
mode
+

Specify the mode of the interlacing. This option can also be specified +as a value alone. See below for a list of values for this option. +

+

Available values are: +

+
+
‘merge, 0’
+

Move odd frames into the upper field, even into the lower field, +generating a double height frame at half frame rate. +

+
 ------> time
+Input:
+Frame 1         Frame 2         Frame 3         Frame 4
+
+11111           22222           33333           44444
+11111           22222           33333           44444
+11111           22222           33333           44444
+11111           22222           33333           44444
+
+Output:
+11111                           33333
+22222                           44444
+11111                           33333
+22222                           44444
+11111                           33333
+22222                           44444
+11111                           33333
+22222                           44444
+
+ +
+
‘drop_even, 1’
+

Only output odd frames, even frames are dropped, generating a frame with +unchanged height at half frame rate. +

+
+
 ------> time
+Input:
+Frame 1         Frame 2         Frame 3         Frame 4
+
+11111           22222           33333           44444
+11111           22222           33333           44444
+11111           22222           33333           44444
+11111           22222           33333           44444
+
+Output:
+11111                           33333
+11111                           33333
+11111                           33333
+11111                           33333
+
+ +
+
‘drop_odd, 2’
+

Only output even frames, odd frames are dropped, generating a frame with +unchanged height at half frame rate. +

+
+
 ------> time
+Input:
+Frame 1         Frame 2         Frame 3         Frame 4
+
+11111           22222           33333           44444
+11111           22222           33333           44444
+11111           22222           33333           44444
+11111           22222           33333           44444
+
+Output:
+                22222                           44444
+                22222                           44444
+                22222                           44444
+                22222                           44444
+
+ +
+
‘pad, 3’
+

Expand each frame to full height, but pad alternate lines with black, +generating a frame with double height at the same input frame rate. +

+
+
 ------> time
+Input:
+Frame 1         Frame 2         Frame 3         Frame 4
+
+11111           22222           33333           44444
+11111           22222           33333           44444
+11111           22222           33333           44444
+11111           22222           33333           44444
+
+Output:
+11111           .....           33333           .....
+.....           22222           .....           44444
+11111           .....           33333           .....
+.....           22222           .....           44444
+11111           .....           33333           .....
+.....           22222           .....           44444
+11111           .....           33333           .....
+.....           22222           .....           44444
+
+ + +
+
‘interleave_top, 4’
+

Interleave the upper field from odd frames with the lower field from +even frames, generating a frame with unchanged height at half frame rate. +

+
+
 ------> time
+Input:
+Frame 1         Frame 2         Frame 3         Frame 4
+
+11111<-         22222           33333<-         44444
+11111           22222<-         33333           44444<-
+11111<-         22222           33333<-         44444
+11111           22222<-         33333           44444<-
+
+Output:
+11111                           33333
+22222                           44444
+11111                           33333
+22222                           44444
+
+ + +
+
‘interleave_bottom, 5’
+

Interleave the lower field from odd frames with the upper field from +even frames, generating a frame with unchanged height at half frame rate. +

+
+
 ------> time
+Input:
+Frame 1         Frame 2         Frame 3         Frame 4
+
+11111           22222<-         33333           44444<-
+11111<-         22222           33333<-         44444
+11111           22222<-         33333           44444<-
+11111<-         22222           33333<-         44444
+
+Output:
+22222                           44444
+11111                           33333
+22222                           44444
+11111                           33333
+
+ + +
+
‘interlacex2, 6’
+

Double frame rate with unchanged height. Frames are inserted each +containing the second temporal field from the previous input frame and +the first temporal field from the next input frame. This mode relies on +the top_field_first flag. Useful for interlaced video displays with no +field synchronisation. +

+
+
 ------> time
+Input:
+Frame 1         Frame 2         Frame 3         Frame 4
+
+11111           22222           33333           44444
+ 11111           22222           33333           44444
+11111           22222           33333           44444
+ 11111           22222           33333           44444
+
+Output:
+11111   22222   22222   33333   33333   44444   44444
+ 11111   11111   22222   22222   33333   33333   44444
+11111   22222   22222   33333   33333   44444   44444
+ 11111   11111   22222   22222   33333   33333   44444
+
+ + +
+
‘mergex2, 7’
+

Move odd frames into the upper field, even into the lower field, +generating a double height frame at same frame rate. +

+
+
 ------> time
+Input:
+Frame 1         Frame 2         Frame 3         Frame 4
+
+11111           22222           33333           44444
+11111           22222           33333           44444
+11111           22222           33333           44444
+11111           22222           33333           44444
+
+Output:
+11111           33333           33333           55555
+22222           22222           44444           44444
+11111           33333           33333           55555
+22222           22222           44444           44444
+11111           33333           33333           55555
+22222           22222           44444           44444
+11111           33333           33333           55555
+22222           22222           44444           44444
+
+ +
+
+ +

Numeric values are deprecated but are accepted for backward +compatibility reasons. +

+

Default mode is merge. +

+
+
flags
+

Specify flags influencing the filter process. +

+

Available value for flags is: +

+
+
low_pass_filter, vlpf
+

Enable linear vertical low-pass filtering in the filter. +Vertical low-pass filtering is required when creating an interlaced +destination from a progressive source which contains high-frequency +vertical detail. Filtering will reduce interlace ’twitter’ and Moire +patterning. +

+
+
complex_filter, cvlpf
+

Enable complex vertical low-pass filtering. +This will slightly less reduce interlace ’twitter’ and Moire +patterning but better retain detail and subjective sharpness impression. +

+
+
bypass_il
+

Bypass already interlaced frames, only adjust the frame rate. +

+
+ +

Vertical low-pass filtering and bypassing already interlaced frames can only be +enabled for mode interleave_top and interleave_bottom. +

+
+
+ +

11.257 tmedian

+

Pick median pixels from several successive input video frames. +

+

The filter accepts the following options: +

+
+
radius
+

Set radius of median filter. +Default is 1. Allowed range is from 1 to 127. +

+
+
planes
+

Set which planes to filter. Default value is 15, by which all planes are processed. +

+
+
percentile
+

Set median percentile. Default value is 0.5. +Default value of 0.5 will pick always median values, while 0 will pick +minimum values, and 1 maximum values. +

+
+ +

11.257.1 Commands

+ +

This filter supports all above options as commands, excluding option radius. +

+

11.258 tmidequalizer

+ +

Apply Temporal Midway Video Equalization effect. +

+

Midway Video Equalization adjusts a sequence of video frames to have the same +histograms, while maintaining their dynamics as much as possible. It’s +useful for e.g. matching exposures from a video frames sequence. +

+

This filter accepts the following option: +

+
+
radius
+

Set filtering radius. Default is 5. Allowed range is from 1 to 127. +

+
+
sigma
+

Set filtering sigma. Default is 0.5. This controls strength of filtering. +Setting this option to 0 effectively does nothing. +

+
+
planes
+

Set which planes to process. Default is 15, which is all available planes. +

+
+ +

11.259 tmix

+ +

Mix successive video frames. +

+

A description of the accepted options follows. +

+
+
frames
+

The number of successive frames to mix. If unspecified, it defaults to 3. +

+
+
weights
+

Specify weight of each input video frame. +Each weight is separated by space. If number of weights is smaller than +number of frames last specified weight will be used for all remaining +unset weights. +

+
+
scale
+

Specify scale, if it is set it will be multiplied with sum +of each weight multiplied with pixel values to give final destination +pixel value. By default scale is auto scaled to sum of weights. +

+
+
planes
+

Set which planes to filter. Default is all. Allowed range is from 0 to 15. +

+
+ +

11.259.1 Examples

+ +
    +
  • Average 7 successive frames: +
    +
    tmix=frames=7:weights="1 1 1 1 1 1 1"
    +
    + +
  • Apply simple temporal convolution: +
    +
    tmix=frames=3:weights="-1 3 -1"
    +
    + +
  • Similar as above but only showing temporal differences: +
    +
    tmix=frames=3:weights="-1 2 -1":scale=1
    +
    +
+ +

11.259.2 Commands

+ +

This filter supports the following commands: +

+
weights
+
scale
+
planes
+

Syntax is same as option with same name. +

+
+ +

11.260 tonemap

+

Tone map colors from different dynamic ranges. +

+

This filter expects data in single precision floating point, as it needs to +operate on (and can output) out-of-range values. Another filter, such as +zscale, is needed to convert the resulting frame to a usable format. +

+

The tonemapping algorithms implemented only work on linear light, so input +data should be linearized beforehand (and possibly correctly tagged). +

+
+
ffmpeg -i INPUT -vf zscale=transfer=linear,tonemap=clip,zscale=transfer=bt709,format=yuv420p OUTPUT
+
+ +

11.260.1 Options

+

The filter accepts the following options. +

+
+
tonemap
+

Set the tone map algorithm to use. +

+

Possible values are: +

+
none
+

Do not apply any tone map, only desaturate overbright pixels. +

+
+
clip
+

Hard-clip any out-of-range values. Use it for perfect color accuracy for +in-range values, while distorting out-of-range values. +

+
+
linear
+

Stretch the entire reference gamut to a linear multiple of the display. +

+
+
gamma
+

Fit a logarithmic transfer between the tone curves. +

+
+
reinhard
+

Preserve overall image brightness with a simple curve, using nonlinear +contrast, which results in flattening details and degrading color accuracy. +

+
+
hable
+

Preserve both dark and bright details better than reinhard, at the cost +of slightly darkening everything. Use it when detail preservation is more +important than color and brightness accuracy. +

+
+
mobius
+

Smoothly map out-of-range values, while retaining contrast and colors for +in-range material as much as possible. Use it when color accuracy is more +important than detail preservation. +

+
+ +

Default is none. +

+
+
param
+

Tune the tone mapping algorithm. +

+

This affects the following algorithms: +

+
none
+

Ignored. +

+
+
linear
+

Specifies the scale factor to use while stretching. +Default to 1.0. +

+
+
gamma
+

Specifies the exponent of the function. +Default to 1.8. +

+
+
clip
+

Specify an extra linear coefficient to multiply into the signal before clipping. +Default to 1.0. +

+
+
reinhard
+

Specify the local contrast coefficient at the display peak. +Default to 0.5, which means that in-gamut values will be about half as bright +as when clipping. +

+
+
hable
+

Ignored. +

+
+
mobius
+

Specify the transition point from linear to mobius transform. Every value +below this point is guaranteed to be mapped 1:1. The higher the value, the +more accurate the result will be, at the cost of losing bright details. +Default to 0.3, which due to the steep initial slope still preserves in-range +colors fairly accurately. +

+
+ +
+
desat
+

Apply desaturation for highlights that exceed this level of brightness. The +higher the parameter, the more color information will be preserved. This +setting helps prevent unnaturally blown-out colors for super-highlights, by +(smoothly) turning into white instead. This makes images feel more natural, +at the cost of reducing information about out-of-range colors. +

+

The default of 2.0 is somewhat conservative and will mostly just apply to +skies or directly sunlit surfaces. A setting of 0.0 disables this option. +

+

This option works only if the input frame has a supported color tag. +

+
+
peak
+

Override signal/nominal/reference peak with this value. Useful when the +embedded peak information in display metadata is not reliable or when tone +mapping from a lower range to a higher range. +

+
+ +

11.261 tpad

+ +

Temporarily pad video frames. +

+

The filter accepts the following options: +

+
+
start
+

Specify number of delay frames before input video stream. Default is 0. +

+
+
stop
+

Specify number of padding frames after input video stream. +Set to -1 to pad indefinitely. Default is 0. +

+
+
start_mode
+

Set kind of frames added to beginning of stream. +Can be either add or clone. +With add frames of solid-color are added. +With clone frames are clones of first frame. +Default is add. +

+
+
stop_mode
+

Set kind of frames added to end of stream. +Can be either add or clone. +With add frames of solid-color are added. +With clone frames are clones of last frame. +Default is add. +

+
+
start_duration, stop_duration
+

Specify the duration of the start/stop delay. See +(ffmpeg-utils)the Time duration section in the ffmpeg-utils(1) manual +for the accepted syntax. +These options override start and stop. Default is 0. +

+
+
color
+

Specify the color of the padded area. For the syntax of this option, +check the (ffmpeg-utils)"Color" section in the ffmpeg-utils +manual. +

+

The default value of color is "black". +

+
+ +

11.262 transpose

+ +

Transpose rows with columns in the input video and optionally flip it. +

+

It accepts the following parameters: +

+
+
dir
+

Specify the transposition direction. +

+

Can assume the following values: +

+
‘0, 4, cclock_flip’
+

Rotate by 90 degrees counterclockwise and vertically flip (default), that is: +

+
L.R     L.l
+. . ->  . .
+l.r     R.r
+
+ +
+
‘1, 5, clock’
+

Rotate by 90 degrees clockwise, that is: +

+
L.R     l.L
+. . ->  . .
+l.r     r.R
+
+ +
+
‘2, 6, cclock’
+

Rotate by 90 degrees counterclockwise, that is: +

+
L.R     R.r
+. . ->  . .
+l.r     L.l
+
+ +
+
‘3, 7, clock_flip’
+

Rotate by 90 degrees clockwise and vertically flip, that is: +

+
L.R     r.R
+. . ->  . .
+l.r     l.L
+
+
+
+ +

For values between 4-7, the transposition is only done if the input +video geometry is portrait and not landscape. These values are +deprecated, the passthrough option should be used instead. +

+

Numerical values are deprecated, and should be dropped in favor of +symbolic constants. +

+
+
passthrough
+

Do not apply the transposition if the input geometry matches the one +specified by the specified value. It accepts the following values: +

+
‘none’
+

Always apply transposition. +

+
‘portrait’
+

Preserve portrait geometry (when height >= width). +

+
‘landscape’
+

Preserve landscape geometry (when width >= height). +

+
+ +

Default value is none. +

+
+ +

For example to rotate by 90 degrees clockwise and preserve portrait +layout: +

+
transpose=dir=1:passthrough=portrait
+
+ +

The command above can also be specified as: +

+
transpose=1:portrait
+
+ +

11.263 trim

+

Trim the input so that the output contains one continuous subpart of the input. +

+

It accepts the following parameters: +

+
start
+

Specify the time of the start of the kept section, i.e. the frame with the +timestamp start will be the first frame in the output. +

+
+
end
+

Specify the time of the first frame that will be dropped, i.e. the frame +immediately preceding the one with the timestamp end will be the last +frame in the output. +

+
+
start_pts
+

This is the same as start, except this option sets the start timestamp +in timebase units instead of seconds. +

+
+
end_pts
+

This is the same as end, except this option sets the end timestamp +in timebase units instead of seconds. +

+
+
duration
+

The maximum duration of the output in seconds. +

+
+
start_frame
+

The number of the first frame that should be passed to the output. +

+
+
end_frame
+

The number of the first frame that should be dropped. +

+
+ +

start, end, and duration are expressed as time +duration specifications; see +(ffmpeg-utils)the Time duration section in the ffmpeg-utils(1) manual +for the accepted syntax. +

+

Note that the first two sets of the start/end options and the duration +option look at the frame timestamp, while the _frame variants simply count the +frames that pass through the filter. Also note that this filter does not modify +the timestamps. If you wish for the output timestamps to start at zero, insert a +setpts filter after the trim filter. +

+

If multiple start or end options are set, this filter tries to be greedy and +keep all the frames that match at least one of the specified constraints. To keep +only the part that matches all the constraints at once, chain multiple trim +filters. +

+

The defaults are such that all the input is kept. So it is possible to set e.g. +just the end values to keep everything before the specified time. +

+

Examples: +

    +
  • Drop everything except the second minute of input: +
    +
    ffmpeg -i INPUT -vf trim=60:120
    +
    + +
  • Keep only the first second: +
    +
    ffmpeg -i INPUT -vf trim=duration=1
    +
    + +
+ +

11.264 unpremultiply

+

Apply alpha unpremultiply effect to input video stream using first plane +of second stream as alpha. +

+

Both streams must have same dimensions and same pixel format. +

+

The filter accepts the following option: +

+
+
planes
+

Set which planes will be processed, unprocessed planes will be copied. +By default value 0xf, all planes will be processed. +

+

If the format has 1 or 2 components, then luma is bit 0. +If the format has 3 or 4 components: +for RGB formats bit 0 is green, bit 1 is blue and bit 2 is red; +for YUV formats bit 0 is luma, bit 1 is chroma-U and bit 2 is chroma-V. +If present, the alpha channel is always the last bit. +

+
+
inplace
+

Do not require 2nd input for processing, instead use alpha plane from input stream. +

+
+ +

11.265 unsharp

+ +

Sharpen or blur the input video. +

+

It accepts the following parameters: +

+
+
luma_msize_x, lx
+

Set the luma matrix horizontal size. It must be an odd integer between +3 and 23. The default value is 5. +

+
+
luma_msize_y, ly
+

Set the luma matrix vertical size. It must be an odd integer between 3 +and 23. The default value is 5. +

+
+
luma_amount, la
+

Set the luma effect strength. It must be a floating point number, reasonable +values lay between -1.5 and 1.5. +

+

Negative values will blur the input video, while positive values will +sharpen it, a value of zero will disable the effect. +

+

Default value is 1.0. +

+
+
chroma_msize_x, cx
+

Set the chroma matrix horizontal size. It must be an odd integer +between 3 and 23. The default value is 5. +

+
+
chroma_msize_y, cy
+

Set the chroma matrix vertical size. It must be an odd integer +between 3 and 23. The default value is 5. +

+
+
chroma_amount, ca
+

Set the chroma effect strength. It must be a floating point number, reasonable +values lay between -1.5 and 1.5. +

+

Negative values will blur the input video, while positive values will +sharpen it, a value of zero will disable the effect. +

+

Default value is 0.0. +

+
+
alpha_msize_x, ax
+

Set the alpha matrix horizontal size. It must be an odd integer +between 3 and 23. The default value is 5. +

+
+
alpha_msize_y, ay
+

Set the alpha matrix vertical size. It must be an odd integer +between 3 and 23. The default value is 5. +

+
+
alpha_amount, aa
+

Set the alpha effect strength. It must be a floating point number, reasonable +values lay between -1.5 and 1.5. +

+

Negative values will blur the input video, while positive values will +sharpen it, a value of zero will disable the effect. +

+

Default value is 0.0. +

+
+
+ +

All parameters are optional and default to the equivalent of the +string ’5:5:1.0:5:5:0.0’. +

+

11.265.1 Examples

+ +
    +
  • Apply strong luma sharpen effect: +
    +
    unsharp=luma_msize_x=7:luma_msize_y=7:luma_amount=2.5
    +
    + +
  • Apply a strong blur of both luma and chroma parameters: +
    +
    unsharp=7:7:-2:7:7:-2
    +
    +
+ +

11.266 untile

+ +

Decompose a video made of tiled images into the individual images. +

+

The frame rate of the output video is the frame rate of the input video +multiplied by the number of tiles. +

+

This filter does the reverse of tile. +

+

The filter accepts the following options: +

+
+
layout
+

Set the grid size (i.e. the number of lines and columns). For the syntax of +this option, check the +(ffmpeg-utils)"Video size" section in the ffmpeg-utils manual. +

+
+ +

11.266.1 Examples

+ +
    +
  • Produce a 1-second video from a still image file made of 25 frames stacked +vertically, like an analogic film reel: +
    +
    ffmpeg -r 1 -i image.jpg -vf untile=1x25 movie.mkv
    +
    +
+ +

11.267 uspp

+ +

Apply ultra slow/simple postprocessing filter that compresses and decompresses +the image at several (or - in the case of quality level 8 - all) +shifts and average the results. +

+

The way this differs from the behavior of spp is that uspp actually encodes & +decodes each case with libavcodec Snow, whereas spp uses a simplified intra only 8x8 +DCT similar to MJPEG. +

+

This filter is not available in ffmpeg versions between 5.0 and 6.0. +

+

The filter accepts the following options: +

+
+
quality
+

Set quality. This option defines the number of levels for averaging. It accepts +an integer in the range 0-8. If set to 0, the filter will have no +effect. A value of 8 means the higher quality. For each increment of +that value the speed drops by a factor of approximately 2. Default value is +3. +

+
+
qp
+

Force a constant quantization parameter. If not set, the filter will use the QP +from the video stream (if available). +

+
+
codec
+

Use specified codec instead of snow. +

+
+ +

11.268 v360

+ +

Convert 360 videos between various formats. +

+

The filter accepts the following options: +

+
+
input
+
output
+

Set format of the input/output video. +

+

Available formats: +

+
+
‘e’
+
‘equirect’
+

Equirectangular projection. +

+
+
‘c3x2’
+
‘c6x1’
+
‘c1x6’
+

Cubemap with 3x2/6x1/1x6 layout. +

+

Format specific options: +

+
+
in_pad
+
out_pad
+

Set padding proportion for the input/output cubemap. Values in decimals. +

+

Example values: +

+
‘0’
+

No padding. +

+
‘0.01’
+

1% of face is padding. For example, with 1920x1280 resolution face size would be 640x640 and padding would be 3 pixels from each side. (640 * 0.01 = 6 pixels) +

+
+ +

Default value is ‘0’. +Maximum value is ‘0.1’. +

+
+
fin_pad
+
fout_pad
+

Set fixed padding for the input/output cubemap. Values in pixels. +

+

Default value is ‘0’. If greater than zero it overrides other padding options. +

+
+
in_forder
+
out_forder
+

Set order of faces for the input/output cubemap. Choose one direction for each position. +

+

Designation of directions: +

+
‘r’
+

right +

+
‘l’
+

left +

+
‘u’
+

up +

+
‘d’
+

down +

+
‘f’
+

forward +

+
‘b’
+

back +

+
+ +

Default value is ‘rludfb’. +

+
+
in_frot
+
out_frot
+

Set rotation of faces for the input/output cubemap. Choose one angle for each position. +

+

Designation of angles: +

+
‘0’
+

0 degrees clockwise +

+
‘1’
+

90 degrees clockwise +

+
‘2’
+

180 degrees clockwise +

+
‘3’
+

270 degrees clockwise +

+
+ +

Default value is ‘000000’. +

+
+ +
+
‘eac’
+

Equi-Angular Cubemap. +

+
+
‘flat’
+
‘gnomonic’
+
‘rectilinear’
+

Regular video. +

+

Format specific options: +

+
h_fov
+
v_fov
+
d_fov
+

Set output horizontal/vertical/diagonal field of view. Values in degrees. +

+

If diagonal field of view is set it overrides horizontal and vertical field of view. +

+
+
ih_fov
+
iv_fov
+
id_fov
+

Set input horizontal/vertical/diagonal field of view. Values in degrees. +

+

If diagonal field of view is set it overrides horizontal and vertical field of view. +

+
+ +
+
‘dfisheye’
+

Dual fisheye. +

+

Format specific options: +

+
h_fov
+
v_fov
+
d_fov
+

Set output horizontal/vertical/diagonal field of view. Values in degrees. +

+

If diagonal field of view is set it overrides horizontal and vertical field of view. +

+
+
ih_fov
+
iv_fov
+
id_fov
+

Set input horizontal/vertical/diagonal field of view. Values in degrees. +

+

If diagonal field of view is set it overrides horizontal and vertical field of view. +

+
+ +
+
‘barrel’
+
‘fb’
+
‘barrelsplit’
+

Facebook’s 360 formats. +

+
+
‘sg’
+

Stereographic format. +

+

Format specific options: +

+
h_fov
+
v_fov
+
d_fov
+

Set output horizontal/vertical/diagonal field of view. Values in degrees. +

+

If diagonal field of view is set it overrides horizontal and vertical field of view. +

+
+
ih_fov
+
iv_fov
+
id_fov
+

Set input horizontal/vertical/diagonal field of view. Values in degrees. +

+

If diagonal field of view is set it overrides horizontal and vertical field of view. +

+
+ +
+
‘mercator’
+

Mercator format. +

+
+
‘ball’
+

Ball format, gives significant distortion toward the back. +

+
+
‘hammer’
+

Hammer-Aitoff map projection format. +

+
+
‘sinusoidal’
+

Sinusoidal map projection format. +

+
+
‘fisheye’
+

Fisheye projection. +

+

Format specific options: +

+
h_fov
+
v_fov
+
d_fov
+

Set output horizontal/vertical/diagonal field of view. Values in degrees. +

+

If diagonal field of view is set it overrides horizontal and vertical field of view. +

+
+
ih_fov
+
iv_fov
+
id_fov
+

Set input horizontal/vertical/diagonal field of view. Values in degrees. +

+

If diagonal field of view is set it overrides horizontal and vertical field of view. +

+
+ +
+
‘pannini’
+

Pannini projection. +

+

Format specific options: +

+
h_fov
+

Set output pannini parameter. +

+
+
ih_fov
+

Set input pannini parameter. +

+
+ +
+
‘cylindrical’
+

Cylindrical projection. +

+

Format specific options: +

+
h_fov
+
v_fov
+
d_fov
+

Set output horizontal/vertical/diagonal field of view. Values in degrees. +

+

If diagonal field of view is set it overrides horizontal and vertical field of view. +

+
+
ih_fov
+
iv_fov
+
id_fov
+

Set input horizontal/vertical/diagonal field of view. Values in degrees. +

+

If diagonal field of view is set it overrides horizontal and vertical field of view. +

+
+ +
+
‘perspective’
+

Perspective projection. (output only) +

+

Format specific options: +

+
v_fov
+

Set perspective parameter. +

+
+ +
+
‘tetrahedron’
+

Tetrahedron projection. +

+
+
‘tsp’
+

Truncated square pyramid projection. +

+
+
‘he’
+
‘hequirect’
+

Half equirectangular projection. +

+
+
‘equisolid’
+

Equisolid format. +

+

Format specific options: +

+
h_fov
+
v_fov
+
d_fov
+

Set output horizontal/vertical/diagonal field of view. Values in degrees. +

+

If diagonal field of view is set it overrides horizontal and vertical field of view. +

+
+
ih_fov
+
iv_fov
+
id_fov
+

Set input horizontal/vertical/diagonal field of view. Values in degrees. +

+

If diagonal field of view is set it overrides horizontal and vertical field of view. +

+
+ +
+
‘og’
+

Orthographic format. +

+

Format specific options: +

+
h_fov
+
v_fov
+
d_fov
+

Set output horizontal/vertical/diagonal field of view. Values in degrees. +

+

If diagonal field of view is set it overrides horizontal and vertical field of view. +

+
+
ih_fov
+
iv_fov
+
id_fov
+

Set input horizontal/vertical/diagonal field of view. Values in degrees. +

+

If diagonal field of view is set it overrides horizontal and vertical field of view. +

+
+ +
+
‘octahedron’
+

Octahedron projection. +

+
+
‘cylindricalea’
+

Cylindrical Equal Area projection. +

+
+ +
+
interp
+

Set interpolation method.
+Note: more complex interpolation methods require much more memory to run. +

+

Available methods: +

+
+
‘near’
+
‘nearest’
+

Nearest neighbour. +

+
‘line’
+
‘linear’
+

Bilinear interpolation. +

+
‘lagrange9’
+

Lagrange9 interpolation. +

+
‘cube’
+
‘cubic’
+

Bicubic interpolation. +

+
‘lanc’
+
‘lanczos’
+

Lanczos interpolation. +

+
‘sp16’
+
‘spline16’
+

Spline16 interpolation. +

+
‘gauss’
+
‘gaussian’
+

Gaussian interpolation. +

+
‘mitchell’
+

Mitchell interpolation. +

+
+ +

Default value is ‘line’. +

+
+
w
+
h
+

Set the output video resolution. +

+

Default resolution depends on formats. +

+
+
in_stereo
+
out_stereo
+

Set the input/output stereo format. +

+
+
‘2d’
+

2D mono +

+
‘sbs’
+

Side by side +

+
‘tb’
+

Top bottom +

+
+ +

Default value is ‘2d’ for input and output format. +

+
+
yaw
+
pitch
+
roll
+

Set rotation for the output video. Values in degrees. +

+
+
rorder
+

Set rotation order for the output video. Choose one item for each position. +

+
+
‘y, Y’
+

yaw +

+
‘p, P’
+

pitch +

+
‘r, R’
+

roll +

+
+ +

Default value is ‘ypr’. +

+
+
h_flip
+
v_flip
+
d_flip
+

Flip the output video horizontally(swaps left-right)/vertically(swaps up-down)/in-depth(swaps back-forward). Boolean values. +

+
+
ih_flip
+
iv_flip
+

Set if input video is flipped horizontally/vertically. Boolean values. +

+
+
in_trans
+

Set if input video is transposed. Boolean value, by default disabled. +

+
+
out_trans
+

Set if output video needs to be transposed. Boolean value, by default disabled. +

+
+
h_offset
+
v_offset
+

Set output horizontal/vertical off-axis offset. Default is set to 0. +Allowed range is from -1 to 1. +

+
+
alpha_mask
+

Build mask in alpha plane for all unmapped pixels by marking them fully transparent. Boolean value, by default disabled. +

+
+
reset_rot
+

Reset rotation of output video. Boolean value, by default disabled. +

+
+ +

11.268.1 Examples

+ +
    +
  • Convert equirectangular video to cubemap with 3x2 layout and 1% padding using bicubic interpolation: +
    +
    ffmpeg -i input.mkv -vf v360=e:c3x2:cubic:out_pad=0.01 output.mkv
    +
    +
  • Extract back view of Equi-Angular Cubemap: +
    +
    ffmpeg -i input.mkv -vf v360=eac:flat:yaw=180 output.mkv
    +
    +
  • Convert transposed and horizontally flipped Equi-Angular Cubemap in side-by-side stereo format to equirectangular top-bottom stereo format: +
    +
    v360=eac:equirect:in_stereo=sbs:in_trans=1:ih_flip=1:out_stereo=tb
    +
    +
+ +

11.268.2 Commands

+ +

This filter supports subset of above options as commands. +

+

11.269 vaguedenoiser

+ +

Apply a wavelet based denoiser. +

+

It transforms each frame from the video input into the wavelet domain, +using Cohen-Daubechies-Feauveau 9/7. Then it applies some filtering to +the obtained coefficients. It does an inverse wavelet transform after. +Due to wavelet properties, it should give a nice smoothed result, and +reduced noise, without blurring picture features. +

+

This filter accepts the following options: +

+
+
threshold
+

The filtering strength. The higher, the more filtered the video will be. +Hard thresholding can use a higher threshold than soft thresholding +before the video looks overfiltered. Default value is 2. +

+
+
method
+

The filtering method the filter will use. +

+

It accepts the following values: +

+
‘hard’
+

All values under the threshold will be zeroed. +

+
+
‘soft’
+

All values under the threshold will be zeroed. All values above will be +reduced by the threshold. +

+
+
‘garrote’
+

Scales or nullifies coefficients - intermediary between (more) soft and +(less) hard thresholding. +

+
+ +

Default is garrote. +

+
+
nsteps
+

Number of times, the wavelet will decompose the picture. Picture can’t +be decomposed beyond a particular point (typically, 8 for a 640x480 +frame - as 2^9 = 512 > 480). Valid values are integers between 1 and 32. Default value is 6. +

+
+
percent
+

Partial of full denoising (limited coefficients shrinking), from 0 to 100. Default value is 85. +

+
+
planes
+

A list of the planes to process. By default all planes are processed. +

+
+
type
+

The threshold type the filter will use. +

+

It accepts the following values: +

+
‘universal’
+

Threshold used is same for all decompositions. +

+
+
‘bayes’
+

Threshold used depends also on each decomposition coefficients. +

+
+ +

Default is universal. +

+
+ +

11.270 varblur

+

Apply variable blur filter by using 2nd video stream to set blur radius. +The 2nd stream must have the same dimensions. +

+

This filter accepts the following options: +

+
+
min_r
+

Set min allowed radius. Allowed range is from 0 to 254. Default is 0. +

+
max_r
+

Set max allowed radius. Allowed range is from 1 to 255. Default is 8. +

+
planes
+

Set which planes to process. By default, all are used. +

+
+ +

The varblur filter also supports the framesync options. +

+

11.270.1 Commands

+ +

This filter supports all the above options as commands. +

+

11.271 vectorscope

+ +

Display 2 color component values in the two dimensional graph (which is called +a vectorscope). +

+

This filter accepts the following options: +

+
+
mode, m
+

Set vectorscope mode. +

+

It accepts the following values: +

+
‘gray’
+
‘tint’
+

Gray values are displayed on graph, higher brightness means more pixels have +same component color value on location in graph. This is the default mode. +

+
+
‘color’
+

Gray values are displayed on graph. Surrounding pixels values which are not +present in video frame are drawn in gradient of 2 color components which are +set by option x and y. The 3rd color component is static. +

+
+
‘color2’
+

Actual color components values present in video frame are displayed on graph. +

+
+
‘color3’
+

Similar as color2 but higher frequency of same values x and y +on graph increases value of another color component, which is luminance by +default values of x and y. +

+
+
‘color4’
+

Actual colors present in video frame are displayed on graph. If two different +colors map to same position on graph then color with higher value of component +not present in graph is picked. +

+
+
‘color5’
+

Gray values are displayed on graph. Similar to color but with 3rd color +component picked from radial gradient. +

+
+ +
+
x
+

Set which color component will be represented on X-axis. Default is 1. +

+
+
y
+

Set which color component will be represented on Y-axis. Default is 2. +

+
+
intensity, i
+

Set intensity, used by modes: gray, color, color3 and color5 for increasing brightness +of color component which represents frequency of (X, Y) location in graph. +

+
+
envelope, e
+
+
‘none’
+

No envelope, this is default. +

+
+
‘instant’
+

Instant envelope, even darkest single pixel will be clearly highlighted. +

+
+
‘peak’
+

Hold maximum and minimum values presented in graph over time. This way you +can still spot out of range values without constantly looking at vectorscope. +

+
+
‘peak+instant’
+

Peak and instant envelope combined together. +

+
+ +
+
graticule, g
+

Set what kind of graticule to draw. +

+
‘none’
+
‘green’
+
‘color’
+
‘invert’
+
+ +
+
opacity, o
+

Set graticule opacity. +

+
+
flags, f
+

Set graticule flags. +

+
+
‘white’
+

Draw graticule for white point. +

+
+
‘black’
+

Draw graticule for black point. +

+
+
‘name’
+

Draw color points short names. +

+
+ +
+
bgopacity, b
+

Set background opacity. +

+
+
lthreshold, l
+

Set low threshold for color component not represented on X or Y axis. +Values lower than this value will be ignored. Default is 0. +Note this value is multiplied with actual max possible value one pixel component +can have. So for 8-bit input and low threshold value of 0.1 actual threshold +is 0.1 * 255 = 25. +

+
+
hthreshold, h
+

Set high threshold for color component not represented on X or Y axis. +Values higher than this value will be ignored. Default is 1. +Note this value is multiplied with actual max possible value one pixel component +can have. So for 8-bit input and high threshold value of 0.9 actual threshold +is 0.9 * 255 = 230. +

+
+
colorspace, c
+

Set what kind of colorspace to use when drawing graticule. +

+
‘auto’
+
‘601’
+
‘709’
+
+

Default is auto. +

+
+
tint0, t0
+
tint1, t1
+

Set color tint for gray/tint vectorscope mode. By default both options are zero. +This means no tint, and output will remain gray. +

+
+ +

11.272 vidstabdetect

+ +

Analyze video stabilization/deshaking. Perform pass 1 of 2, see +vidstabtransform for pass 2. +

+

This filter generates a file with relative translation and rotation +transform information about subsequent frames, which is then used by +the vidstabtransform filter. +

+

To enable compilation of this filter you need to configure FFmpeg with +--enable-libvidstab. +

+

This filter accepts the following options: +

+
+
result
+

Set the path to the file used to write the transforms information. +Default value is transforms.trf. +

+
+
shakiness
+

Set how shaky the video is and how quick the camera is. It accepts an +integer in the range 1-10, a value of 1 means little shakiness, a +value of 10 means strong shakiness. Default value is 5. +

+
+
accuracy
+

Set the accuracy of the detection process. It must be a value in the +range 1-15. A value of 1 means low accuracy, a value of 15 means high +accuracy. Default value is 15. +

+
+
stepsize
+

Set stepsize of the search process. The region around minimum is +scanned with 1 pixel resolution. Default value is 6. +

+
+
mincontrast
+

Set minimum contrast. Below this value a local measurement field is +discarded. Must be a floating point value in the range 0-1. Default +value is 0.3. +

+
+
tripod
+

Set reference frame number for tripod mode. +

+

If enabled, the motion of the frames is compared to a reference frame +in the filtered stream, identified by the specified number. The idea +is to compensate all movements in a more-or-less static scene and keep +the camera view absolutely still. +

+

If set to 0, it is disabled. The frames are counted starting from 1. +

+
+
show
+

Show fields and transforms in the resulting frames. It accepts an +integer in the range 0-2. Default value is 0, which disables any +visualization. +

+
+
fileformat
+

Format for the transforms data file to be written. +Acceptable values are +

+
+
‘ascii’
+

Human-readable plain text +

+
+
‘binary’
+

Binary format, roughly 40% smaller than ascii. (default) +

+
+ +
+
+ +

11.272.1 Examples

+ +
    +
  • Use default values: +
    +
    vidstabdetect
    +
    + +
  • Analyze strongly shaky movie and put the results in file +mytransforms.trf: +
    +
    vidstabdetect=shakiness=10:accuracy=15:result="mytransforms.trf"
    +
    + +
  • Visualize the result of internal transformations in the resulting +video: +
    +
    vidstabdetect=show=1
    +
    + +
  • Analyze a video with medium shakiness using ffmpeg: +
    +
    ffmpeg -i input -vf vidstabdetect=shakiness=5:show=1 dummy.avi
    +
    +
+ +

11.273 vidstabtransform

+ +

Video stabilization/deshaking: pass 2 of 2, +see vidstabdetect for pass 1. +

+

Read a file with transform information for each frame and +apply/compensate them. Together with the vidstabdetect +filter this can be used to deshake videos. See also +http://public.hronopik.de/vid.stab. It is important to also use +the unsharp filter, see below. +

+

To enable compilation of this filter you need to configure FFmpeg with +--enable-libvidstab. +

+

11.273.1 Options

+ +
+
input
+

Set path to the file used to read the transforms. Default value is +transforms.trf. +

+
+
smoothing
+

Set the number of frames (value*2 + 1) used for lowpass filtering the +camera movements. Default value is 10. +

+

For example a number of 10 means that 21 frames are used (10 in the +past and 10 in the future) to smoothen the motion in the video. A +larger value leads to a smoother video, but limits the acceleration of +the camera (pan/tilt movements). 0 is a special case where a static +camera is simulated. +

+
+
optalgo
+

Set the camera path optimization algorithm. +

+

Accepted values are: +

+
‘gauss’
+

gaussian kernel low-pass filter on camera motion (default) +

+
‘avg’
+

averaging on transformations +

+
+ +
+
maxshift
+

Set maximal number of pixels to translate frames. Default value is -1, +meaning no limit. +

+
+
maxangle
+

Set maximal angle in radians (degree*PI/180) to rotate frames. Default +value is -1, meaning no limit. +

+
+
crop
+

Specify how to deal with borders that may be visible due to movement +compensation. +

+

Available values are: +

+
‘keep’
+

keep image information from previous frame (default) +

+
‘black’
+

fill the border black +

+
+ +
+
invert
+

Invert transforms if set to 1. Default value is 0. +

+
+
relative
+

Consider transforms as relative to previous frame if set to 1, +absolute if set to 0. Default value is 0. +

+
+
zoom
+

Set percentage to zoom. A positive value will result in a zoom-in +effect, a negative value in a zoom-out effect. Default value is 0 (no +zoom). +

+
+
optzoom
+

Set optimal zooming to avoid borders. +

+

Accepted values are: +

+
‘0’
+

disabled +

+
‘1’
+

optimal static zoom value is determined (only very strong movements +will lead to visible borders) (default) +

+
‘2’
+

optimal adaptive zoom value is determined (no borders will be +visible), see zoomspeed +

+
+ +

Note that the value given at zoom is added to the one calculated here. +

+
+
zoomspeed
+

Set percent to zoom maximally each frame (enabled when +optzoom is set to 2). Range is from 0 to 5, default value is +0.25. +

+
+
interpol
+

Specify type of interpolation. +

+

Available values are: +

+
‘no’
+

no interpolation +

+
‘linear’
+

linear only horizontal +

+
‘bilinear’
+

linear in both directions (default) +

+
‘bicubic’
+

cubic in both directions (slow) +

+
+ +
+
tripod
+

Enable virtual tripod mode if set to 1, which is equivalent to +relative=0:smoothing=0. Default value is 0. +

+

Use also tripod option of vidstabdetect. +

+
+
debug
+

Increase log verbosity if set to 1. Also the detected global motions +are written to the temporary file global_motions.trf. Default +value is 0. +

+
+ +

11.273.2 Examples

+ +
    +
  • Use ffmpeg for a typical stabilization with default values: +
    +
    ffmpeg -i inp.mpeg -vf vidstabtransform,unsharp=5:5:0.8:3:3:0.4 inp_stabilized.mpeg
    +
    + +

    Note the use of the unsharp filter which is always recommended. +

    +
  • Zoom in a bit more and load transform data from a given file: +
    +
    vidstabtransform=zoom=5:input="mytransforms.trf"
    +
    + +
  • Smoothen the video even more: +
    +
    vidstabtransform=smoothing=30
    +
    +
+ +

11.274 vflip

+ +

Flip the input video vertically. +

+

For example, to vertically flip a video with ffmpeg: +

+
ffmpeg -i in.avi -vf "vflip" out.avi
+
+ +

11.275 vfrdet

+ +

Detect variable frame rate video. +

+

This filter tries to detect if the input is variable or constant frame rate. +

+

At end it will output number of frames detected as having variable delta pts, +and ones with constant delta pts. +If there was frames with variable delta, than it will also show min, max and +average delta encountered. +

+

11.276 vibrance

+ +

Boost or alter saturation. +

+

The filter accepts the following options: +

+
intensity
+

Set strength of boost if positive value or strength of alter if negative value. +Default is 0. Allowed range is from -2 to 2. +

+
+
rbal
+

Set the red balance. Default is 1. Allowed range is from -10 to 10. +

+
+
gbal
+

Set the green balance. Default is 1. Allowed range is from -10 to 10. +

+
+
bbal
+

Set the blue balance. Default is 1. Allowed range is from -10 to 10. +

+
+
rlum
+

Set the red luma coefficient. +

+
+
glum
+

Set the green luma coefficient. +

+
+
blum
+

Set the blue luma coefficient. +

+
+
alternate
+

If intensity is negative and this is set to 1, colors will change, +otherwise colors will be less saturated, more towards gray. +

+
+ +

11.276.1 Commands

+ +

This filter supports the all above options as commands. +

+

11.277 vif

+ +

Obtain the average VIF (Visual Information Fidelity) between two input videos. +

+

This filter takes two input videos. +

+

Both input videos must have the same resolution and pixel format for +this filter to work correctly. Also it assumes that both inputs +have the same number of frames, which are compared one by one. +

+

The obtained average VIF score is printed through the logging system. +

+

The filter stores the calculated VIF score of each frame. +

+

This filter also supports the framesync options. +

+

In the below example the input file main.mpg being processed is compared +with the reference file ref.mpg. +

+
+
ffmpeg -i main.mpg -i ref.mpg -lavfi vif -f null -
+
+ +

11.278 vignette

+ +

Make or reverse a natural vignetting effect. +

+

The filter accepts the following options: +

+
+
angle, a
+

Set lens angle expression as a number of radians. +

+

The value is clipped in the [0,PI/2] range. +

+

Default value: "PI/5" +

+
+
x0
+
y0
+

Set center coordinates expressions. Respectively "w/2" and "h/2" +by default. +

+
+
mode
+

Set forward/backward mode. +

+

Available modes are: +

+
‘forward’
+

The larger the distance from the central point, the darker the image becomes. +

+
+
‘backward’
+

The larger the distance from the central point, the brighter the image becomes. +This can be used to reverse a vignette effect, though there is no automatic +detection to extract the lens angle and other settings (yet). It can +also be used to create a burning effect. +

+
+ +

Default value is ‘forward’. +

+
+
eval
+

Set evaluation mode for the expressions (angle, x0, y0). +

+

It accepts the following values: +

+
‘init’
+

Evaluate expressions only once during the filter initialization. +

+
+
‘frame’
+

Evaluate expressions for each incoming frame. This is way slower than the +‘init’ mode since it requires all the scalers to be re-computed, but it +allows advanced dynamic expressions. +

+
+ +

Default value is ‘init’. +

+
+
dither
+

Set dithering to reduce the circular banding effects. Default is 1 +(enabled). +

+
+
aspect
+

Set vignette aspect. This setting allows one to adjust the shape of the vignette. +Setting this value to the SAR of the input will make a rectangular vignetting +following the dimensions of the video. +

+

Default is 1/1. +

+
+ +

11.278.1 Expressions

+ +

The alpha, x0 and y0 expressions can contain the +following parameters. +

+
+
w
+
h
+

input width and height +

+
+
n
+

the number of input frame, starting from 0 +

+
+
pts
+

the PTS (Presentation TimeStamp) time of the filtered video frame, expressed in +TB units, NAN if undefined +

+
+
r
+

frame rate of the input video, NAN if the input frame rate is unknown +

+
+
t
+

the PTS (Presentation TimeStamp) of the filtered video frame, +expressed in seconds, NAN if undefined +

+
+
tb
+

time base of the input video +

+
+ + +

11.278.2 Examples

+ +
    +
  • Apply simple strong vignetting effect: +
    +
    vignette=PI/4
    +
    + +
  • Make a flickering vignetting: +
    +
    vignette='PI/4+random(1)*PI/50':eval=frame
    +
    + +
+ +

11.279 vmafmotion

+ +

Obtain the average VMAF motion score of a video. +It is one of the component metrics of VMAF. +

+

The obtained average motion score is printed through the logging system. +

+

The filter accepts the following options: +

+
+
stats_file
+

If specified, the filter will use the named file to save the motion score of +each frame with respect to the previous frame. +When filename equals "-" the data is sent to standard output. +

+
+ +

Example: +

+
ffmpeg -i ref.mpg -vf vmafmotion -f null -
+
+ +

11.280 vpp_amf

+ +

Scale (resize) and convert colorspace, transfer characteristics or color primaries for the input video, using AMD Advanced Media Framework library for hardware acceleration. +Setting the output width and height works in the same way as for the scale filter. +

+

The filter accepts the following options: +

+
w
+
h
+

Set the output video dimension expression. Default value is the input dimension. +

+

Allows for the same expressions as the scale filter. +

+
+
scale_type
+

Sets the algorithm used for scaling: +

+
+
bilinear
+

Bilinear +

+

This is the default. +

+
+
bicubic
+

Bicubic +

+
+
+ +
+
format
+

Controls the output pixel format. By default, or if none is specified, the input +pixel format is used. +

+ +
+
force_original_aspect_ratio
+
force_divisible_by
+

Work the same as the identical scale filter options. +

+
+
reset_sar
+

Works the same as the identical scale filter option. +

+
+
in_color_range
+

Override input color range. +

+
+
out_color_range
+

Specify output color range. +

+

The accepted values for in_trc and out_trc are: +

+
‘studio’
+

Studio (or restricted, or MPEG) color range. +

+
+
‘full’
+

Full (or JPEG) color range. +

+
+
+ +
+
color_profile
+

Specify all color properties at once. +

+

The accepted values are: +

+
‘bt601’
+

BT.601 +

+
+
‘bt709’
+

BT.709 +

+
+
‘bt2020’
+

BT.2020 +

+
+
+ +
+
in_trc
+

Override input transfer characteristics. +

+
+
out_trc
+

Specify output transfer characteristics. +

+

The accepted values for in_trc and out_trc are: +

+
‘bt709’
+

BT.709 +

+
+
‘gamma22’
+

Constant gamma of 2.2 +

+
+
‘gamma28’
+

Constant gamma of 2.8 +

+
+
‘smpte170m’
+

SMPTE-170M +

+
+
‘smpte240m’
+

SMPTE-240M +

+
+
‘linear’
+

Linear +

+
+
‘log’
+

LOG +

+
+
‘log-sqrt’
+

LOG_SQRT +

+
+
‘iec61966-2-4’
+

iec61966-2-4 +

+
+
‘bt1361-ecg’
+

BT1361_ECG +

+
+
‘iec61966-2-1’
+

iec61966-2-1 +

+
+
‘bt2020-10’
+

BT.2020 for 10-bits content +

+
+
‘bt2020-12’
+

BT.2020 for 12-bits content +

+
+
‘smpte2084’
+

SMPTE2084 +

+
+
‘smpte428’
+

SMPTE428 +

+
+
‘arib-std-b67’
+

ARIB_STD_B67 +

+
+
+ +
+
in_primaries
+

Override input color primaries. +

+
+
out_primaries
+

Specify output color primaries. +

+

The accepted values for in_primaries and out_primaries are: +

+
‘bt709’
+

BT.709 +

+
+
‘bt470m’
+

BT.470M +

+
+
‘bt470bg’
+

BT.470BG or BT.601-6 625 +

+
+
‘smpte170m’
+

SMPTE-170M or BT.601-6 525 +

+
+
‘smpte240m’
+

SMPTE-240M +

+
+
‘film’
+

film +

+
+
‘bt2020’
+

BT.2020 +

+
+
‘smpte428’
+

SMPTE-428 +

+
+
‘smpte431’
+

SMPTE-431 +

+
+
‘smpte432’
+

SMPTE-432 +

+
+
‘jedec-p22’
+

JEDEC P22 phosphors +

+
+
+
+
+ +

11.280.1 Examples

+ +
    +
  • Scale input to 720p, keeping aspect ratio and ensuring the output is yuv420p. +
    +
    vpp_amf=-2:720:format=yuv420p
    +
    + +
  • Upscale to 4K and change color profile to bt2020. +
    +
    vpp_amf=4096:2160:color_profile=bt2020
    +
    + +
  • Override input primaries and input transfer characteristics, change both to bt709. + +
    +
    vpp_amf=color_profile=bt2020:in_trc=smpte2084:in_primaries=bt2020:out_trc=bt709:out_primaries=bt709
    +
    +
+ +

11.281 vstack

+

Stack input videos vertically. +

+

All streams must be of same pixel format and of same width. +

+

Note that this filter is faster than using overlay and pad filter +to create same output. +

+

The filter accepts the following options: +

+
+
inputs
+

Set number of input streams. Default is 2. +

+
+
shortest
+

If set to 1, force the output to terminate when the shortest input +terminates. Default value is 0. +

+
+ +

11.282 w3fdif

+ +

Deinterlace the input video ("w3fdif" stands for "Weston 3 Field +Deinterlacing Filter"). +

+

Based on the process described by Martin Weston for BBC R&D, and +implemented based on the de-interlace algorithm written by Jim +Easterbrook for BBC R&D, the Weston 3 field deinterlacing filter +uses filter coefficients calculated by BBC R&D. +

+

This filter uses field-dominance information in frame to decide which +of each pair of fields to place first in the output. +If it gets it wrong use setfield filter before w3fdif filter. +

+

There are two sets of filter coefficients, so called "simple" +and "complex". Which set of filter coefficients is used can +be set by passing an optional parameter: +

+
+
filter
+

Set the interlacing filter coefficients. Accepts one of the following values: +

+
+
‘simple’
+

Simple filter coefficient set. +

+
‘complex’
+

More-complex filter coefficient set. +

+
+

Default value is ‘complex’. +

+
+
mode
+

The interlacing mode to adopt. It accepts one of the following values: +

+
+
frame
+

Output one frame for each frame. +

+
field
+

Output one frame for each field. +

+
+ +

The default value is field. +

+
+
parity
+

The picture field parity assumed for the input interlaced video. It accepts one +of the following values: +

+
+
tff
+

Assume the top field is first. +

+
bff
+

Assume the bottom field is first. +

+
auto
+

Enable automatic detection of field parity. +

+
+ +

The default value is auto. +If the interlacing is unknown or the decoder does not export this information, +top field first will be assumed. +

+
+
deint
+

Specify which frames to deinterlace. Accepts one of the following values: +

+
+
‘all’
+

Deinterlace all frames, +

+
‘interlaced’
+

Only deinterlace frames marked as interlaced. +

+
+ +

Default value is ‘all’. +

+
+ +

11.282.1 Commands

+

This filter supports same commands as options. +

+

11.283 waveform

+

Video waveform monitor. +

+

The waveform monitor plots color component intensity. By default luma +only. Each column of the waveform corresponds to a column of pixels in the +source video. +

+

It accepts the following options: +

+
+
mode, m
+

Can be either row, or column. Default is column. +In row mode, the graph on the left side represents color component value 0 and +the right side represents value = 255. In column mode, the top side represents +color component value = 0 and bottom side represents value = 255. +

+
+
intensity, i
+

Set intensity. Smaller values are useful to find out how many values of the same +luminance are distributed across input rows/columns. +Default value is 0.04. Allowed range is [0, 1]. +

+
+
mirror, r
+

Set mirroring mode. 0 means unmirrored, 1 means mirrored. +In mirrored mode, higher values will be represented on the left +side for row mode and at the top for column mode. Default is +1 (mirrored). +

+
+
display, d
+

Set display mode. +It accepts the following values: +

+
‘overlay’
+

Presents information identical to that in the parade, except +that the graphs representing color components are superimposed directly +over one another. +

+

This display mode makes it easier to spot relative differences or similarities +in overlapping areas of the color components that are supposed to be identical, +such as neutral whites, grays, or blacks. +

+
+
‘stack’
+

Display separate graph for the color components side by side in +row mode or one below the other in column mode. +

+
+
‘parade’
+

Display separate graph for the color components side by side in +column mode or one below the other in row mode. +

+

Using this display mode makes it easy to spot color casts in the highlights +and shadows of an image, by comparing the contours of the top and the bottom +graphs of each waveform. Since whites, grays, and blacks are characterized +by exactly equal amounts of red, green, and blue, neutral areas of the picture +should display three waveforms of roughly equal width/height. If not, the +correction is easy to perform by making level adjustments the three waveforms. +

+
+

Default is stack. +

+
+
components, c
+

Set which color components to display. Default is 1, which means only luma +or red color component if input is in RGB colorspace. If is set for example to +7 it will display all 3 (if) available color components. +

+
+
envelope, e
+
+
‘none’
+

No envelope, this is default. +

+
+
‘instant’
+

Instant envelope, minimum and maximum values presented in graph will be easily +visible even with small step value. +

+
+
‘peak’
+

Hold minimum and maximum values presented in graph across time. This way you +can still spot out of range values without constantly looking at waveforms. +

+
+
‘peak+instant’
+

Peak and instant envelope combined together. +

+
+ +
+
filter, f
+
+
‘lowpass’
+

No filtering, this is default. +

+
+
‘flat’
+

Luma and chroma combined together. +

+
+
‘aflat’
+

Similar as above, but shows difference between blue and red chroma. +

+
+
‘xflat’
+

Similar as above, but use different colors. +

+
+
‘yflat’
+

Similar as above, but again with different colors. +

+
+
‘chroma’
+

Displays only chroma. +

+
+
‘color’
+

Displays actual color value on waveform. +

+
+
‘acolor’
+

Similar as above, but with luma showing frequency of chroma values. +

+
+ +
+
graticule, g
+

Set which graticule to display. +

+
+
‘none’
+

Do not display graticule. +

+
+
‘green’
+

Display green graticule showing legal broadcast ranges. +

+
+
‘orange’
+

Display orange graticule showing legal broadcast ranges. +

+
+
‘invert’
+

Display invert graticule showing legal broadcast ranges. +

+
+ +
+
opacity, o
+

Set graticule opacity. +

+
+
flags, fl
+

Set graticule flags. +

+
+
‘numbers’
+

Draw numbers above lines. By default enabled. +

+
+
‘dots’
+

Draw dots instead of lines. +

+
+ +
+
scale, s
+

Set scale used for displaying graticule. +

+
+
‘digital’
+
‘millivolts’
+
‘ire’
+
+

Default is digital. +

+
+
bgopacity, b
+

Set background opacity. +

+
+
tint0, t0
+
tint1, t1
+

Set tint for output. +Only used with lowpass filter and when display is not overlay and input +pixel formats are not RGB. +

+
+
fitmode, fm
+

Set sample aspect ratio of video output frames. +Can be used to configure waveform so it is not +stretched too much in one of directions. +

+
+
‘none’
+

Set sample aspect ration to 1/1. +

+
‘size’
+

Set sample aspect ratio to match input size of video +

+
+

Default is ‘none’. +

+
+
input
+

Set input formats for filter to pick from. +Can be ‘all’, for selecting from all available formats, +or ‘first’, for selecting first available format. +Default is ‘first’. +

+
+ +

11.284 weave, doubleweave

+ +

The weave takes a field-based video input and join +each two sequential fields into single frame, producing a new double +height clip with half the frame rate and half the frame count. +

+

The doubleweave works same as weave but without +halving frame rate and frame count. +

+

It accepts the following option: +

+
+
first_field
+

Set first field. Available values are: +

+
+
‘top, t’
+

Set the frame as top-field-first. +

+
+
‘bottom, b’
+

Set the frame as bottom-field-first. +

+
+
+
+ +

11.284.1 Examples

+ +
    +
  • Interlace video using select and separatefields filter: +
    +
    separatefields,select=eq(mod(n,4),0)+eq(mod(n,4),3),weave
    +
    +
+ +

11.285 xbr

+

Apply the xBR high-quality magnification filter which is designed for pixel +art. It follows a set of edge-detection rules, see +https://forums.libretro.com/t/xbr-algorithm-tutorial/123. +

+

It accepts the following option: +

+
+
n
+

Set the scaling dimension: 2 for 2xBR, 3 for +3xBR and 4 for 4xBR. +Default is 3. +

+
+ +

11.286 xcorrelate

+

Apply normalized cross-correlation between first and second input video stream. +

+

Second input video stream dimensions must be lower than first input video stream. +

+

The filter accepts the following options: +

+
+
planes
+

Set which planes to process. +

+
+
secondary
+

Set which secondary video frames will be processed from second input video stream, +can be first or all. Default is all. +

+
+ +

The xcorrelate filter also supports the framesync options. +

+

11.287 xfade

+ +

Apply cross fade from one input video stream to another input video stream. +The cross fade is applied for specified duration. +

+

Both inputs must be constant frame-rate and have the same resolution, pixel format, +frame rate and timebase. +

+

The filter accepts the following options: +

+
+
transition
+

Set one of available transition effects: +

+
+
‘custom’
+
‘fade’
+
‘wipeleft’
+
‘wiperight’
+
‘wipeup’
+
‘wipedown’
+
‘slideleft’
+
‘slideright’
+
‘slideup’
+
‘slidedown’
+
‘circlecrop’
+
‘rectcrop’
+
‘distance’
+
‘fadeblack’
+
‘fadewhite’
+
‘radial’
+
‘smoothleft’
+
‘smoothright’
+
‘smoothup’
+
‘smoothdown’
+
‘circleopen’
+
‘circleclose’
+
‘vertopen’
+
‘vertclose’
+
‘horzopen’
+
‘horzclose’
+
‘dissolve’
+
‘pixelize’
+
‘diagtl’
+
‘diagtr’
+
‘diagbl’
+
‘diagbr’
+
‘hlslice’
+
‘hrslice’
+
‘vuslice’
+
‘vdslice’
+
‘hblur’
+
‘fadegrays’
+
‘wipetl’
+
‘wipetr’
+
‘wipebl’
+
‘wipebr’
+
‘squeezeh’
+
‘squeezev’
+
‘zoomin’
+
‘fadefast’
+
‘fadeslow’
+
‘hlwind’
+
‘hrwind’
+
‘vuwind’
+
‘vdwind’
+
‘coverleft’
+
‘coverright’
+
‘coverup’
+
‘coverdown’
+
‘revealleft’
+
‘revealright’
+
‘revealup’
+
‘revealdown’
+
+

Default transition effect is fade. +

+
+
duration
+

Set cross fade duration in seconds. +Range is 0 to 60 seconds. +Default duration is 1 second. +

+
+
offset
+

Set cross fade start relative to first input stream in seconds. +Default offset is 0. +

+
+
expr
+

Set expression for custom transition effect. +

+

The expressions can use the following variables and functions: +

+
+
X
+
Y
+

The coordinates of the current sample. +

+
+
W
+
H
+

The width and height of the image. +

+
+
P
+

Progress of transition effect. +

+
+
PLANE
+

Currently processed plane. +

+
+
A
+

Return value of first input at current location and plane. +

+
+
B
+

Return value of second input at current location and plane. +

+
+
a0(x, y)
+
a1(x, y)
+
a2(x, y)
+
a3(x, y)
+

Return the value of the pixel at location (x,y) of the +first/second/third/fourth component of first input. +

+
+
b0(x, y)
+
b1(x, y)
+
b2(x, y)
+
b3(x, y)
+

Return the value of the pixel at location (x,y) of the +first/second/third/fourth component of second input. +

+
+
+
+ +

11.287.1 Examples

+ +
    +
  • Cross fade from one input video to another input video, with fade transition and duration of transition +of 2 seconds starting at offset of 5 seconds: +
    +
    ffmpeg -i first.mp4 -i second.mp4 -filter_complex xfade=transition=fade:duration=2:offset=5 output.mp4
    +
    +
+ +

11.288 xmedian

+

Pick median pixels from several input videos. +

+

The filter accepts the following options: +

+
+
inputs
+

Set number of inputs. +Default is 3. Allowed range is from 3 to 255. +If number of inputs is even number, than result will be mean value between two median values. +

+
+
planes
+

Set which planes to filter. Default value is 15, by which all planes are processed. +

+
+
percentile
+

Set median percentile. Default value is 0.5. +Default value of 0.5 will pick always median values, while 0 will pick +minimum values, and 1 maximum values. +

+
+ +

11.288.1 Commands

+ +

This filter supports all above options as commands, excluding option inputs. +

+

11.289 xpsnr

+ +

Obtain the average (across all input frames) and minimum (across all color plane averages) +eXtended Perceptually weighted peak Signal-to-Noise Ratio (XPSNR) between two input videos. +

+

The XPSNR is a low-complexity psychovisually motivated distortion measurement algorithm for +assessing the difference between two video streams or images. This is especially useful for +objectively quantifying the distortions caused by video and image codecs, as an alternative +to a formal subjective test. The logarithmic XPSNR output values are in a similar range as +those of traditional psnr assessments but better reflect human impressions of visual +coding quality. More details on the XPSNR measure, which essentially represents a blockwise +weighted variant of the PSNR measure, can be found in the following freely available papers: +

+
    +
  • C. R. Helmrich, M. Siekmann, S. Becker, S. Bosse, D. Marpe, and T. Wiegand, "XPSNR: A +Low-Complexity Extension of the Perceptually Weighted Peak Signal-to-Noise Ratio for +High-Resolution Video Quality Assessment," in Proc. IEEE Int. Conf. Acoustics, Speech, +Sig. Process. (ICASSP), virt./online, May 2020. www.ecodis.de/xpsnr.htm + +
  • C. R. Helmrich, S. Bosse, H. Schwarz, D. Marpe, and T. Wiegand, "A Study of the +Extended Perceptually Weighted Peak Signal-to-Noise Ratio (XPSNR) for Video Compression +with Different Resolutions and Bit Depths," ITU Journal: ICT Discoveries, vol. 3, no. +1, pp. 65 - 72, May 2020. http://handle.itu.int/11.1002/pub/8153d78b-en +
+ +

When publishing the results of XPSNR assessments obtained using, e.g., this FFmpeg filter, a +reference to the above papers as a means of documentation is strongly encouraged. The filter +requires two input videos. The first input is considered a (usually not distorted) reference +source and is passed unchanged to the output, whereas the second input is a (distorted) test +signal. Except for the bit depth, these two video inputs must have the same pixel format. In +addition, for best performance, both compared input videos should be in YCbCr color format. +

+

The obtained overall XPSNR values mentioned above are printed through the logging system. In +case of input with multiple color planes, we suggest reporting of the minimum XPSNR average. +

+

The following parameter, which behaves like the one for the psnr filter, is accepted: +

+
+
stats_file, f
+

If specified, the filter will use the named file to save the XPSNR value of each individual +frame and color plane. When the file name equals "-", that data is sent to standard output. +

+
+ +

This filter also supports the framesync options. +

+

11.289.1 Examples

+
    +
  • XPSNR analysis of two 1080p HD videos, ref_source.yuv and test_video.yuv, both at 24 frames +per second, with color format 4:2:0, bit depth 8, and output of a logfile named "xpsnr.log": +
    +
    ffmpeg -s 1920x1080 -framerate 24 -pix_fmt yuv420p -i ref_source.yuv -s 1920x1080 -framerate
    +24 -pix_fmt yuv420p -i test_video.yuv -lavfi xpsnr="stats_file=xpsnr.log" -f null -
    +
    + +
  • XPSNR analysis of two 2160p UHD videos, ref_source.yuv with bit depth 8 and test_video.yuv +with bit depth 10, both at 60 frames per second with color format 4:2:0, no logfile output: +
    +
    ffmpeg -s 3840x2160 -framerate 60 -pix_fmt yuv420p -i ref_source.yuv -s 3840x2160 -framerate
    +60 -pix_fmt yuv420p10le -i test_video.yuv -lavfi xpsnr="stats_file=-" -f null -
    +
    +
+ +

11.290 xstack

+

Stack video inputs into custom layout. +

+

All streams must be of same pixel format. +

+

The filter accepts the following options: +

+
+
inputs
+

Set number of input streams. Default is 2. +

+
+
layout
+

Specify layout of inputs. +This option requires the desired layout configuration to be explicitly set by the user. +This sets position of each video input in output. Each input +is separated by ’|’. +The first number represents the column, and the second number represents the row. +Numbers start at 0 and are separated by ’_’. Optionally one can use wX and hX, +where X is video input from which to take width or height. +Multiple values can be used when separated by ’+’. In such +case values are summed together. +

+

Note that if inputs are of different sizes gaps may appear, as not all of +the output video frame will be filled. Similarly, videos can overlap each +other if their position doesn’t leave enough space for the full frame of +adjoining videos. +

+

For 2 inputs, a default layout of 0_0|w0_0 (equivalent to +grid=2x1) is set. In all other cases, a layout or a grid must be set by +the user. Either grid or layout can be specified at a time. +Specifying both will result in an error. +

+
+
grid
+

Specify a fixed size grid of inputs. +This option is used to create a fixed size grid of the input streams. Set the +grid size in the form COLUMNSxROWS. There must be ROWS * COLUMNS +input streams and they will be arranged as a grid with ROWS rows and +COLUMNS columns. When using this option, each input stream within a row +must have the same height and all the rows must have the same width. +

+

If grid is set, then inputs option is ignored and is implicitly +set to ROWS * COLUMNS. +

+

For 2 inputs, a default grid of 2x1 (equivalent to +layout=0_0|w0_0) is set. In all other cases, a layout or a grid must be +set by the user. Either grid or layout can be specified at a time. +Specifying both will result in an error. +

+
+
shortest
+

If set to 1, force the output to terminate when the shortest input +terminates. Default value is 0. +

+
+
fill
+

If set to valid color, all unused pixels will be filled with that color. +By default fill is set to none, so it is disabled. +

+
+ +

11.290.1 Examples

+ +
    +
  • Display 4 inputs into 2x2 grid. + +

    Layout: +

    +
    input1(0, 0)  | input3(w0, 0)
    +input2(0, h0) | input4(w0, h0)
    +
    + +
    +
    xstack=inputs=4:layout=0_0|0_h0|w0_0|w0_h0
    +
    + +

    Note that if inputs are of different sizes, gaps or overlaps may occur. +

    +
  • Display 4 inputs into 1x4 grid. + +

    Layout: +

    +
    input1(0, 0)
    +input2(0, h0)
    +input3(0, h0+h1)
    +input4(0, h0+h1+h2)
    +
    + +
    +
    xstack=inputs=4:layout=0_0|0_h0|0_h0+h1|0_h0+h1+h2
    +
    + +

    Note that if inputs are of different widths, unused space will appear. +

    +
  • Display 9 inputs into 3x3 grid. + +

    Layout: +

    +
    input1(0, 0)       | input4(w0, 0)      | input7(w0+w3, 0)
    +input2(0, h0)      | input5(w0, h0)     | input8(w0+w3, h0)
    +input3(0, h0+h1)   | input6(w0, h0+h1)  | input9(w0+w3, h0+h1)
    +
    + +
    +
    xstack=inputs=9:layout=0_0|0_h0|0_h0+h1|w0_0|w0_h0|w0_h0+h1|w0+w3_0|w0+w3_h0|w0+w3_h0+h1
    +
    + +

    Note that if inputs are of different sizes, gaps or overlaps may occur. +

    +
  • Display 16 inputs into 4x4 grid. + +

    Layout: +

    +
    input1(0, 0)       | input5(w0, 0)       | input9 (w0+w4, 0)       | input13(w0+w4+w8, 0)
    +input2(0, h0)      | input6(w0, h0)      | input10(w0+w4, h0)      | input14(w0+w4+w8, h0)
    +input3(0, h0+h1)   | input7(w0, h0+h1)   | input11(w0+w4, h0+h1)   | input15(w0+w4+w8, h0+h1)
    +input4(0, h0+h1+h2)| input8(w0, h0+h1+h2)| input12(w0+w4, h0+h1+h2)| input16(w0+w4+w8, h0+h1+h2)
    +
    + +
    +
    xstack=inputs=16:layout=0_0|0_h0|0_h0+h1|0_h0+h1+h2|w0_0|w0_h0|w0_h0+h1|w0_h0+h1+h2|w0+w4_0|
    +w0+w4_h0|w0+w4_h0+h1|w0+w4_h0+h1+h2|w0+w4+w8_0|w0+w4+w8_h0|w0+w4+w8_h0+h1|w0+w4+w8_h0+h1+h2
    +
    + +

    Note that if inputs are of different sizes, gaps or overlaps may occur. +

    +
+ +

11.291 yadif

+ +

Deinterlace the input video ("yadif" means "yet another deinterlacing +filter"). +

+

It accepts the following parameters: +

+ +
+
mode
+

The interlacing mode to adopt. It accepts one of the following values: +

+
+
0, send_frame
+

Output one frame for each frame. +

+
1, send_field
+

Output one frame for each field. +

+
2, send_frame_nospatial
+

Like send_frame, but it skips the spatial interlacing check. +

+
3, send_field_nospatial
+

Like send_field, but it skips the spatial interlacing check. +

+
+ +

The default value is send_frame. +

+
+
parity
+

The picture field parity assumed for the input interlaced video. It accepts one +of the following values: +

+
+
0, tff
+

Assume the top field is first. +

+
1, bff
+

Assume the bottom field is first. +

+
-1, auto
+

Enable automatic detection of field parity. +

+
+ +

The default value is auto. +If the interlacing is unknown or the decoder does not export this information, +top field first will be assumed. +

+
+
deint
+

Specify which frames to deinterlace. Accepts one of the following +values: +

+
+
0, all
+

Deinterlace all frames. +

+
1, interlaced
+

Only deinterlace frames marked as interlaced. +

+
+ +

The default value is all. +

+
+ +

11.292 yaepblur

+ +

Apply blur filter while preserving edges ("yaepblur" means "yet another edge preserving blur filter"). +The algorithm is described in +"J. S. Lee, Digital image enhancement and noise filtering by use of local statistics, IEEE Trans. Pattern Anal. Mach. Intell. PAMI-2, 1980." +

+

It accepts the following parameters: +

+
+
radius, r
+

Set the window radius. Default value is 3. +

+
+
planes, p
+

Set which planes to filter. Default is only the first plane. +

+
+
sigma, s
+

Set blur strength. Default value is 128. +

+
+ +

11.292.1 Commands

+

This filter supports same commands as options. +

+

11.293 zoompan

+ +

Apply Zoom & Pan effect. +

+

This filter accepts the following options: +

+
+
zoom, z
+

Set the zoom expression. Range is 1-10. Default is 1. +

+
+
x
+
y
+

Set the x and y expression. Default is 0. +

+
+
d
+

Set the duration expression in number of frames. +This sets for how many number of frames effect will last for +single input image. Default is 90. +

+
+
s
+

Set the output image size, default is ’hd720’. +

+
+
fps
+

Set the output frame rate, default is ’25’. +

+
+ +

Each expression can contain the following constants: +

+
+
in_w, iw
+

Input width. +

+
+
in_h, ih
+

Input height. +

+
+
out_w, ow
+

Output width. +

+
+
out_h, oh
+

Output height. +

+
+
in
+

Input frame count. +

+
+
on
+

Output frame count. +

+
+
in_time, it
+

The input timestamp expressed in seconds. It’s NAN if the input timestamp is unknown. +

+
+
out_time, time, ot
+

The output timestamp expressed in seconds. +

+
+
x
+
y
+

Last calculated ’x’ and ’y’ position from ’x’ and ’y’ expression +for current input frame. +

+
+
px
+
py
+

’x’ and ’y’ of last output frame of previous input frame or 0 when there was +not yet such frame (first input frame). +

+
+
zoom
+

Last calculated zoom from ’z’ expression for current input frame. +

+
+
pzoom
+

Last calculated zoom of last output frame of previous input frame. +

+
+
duration
+

Number of output frames for current input frame. Calculated from ’d’ expression +for each input frame. +

+
+
pduration
+

number of output frames created for previous input frame +

+
+
a
+

Rational number: input width / input height +

+
+
sar
+

sample aspect ratio +

+
+
dar
+

display aspect ratio +

+
+
+ +

11.293.1 Examples

+ +
    +
  • Zoom in up to 1.5x and pan at same time to some spot near center of picture: +
    +
    zoompan=z='min(zoom+0.0015,1.5)':d=700:x='if(gte(zoom,1.5),x,x+1/a)':y='if(gte(zoom,1.5),y,y+1)':s=640x360
    +
    + +
  • Zoom in up to 1.5x and pan always at center of picture: +
    +
    zoompan=z='min(zoom+0.0015,1.5)':d=700:x='iw/2-(iw/zoom/2)':y='ih/2-(ih/zoom/2)'
    +
    + +
  • Same as above but without pausing: +
    +
    zoompan=z='min(max(zoom,pzoom)+0.0015,1.5)':d=1:x='iw/2-(iw/zoom/2)':y='ih/2-(ih/zoom/2)'
    +
    + +
  • Zoom in 2x into center of picture only for the first second of the input video: +
    +
    zoompan=z='if(between(in_time,0,1),2,1)':d=1:x='iw/2-(iw/zoom/2)':y='ih/2-(ih/zoom/2)'
    +
    + +
+ +

11.294 zscale

+

Scale (resize) the input video, using the z.lib library: +https://github.com/sekrit-twc/zimg. To enable compilation of this +filter, you need to configure FFmpeg with --enable-libzimg. +

+

The zscale filter forces the output display aspect ratio to be the same +as the input, by changing the output sample aspect ratio. +

+

If the input image format is different from the format requested by +the next filter, the zscale filter will convert the input to the +requested format. +

+

11.294.1 Options

+

The filter accepts the following options. +

+
+
width, w
+
height, h
+

Set the output video dimension expression. Default value is the input +dimension. +

+

If the width or w value is 0, the input width is used for +the output. If the height or h value is 0, the input height +is used for the output. +

+

If one and only one of the values is -n with n >= 1, the zscale filter +will use a value that maintains the aspect ratio of the input image, +calculated from the other specified dimension. After that it will, +however, make sure that the calculated dimension is divisible by n and +adjust the value if necessary. +

+

If both values are -n with n >= 1, the behavior will be identical to +both values being set to 0 as previously detailed. +

+

See below for the list of accepted constants for use in the dimension +expression. +

+
+
size, s
+

Set the video size. For the syntax of this option, check the +(ffmpeg-utils)"Video size" section in the ffmpeg-utils manual. +

+
+
dither, d
+

Set the dither type. +

+

Possible values are: +

+
none
+
ordered
+
random
+
error_diffusion
+
+ +

Default is none. +

+
+
filter, f
+

Set the resize filter type. +

+

Possible values are: +

+
point
+
bilinear
+
bicubic
+
spline16
+
spline36
+
spline64
+
lanczos
+
+ +

Default is bilinear. +

+
+
range, r
+

Set the color range. +

+

Possible values are: +

+
input
+
limited
+
full
+
+ +

Default is same as input. +

+
+
primaries, p
+

Set the color primaries. +

+

Possible values are: +

+
input
+
709
+
unspecified
+
170m
+
240m
+
2020
+
+ +

Default is same as input. +

+
+
transfer, t
+

Set the transfer characteristics. +

+

Possible values are: +

+
input
+
709
+
unspecified
+
601
+
linear
+
2020_10
+
2020_12
+
smpte2084
+
iec61966-2-1
+
arib-std-b67
+
+ +

Default is same as input. +

+
+
matrix, m
+

Set the colorspace matrix. +

+

Possible value are: +

+
input
+
709
+
unspecified
+
470bg
+
170m
+
2020_ncl
+
2020_cl
+
+ +

Default is same as input. +

+
+
rangein, rin
+

Set the input color range. +

+

Possible values are: +

+
input
+
limited
+
full
+
+ +

Default is same as input. +

+
+
primariesin, pin
+

Set the input color primaries. +

+

Possible values are: +

+
input
+
709
+
unspecified
+
170m
+
240m
+
2020
+
+ +

Default is same as input. +

+
+
transferin, tin
+

Set the input transfer characteristics. +

+

Possible values are: +

+
input
+
709
+
unspecified
+
601
+
linear
+
2020_10
+
2020_12
+
+ +

Default is same as input. +

+
+
matrixin, min
+

Set the input colorspace matrix. +

+

Possible value are: +

+
input
+
709
+
unspecified
+
470bg
+
170m
+
2020_ncl
+
2020_cl
+
+ +
+
chromal, c
+

Set the output chroma location. +

+

Possible values are: +

+
input
+
left
+
center
+
topleft
+
top
+
bottomleft
+
bottom
+
+ +
+
chromalin, cin
+

Set the input chroma location. +

+

Possible values are: +

+
input
+
left
+
center
+
topleft
+
top
+
bottomleft
+
bottom
+
+ +
+
npl
+

Set the nominal peak luminance. +

+
+
param_a
+

Parameter A for scaling filters. Parameter "b" for bicubic, and the number of +filter taps for lanczos. +

+
+
param_b
+

Parameter B for scaling filters. Parameter "c" for bicubic. +

+
+ +

The values of the w and h options are expressions +containing the following constants: +

+
+
in_w
+
in_h
+

The input width and height +

+
+
iw
+
ih
+

These are the same as in_w and in_h. +

+
+
out_w
+
out_h
+

The output (scaled) width and height +

+
+
ow
+
oh
+

These are the same as out_w and out_h +

+
+
a
+

The same as iw / ih +

+
+
sar
+

input sample aspect ratio +

+
+
dar
+

The input display aspect ratio. Calculated from (iw / ih) * sar. +

+
+
hsub
+
vsub
+

horizontal and vertical input chroma subsample values. For example for the +pixel format "yuv422p" hsub is 2 and vsub is 1. +

+
+
ohsub
+
ovsub
+

horizontal and vertical output chroma subsample values. For example for the +pixel format "yuv422p" hsub is 2 and vsub is 1. +

+
+ +

11.294.2 Commands

+ +

This filter supports the following commands: +

+
width, w
+
height, h
+

Set the output video dimension expression. +The command accepts the same syntax of the corresponding option. +

+

If the specified expression is not valid, it is kept at its current +value. +

+
+ + +

12 CUDA Video Filters

+ +

To enable CUDA and/or NPP filters please refer to configuration guidelines for CUDA and for CUDA NPP filters. +

+

Running CUDA filters requires you to initialize a hardware device and to pass that device to all filters in any filter graph. +

+
-init_hw_device cuda[=name][:device[,key=value...]]
+

Initialise a new hardware device of type cuda called name, using the +given device parameters. +

+
+
-filter_hw_device name
+

Pass the hardware device called name to all filters in any filter graph. +

+
+
+ +

For more detailed information see https://www.ffmpeg.org/ffmpeg.html#Advanced-Video-options +

+
    +
  • Example of initializing second CUDA device on the system and running scale_cuda and bilateral_cuda filters. +
    +
    ./ffmpeg -hwaccel cuda -hwaccel_output_format cuda -i input.mp4 -init_hw_device cuda:1 -filter_complex \
    +"[0:v]scale_cuda=format=yuv444p[scaled_video];[scaled_video]bilateral_cuda=window_size=9:sigmaS=3.0:sigmaR=50.0" \
    +-an -sn -c:v h264_nvenc -cq 20 out.mp4
    +
    +
+ +

Since CUDA filters operate exclusively on GPU memory, frame data must sometimes be uploaded (hwupload) to hardware surfaces associated with the appropriate CUDA device before processing, and downloaded (hwdownload) back to normal memory afterward, if required. Whether hwupload or hwdownload is necessary depends on the specific workflow: +

+
    +
  • If the input frames are already in GPU memory (e.g., when using -hwaccel cuda or -hwaccel_output_format cuda), explicit use of hwupload is not needed, as the data is already in the appropriate memory space. +
  • If the input frames are in CPU memory (e.g., software-decoded frames or frames processed by CPU-based filters), it is necessary to use hwupload to transfer the data to GPU memory for CUDA processing. +
  • If the output of the CUDA filters needs to be further processed by software-based filters or saved in a format not supported by GPU-based encoders, hwdownload is required to transfer the data back to CPU memory. +
+

Note that hwupload uploads data to a surface with the same layout as the software frame, so it may be necessary to add a format filter immediately before hwupload to ensure the input is in the correct format. Similarly, hwdownload may not support all output formats, so an additional format filter may need to be inserted immediately after hwdownload in the filter graph to ensure compatibility. +

+

12.1 CUDA

+

Below is a description of the currently available Nvidia CUDA video filters. +

+

Prerequisites: +

    +
  • Install Nvidia CUDA Toolkit +
+ +

Note: If FFmpeg detects the Nvidia CUDA Toolkit during configuration, it will enable CUDA filters automatically without requiring any additional flags. If you want to explicitly enable them, use the following options: +

+
    +
  • Configure FFmpeg with --enable-cuda-nvcc --enable-nonfree. +
  • Configure FFmpeg with --enable-cuda-llvm. Additional requirement: llvm lib must be installed. +
+ +

12.1.1 bilateral_cuda

+

CUDA accelerated bilateral filter, an edge preserving filter. +This filter is mathematically accurate thanks to the use of GPU acceleration. +For best output quality, use one to one chroma subsampling, i.e. yuv444p format. +

+

The filter accepts the following options: +

+
sigmaS
+

Set sigma of gaussian function to calculate spatial weight, also called sigma space. +Allowed range is 0.1 to 512. Default is 0.1. +

+
+
sigmaR
+

Set sigma of gaussian function to calculate color range weight, also called sigma color. +Allowed range is 0.1 to 512. Default is 0.1. +

+
+
window_size
+

Set window size of the bilateral function to determine the number of neighbours to loop on. +If the number entered is even, one will be added automatically. +Allowed range is 1 to 255. Default is 1. +

+
+

12.1.1.1 Examples

+ +
    +
  • Apply the bilateral filter on a video. + +
    +
    ./ffmpeg -v verbose \
    +-hwaccel cuda -hwaccel_output_format cuda -i input.mp4  \
    +-init_hw_device cuda \
    +-filter_complex \
    +" \
    +[0:v]scale_cuda=format=yuv444p[scaled_video];
    +[scaled_video]bilateral_cuda=window_size=9:sigmaS=3.0:sigmaR=50.0" \
    +-an -sn -c:v h264_nvenc -cq 20 out.mp4
    +
    + +
+ +

12.1.2 bwdif_cuda

+ +

Deinterlace the input video using the bwdif algorithm, but implemented +in CUDA so that it can work as part of a GPU accelerated pipeline with nvdec +and/or nvenc. +

+

It accepts the following parameters: +

+
+
mode
+

The interlacing mode to adopt. It accepts one of the following values: +

+
+
0, send_frame
+

Output one frame for each frame. +

+
1, send_field
+

Output one frame for each field. +

+
+ +

The default value is send_field. +

+
+
parity
+

The picture field parity assumed for the input interlaced video. It accepts one +of the following values: +

+
+
0, tff
+

Assume the top field is first. +

+
1, bff
+

Assume the bottom field is first. +

+
-1, auto
+

Enable automatic detection of field parity. +

+
+ +

The default value is auto. +If the interlacing is unknown or the decoder does not export this information, +top field first will be assumed. +

+
+
deint
+

Specify which frames to deinterlace. Accepts one of the following +values: +

+
+
0, all
+

Deinterlace all frames. +

+
1, interlaced
+

Only deinterlace frames marked as interlaced. +

+
+ +

The default value is all. +

+
+ +

12.1.3 chromakey_cuda

+

CUDA accelerated YUV colorspace color/chroma keying. +

+

This filter works like normal chromakey filter but operates on CUDA frames. +for more details and parameters see chromakey. +

+

12.1.3.1 Examples

+ +
    +
  • Make all the green pixels in the input video transparent and use it as an overlay for another video: + +
    +
    ./ffmpeg \
    +    -hwaccel cuda -hwaccel_output_format cuda -i input_green.mp4  \
    +    -hwaccel cuda -hwaccel_output_format cuda -i base_video.mp4 \
    +    -init_hw_device cuda \
    +    -filter_complex \
    +    " \
    +        [0:v]chromakey_cuda=0x25302D:0.1:0.12:1[overlay_video]; \
    +        [1:v]scale_cuda=format=yuv420p[base]; \
    +        [base][overlay_video]overlay_cuda" \
    +    -an -sn -c:v h264_nvenc -cq 20 output.mp4
    +
    + +
  • Process two software sources, explicitly uploading the frames: + +
    +
    ./ffmpeg -init_hw_device cuda=cuda -filter_hw_device cuda \
    +    -f lavfi -i color=size=800x600:color=white,format=yuv420p \
    +    -f lavfi -i yuvtestsrc=size=200x200,format=yuv420p \
    +    -filter_complex \
    +    " \
    +        [0]hwupload[under]; \
    +        [1]hwupload,chromakey_cuda=green:0.1:0.12[over]; \
    +        [under][over]overlay_cuda" \
    +    -c:v hevc_nvenc -cq 18 -preset slow output.mp4
    +
    + +
+ +

12.1.4 colorspace_cuda

+ +

CUDA accelerated implementation of the colorspace filter. +

+

It is by no means feature complete compared to the software colorspace filter, +and at the current time only supports color range conversion between jpeg/full +and mpeg/limited range. +

+

The filter accepts the following options: +

+
+
range
+

Specify output color range. +

+

The accepted values are: +

+
‘tv’
+

TV (restricted) range +

+
+
‘mpeg’
+

MPEG (restricted) range +

+
+
‘pc’
+

PC (full) range +

+
+
‘jpeg’
+

JPEG (full) range +

+
+
+ +
+
+ +

12.1.5 overlay_cuda

+ +

Overlay one video on top of another. +

+

This is the CUDA variant of the overlay filter. +It only accepts CUDA frames. The underlying input pixel formats have to match. +

+

It takes two inputs and has one output. The first input is the "main" +video on which the second input is overlaid. +

+

It accepts the following parameters: +

+
+
x
+
y
+

Set expressions for the x and y coordinates of the overlaid video +on the main video. +

+

They can contain the following parameters: +

+
+
main_w, W
+
main_h, H
+

The main input width and height. +

+
+
overlay_w, w
+
overlay_h, h
+

The overlay input width and height. +

+
+
x
+
y
+

The computed values for x and y. They are evaluated for +each new frame. +

+
+
n
+

The ordinal index of the main input frame, starting from 0. +

+
+
pos
+

The byte offset position in the file of the main input frame, NAN if unknown. +Deprecated, do not use. +

+
+
t
+

The timestamp of the main input frame, expressed in seconds, NAN if unknown. +

+
+
+ +

Default value is "0" for both expressions. +

+
+
eval
+

Set when the expressions for x and y are evaluated. +

+

It accepts the following values: +

+
init
+

Evaluate expressions once during filter initialization or +when a command is processed. +

+
+
frame
+

Evaluate expressions for each incoming frame +

+
+ +

Default value is frame. +

+
+
eof_action
+

See framesync. +

+
+
shortest
+

See framesync. +

+
+
repeatlast
+

See framesync. +

+
+
+ +

This filter also supports the framesync options. +

+ +

12.1.6 pad_cuda

+

Add paddings to an input video stream using CUDA. +

+

This filter is the CUDA-accelerated version of the pad filter. It accepts the same options and expressions and provides the same core functionality. +For a detailed description of available options, please see the documentation for the pad filter. +

+

12.1.6.1 Examples

+ +
    +
  • Add a 200-pixel black border to all sides of a video frame: +
    +
    ffmpeg -hwaccel cuda -hwaccel_output_format cuda -i input.mp4 -vf "pad_cuda=w=iw+400:h=ih+400:x=200:y=200" -c:v h264_nvenc out.mp4
    +
    + +
  • Pad the input video to a 16:9 aspect ratio, filling with the color "blue": +
    +
    ffmpeg -hwaccel cuda -hwaccel_output_format cuda -i input.mp4 -vf "pad_cuda=w=ih*16/9/sar:h=ih:x=(ow-iw)/2:y=(oh-ih)/2:color=blue" -c:v h264_nvenc out.mp4
    +
    +
+ +

12.1.7 scale_cuda

+ +

Scale (resize) and convert (pixel format) the input video, using accelerated CUDA kernels. +Setting the output width and height works in the same way as for the scale filter. +

+

The filter accepts the following options: +

+
w
+
h
+

Set the output video dimension expression. Default value is the input dimension. +

+

Allows for the same expressions as the scale filter. +

+
+
interp_algo
+

Sets the algorithm used for scaling: +

+
+
nearest
+

Nearest neighbour +

+

Used by default if input parameters match the desired output. +

+
+
bilinear
+

Bilinear +

+
+
bicubic
+

Bicubic +

+

This is the default. +

+
+
lanczos
+

Lanczos +

+
+
+ +
+
format
+

Controls the output pixel format. By default, or if none is specified, the input +pixel format is used. +

+

The filter does not support converting between YUV and RGB pixel formats. +

+
+
passthrough
+

If set to 0, every frame is processed, even if no conversion is necessary. +This mode can be useful to use the filter as a buffer for a downstream +frame-consumer that exhausts the limited decoder frame pool. +

+

If set to 1, frames are passed through as-is if they match the desired output +parameters. This is the default behaviour. +

+
+
param
+

Algorithm-Specific parameter. +

+

Affects the curves of the bicubic algorithm. +

+
+
force_original_aspect_ratio
+
force_divisible_by
+

Work the same as the identical scale filter options. +

+
+
reset_sar
+

Works the same as the identical scale filter option. +

+
+
+ +

12.1.7.1 Examples

+ +
    +
  • Scale input to 720p, keeping aspect ratio and ensuring the output is yuv420p. +
    +
    scale_cuda=-2:720:format=yuv420p
    +
    + +
  • Upscale to 4K using nearest neighbour algorithm. +
    +
    scale_cuda=4096:2160:interp_algo=nearest
    +
    + +
  • Don’t do any conversion or scaling, but copy all input frames into newly allocated ones. +This can be useful to deal with a filter and encode chain that otherwise exhausts the +decoders frame pool. +
    +
    scale_cuda=passthrough=0
    +
    +
+ +

12.1.8 thumbnail_cuda

+ +

Select the most representative frame in a given sequence of consecutive frames using CUDA. +

+

The filter accepts the following options: +

+
+
n
+

Set the frames batch size to analyze; in a set of n frames, the filter +will pick one of them, and then handle the next batch of n frames until +the end. Default is 100. +

+
+ +

Since the filter keeps track of the whole frames sequence, a bigger n +value will result in a higher memory usage, so a high value is not recommended. +

+

12.1.8.1 Example

+ +
    +
  • Thumbnails are extracted from every n=150-frame batch, selecting one per batch. Chosen frames are then scaled with scale_cuda. +
    +
    ./ffmpeg  -hwaccel cuda -hwaccel_output_format cuda  -i ./input.mp4 -vf "thumbnail_cuda=150,scale_cuda=1920:1080,hwdownload,format=nv12" ./output/out%03d.png
    +
    + +
+ +

12.1.9 transpose_cuda

+ +

Transpose rows with columns in the input video and optionally flip it. +For more in depth examples see the transpose video filter, which shares mostly the same options. +

+

It accepts the following parameters: +

+
+
dir
+

Specify the transposition direction. +

+

Can assume the following values: +

+
‘cclock_flip’
+

Rotate by 90 degrees counterclockwise and vertically flip. (default) +

+
+
‘clock’
+

Rotate by 90 degrees clockwise. +

+
+
‘cclock’
+

Rotate by 90 degrees counterclockwise. +

+
+
‘clock_flip’
+

Rotate by 90 degrees clockwise and vertically flip. +

+
+
‘reversal’
+

Rotate by 180 degrees. +

+
+
‘hflip’
+

Flip horizontally. +

+
+
‘vflip’
+

Flip vertically. +

+
+ +
+
passthrough
+

Do not apply the transposition if the input geometry matches the one +specified by the specified value. It accepts the following values: +

+
‘none’
+

Always apply transposition. (default) +

+
‘portrait’
+

Preserve portrait geometry (when height >= width). +

+
‘landscape’
+

Preserve landscape geometry (when width >= height). +

+
+ +
+
+ +

12.1.10 yadif_cuda

+ +

Deinterlace the input video using the yadif algorithm, but implemented +in CUDA so that it can work as part of a GPU accelerated pipeline with nvdec +and/or nvenc. +

+

It accepts the following parameters: +

+ +
+
mode
+

The interlacing mode to adopt. It accepts one of the following values: +

+
+
0, send_frame
+

Output one frame for each frame. +

+
1, send_field
+

Output one frame for each field. +

+
2, send_frame_nospatial
+

Like send_frame, but it skips the spatial interlacing check. +

+
3, send_field_nospatial
+

Like send_field, but it skips the spatial interlacing check. +

+
+ +

The default value is send_frame. +

+
+
parity
+

The picture field parity assumed for the input interlaced video. It accepts one +of the following values: +

+
+
0, tff
+

Assume the top field is first. +

+
1, bff
+

Assume the bottom field is first. +

+
-1, auto
+

Enable automatic detection of field parity. +

+
+ +

The default value is auto. +If the interlacing is unknown or the decoder does not export this information, +top field first will be assumed. +

+
+
deint
+

Specify which frames to deinterlace. Accepts one of the following +values: +

+
+
0, all
+

Deinterlace all frames. +

+
1, interlaced
+

Only deinterlace frames marked as interlaced. +

+
+ +

The default value is all. +

+
+ +

12.2 CUDA NPP

+

Below is a description of the currently available NVIDIA Performance Primitives (libnpp) video filters. +

+

Prerequisites: +

    +
  • Install Nvidia CUDA Toolkit +
  • Install libnpp +
+ +

To enable CUDA NPP filters: +

+
    +
  • Configure FFmpeg with --enable-nonfree --enable-libnpp. +
+ + +

12.2.1 scale_npp

+ +

Use the NVIDIA Performance Primitives (libnpp) to perform scaling and/or pixel +format conversion on CUDA video frames. Setting the output width and height +works in the same way as for the scale filter. +

+

The following additional options are accepted: +

+
format
+

The pixel format of the output CUDA frames. If set to the string "same" (the +default), the input format will be kept. Note that automatic format negotiation +and conversion is not yet supported for hardware frames +

+
+
interp_algo
+

The interpolation algorithm used for resizing. One of the following: +

+
nn
+

Nearest neighbour. +

+
+
linear
+
cubic
+
cubic2p_bspline
+

2-parameter cubic (B=1, C=0) +

+
+
cubic2p_catmullrom
+

2-parameter cubic (B=0, C=1/2) +

+
+
cubic2p_b05c03
+

2-parameter cubic (B=1/2, C=3/10) +

+
+
super
+

Supersampling +

+
+
lanczos
+
+ +
+
force_original_aspect_ratio
+

Enable decreasing or increasing output video width or height if necessary to +keep the original aspect ratio. Possible values: +

+
+
‘disable’
+

Scale the video as specified and disable this feature. +

+
+
‘decrease’
+

The output video dimensions will automatically be decreased if needed. +

+
+
‘increase’
+

The output video dimensions will automatically be increased if needed. +

+
+
+ +

One useful instance of this option is that when you know a specific device’s +maximum allowed resolution, you can use this to limit the output video to +that, while retaining the aspect ratio. For example, device A allows +1280x720 playback, and your video is 1920x800. Using this option (set it to +decrease) and specifying 1280x720 to the command line makes the output +1280x533. +

+

Please note that this is a different thing than specifying -1 for w +or h, you still need to specify the output resolution for this option +to work. +

+
+
force_divisible_by
+

Ensures that both the output dimensions, width and height, are divisible by the +given integer when used together with force_original_aspect_ratio. This +works similar to using -n in the w and h options. +

+

This option respects the value set for force_original_aspect_ratio, +increasing or decreasing the resolution accordingly. The video’s aspect ratio +may be slightly modified. +

+

This option can be handy if you need to have a video fit within or exceed +a defined resolution using force_original_aspect_ratio but also have +encoder restrictions on width or height divisibility. +

+
+
reset_sar
+

Works the same as the identical scale filter option. +

+
+
eval
+

Specify when to evaluate width and height expression. It accepts the following values: +

+
+
‘init’
+

Only evaluate expressions once during the filter initialization or when a command is processed. +

+
+
‘frame’
+

Evaluate expressions for each incoming frame. +

+
+
+ +
+
+ +

The values of the w and h options are expressions +containing the following constants: +

+
+
in_w
+
in_h
+

The input width and height +

+
+
iw
+
ih
+

These are the same as in_w and in_h. +

+
+
out_w
+
out_h
+

The output (scaled) width and height +

+
+
ow
+
oh
+

These are the same as out_w and out_h +

+
+
a
+

The same as iw / ih +

+
+
sar
+

input sample aspect ratio +

+
+
dar
+

The input display aspect ratio. Calculated from (iw / ih) * sar. +

+
+
n
+

The (sequential) number of the input frame, starting from 0. +Only available with eval=frame. +

+
+
t
+

The presentation timestamp of the input frame, expressed as a number of +seconds. Only available with eval=frame. +

+
+
pos
+

The position (byte offset) of the frame in the input stream, or NaN if +this information is unavailable and/or meaningless (for example in case of synthetic video). +Only available with eval=frame. +Deprecated, do not use. +

+
+ +

12.2.2 scale2ref_npp

+ +

Use the NVIDIA Performance Primitives (libnpp) to scale (resize) the input +video, based on a reference video. +

+

See the scale_npp filter for available options, scale2ref_npp supports the same +but uses the reference video instead of the main input as basis. scale2ref_npp +also supports the following additional constants for the w and +h options: +

+
+
main_w
+
main_h
+

The main input video’s width and height +

+
+
main_a
+

The same as main_w / main_h +

+
+
main_sar
+

The main input video’s sample aspect ratio +

+
+
main_dar, mdar
+

The main input video’s display aspect ratio. Calculated from +(main_w / main_h) * main_sar. +

+
+
main_n
+

The (sequential) number of the main input frame, starting from 0. +Only available with eval=frame. +

+
+
main_t
+

The presentation timestamp of the main input frame, expressed as a number of +seconds. Only available with eval=frame. +

+
+
main_pos
+

The position (byte offset) of the frame in the main input stream, or NaN if +this information is unavailable and/or meaningless (for example in case of synthetic video). +Only available with eval=frame. +

+
+ +

12.2.2.1 Examples

+ +
    +
  • Scale a subtitle stream (b) to match the main video (a) in size before overlaying +
    +
    'scale2ref_npp[b][a];[a][b]overlay_cuda'
    +
    + +
  • Scale a logo to 1/10th the height of a video, while preserving its display aspect ratio. +
    +
    [logo-in][video-in]scale2ref_npp=w=oh*mdar:h=ih/10[logo-out][video-out]
    +
    +
+ +

12.2.3 sharpen_npp

+

Use the NVIDIA Performance Primitives (libnpp) to perform image sharpening with +border control. +

+

The following additional options are accepted: +

+
border_type
+

Type of sampling to be used ad frame borders. One of the following: +

+
replicate
+

Replicate pixel values. +

+
+
+
+
+ +

12.2.4 transpose_npp

+ +

Transpose rows with columns in the input video and optionally flip it. +For more in depth examples see the transpose video filter, which shares mostly the same options. +

+

It accepts the following parameters: +

+
+
dir
+

Specify the transposition direction. +

+

Can assume the following values: +

+
‘cclock_flip’
+

Rotate by 90 degrees counterclockwise and vertically flip. (default) +

+
+
‘clock’
+

Rotate by 90 degrees clockwise. +

+
+
‘cclock’
+

Rotate by 90 degrees counterclockwise. +

+
+
‘clock_flip’
+

Rotate by 90 degrees clockwise and vertically flip. +

+
+ +
+
passthrough
+

Do not apply the transposition if the input geometry matches the one +specified by the specified value. It accepts the following values: +

+
‘none’
+

Always apply transposition. (default) +

+
‘portrait’
+

Preserve portrait geometry (when height >= width). +

+
‘landscape’
+

Preserve landscape geometry (when width >= height). +

+
+ +
+
+ + +

13 OpenCL Video Filters

+ +

Below is a description of the currently available OpenCL video filters. +

+

To enable compilation of these filters you need to configure FFmpeg with +--enable-opencl. +

+

Running OpenCL filters requires you to initialize a hardware device and to pass that device to all filters in any filter graph. +

+
-init_hw_device opencl[=name][:device[,key=value...]]
+

Initialise a new hardware device of type opencl called name, using the +given device parameters. +

+
+
-filter_hw_device name
+

Pass the hardware device called name to all filters in any filter graph. +

+
+
+ +

For more detailed information see https://www.ffmpeg.org/ffmpeg.html#Advanced-Video-options +

+
    +
  • Example of choosing the first device on the second platform and running avgblur_opencl filter with default parameters on it. +
    +
    -init_hw_device opencl=gpu:1.0 -filter_hw_device gpu -i INPUT -vf "hwupload, avgblur_opencl, hwdownload" OUTPUT
    +
    +
+ +

Since OpenCL filters are not able to access frame data in normal memory, all frame data needs to be uploaded(hwupload) to hardware surfaces connected to the appropriate device before being used and then downloaded(hwdownload) back to normal memory. Note that hwupload will upload to a surface with the same layout as the software frame, so it may be necessary to add a format filter immediately before to get the input into the right format and hwdownload does not support all formats on the output - it may be necessary to insert an additional format filter immediately following in the graph to get the output in a supported format. +

+

13.1 avgblur_opencl

+ +

Apply average blur filter. +

+

The filter accepts the following options: +

+
+
sizeX
+

Set horizontal radius size. +Range is [1, 1024] and default value is 1. +

+
+
planes
+

Set which planes to filter. Default value is 0xf, by which all planes are processed. +

+
+
sizeY
+

Set vertical radius size. Range is [1, 1024] and default value is 0. If zero, sizeX value will be used. +

+
+ +

13.1.1 Example

+ +
    +
  • Apply average blur filter with horizontal and vertical size of 3, setting each pixel of the output to the average value of the 7x7 region centered on it in the input. For pixels on the edges of the image, the region does not extend beyond the image boundaries, and so out-of-range coordinates are not used in the calculations. +
    +
    -i INPUT -vf "hwupload, avgblur_opencl=3, hwdownload" OUTPUT
    +
    +
+ +

13.2 boxblur_opencl

+ +

Apply a boxblur algorithm to the input video. +

+

It accepts the following parameters: +

+
+
luma_radius, lr
+
luma_power, lp
+
chroma_radius, cr
+
chroma_power, cp
+
alpha_radius, ar
+
alpha_power, ap
+
+ +

A description of the accepted options follows. +

+
+
luma_radius, lr
+
chroma_radius, cr
+
alpha_radius, ar
+

Set an expression for the box radius in pixels used for blurring the +corresponding input plane. +

+

The radius value must be a non-negative number, and must not be +greater than the value of the expression min(w,h)/2 for the +luma and alpha planes, and of min(cw,ch)/2 for the chroma +planes. +

+

Default value for luma_radius is "2". If not specified, +chroma_radius and alpha_radius default to the +corresponding value set for luma_radius. +

+

The expressions can contain the following constants: +

+
w
+
h
+

The input width and height in pixels. +

+
+
cw
+
ch
+

The input chroma image width and height in pixels. +

+
+
hsub
+
vsub
+

The horizontal and vertical chroma subsample values. For example, for the +pixel format "yuv422p", hsub is 2 and vsub is 1. +

+
+ +
+
luma_power, lp
+
chroma_power, cp
+
alpha_power, ap
+

Specify how many times the boxblur filter is applied to the +corresponding plane. +

+

Default value for luma_power is 2. If not specified, +chroma_power and alpha_power default to the +corresponding value set for luma_power. +

+

A value of 0 will disable the effect. +

+
+ +

13.2.1 Examples

+ +

Apply boxblur filter, setting each pixel of the output to the average value of box-radiuses luma_radius, chroma_radius, alpha_radius for each plane respectively. The filter will apply luma_power, chroma_power, alpha_power times onto the corresponding plane. For pixels on the edges of the image, the radius does not extend beyond the image boundaries, and so out-of-range coordinates are not used in the calculations. +

+
    +
  • Apply a boxblur filter with the luma, chroma, and alpha radius +set to 2 and luma, chroma, and alpha power set to 3. The filter will run 3 times with box-radius set to 2 for every plane of the image. +
    +
    -i INPUT -vf "hwupload, boxblur_opencl=luma_radius=2:luma_power=3, hwdownload" OUTPUT
    +-i INPUT -vf "hwupload, boxblur_opencl=2:3, hwdownload" OUTPUT
    +
    + +
  • Apply a boxblur filter with luma radius set to 2, luma_power to 1, chroma_radius to 4, chroma_power to 5, alpha_radius to 3 and alpha_power to 7. + +

    For the luma plane, a 2x2 box radius will be run once. +

    +

    For the chroma plane, a 4x4 box radius will be run 5 times. +

    +

    For the alpha plane, a 3x3 box radius will be run 7 times. +

    +
    -i INPUT -vf "hwupload, boxblur_opencl=2:1:4:5:3:7, hwdownload" OUTPUT
    +
    +
+ +

13.3 colorkey_opencl

+

RGB colorspace color keying. +

+

The filter accepts the following options: +

+
+
color
+

The color which will be replaced with transparency. +

+
+
similarity
+

Similarity percentage with the key color. +

+

0.01 matches only the exact key color, while 1.0 matches everything. +

+
+
blend
+

Blend percentage. +

+

0.0 makes pixels either fully transparent, or not transparent at all. +

+

Higher values result in semi-transparent pixels, with a higher transparency +the more similar the pixels color is to the key color. +

+
+ +

13.3.1 Examples

+ +
    +
  • Make every semi-green pixel in the input transparent with some slight blending: +
    +
    -i INPUT -vf "hwupload, colorkey_opencl=green:0.3:0.1, hwdownload" OUTPUT
    +
    +
+ +

13.4 convolution_opencl

+ +

Apply convolution of 3x3, 5x5, 7x7 matrix. +

+

The filter accepts the following options: +

+
+
0m
+
1m
+
2m
+
3m
+

Set matrix for each plane. +Matrix is sequence of 9, 25 or 49 signed numbers. +Default value for each plane is 0 0 0 0 1 0 0 0 0. +

+
+
0rdiv
+
1rdiv
+
2rdiv
+
3rdiv
+

Set multiplier for calculated value for each plane. +If unset or 0, it will be sum of all matrix elements. +The option value must be a float number greater or equal to 0.0. Default value is 1.0. +

+
+
0bias
+
1bias
+
2bias
+
3bias
+

Set bias for each plane. This value is added to the result of the multiplication. +Useful for making the overall image brighter or darker. +The option value must be a float number greater or equal to 0.0. Default value is 0.0. +

+
+
+ +

13.4.1 Examples

+ +
    +
  • Apply sharpen: +
    +
    -i INPUT -vf "hwupload, convolution_opencl=0 -1 0 -1 5 -1 0 -1 0:0 -1 0 -1 5 -1 0 -1 0:0 -1 0 -1 5 -1 0 -1 0:0 -1 0 -1 5 -1 0 -1 0, hwdownload" OUTPUT
    +
    + +
  • Apply blur: +
    +
    -i INPUT -vf "hwupload, convolution_opencl=1 1 1 1 1 1 1 1 1:1 1 1 1 1 1 1 1 1:1 1 1 1 1 1 1 1 1:1 1 1 1 1 1 1 1 1:1/9:1/9:1/9:1/9, hwdownload" OUTPUT
    +
    + +
  • Apply edge enhance: +
    +
    -i INPUT -vf "hwupload, convolution_opencl=0 0 0 -1 1 0 0 0 0:0 0 0 -1 1 0 0 0 0:0 0 0 -1 1 0 0 0 0:0 0 0 -1 1 0 0 0 0:5:1:1:1:0:128:128:128, hwdownload" OUTPUT
    +
    + +
  • Apply edge detect: +
    +
    -i INPUT -vf "hwupload, convolution_opencl=0 1 0 1 -4 1 0 1 0:0 1 0 1 -4 1 0 1 0:0 1 0 1 -4 1 0 1 0:0 1 0 1 -4 1 0 1 0:5:5:5:1:0:128:128:128, hwdownload" OUTPUT
    +
    + +
  • Apply laplacian edge detector which includes diagonals: +
    +
    -i INPUT -vf "hwupload, convolution_opencl=1 1 1 1 -8 1 1 1 1:1 1 1 1 -8 1 1 1 1:1 1 1 1 -8 1 1 1 1:1 1 1 1 -8 1 1 1 1:5:5:5:1:0:128:128:0, hwdownload" OUTPUT
    +
    + +
  • Apply emboss: +
    +
    -i INPUT -vf "hwupload, convolution_opencl=-2 -1 0 -1 1 1 0 1 2:-2 -1 0 -1 1 1 0 1 2:-2 -1 0 -1 1 1 0 1 2:-2 -1 0 -1 1 1 0 1 2, hwdownload" OUTPUT
    +
    +
+ +

13.5 erosion_opencl

+ +

Apply erosion effect to the video. +

+

This filter replaces the pixel by the local(3x3) minimum. +

+

It accepts the following options: +

+
+
threshold0
+
threshold1
+
threshold2
+
threshold3
+

Limit the maximum change for each plane. Range is [0, 65535] and default value is 65535. +If 0, plane will remain unchanged. +

+
+
coordinates
+

Flag which specifies the pixel to refer to. +Range is [0, 255] and default value is 255, i.e. all eight pixels are used. +

+

Flags to local 3x3 coordinates region centered on x: +

+

1 2 3 +

+

4 x 5 +

+

6 7 8 +

+
+ +

13.5.1 Example

+ +
    +
  • Apply erosion filter with threshold0 set to 30, threshold1 set 40, threshold2 set to 50 and coordinates set to 231, setting each pixel of the output to the local minimum between pixels: 1, 2, 3, 6, 7, 8 of the 3x3 region centered on it in the input. If the difference between input pixel and local minimum is more then threshold of the corresponding plane, output pixel will be set to input pixel - threshold of corresponding plane. +
    +
    -i INPUT -vf "hwupload, erosion_opencl=30:40:50:coordinates=231, hwdownload" OUTPUT
    +
    +
+ +

13.6 deshake_opencl

+

Feature-point based video stabilization filter. +

+

The filter accepts the following options: +

+
+
tripod
+

Simulates a tripod by preventing any camera movement whatsoever from the original frame. Defaults to 0. +

+
+
debug
+

Whether or not additional debug info should be displayed, both in the processed output and in the console. +

+

Note that in order to see console debug output you will also need to pass -v verbose to ffmpeg. +

+

Viewing point matches in the output video is only supported for RGB input. +

+

Defaults to 0. +

+
+
adaptive_crop
+

Whether or not to do a tiny bit of cropping at the borders to cut down on the amount of mirrored pixels. +

+

Defaults to 1. +

+
+
refine_features
+

Whether or not feature points should be refined at a sub-pixel level. +

+

This can be turned off for a slight performance gain at the cost of precision. +

+

Defaults to 1. +

+
+
smooth_strength
+

The strength of the smoothing applied to the camera path from 0.0 to 1.0. +

+

1.0 is the maximum smoothing strength while values less than that result in less smoothing. +

+

0.0 causes the filter to adaptively choose a smoothing strength on a per-frame basis. +

+

Defaults to 0.0. +

+
+
smooth_window_multiplier
+

Controls the size of the smoothing window (the number of frames buffered to determine motion information from). +

+

The size of the smoothing window is determined by multiplying the framerate of the video by this number. +

+

Acceptable values range from 0.1 to 10.0. +

+

Larger values increase the amount of motion data available for determining how to smooth the camera path, +potentially improving smoothness, but also increase latency and memory usage. +

+

Defaults to 2.0. +

+
+
+ +

13.6.1 Examples

+ +
    +
  • Stabilize a video with a fixed, medium smoothing strength: +
    +
    -i INPUT -vf "hwupload, deshake_opencl=smooth_strength=0.5, hwdownload" OUTPUT
    +
    + +
  • Stabilize a video with debugging (both in console and in rendered video): +
    +
    -i INPUT -filter_complex "[0:v]format=rgba, hwupload, deshake_opencl=debug=1, hwdownload, format=rgba, format=yuv420p" -v verbose OUTPUT
    +
    +
+ +

13.7 dilation_opencl

+ +

Apply dilation effect to the video. +

+

This filter replaces the pixel by the local(3x3) maximum. +

+

It accepts the following options: +

+
+
threshold0
+
threshold1
+
threshold2
+
threshold3
+

Limit the maximum change for each plane. Range is [0, 65535] and default value is 65535. +If 0, plane will remain unchanged. +

+
+
coordinates
+

Flag which specifies the pixel to refer to. +Range is [0, 255] and default value is 255, i.e. all eight pixels are used. +

+

Flags to local 3x3 coordinates region centered on x: +

+

1 2 3 +

+

4 x 5 +

+

6 7 8 +

+
+ +

13.7.1 Example

+ +
    +
  • Apply dilation filter with threshold0 set to 30, threshold1 set 40, threshold2 set to 50 and coordinates set to 231, setting each pixel of the output to the local maximum between pixels: 1, 2, 3, 6, 7, 8 of the 3x3 region centered on it in the input. If the difference between input pixel and local maximum is more then threshold of the corresponding plane, output pixel will be set to input pixel + threshold of corresponding plane. +
    +
    -i INPUT -vf "hwupload, dilation_opencl=30:40:50:coordinates=231, hwdownload" OUTPUT
    +
    +
+ +

13.8 nlmeans_opencl

+ +

Non-local Means denoise filter through OpenCL, this filter accepts same options as nlmeans. +

+

13.9 overlay_opencl

+ +

Overlay one video on top of another. +

+

It takes two inputs and has one output. The first input is the "main" video on which the second input is overlaid. +This filter requires same memory layout for all the inputs. So, format conversion may be needed. +

+

The filter accepts the following options: +

+
+
x
+

Set the x coordinate of the overlaid video on the main video. +Default value is 0. +

+
+
y
+

Set the y coordinate of the overlaid video on the main video. +Default value is 0. +

+
+
+ +

13.9.1 Examples

+ +
    +
  • Overlay an image LOGO at the top-left corner of the INPUT video. Both inputs are yuv420p format. +
    +
    -i INPUT -i LOGO -filter_complex "[0:v]hwupload[a], [1:v]format=yuv420p, hwupload[b], [a][b]overlay_opencl, hwdownload" OUTPUT
    +
    +
  • The inputs have same memory layout for color channels , the overlay has additional alpha plane, like INPUT is yuv420p, and the LOGO is yuva420p. +
    +
    -i INPUT -i LOGO -filter_complex "[0:v]hwupload[a], [1:v]format=yuva420p, hwupload[b], [a][b]overlay_opencl, hwdownload" OUTPUT
    +
    + +
+ +

13.10 pad_opencl

+ +

Add paddings to the input image, and place the original input at the +provided x, y coordinates. +

+

It accepts the following options: +

+
+
width, w
+
height, h
+

Specify an expression for the size of the output image with the +paddings added. If the value for width or height is 0, the +corresponding input size is used for the output. +

+

The width expression can reference the value set by the +height expression, and vice versa. +

+

The default value of width and height is 0. +

+
+
x
+
y
+

Specify the offsets to place the input image at within the padded area, +with respect to the top/left border of the output image. +

+

The x expression can reference the value set by the y +expression, and vice versa. +

+

The default value of x and y is 0. +

+

If x or y evaluate to a negative number, they’ll be changed +so the input image is centered on the padded area. +

+
+
color
+

Specify the color of the padded area. For the syntax of this option, +check the (ffmpeg-utils)"Color" section in the ffmpeg-utils +manual. +

+
+
aspect
+

Pad to an aspect instead to a resolution. +

+
+ +

The value for the width, height, x, and y +options are expressions containing the following constants: +

+
+
in_w
+
in_h
+

The input video width and height. +

+
+
iw
+
ih
+

These are the same as in_w and in_h. +

+
+
out_w
+
out_h
+

The output width and height (the size of the padded area), as +specified by the width and height expressions. +

+
+
ow
+
oh
+

These are the same as out_w and out_h. +

+
+
x
+
y
+

The x and y offsets as specified by the x and y +expressions, or NAN if not yet specified. +

+
+
a
+

same as iw / ih +

+
+
sar
+

input sample aspect ratio +

+
+
dar
+

input display aspect ratio, it is the same as (iw / ih) * sar +

+
+ +

13.11 prewitt_opencl

+ +

Apply the Prewitt operator (https://en.wikipedia.org/wiki/Prewitt_operator) to input video stream. +

+

The filter accepts the following option: +

+
+
planes
+

Set which planes to filter. Default value is 0xf, by which all planes are processed. +

+
+
scale
+

Set value which will be multiplied with filtered result. +Range is [0.0, 65535] and default value is 1.0. +

+
+
delta
+

Set value which will be added to filtered result. +Range is [-65535, 65535] and default value is 0.0. +

+
+ +

13.11.1 Example

+ +
    +
  • Apply the Prewitt operator with scale set to 2 and delta set to 10. +
    +
    -i INPUT -vf "hwupload, prewitt_opencl=scale=2:delta=10, hwdownload" OUTPUT
    +
    +
+ +

13.12 program_opencl

+ +

Filter video using an OpenCL program. +

+
+
source
+

OpenCL program source file. +

+
+
kernel
+

Kernel name in program. +

+
+
inputs
+

Number of inputs to the filter. Defaults to 1. +

+
+
size, s
+

Size of output frames. Defaults to the same as the first input. +

+
+
+ +

The program_opencl filter also supports the framesync options. +

+

The program source file must contain a kernel function with the given name, +which will be run once for each plane of the output. Each run on a plane +gets enqueued as a separate 2D global NDRange with one work-item for each +pixel to be generated. The global ID offset for each work-item is therefore +the coordinates of a pixel in the destination image. +

+

The kernel function needs to take the following arguments: +

    +
  • Destination image, __write_only image2d_t. + +

    This image will become the output; the kernel should write all of it. +

  • Frame index, unsigned int. + +

    This is a counter starting from zero and increasing by one for each frame. +

  • Source images, __read_only image2d_t. + +

    These are the most recent images on each input. The kernel may read from +them to generate the output, but they can’t be written to. +

+ +

Example programs: +

+
    +
  • Copy the input to the output (output must be the same size as the input). +
    __kernel void copy(__write_only image2d_t destination,
    +                   unsigned int index,
    +                   __read_only  image2d_t source)
    +{
    +    const sampler_t sampler = CLK_NORMALIZED_COORDS_FALSE;
    +
    +    int2 location = (int2)(get_global_id(0), get_global_id(1));
    +
    +    float4 value = read_imagef(source, sampler, location);
    +
    +    write_imagef(destination, location, value);
    +}
    +
    +
  • Apply a simple transformation, rotating the input by an amount increasing +with the index counter. Pixel values are linearly interpolated by the +sampler, and the output need not have the same dimensions as the input. +
    __kernel void rotate_image(__write_only image2d_t dst,
    +                           unsigned int index,
    +                           __read_only  image2d_t src)
    +{
    +    const sampler_t sampler = (CLK_NORMALIZED_COORDS_FALSE |
    +                               CLK_FILTER_LINEAR);
    +
    +    float angle = (float)index / 100.0f;
    +
    +    float2 dst_dim = convert_float2(get_image_dim(dst));
    +    float2 src_dim = convert_float2(get_image_dim(src));
    +
    +    float2 dst_cen = dst_dim / 2.0f;
    +    float2 src_cen = src_dim / 2.0f;
    +
    +    int2   dst_loc = (int2)(get_global_id(0), get_global_id(1));
    +
    +    float2 dst_pos = convert_float2(dst_loc) - dst_cen;
    +    float2 src_pos = {
    +        cos(angle) * dst_pos.x - sin(angle) * dst_pos.y,
    +        sin(angle) * dst_pos.x + cos(angle) * dst_pos.y
    +    };
    +    src_pos = src_pos * src_dim / dst_dim;
    +
    +    float2 src_loc = src_pos + src_cen;
    +
    +    if (src_loc.x < 0.0f      || src_loc.y < 0.0f ||
    +        src_loc.x > src_dim.x || src_loc.y > src_dim.y)
    +        write_imagef(dst, dst_loc, 0.5f);
    +    else
    +        write_imagef(dst, dst_loc, read_imagef(src, sampler, src_loc));
    +}
    +
    +
  • Blend two inputs together, with the amount of each input used varying +with the index counter. +
    __kernel void blend_images(__write_only image2d_t dst,
    +                           unsigned int index,
    +                           __read_only  image2d_t src1,
    +                           __read_only  image2d_t src2)
    +{
    +    const sampler_t sampler = (CLK_NORMALIZED_COORDS_FALSE |
    +                               CLK_FILTER_LINEAR);
    +
    +    float blend = (cos((float)index / 50.0f) + 1.0f) / 2.0f;
    +
    +    int2  dst_loc = (int2)(get_global_id(0), get_global_id(1));
    +    int2 src1_loc = dst_loc * get_image_dim(src1) / get_image_dim(dst);
    +    int2 src2_loc = dst_loc * get_image_dim(src2) / get_image_dim(dst);
    +
    +    float4 val1 = read_imagef(src1, sampler, src1_loc);
    +    float4 val2 = read_imagef(src2, sampler, src2_loc);
    +
    +    write_imagef(dst, dst_loc, val1 * blend + val2 * (1.0f - blend));
    +}
    +
    +
+ +

13.13 remap_opencl

+ +

Remap pixels using 2nd: Xmap and 3rd: Ymap input video stream. +

+

Destination pixel at position (X, Y) will be picked from source (x, y) position +where x = Xmap(X, Y) and y = Ymap(X, Y). If mapping values are out of range, zero +value for pixel will be used for destination pixel. +

+

Xmap and Ymap input video streams must be of same dimensions. Output video stream +will have Xmap/Ymap video stream dimensions. +Xmap and Ymap input video streams are 32bit float pixel format, single channel. +

+
+
interp
+

Specify interpolation used for remapping of pixels. +Allowed values are near and linear. +Default value is linear. +

+
+
fill
+

Specify the color of the unmapped pixels. For the syntax of this option, +check the (ffmpeg-utils)"Color" section in the ffmpeg-utils +manual. Default color is black. +

+
+ +

13.14 roberts_opencl

+

Apply the Roberts cross operator (https://en.wikipedia.org/wiki/Roberts_cross) to input video stream. +

+

The filter accepts the following option: +

+
+
planes
+

Set which planes to filter. Default value is 0xf, by which all planes are processed. +

+
+
scale
+

Set value which will be multiplied with filtered result. +Range is [0.0, 65535] and default value is 1.0. +

+
+
delta
+

Set value which will be added to filtered result. +Range is [-65535, 65535] and default value is 0.0. +

+
+ +

13.14.1 Example

+ +
    +
  • Apply the Roberts cross operator with scale set to 2 and delta set to 10 +
    +
    -i INPUT -vf "hwupload, roberts_opencl=scale=2:delta=10, hwdownload" OUTPUT
    +
    +
+ +

13.15 sobel_opencl

+ +

Apply the Sobel operator (https://en.wikipedia.org/wiki/Sobel_operator) to input video stream. +

+

The filter accepts the following option: +

+
+
planes
+

Set which planes to filter. Default value is 0xf, by which all planes are processed. +

+
+
scale
+

Set value which will be multiplied with filtered result. +Range is [0.0, 65535] and default value is 1.0. +

+
+
delta
+

Set value which will be added to filtered result. +Range is [-65535, 65535] and default value is 0.0. +

+
+ +

13.15.1 Example

+ +
    +
  • Apply sobel operator with scale set to 2 and delta set to 10 +
    +
    -i INPUT -vf "hwupload, sobel_opencl=scale=2:delta=10, hwdownload" OUTPUT
    +
    +
+ +

13.16 tonemap_opencl

+ +

Perform HDR(PQ/HLG) to SDR conversion with tone-mapping. +

+

It accepts the following parameters: +

+
+
tonemap
+

Specify the tone-mapping operator to be used. Same as tonemap option in tonemap. +

+
+
param
+

Tune the tone mapping algorithm. same as param option in tonemap. +

+
+
desat
+

Apply desaturation for highlights that exceed this level of brightness. The +higher the parameter, the more color information will be preserved. This +setting helps prevent unnaturally blown-out colors for super-highlights, by +(smoothly) turning into white instead. This makes images feel more natural, +at the cost of reducing information about out-of-range colors. +

+

The default value is 0.5, and the algorithm here is a little different from +the cpu version tonemap currently. A setting of 0.0 disables this option. +

+
+
threshold
+

The tonemapping algorithm parameters is fine-tuned per each scene. And a threshold +is used to detect whether the scene has changed or not. If the distance between +the current frame average brightness and the current running average exceeds +a threshold value, we would re-calculate scene average and peak brightness. +The default value is 0.2. +

+
+
format
+

Specify the output pixel format. +

+

Currently supported formats are: +

+
p010
+
nv12
+
+ +
+
range, r
+

Set the output color range. +

+

Possible values are: +

+
tv/mpeg
+
pc/jpeg
+
+ +

Default is same as input. +

+
+
primaries, p
+

Set the output color primaries. +

+

Possible values are: +

+
bt709
+
bt2020
+
+ +

Default is same as input. +

+
+
transfer, t
+

Set the output transfer characteristics. +

+

Possible values are: +

+
bt709
+
bt2020
+
+ +

Default is bt709. +

+
+
matrix, m
+

Set the output colorspace matrix. +

+

Possible value are: +

+
bt709
+
bt2020
+
+ +

Default is same as input. +

+
+
+ +

13.16.1 Example

+ +
    +
  • Convert HDR(PQ/HLG) video to bt2020-transfer-characteristic p010 format using linear operator. +
    +
    -i INPUT -vf "format=p010,hwupload,tonemap_opencl=t=bt2020:tonemap=linear:format=p010,hwdownload,format=p010" OUTPUT
    +
    +
+ +

13.17 unsharp_opencl

+ +

Sharpen or blur the input video. +

+

It accepts the following parameters: +

+
+
luma_msize_x, lx
+

Set the luma matrix horizontal size. +Range is [1, 23] and default value is 5. +

+
+
luma_msize_y, ly
+

Set the luma matrix vertical size. +Range is [1, 23] and default value is 5. +

+
+
luma_amount, la
+

Set the luma effect strength. +Range is [-10, 10] and default value is 1.0. +

+

Negative values will blur the input video, while positive values will +sharpen it, a value of zero will disable the effect. +

+
+
chroma_msize_x, cx
+

Set the chroma matrix horizontal size. +Range is [1, 23] and default value is 5. +

+
+
chroma_msize_y, cy
+

Set the chroma matrix vertical size. +Range is [1, 23] and default value is 5. +

+
+
chroma_amount, ca
+

Set the chroma effect strength. +Range is [-10, 10] and default value is 0.0. +

+

Negative values will blur the input video, while positive values will +sharpen it, a value of zero will disable the effect. +

+
+
+ +

All parameters are optional and default to the equivalent of the +string ’5:5:1.0:5:5:0.0’. +

+

13.17.1 Examples

+ +
    +
  • Apply strong luma sharpen effect: +
    +
    -i INPUT -vf "hwupload, unsharp_opencl=luma_msize_x=7:luma_msize_y=7:luma_amount=2.5, hwdownload" OUTPUT
    +
    + +
  • Apply a strong blur of both luma and chroma parameters: +
    +
    -i INPUT -vf "hwupload, unsharp_opencl=7:7:-2:7:7:-2, hwdownload" OUTPUT
    +
    +
+ +

13.18 xfade_opencl

+ +

Cross fade two videos with custom transition effect by using OpenCL. +

+

It accepts the following options: +

+
+
transition
+

Set one of possible transition effects. +

+
+
custom
+

Select custom transition effect, the actual transition description +will be picked from source and kernel options. +

+
+
fade
+
wipeleft
+
wiperight
+
wipeup
+
wipedown
+
slideleft
+
slideright
+
slideup
+
slidedown
+
+

Default transition is fade. +

+
+ +
+
source
+

OpenCL program source file for custom transition. +

+
+
kernel
+

Set name of kernel to use for custom transition from program source file. +

+
+
duration
+

Set duration of video transition. +

+
+
offset
+

Set time of start of transition relative to first video. +

+
+ +

The program source file must contain a kernel function with the given name, +which will be run once for each plane of the output. Each run on a plane +gets enqueued as a separate 2D global NDRange with one work-item for each +pixel to be generated. The global ID offset for each work-item is therefore +the coordinates of a pixel in the destination image. +

+

The kernel function needs to take the following arguments: +

    +
  • Destination image, __write_only image2d_t. + +

    This image will become the output; the kernel should write all of it. +

    +
  • First Source image, __read_only image2d_t. +Second Source image, __read_only image2d_t. + +

    These are the most recent images on each input. The kernel may read from +them to generate the output, but they can’t be written to. +

    +
  • Transition progress, float. This value is always between 0 and 1 inclusive. +
+ +

Example programs: +

+
    +
  • Apply dots curtain transition effect: +
    __kernel void blend_images(__write_only image2d_t dst,
    +                           __read_only  image2d_t src1,
    +                           __read_only  image2d_t src2,
    +                           float progress)
    +{
    +    const sampler_t sampler = (CLK_NORMALIZED_COORDS_FALSE |
    +                               CLK_FILTER_LINEAR);
    +    int2  p = (int2)(get_global_id(0), get_global_id(1));
    +    float2 rp = (float2)(get_global_id(0), get_global_id(1));
    +    float2 dim = (float2)(get_image_dim(src1).x, get_image_dim(src1).y);
    +    rp = rp / dim;
    +
    +    float2 dots = (float2)(20.0, 20.0);
    +    float2 center = (float2)(0,0);
    +    float2 unused;
    +
    +    float4 val1 = read_imagef(src1, sampler, p);
    +    float4 val2 = read_imagef(src2, sampler, p);
    +    bool next = distance(fract(rp * dots, &unused), (float2)(0.5, 0.5)) < (progress / distance(rp, center));
    +
    +    write_imagef(dst, p, next ? val1 : val2);
    +}
    +
    +
+ + +

14 VAAPI Video Filters

+ +

VAAPI Video filters are usually used with VAAPI decoder and VAAPI encoder. Below is a description of VAAPI video filters. +

+

To enable compilation of these filters you need to configure FFmpeg with +--enable-vaapi. +

+

To use vaapi filters, you need to setup the vaapi device correctly. For more information, please read https://trac.ffmpeg.org/wiki/Hardware/VAAPI +

+

14.1 overlay_vaapi

+ +

Overlay one video on the top of another. +

+

It takes two inputs and has one output. The first input is the "main" video on which the second input is overlaid. +

+

The filter accepts the following options: +

+
+
x
+
y
+

Set expressions for the x and y coordinates of the overlaid video +on the main video. +

+

Default value is "0" for both expressions. +

+
+
w
+
h
+

Set expressions for the width and height the overlaid video +on the main video. +

+

Default values are ’overlay_iw’ for ’w’ and ’overlay_ih*w/overlay_iw’ for ’h’. +

+

The expressions can contain the following parameters: +

+
+
main_w, W
+
main_h, H
+

The main input width and height. +

+
+
overlay_iw
+
overlay_ih
+

The overlay input width and height. +

+
+
overlay_w, w
+
overlay_h, h
+

The overlay output width and height. +

+
+
overlay_x, x
+
overlay_y, y
+

Position of the overlay layer inside of main +

+
+
+ +
+
alpha
+

Set transparency of overlaid video. Allowed range is 0.0 to 1.0. +Higher value means lower transparency. +Default value is 1.0. +

+
+
eof_action
+

See framesync. +

+
+
shortest
+

See framesync. +

+
+
repeatlast
+

See framesync. +

+
+
+ +

This filter also supports the framesync options. +

14.1.1 Examples

+ +
    +
  • Overlay an image LOGO at the top-left corner of the INPUT video. Both inputs for this filter are yuv420p format. +
    +
    -i INPUT -i LOGO -filter_complex "[0:v]hwupload[a], [1:v]format=yuv420p, hwupload[b], [a][b]overlay_vaapi" OUTPUT
    +
    +
  • Overlay an image LOGO at the offset (200, 100) from the top-left corner of the INPUT video. +The inputs have same memory layout for color channels, the overlay has additional alpha plane, like INPUT is yuv420p, and the LOGO is yuva420p. +
    +
    -i INPUT -i LOGO -filter_complex "[0:v]hwupload[a], [1:v]format=yuva420p, hwupload[b], [a][b]overlay_vaapi=x=200:y=100:w=400:h=300:alpha=1.0, hwdownload, format=nv12" OUTPUT
    +
    + +
+ +

14.2 tonemap_vaapi

+ +

Perform HDR-to-SDR or HDR-to-HDR tone-mapping. +It currently only accepts HDR10 as input. +

+

It accepts the following parameters: +

+
+
format
+

Specify the output pixel format. +

+

Default is nv12 for HDR-to-SDR tone-mapping and p010 for HDR-to-HDR +tone-mapping. +

+
+
primaries, p
+

Set the output color primaries. +

+

Default is bt709 for HDR-to-SDR tone-mapping and same as input for HDR-to-HDR +tone-mapping. +

+
+
transfer, t
+

Set the output transfer characteristics. +

+

Default is bt709 for HDR-to-SDR tone-mapping and same as input for HDR-to-HDR +tone-mapping. +

+
+
matrix, m
+

Set the output colorspace matrix. +

+

Default is bt709 for HDR-to-SDR tone-mapping and same as input for HDR-to-HDR +tone-mapping. +

+
+
display
+

Set the output mastering display colour volume. It is given by a ’|’-separated +list of two values, two values are space separated. It set display primaries +x & y in G, B, R order, then white point x & y, the nominal minimum & maximum +display luminances. +

+

HDR-to-HDR tone-mapping will be performed when this option is set. +

+
+
light
+

Set the output content light level information. It accepts 2 space-separated +values, the first input is the maximum light level and the second input is +the maximum average light level. +

+

It is ignored for HDR-to-SDR tone-mapping, and optional for HDR-to-HDR +tone-mapping. +

+
+
+ +

14.2.1 Example

+ +
    +
  • Convert HDR(HDR10) video to bt2020-transfer-characteristic p010 format +
    +
    tonemap_vaapi=format=p010:t=bt2020-10
    +
    +
  • Convert HDR video to HDR video +
    +
    tonemap_vaapi=display=7500\ 3000|34000\ 16000|13250\ 34500|15635\ 16450|500\ 10000000
    +
    +
+ +

14.3 hstack_vaapi

+

Stack input videos horizontally. +

+

This is the VA-API variant of the hstack filter, each input stream may +have different height, this filter will scale down/up each input stream while +keeping the original aspect. +

+

It accepts the following options: +

+
+
inputs
+

See hstack. +

+
+
shortest
+

See hstack. +

+
+
height
+

Set height of output. If set to 0, this filter will set height of output to +height of the first input stream. Default value is 0. +

+
+ +

14.4 vstack_vaapi

+

Stack input videos vertically. +

+

This is the VA-API variant of the vstack filter, each input stream may +have different width, this filter will scale down/up each input stream while +keeping the original aspect. +

+

It accepts the following options: +

+
+
inputs
+

See vstack. +

+
+
shortest
+

See vstack. +

+
+
width
+

Set width of output. If set to 0, this filter will set width of output to +width of the first input stream. Default value is 0. +

+
+ +

14.5 xstack_vaapi

+

Stack video inputs into custom layout. +

+

This is the VA-API variant of the xstack filter, each input stream may +have different size, this filter will scale down/up each input stream to the +given output size, or the size of the first input stream. +

+

It accepts the following options: +

+
+
inputs
+

See xstack. +

+
+
shortest
+

See xstack. +

+
+
layout
+

See xstack. +Moreover, this permits the user to supply output size for each input stream. +

+
xstack_vaapi=inputs=4:layout=0_0_1920x1080|0_h0_1920x1080|w0_0_1920x1080|w0_h0_1920x1080
+
+ +
+
grid
+

See xstack. +

+
+
grid_tile_size
+

Set output size for each input stream when grid is set. If this option +is not set, this filter will set output size by default to the size of the +first input stream. For the syntax of this option, check the +(ffmpeg-utils)"Video size" section in the ffmpeg-utils manual. +

+
+
fill
+

See xstack. +

+
+ +

14.6 pad_vaapi

+ +

Add paddings to the input image, and place the original input at the +provided x, y coordinates. +

+

It accepts the following options: +

+
+
width, w
+
height, h
+

Specify an expression for the size of the output image with the +paddings added. If the value for width or height is 0, the +corresponding input size is used for the output. +

+

The width expression can reference the value set by the +height expression, and vice versa. +

+

The default value of width and height is 0. +

+
+
x
+
y
+

Specify the offsets to place the input image at within the padded area, +with respect to the top/left border of the output image. +

+

The x expression can reference the value set by the y +expression, and vice versa. +

+

The default value of x and y is 0. +

+

If x or y evaluate to a negative number, they’ll be changed +so the input image is centered on the padded area. +

+
+
color
+

Specify the color of the padded area. For the syntax of this option, +check the (ffmpeg-utils)"Color" section in the ffmpeg-utils +manual. +

+
+
aspect
+

Pad to an aspect instead to a resolution. +

+
+ +

The value for the width, height, x, and y +options are expressions containing the following constants: +

+
+
in_w
+
in_h
+

The input video width and height. +

+
+
iw
+
ih
+

These are the same as in_w and in_h. +

+
+
out_w
+
out_h
+

The output width and height (the size of the padded area), as +specified by the width and height expressions. +

+
+
ow
+
oh
+

These are the same as out_w and out_h. +

+
+
x
+
y
+

The x and y offsets as specified by the x and y +expressions, or NAN if not yet specified. +

+
+
a
+

same as iw / ih +

+
+
sar
+

input sample aspect ratio +

+
+
dar
+

input display aspect ratio, it is the same as (iw / ih) * sar +

+
+ +

14.7 drawbox_vaapi

+ +

Draw a colored box on the input image. +

+

It accepts the following parameters: +

+
+
x
+
y
+

The expressions which specify the top left corner coordinates of the box. It defaults to 0. +

+
+
width, w
+
height, h
+

The expressions which specify the width and height of the box; if 0 they are interpreted as +the input width and height. It defaults to 0. +

+
+
color, c
+

Specify the color of the box to write. For the general syntax of this option, +check the (ffmpeg-utils)"Color" section in the ffmpeg-utils manual. +

+
+
thickness, t
+

The expression which sets the thickness of the box edge. +A value of fill will create a filled box. Default value is 3. +

+

See below for the list of accepted constants. +

+
+
replace
+

With value 1, the pixels of the painted box will overwrite the video’s color and alpha pixels. +Default is 0, which composites the box onto the input video. +

+
+ +

The parameters for x, y, w and h and t are expressions containing the +following constants: +

+
+
in_h, ih
+
in_w, iw
+

The input width and height. +

+
+
x
+
y
+

The x and y offset coordinates where the box is drawn. +

+
+
w
+
h
+

The width and height of the drawn box. +

+
+
t
+

The thickness of the drawn box. +

+
+
+ +

14.7.1 Examples

+ +
    +
  • Draw a black box around the edge of the input image: +
    +
    drawbox
    +
    + +
  • Draw a box with color red and an opacity of 50%: +
    +
    drawbox=10:20:200:60:red@0.5
    +
    + +

    The previous example can be specified as: +

    +
    drawbox=x=10:y=20:w=200:h=60:color=red@0.5
    +
    + +
  • Fill the box with pink color: +
    +
    drawbox=x=10:y=10:w=100:h=100:color=pink@0.5:t=fill
    +
    + +
  • Draw a 2-pixel red 2.40:1 mask: +
    +
    drawbox=x=-t:y=0.5*(ih-iw/2.4)-t:w=iw+t*2:h=iw/2.4+t*2:t=2:c=red
    +
    +
+ + +

15 VideoToolbox Video Filters

+ +

Below is a description of the currently available VideoToolbox video filters. +

+

VideoToolbox filter depends on VideoToolbox framework, and is auto detected +when building ffmpeg for an Apple platform such as macOS. Add +--enable-videotoolbox to configure if autodetect is disabled. +

+

15.1 scale_vt

+ +

Scale and convert the color parameters using VTPixelTransferSession. +

+

The filter accepts the following options: +

+
w
+
h
+

Set the output video dimension expression. Default value is the input dimension. +

+
+
color_matrix
+

Set the output colorspace matrix. +

+
+
color_primaries
+

Set the output color primaries. +

+
+
color_transfer
+

Set the output transfer characteristics. +

+
+
+ +

15.1.1 Examples

+ +
    +
  • Perform HDR to SDR conversion, and scale to half size of input +
    +
    ffmpeg -hwaccel videotoolbox \
    +        -hwaccel_output_format videotoolbox_vld \
    +        -i hdr.mov \
    +        -c:v hevc_videotoolbox \
    +        -profile:v main \
    +        -b:v 3M \
    +        -vf scale_vt=w=iw/2:h=ih/2:color_matrix=bt709:color_primaries=bt709:color_transfer=bt709 \
    +        -c:a copy \
    +        -tag:v hvc1 \
    +        sdr.mp4
    +
    + +
+ +

15.2 transpose_vt

+ +

Transpose rows with columns in the input video and optionally flip it. +For more in depth examples see the transpose video filter, which shares mostly the same options. +

+

It accepts the following parameters: +

+
+
dir
+

Specify the transposition direction. +

+

Can assume the following values: +

+
‘cclock_flip’
+

Rotate by 90 degrees counterclockwise and vertically flip. (default) +

+
+
‘clock’
+

Rotate by 90 degrees clockwise. +

+
+
‘cclock’
+

Rotate by 90 degrees counterclockwise. +

+
+
‘clock_flip’
+

Rotate by 90 degrees clockwise and vertically flip. +

+
+
‘hflip’
+

Flip the input video horizontally. +

+
+
‘vflip’
+

Flip the input video vertically. +

+
+
+ +
+
passthrough
+

Do not apply the transposition if the input geometry matches the one +specified by the specified value. It accepts the following values: +

+
‘none’
+

Always apply transposition. (default) +

+
‘portrait’
+

Preserve portrait geometry (when height >= width). +

+
‘landscape’
+

Preserve landscape geometry (when width >= height). +

+
+ +
+
+ + +

16 Vulkan Video Filters

+ +

Below is a description of the currently available Vulkan video filters. +

+

To enable compilation of these filters you need to configure FFmpeg with +--enable-vulkan and either --enable-libglslang or --enable-libshaderc. +

+

Running Vulkan filters requires you to initialize a hardware device and to pass that device to all filters in any filter graph. +

+
-init_hw_device vulkan[=name][:device[,key=value...]]
+

Initialise a new hardware device of type vulkan called name, using the +given device parameters and options in key=value. The following options +are supported: +

+
+
debug
+

Switches validation layers on if set to 1. +

+
+
linear_images
+

Allocates linear images. Does not apply to decoding. +

+
+
disable_multiplane
+

Disables multiplane images. Does not apply to decoding. +

+
+
avoid_host_import
+

Avoids the use of dynamic host memory imports, in favor of a regular memcpy() +into a previously mapped buffer. +

+
+ +
+
-filter_hw_device name
+

Pass the hardware device called name to all filters in any filter graph. +

+
+
+ +

For more detailed information see https://www.ffmpeg.org/ffmpeg.html#Advanced-Video-options +

+
    +
  • Example of choosing the first device and running nlmeans_vulkan filter with default parameters on it. +
    +
    -init_hw_device vulkan=vk:0 -filter_hw_device vk -i INPUT -vf "hwupload,nlmeans_vulkan,hwdownload" OUTPUT
    +
    +
+ +

As Vulkan filters are not able to access frame data in normal memory, all frame data needs to be uploaded (hwupload) to hardware surfaces connected to the appropriate device before being used and then downloaded (hwdownload) back to normal memory. Note that hwupload will upload to a frame with the same layout as the software frame, so it may be necessary to add a format filter immediately before to get the input into the right format and hwdownload does not support all formats on the output - it is usually necessary to insert an additional format filter immediately following in the graph to get the output in a supported format. +

+

16.1 avgblur_vulkan

+ +

Apply an average blur filter, implemented on the GPU using Vulkan. +

+

The filter accepts the following options: +

+
+
sizeX
+

Set horizontal radius size. +Range is [1, 32] and default value is 3. +

+
+
sizeY
+

Set vertical radius size. Range is [1, 32] and default value is 3. +

+
+
planes
+

Set which planes to filter. Default value is 0xf, by which all planes are processed. +

+
+ +

16.2 blend_vulkan

+ +

Blend two Vulkan frames into each other. +

+

The blend filter takes two input streams and outputs one +stream, the first input is the "top" layer and second input is +"bottom" layer. By default, the output terminates when the longest input terminates. +

+

A description of the accepted options follows. +

+
+
c0_mode
+
c1_mode
+
c2_mode
+
c3_mode
+
all_mode
+

Set blend mode for specific pixel component or all pixel components in case +of all_mode. Default value is normal. +

+

Available values for component modes are: +

+
‘normal’
+
‘multiply’
+
+ +
+
+ +

16.3 bwdif_vulkan

+ +

Deinterlacer using bwdif, the "Bob Weaver Deinterlacing Filter" algorithm, implemented +on the GPU using Vulkan. +

+

It accepts the following parameters: +

+
+
mode
+

The interlacing mode to adopt. It accepts one of the following values: +

+
+
0, send_frame
+

Output one frame for each frame. +

+
1, send_field
+

Output one frame for each field. +

+
+ +

The default value is send_field. +

+
+
parity
+

The picture field parity assumed for the input interlaced video. It accepts one +of the following values: +

+
+
0, tff
+

Assume the top field is first. +

+
1, bff
+

Assume the bottom field is first. +

+
-1, auto
+

Enable automatic detection of field parity. +

+
+ +

The default value is auto. +If the interlacing is unknown or the decoder does not export this information, +top field first will be assumed. +

+
+
deint
+

Specify which frames to deinterlace. Accepts one of the following +values: +

+
+
0, all
+

Deinterlace all frames. +

+
1, interlaced
+

Only deinterlace frames marked as interlaced. +

+
+ +

The default value is all. +

+
+ +

16.4 chromaber_vulkan

+ +

Apply an effect that emulates chromatic aberration. Works best with RGB inputs, +but provides a similar effect with YCbCr inputs too. +

+
+
dist_x
+

Horizontal displacement multiplier. Each chroma pixel’s position will be multiplied +by this amount, starting from the center of the image. Default is 0. +

+
+
dist_y
+

Similarly, this sets the vertical displacement multiplier. Default is 0. +

+
+
+ +

16.5 color_vulkan

+ +

Video source that creates a Vulkan frame of a solid color. +Useful for benchmarking, or overlaying. +

+

It accepts the following parameters: +

+
+
color
+

The color to use. Either a name, or a hexadecimal value. +The default value is black. +

+
+
size
+

The size of the output frame. Default value is 1920x1080. +

+
+
rate
+

The framerate to output at. Default value is 60 frames per second. +

+
+
duration
+

The video duration. Default value is -0.000001. +

+
+
sar
+

The video signal aspect ratio. Default value is 1/1. +

+
+
format
+

The pixel format of the output Vulkan frames. Default value is yuv444p. +

+
+
out_range
+

Set the output YCbCr sample range. +

+

This allows the autodetected value to be overridden as well as allows forcing +a specific value used for the output and encoder. If not specified, the +range depends on the pixel format. Possible values: +

+
+
‘auto/unknown’
+

Choose automatically. +

+
+
‘jpeg/full/pc’
+

Set full range (0-255 in case of 8-bit luma). +

+
+
‘mpeg/limited/tv’
+

Set "MPEG" range (16-235 in case of 8-bit luma). +

+
+ +
+
+ +

16.6 vflip_vulkan

+ +

Flips an image vertically. +

+

16.7 hflip_vulkan

+ +

Flips an image horizontally. +

+

16.8 flip_vulkan

+ +

Flips an image along both the vertical and horizontal axis. +

+

16.9 gblur_vulkan

+ +

Apply Gaussian blur filter on Vulkan frames. +

+

The filter accepts the following options: +

+
+
sigma
+

Set horizontal sigma, standard deviation of Gaussian blur. Default is 0.5. +

+
+
sigmaV
+

Set vertical sigma, if negative it will be same as sigma. +Default is -1. +

+
+
planes
+

Set which planes to filter. By default all planes are filtered. +

+
+
size
+

Set the kernel size along the horizontal axis. Default is 19. +

+
+
sizeV
+

Set the kernel size along the vertical axis. Default is 0, +which sets to use the same value as size. +

+
+
+ +

16.10 nlmeans_vulkan

+ +

Denoise frames using Non-Local Means algorithm, implemented on the GPU using +Vulkan. +Supports more pixel formats than nlmeans or nlmeans_opencl, including +alpha channel support. +

+

The filter accepts the following options. +

+
+
s
+

Set denoising strength for all components. Default is 1.0. Must be in range [0.0, 100.0]. +

+
+
p
+

Set patch size for all planes. Default is 7. Must be odd number in range [0, 99]. +

+
+
r
+

Set research size. Default is 15. Must be odd number in range [0, 99]. +

+
+
t
+

Set parallelism. Default is 8. Must be a number in the range [1, 64]. +Larger values will use more VRAM but may not result in greater speed. +The optimal value is hardware and input dependent. +

+
+
s0
+
s1
+
s2
+
s3
+

Set denoising strength for a specific component. Default is 1.0, equal to s. +Must be in range [0.0, 100.0]. 0.0 disables denoising in that component. +

+
+
p0
+
p1
+
p2
+
p3
+

Set patch size for a specific component. Default is 7, equal to p. +Must be odd number in range [0, 99]. +

+
+
+ +

16.11 overlay_vulkan

+ +

Overlay one video on top of another. +

+

It takes two inputs and has one output. The first input is the "main" video on which the second input is overlaid. +This filter requires all inputs to use the same pixel format. So, format conversion may be needed. +

+

The filter accepts the following options: +

+
+
x
+

Set the x coordinate of the overlaid video on the main video. +Default value is 0. +

+
+
y
+

Set the y coordinate of the overlaid video on the main video. +Default value is 0. +

+
+
+ +

16.12 transpose_vulkan

+ +

Transpose rows with columns in the input video and optionally flip it. +For more in depth examples see the transpose video filter, which shares mostly the same options. +

+

It accepts the following parameters: +

+
+
dir
+

Specify the transposition direction. +

+

Can assume the following values: +

+
‘cclock_flip’
+

Rotate by 90 degrees counterclockwise and vertically flip. (default) +

+
+
‘clock’
+

Rotate by 90 degrees clockwise. +

+
+
‘cclock’
+

Rotate by 90 degrees counterclockwise. +

+
+
‘clock_flip’
+

Rotate by 90 degrees clockwise and vertically flip. +

+
+ +
+
passthrough
+

Do not apply the transposition if the input geometry matches the one +specified by the specified value. It accepts the following values: +

+
‘none’
+

Always apply transposition. (default) +

+
‘portrait’
+

Preserve portrait geometry (when height >= width). +

+
‘landscape’
+

Preserve landscape geometry (when width >= height). +

+
+ +
+
+ + +

17 QSV Video Filters

+ +

Below is a description of the currently available QSV video filters. +

+

To enable compilation of these filters you need to configure FFmpeg with +--enable-libmfx or --enable-libvpl. +

+

To use QSV filters, you need to setup the QSV device correctly. For more information, please read https://trac.ffmpeg.org/wiki/Hardware/QuickSync +

+

17.1 hstack_qsv

+

Stack input videos horizontally. +

+

This is the QSV variant of the hstack filter, each input stream may +have different height, this filter will scale down/up each input stream while +keeping the original aspect. +

+

It accepts the following options: +

+
+
inputs
+

See hstack. +

+
+
shortest
+

See hstack. +

+
+
height
+

Set height of output. If set to 0, this filter will set height of output to +height of the first input stream. Default value is 0. +

+
+ +

17.2 vstack_qsv

+

Stack input videos vertically. +

+

This is the QSV variant of the vstack filter, each input stream may +have different width, this filter will scale down/up each input stream while +keeping the original aspect. +

+

It accepts the following options: +

+
+
inputs
+

See vstack. +

+
+
shortest
+

See vstack. +

+
+
width
+

Set width of output. If set to 0, this filter will set width of output to +width of the first input stream. Default value is 0. +

+
+ +

17.3 xstack_qsv

+

Stack video inputs into custom layout. +

+

This is the QSV variant of the xstack filter. +

+

It accepts the following options: +

+
+
inputs
+

See xstack. +

+
+
shortest
+

See xstack. +

+
+
layout
+

See xstack. +Moreover, this permits the user to supply output size for each input stream. +

+
xstack_qsv=inputs=4:layout=0_0_1920x1080|0_h0_1920x1080|w0_0_1920x1080|w0_h0_1920x1080
+
+ +
+
grid
+

See xstack. +

+
+
grid_tile_size
+

Set output size for each input stream when grid is set. If this option +is not set, this filter will set output size by default to the size of the +first input stream. For the syntax of this option, check the +(ffmpeg-utils)"Video size" section in the ffmpeg-utils manual. +

+
+
fill
+

See xstack. +

+
+ + +

18 Video Sources

+ +

Below is a description of the currently available video sources. +

+

18.1 buffer

+ +

Buffer video frames, and make them available to the filter chain. +

+

This source is mainly intended for a programmatic use, in particular +through the interface defined in libavfilter/buffersrc.h. +

+

It accepts the following parameters: +

+
+
video_size
+

Specify the size (width and height) of the buffered video frames. For the +syntax of this option, check the +(ffmpeg-utils)"Video size" section in the ffmpeg-utils manual. +

+
+
width
+

The input video width. +

+
+
height
+

The input video height. +

+
+
pix_fmt
+

A string representing the pixel format of the buffered video frames. +It may be a number corresponding to a pixel format, or a pixel format +name. +

+
+
time_base
+

Specify the timebase assumed by the timestamps of the buffered frames. +

+
+
frame_rate
+

Specify the frame rate expected for the video stream. +

+
+
colorspace
+

A string representing the color space of the buffered video frames. +It may be a number corresponding to a color space, or a color space +name. +

+
+
range
+

A string representing the color range of the buffered video frames. +It may be a number corresponding to a color range, or a color range +name. +

+
+
alpha_mode
+

A string representing the alpha mode of the buffered video frames. +It may be a number corresponding to an alpha mode, or an alpha mode +name. +

+
+
pixel_aspect, sar
+

The sample (pixel) aspect ratio of the input video. +

+
+
hw_frames_ctx
+

When using a hardware pixel format, this should be a reference to an +AVHWFramesContext describing input frames. +

+
+ +

For example: +

+
buffer=width=320:height=240:pix_fmt=yuv410p:time_base=1/24:sar=1
+
+ +

will instruct the source to accept video frames with size 320x240 and +with format "yuv410p", assuming 1/24 as the timestamps timebase and +square pixels (1:1 sample aspect ratio). +Since the pixel format with name "yuv410p" corresponds to the number 6 +(check the enum AVPixelFormat definition in libavutil/pixfmt.h), +this example corresponds to: +

+
buffer=size=320x240:pixfmt=6:time_base=1/24:pixel_aspect=1/1
+
+ +

Alternatively, the options can be specified as a flat string, but this +syntax is deprecated: +

+

width:height:pix_fmt:time_base.num:time_base.den:pixel_aspect.num:pixel_aspect.den +

+

18.2 cellauto

+ +

Create a pattern generated by an elementary cellular automaton. +

+

The initial state of the cellular automaton can be defined through the +filename and pattern options. If such options are +not specified an initial state is created randomly. +

+

At each new frame a new row in the video is filled with the result of +the cellular automaton next generation. The behavior when the whole +frame is filled is defined by the scroll option. +

+

This source accepts the following options: +

+
+
filename, f
+

Read the initial cellular automaton state, i.e. the starting row, from +the specified file. +In the file, each non-whitespace character is considered an alive +cell, a newline will terminate the row, and further characters in the +file will be ignored. +

+
+
pattern, p
+

Read the initial cellular automaton state, i.e. the starting row, from +the specified string. +

+

Each non-whitespace character in the string is considered an alive +cell, a newline will terminate the row, and further characters in the +string will be ignored. +

+
+
rate, r
+

Set the video rate, that is the number of frames generated per second. +Default is 25. +

+
+
random_fill_ratio, ratio
+

Set the random fill ratio for the initial cellular automaton row. It +is a floating point number value ranging from 0 to 1, defaults to +1/PHI. +

+

This option is ignored when a file or a pattern is specified. +

+
+
random_seed, seed
+

Set the seed for filling randomly the initial row, must be an integer +included between 0 and UINT32_MAX. If not specified, or if explicitly +set to -1, the filter will try to use a good random seed on a best +effort basis. +

+
+
rule
+

Set the cellular automaton rule, it is a number ranging from 0 to 255. +Default value is 110. +

+
+
size, s
+

Set the size of the output video. For the syntax of this option, check the +(ffmpeg-utils)"Video size" section in the ffmpeg-utils manual. +

+

If filename or pattern is specified, the size is set +by default to the width of the specified initial state row, and the +height is set to width * PHI. +

+

If size is set, it must contain the width of the specified +pattern string, and the specified pattern will be centered in the +larger row. +

+

If a filename or a pattern string is not specified, the size value +defaults to "320x518" (used for a randomly generated initial state). +

+
+
scroll
+

If set to 1, scroll the output upward when all the rows in the output +have been already filled. If set to 0, the new generated row will be +written over the top row just after the bottom row is filled. +Defaults to 1. +

+
+
start_full, full
+

If set to 1, completely fill the output with generated rows before +outputting the first frame. +This is the default behavior, for disabling set the value to 0. +

+
+
stitch
+

If set to 1, stitch the left and right row edges together. +This is the default behavior, for disabling set the value to 0. +

+
+ +

18.2.1 Examples

+ +
    +
  • Read the initial state from pattern, and specify an output of +size 200x400. +
    +
    cellauto=f=pattern:s=200x400
    +
    + +
  • Generate a random initial row with a width of 200 cells, with a fill +ratio of 2/3: +
    +
    cellauto=ratio=2/3:s=200x200
    +
    + +
  • Create a pattern generated by rule 18 starting by a single alive cell +centered on an initial row with width 100: +
    +
    cellauto=p=@:s=100x400:full=0:rule=18
    +
    + +
  • Specify a more elaborated initial pattern: +
    +
    cellauto=p='@@ @ @@':s=100x400:full=0:rule=18
    +
    + +
+ +

18.3 coreimagesrc

+

Video source generated on GPU using Apple’s CoreImage API on OSX. +

+

This video source is a specialized version of the coreimage video filter. +Use a core image generator at the beginning of the applied filterchain to +generate the content. +

+

The coreimagesrc video source accepts the following options: +

+
list_generators
+

List all available generators along with all their respective options as well as +possible minimum and maximum values along with the default values. +

+
list_generators=true
+
+ +
+
size, s
+

Specify the size of the sourced video. For the syntax of this option, check the +(ffmpeg-utils)"Video size" section in the ffmpeg-utils manual. +The default value is 320x240. +

+
+
rate, r
+

Specify the frame rate of the sourced video, as the number of frames +generated per second. It has to be a string in the format +frame_rate_num/frame_rate_den, an integer number, a floating point +number or a valid video frame rate abbreviation. The default value is +"25". +

+
+
sar
+

Set the sample aspect ratio of the sourced video. +

+
+
duration, d
+

Set the duration of the sourced video. See +(ffmpeg-utils)the Time duration section in the ffmpeg-utils(1) manual +for the accepted syntax. +

+

If not specified, or the expressed duration is negative, the video is +supposed to be generated forever. +

+
+ +

Additionally, all options of the coreimage video filter are accepted. +A complete filterchain can be used for further processing of the +generated input without CPU-HOST transfer. See coreimage documentation +and examples for details. +

+

18.3.1 Examples

+ +
    +
  • Use CIQRCodeGenerator to create a QR code for the FFmpeg homepage, +given as complete and escaped command-line for Apple’s standard bash shell: +
    +
    ffmpeg -f lavfi -i coreimagesrc=s=100x100:filter=CIQRCodeGenerator@inputMessage=https\\\\\://FFmpeg.org/@inputCorrectionLevel=H -frames:v 1 QRCode.png
    +
    +

    This example is equivalent to the QRCode example of coreimage without the +need for a nullsrc video source. +

+ + +

18.4 ddagrab

+ +

Captures the Windows Desktop via Desktop Duplication API. +

+

The filter exclusively returns D3D11 Hardware Frames, for on-gpu encoding +or processing. So an explicit hwdownload is needed for any kind of +software processing. +

+

It accepts the following options: +

+
+
output_idx
+

DXGI Output Index to capture. +

+

Usually corresponds to the index Windows has given the screen minus one, +so it’s starting at 0. +

+

Defaults to output 0. +

+
+
draw_mouse
+

Whether to draw the mouse cursor. +

+

Defaults to true. +

+

Only affects hardware cursors. If a game or application renders its own cursor, +it’ll always be captured. +

+
+
framerate
+

Maximum framerate at which the desktop will be captured - the interval between +successive frames will not be smaller than the inverse of the framerate. When +dup_frames is true (the default) and the desktop is not being updated +often enough, the filter will duplicate a previous frame. Note that there is no +background buffering going on, so when the filter is not polled often enough +then the actual inter-frame interval may be significantly larger. +

+

Defaults to 30 FPS. +

+
+
video_size
+

Specify the size of the captured video. +

+

Defaults to the full size of the screen. +

+

Cropped from the bottom/right if smaller than screen size. +

+
+
offset_x
+

Horizontal offset of the captured video. +

+
+
offset_y
+

Vertical offset of the captured video. +

+
+
output_fmt
+

Desired filter output format. +Defaults to 8 Bit BGRA. +

+

It accepts the following values: +

+
‘auto’
+

Passes all supported output formats to DDA and returns what DDA decides to use. +

+
‘8bit’
+
‘bgra’
+

8 Bit formats always work, and DDA will convert to them if necessary. +

+
‘10bit’
+
‘x2bgr10’
+

Filter initialization will fail if 10 bit format is requested but unavailable. +

+
+ +
+
dup_frames
+

When this option is set to true (the default), the filter will duplicate frames +when the desktop has not been updated in order to maintain approximately +constant target framerate. When this option is set to false, the filter will +wait for the desktop to be updated (inter-frame intervals may vary significantly +in this case). +

+
+
+ +

18.4.1 Examples

+ +

Capture primary screen and encode using nvenc: +

+
ffmpeg -f lavfi -i ddagrab -c:v h264_nvenc -cq 18 output.mp4
+
+ +

You can also skip the lavfi device and directly use the filter. +Also demonstrates downloading the frame and encoding with libx264. +Explicit output format specification is required in this case: +

+
ffmpeg -filter_complex ddagrab=output_idx=1:framerate=60,hwdownload,format=bgra -c:v libx264 -crf 18 output.mp4
+
+ +

If you want to capture only a subsection of the desktop, this can be achieved +by specifying a smaller size and its offsets into the screen: +

+
ddagrab=video_size=800x600:offset_x=100:offset_y=100
+
+ + +

18.5 gfxcapture

+ +

Capture windows or monitors using Windows.Graphics.Capture API. +

+

This source provides low overhead capture of application windows or entire monitors. +The filter outputs hardware frames in d3d11 format; use hwdownload,format= +if system memory frames are required. +

+

The window to be captured can be selected via regular expressions on its title, +class name or backing executable name, by explicit native handles, or by monitor +index or explicit native handle. A window must match all provided expressions to be +selected. The first matching window will be picked, in whatever order Windows +returns them. +

+

Explicit handles (hwnd, hmonitor) override pattern or index +based selection. If neither handles nor a monitor index are given, the first +window matching the provided regular expressions is captured. +

+

This source does NOT hold a stable FPS. It returns frames at whatever rate the compositor +provides them, only capped by the max_framerate. +If you need a stable rate, you need to add an fps filter to drop/duplicate frames as needed. +

+

If the capture source disappears mid-capture (window closed, monitor disconnected), the filter will return EOF. +

+

This source accepts the following options: +

+
window_title
+

ECMAScript regular expression matched against the window title. Supports a +PCRE style (?i) prefix for case-insensitive matching. +

+
+
window_class
+

As window_title, but matched against the window class name. +

+
+
window_exe
+

As window_title, but matched against the executable file name of the +window’s process. +

+
+
monitor_idx
+

Zero-based index of the monitor to capture. +

+

Can also be set to window to capture the monitor the selected window +is displayed on at filter initialization time. +

+
+
hwnd
+

Explicit native window handle (HWND). +

+
+
hmonitor
+

Explicit native monitor handle (HMONITOR). +

+
+
capture_cursor
+

Capture the mouse cursor. Enabled by default. +

+
+
capture_border
+

Capture the full area of the window, including its window decorarions/border. +Disabled by default. +

+
+
display_border
+

Draw a yellow highlight border around the captured window. Disabled by default. +

+
+
max_framerate
+

Maximum capture frame rate. Accepts a video rate (e.g. 30, 60/1, +24000/1001). The default is 60 FPS. +The actual rate is the rate at which the compositor renders the window/monitor, +capped by this option. +

+
+
width
+

Force the output canvas width. If zero (default) the initial captured source +width is used. If a negative number is provided, the width will be rounded +down to the next multiple of that number. +

+

See resize_mode. +

+
+
height
+

Force the output canvas height. If zero (default) the initial captured source +height is used. If a negative number is provided, the height will be rounded +down to the next multiple of that number. +

+

See resize_mode. +

+
+
crop_left
+

Crop this many pixels from the left side of the captured frames. +

+
+
crop_top
+

Crop this many pixels from the left side of the captured frames. +

+
+
crop_right
+

Crop this many pixels from the left side of the captured frames. +

+
+
crop_bottom
+

Crop this many pixels from the left side of the captured frames. +

+
+
premultiplied
+

If set to 1, return frames with premultiplied alpha. Default is 0 (straight +alpha). +

+
+
resize_mode
+

Defines how the captured content is fitted into the output canvas size. +Possible values: +

+
‘crop’
+

Crop (or pad with black) to the canvas size. (default) +

+
‘scale’
+

Scale the source to fill the canvas, potentially altering aspect ratio. +

+
‘scale_aspect’
+

Scale the source to fit inside the canvas while preserving aspect ratio. +Remaining area is filled with black. +

+
+ +
+
scale_mode
+

Scaling algorithm used when resizing is required. +

+

Possible values: +

+
‘point’
+

Nearest neighbour (pixelated) scaling. +

+
‘bilinear’
+

Bilinear filtering. (default) +

+
‘bicubic’
+

Bicubic filtering. Potentially more blurry, but fewer scaling artifacts depending on contents. +

+
+ +
+
output_fmt
+

Desired output pixel format inside the D3D11 hardware frames. +

+

Possible values: +

+
‘bgra’
+
‘8bit’
+

8 bit BGRA output (default) +

+
‘x2bgr10’
+
‘10bit’
+

10 bit BGR output +

+
‘rgbaf16’
+
‘16bit’
+

16bit float RGBA output +

+
+ +
+
+ +

18.5.1 Examples

+
    +
  • Capture a window by title (case-insensitive) at a maximum of 60 fps: +
    +
    ffmpeg -filter_complex gfxcapture=window_title='(?i)My Application':max_framerate=60,hwdownload,format=bgra,format=yuv420p -c:v libx264 -crf 15 capture.mp4
    +
    + +
  • Capture monitor 1 at native refresh, 10bit color depth, scale to 1920x1080 preserving aspect: +
    +
    ffmpeg -filter_complex gfxcapture=monitor_idx=1:width=1920:height=1080:resize_mode=scale_aspect:output_fmt=10bit -c:v hevc_nvenc -cq 15 capture.mp4
    +
    + +
  • Capture a window by executable name, draw border, force point scaling, fixed 60 fps: +
    +
    ffmpeg -filter_complex gfxcapture=window_exe='^firefox.exe$':display_border=1:scale_mode=point,fps=60 -rc qvbr -qvbr_quality_level 15 -c:v h264_amf capture.mp4
    +
    + +
+ + +

18.6 gradients

+

Generate several gradients. +

+
+
size, s
+

Set frame size. For the syntax of this option, check the (ffmpeg-utils)"Video +size" section in the ffmpeg-utils manual. Default value is "640x480". +

+
+
rate, r
+

Set frame rate, expressed as number of frames per second. Default +value is "25". +

+
+
c0, c1, c2, c3, c4, c5, c6, c7
+

Set 8 colors. Default values for colors is to pick random one. +

+
+
x0, y0, y0, y1
+

Set gradient line source and destination points. If negative or out of range, random ones +are picked. +

+
+
nb_colors, n
+

Set number of colors to use at once. Allowed range is from 2 to 8. Default value is 2. +

+
+
seed
+

Set seed for picking gradient line points. +

+
+
duration, d
+

Set the duration of the sourced video. See +(ffmpeg-utils)the Time duration section in the ffmpeg-utils(1) manual +for the accepted syntax. +

+

If not specified, or the expressed duration is negative, the video is +supposed to be generated forever. +

+
+
speed
+

Set speed of gradients rotation. +

+
+
type, t
+

Set type of gradients. +Available values are: +

+
‘linear’
+
‘radial’
+
‘circular’
+
‘spiral’
+
‘square’
+
+ +

Default type is linear. +

+
+ +

18.6.1 Commands

+ +

This source supports the some above options as commands. +

+

18.7 mandelbrot

+ +

Generate a Mandelbrot set fractal, and progressively zoom towards the +point specified with start_x and start_y. +

+

This source accepts the following options: +

+
+
end_pts
+

Set the terminal pts value. Default value is 400. +

+
+
end_scale
+

Set the terminal scale value. +Must be a floating point value. Default value is 0.3. +

+
+
inner
+

Set the inner coloring mode, that is the algorithm used to draw the +Mandelbrot fractal internal region. +

+

It shall assume one of the following values: +

+
black
+

Set black mode. +

+
convergence
+

Show time until convergence. +

+
mincol
+

Set color based on point closest to the origin of the iterations. +

+
period
+

Set period mode. +

+
+ +

Default value is mincol. +

+
+
bailout
+

Set the bailout value. Default value is 10.0. +

+
+
maxiter
+

Set the maximum of iterations performed by the rendering +algorithm. Default value is 7189. +

+
+
outer
+

Set outer coloring mode. +It shall assume one of following values: +

+
iteration_count
+

Set iteration count mode. +

+
normalized_iteration_count
+

set normalized iteration count mode. +

+
+

Default value is normalized_iteration_count. +

+
+
rate, r
+

Set frame rate, expressed as number of frames per second. Default +value is "25". +

+
+
size, s
+

Set frame size. For the syntax of this option, check the (ffmpeg-utils)"Video +size" section in the ffmpeg-utils manual. Default value is "640x480". +

+
+
start_scale
+

Set the initial scale value. Default value is 3.0. +

+
+
start_x
+

Set the initial x position. Must be a floating point value between +-100 and 100. Default value is -0.743643887037158704752191506114774. +

+
+
start_y
+

Set the initial y position. Must be a floating point value between +-100 and 100. Default value is -0.131825904205311970493132056385139. +

+
+ +

18.8 mptestsrc

+ +

Generate various test patterns, as generated by the MPlayer test filter. +

+

The size of the generated video is fixed, and is 512x512. +This source is useful in particular for testing encoding features. +

+

This source accepts the following options: +

+
+
rate, r
+

Specify the frame rate of the sourced video, as the number of frames +generated per second. It has to be a string in the format +frame_rate_num/frame_rate_den, an integer number, a floating point +number or a valid video frame rate abbreviation. The default value is +"25". +

+
+
duration, d
+

Set the duration of the sourced video. See +(ffmpeg-utils)the Time duration section in the ffmpeg-utils(1) manual +for the accepted syntax. +

+

If not specified, or the expressed duration is negative, the video is +supposed to be generated forever. +

+
+
test, t
+
+

Set the number or the name of the test to perform. Supported tests are: +

+
dc_luma
+
dc_chroma
+
freq_luma
+
freq_chroma
+
amp_luma
+
amp_chroma
+
cbp
+
mv
+
ring1
+
ring2
+
all
+
max_frames, m
+

Set the maximum number of frames generated for each test, default value is 30. +

+
+
+ +

Default value is "all", which will cycle through the list of all tests. +

+
+ +

Some examples: +

+
mptestsrc=t=dc_luma
+
+ +

will generate a "dc_luma" test pattern. +

+

18.9 frei0r_src

+ +

Provide a frei0r source. +

+

To enable compilation of this filter you need to install the frei0r +header and configure FFmpeg with --enable-frei0r. +

+

This source accepts the following parameters: +

+
+
size
+

The size of the video to generate. For the syntax of this option, check the +(ffmpeg-utils)"Video size" section in the ffmpeg-utils manual. +

+
+
framerate
+

The framerate of the generated video. It may be a string of the form +num/den or a frame rate abbreviation. +

+
+
filter_name
+

The name to the frei0r source to load. For more information regarding frei0r and +how to set the parameters, read the frei0r section in the video filters +documentation. +

+
+
filter_params
+

A ’|’-separated list of parameters to pass to the frei0r source. +

+
+
+ +

For example, to generate a frei0r partik0l source with size 200x200 +and frame rate 10 which is overlaid on the overlay filter main input: +

+
frei0r_src=size=200x200:framerate=10:filter_name=partik0l:filter_params=1234 [overlay]; [in][overlay] overlay
+
+ +

18.10 life

+ +

Generate a life pattern. +

+

This source is based on a generalization of John Conway’s life game. +

+

The sourced input represents a life grid, each pixel represents a cell +which can be in one of two possible states, alive or dead. Every cell +interacts with its eight neighbours, which are the cells that are +horizontally, vertically, or diagonally adjacent. +

+

At each interaction the grid evolves according to the adopted rule, +which specifies the number of neighbor alive cells which will make a +cell stay alive or born. The rule option allows one to specify +the rule to adopt. +

+

This source accepts the following options: +

+
+
filename, f
+

Set the file from which to read the initial grid state. In the file, +each non-whitespace character is considered an alive cell, and newline +is used to delimit the end of each row. +

+

If this option is not specified, the initial grid is generated +randomly. +

+
+
rate, r
+

Set the video rate, that is the number of frames generated per second. +Default is 25. +

+
+
random_fill_ratio, ratio
+

Set the random fill ratio for the initial random grid. It is a +floating point number value ranging from 0 to 1, defaults to 1/PHI. +It is ignored when a file is specified. +

+
+
random_seed, seed
+

Set the seed for filling the initial random grid, must be an integer +included between 0 and UINT32_MAX. If not specified, or if explicitly +set to -1, the filter will try to use a good random seed on a best +effort basis. +

+
+
rule
+

Set the life rule. +

+

A rule can be specified with a code of the kind "SNS/BNB", +where NS and NB are sequences of numbers in the range 0-8, +NS specifies the number of alive neighbor cells which make a +live cell stay alive, and NB the number of alive neighbor cells +which make a dead cell to become alive (i.e. to "born"). +"s" and "b" can be used in place of "S" and "B", respectively. +

+

Alternatively a rule can be specified by an 18-bits integer. The 9 +high order bits are used to encode the next cell state if it is alive +for each number of neighbor alive cells, the low order bits specify +the rule for "borning" new cells. Higher order bits encode for an +higher number of neighbor cells. +For example the number 6153 = (12<<9)+9 specifies a stay alive +rule of 12 and a born rule of 9, which corresponds to "S23/B03". +

+

Default value is "S23/B3", which is the original Conway’s game of life +rule, and will keep a cell alive if it has 2 or 3 neighbor alive +cells, and will born a new cell if there are three alive cells around +a dead cell. +

+
+
size, s
+

Set the size of the output video. For the syntax of this option, check the +(ffmpeg-utils)"Video size" section in the ffmpeg-utils manual. +

+

If filename is specified, the size is set by default to the +same size of the input file. If size is set, it must contain +the size specified in the input file, and the initial grid defined in +that file is centered in the larger resulting area. +

+

If a filename is not specified, the size value defaults to "320x240" +(used for a randomly generated initial grid). +

+
+
stitch
+

If set to 1, stitch the left and right grid edges together, and the +top and bottom edges also. Defaults to 1. +

+
+
mold
+

Set cell mold speed. If set, a dead cell will go from death_color to +mold_color with a step of mold. mold can have a +value from 0 to 255. +

+
+
life_color
+

Set the color of living (or new born) cells. +

+
+
death_color
+

Set the color of dead cells. If mold is set, this is the first color +used to represent a dead cell. +

+
+
mold_color
+

Set mold color, for definitely dead and moldy cells. +

+

For the syntax of these 3 color options, check the (ffmpeg-utils)"Color" section in the +ffmpeg-utils manual. +

+
+ +

18.10.1 Examples

+ +
    +
  • Read a grid from pattern, and center it on a grid of size +300x300 pixels: +
    +
    life=f=pattern:s=300x300
    +
    + +
  • Generate a random grid of size 200x200, with a fill ratio of 2/3: +
    +
    life=ratio=2/3:s=200x200
    +
    + +
  • Specify a custom rule for evolving a randomly generated grid: +
    +
    life=rule=S14/B34
    +
    + +
  • Full example with slow death effect (mold) using ffplay: +
    +
    ffplay -f lavfi life=s=300x200:mold=10:r=60:ratio=0.1:death_color=#C83232:life_color=#00ff00,scale=1200:800:flags=16
    +
    +
+ +

18.11 perlin

+

Generate Perlin noise. +

+

Perlin noise is a kind of noise with local continuity in space. This +can be used to generate patterns with continuity in space and time, +e.g. to simulate smoke, fluids, or terrain. +

+

In case more than one octave is specified through the octaves +option, Perlin noise is generated as a sum of components, each one +with doubled frequency. In this case the persistence option +specify the ratio of the amplitude with respect to the previous +component. More octave components enable to specify more high +frequency details in the generated noise (e.g. small size variations +due to boulders in a generated terrain). +

+

18.11.1 Options

+
+
size, s
+

Specify the size (width and height) of the buffered video frames. For the +syntax of this option, check the +(ffmpeg-utils)"Video size" section in the ffmpeg-utils manual. +Default value is 320x240. +

+
+
rate, r
+

Specify the frame rate expected for the video stream, expressed as a +number of frames per second. Default value is 25. +

+
+
octaves
+

Specify the total number of components making up the noise, each one +with doubled frequency. Default value is 1. +

+
+
persistence
+

Set the ratio used to compute the amplitude of the next octave +component with respect to the previous component amplitude. Default +value is 1. +

+
+
xscale
+
yscale
+

Define a scale factor used to multiple the x, y coordinates. This can +be useful to define an effect with a pattern stretched along the x or +y axis. Default value is 1. +

+
+
tscale
+

Define a scale factor used to multiple the time coordinate. This can +be useful to change the time variation speed. Default value is 1. +

+
+
random_mode
+

Set random mode used to compute initial pattern. +

+

Supported values are: +

+
random
+

Compute and use random seed. +

+
+
ken
+

Use the predefined initial pattern defined by Ken Perlin in the +original article, can be useful to compare the output with other +sources. +

+
+
seed
+

Use the value specified by random_seed option. +

+
+ +

Default value is random. +

+
+
random_seed, seed
+

When random_mode is set to random_seed, use this value +to compute the initial pattern. Default value is 0. +

+
+ +

18.11.2 Examples

+
    +
  • Generate single component: +
    +
    perlin
    +
    + +
  • Use Perlin noise with 7 components, each one with a halved contribution +to total amplitude: +
    +
    perlin=octaves=7:persistence=0.5
    +
    + +
  • Chain Perlin noise with the lutyuv to generate a black&white +effect: +
    +
    perlin=octaves=3:tscale=0.3,lutyuv=y='if(lt(val\,128)\,255\,0)'
    +
    + +
  • Stretch noise along the y axis, and convert gray level to red-only +signal: +
    +
    perlin=octaves=7:tscale=0.4:yscale=0.3,lutrgb=r=val:b=0:g=0
    +
    +
+ +

18.12 qrencodesrc

+ +

Generate a QR code using the libqrencode library (see +https://fukuchi.org/works/qrencode/). +

+

To enable the compilation of this source, you need to configure FFmpeg with +--enable-libqrencode. +

+

The QR code is generated from the provided text or text pattern. The +corresponding QR code is scaled and put in the video output according to the +specified output size options. +

+

In case no text is specified, the QR code is not generated, but an empty colored +output is returned instead. +

+

This source accepts the following options: +

+
+
qrcode_width, q
+
padded_qrcode_width, Q
+

Specify an expression for the width of the rendered QR code, with and without +padding. The qrcode_width expression can reference the value set by the +padded_qrcode_width expression, and vice versa. +By default padded_qrcode_width is set to qrcode_width, meaning that +there is no padding. +

+

These expressions are evaluated only once, when initializing the source. +See the qrencode Expressions section for details. +

+

Note that some of the constants are missing for the source (for example the +x or t or ¸n), since they only makes sense when evaluating the +expression for each frame rather than at initialization time. +

+
+
rate, r
+

Specify the frame rate of the sourced video, as the number of frames +generated per second. It has to be a string in the format +frame_rate_num/frame_rate_den, an integer number, a floating point +number or a valid video frame rate abbreviation. The default value is +"25". +

+
+
case_sensitive, cs
+

Instruct libqrencode to use case sensitive encoding. This is enabled by +default. This can be disabled to reduce the QR encoding size. +

+
+
level, l
+

Specify the QR encoding error correction level. With an higher correction level, +the encoding size will increase but the code will be more robust to corruption. +Lower level is L. +

+

It accepts the following values: +

+
‘L’
+
‘M’
+
‘Q’
+
‘H’
+
+ +
+
expansion
+

Select how the input text is expanded. Can be either none, or +normal (default). See the qrencode Text expansion +section for details. +

+
+
text
+
textfile
+

Define the text to be rendered. In case neither is specified, no QR is encoded +(just an empty colored frame). +

+

In case expansion is enabled, the text is treated as a text template, using the +qrencode expansion mechanism. See the qrencode +Text expansion section for details. +

+
+
background_color, bc
+
foreground_color, fc
+

Set the QR code and background color. The default value of +foreground_color is "black", the default value of background_color +is "white". +

+

For the syntax of the color options, check the (ffmpeg-utils)"Color" +section in the ffmpeg-utils manual. +

+
+ +

18.12.1 Examples

+ +
    +
  • Generate a QR code encoding the specified text with the default size: +
    +
    qrencodesrc=text=www.ffmpeg.org
    +
    + +
  • Same as below, but select blue on pink colors: +
    +
    qrencodesrc=text=www.ffmpeg.org:bc=pink:fc=blue
    +
    + +
  • Generate a QR code with width of 200 pixels and padding, making the padded width +4/3 of the QR code width: +
    +
    qrencodesrc=text=www.ffmpeg.org:q=200:Q=4/3*q
    +
    + +
  • Generate a QR code with padded width of 200 pixels and padding, making the QR +code width 3/4 of the padded width: +
    +
    qrencodesrc=text=www.ffmpeg.org:Q=200:q=3/4*Q
    +
    + +
  • Generate a QR code encoding the frame number: +
    +
    qrencodesrc=text=%{n}
    +
    + +
  • Generate a QR code encoding the GMT timestamp: +
    +
    qrencodesrc=text=%{gmtime}
    +
    + +
  • Generate a QR code encoding the timestamp expressed as a float: +
    +
    qrencodesrc=text=%{pts}
    +
    + +
+ +

18.13 allrgb, allyuv, color, colorchart, colorspectrum, haldclutsrc, nullsrc, pal75bars, pal100bars, rgbtestsrc, smptebars, smptehdbars, testsrc, testsrc2, yuvtestsrc

+ +

The allrgb source returns frames of size 4096x4096 of all rgb colors. +

+

The allyuv source returns frames of size 4096x4096 of all yuv colors. +

+

The color source provides an uniformly colored input. +

+

The colorchart source provides a colors checker chart. +

+

The colorspectrum source provides a color spectrum input. +

+

The haldclutsrc source provides an identity Hald CLUT. See also +haldclut filter. +

+

The nullsrc source returns unprocessed video frames. It is +mainly useful to be employed in analysis / debugging tools, or as the +source for filters which ignore the input data. +

+

The pal75bars source generates a color bars pattern, based on +EBU PAL recommendations with 75% color levels. +

+

The pal100bars source generates a color bars pattern, based on +EBU PAL recommendations with 100% color levels. +

+

The rgbtestsrc source generates an RGB test pattern useful for +detecting RGB vs BGR issues. You should see a red, green and blue +stripe from top to bottom. +

+

The smptebars source generates a color bars pattern, based on +the SMPTE Engineering Guideline EG 1-1990. +

+

The smptehdbars source generates a color bars pattern, based on +the SMPTE RP 219-2002. +

+

The testsrc source generates a test video pattern, showing a +color pattern, a scrolling gradient and a timestamp. This is mainly +intended for testing purposes. +

+

The testsrc2 source is similar to testsrc, but supports more +pixel formats instead of just rgb24. This allows using it as an +input for other tests without requiring a format conversion. +

+

The yuvtestsrc source generates an YUV test pattern. You should +see a y, cb and cr stripe from top to bottom. +

+

The sources accept the following parameters: +

+
+
level
+

Specify the level of the Hald CLUT, only available in the haldclutsrc +source. A level of N generates a picture of N*N*N by N*N*N +pixels to be used as identity matrix for 3D lookup tables. Each component is +coded on a 1/(N*N) scale. +

+
+
color, c
+

Specify the color of the source, only available in the color +source. For the syntax of this option, check the +(ffmpeg-utils)"Color" section in the ffmpeg-utils manual. +

+
+
size, s
+

Specify the size of the sourced video. For the syntax of this option, check the +(ffmpeg-utils)"Video size" section in the ffmpeg-utils manual. +The default value is 320x240. +

+

This option is not available with the allrgb, allyuv, and +haldclutsrc filters. +

+
+
rate, r
+

Specify the frame rate of the sourced video, as the number of frames +generated per second. It has to be a string in the format +frame_rate_num/frame_rate_den, an integer number, a floating point +number or a valid video frame rate abbreviation. The default value is +"25". +

+
+
duration, d
+

Set the duration of the sourced video. See +(ffmpeg-utils)the Time duration section in the ffmpeg-utils(1) manual +for the accepted syntax. +

+

If not specified, or the expressed duration is negative, the video is +supposed to be generated forever. +

+

Since the frame rate is used as time base, all frames including the last one +will have their full duration. If the specified duration is not a multiple +of the frame duration, it will be rounded up. +

+
+
sar
+

Set the sample aspect ratio of the sourced video. +

+
+
alpha
+

Specify the alpha (opacity) of the background, only available in the +testsrc2 source. The value must be between 0 (fully transparent) and +255 (fully opaque, the default). +

+
+
decimals, n
+

Set the number of decimals to show in the timestamp, only available in the +testsrc source. +

+

The displayed timestamp value will correspond to the original +timestamp value multiplied by the power of 10 of the specified +value. Default value is 0. +

+
+
type
+

Set the type of the color spectrum, only available in the +colorspectrum source. Can be one of the following: +

+
+
‘black’
+
‘white’
+
‘all’
+
+ +
+
patch_size
+

Set patch size of single color patch, only available in the +colorchart source. Default is 64x64. +

+
+
preset
+

Set colorchecker colors preset, only available in the +colorchart source. +

+

Available values are: +

+
‘reference’
+
‘skintones’
+
+ +

Default value is reference. +

+
+ +

18.13.1 Examples

+ +
    +
  • Generate a video with a duration of 5.3 seconds, with size +176x144 and a frame rate of 10 frames per second: +
    +
    testsrc=duration=5.3:size=qcif:rate=10
    +
    + +
  • The following graph description will generate a red source +with an opacity of 0.2, with size "qcif" and a frame rate of 10 +frames per second: +
    +
    color=c=red@0.2:s=qcif:r=10
    +
    + +
  • If the input content is to be ignored, nullsrc can be used. The +following command generates noise in the luma plane by employing +the geq filter: +
    +
    nullsrc=s=256x256, geq=random(1)*255:128:128
    +
    +
+ +

18.13.2 Commands

+ +

The color source supports the following commands: +

+
+
c, color
+

Set the color of the created image. Accepts the same syntax of the +corresponding color option. +

+
+ +

18.14 openclsrc

+ +

Generate video using an OpenCL program. +

+
+
source
+

OpenCL program source file. +

+
+
kernel
+

Kernel name in program. +

+
+
size, s
+

Size of frames to generate. This must be set. +

+
+
format
+

Pixel format to use for the generated frames. This must be set. +

+
+
rate, r
+

Number of frames generated every second. Default value is ’25’. +

+
+
+ +

For details of how the program loading works, see the program_opencl +filter. +

+

Example programs: +

+
    +
  • Generate a colour ramp by setting pixel values from the position of the pixel +in the output image. (Note that this will work with all pixel formats, but +the generated output will not be the same.) +
    __kernel void ramp(__write_only image2d_t dst,
    +                   unsigned int index)
    +{
    +    int2 loc = (int2)(get_global_id(0), get_global_id(1));
    +
    +    float4 val;
    +    val.xy = val.zw = convert_float2(loc) / convert_float2(get_image_dim(dst));
    +
    +    write_imagef(dst, loc, val);
    +}
    +
    +
  • Generate a Sierpinski carpet pattern, panning by a single pixel each frame. +
    __kernel void sierpinski_carpet(__write_only image2d_t dst,
    +                                unsigned int index)
    +{
    +    int2 loc = (int2)(get_global_id(0), get_global_id(1));
    +
    +    float4 value = 0.0f;
    +    int x = loc.x + index;
    +    int y = loc.y + index;
    +    while (x > 0 || y > 0) {
    +        if (x % 3 == 1 && y % 3 == 1) {
    +            value = 1.0f;
    +            break;
    +        }
    +        x /= 3;
    +        y /= 3;
    +    }
    +
    +    write_imagef(dst, loc, value);
    +}
    +
    +
+ +

18.15 sierpinski

+ +

Generate a Sierpinski carpet/triangle fractal, and randomly pan around. +

+

This source accepts the following options: +

+
+
size, s
+

Set frame size. For the syntax of this option, check the (ffmpeg-utils)"Video +size" section in the ffmpeg-utils manual. Default value is "640x480". +

+
+
rate, r
+

Set frame rate, expressed as number of frames per second. Default +value is "25". +

+
+
seed
+

Set seed which is used for random panning. +

+
+
jump
+

Set max jump for single pan destination. Allowed range is from 1 to 10000. +

+
+
type
+

Set fractal type, can be default carpet or triangle. +

+
+ +

18.16 zoneplate

+

Generate a zoneplate test video pattern. +

+

This source accepts the following options: +

+
+
size, s
+

Set frame size. For the syntax of this option, check the (ffmpeg-utils)"Video +size" section in the ffmpeg-utils manual. Default value is "320x240". +

+
+
rate, r
+

Set frame rate, expressed as number of frames per second. Default +value is "25". +

+
+
duration, d
+

Set the duration of the sourced video. See +(ffmpeg-utils)the Time duration section in the ffmpeg-utils(1) manual +for the accepted syntax. +

+

If not specified, or the expressed duration is negative, the video is +supposed to be generated forever. +

+
+
sar
+

Set the sample aspect ratio of the sourced video. +

+
+
precision
+

Set precision in bits for look-up table for sine calculations. Default value is 10. +Allowed range is from 4 to 16. +

+
+
xo
+

Set horizontal axis offset for output signal. Default value is 0. +

+
+
yo
+

Set vertical axis offset for output signal. Default value is 0. +

+
+
to
+

Set time axis offset for output signal. Default value is 0. +

+
+
k0
+

Set 0-order, constant added to signal phase. Default value is 0. +

+
+
kx
+

Set 1-order, phase factor multiplier for horizontal axis. Default value is 0. +

+
+
ky
+

Set 1-order, phase factor multiplier for vertical axis. Default value is 0. +

+
+
kt
+

Set 1-order, phase factor multiplier for time axis. Default value is 0. +

+
+
kxt, kyt, kxy
+

Set phase factor multipliers for combination of spatial and temporal axis. +Default value is 0. +

+
+
kx2
+

Set 2-order, phase factor multiplier for horizontal axis. Default value is 0. +

+
+
ky2
+

Set 2-order, phase factor multiplier for vertical axis. Default value is 0. +

+
+
kt2
+

Set 2-order, phase factor multiplier for time axis. Default value is 0. +

+
+
ku
+

Set the constant added to final phase to produce chroma-blue component of signal. +Default value is 0. +

+
+
kv
+

Set the constant added to final phase to produce chroma-red component of signal. +Default value is 0. +

+
+ +

18.16.1 Commands

+ +

This source supports the some above options as commands. +

+

18.16.2 Examples

+ +
    +
  • Generate horizontal color sine sweep: +
    +
    zoneplate=ku=512:kv=0:kt2=0:kx2=256:s=wvga:xo=-426:kt=11
    +
    + +
  • Generate vertical color sine sweep: +
    +
    zoneplate=ku=512:kv=0:kt2=0:ky2=156:s=wvga:yo=-240:kt=11
    +
    + +
  • Generate circular zone-plate: +
    +
    zoneplate=ku=512:kv=100:kt2=0:ky2=256:kx2=556:s=wvga:yo=0:kt=11
    +
    +
+ + +

19 Video Sinks

+ +

Below is a description of the currently available video sinks. +

+

19.1 buffersink

+ +

Buffer video frames, and make them available to the end of the filter +graph. +

+

This sink is mainly intended for programmatic use, in particular +through the interface defined in libavfilter/buffersink.h +or the options system. +

+

It accepts a pointer to an AVBufferSinkContext structure, which +defines the incoming buffers’ formats, to be passed as the opaque +parameter to avfilter_init_filter for initialization. +

+

19.2 nullsink

+ +

Null video sink: do absolutely nothing with the input video. It is +mainly useful as a template and for use in analysis / debugging +tools. +

+ +

20 Multimedia Filters

+ +

Below is a description of the currently available multimedia filters. +

+

20.1 a3dscope

+ +

Convert input audio to 3d scope video output. +

+

The filter accepts the following options: +

+
+
rate, r
+

Set frame rate, expressed as number of frames per second. Default +value is "25". +

+
+
size, s
+

Specify the video size for the output. For the syntax of this option, check the +(ffmpeg-utils)"Video size" section in the ffmpeg-utils manual. +Default value is hd720. +

+
+
fov
+

Set the camera field of view. Default is 90 degrees. +Allowed range is from 40 to 150. +

+
+
roll
+

Set the camera roll. +

+
+
pitch
+

Set the camera pitch. +

+
+
yaw
+

Set the camera yaw. +

+
+
xzoom
+

Set the camera zoom on X-axis. +

+
+
yzoom
+

Set the camera zoom on Y-axis. +

+
+
zzoom
+

Set the camera zoom on Z-axis. +

+
+
xpos
+

Set the camera position on X-axis. +

+
+
ypos
+

Set the camera position on Y-axis. +

+
+
zpos
+

Set the camera position on Z-axis. +

+
+
length
+

Set the length of displayed audio waves in number of frames. +

+
+ +

20.1.1 Commands

+ +

Filter supports the some above options as commands. +

+

20.2 abitscope

+ +

Convert input audio to a video output, displaying the audio bit scope. +

+

The filter accepts the following options: +

+
+
rate, r
+

Set frame rate, expressed as number of frames per second. Default +value is "25". +

+
+
size, s
+

Specify the video size for the output. For the syntax of this option, check the +(ffmpeg-utils)"Video size" section in the ffmpeg-utils manual. +Default value is 1024x256. +

+
+
colors
+

Specify list of colors separated by space or by ’|’ which will be used to +draw channels. Unrecognized or missing colors will be replaced +by white color. +

+
+
mode, m
+

Set output mode. Can be bars or trace. Default is bars. +

+
+ +

20.3 adrawgraph

+

Draw a graph using input audio metadata. +

+

See drawgraph +

+

20.4 agraphmonitor

+ +

See graphmonitor. +

+

20.5 ahistogram

+ +

Convert input audio to a video output, displaying the volume histogram. +

+

The filter accepts the following options: +

+
+
dmode
+

Specify how histogram is calculated. +

+

It accepts the following values: +

+
‘single’
+

Use single histogram for all channels. +

+
‘separate’
+

Use separate histogram for each channel. +

+
+

Default is single. +

+
+
rate, r
+

Set frame rate, expressed as number of frames per second. Default +value is "25". +

+
+
size, s
+

Specify the video size for the output. For the syntax of this option, check the +(ffmpeg-utils)"Video size" section in the ffmpeg-utils manual. +Default value is hd720. +

+
+
scale
+

Set display scale. +

+

It accepts the following values: +

+
‘log’
+

logarithmic +

+
‘sqrt’
+

square root +

+
‘cbrt’
+

cubic root +

+
‘lin’
+

linear +

+
‘rlog’
+

reverse logarithmic +

+
+

Default is log. +

+
+
ascale
+

Set amplitude scale. +

+

It accepts the following values: +

+
‘log’
+

logarithmic +

+
‘lin’
+

linear +

+
+

Default is log. +

+
+
acount
+

Set how much frames to accumulate in histogram. +Default is 1. Setting this to -1 accumulates all frames. +

+
+
rheight
+

Set histogram ratio of window height. +

+
+
slide
+

Set sonogram sliding. +

+

It accepts the following values: +

+
‘replace’
+

replace old rows with new ones. +

+
‘scroll’
+

scroll from top to bottom. +

+
+

Default is replace. +

+
+
hmode
+

Set histogram mode. +

+

It accepts the following values: +

+
‘abs’
+

Use absolute values of samples. +

+
‘sign’
+

Use untouched values of samples. +

+
+

Default is abs. +

+
+ +

20.6 aphasemeter

+ +

Measures phase of input audio, which is exported as metadata lavfi.aphasemeter.phase, +representing mean phase of current audio frame. A video output can also be produced and is +enabled by default. The audio is passed through as first output. +

+

Audio will be rematrixed to stereo if it has a different channel layout. Phase value is in +range [-1, 1] where -1 means left and right channels are completely out of phase +and 1 means channels are in phase. +

+

The filter accepts the following options, all related to its video output: +

+
+
rate, r
+

Set the output frame rate. Default value is 25. +

+
+
size, s
+

Set the video size for the output. For the syntax of this option, check the +(ffmpeg-utils)"Video size" section in the ffmpeg-utils manual. +Default value is 800x400. +

+
+
rc
+
gc
+
bc
+

Specify the red, green, blue contrast. Default values are 2, +7 and 1. +Allowed range is [0, 255]. +

+
+
mpc
+

Set color which will be used for drawing median phase. If color is +none which is default, no median phase value will be drawn. +

+
+
video
+

Enable video output. Default is enabled. +

+
+ +

20.6.1 phasing detection

+ +

The filter also detects out of phase and mono sequences in stereo streams. +It logs the sequence start, end and duration when it lasts longer or as long as the minimum set. +

+

The filter accepts the following options for this detection: +

+
+
phasing
+

Enable mono and out of phase detection. Default is disabled. +

+
+
tolerance, t
+

Set phase tolerance for mono detection, in amplitude ratio. Default is 0. +Allowed range is [0, 1]. +

+
+
angle, a
+

Set angle threshold for out of phase detection, in degree. Default is 170. +Allowed range is [90, 180]. +

+
+
duration, d
+

Set mono or out of phase duration until notification, expressed in seconds. Default is 2. +

+
+ +

20.6.2 Examples

+ +
    +
  • Complete example with ffmpeg to detect 1 second of mono with 0.001 phase tolerance: +
    +
    ffmpeg -i stereo.wav -af aphasemeter=video=0:phasing=1:duration=1:tolerance=0.001 -f null -
    +
    +
+ +

20.7 avectorscope

+ +

Convert input audio to a video output, representing the audio vector +scope. +

+

The filter is used to measure the difference between channels of stereo +audio stream. A monaural signal, consisting of identical left and right +signal, results in straight vertical line. Any stereo separation is visible +as a deviation from this line, creating a Lissajous figure. +If the straight (or deviation from it) but horizontal line appears this +indicates that the left and right channels are out of phase. +

+

The filter accepts the following options: +

+
+
mode, m
+

Set the vectorscope mode. +

+

Available values are: +

+
‘lissajous’
+

Lissajous rotated by 45 degrees. +

+
+
‘lissajous_xy’
+

Same as above but not rotated. +

+
+
‘polar’
+

Shape resembling half of circle. +

+
+ +

Default value is ‘lissajous’. +

+
+
size, s
+

Set the video size for the output. For the syntax of this option, check the +(ffmpeg-utils)"Video size" section in the ffmpeg-utils manual. +Default value is 400x400. +

+
+
rate, r
+

Set the output frame rate. Default value is 25. +

+
+
rc
+
gc
+
bc
+
ac
+

Specify the red, green, blue and alpha contrast. Default values are 40, +160, 80 and 255. +Allowed range is [0, 255]. +

+
+
rf
+
gf
+
bf
+
af
+

Specify the red, green, blue and alpha fade. Default values are 15, +10, 5 and 5. +Allowed range is [0, 255]. +

+
+
zoom
+

Set the zoom factor. Default value is 1. Allowed range is [0, 10]. +Values lower than 1 will auto adjust zoom factor to maximal possible value. +

+
+
draw
+

Set the vectorscope drawing mode. +

+

Available values are: +

+
‘dot’
+

Draw dot for each sample. +

+
+
‘line’
+

Draw line between previous and current sample. +

+
+
‘aaline’
+

Draw anti-aliased line between previous and current sample. +

+
+ +

Default value is ‘dot’. +

+
+
scale
+

Specify amplitude scale of audio samples. +

+

Available values are: +

+
‘lin’
+

Linear. +

+
+
‘sqrt’
+

Square root. +

+
+
‘cbrt’
+

Cubic root. +

+
+
‘log’
+

Logarithmic. +

+
+ +
+
swap
+

Swap left channel axis with right channel axis. +

+
+
mirror
+

Mirror axis. +

+
+
‘none’
+

No mirror. +

+
+
‘x’
+

Mirror only x axis. +

+
+
‘y’
+

Mirror only y axis. +

+
+
‘xy’
+

Mirror both axis. +

+
+ +
+
+ +

20.7.1 Examples

+ +
    +
  • Complete example using ffplay: +
    +
    ffplay -f lavfi 'amovie=input.mp3, asplit [a][out1];
    +             [a] avectorscope=zoom=1.3:rc=2:gc=200:bc=10:rf=1:gf=8:bf=7 [out0]'
    +
    +
+ +

20.7.2 Commands

+ +

This filter supports the all above options as commands except options size and rate. +

+

20.8 bench, abench

+ +

Benchmark part of a filtergraph. +

+

The filter accepts the following options: +

+
+
action
+

Start or stop a timer. +

+

Available values are: +

+
‘start’
+

Get the current time, set it as frame metadata (using the key +lavfi.bench.start_time), and forward the frame to the next filter. +

+
+
‘stop’
+

Get the current time and fetch the lavfi.bench.start_time metadata from +the input frame metadata to get the time difference. Time difference, average, +maximum and minimum time (respectively t, avg, max and +min) are then printed. The timestamps are expressed in seconds. +

+
+
+
+ +

20.8.1 Examples

+ +
    +
  • Benchmark selectivecolor filter: +
    +
    bench=start,selectivecolor=reds=-.2 .12 -.49,bench=stop
    +
    +
+ +

20.9 concat

+ +

Concatenate audio and video streams, joining them together one after the +other. +

+

The filter works on segments of synchronized video and audio streams. All +segments must have the same number of streams of each type, and that will +also be the number of streams at output. +

+

The filter accepts the following options: +

+
+
n
+

Set the number of segments. Default is 2. +

+
+
v
+

Set the number of output video streams, that is also the number of video +streams in each segment. Default is 1. +

+
+
a
+

Set the number of output audio streams, that is also the number of audio +streams in each segment. Default is 0. +

+
+
unsafe
+

Activate unsafe mode: do not fail if segments have a different format. +

+
+
+ +

The filter has v+a outputs: first v video outputs, then +a audio outputs. +

+

There are nx(v+a) inputs: first the inputs for the first +segment, in the same order as the outputs, then the inputs for the second +segment, etc. +

+

Related streams do not always have exactly the same duration, for various +reasons including codec frame size or sloppy authoring. For that reason, +related synchronized streams (e.g. a video and its audio track) should be +concatenated at once. The concat filter will use the duration of the longest +stream in each segment (except the last one), and if necessary pad shorter +audio streams with silence. +

+

For this filter to work correctly, all segments must start at timestamp 0. +

+

All corresponding streams must have the same parameters in all segments; the +filtering system will automatically select a common pixel format for video +streams, and a common sample format, sample rate and channel layout for +audio streams, but other settings, such as resolution, must be converted +explicitly by the user. +

+

Different frame rates are acceptable but will result in variable frame rate +at output; be sure to configure the output file to handle it. +

+

20.9.1 Examples

+ +
    +
  • Concatenate an opening, an episode and an ending, all in bilingual version +(video in stream 0, audio in streams 1 and 2): +
    +
    ffmpeg -i opening.mkv -i episode.mkv -i ending.mkv -filter_complex \
    +  '[0:0] [0:1] [0:2] [1:0] [1:1] [1:2] [2:0] [2:1] [2:2]
    +   concat=n=3:v=1:a=2 [v] [a1] [a2]' \
    +  -map '[v]' -map '[a1]' -map '[a2]' output.mkv
    +
    + +
  • Concatenate two parts, handling audio and video separately, using the +(a)movie sources, and adjusting the resolution: +
    +
    movie=part1.mp4, scale=512:288 [v1] ; amovie=part1.mp4 [a1] ;
    +movie=part2.mp4, scale=512:288 [v2] ; amovie=part2.mp4 [a2] ;
    +[v1] [v2] concat [outv] ; [a1] [a2] concat=v=0:a=1 [outa]
    +
    +

    Note that a desync will happen at the stitch if the audio and video streams +do not have exactly the same duration in the first file. +

    +
+ +

20.9.2 Commands

+ +

This filter supports the following commands: +

+
next
+

Close the current segment and step to the next one +

+
+ +

20.10 ebur128

+ +

EBU R128 scanner filter. This filter takes an audio stream and analyzes its loudness +level. By default, it logs a message at a frequency of 10Hz with the +Momentary loudness (identified by M), Short-term loudness (S), +Integrated loudness (I) and Loudness Range (LRA). +

+

The filter can only analyze streams which have +sample format is double-precision floating point. The input stream will be converted to +this specification, if needed. Users may need to insert aformat and/or aresample filters +after this filter to obtain the original parameters. +

+

The filter also has a video output (see the video option) with a real +time graph to observe the loudness evolution. The graphic contains the logged +message mentioned above, so it is not printed anymore when this option is set, +unless the verbose logging is set. The main graphing area contains the +short-term loudness (3 seconds of analysis), and the gauge on the right is for +the momentary loudness (400 milliseconds), but can optionally be configured +to instead display short-term loudness (see gauge). +

+

The green area marks a +/- 1LU target range around the target loudness +(-23LUFS by default, unless modified through target). +

+

More information about the Loudness Recommendation EBU R128 on +http://tech.ebu.ch/loudness. +

+

The filter accepts the following options: +

+
+
video
+

Activate the video output. The audio stream is passed unchanged whether this +option is set or no. The video stream will be the first output stream if +activated. Default is 0. +

+
+
size
+

Set the video size. This option is for video only. For the syntax of this +option, check the +(ffmpeg-utils)"Video size" section in the ffmpeg-utils manual. +Default and minimum resolution is 640x480. +

+
+
meter
+

Set the EBU scale meter. Default is 9. Common values are 9 and +18, respectively for EBU scale meter +9 and EBU scale meter +18. Any +other integer value between this range is allowed. +

+
+
metadata
+

Set metadata injection. If set to 1, the audio input will be segmented +into 100ms output frames, each of them containing various loudness information +in metadata. All the metadata keys are prefixed with lavfi.r128.. +

+

Default is 0. +

+
+
framelog
+

Force the frame logging level. +

+

Available values are: +

+
‘quiet’
+

logging disabled +

+
‘info’
+

information logging level +

+
‘verbose’
+

verbose logging level +

+
+ +

By default, the logging level is set to info. If the video or +the metadata options are set, it switches to verbose. +

+
+
peak
+

Set peak mode(s). +

+

Available modes can be cumulated (the option is a flag type). Possible +values are: +

+
‘none’
+

Disable any peak mode (default). +

+
‘sample’
+

Enable sample-peak mode. +

+

Simple peak mode looking for the higher sample value. It logs a message +for sample-peak (identified by SPK). +

+
‘true’
+

Enable true-peak mode. +

+

If enabled, the peak lookup is done on an over-sampled version of the input +stream for better peak accuracy. It logs a message for true-peak. +(identified by TPK) and true-peak per frame (identified by FTPK). +This mode requires a build with libswresample. +

+
+ +
+
dualmono
+

Treat mono input files as "dual mono". If a mono file is intended for playback +on a stereo system, its EBU R128 measurement will be perceptually incorrect. +If set to true, this option will compensate for this effect. +Multi-channel input files are not affected by this option. +

+
+
panlaw
+

Set a specific pan law to be used for the measurement of dual mono files. +This parameter is optional, and has a default value of -3.01dB. +

+
+
target
+

Set a specific target level (in LUFS) used as relative zero in the visualization. +This parameter is optional and has a default value of -23LUFS as specified +by EBU R128. However, material published online may prefer a level of -16LUFS +(e.g. for use with podcasts or video platforms). +

+
+
gauge
+

Set the value displayed by the gauge. Valid values are momentary and s +shortterm. By default the momentary value will be used, but in certain +scenarios it may be more useful to observe the short term value instead (e.g. +live mixing). +

+
+
scale
+

Sets the display scale for the loudness. Valid parameters are absolute +(in LUFS) or relative (LU) relative to the target. This only affects the +video output, not the summary or continuous log output. +

+
+
integrated
+

Read-only exported value for measured integrated loudness, in LUFS. +

+
+
range
+

Read-only exported value for measured loudness range, in LU. +

+
+
lra_low
+

Read-only exported value for measured LRA low, in LUFS. +

+
+
lra_high
+

Read-only exported value for measured LRA high, in LUFS. +

+
+
sample_peak
+

Read-only exported value for measured sample peak, in dBFS. +

+
+
true_peak
+

Read-only exported value for measured true peak, in dBFS. +

+
+ +

20.10.1 Examples

+ +
    +
  • Real-time graph using ffplay, with a EBU scale meter +18: +
    +
    ffplay -f lavfi -i "amovie=input.mp3,ebur128=video=1:meter=18 [out0][out1]"
    +
    + +
  • Run an analysis with ffmpeg: +
    +
    ffmpeg -nostats -i input.mp3 -filter_complex ebur128 -f null -
    +
    +
+ +

20.11 interleave, ainterleave

+ +

Temporally interleave frames from several inputs. +

+

interleave works with video inputs, ainterleave with audio. +

+

These filters read frames from several inputs and send the oldest +queued frame to the output. +

+

Input streams must have well defined, monotonically increasing frame +timestamp values. +

+

In order to submit one frame to output, these filters need to enqueue +at least one frame for each input, so they cannot work in case one +input is not yet terminated and will not receive incoming frames. +

+

For example consider the case when one input is a select filter +which always drops input frames. The interleave filter will keep +reading from that input, but it will never be able to send new frames +to output until the input sends an end-of-stream signal. +

+

Also, depending on inputs synchronization, the filters will drop +frames in case one input receives more frames than the other ones, and +the queue is already filled. +

+

These filters accept the following options: +

+
+
nb_inputs, n
+

Set the number of different inputs, it is 2 by default. +

+
+
duration
+

How to determine the end-of-stream. +

+
+
longest
+

The duration of the longest input. (default) +

+
+
shortest
+

The duration of the shortest input. +

+
+
first
+

The duration of the first input. +

+
+ +
+
+ +

20.11.1 Examples

+ +
    +
  • Interleave frames belonging to different streams using ffmpeg: +
    +
    ffmpeg -i bambi.avi -i pr0n.mkv -filter_complex "[0:v][1:v] interleave" out.avi
    +
    + +
  • Add flickering blur effect: +
    +
    select='if(gt(random(0), 0.2), 1, 2)':n=2 [tmp], boxblur=2:2, [tmp] interleave
    +
    +
+ +

20.12 latency, alatency

+

Measure filtering latency. +

+

Report previous filter filtering latency, delay in number of audio samples for audio filters +or number of video frames for video filters. +

+

On end of input stream, filter will report min and max measured latency for previous running filter +in filtergraph. +

+

20.13 metadata, ametadata

+ +

Manipulate frame metadata. +

+

This filter accepts the following options: +

+
+
mode
+

Set mode of operation of the filter. +

+

Can be one of the following: +

+
+
‘select’
+

If both value and key is set, select frames +which have such metadata. If only key is set, select +every frame that has such key in metadata. +

+
+
‘add’
+

Add new metadata key and value. If key is already available +do nothing. +

+
+
‘modify’
+

Modify value of already present key. +

+
+
‘delete’
+

If value is set, delete only keys that have such value. +Otherwise, delete key. If key is not set, delete all metadata values in +the frame. +

+
+
‘print’
+

Print key and its value if metadata was found. If key is not set print all +metadata values available in frame. +

+
+ +
+
key
+

Set key used with all modes. Must be set for all modes except print and delete. +

+
+
value
+

Set metadata value which will be used. This option is mandatory for +modify and add mode. +

+
+
function
+

Which function to use when comparing metadata value and value. +

+

Can be one of following: +

+
+
‘same_str’
+

Values are interpreted as strings, returns true if metadata value is same as value. +

+
+
‘starts_with’
+

Values are interpreted as strings, returns true if metadata value starts with +the value option string. +

+
+
‘less’
+

Values are interpreted as floats, returns true if metadata value is less than value. +

+
+
‘equal’
+

Values are interpreted as floats, returns true if value is equal with metadata value. +

+
+
‘greater’
+

Values are interpreted as floats, returns true if metadata value is greater than value. +

+
+
‘expr’
+

Values are interpreted as floats, returns true if expression from option expr +evaluates to true. +

+
+
‘ends_with’
+

Values are interpreted as strings, returns true if metadata value ends with +the value option string. +

+
+ +
+
expr
+

Set expression which is used when function is set to expr. +The expression is evaluated through the eval API and can contain the following +constants: +

+
+
VALUE1, FRAMEVAL
+

Float representation of value from metadata key. +

+
+
VALUE2, USERVAL
+

Float representation of value as supplied by user in value option. +

+
+ +
+
file
+

If specified in print mode, output is written to the named file. Instead of +plain filename any writable url can be specified. Filename “-” is a shorthand +for standard output. If file option is not set, output is written to the log +with AV_LOG_INFO loglevel. +

+
+
direct
+

Reduces buffering in print mode when output is written to a URL set using file. +

+
+
+ +

20.13.1 Examples

+ +
    +
  • Print all metadata values for frames with key lavfi.signalstats.YDIF with values +between 0 and 1. +
    +
    signalstats,metadata=print:key=lavfi.signalstats.YDIF:value=0:function=expr:expr='between(VALUE1,0,1)'
    +
    +
  • Print silencedetect output to file metadata.txt. +
    +
    silencedetect,ametadata=mode=print:file=metadata.txt
    +
    +
  • Direct all metadata to a pipe with file descriptor 4. +
    +
    metadata=mode=print:file='pipe\:4'
    +
    +
+ +

20.14 perms, aperms

+ +

Set read/write permissions for the output frames. +

+

These filters are mainly aimed at developers to test direct path in the +following filter in the filtergraph. +

+

The filters accept the following options: +

+
+
mode
+

Select the permissions mode. +

+

It accepts the following values: +

+
‘none’
+

Do nothing. This is the default. +

+
‘ro’
+

Set all the output frames read-only. +

+
‘rw’
+

Set all the output frames directly writable. +

+
‘toggle’
+

Make the frame read-only if writable, and writable if read-only. +

+
‘random’
+

Set each output frame read-only or writable randomly. +

+
+ +
+
seed
+

Set the seed for the random mode, must be an integer included between +0 and UINT32_MAX. If not specified, or if explicitly set to +-1, the filter will try to use a good random seed on a best effort +basis. +

+
+ +

Note: in case of auto-inserted filter between the permission filter and the +following one, the permission might not be received as expected in that +following filter. Inserting a format or aformat filter before the +perms/aperms filter can avoid this problem. +

+

20.15 realtime, arealtime

+ +

Slow down filtering to match real time approximately. +

+

These filters will pause the filtering for a variable amount of time to +match the output rate with the input timestamps. +They are similar to the re option to ffmpeg. +

+

They accept the following options: +

+
+
limit
+

Time limit for the pauses. Any pause longer than that will be considered +a timestamp discontinuity and reset the timer. Default is 2 seconds. +

+
speed
+

Speed factor for processing. The value must be a float larger than zero. +Values larger than 1.0 will result in faster than realtime processing, +smaller will slow processing down. The limit is automatically adapted +accordingly. Default is 1.0. +

+

A processing speed faster than what is possible without these filters cannot +be achieved. +

+
+ +

20.15.1 Commands

+ +

Both filters supports the all above options as commands. +

+

20.16 segment, asegment

+ +

Split single input stream into multiple streams. +

+

This filter does opposite of concat filters. +

+

segment works on video frames, asegment on audio samples. +

+

This filter accepts the following options: +

+
+
timestamps
+

Timestamps of output segments separated by ’|’. The first segment will run +from the beginning of the input stream. The last segment will run until +the end of the input stream +

+
+
frames, samples
+

Exact frame/sample count to split the segments. +

+
+ +

In all cases, prefixing an each segment with ’+’ will make it relative to the +previous segment. +

+

20.16.1 Examples

+ +
    +
  • Split input audio stream into three output audio streams, starting at start of input audio stream +and storing that in 1st output audio stream, then following at 60th second and storing than in 2nd +output audio stream, and last after 150th second of input audio stream store in 3rd output audio stream: +
    +
    asegment=timestamps="60|150"
    +
    + +
+ +

20.17 select, aselect

+ +

Select frames to pass in output. +

+

This filter accepts the following options: +

+
+
expr, e
+

Set expression, which is evaluated for each input frame. +

+

If the expression is evaluated to zero, the frame is discarded. +

+

If the evaluation result is negative or NaN, the frame is sent to the +first output; otherwise it is sent to the output with index +ceil(val)-1, assuming that the input index starts from 0. +

+

For example a value of 1.2 corresponds to the output with index +ceil(1.2)-1 = 2-1 = 1, that is the second output. +

+
+
outputs, n
+

Set the number of outputs. The output to which to send the selected +frame is based on the result of the evaluation. Default value is 1. +

+
+ +

The expression can contain the following constants: +

+
+
n
+

The (sequential) number of the filtered frame, starting from 0. +

+
+
selected_n
+

The (sequential) number of the selected frame, starting from 0. +

+
+
prev_selected_n
+

The sequential number of the last selected frame. It’s NAN if undefined. +

+
+
TB
+

The timebase of the input timestamps. +

+
+
pts
+

The PTS (Presentation TimeStamp) of the filtered frame, +expressed in TB units. It’s NAN if undefined. +

+
+
t
+

The PTS of the filtered frame, +expressed in seconds. It’s NAN if undefined. +

+
+
prev_pts
+

The PTS of the previously filtered frame. It’s NAN if undefined. +

+
+
prev_selected_pts
+

The PTS of the last previously filtered frame. It’s NAN if undefined. +

+
+
prev_selected_t
+

The PTS of the last previously selected frame, expressed in seconds. It’s NAN if undefined. +

+
+
start_pts
+

The first PTS in the stream which is not NAN. It remains NAN if not found. +

+
+
start_t
+

The first PTS, in seconds, in the stream which is not NAN. It remains NAN if not found. +

+
+
pict_type (video only)
+

The type of the filtered frame. It can assume one of the following +values: +

+
I
+
P
+
B
+
S
+
SI
+
SP
+
BI
+
+ +
+
interlace_type (video only)
+

The frame interlace type. It can assume one of the following values: +

+
PROGRESSIVE
+

The frame is progressive (not interlaced). +

+
TOPFIRST
+

The frame is top-field-first. +

+
BOTTOMFIRST
+

The frame is bottom-field-first. +

+
+ +
+
consumed_sample_n (audio only)
+

the number of selected samples before the current frame +

+
+
samples_n (audio only)
+

the number of samples in the current frame +

+
+
sample_rate (audio only)
+

the input sample rate +

+
+
key
+

This is 1 if the filtered frame is a key-frame, 0 otherwise. +

+
+
pos
+

the position in the file of the filtered frame, -1 if the information +is not available (e.g. for synthetic video); deprecated, do not use +

+
+
scene (video only)
+

value between 0 and 1 to indicate a new scene; a low value reflects a low +probability for the current frame to introduce a new scene, while a higher +value means the current frame is more likely to be one (see the example below) +

+
+
concatdec_select
+

The concat demuxer can select only part of a concat input file by setting an +inpoint and an outpoint, but the output packets may not be entirely contained +in the selected interval. By using this variable, it is possible to skip frames +generated by the concat demuxer which are not exactly contained in the selected +interval. +

+

This works by comparing the frame pts against the lavf.concat.start_time +and the lavf.concat.duration packet metadata values which are also +present in the decoded frames. +

+

The concatdec_select variable is -1 if the frame pts is at least +start_time and either the duration metadata is missing or the frame pts is less +than start_time + duration, 0 otherwise, and NaN if the start_time metadata is +missing. +

+

That basically means that an input frame is selected if its pts is within the +interval set by the concat demuxer. +

+
+
iw (video only)
+

Represents the width of the input video frame. +

+
+
ih (video only)
+

Represents the height of the input video frame. +

+
+
view (video only)
+

View ID for multi-view video. +

+
+
+ +

The default value of the select expression is "1". +

+

20.17.1 Examples

+ +
    +
  • Select all frames in input: +
    +
    select
    +
    + +

    The example above is the same as: +

    +
    select=1
    +
    + +
  • Skip all frames: +
    +
    select=0
    +
    + +
  • Select only I-frames: +
    +
    select='eq(pict_type\,I)'
    +
    + +
  • Select one frame every 100: +
    +
    select='not(mod(n\,100))'
    +
    + +
  • Select only frames contained in the 10-20 time interval: +
    +
    select=between(t\,10\,20)
    +
    + +
  • Select only I-frames contained in the 10-20 time interval: +
    +
    select=between(t\,10\,20)*eq(pict_type\,I)
    +
    + +
  • Select frames with a minimum distance of 10 seconds: +
    +
    select='isnan(prev_selected_t)+gte(t-prev_selected_t\,10)'
    +
    + +
  • Use aselect to select only audio frames with samples number > 100: +
    +
    aselect='gt(samples_n\,100)'
    +
    + +
  • Create a mosaic of the first scenes: +
    +
    ffmpeg -i video.avi -vf select='gt(scene\,0.4)',scale=160:120,tile -frames:v 1 preview.png
    +
    + +

    Comparing scene against a value between 0.3 and 0.5 is generally a sane +choice. +

    +
  • Send even and odd frames to separate outputs, and compose them: +
    +
    select=n=2:e='mod(n, 2)+1' [odd][even]; [odd] pad=h=2*ih [tmp]; [tmp][even] overlay=y=h
    +
    + +
  • Select useful frames from an ffconcat file which is using inpoints and +outpoints but where the source files are not intra frame only. +
    +
    ffmpeg -copyts -vsync 0 -segment_time_metadata 1 -i input.ffconcat -vf select=concatdec_select -af aselect=concatdec_select output.avi
    +
    +
+ +

20.18 sendcmd, asendcmd

+ +

Send commands to filters in the filtergraph. +

+

These filters read commands to be sent to other filters in the +filtergraph. +

+

sendcmd must be inserted between two video filters, +asendcmd must be inserted between two audio filters, but apart +from that they act the same way. +

+

The specification of commands can be provided in the filter arguments +with the commands option, or in a file specified by the +filename option. +

+

These filters accept the following options: +

+
commands, c
+

Set the commands to be read and sent to the other filters. +

+
filename, f
+

Set the filename of the commands to be read and sent to the other +filters. +

+
+ +

20.18.1 Commands syntax

+ +

A commands description consists of a sequence of interval +specifications, comprising a list of commands to be executed when a +particular event related to that interval occurs. The occurring event +is typically the current frame time entering or leaving a given time +interval. +

+

An interval is specified by the following syntax: +

+
START[-END] COMMANDS;
+
+ +

The time interval is specified by the START and END times. +END is optional and defaults to the maximum time. +

+

The current frame time is considered within the specified interval if +it is included in the interval [START, END), that is when +the time is greater or equal to START and is lesser than +END. +

+

COMMANDS consists of a sequence of one or more command +specifications, separated by ",", relating to that interval. The +syntax of a command specification is given by: +

+
[FLAGS] TARGET COMMAND ARG
+
+ +

FLAGS is optional and specifies the type of events relating to +the time interval which enable sending the specified command, and must +be a non-null sequence of identifier flags separated by "+" or "|" and +enclosed between "[" and "]". +

+

The following flags are recognized: +

+
enter
+

The command is sent when the current frame timestamp enters the +specified interval. In other words, the command is sent when the +previous frame timestamp was not in the given interval, and the +current is. +

+
+
leave
+

The command is sent when the current frame timestamp leaves the +specified interval. In other words, the command is sent when the +previous frame timestamp was in the given interval, and the +current is not. +

+
+
expr
+

The command ARG is interpreted as expression and result of +expression is passed as ARG. +

+

The expression is evaluated through the eval API and can contain the following +constants: +

+
+
POS
+

Original position in the file of the frame, or undefined if undefined +for the current frame. Deprecated, do not use. +

+
+
PTS
+

The presentation timestamp in input. +

+
+
N
+

The count of the input frame for video or audio, starting from 0. +

+
+
T
+

The time in seconds of the current frame. +

+
+
TS
+

The start time in seconds of the current command interval. +

+
+
TE
+

The end time in seconds of the current command interval. +

+
+
TI
+

The interpolated time of the current command interval, TI = (T - TS) / (TE - TS). +

+
+
W
+

The video frame width. +

+
+
H
+

The video frame height. +

+
+ +
+
+ +

If FLAGS is not specified, a default value of [enter] is +assumed. +

+

TARGET specifies the target of the command, usually the name of +the filter class or a specific filter instance name. +

+

COMMAND specifies the name of the command for the target filter. +

+

ARG is optional and specifies the optional list of argument for +the given COMMAND. +

+

Between one interval specification and another, whitespaces, or +sequences of characters starting with # until the end of line, +are ignored and can be used to annotate comments. +

+

A simplified BNF description of the commands specification syntax +follows: +

+
COMMAND_FLAG  ::= "enter" | "leave"
+COMMAND_FLAGS ::= COMMAND_FLAG [(+|"|")COMMAND_FLAG]
+COMMAND       ::= ["[" COMMAND_FLAGS "]"] TARGET COMMAND [ARG]
+COMMANDS      ::= COMMAND [,COMMANDS]
+INTERVAL      ::= START[-END] COMMANDS
+INTERVALS     ::= INTERVAL[;INTERVALS]
+
+ +

20.18.2 Examples

+ +
    +
  • Specify audio tempo change at second 4: +
    +
    asendcmd=c='4.0 atempo tempo 1.5',atempo
    +
    + +
  • Target a specific filter instance: +
    +
    asendcmd=c='4.0 atempo@my tempo 1.5',atempo@my
    +
    + +
  • Specify a list of drawtext and hue commands in a file. +
    +
    # show text in the interval 5-10
    +5.0-10.0 [enter] drawtext reinit 'fontfile=FreeSerif.ttf:text=hello world',
    +         [leave] drawtext reinit 'fontfile=FreeSerif.ttf:text=';
    +
    +# desaturate the image in the interval 15-20
    +15.0-20.0 [enter] hue s 0,
    +          [enter] drawtext reinit 'fontfile=FreeSerif.ttf:text=nocolor',
    +          [leave] hue s 1,
    +          [leave] drawtext reinit 'fontfile=FreeSerif.ttf:text=color';
    +
    +# apply an exponential saturation fade-out effect, starting from time 25
    +25 [enter] hue s exp(25-t)
    +
    + +

    A filtergraph allowing to read and process the above command list +stored in a file test.cmd, can be specified with: +

    +
    sendcmd=f=test.cmd,drawtext=fontfile=FreeSerif.ttf:text='',hue
    +
    +
+ +

20.19 setpts, asetpts

+ +

Change the PTS (presentation timestamp) of the input frames. +

+

setpts works on video frames, asetpts on audio frames. +

+

This filter accepts the following options: +

+
+
expr
+

The expression which is evaluated for each frame to construct its timestamp. +

+
+
strip_fps (video only)
+

Boolean option which determines if the original framerate and frame duration +metadata is unset. If set to true, be advised that a sane frame rate should be +explicitly specified if output is sent to a constant frame rate muxer. +Default is false. +

+
+
+ +

The expression is evaluated through the eval API and can contain the following +constants: +

+
+
FRAME_RATE, FR
+

frame rate, only defined for constant frame-rate video +

+
+
PTS
+

The presentation timestamp in input +

+
+
N
+

The count of the input frame for video or the number of consumed samples, +not including the current frame for audio, starting from 0. +

+
+
NB_CONSUMED_SAMPLES
+

The number of consumed samples, not including the current frame (only +audio) +

+
+
NB_SAMPLES, S
+

The number of samples in the current frame (only audio) +

+
+
SAMPLE_RATE, SR
+

The audio sample rate. +

+
+
STARTPTS
+

The PTS of the first frame. +

+
+
STARTT
+

the time in seconds of the first frame +

+
+
INTERLACED
+

State whether the current frame is interlaced. +

+
+
T
+

the time in seconds of the current frame +

+
+
POS
+

original position in the file of the frame, or undefined if undefined +for the current frame; deprecated, do not use +

+
+
PREV_INPTS
+

The previous input PTS. +

+
+
PREV_INT
+

previous input time in seconds +

+
+
PREV_OUTPTS
+

The previous output PTS. +

+
+
PREV_OUTT
+

previous output time in seconds +

+
+
RTCTIME
+

The wallclock (RTC) time in microseconds. This is deprecated, use time(0) +instead. +

+
+
RTCSTART
+

The wallclock (RTC) time at the start of the movie in microseconds. +

+
+
TB
+

The timebase of the input timestamps. +

+
+
T_CHANGE
+

Time of the first frame after command was applied or time of the first frame if no commands. +

+
+
+ +

20.19.1 Examples

+ +
    +
  • Start counting PTS from zero +
    +
    setpts=PTS-STARTPTS
    +
    + +
  • Apply fast motion effect: +
    +
    setpts=0.5*PTS
    +
    + +
  • Apply slow motion effect: +
    +
    setpts=2.0*PTS
    +
    + +
  • Set fixed rate of 25 frames per second: +
    +
    setpts=N/(25*TB)
    +
    + +
  • Apply a random jitter effect of +/-100 TB units: +
    +
    setpts=PTS+randomi(0, -100\,100)
    +
    + +
  • Set fixed rate 25 fps with some jitter: +
    +
    setpts='1/(25*TB) * (N + 0.05 * sin(N*2*PI/25))'
    +
    + +
  • Apply an offset of 10 seconds to the input PTS: +
    +
    setpts=PTS+10/TB
    +
    + +
  • Generate timestamps from a "live source" and rebase onto the current timebase: +
    +
    setpts='(RTCTIME - RTCSTART) / (TB * 1000000)'
    +
    + +
  • Generate timestamps by counting samples: +
    +
    asetpts=N/SR/TB
    +
    + +
+ +

20.19.2 Commands

+ +

Both filters support all above options as commands. +

+

20.20 setrange

+ +

Force color range for the output video frame. +

+

The setrange filter marks the color range property for the +output frames. It does not change the input frame, but only sets the +corresponding property, which affects how the frame is treated by +following filters. +

+

The filter accepts the following options: +

+
+
range
+

Available values are: +

+
+
‘auto’
+

Keep the same color range property. +

+
+
‘unspecified, unknown’
+

Set the color range as unspecified. +

+
+
‘limited, tv, mpeg’
+

Set the color range as limited. +

+
+
‘full, pc, jpeg’
+

Set the color range as full. +

+
+
+
+ +

20.21 settb, asettb

+ +

Set the timebase to use for the output frames timestamps. +It is mainly useful for testing timebase configuration. +

+

It accepts the following parameters: +

+
+
expr, tb
+

The expression which is evaluated into the output timebase. +

+
+
+ +

The value for tb is an arithmetic expression representing a +rational. The expression can contain the constants "AVTB" (the default +timebase), "intb" (the input timebase) and "sr" (the sample rate, +audio only). Default value is "intb". +

+

20.21.1 Examples

+ +
    +
  • Set the timebase to 1/25: +
    +
    settb=expr=1/25
    +
    + +
  • Set the timebase to 1/10: +
    +
    settb=expr=0.1
    +
    + +
  • Set the timebase to 1001/1000: +
    +
    settb=1+0.001
    +
    + +
  • Set the timebase to 2*intb: +
    +
    settb=2*intb
    +
    + +
  • Set the default timebase value: +
    +
    settb=AVTB
    +
    +
+ +

20.22 showcqt

+

Convert input audio to a video output representing frequency spectrum +logarithmically using Brown-Puckette constant Q transform algorithm with +direct frequency domain coefficient calculation (but the transform itself +is not really constant Q, instead the Q factor is actually variable/clamped), +with musical tone scale, from E0 to D#10. +

+

The filter accepts the following options: +

+
+
size, s
+

Specify the video size for the output. It must be even. For the syntax of this option, +check the (ffmpeg-utils)"Video size" section in the ffmpeg-utils manual. +Default value is 1920x1080. +

+
+
fps, rate, r
+

Set the output frame rate. Default value is 25. +

+
+
bar_h
+

Set the bargraph height. It must be even. Default value is -1 which +computes the bargraph height automatically. +

+
+
axis_h
+

Set the axis height. It must be even. Default value is -1 which computes +the axis height automatically. +

+
+
sono_h
+

Set the sonogram height. It must be even. Default value is -1 which +computes the sonogram height automatically. +

+
+
fullhd
+

Set the fullhd resolution. This option is deprecated, use size, s +instead. Default value is 1. +

+
+
sono_v, volume
+

Specify the sonogram volume expression. It can contain variables: +

+
bar_v
+

the bar_v evaluated expression +

+
frequency, freq, f
+

the frequency where it is evaluated +

+
timeclamp, tc
+

the value of timeclamp option +

+
+

and functions: +

+
a_weighting(f)
+

A-weighting of equal loudness +

+
b_weighting(f)
+

B-weighting of equal loudness +

+
c_weighting(f)
+

C-weighting of equal loudness. +

+
+

Default value is 16. +

+
+
bar_v, volume2
+

Specify the bargraph volume expression. It can contain variables: +

+
sono_v
+

the sono_v evaluated expression +

+
frequency, freq, f
+

the frequency where it is evaluated +

+
timeclamp, tc
+

the value of timeclamp option +

+
+

and functions: +

+
a_weighting(f)
+

A-weighting of equal loudness +

+
b_weighting(f)
+

B-weighting of equal loudness +

+
c_weighting(f)
+

C-weighting of equal loudness. +

+
+

Default value is sono_v. +

+
+
sono_g, gamma
+

Specify the sonogram gamma. Lower gamma makes the spectrum more contrast, +higher gamma makes the spectrum having more range. Default value is 3. +Acceptable range is [1, 7]. +

+
+
bar_g, gamma2
+

Specify the bargraph gamma. Default value is 1. Acceptable range is +[1, 7]. +

+
+
bar_t
+

Specify the bargraph transparency level. Lower value makes the bargraph sharper. +Default value is 1. Acceptable range is [0, 1]. +

+
+
timeclamp, tc
+

Specify the transform timeclamp. At low frequency, there is trade-off between +accuracy in time domain and frequency domain. If timeclamp is lower, +event in time domain is represented more accurately (such as fast bass drum), +otherwise event in frequency domain is represented more accurately +(such as bass guitar). Acceptable range is [0.002, 1]. Default value is 0.17. +

+
+
attack
+

Set attack time in seconds. The default is 0 (disabled). Otherwise, it +limits future samples by applying asymmetric windowing in time domain, useful +when low latency is required. Accepted range is [0, 1]. +

+
+
basefreq
+

Specify the transform base frequency. Default value is 20.01523126408007475, +which is frequency 50 cents below E0. Acceptable range is [10, 100000]. +

+
+
endfreq
+

Specify the transform end frequency. Default value is 20495.59681441799654, +which is frequency 50 cents above D#10. Acceptable range is [10, 100000]. +

+
+
coeffclamp
+

This option is deprecated and ignored. +

+
+
tlength
+

Specify the transform length in time domain. Use this option to control accuracy +trade-off between time domain and frequency domain at every frequency sample. +It can contain variables: +

+
frequency, freq, f
+

the frequency where it is evaluated +

+
timeclamp, tc
+

the value of timeclamp option. +

+
+

Default value is 384*tc/(384+tc*f). +

+
+
count
+

Specify the transform count for every video frame. Default value is 6. +Acceptable range is [1, 30]. +

+
+
fcount
+

Specify the transform count for every single pixel. Default value is 0, +which makes it computed automatically. Acceptable range is [0, 10]. +

+
+
fontfile
+

Specify font file for use with freetype to draw the axis. If not specified, +use embedded font. Note that drawing with font file or embedded font is not +implemented with custom basefreq and endfreq, use axisfile +option instead. +

+
+
font
+

Specify fontconfig pattern. This has lower priority than fontfile. The +: in the pattern may be replaced by | to avoid unnecessary +escaping. +

+
+
fontcolor
+

Specify font color expression. This is arithmetic expression that should return +integer value 0xRRGGBB. It can contain variables: +

+
frequency, freq, f
+

the frequency where it is evaluated +

+
timeclamp, tc
+

the value of timeclamp option +

+
+

and functions: +

+
midi(f)
+

midi number of frequency f, some midi numbers: E0(16), C1(24), C2(36), A4(69) +

+
r(x), g(x), b(x)
+

red, green, and blue value of intensity x. +

+
+

Default value is st(0, (midi(f)-59.5)/12); +st(1, if(between(ld(0),0,1), 0.5-0.5*cos(2*PI*ld(0)), 0)); +r(1-ld(1)) + b(ld(1)). +

+
+
axisfile
+

Specify image file to draw the axis. This option override fontfile and +fontcolor option. +

+
+
axis, text
+

Enable/disable drawing text to the axis. If it is set to 0, drawing to +the axis is disabled, ignoring fontfile and axisfile option. +Default value is 1. +

+
+
csp
+

Set colorspace. The accepted values are: +

+
‘unspecified’
+

Unspecified (default) +

+
+
‘bt709’
+

BT.709 +

+
+
‘fcc’
+

FCC +

+
+
‘bt470bg’
+

BT.470BG or BT.601-6 625 +

+
+
‘smpte170m’
+

SMPTE-170M or BT.601-6 525 +

+
+
‘smpte240m’
+

SMPTE-240M +

+
+
‘bt2020ncl’
+

BT.2020 with non-constant luminance +

+
+
+ +
+
cscheme
+

Set spectrogram color scheme. This is list of floating point values with format +left_r|left_g|left_b|right_r|right_g|right_b. +The default is 1|0.5|0|0|0.5|1. +

+
+
+ +

20.22.1 Examples

+ +
    +
  • Playing audio while showing the spectrum: +
    +
    ffplay -f lavfi 'amovie=a.mp3, asplit [a][out1]; [a] showcqt [out0]'
    +
    + +
  • Same as above, but with frame rate 30 fps: +
    +
    ffplay -f lavfi 'amovie=a.mp3, asplit [a][out1]; [a] showcqt=fps=30:count=5 [out0]'
    +
    + +
  • Playing at 1280x720: +
    +
    ffplay -f lavfi 'amovie=a.mp3, asplit [a][out1]; [a] showcqt=s=1280x720:count=4 [out0]'
    +
    + +
  • Disable sonogram display: +
    +
    sono_h=0
    +
    + +
  • A1 and its harmonics: A1, A2, (near)E3, A3: +
    +
    ffplay -f lavfi 'aevalsrc=0.1*sin(2*PI*55*t)+0.1*sin(4*PI*55*t)+0.1*sin(6*PI*55*t)+0.1*sin(8*PI*55*t),
    +                 asplit[a][out1]; [a] showcqt [out0]'
    +
    + +
  • Same as above, but with more accuracy in frequency domain: +
    +
    ffplay -f lavfi 'aevalsrc=0.1*sin(2*PI*55*t)+0.1*sin(4*PI*55*t)+0.1*sin(6*PI*55*t)+0.1*sin(8*PI*55*t),
    +                 asplit[a][out1]; [a] showcqt=timeclamp=0.5 [out0]'
    +
    + +
  • Custom volume: +
    +
    bar_v=10:sono_v=bar_v*a_weighting(f)
    +
    + +
  • Custom gamma, now spectrum is linear to the amplitude. +
    +
    bar_g=2:sono_g=2
    +
    + +
  • Custom tlength equation: +
    +
    tc=0.33:tlength='st(0,0.17); 384*tc / (384 / ld(0) + tc*f /(1-ld(0))) + 384*tc / (tc*f / ld(0) + 384 /(1-ld(0)))'
    +
    + +
  • Custom fontcolor and fontfile, C-note is colored green, others are colored blue: +
    +
    fontcolor='if(mod(floor(midi(f)+0.5),12), 0x0000FF, g(1))':fontfile=myfont.ttf
    +
    + +
  • Custom font using fontconfig: +
    +
    font='Courier New,Monospace,mono|bold'
    +
    + +
  • Custom frequency range with custom axis using image file: +
    +
    axisfile=myaxis.png:basefreq=40:endfreq=10000
    +
    +
+ +

20.23 showcwt

+ +

Convert input audio to video output representing frequency spectrum +using Continuous Wavelet Transform and Morlet wavelet. +

+

The filter accepts the following options: +

+
+
size, s
+

Specify the video size for the output. For the syntax of this option, +check the (ffmpeg-utils)"Video size" section in the ffmpeg-utils manual. +Default value is 640x512. +

+
+
rate, r
+

Set the output frame rate. Default value is 25. +

+
+
scale
+

Set the frequency scale used. Allowed values are: +

+
+
linear
+
log
+
bark
+
mel
+
erbs
+
sqrt
+
cbrt
+
qdrt
+
fm
+
+

Default value is linear. +

+
+
iscale
+

Set the intensity scale used. Allowed values are: +

+
+
linear
+
log
+
sqrt
+
cbrt
+
qdrt
+
+

Default value is log. +

+
+
min
+

Set the minimum frequency that will be used in output. +Default is 20 Hz. +

+
+
max
+

Set the maximum frequency that will be used in output. +Default is 20000 Hz. The real frequency upper limit +depends on input audio’s sample rate and such will be enforced +on this value when it is set to value greater than Nyquist frequency. +

+
+
imin
+

Set the minimum intensity that will be used in output. +

+
+
imax
+

Set the maximum intensity that will be used in output. +

+
+
logb
+

Set the logarithmic basis for brightness strength when +mapping calculated magnitude values to pixel values. +Allowed range is from 0 to 1. +Default value is 0.0001. +

+
+
deviation
+

Set the frequency deviation. +Lower values than 1 are more frequency oriented, +while higher values than 1 are more time oriented. +Allowed range is from 0 to 10. +Default value is 1. +

+
+
pps
+

Set the number of pixel output per each second in one row. +Allowed range is from 1 to 1024. +Default value is 64. +

+
+
mode
+

Set the output visual mode. Allowed values are: +

+
+
magnitude
+

Show magnitude. +

+
phase
+

Show only phase. +

+
magphase
+

Show combination of magnitude and phase. +Magnitude is mapped to brightness and phase to color. +

+
channel
+

Show unique color per channel magnitude. +

+
stereo
+

Show unique color per stereo difference. +

+
+ +

Default value is magnitude. +

+
+
slide
+

Set the output slide method. Allowed values are: +

+
+
replace
+
scroll
+
frame
+
+ +
+
direction
+

Set the direction method for output slide method. Allowed values are: +

+
+
lr
+

Direction from left to right. +

+
rl
+

Direction from right to left. +

+
ud
+

Direction from up to down. +

+
du
+

Direction from down to up. +

+
+ +
+
bar
+

Set the ratio of bargraph display to display size. Default is 0. +

+
+
rotation
+

Set color rotation, must be in [-1.0, 1.0] range. +Default value is 0. +

+
+ +

20.24 showfreqs

+ +

Convert input audio to video output representing the audio power spectrum. +Audio amplitude is on Y-axis while frequency is on X-axis. +

+

The filter accepts the following options: +

+
+
size, s
+

Specify size of video. For the syntax of this option, check the +(ffmpeg-utils)"Video size" section in the ffmpeg-utils manual. +Default is 1024x512. +

+
+
rate, r
+

Set video rate. Default is 25. +

+
+
mode
+

Set display mode. +This set how each frequency bin will be represented. +

+

It accepts the following values: +

+
‘line’
+
‘bar’
+
‘dot’
+
+

Default is bar. +

+
+
ascale
+

Set amplitude scale. +

+

It accepts the following values: +

+
‘lin’
+

Linear scale. +

+
+
‘sqrt’
+

Square root scale. +

+
+
‘cbrt’
+

Cubic root scale. +

+
+
‘log’
+

Logarithmic scale. +

+
+

Default is log. +

+
+
fscale
+

Set frequency scale. +

+

It accepts the following values: +

+
‘lin’
+

Linear scale. +

+
+
‘log’
+

Logarithmic scale. +

+
+
‘rlog’
+

Reverse logarithmic scale. +

+
+

Default is lin. +

+
+
win_size
+

Set window size. Allowed range is from 16 to 65536. +

+

Default is 2048 +

+
+
win_func
+

Set windowing function. +

+

It accepts the following values: +

+
‘rect’
+
‘bartlett’
+
‘hanning’
+
‘hamming’
+
‘blackman’
+
‘welch’
+
‘flattop’
+
‘bharris’
+
‘bnuttall’
+
‘bhann’
+
‘sine’
+
‘nuttall’
+
‘lanczos’
+
‘gauss’
+
‘tukey’
+
‘dolph’
+
‘cauchy’
+
‘parzen’
+
‘poisson’
+
‘bohman’
+
‘kaiser’
+
+

Default is hanning. +

+
+
overlap
+

Set window overlap. In range [0, 1]. Default is 1, +which means optimal overlap for selected window function will be picked. +

+
+
averaging
+

Set time averaging. Setting this to 0 will display current maximal peaks. +Default is 1, which means time averaging is disabled. +

+
+
colors
+

Specify list of colors separated by space or by ’|’ which will be used to +draw channel frequencies. Unrecognized or missing colors will be replaced +by white color. +

+
+
cmode
+

Set channel display mode. +

+

It accepts the following values: +

+
‘combined’
+
‘separate’
+
+

Default is combined. +

+
+
minamp
+

Set minimum amplitude used in log amplitude scaler. +

+
+
data
+

Set data display mode. +

+

It accepts the following values: +

+
‘magnitude’
+
‘phase’
+
‘delay’
+
+

Default is magnitude. +

+
+
channels
+

Set channels to use when processing audio. By default all are processed. +

+
+ +

20.25 showspatial

+ +

Convert stereo input audio to a video output, representing the spatial relationship +between two channels. +

+

The filter accepts the following options: +

+
+
size, s
+

Specify the video size for the output. For the syntax of this option, check the +(ffmpeg-utils)"Video size" section in the ffmpeg-utils manual. +Default value is 512x512. +

+
+
win_size
+

Set window size. Allowed range is from 1024 to 65536. Default size is 4096. +

+
+
win_func
+

Set window function. +

+

It accepts the following values: +

+
‘rect’
+
‘bartlett’
+
‘hann’
+
‘hanning’
+
‘hamming’
+
‘blackman’
+
‘welch’
+
‘flattop’
+
‘bharris’
+
‘bnuttall’
+
‘bhann’
+
‘sine’
+
‘nuttall’
+
‘lanczos’
+
‘gauss’
+
‘tukey’
+
‘dolph’
+
‘cauchy’
+
‘parzen’
+
‘poisson’
+
‘bohman’
+
‘kaiser’
+
+ +

Default value is hann. +

+
+
rate, r
+

Set output framerate. +

+
+ +

20.26 showspectrum

+ +

Convert input audio to a video output, representing the audio frequency +spectrum. +

+

The filter accepts the following options: +

+
+
size, s
+

Specify the video size for the output. For the syntax of this option, check the +(ffmpeg-utils)"Video size" section in the ffmpeg-utils manual. +Default value is 640x512. +

+
+
slide
+

Specify how the spectrum should slide along the window. +

+

It accepts the following values: +

+
‘replace’
+

the samples start again on the left when they reach the right +

+
‘scroll’
+

the samples scroll from right to left +

+
‘fullframe’
+

frames are only produced when the samples reach the right +

+
‘rscroll’
+

the samples scroll from left to right +

+
‘lreplace’
+

the samples start again on the right when they reach the left +

+
+ +

Default value is replace. +

+
+
mode
+

Specify display mode. +

+

It accepts the following values: +

+
‘combined’
+

all channels are displayed in the same row +

+
‘separate’
+

all channels are displayed in separate rows +

+
+ +

Default value is ‘combined’. +

+
+
color
+

Specify display color mode. +

+

It accepts the following values: +

+
‘channel’
+

each channel is displayed in a separate color +

+
‘intensity’
+

each channel is displayed using the same color scheme +

+
‘rainbow’
+

each channel is displayed using the rainbow color scheme +

+
‘moreland’
+

each channel is displayed using the moreland color scheme +

+
‘nebulae’
+

each channel is displayed using the nebulae color scheme +

+
‘fire’
+

each channel is displayed using the fire color scheme +

+
‘fiery’
+

each channel is displayed using the fiery color scheme +

+
‘fruit’
+

each channel is displayed using the fruit color scheme +

+
‘cool’
+

each channel is displayed using the cool color scheme +

+
‘magma’
+

each channel is displayed using the magma color scheme +

+
‘green’
+

each channel is displayed using the green color scheme +

+
‘viridis’
+

each channel is displayed using the viridis color scheme +

+
‘plasma’
+

each channel is displayed using the plasma color scheme +

+
‘cividis’
+

each channel is displayed using the cividis color scheme +

+
‘terrain’
+

each channel is displayed using the terrain color scheme +

+
+ +

Default value is ‘channel’. +

+
+
scale
+

Specify scale used for calculating intensity color values. +

+

It accepts the following values: +

+
‘lin’
+

linear +

+
‘sqrt’
+

square root, default +

+
‘cbrt’
+

cubic root +

+
‘log’
+

logarithmic +

+
‘4thrt’
+

4th root +

+
‘5thrt’
+

5th root +

+
+ +

Default value is ‘sqrt’. +

+
+
fscale
+

Specify frequency scale. +

+

It accepts the following values: +

+
‘lin’
+

linear +

+
‘log’
+

logarithmic +

+
+ +

Default value is ‘lin’. +

+
+
saturation
+

Set saturation modifier for displayed colors. Negative values provide +alternative color scheme. 0 is no saturation at all. +Saturation must be in [-10.0, 10.0] range. +Default value is 1. +

+
+
win_func
+

Set window function. +

+

It accepts the following values: +

+
‘rect’
+
‘bartlett’
+
‘hann’
+
‘hanning’
+
‘hamming’
+
‘blackman’
+
‘welch’
+
‘flattop’
+
‘bharris’
+
‘bnuttall’
+
‘bhann’
+
‘sine’
+
‘nuttall’
+
‘lanczos’
+
‘gauss’
+
‘tukey’
+
‘dolph’
+
‘cauchy’
+
‘parzen’
+
‘poisson’
+
‘bohman’
+
‘kaiser’
+
+ +

Default value is hann. +

+
+
orientation
+

Set orientation of time vs frequency axis. Can be vertical or +horizontal. Default is vertical. +

+
+
overlap
+

Set ratio of overlap window. Default value is 0. +When value is 1 overlap is set to recommended size for specific +window function currently used. +

+
+
gain
+

Set scale gain for calculating intensity color values. +Default value is 1. +

+
+
data
+

Set which data to display. Can be magnitude, default or phase, +or unwrapped phase: uphase. +

+
+
rotation
+

Set color rotation, must be in [-1.0, 1.0] range. +Default value is 0. +

+
+
start
+

Set start frequency from which to display spectrogram. Default is 0. +

+
+
stop
+

Set stop frequency to which to display spectrogram. Default is 0. +

+
+
fps
+

Set upper frame rate limit. Default is auto, unlimited. +

+
+
legend
+

Draw time and frequency axes and legends. Default is disabled. +

+
+
drange
+

Set dynamic range used to calculate intensity color values. Default is 120 dBFS. +Allowed range is from 10 to 200. +

+
+
limit
+

Set upper limit of input audio samples volume in dBFS. Default is 0 dBFS. +Allowed range is from -100 to 100. +

+
+
opacity
+

Set opacity strength when using pixel format output with alpha component. +

+
+ +

The usage is very similar to the showwaves filter; see the examples in that +section. +

+

20.26.1 Examples

+ +
    +
  • Large window with logarithmic color scaling: +
    +
    showspectrum=s=1280x480:scale=log
    +
    + +
  • Complete example for a colored and sliding spectrum per channel using ffplay: +
    +
    ffplay -f lavfi 'amovie=input.mp3, asplit [a][out1];
    +             [a] showspectrum=mode=separate:color=intensity:slide=1:scale=cbrt [out0]'
    +
    +
+ +

20.27 showspectrumpic

+ +

Convert input audio to a single video frame, representing the audio frequency +spectrum. +

+

The filter accepts the following options: +

+
+
size, s
+

Specify the video size for the output. For the syntax of this option, check the +(ffmpeg-utils)"Video size" section in the ffmpeg-utils manual. +Default value is 4096x2048. +

+
+
mode
+

Specify display mode. +

+

It accepts the following values: +

+
‘combined’
+

all channels are displayed in the same row +

+
‘separate’
+

all channels are displayed in separate rows +

+
+

Default value is ‘combined’. +

+
+
color
+

Specify display color mode. +

+

It accepts the following values: +

+
‘channel’
+

each channel is displayed in a separate color +

+
‘intensity’
+

each channel is displayed using the same color scheme +

+
‘rainbow’
+

each channel is displayed using the rainbow color scheme +

+
‘moreland’
+

each channel is displayed using the moreland color scheme +

+
‘nebulae’
+

each channel is displayed using the nebulae color scheme +

+
‘fire’
+

each channel is displayed using the fire color scheme +

+
‘fiery’
+

each channel is displayed using the fiery color scheme +

+
‘fruit’
+

each channel is displayed using the fruit color scheme +

+
‘cool’
+

each channel is displayed using the cool color scheme +

+
‘magma’
+

each channel is displayed using the magma color scheme +

+
‘green’
+

each channel is displayed using the green color scheme +

+
‘viridis’
+

each channel is displayed using the viridis color scheme +

+
‘plasma’
+

each channel is displayed using the plasma color scheme +

+
‘cividis’
+

each channel is displayed using the cividis color scheme +

+
‘terrain’
+

each channel is displayed using the terrain color scheme +

+
+

Default value is ‘intensity’. +

+
+
scale
+

Specify scale used for calculating intensity color values. +

+

It accepts the following values: +

+
‘lin’
+

linear +

+
‘sqrt’
+

square root, default +

+
‘cbrt’
+

cubic root +

+
‘log’
+

logarithmic +

+
‘4thrt’
+

4th root +

+
‘5thrt’
+

5th root +

+
+

Default value is ‘log’. +

+
+
fscale
+

Specify frequency scale. +

+

It accepts the following values: +

+
‘lin’
+

linear +

+
‘log’
+

logarithmic +

+
+ +

Default value is ‘lin’. +

+
+
saturation
+

Set saturation modifier for displayed colors. Negative values provide +alternative color scheme. 0 is no saturation at all. +Saturation must be in [-10.0, 10.0] range. +Default value is 1. +

+
+
win_func
+

Set window function. +

+

It accepts the following values: +

+
‘rect’
+
‘bartlett’
+
‘hann’
+
‘hanning’
+
‘hamming’
+
‘blackman’
+
‘welch’
+
‘flattop’
+
‘bharris’
+
‘bnuttall’
+
‘bhann’
+
‘sine’
+
‘nuttall’
+
‘lanczos’
+
‘gauss’
+
‘tukey’
+
‘dolph’
+
‘cauchy’
+
‘parzen’
+
‘poisson’
+
‘bohman’
+
‘kaiser’
+
+

Default value is hann. +

+
+
orientation
+

Set orientation of time vs frequency axis. Can be vertical or +horizontal. Default is vertical. +

+
+
gain
+

Set scale gain for calculating intensity color values. +Default value is 1. +

+
+
legend
+

Draw time and frequency axes and legends. Default is enabled. +

+
+
rotation
+

Set color rotation, must be in [-1.0, 1.0] range. +Default value is 0. +

+
+
start
+

Set start frequency from which to display spectrogram. Default is 0. +

+
+
stop
+

Set stop frequency to which to display spectrogram. Default is 0. +

+
+
drange
+

Set dynamic range used to calculate intensity color values. Default is 120 dBFS. +Allowed range is from 10 to 200. +

+
+
limit
+

Set upper limit of input audio samples volume in dBFS. Default is 0 dBFS. +Allowed range is from -100 to 100. +

+
+
opacity
+

Set opacity strength when using pixel format output with alpha component. +

+
+ +

20.27.1 Examples

+ +
    +
  • Extract an audio spectrogram of a whole audio track +in a 1024x1024 picture using ffmpeg: +
    +
    ffmpeg -i audio.flac -lavfi showspectrumpic=s=1024x1024 spectrogram.png
    +
    +
+ +

20.28 showvolume

+ +

Convert input audio volume to a video output. +

+

The filter accepts the following options: +

+
+
rate, r
+

Set video rate. +

+
+
b
+

Set border width, allowed range is [0, 5]. Default is 1. +

+
+
w
+

Set channel width, allowed range is [80, 8192]. Default is 400. +

+
+
h
+

Set channel height, allowed range is [1, 900]. Default is 20. +

+
+
f
+

Set fade, allowed range is [0, 1]. Default is 0.95. +

+
+
c
+

Set volume color expression. +

+

The expression can use the following variables: +

+
+
VOLUME
+

Current max volume of channel in dB. +

+
+
PEAK
+

Current peak. +

+
+
CHANNEL
+

Current channel number, starting from 0. +

+
+ +
+
t
+

If set, displays channel names. Default is enabled. +

+
+
v
+

If set, displays volume values. Default is enabled. +

+
+
o
+

Set orientation, can be horizontal: h or vertical: v, +default is h. +

+
+
s
+

Set step size, allowed range is [0, 5]. Default is 0, which means +step is disabled. +

+
+
p
+

Set background opacity, allowed range is [0, 1]. Default is 0. +

+
+
m
+

Set metering mode, can be peak: p or rms: r, +default is p. +

+
+
ds
+

Set display scale, can be linear: lin or log: log, +default is lin. +

+
+
dm
+

In second. +If set to > 0., display a line for the max level +in the previous seconds. +default is disabled: 0. +

+
+
dmc
+

The color of the max line. Use when dm option is set to > 0. +default is: orange +

+
+ +

20.29 showwaves

+ +

Convert input audio to a video output, representing the samples waves. +

+

The filter accepts the following options: +

+
+
size, s
+

Specify the video size for the output. For the syntax of this option, check the +(ffmpeg-utils)"Video size" section in the ffmpeg-utils manual. +Default value is 600x240. +

+
+
mode
+

Set display mode. +

+

Available values are: +

+
‘point’
+

Draw a point for each sample. +

+
+
‘line’
+

Draw a vertical line for each sample. +

+
+
‘p2p’
+

Draw a point for each sample and a line between them. +

+
+
‘cline’
+

Draw a centered vertical line for each sample. +

+
+ +

Default value is point. +

+
+
n
+

Set the number of samples which are printed on the same column. A +larger value will decrease the frame rate. Must be a positive +integer. This option can be set only if the value for rate +is not explicitly specified. +

+
+
rate, r
+

Set the (approximate) output frame rate. This is done by setting the +option n. Default value is "25". +

+
+
split_channels
+

Set if channels should be drawn separately or overlap. Default value is 0. +

+
+
colors
+

Set colors separated by ’|’ which are going to be used for drawing of each channel. +

+
+
scale
+

Set amplitude scale. +

+

Available values are: +

+
‘lin’
+

Linear. +

+
+
‘log’
+

Logarithmic. +

+
+
‘sqrt’
+

Square root. +

+
+
‘cbrt’
+

Cubic root. +

+
+ +

Default is linear. +

+
+
draw
+

Set the draw mode. This is mostly useful to set for high n. +

+

Available values are: +

+
‘scale’
+

Scale pixel values for each drawn sample. +

+
+
‘full’
+

Draw every sample directly. +

+
+ +

Default value is scale. +

+
+ +

20.29.1 Examples

+ +
    +
  • Output the input file audio and the corresponding video representation +at the same time: +
    +
    amovie=a.mp3,asplit[out0],showwaves[out1]
    +
    + +
  • Create a synthetic signal and show it with showwaves, forcing a +frame rate of 30 frames per second: +
    +
    aevalsrc=sin(1*2*PI*t)*sin(880*2*PI*t):cos(2*PI*200*t),asplit[out0],showwaves=r=30[out1]
    +
    +
+ +

20.30 showwavespic

+ +

Convert input audio to a single video frame, representing the samples waves. +

+

The filter accepts the following options: +

+
+
size, s
+

Specify the video size for the output. For the syntax of this option, check the +(ffmpeg-utils)"Video size" section in the ffmpeg-utils manual. +Default value is 600x240. +

+
+
split_channels
+

Set if channels should be drawn separately or overlap. Default value is 0. +

+
+
colors
+

Set colors separated by ’|’ which are going to be used for drawing of each channel. +

+
+
scale
+

Set amplitude scale. +

+

Available values are: +

+
‘lin’
+

Linear. +

+
+
‘log’
+

Logarithmic. +

+
+
‘sqrt’
+

Square root. +

+
+
‘cbrt’
+

Cubic root. +

+
+ +

Default is linear. +

+
+
draw
+

Set the draw mode. +

+

Available values are: +

+
‘scale’
+

Scale pixel values for each drawn sample. +

+
+
‘full’
+

Draw every sample directly. +

+
+ +

Default value is scale. +

+
+
filter
+

Set the filter mode. +

+

Available values are: +

+
‘average’
+

Use average samples values for each drawn sample. +

+
+
‘peak’
+

Use peak samples values for each drawn sample. +

+
+ +

Default value is average. +

+
+ +

20.30.1 Examples

+ +
    +
  • Extract a channel split representation of the wave form of a whole audio track +in a 1024x800 picture using ffmpeg: +
    +
    ffmpeg -i audio.flac -lavfi showwavespic=split_channels=1:s=1024x800 waveform.png
    +
    +
+ +

20.31 sidedata, asidedata

+ +

Delete frame side data, or select frames based on it. +

+

This filter accepts the following options: +

+
+
mode
+

Set mode of operation of the filter. +

+

Can be one of the following: +

+
+
‘select’
+

Select every frame with side data of type. +

+
+
‘delete’
+

Delete side data of type. If type is not set, delete all side +data in the frame. +

+
+
+ +
+
type
+

Set side data type used with all modes. Must be set for select mode. +

+

Possible values are: +

+
+
‘PANSCAN’
+
‘A53_CC’
+
‘STEREO3D’
+
‘MATRIXENCODING’
+
‘DOWNMIX_INFO’
+
‘REPLAYGAIN’
+
‘DISPLAYMATRIX’
+
‘AFD’
+
‘MOTION_VECTORS’
+
‘SKIP_SAMPLES’
+
‘AUDIO_SERVICE_TYPE’
+
‘MASTERING_DISPLAY_METADATA’
+
‘GOP_TIMECODE’
+
‘SPHERICAL’
+
‘CONTENT_LIGHT_LEVEL’
+
‘ICC_PROFILE’
+
‘S12M_TIMECODE’
+
‘DYNAMIC_HDR_PLUS’
+
‘REGIONS_OF_INTEREST’
+
‘VIDEO_ENC_PARAMS’
+
‘SEI_UNREGISTERED’
+
‘FILM_GRAIN_PARAMS’
+
‘DETECTION_BOUNDING_BOXES’
+
‘DETECTION_BBOXES’
+
‘DOVI_RPU_BUFFER’
+
‘DOVI_METADATA’
+
‘DYNAMIC_HDR_VIVID’
+
‘AMBIENT_VIEWING_ENVIRONMENT’
+
‘VIDEO_HINT’
+
+ +
+
+ +

20.32 spectrumsynth

+ +

Synthesize audio from 2 input video spectrums, first input stream represents +magnitude across time and second represents phase across time. +The filter will transform from frequency domain as displayed in videos back +to time domain as presented in audio output. +

+

This filter is primarily created for reversing processed showspectrum +filter outputs, but can synthesize sound from other spectrograms too. +But in such case results are going to be poor if the phase data is not +available, because in such cases phase data need to be recreated, usually +it’s just recreated from random noise. +For best results use gray only output (channel color mode in +showspectrum filter) and log scale for magnitude video and +lin scale for phase video. To produce phase, for 2nd video, use +data option. Inputs videos should generally use fullframe +slide mode as that saves resources needed for decoding video. +

+

The filter accepts the following options: +

+
+
sample_rate
+

Specify sample rate of output audio, the sample rate of audio from which +spectrum was generated may differ. +

+
+
channels
+

Set number of channels represented in input video spectrums. +

+
+
scale
+

Set scale which was used when generating magnitude input spectrum. +Can be lin or log. Default is log. +

+
+
slide
+

Set slide which was used when generating inputs spectrums. +Can be replace, scroll, fullframe or rscroll. +Default is fullframe. +

+
+
win_func
+

Set window function used for resynthesis. +

+
+
overlap
+

Set window overlap. In range [0, 1]. Default is 1, +which means optimal overlap for selected window function will be picked. +

+
+
orientation
+

Set orientation of input videos. Can be vertical or horizontal. +Default is vertical. +

+
+ +

20.32.1 Examples

+ +
    +
  • First create magnitude and phase videos from audio, assuming audio is stereo with 44100 sample rate, +then resynthesize videos back to audio with spectrumsynth: +
    +
    ffmpeg -i input.flac -lavfi showspectrum=mode=separate:scale=log:overlap=0.875:color=channel:slide=fullframe:data=magnitude -an -c:v rawvideo magnitude.nut
    +ffmpeg -i input.flac -lavfi showspectrum=mode=separate:scale=lin:overlap=0.875:color=channel:slide=fullframe:data=phase -an -c:v rawvideo phase.nut
    +ffmpeg -i magnitude.nut -i phase.nut -lavfi spectrumsynth=channels=2:sample_rate=44100:win_func=hann:overlap=0.875:slide=fullframe output.flac
    +
    +
+ +

20.33 split, asplit

+ +

Split input into several identical outputs. +

+

asplit works with audio input, split with video. +

+

The filter accepts a single parameter which specifies the number of outputs. If +unspecified, it defaults to 2. +

+

20.33.1 Examples

+ +
    +
  • Create two separate outputs from the same input: +
    +
    [in] split [out0][out1]
    +
    + +
  • To create 3 or more outputs, you need to specify the number of +outputs, like in: +
    +
    [in] asplit=3 [out0][out1][out2]
    +
    + +
  • Create two separate outputs from the same input, one cropped and +one padded: +
    +
    [in] split [splitout1][splitout2];
    +[splitout1] crop=100:100:0:0    [cropout];
    +[splitout2] pad=200:200:100:100 [padout];
    +
    + +
  • Create 5 copies of the input audio with ffmpeg: +
    +
    ffmpeg -i INPUT -filter_complex asplit=5 OUTPUT
    +
    +
+ +

20.34 zmq, azmq

+ +

Receive commands sent through a libzmq client, and forward them to +filters in the filtergraph. +

+

zmq and azmq work as a pass-through filters. zmq +must be inserted between two video filters, azmq between two +audio filters. Both are capable to send messages to any filter type. +

+

To enable these filters you need to install the libzmq library and +headers and configure FFmpeg with --enable-libzmq. +

+

For more information about libzmq see: +http://www.zeromq.org/ +

+

The zmq and azmq filters work as a libzmq server, which +receives messages sent through a network interface defined by the +bind_address (or the abbreviation "b") option. +Default value of this option is tcp://localhost:5555. You may +want to alter this value to your needs, but do not forget to escape any +’:’ signs (see filtergraph escaping). +

+

The received message must be in the form: +

+
TARGET COMMAND [ARG]
+
+ +

TARGET specifies the target of the command, usually the name of +the filter class or a specific filter instance name. The default +filter instance name uses the pattern ‘Parsed_<filter_name>_<index>’, +but you can override this by using the ‘filter_name@id’ syntax +(see Filtergraph syntax). +

+

COMMAND specifies the name of the command for the target filter. +

+

ARG is optional and specifies the optional argument list for the +given COMMAND. +

+

Upon reception, the message is processed and the corresponding command +is injected into the filtergraph. Depending on the result, the filter +will send a reply to the client, adopting the format: +

+
ERROR_CODE ERROR_REASON
+MESSAGE
+
+ +

MESSAGE is optional. +

+

20.34.1 Examples

+ +

Look at tools/zmqsend for an example of a zmq client which can +be used to send commands processed by these filters. +

+

Consider the following filtergraph generated by ffplay. +In this example the last overlay filter has an instance name. All other +filters will have default instance names. +

+
+
ffplay -dumpgraph 1 -f lavfi "
+color=s=100x100:c=red  [l];
+color=s=100x100:c=blue [r];
+nullsrc=s=200x100, zmq [bg];
+[bg][l]   overlay     [bg+l];
+[bg+l][r] overlay@my=x=100 "
+
+ +

To change the color of the left side of the video, the following +command can be used: +

+
echo Parsed_color_0 c yellow | tools/zmqsend
+
+ +

To change the right side: +

+
echo Parsed_color_1 c pink | tools/zmqsend
+
+ +

To change the position of the right side: +

+
echo overlay@my x 150 | tools/zmqsend
+
+ + + +

21 Multimedia Sources

+ +

Below is a description of the currently available multimedia sources. +

+

21.1 amovie

+ +

This is the same as movie source, except it selects an audio +stream by default. +

+

21.2 avsynctest

+

Generate an Audio/Video Sync Test. +

+

Generated stream periodically shows flash video frame and emits beep in audio. +Useful to inspect A/V sync issues. +

+

It accepts the following options: +

+
+
size, s
+

Set output video size. Default value is hd720. +

+
+
framerate, fr
+

Set output video frame rate. Default value is 30. +

+
+
samplerate, sr
+

Set output audio sample rate. Default value is 44100. +

+
+
amplitude, a
+

Set output audio beep amplitude. Default value is 0.7. +

+
+
period, p
+

Set output audio beep period in seconds. Default value is 3. +

+
+
delay, dl
+

Set output video flash delay in number of frames. Default value is 0. +

+
+
cycle, c
+

Enable cycling of video delays, by default is disabled. +

+
+
duration, d
+

Set stream output duration. By default duration is unlimited. +

+
+
fg, bg, ag
+

Set foreground/background/additional color. +

+
+ +

21.2.1 Commands

+ +

This source supports the some above options as commands. +

+

21.3 movie

+ +

Read audio and/or video stream(s) from a movie container. +

+

It accepts the following parameters: +

+
+
filename
+

The name of the resource to read (not necessarily a file; it can also be a +device or a stream accessed through some protocol). +

+
+
format_name, f
+

Specifies the format assumed for the movie to read, and can be either +the name of a container or an input device. If not specified, the +format is guessed from movie_name or by probing. +

+
+
seek_point, sp
+

Specifies the seek point in seconds. The frames will be output +starting from this seek point. The parameter is evaluated with +av_strtod, so the numerical value may be suffixed by an IS +postfix. The default value is "0". +

+
+
streams, s
+

Specifies the streams to read. Several streams can be specified, +separated by "+". The source will then have as many outputs, in the +same order. The syntax is explained in the (ffmpeg)"Stream specifiers" +section in the ffmpeg manual. Two special names, "dv" and "da" specify +respectively the default (best suited) video and audio stream. Default +is "dv", or "da" if the filter is called as "amovie". +

+
+
stream_index, si
+

Specifies the index of the video stream to read. If the value is -1, +the most suitable video stream will be automatically selected. The default +value is "-1". Deprecated. If the filter is called "amovie", it will select +audio instead of video. +

+
+
loop
+

Specifies how many times to read the stream in sequence. +If the value is 0, the stream will be looped infinitely. +Default value is "1". +

+

Note that when the movie is looped the source timestamps are not +changed, so it will generate non monotonically increasing timestamps. +

+
+
discontinuity
+

Specifies the time difference between frames above which the point is +considered a timestamp discontinuity which is removed by adjusting the later +timestamps. +

+
+
dec_threads
+

Specifies the number of threads for decoding +

+
+
format_opts
+

Specify format options for the opened file. Format options can be specified +as a list of key=value pairs separated by ’:’. The following example +shows how to add protocol_whitelist and protocol_blacklist options: +

+
ffplay -f lavfi
+"movie=filename='1.sdp':format_opts='protocol_whitelist=file,rtp,udp\:protocol_blacklist=http'"
+
+
+
+ +

It allows overlaying a second video on top of the main input of +a filtergraph, as shown in this graph: +

+
input -----------> deltapts0 --> overlay --> output
+                                    ^
+                                    |
+movie --> scale--> deltapts1 -------+
+
+

21.3.1 Examples

+ +
    +
  • Skip 3.2 seconds from the start of the AVI file in.avi, and overlay it +on top of the input labelled "in": +
    +
    movie=in.avi:seek_point=3.2, scale=180:-1, setpts=PTS-STARTPTS [over];
    +[in] setpts=PTS-STARTPTS [main];
    +[main][over] overlay=16:16 [out]
    +
    + +
  • Read from a video4linux2 device, and overlay it on top of the input +labelled "in": +
    +
    movie=/dev/video0:f=video4linux2, scale=180:-1, setpts=PTS-STARTPTS [over];
    +[in] setpts=PTS-STARTPTS [main];
    +[main][over] overlay=16:16 [out]
    +
    + +
  • Read the first video stream and the audio stream with id 0x81 from +dvd.vob; the video is connected to the pad named "video" and the audio is +connected to the pad named "audio": +
    +
    movie=dvd.vob:s=v:0+#0x81 [video] [audio]
    +
    +
+ +

21.3.2 Commands

+ +

Both movie and amovie support the following commands: +

+
seek
+

Perform seek using "av_seek_frame". +The syntax is: seek stream_index|timestamp|flags +

    +
  • stream_index: If stream_index is -1, a default +stream is selected, and timestamp is automatically converted +from AV_TIME_BASE units to the stream specific time_base. +
  • timestamp: Timestamp in AVStream.time_base units +or, if no stream is specified, in AV_TIME_BASE units. +
  • flags: Flags which select direction and seeking mode. +
+ +
+
get_duration
+

Get movie duration in AV_TIME_BASE units. +

+
+
+ + +

22 See Also

+ +

ffmpeg, ffplay, ffprobe, +libavfilter +

+ +

23 Authors

+ +

The FFmpeg developers. +

+

For details about the authorship, see the Git history of the project +(https://git.ffmpeg.org/ffmpeg), e.g. by typing the command +git log in the FFmpeg source directory, or browsing the +online repository at https://git.ffmpeg.org/ffmpeg. +

+

Maintainers for the specific components are listed in the file +MAINTAINERS in the source code tree. +

+ + +

+ This document was generated on May 17, 2026 using makeinfo. +

+ +

Hosting provided by telepoint.bg

+
+
+
+ + + + + + + + + \ No newline at end of file diff --git a/research/live-adaptive-sync/mirrors/google-temporal-synchronization-multiple-audio.html b/research/live-adaptive-sync/mirrors/google-temporal-synchronization-multiple-audio.html new file mode 100644 index 0000000..5493ae5 --- /dev/null +++ b/research/live-adaptive-sync/mirrors/google-temporal-synchronization-multiple-audio.html @@ -0,0 +1,4497 @@ + + + + + + + + + + + + + + + + Temporal Synchronization of Multiple Audio Signals + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + + +
+
+ + + + + + +
+
+

Temporal Synchronization of Multiple Audio Signals

+
+ +
+ + Julius Kammerl + +
+ +
+ + +
+ + Neil Birkbeck + +
+ +
+ + +
+ + Sasi Inguva + +
+ +
+ + +
+ + Damien Kelly + +
+ +
+ + +
+ + Andy Crawford + +
+ +
+ + +
+ + Hugh Denman + +
+ +
+ + +
+ + Anil Kokaram + +
+ +
+ + + + + + +
+
+ Proceedings of the International Conference on Signal Processing (ICASSP), Florence, Italy (2014) +
+ +
+
+ + + + + + + + + + + + + + + + + + + + + + +Google Scholar + + + + + + + + + + + +
+ + + + + + + + + + + + + + + + + + + + + +
+ + + +
+
+
+
+
+
+
+
+

Abstract

+
+
+ Given the proliferation of consumer media recording devices, events often give rise to a large number of recordings. These recordings are taken from different spatial positions and do not have reliable timestamp information. In this paper, we present two robust graph-based approaches for synchronizing multiple audio signals. The graphs are constructed atop the over-determined system resulting from pairwise signal comparison using cross-correlation of audio features. The first approach uses a Minimum Spanning Tree (MST) technique, while the second uses Belief Propagation (BP) to solve the system. Both approaches can provide excellent solutions and robustness to pairwise outliers, however the MST approach is much less complex than BP. In addition, an experimental comparison of audio features-based synchronization shows that spectral flatness outperforms the zero-crossing rate and signal energy. +
+
+
+ +
+ +
+ + + + + + + + + + + + + + + + +
+ + + + + + + + + + + + + + +
+ × +
+
+ +
+
+
+ + + + + + + + + + + + + + + + + diff --git a/research/live-adaptive-sync/mirrors/grondin-glass-2018-gcc-phat-close-mics.pdf b/research/live-adaptive-sync/mirrors/grondin-glass-2018-gcc-phat-close-mics.pdf new file mode 100644 index 0000000..48c1062 Binary files /dev/null and b/research/live-adaptive-sync/mirrors/grondin-glass-2018-gcc-phat-close-mics.pdf differ diff --git a/research/live-adaptive-sync/mirrors/online-estimation-sampling-rate-offsets-2021.pdf b/research/live-adaptive-sync/mirrors/online-estimation-sampling-rate-offsets-2021.pdf new file mode 100644 index 0000000..5126a62 Binary files /dev/null and b/research/live-adaptive-sync/mirrors/online-estimation-sampling-rate-offsets-2021.pdf differ diff --git a/research/live-adaptive-sync/mirrors/schmalenstroeer-haeb-umbach-2013-pretrained-clock-skew.pdf b/research/live-adaptive-sync/mirrors/schmalenstroeer-haeb-umbach-2013-pretrained-clock-skew.pdf new file mode 100644 index 0000000..d20f94b Binary files /dev/null and b/research/live-adaptive-sync/mirrors/schmalenstroeer-haeb-umbach-2013-pretrained-clock-skew.pdf differ diff --git a/research/live-adaptive-sync/mirrors/soxr-wiki-home.html b/research/live-adaptive-sync/mirrors/soxr-wiki-home.html new file mode 100644 index 0000000..0f6e3b4 --- /dev/null +++ b/research/live-adaptive-sync/mirrors/soxr-wiki-home.html @@ -0,0 +1,1618 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + The SoX Resampler library / Wiki / Home + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + +
+
+ + + + + + + + + + + + + + + + + +
+ +
+
+ + +
+
+ +
+
+ +
+ +
+
+
+
+ + +
+ + +
+ + + + + + + + + + + + + + + + + + + + + + + + + + + +
+
+
+ + + + + + + + +
+ + + + + +
+ + + + +
+ +
+ + +
+ +
+ + + +
+
+ + + + + + + Menu + ▾ + ▴ + + + + + + + + + + + + + + + + + + +
+

+ Home + + + + + + + + + + + + + +

+ + +
+ +
+ + + + + + + + + + + + robs + + + + + + + +
+ +
+ + +
+ + + + +

News

+
    +
  • 2018-02-24 Maintenance release, version 0.1.3. Details.
  • +
  • 2015-09-05 Maintenance release, version 0.1.2. Details.
  • +
  • 2013-03-03 Maintenance release, version 0.1.1. Details.
  • +
  • 2013-01-19 First public release, version 0.1.0. Thanks for helping to make this release happen go to the good folk at the FFmpeg and Audacity projects, and to Benjamin Drung.
  • +
+

Documentation

+ +

FAQ

+

How do I unpack a .xz file?
+On a unix-like system, you should use ‘xz’ (or xz-enabled ‘tar’); on MS-windows, 7-Zip will do the job.

+

***How should I create my first application using the resampler? ***
+Have a look at the provided examples: each takes a different approach to interfacing with the library. Choose the one which seems most suitable for your application, then base your application code on the example (you can freely copy the example code into your application, if you like).

+

How can I try out the example code with real audio files?
+For simplicity, the examples work with ‘raw’ audio files (or piped streams), but you can easily convert between raw and other formats using the ‘sox’ command-line utility. For example, using example #3 to convert a stereo ‘wav’ file from 44100 to 48000 Hz:

+
sox input.wav input.f32
+./3-options-input-fn 44100 48000 2 <input.f32 >output.f32
+sox -c 2 -r 48000 output.f32 output.wav
+
+ +

(the ./ should be omitted on MS-windows)

+

How should soxr_quality_spec.flags be set?
+For constant-rate resampling, the flags are mainly for test purposes so should normally be set to zero. For variable-rate resampling, set flags to SOXR_VR (see example #5).

+

Why is my resampled output slightly shorter than it should be?
+A resampler holds a few samples in memory to meet the needs of the resampling filter. In order to ‘flush’ the last few samples through to the output, the resampler must be told to do so (i.e. by informing it that no more input samples are or will be available), then the application must continue to read output samples until no more are available. See examples # 2–4.

+ + + +
+ + +
+ + + + + + + +
+ + + + +
+
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+
+ +
+
+
+ MongoDB Logo + MongoDB +
+
+
+
+
+
+
+ +
+
+
+ + + + + + + + + + + +
+ + +
+ + + +
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/research/live-adaptive-sync/mirrors/synchronization-time-varying-sro-speaker-changes.pdf b/research/live-adaptive-sync/mirrors/synchronization-time-varying-sro-speaker-changes.pdf new file mode 100644 index 0000000..da189eb Binary files /dev/null and b/research/live-adaptive-sync/mirrors/synchronization-time-varying-sro-speaker-changes.pdf differ diff --git a/research/live-adaptive-sync/mirrors/wang-doclo-2016-correlation-maximization-sro.pdf b/research/live-adaptive-sync/mirrors/wang-doclo-2016-correlation-maximization-sro.pdf new file mode 100644 index 0000000..cd1e704 Binary files /dev/null and b/research/live-adaptive-sync/mirrors/wang-doclo-2016-correlation-maximization-sro.pdf differ diff --git a/research/live-adaptive-sync/summary.md b/research/live-adaptive-sync/summary.md new file mode 100644 index 0000000..20ff15e --- /dev/null +++ b/research/live-adaptive-sync/summary.md @@ -0,0 +1,123 @@ +# Live Adaptive Sync: Distilled Research + +## Objective + +Find prior work that supports synchronizing independent live microphone feeds well enough to build a coherent spatial bus for AquaSynth/Ambisonics. + +## Current Mechanism + +Mimir currently launches separate FFmpeg capture processes and SRT endpoints. That preserves OBS mix control but does not give us one clocked multichannel stream. For Ambisonics, independent device clocks create two separate problems: + +- initial offset: one stream starts late or has more device/buffer latency +- sampling-rate offset: one device runs slightly faster/slower, so offset grows over time + +The research literature calls the second problem **sampling-rate offset** or **SRO**, often measured in parts per million. + +## Main Finding + +Live adaptive sync is a known problem in wireless/distributed acoustic sensor networks. The common model is: + +1. estimate clock drift/SRO between independent nodes +2. compensate by resampling, STFT-domain phase correction, or clock control when hardware permits +3. estimate current TDOA/delay separately for localization or beamforming + +Wang and Doclo model SRO as linear phase drift in the STFT domain and note that independent microphone nodes have their own ADC/clock sources; because oscillator tolerances differ, sampling-rate mismatch is inevitable [Wang/Doclo 2016]. They also spell out the practical compensation options: time-domain resampling, interpolation filters, sinc/Lagrange-style correction, or STFT-domain compensation [Wang/Doclo 2016]. + +Didier et al. report SROs up to hundreds of ppm in common heterogeneous devices and emphasize that SRO causes increasing time drift, breaking coherent signal processing [Didier 2023]. Their method estimates SRO from coherence drift and compensates with frequency-domain phase shifts, while also detecting full-sample drifts in a WOLA/STFT processing chain [Didier 2023]. + +Schmalenstroeer and Haeb-Umbach show the control-loop version: timestamp exchange gives a first clock-frequency estimate, a Kalman filter smooths it, and then one node's sampling frequency is adjusted; their long-term tests kept oscillator deviation under half a sample, enabling cross-device TDOA estimation [Schmalenstroeer 2013]. That is closest to "lock them into sync," but it requires either controllable clocks or an equivalent software resampling actuator. + +## Architecture We Can Steal Without Shame + +```mermaid +flowchart TD + A["Raw mic streams"] --> B["Ring buffers with monotonic local timestamps"] + C["Calibration ping / chirp"] --> D["Initial offset, polarity, gain, impulse response"] + B --> E["Passive coherence / cross-correlation tracker"] + D --> E + E --> F["SRO estimator: ppm drift"] + E --> G["Delay/TDOA estimator: current sample offset"] + F --> H["Adaptive resampler per non-reference device"] + G --> I["Delay line / fractional delay correction"] + H --> I + I --> J["Aligned multichannel frame bus"] + J --> K["Spatial estimator / source encoder"] + K --> L["Ambisonic bus into AquaSynth Faust DSP"] +``` + +## Design Implications + +- Pick one reference stream. + One mic/interface becomes the local time authority. Every other stream is resampled toward it. This is not philosophical. Somebody has to be the clock or the whole room starts freelancing. + +- Separate the slow and fast loops. + SRO is a slow ppm-level estimate. Current TDOA/delay is a frame-level estimate. If one loop owns both, it will overreact to speech movement and reverberation. + +- Use pings for initialization and health checks, not as the only sync. + A chirp/MLS/pulse can estimate fixed latency, polarity, gain, and rough impulse response. It does not prevent future drift. Periodic pings can re-anchor the system, but they are intrusive unless hidden in a controlled band and supported by the microphones/signal path. + +- Prefer passive SRO tracking during speech. + Coherence-drift and STFT phase-drift methods are designed for exactly this: estimate rate mismatch from live audio without requiring dedicated sync tones [Wang/Doclo 2016; Didier 2023]. + +- Use GCC-PHAT for TDOA/localization, not clock locking. + GCC-PHAT is a standard TDOA/DOA tool and has real-time variants studied for close microphone pairs [Grondin/Glass 2018]. It estimates arrival-time differences; it does not by itself distinguish source movement from clock drift over long windows. + +- Resampling is the actuator. + If hardware clocks cannot be controlled, the software actuator is adaptive sample-rate conversion. FFmpeg's `aresample` can stretch/squeeze audio to match timestamps, and libsoxr supports variable-rate resampling via `SOXR_VR` [FFmpeg Filters; SoXR]. For this project, a custom live engine will probably need an explicit variable-ratio resampler rather than hoping FFmpeg guesses the right timestamps. + +## Candidate V1 Algorithm + +1. Capture every mic into its own ring buffer at nominal 48 kHz. +2. Select the shielded cardioid or a stable interface channel as reference. +3. Run startup calibration: + - play/capture broadband chirp or sharp pulse + - estimate fixed delay per mic by cross-correlation + - estimate polarity and gain + - store confidence and impulse-response sketch +4. Process live in frames, probably 10-20 ms with overlap for analysis. +5. For each non-reference stream: + - estimate short-window delay with GCC-PHAT/cross-correlation + - estimate longer-window SRO from phase/coherence drift + - smooth SRO with a Kalman/PLL-style filter + - drive a variable-rate resampler plus fractional delay line +6. Emit aligned multichannel frames into the spatial encoder. +7. Track health: + - estimated ppm + - residual delay error + - correlation/coherence confidence + - dropouts and buffer pressure + - "sync lost" state when confidence collapses + +## Failure Modes + +- Moving speakers can look like drift if the estimator uses the wrong window. +- Reverberation and camera mic processing can corrupt phase/TDOA estimates. +- USB/camera mics may apply AGC, noise suppression, compression, or hidden resampling. +- Bluetooth paths are probably disqualified for coherent spatial work. +- Ultrasonic pings may be eaten by mic frequency response, codecs, driver filtering, or camera audio processing. +- Separate FFmpeg processes may not expose enough timing authority for tight adaptive control. + +## Recommended Next Cut + +Do not start with a full arbitrary-array ambisonic encoder. First build a sync harness: + +- two live mic inputs +- startup chirp calibration +- live SRO estimate in ppm +- adaptive resampler correction +- residual delay meter +- recorded before/after WAV evidence + +Once two mics can stay aligned for 20-30 minutes, expand to four. Only then should the output feed the Ambisonic/AquaSynth graph. Spatial DSP built on drifting channels is just a confident hallucination with XLR cables. + +## Citations + +- [Wang/Doclo 2016] L. Wang and S. Doclo, "Correlation Maximization-Based Sampling Rate Offset Estimation for Distributed Microphone Arrays." Mirror: `mirrors/wang-doclo-2016-correlation-maximization-sro.pdf`. +- [Schmalenstroeer 2013] J. Schmalenstroeer and R. Haeb-Umbach, "Sampling Rate Synchronisation in Acoustic Sensor Networks with a Pre-Trained Clock Skew Error Model." Mirror: `mirrors/schmalenstroeer-haeb-umbach-2013-pretrained-clock-skew.pdf`. +- [Didier 2023] P. Didier, T. van Waterschoot, S. Doclo, and M. Moonen, "Sampling Rate Offset Estimation and Compensation for Distributed Adaptive Node-Specific Signal Estimation in Wireless Acoustic Sensor Networks." Mirror: `mirrors/didier-2023-sro-estimation-compensation-wasn.pdf`. +- [Time-varying SRO] "On Synchronization of Wireless Acoustic Sensor Networks in the Presence of Time-varying Sampling Rate Offsets and Speaker Changes." Mirror: `mirrors/synchronization-time-varying-sro-speaker-changes.pdf`. +- [Online SRO 2021] "Online Estimation of Sampling Rate Offsets." Mirror: `mirrors/online-estimation-sampling-rate-offsets-2021.pdf`. +- [Google Sync] A. Shrestha et al., "Temporal Synchronization of Multiple Audio Signals." Mirror: `mirrors/google-temporal-synchronization-multiple-audio.html`. +- [Grondin/Glass 2018] F. Grondin and J. Glass, "A Study of the Complexity and Accuracy of Direction of Arrival Estimation Methods Based on GCC-PHAT for a Pair of Close Microphones." Mirror: `mirrors/grondin-glass-2018-gcc-phat-close-mics.pdf`. +- [FFmpeg Filters] FFmpeg Filters Documentation. Mirror: `mirrors/ffmpeg-filters.html`. +- [SoXR] SoX Resampler library / libsoxr wiki. Mirror: `mirrors/soxr-wiki-home.html`. diff --git a/research/playback-calibration/bibliography.md b/research/playback-calibration/bibliography.md new file mode 100644 index 0000000..c0f47eb --- /dev/null +++ b/research/playback-calibration/bibliography.md @@ -0,0 +1,28 @@ +# Bibliography: Playback Calibration And Loudspeaker Rendering + +Mirrored copies live under `research/playback-calibration/mirrors/`. + +## Ambisonic Loudspeaker Decoding + +- [IEM Plug-in Suite, "AllRADecoder Guide"](https://plugins.iem.at/docs/allradecoder/) + Mirror: `mirrors/iem-allradecoder-guide.html` + +- [IEM Plug-in Suite, "Configuration Files"](https://plugins.iem.at/docs/configurationfiles/) + Mirror: `mirrors/iem-configuration-files.html` + +- [IEM Plug-in Suite, "Plug-in Descriptions"](https://plugins.iem.at/docs/plugindescriptions/) + Mirror: `mirrors/iem-plugin-descriptions.html` + +- [RingBuffer, "Decoding to Loudspeaker Setups"](https://ringbuffer.org/spatial_audio/iem-reaper/decoding-loudspeaker-iem/) + Mirror: `mirrors/ringbuffer-decoding-loudspeaker-iem.html` + +- [SPARTA, "Overview"](https://leomccormack.github.io/sparta-site/docs/plugins/overview/) + Mirror: `mirrors/sparta-overview.html` + +- [ICST Ambisonics, "How it Works"](https://ambisonics.ch/icst-ambisonics-plugins/03_how_it_works/) + Mirror: `mirrors/icst-how-it-works.html` + +## Measurement + +- [FLUX:: MiRA, "Impulse response measurement"](https://doc.flux.audio/mira/Impulse_response_measurement.html) + Mirror: `mirrors/flux-mira-impulse-response-measurement.html` diff --git a/research/playback-calibration/mirrors/flux-mira-impulse-response-measurement.html b/research/playback-calibration/mirrors/flux-mira-impulse-response-measurement.html new file mode 100644 index 0000000..6d4ec4a --- /dev/null +++ b/research/playback-calibration/mirrors/flux-mira-impulse-response-measurement.html @@ -0,0 +1,1099 @@ + + + + + + + + + +Impulse response measurement – MiRA + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+
+ + +
+ +
+ + +
+ + + +
+ +
+
+

Impulse response measurement

+
+ + + +
+ + + + +
+ + + +
+ + +
+

Introduction

+

The impulse response of a system is the signal obtained at the output when feeding a click (also termed impulse, spike or Dirac) its input. It is a fundamental tool to describe the time properties of a linear system.

+

Combined with the transfer function, impulse response measurement is essential in characterizing the acoustics of a studio, concert hall or venue, from which synthetic figures such as reverberation time are derived. Determining the impulse response of an amplifier and loudspeaker in tandem can also serve to assess their performance.

+

A pass-trough device, or equivalently, a completely dead space such as an anechoic chamber, exhibits a unit impulse response, whose value at zero time is gain, and is zero at all other instants.

+
+
+

Impulse response display example

+
Impulse response display example
+
+
+
+

Analyze/freeze

+

The button toggles the impulse response real-time update on and off.

+
+
+

Delay Unit

+

Change the unit in which the time delay is expressed. It can be expressed as:

+
    +
  • samples
  • +
  • milliseconds
  • +
  • meters
  • +
  • feet
  • +
+
+
+
+

General procedure

+

Impulse response (IR) measurement requires that sufficient samples be accumulated before the actual computation is ready, depending on the values of the Max Length and Time averaging Time averaging settings. The user interface displays a message indicating the remaining time before the display is ready, whenever the related settings are changed or the reset button is pressed.

+

Because the software cannot detect whenever you make changes to the analyzed system, you need to press the Reset button in the setup or wait for the display to stabilize before reading the display.

+

Once your test setup is ready, press the ‘Reset’ button and wait for the display showing the remaining time to disappear, at which point the IR display is ready. When a sufficient amount of samples has been accumulated, IR computation goes on as long as the ‘Run’ button is active, and updated with new incoming samples.

+
+
+
+ +
+
+Note +
+
+
+

Make sure the actual impulse response is shorter than the maximum specified time, otherwise mild to severe time-aliasing will occur, and the measurement will not be reliable.

+

A good rule of thumb is to set the Max length parameter to twice that of the estimated RT60 of the room.

+

If in doubt, raise the Max length setting until the impulse response curve does not change, and check the tail of the curve does indeed fall to zero.

+
+
+
+
+

Time averaging

+

The time averaging function computes the mean of several IR measurements over time, which is very useful to filter out noise and other artifacts. It is enabled by default as this gives better display stability and measurement robustness, however averaging also slows down the reactivity of the display to incoming variations, so you can disable it if needed.

+

When IR averaging is enabled, a message is shown giving the number of currently computed impulse responses versus averaging length. The display switches to show the mean confidence percentage when ready.

+
+
+

Settings

+
+

Main

+

Impulse response setup options

+ ++++ + + + + + + + + + + + + + + + + +
NameDescription
RunToggles impulse response live update on and off. Default is on.
You can temporarily freeze the impulse response with this button, to examine it in detail at your leisure, without worrying about changing external conditions.
Disabling ‘Run’ is equivalent to freezing the measurement, and leaves the averaging buffer as is.
ResetResets the impulse response computation, including the averaging buffer.1
+ +
+

+
+
+

Time

+ ++++ + + + + + + + + + + + + + + + + +
NameDescription
Max lengthSets the maximum length of the impulse response in seconds.
If the reverberation time in your room exceeds this value, time-aliasing will occur, meaning that the impulse response computation will be incorrect and some of the reverberation tail might end up at the start of the display.
The default value is 0.3s.
Increasing this value not only requires more processing power, it also increases the time needed to wait for the display to be updated, as the calculations involved need a greater amount of incoming audio samples to be processed.
Combining time averaging and a long length setting means you’ll have to wait 30 seconds or so for the display to stabilize, so you should really do this if you need to or do not mind waiting.
Time averagingAccumulates several impulse response measurements and averages them before display.
This allows for more precise measurements and lessens the effect of spurious acoustic noise interfering with the measurement, but it also means having to wait longer for the measurement to be ready.
+
+
+

Scale

+ ++++ + + + + + + + + + + + + +
NameDescription
Range modeToggles auto-scaling the vertical axis to the effective range of the impulse response data in the current timeframe.
It functions as an automatic zoom alongside the vertical axis, which can be useful for hands-free operation.
+
+
+

Other

+ ++++ + + + + + + + + + + + + + + + + + + + + + + + + +
NameDescription
Zoom XAdjusts the horizontal axis zoom factor, which can also be changed by clicking inside the impulse response display itself and rotating the mouse center wheel up and down (scroll in / out), if your mouse has this feature.
Zoom Y-/+Adjusts the vertical axis zoom factor.
UnitChange the unit used on the x-axis of the IR graph. Default is “ms”. Possible options are:
- smp (samples)
- ms (milliseconds)
- s (seconds)
WidthChange the thickness of the impulse response curve. Default is 1.5 pixels.
+ + +
+
+
+
+
    +
  1. If you are using a lengthy averaging setting and have just changed your setup, you can reset the entire impulse response to immediately forget previous measurements.↩︎

  2. +
+
+ +
+ + +
+ + + + + + \ No newline at end of file diff --git a/research/playback-calibration/mirrors/icst-how-it-works.html b/research/playback-calibration/mirrors/icst-how-it-works.html new file mode 100644 index 0000000..fd74f31 --- /dev/null +++ b/research/playback-calibration/mirrors/icst-how-it-works.html @@ -0,0 +1,31 @@ +How it Works | ICST Ambisonics

How it Works

Level: Beginner | Audience: Composer, technician, student, studio user.

Use this page when you want to understand the basic logic of the ICST Ambisonics workflow before building or modifying a session.

The signal chain in one minute

The standard production chain is:

Source -> Encoder -> Bformat Master -> Decoder -> Speakers

Optional monitoring branch:

Bformat Master -> Binaural Decoder -> Headphones

The most important rule is this:

  • the encoder creates the HOA / B-format field
  • the Bformat Master is the central collection and render point
  • the decoder is for monitoring over loudspeakers
  • headphone monitoring should use a separate binaural decoder
  • final export should come from the Bformat Master, not from the decoder output

A helpful way to think about Ambisonics is this: the system does not primarily store speakers or objects, but a sound field representation. +The decoder does not “create space” afterwards. It projects an already encoded spatial field onto a concrete loudspeaker setup.

What each part does

  • Source
    A mono or multichannel sound source that you want to place in space.
  • Encoder
    Positions or moves the source inside the Ambisonics field.
  • Bformat Master
    Receives the encoded HOA signal and acts as the main recording and rendering point.
  • Decoder
    Translates the B-format field into loudspeaker signals for the actual room.
  • Binaural decoder
    Converts the same HOA signal into headphone monitoring without changing the loudspeaker decoder setup.

Required vs optional

Required for a basic loudspeaker setup:

  • source track
  • encoder
  • Bformat Master
  • decoder

Optional but often useful:

  • binaural monitoring path
  • B-format player track
  • track templates or project templates
  • OSC control
  • MultiDecoder for layered or segmented arrays

Typical REAPER structure

The following image shows a typical Ambisonics workflow:

01_easyworkflow

The next image shows the ICST plugin signal flow:

0_workflow_

In REAPER, the signal flow often appears like this:

03_reaper_workflow

Typical track roles:

  • Decoder
  • Bformat Master
  • B-format (ambiX) Player
  • MultiEncoder with mono child tracks

Ambi_Signalflow

Which encoder should you use?

Use MonoEncoder when:

  • you want to position or automate one source at a time
  • you are learning the workflow
  • you want the clearest routing per source

Use MultiEncoder when:

  • you want to manage several sources in one interface
  • you need grouped motion or choreographic movement
  • you want a template-based setup with multiple source tracks already prepared

Common misunderstandings

  • The decoder is not the render target.
    It is mainly the loudspeaker-monitoring stage.
  • The Bformat Master is not just another bus.
    It is the central HOA signal that should stay stable and clearly named.
  • Binaural monitoring is not the same as the loudspeaker decoder.
    Treat it as a separate listening branch.
  • Higher order is not automatically better.
    The HOA order should match the real loudspeaker density and the production goal.

Next steps

\ No newline at end of file diff --git a/research/playback-calibration/mirrors/iem-allradecoder-guide.html b/research/playback-calibration/mirrors/iem-allradecoder-guide.html new file mode 100644 index 0000000..71af1a2 --- /dev/null +++ b/research/playback-calibration/mirrors/iem-allradecoder-guide.html @@ -0,0 +1,185 @@ + + + + + + + + + AllRADecoder Guide + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ +
+
+ + + + +
+
+
+
+
+

AllRADecoder Guide

+

+ This guide shows how to create a decoder for any desired loudspeaker layout using the AllRAD approach. +

+ +
+
+ + + + + + +

General Information

+

This plug-in let’s you create an Ambisonic decoder for any desired loudspeaker layout and Ambisonic orders up to 7th. The resulting decoder is automatically applied to the incoming Ambisonic audio stream, so it can be directly inserted into a decoder track of your audio project. You might also want to use it as a standalone application to design a decoder and write it into a JSON configuration file for the use in other plug-ins (e.g. SimpleDecoder).

+

A theoretical description of the AllRAD approach can be found here.

+

How to start

+

The first thing you want to do is inserting the loudspeaker coordinates (azimuth and elevation) and the channel numbers by clicking ‘add loudspeaker’. The fields ‘radius’ and ‘gain’ will only affect imaginary loudspeakers (see below). Loudspeaker at the same level as the listener’s head should by specified by an elevation angle of 0°.

+

You can click the Noise-button which will send a short noise-signal to the corresponding channel. That way you can verify whether your routing is correct.

+

The loudspeakers will appear both in the 3D visualization (top left) and in the loudness map (bottom left). The selected loudspeaker is visualized in green and imaginary ones in orange, while the others are blue. You can rotate the 3D view by dragging, and zoom in/out using the mouse wheel. As opposed to imaginary loudspeakers, physical loudspeaker directions are always visualized with a radius of one.

+

You can rotate the whole layout around the z-axis by clicking the rotate button. This will add a constant value to all azimuth angles.

+

After inserting enough loudspeakers, the layout is going to be triangulated immediately and it is also displayed in the 3D visualization. Any problems with your layout will be shown in the message area below the loudspeaker list.

+

Adding Imaginary Loudspeakers

+

Sometimes, a loudspeaker layout cannot be triangulated in a non-ambiguous way. With the AllRAD approach, imaginary loudspeakers can be added to circumvent these cases.

+

For example, almost every layout without loudspeakers in the lower hemisphere has an ambiguous ground plane. Inserting an imaginary loudspeaker just below the listener will prevent that case. The following layout depicts that example. You also might want to set the gain of that specific imaginary loudspeaker to zero, otherwise sources from below will be reproduced by all of the floor level loudspeakers. Download the configuration file for this layout here: layoutWithOneImaginaryLoudspeaker.json

+

Without imaginary loudspeakers

+

However, in that layout there are several other planes, which could be divided into triangles in a different way: the top plane, left and right side, and also the plane right behind the listener. Additionally, there is no way the top plane can be triangulated symmetrically. It’s advisable to add imaginary loudspeakers here, too. In the following layout four additional imaginary loudspeakers were added. The configuration can be downloaded here: layoutWithAdditionalImaginaryLoudspeakers.json

+

With imaginary loudspeakers

+

The gain is left unchanged, as one does not want silent spots in those areas. However, for loudspeaker #14 the radius had to be altered. Otherwise, there would have been a connection between two imaginary loudspeakers, which is not supported by the AllRAD approach.

+

Choosing a suitable Ambisonic order

+

The choice of the Ambisonic order is a trade-off between constant source width and smooth panning (low order) and precise localization and bumpier panning getting more similar to VBAP (high order). It’s typical to choose 2nd order for 60° spaced loudspeaker layouts, 3rd for 45°, and 5th order for 30°, respectively. 7th order is the highest you can choose and is suitable for 15° spaced layouts.

+

Energy distribution and rE source width map

+

When a decoder was calculated successfully, it’s energy distribution is shown in the loudness map in the bottom left (red). Peak values are visualized in red (up to +3dB) while areas of low energy (down to -3dB) are transparent. The choice of the Ambisonic order will influence the energy distribution.

+

When double-clicking on the map, it will display the acos-rE source width (green). The brighter the color, the wider the image of a source which is panned to that direction. In general, sources at a loudspeaker position will be smaller (less green) than sources between loudspeakers, as several loudspeaker will try to reproduce the signal.

+

Export Layouts and Decoders

+

You can export your layout to a JSON file by clicking the ’export’ button. Additionally, you can check the ‘Export Decoder’ checkbox to also write the calculated (don’t forget to calculate it first) decoder matrix to the configuration file, which e.g. can be used with the SimpleDecoder plug-in.

+

Future Features

+

In future implementations there will be more analysis than the loudness (energy) map, with maps of the rE-vector-based analysis of directional mapping and perceived width. Also the AllRAD2 approach will be implemented soon. It brings improvements to a panning-invariant loudness on typically irregular layout as the 4+5+0 layout from ITU BS.2051 with dense loudspeaker spacing only in front.

+ + + +
+
+
+
+
+ +

Website created with Hugo using the kube theme.

+
+ +
+ + + + + + + + diff --git a/research/playback-calibration/mirrors/iem-configuration-files.html b/research/playback-calibration/mirrors/iem-configuration-files.html new file mode 100644 index 0000000..3a081f4 --- /dev/null +++ b/research/playback-calibration/mirrors/iem-configuration-files.html @@ -0,0 +1,350 @@ + + + + + + + + + Configuration Files + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ +
+
+ + + + +
+
+
+
+
+

Configuration Files

+

+ This guide shows how the configuration files are structured. +

+ +
+
+ + + + + + +

General Information

+

All the configuration files for the SimpleDecoder and MatrixMultiplier are using the JSON format. See https://json.org for detailed information about this format. We are aware that we introduce a breaking change by switching from Kronlachner’s configuration files to the JSON format. However, we believe that the JSON format is more future proof and more expandable, especially for the features we’ve planned to introduce in the future (e.g. MultibandDecoder). Additionally, it is easier to en- and decode, is human-readable and decent text editors will alert you, if the data is not JSON conform.

+

The idea is to have one configuration file for your whole project, including information about decoding, routing, positions of loudspeakers and additional matrix operations.

+

Structure of Configuration Files

+

The basic structure only holds name and description attributes of the configuration and looks like this:

+
{
+"Name" : "name of this configuration",
+"Description" : "description text"
+}
+

It can be extended by adding objects to the configuration. For now, only the following objects are defined:

+ +

Each of them has some more sub-objects, described below.

+

So a decoder configuration could look like this (note, that you have to add the necessary attributes to the decoder object):

+
{
+"Name" : "name of this configuration",
+"Description" : "description text",
+"Decoder" : {
+            <put decoder information here>
+            }
+}
+

There will be more objects in the future. For example definitions of loudspeakers to visualize their position; or a MultibandDecoder consisting of an array of Decoders with additional information about frequency bands.

+

The TransformationMatrix Object

+

With the TransformationMatrix object you can matrix transformation functionality to your configuration. You can define the following attributes:

+
    +
  • Name: The name of your matrix as a string. (optional)
  • +
  • Description: The description of your matrix as a string. (optional)
  • +
  • TransformationMatrix: The transformation matrix given as a two-dimensional array of floats. See example below.
  • +
+

The Decoder Object

+

With the Decoder object you can add a decoder functionality to your configuration. You can define the following attributes:

+
    +
  • Name: The name of your decoder as a string. (optional)
  • +
  • Description: The description of your decoder as a string. (optional)
  • +
  • ExpectedInputNormalization: Define the expected normalization of the Ambisonic input signal. Can be "n3d" or "sn3d".
  • +
  • Weights: Define what kind of weighting your decoder uses. Can be none, inPhase or maxrE.
  • +
  • WeightsAlreadyApplied: Define, whether the weights are already applied to the matrix (bool value: true or false).
  • +
  • SubwooferChannel : Define the subwoofer channel as an integer. If defined, the SimpleDecoder will use the discrete subwoofer mode and route the filtered input signal to that channel. (optional)
  • +
  • Matrix: The decoder matrix given as a two-dimensional array of floats. See How-to and example below.
  • +
  • Routing: Define an output routing as an array of integers. Each loudspeaker signal will be routed to the declared channel number. See example below. (optional)
  • +
+
Notes
+

Specifying the weights-attribute is necessary for the variable order functionality. The decoder plug-in will automatically apply the correct weights according to the selected Ambisonic order of the input signal. If the weights are already applied to the matrix, the decoder will extract the Ambisonic order from the matrix’ input size and internally create a matrix without the weights.

+

The LoudspeakerLayout Object

+

The LoudspeakerLayout object holds information about a loudspeaker layout, which can be loaded into the AllRADecoder plug-in, e.g. to create a decoder. You can define the following attributes:

+
    +
  • Name: The name of your loudspeaker layout as a string. (optional)
  • +
  • Description: The description of your loudspeaker layout as a string. (optional)
  • +
  • Loudspeakers: This is an array holding the list of loudspeakers.
  • +
+

Each loudspeaker within the Loudspeakers object (array) holds information about its coordinates in the spherical representation, a flag whether it is an imaginary loudspeaker (important for the AllRAD approach), a channel number for routing a signal to, and a gain value.

+

Here’s a list of these attributes:

+
    +
  • Azimuth: The azimuth angle of the loudspeaker in degrees as a float value. Front would be 0°, right -90°, left 90° (right-hand-rule).
  • +
  • Elevation: The elevation angle in degrees as a float value. Horizontal plane is at 0°, zenith at 90° and nadir at -90°. Using values higher than 90° or lower than -90° is not recommended.
  • +
  • Radius: Distance from the listening position in meters as a float value.
  • +
  • IsImaginary: Flag whether or not the loudspeaker is imaginary (bool value: true or false).
  • +
  • Channel: The channel number the signals will be routed to as an integer value.
  • +
  • Gain: A linear gain as a float value.
  • +
+
Notes
+

The AllRADecoder plug-in uses the Gain and Radius only for imaginary loudspeakers, so the gain won’t affect real loudspeakers. See the AllRADecoder Guide for more information. You can also use the AllRADecoder to export the loudspeaker layout to a configuration file.

+

How to create a Decoder Matrix

+

Next to a well designed loudspeaker layout, a decent Ambisonic decoder is essential for playing back an Ambisonic audio signal. Creating a decoder is not a trivial and there are several strategies to do so. At the IEM we use the AllRAD strategy to create our decoders. We created the AllRADecoder plug-in for designing an Ambisonic Decoder which suits your loudspeaker layout. As the name implies, it also uses the AllRAD approach. See the AllRADecoder Guide for a how-to.

+

As an alternative, the Ambisonic Decoder ToolBox is a handy tool for Matlab and Octave, which also supports that All-Round Ambisonic Decoding. You can use it to create a decoder matrix and export it with Matlab’s jsonencode.

+

Example Decoder Configuration

+

This configuration holds a Decoder object, whose matrix is of size 12x36 (output: 12 loudspeakers; input: 5th order Ambisonics). The output channel 4 is skipped by the routing, as that channel is the one the subwoofer is connected to.

+
{
+"Name" : "Produktionsstudio decoder",
+"Description" : "",
+"Decoder" : {
+        "Name" : "Produktionsstudio decoder o5",
+        "Description" : "This is a decoder for the IEM Produktionsstudio.",
+        "ExpectedInputNormalization" : "n3d",
+        "Weights" : "maxrE",
+        "WeightsAlreadyApplied" : true,
+        "SubwooferChannel" : 4,
+        "Matrix" :
+                [[0.524667,0.469572,-0.209473,0.540143,0.535468,-0.176045,-0.20627,-0.207037,0.07898,
+                0.278071,-0.171502,-0.07617,0.080337,-0.087312,-0.031127,-0.168645,0.052887,-0.068575,
+                -0.033469,0.013674,0.034073,0.018059,-0.004703,0.03349,-0.135629,-0.00736,-0.01102,
+                -0.003696,-0.006332,0.01286,0.000698,0.015034,0.000453,0.00238,0.013449,-0.038748],
+
+                [0.52467,-0.469604,-0.20946,0.540139,-0.535474,0.17603,-0.2063,-0.207068,0.078934,
+                -0.278033,0.171505,0.076174,0.080367,-0.087303,-0.031177,-0.168678,-0.052849,0.068601,
+                0.033452,-0.013671,0.034052,0.018066,-0.004701,0.033452,-0.135628,0.007376,0.011046,
+                0.003689,0.006338,-0.012852,0.000708,0.015035,0.000457,0.002368,0.013438,-0.038739],
+
+                [0.48232,9E-6,-0.046894,0.673631,5E-6,-2E-6,-0.19333,-0.042387,0.538062,
+                -6E-6,-1E-6,2E-5,-0.030678,-0.09633,-0.027138,0.356096,-1.2E-5,-1E-6,
+                2.9E-5,-6E-6,0.006201,-0.048245,-0.016251,-0.020807,0.186287,-9E-6,-2E-6,
+                1.7E-5,-6E-6,-3E-6,0.014683,-0.0051,-0.023478,0.004609,-0.013344,0.06439],
+
+                [0.85076,0.503559,-0.266537,-0.952247,-0.541567,-0.191062,-0.337222,0.256713,0.330954,
+                0.232285,0.174682,-0.082557,0.078565,0.147907,-0.038838,0.043984,-8E-6,-0.05472,
+                0.034604,0.016569,0.042964,0.003534,-0.010906,-0.036433,-0.073705,-0.028779,-6E-6,
+                -0.003308,0.00572,0.014052,0.007977,-0.014555,-0.011586,-0.006084,0.011077,0.01356],
+
+                [0.850906,-0.503624,-0.26653,-0.952374,0.54162,0.191019,-0.337233,0.256664,0.330977,
+                -0.232295,-0.174643,0.08258,0.078468,0.147914,-0.038837,0.043988,1.8E-5,0.05471,
+                -0.034603,-0.016557,0.042992,0.003586,-0.010914,-0.03641,-0.073692,0.028764,2E-6,
+                0.003306,-0.005728,-0.014047,0.007974,-0.014559,-0.011597,-0.006063,0.011066,0.013551],
+
+                [0.75186,1,-0.156013,0.078043,0.104081,-0.172615,-0.296441,-0.029433,-0.689955,
+                -0.346121,-0.035164,-0.144883,0.009708,-0.012092,0.089756,-0.071726,-0.022223,0.030376,
+                -0.005582,-0.03736,0.022544,0.002024,0.022081,0.020618,0.109559,0.011133,0.0051,
+                -0.006416,-0.001135,0.003645,0.015434,0.002047,0.023454,0.000338,-0.005592,-0.003701],
+
+                [0.751767,-0.999922,-0.156027,0.07806,-0.104104,0.172666,-0.296466,-0.029437,-0.689946,
+                0.346149,0.035161,0.144916,0.009764,-0.012099,0.08978,-0.071742,0.022225,-0.030366,
+                0.005591,0.037336,0.02256,0.002004,0.022082,0.020616,0.109586,-0.011143,-0.0051,
+                0.006428,0.001146,-0.003654,0.015432,0.002053,0.023463,0.000337,-0.005577,-0.003705],
+
+                [0.637174,0.451681,0.620473,0.452977,0.287188,0.468269,0.139043,0.468783,0.001525,
+                0.051188,0.271546,0.194653,-0.14563,0.194029,0.000703,-0.04915,0.000835,0.034811,
+                0.09865,0.001314,-0.108523,0.000868,-0.000392,-0.033847,0.00361,0.006586,0.000343,
+                0.002819,0.001608,-0.01643,-0.017683,-0.016347,-0.000279,-0.00303,0.005699,0.006292],
+
+                [0.637184,-0.451677,0.62047,0.452974,-0.287146,-0.46826,0.13902,0.468777,0.001515,
+                -0.051165,-0.2715,-0.194646,-0.145654,0.194027,0.000686,-0.049135,-0.00084,-0.034777,
+                -0.09864,-0.001314,-0.108533,0.000869,-0.000403,-0.033837,0.003608,-0.006587,-0.000337,
+                -0.002805,-0.001621,0.016428,-0.017681,-0.016347,-0.000283,-0.003032,0.005699,0.006286],
+
+                [0.638428,0.453622,0.620934,-0.453725,-0.288777,0.469057,0.138133,-0.469196,3E-6,
+                0.050871,-0.272325,0.193751,-0.146243,-0.193823,2.8E-5,0.050953,3.7E-5,0.034674,
+                -0.098291,0.000659,-0.108277,-0.000645,1.8E-5,0.034723,0.003108,0.006353,1.2E-5,
+                0.002849,-0.001321,-0.016311,-0.017459,0.016335,-6E-6,0.002865,0.005478,-0.006362],
+
+                [0.638375,-0.453609,0.620938,-0.453603,0.288705,-0.469075,0.138194,-0.469142,-6.4E-5,
+                -0.05085,0.272288,-0.193774,-0.146229,-0.19386,-5E-6,0.050949,-2.4E-5,-0.03467,
+                0.098293,-0.000651,-0.108301,-0.000667,3E-5,0.034709,0.003096,-0.006357,-1E-6,
+                -0.002853,0.001324,0.016324,-0.017463,0.016343,4E-6,0.002853,0.005472,-0.006355],
+
+                [0.413602,1.9E-5,0.610689,5.3E-5,1.9E-5,3E-5,0.565073,7.3E-5,1E-5,
+                1E-6,2.7E-5,2.6E-5,0.38914,5.4E-5,1.7E-5,-2.1E-5,2E-6,1E-6,
+                1.9E-5,1.3E-5,0.192325,1.8E-5,1.6E-5,-2.8E-5,-0.0034,3E-6,3E-6,
+                1E-6,5E-6,1E-6,0.054697,-3E-6,8E-6,-1.7E-5,-0.004302,-3E-6]],
+
+        "Routing" : [1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 12, 13]
+        }
+}
+

Example TransformationMatrix Configuration

+

This example shows a simple configuration which uses a TransformationMatrix.

+
{
+    "Name" : "Left/right flip",
+    "Description" : "This configuration contains a simple left/right flip matrix.",
+    "TransformationMatrix" : {
+                              "Name" : "Flip: left/right o5",
+                              "Description" : "This matrix mirrors a 5th order Ambisonic signal across the xy-plane (left right flip)",
+                              "Matrix" : [[1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],
+                                          [0,-1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],
+                                          [0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],
+                                          [0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],
+                                          [0,0,0,0,-1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],
+                                          [0,0,0,0,0,-1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],
+                                          [0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],
+                                          [0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],
+                                          [0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],
+                                          [0,0,0,0,0,0,0,0,0,-1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],
+                                          [0,0,0,0,0,0,0,0,0,0,-1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],
+                                          [0,0,0,0,0,0,0,0,0,0,0,-1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],
+                                          [0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],
+                                          [0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],
+                                          [0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],
+                                          [0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],
+                                          [0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,-1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],
+                                          [0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,-1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],
+                                          [0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,-1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],
+                                          [0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,-1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],
+                                          [0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],
+                                          [0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0],
+                                          [0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0],
+                                          [0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0],
+                                          [0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0],
+                                          [0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,-1,0,0,0,0,0,0,0,0,0,0],
+                                          [0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,-1,0,0,0,0,0,0,0,0,0],
+                                          [0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,-1,0,0,0,0,0,0,0,0],
+                                          [0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,-1,0,0,0,0,0,0,0],
+                                          [0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,-1,0,0,0,0,0,0],
+                                          [0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0],
+                                          [0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0],
+                                          [0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0],
+                                          [0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0],
+                                          [0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0],
+                                          [0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1]]
+                             }
+}
+
+ + +
+
+
+
+
+ +

Website created with Hugo using the kube theme.

+
+ +
+ + + + + + + + diff --git a/research/playback-calibration/mirrors/iem-plugin-descriptions.html b/research/playback-calibration/mirrors/iem-plugin-descriptions.html new file mode 100644 index 0000000..d461d4c --- /dev/null +++ b/research/playback-calibration/mirrors/iem-plugin-descriptions.html @@ -0,0 +1,247 @@ + + + + + + + + + Plug-in Descriptions + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ +
+
+ + + + +
+
+
+
+
+

Plug-in Descriptions

+

+ Here you find a list of available plug-ins in alphabetical order and their description. +

+ +
+
+ + + + + + +

AllRADecoder

+

Author Daniel Rudrich and Franz Zotter Type Decoder Input Ambisonic signals Output Audio signals

+

With the AllRADecoder you can design an Ambisonic decoder for an arbitrary loudspeaker layout using the AllRAD approach. More information can be found in the AllRADecoder Guide.

+

BinauralDecoder

+

Author Daniel Rudrich Type Binaural Decoder Input Ambisonic signals Output Binaural headphone signals

+

The BinauralDecoder renders the Ambisonic input signal to a binaural headphone signal using the MagLS approach proposed in [1]. The used HRTFs come from the Neumann KU 100 dummy head. Additionally, you can apply headphone equalizations from measurements by Benjamin Bernschütz et. al..

+

In contrast to conventional binaural decoders, the BinauralDecoder does not use virtual loudspeakers, but converts the Ambisonic signals directly to binaural headphone signals, with help of pre-processed HRTFs. They are processed in way, that the original frequency response of the HRTFs is maintained as best as possible, when used with Ambisonics.

+

[1] Schoerkhuber, Christian; Zaunschirm, Markus; Hoeldrich, Robert. “Binaural Rendering of Ambisonic Signals via Magnitude Least Squares”, Fortschritte der Akustik, DAGA, 2018

+

CoordinateConverter

+

Author Daniel Rudrich Type Tool Input N/A Output N/A

+

The CoordinateConverter converts VST parameters from a spherical representation to a cartesian, and vice versa. There’s no audio processing, just the conversion of parameters. A typical application would be the conversion of xyz-position data (e.g. in an automation) to a spherical representation (azimuth, elevation, radius) for the use of Ambisonic encoders.

+

Another example would be the usage with the RoomEncoder: With using the sphere panner (spherical coordinates) and mapping the calculated cartesian coordinates from the CoordinateConverter to the source position of the RoomEncoder, you can make a source rotate around the listener in the virtual room, maintaining the radius. For that you will need to set the radius normalization to any desired maximum value, and set x- and y-range to 15.0m, z-range to 10.0m, as these values are the parameter limits of the source’s x-, y- and z-position. Adjusting the ranges is necessary, as VST parameters are normlized to a range from 0 to 1.

+

DirectionalCompressor

+

Author Daniel Rudrich Type Dynamics Input Ambisonic signals Output Ambisonic signals

+

The DirectionalCompressor is a more sophisticated Ambisonic compressor/limiter which lets you control the dynamics for different spatial regions. Example: a singer in the front could be compressed more than the rest. You can realize a ducking effect, with which the singer’s level can control the level of everything else. +Use the sphere panner and width settings to create a spatial mask. You can listen to the masked and unmasked signals by selecting them in the combobox. There are two identical compressors available. In each you can choose between three input signals (full = omni, masked signal, and unmasked signal = the rest). You can also select the target to which the compressor gains are applied to.

+

DirectivityShaper

+

Author Daniel Rudrich Type Directivity encoder Input Mono audio signal Output Directivity signals

+

This plugin let’s you filter your input signal into four independent bands, to which different directivity patterns can be applied to. Also the bands can be spatially panned.

+

For each band, you can choose from four types of filters (all-pass, low-pass, band-pass, high-pass) blend continuously between different directivity orders, as well as different shapes: basic, maxRE and in-phase. The latter makes sure that there will be no side-lobes, at cost of a wider main-lobe. Use the sphere in the upper left to pan the individual bands. The black dot represents the probe, with which you can measure the frequency-response for that direction (white line in the filter visualizer). You can also lock the relative orientations of all bands to the probe, by enabling the “lock orientations”-button. Now every band will rotate according to the probe’s orientation. +See the DirectivityShaper Guide for a detailed manual of the plug-in.

+

DistanceCompensator

+

Author Daniel Rudrich Type Loudspeaker compensation Input Audio signals Output Audio signals

+

With the DistanceCompensator you can insert the distances from the listening position to each loudspeaker and the plug-in will calculate the needed delays and gains in order to compensate for distance differences. So this plug-in is best place at the end of your signal chain (e.g. right after the decoders).

+

Gain and delay compensation can be enabled/disabled separately and their calculation parameters can also be set. Regarding delays, the speed of sound can be specified, which is 343.2m/s (at 20°C) per default. The gains are calculated using a distance-ratio-exponent, which has a value of 1.0 assuming free field conditions. In rooms, smaller values might be more adequate due to wall reflections. Additionally, you can specify whether there should be only attenuations or using the zero-mean setting, which also will have positive gains applied to loudspeakers further away from the listener.

+

It’s also possible to load the distance information via configuration files holding a LoudspeakerLayout object (see the Configuration Files Guide for more information). Once loaded a LoudspeakerLayout, you can define the reference position to which the loudspeakers are compensated for.

+

DualDelay

+

Author Daniel Rudrich Type Delay Effect Input Ambisonic signals Output Ambisonic signals

+

The DualDelay is the plug-in-version of our RotationalDelay we first used for the Al Di Meola concert. There are two delay-lines, which can be configured independently. The most exciting feature is the rotation or warping of the whole sound field every single time it runs through the delay-line. The delay’s timbre can be shaped with the high- and low-pass filters. The output of each delay-line can be fed back to itself or/and to the other one and mixed to the output signal. Delay times can be synced to the host BPM.

+

EnergyVisualizer

+

Author Daniel Rudrich Type Visualizer Input Ambisonic signals Output Ambisonic signals

+

The EnergyVisualizer visualizes the energy distribution on the sphere of the Ambisonic input signal using a Hammer-Aitoff projection (a spherical projection which is area-preserving). The visualized dynamic range can be adjusted between 10 and 60dB, the range can be moved up or down using the slider next to the projection. The energy levels are color-coded with a perceptually motivated colormap. The used colormap is depicted next to the projection. As the Ambisonic Signal is decoded using 480 virtual loudspeakers with a subsequent RMS calculation and OpenGL visualization, the EnergyVisualizer is quite computation-intensive. Make sure you use only a few of them. The processing will be automatically deactivated when the GUI window is closed.

+

FdnReverb

+

Author Sebastian Grill Type Reverb Input Audio/Ambisonic signals Output Audio/Ambisonic signals

+

The FdnReverb is a Feedback-Delay-Network reverberation. You can use it with single audio signals but also with Ambisonic signals, leading to a spatial and temporal diffuse reverberation. You can set the overall reverberation time and refine it frequency-dependently with the help of two shelving-filters. With the fade-in control you can make the effect even more diffuse by emplyoing a second network in parallel. Make sure the fade-in time is below the overall reverberation time.

+

GranularEncoder

+

Author Stefan Riedel Type Encoder/Synthesizer Input Mono or stereo audio signals Output Ambisonic signals

+

The GranularEncoder takes a mono or stereo input signal, and encodes short ‘grains’ of audio into the Ambisonic domain. The grains are distributed around a controllable center direction, and you are able to specify the size and shape of the distribution. The ‘Mix’ parameter allows to blend between standard encoding (dry) and granular encoding (wet). This allows to use GranularEncoder either as an Ambisonic granulator (wet setting) or as an Ambisonic grain delay (mixed setting). The plug-in comes with integrated modulators for the grain parameters (cf. small sliders), which are useful to vary e.g. the grain seed ‘Position’ in the internal audio buffer.

+

To get a visual impression of the grain distribution it is recommended to use the EnergyVisualizer plug-in, placed e.g. on your Ambisonic bus before decoding. A detailed description can be found in the GranularEncoder Guide.

+

MatrixMultiplier

+

Author Daniel Rudrich Type Matrix Transformation Input Audio/Ambisonic signals Output Audio/Ambisonic signals

+

With the MatrixMultiplier you can load a configuration which contains a TransformationMatrix object (see the Configuration Files Guide) which will be applied to the input signal.

+

MultiBandCompressor

+

Author Markus Huber Type Dynamics Input Ambisonic signals Output Ambisonic signals

+

The MultiBandCompressor splits your Ambisonic signal into four bands and compresses them individually. The filters’ crossover frequencies can be adjusted to taste, and the individual compressors are fully configurable as well, making use of the parameters threshold, knee, ratio, attac, release, and make-up gain. With the master controls you can adjust the parameters of all compressors simultaneously.

+

The used filters are Linkwitz-Rilely crossovers, so the magnitude response of your mix will be retained, without introducing any phase issues. The plug-in will also retain the spatial image of your mix, by analyzing the omni-channel (W) of each band and applying the same gain reduction to all channels.

+

MultiEncoder

+

Author Daniel Rudrich Type Encoder Input Audio signals Output Ambisonic signals

+

With the MultiEncoder you can encode multiple sources with just one plug-in. Select your desired number of input sources in the upper left corner (up to 64). Every source can be panned, muted and soloed individually. Also the gain and label color can be adjusted. Enable the „Lock Directions“-button to let the sources stick to the Master-Panner, which can be controlled separately.

+

Double-click on the sphere, to switch to a linear elevation representation, to get a higher resolution at the horizon. Every ring represents 15° of elevation.

+

MultiEQ

+

Author Daniel Rudrich Type Filter Input Audio signals Output Audio signals

+

The MultiEQ is a simple multi-channel equalizer, filtering up to 64 audio channels. Each of the six filters can be switched on and of, and different filter types can be selected. The first and last filter offer additional filter types: low-/high-pass of different filter orders. The 4th order HP / LP can be used to create a Linkwitz-Riley crossover, good for splitting audio into two bands, with a constant magnitude response when added together.

+

OmniCompressor

+

Author Daniel Rudrich Type Dynamics Input Ambisonic signals Output Ambisonic signals

+

The OmniCompressor is an Ambisonic compressor which can also be used as a limiter. It works like a regular audio compressor. It uses the first input channel (W-channel, hence the name ‘Omni’) as the driving signal and applies the calculated gains to all the whole Ambisonic signal. When the ratio is set to infinity and the attack time is set to 0.0 ms, the compressor acts as a brick-wall limiter. With that setting you might want to enable the lookahead feature to avoid unwanted distortions. This introduces a delay of 5ms to your signal. There is a hidden parameter (when disabling the plug-ins GUI in your DAW) with which you can deactivate latency reporting to your DAW to compensate for it. This improves the playback performance of your project, however the delay won’t be compensated for with that setting.

+

ProbeDecoder

+

Author Daniel Rudrich Type Analysis tool/Decoder Input Ambisonic signals Output Mono audio signal

+

The ProbeDecoder is a little tool to sample/decode the Ambisonic input signal for one specific direction and listen to the output, mostly used for testing purposes.

+

RoomEncoder

+

Author Daniel Rudrich Type Encoder+Reverb Input Mono audio signal or Directivity signals Output Ambisonic signals

+

The RoomEncoder is so far the most complex and computation-intensive plug-in within the IEM Plug-in Suite. With it, you can put a source and a listener into a virtual shoebox-shaped room and render over 200 wall reflections. You can move the source and the listener freely within the room, which will automatically generate the Doppler-shift effect. Use the overall reflection coefficient and the low-shelve and high-shelve filters to change the character of the reflections. With the additional attenuation controls, you can add an additional attenuation for each of the six walls. E.g. you can get rid of the upper wall by completly turning down the ceiling attenuation.

+

But thats not all: The RoomEncoder takes DirectivitySignals (or when set to 0th order also monophonic signals) as an input signal. This means that your source can have a frequency-dependent and time-variable (e.g. changing the direction of the source) directivity. The DirectivityShaper plugin is a good tool to create various directivity patterns.

+

You can sync several RoomEncoder instances to render the same room using one of the four synchronization channels. You can also select, which parameters to sync (listener position, room size, and reflection properties).

+

SceneRotator

+

Author Daniel Rudrich Type Rotator Input Ambisonic signals Output Ambisonic signals

+

The SceneRotator is an Ambisonic rotator plug-in, which rotates the Ambisonic scene, hence the name! It can be used with Yaw-Pitch-Roll rotation data, or alternativeley with Quaternions. Each of the three rotation axes can be flipped, and aso the Quaternion data can be inverted. Rotations then happen in reversed direction. You can also select the order of rotations. Ambisonic rotation is very useful in combination with binaural playback, so head-movements can be compensated for.

+

SimpleDecoder

+

Author Daniel Rudrich Type Decoder Input Ambisonic signals Output Audio signals

+

The SimpleDecoder reads JSON configuration files and decodes the Ambisonic input signal to loudspeaker signals. You can create these configuration files with the AllRADecoder. For more information about the structure of the configuration files read the Configuration Files Guide.

+

When setting the subwoofer mode to discrete or virtual the Ambisonic input signal is filtered into two bands (using 4th order high-/low-pass). The upper band is decoded according to the loaded configuration file. If set to discrete, the lower is sent to the defined subwoofer channel. You will get a warning, if you choose a channel, which is already used by the decoder. If set to virtual, the low-pass signal is passed to all loudspeakers, creating a virtual subwoofer.

+

StereoEncoder

+

Author Daniel Rudrich Type Encoder Input Mono or stereo audio signals Output Ambisonic signals

+

Use the StereoEncoder plug-in to encode mono or stereo audio signals into the Ambisonic domain. In the upper right corner you can set your desired Ambisonic order and normalization. Make sure your bus size is big enough. Set to Auto-mode and the plug-in adapts to changes in bus size by using the highest possible order.

+

Use the Azimuth, Elevation and Roll sliders to pan your source. With the Width slider you can separate the two input channels. You can, of course, use the StereoEncoder also for mono sources. Additionally, there is a quaternion input you can use as an alternative to Azimuth and Elevation, e.g. for orientation data of (head-)trackers.

+

Use the following modifiers and the mouse-wheel to pan with the sphere panner:

+
    +
  • ⌘/CTRL : Azimuth
  • +
  • ALT : Elevation
  • +
  • CMD / CTRL + ALT : Roll
  • +
  • SHIFT : Width
  • +
+

Double-click on the sphere, to switch to a linear elevation representation, to get a higher resolution at the horizon. Every ring represents 15° of elevation.

+

There are some key-combinations to directly pan to a specific direction: SHIFT + F -> Front, SHIFT + U -> Up, SHIFT + L -> Left, and so on… you can figure out the rest :-)

+

ToolBox

+

Author Daniel Rudrich Type Ambisonic tool Input Ambisonic signals Output Ambisonic signals

+

This is a small plug-in with a growing set of handy features. For now, you can flip the Ambisonic input signal along the x, y and/or z axis. You can also use it as an adapter between plug-ins with an automatic order settings and plug-ins with fixed order settings. The LOA (lower order Ambisonics) Weighting feature lets you mix Ambisonic signals with different orders. Example: use the ToolBox with the setting 1st (input), 5th (output) and maxrE weights for a FOA microphone signal (which is LOA) to include it in your 5th order mix. Note, that the weights have to match you decoder’s weights. The gain control boosts or attenuates the overall audio level.

+ + + +
+
+
+
+
+ +

Website created with Hugo using the kube theme.

+
+ +
+ + + + + + + + diff --git a/research/playback-calibration/mirrors/ringbuffer-decoding-loudspeaker-iem.html b/research/playback-calibration/mirrors/ringbuffer-decoding-loudspeaker-iem.html new file mode 100644 index 0000000..8b1dc87 --- /dev/null +++ b/research/playback-calibration/mirrors/ringbuffer-decoding-loudspeaker-iem.html @@ -0,0 +1,162 @@ + + + + + +Decoding to Loudspeaker Setups | RingBuffer + + + + + + + + + + + + + + + + + + + + +Skip to main content + + +
+
+
+ +
+ +
+
+ +
+ +
+
+
+
+ +
+
+ + + +

Decoding to Loudspeaker Setups

+ + + + +
+
+

Creating a Decoder

+

Decoder can be created using the allrad decoder. In there the coordinats for each speaker can be set. The decoder requires a "body" meaning that a purely two dimensional speaker setup will not work. For that Imaginary speaker can be placed. Audio that is rendered on it is send to all connected speakers. To avoid this a gain factor can be applied.

+
+
+

Using a Decoder

+

Route an ambisonics signal into the decoder channel. Make sure to send all channels (and not just 1/2).

+

Allrad Decoder itself can already process and decode ambisonics. But you can also export the created decoder and speakerlayouts for use in other IEM-Plugins.

+

The "SimpleDecoder" can import decoder files. With this plugin you are able to use a discrete subwoofer without extra signal splitting and processing and also a so called virtual subwoofer which sends the low frequency audio to all speakers.

+
+
+

Distance Compensator

+

A normal ambisonics decoder does only decode the direction of the incoming soundfield. So this would only work cleanly on a perfect sphere or circle. In realworld scenario this is barely the case, the distance from the speaker to the listener might diverge leading to different level and delays of the signal at the listeners position. To compensate for that effect the "DistanceCompensator" can be used. You can load the speaker layout file into it or adjust the distance manually.

+
+
+
+
+
+Contents © Henrik von Coler 2022 - Contact +
+
+
+
+ + + + + diff --git a/research/playback-calibration/mirrors/sparta-overview.html b/research/playback-calibration/mirrors/sparta-overview.html new file mode 100644 index 0000000..dd6c71f --- /dev/null +++ b/research/playback-calibration/mirrors/sparta-overview.html @@ -0,0 +1,10 @@ +Overview - SPARTA +

Overview

SPARTA is a collection of flexible audio plug-ins for spatial audio production, editing, and playback. The plugins are developed primarily by members of the Acoustics Lab at Aalto University, Finland. However, they have also received contributions from many other researchers, developers, and enthusiasts, from around the world.

Parametric rendering

The SPARTA installer also includes the COMPASS suite, HO-DirAC suite, CroPaC Binaural Decoder, and the HO-SIRR room impulse response renderer. +These plug-ins are restricted to lower order Ambisonics input. However, they also leverage parametric processing and are adaptive, which allows them to go beyond the capabilities of conventional linear Ambisonics algorithms. In many cases, parametric rendering of first-order Ambisonics can often match, or exceed, the perceptual performance of traditional renderers operating at much higher Ambisonic orders.

All included plug-ins

SPARTA

Learn more about the SPARTA Suite →

  • sparta_ambiBIN - A binaural ambisonic decoder (up to 10th order) with a built-in SOFA loader and head-tracking support via OSC messages. Includes: Least-Squares (LS), spatial re-sampling (SPR, virtual loudspeakers), time-alignment (TA), and magnitude least-squares (Mag-LS) decoding options.
  • sparta_ambiDEC - A frequency-dependent loudspeaker ambisonic decoder (up to 10th order) with user specifiable loudspeaker directions (up to 128), which may be optionally imported via JSON configuration files. Includes: All-Round (AllRAD), Energy-Preserving (EPAD), Spatial (SAD), and Mode-Matching (MMD) ambisonic decoding options. The loudspeaker signals may be optionally binauralised for headphone playback.
  • sparta_ambiDRC - A frequency-dependent dynamic range compressor (DRC) for ambisonic signals (up to 10th order).
  • sparta_ambiENC - An ambisonic encoder/panner (up to 10th order), with support for up to 128 input channels; the directions for which may also be imported via JSON configuration files.
  • sparta_ambiRoomSim - A shoebox room simulator based on the image-source method, supporting multiple sources and ambisonic receivers.
  • sparta_array2sh - A microphone array Ambisonics encoder (up to 10th order), with presets for several commercially available A-format and higher-order microphone arrays. The plug-in can also present objective evaluation metrics for the currently selected configuration.
  • sparta_beamformer - A spherical harmonic domain beamforming plug-in with multiple beamforming strategies (up to 128 output beams).
  • sparta_binauraliser - A binaural panner (up to 128 input channels) with a built-in SOFA loader and head-tracking support via OSC messages.
  • sparta_binauraliserNF - Binauraliser, with the addition of proximity filtering for near field sound sources.
  • sparta_decorrelator - A simple multi-channel signal decorrelator (up to 128 input channels).
  • sparta_dirass - A sound-field visualiser based on re-assigning the energy of beamformers. This re-assignment is based on DoA estimates extracted from “spatially-constrained” regions, which are centred around each beamformer look-direction.
  • sparta_matrixconv - A matrix convolver with an optional partitioned convolution mode. The user need only specify the number of inputs and load the filters via a wav file.
  • sparta_multiconv - A multi-channel convolver with an optional partitioned convolution mode. Unlike “MatrixConv”, this plug-in does not perform any matrixing. Instead, each input channel is convolved with the respective filter; i.e. numInputs = numFilters = numOutputs.
  • sparta_6DoFconv - A time-varying partitioned convolution multi-channel convolver for SOFA files containing RIRs with multiple listener positions.
  • sparta_panner - A 3D loudspeaker array panner using the VBAP method (up to 128 inputs and outputs).
  • sparta_pitchshifter - A simple multi-channel pitch shifter (up to 128 channels) based on the phase vocoder approach.
  • sparta_powermap - A sound-field visualisation plug-in based on ambisonic signals as input (up to 10th order), with PWD/MVDR/MUSIC/Min-Norm options.
  • sparta_rotator - A flexible ambisonic rotator (up to 10th order) with head-tracking support via OSC messages.
  • sparta_sldoa - A frequency-dependent sound-field visualiser (up to 10th order), based on depicting the direction-of-arrival (DoA) estimates derived from spatially localised active-intensity vectors. The low frequency estimates are shown with blue icons, mid-frequencies with green, and high-frequencies with red.
  • sparta_spreader - An arbitrary array (e.g. HRIRs or microphone array IRs) panner with coherent and incoherent spreading options.

COMPASS

Learn more about the COMPASS Suite →

  • compass_binaural - An adaptive binaural ambisonic decoder (up to 3rd order input) based on the parametric COMPASS model, with a built-in SOFA loader and head-tracking support via OSC messages.
  • compass_binauralVR - Same as the compass_binaural plug-in, but also supporting listener translation around the receiver position and support for multiple simultaneous listeners.
  • compass_decoder - A parametrically enhanced loudspeaker ambisonic decoder (up to 3rd order input).
  • compass_6dof - A six degrees-of-freedom (6DoF) renderer based on multiple Ambisonic receivers as input, which supports listener translation both within and beyond the convex hull of the receiver arrangement.
  • compass_gravitator - A parametric sound-field focussing plug-in.
  • compass_sidechain - A plug-in that manipulates the spatial properties of one Ambisonic recording based on the spatial analysis of a different recording.
  • compass_spatedit - A flexible spatial editing plug-in.
  • compass_tracker - A multiple target acoustic tracker which can optionally steer a beamformer towards each target.
  • compass_upmixer - An Ambisonic upmixer (1-3rd order input, 2-7th order output).

HO-DirAC

Learn more about the HO-DirAC Suite →

  • hodirac_binaural - An adaptive binaural ambisonic decoder (up to 3rd order input) based on the parametric HO-DirAC model, with a built-in SOFA loader and head-tracking support via OSC messages.
  • hodirac_decoder - A parametrically enhanced loudspeaker ambisonic decoder (up to 3rd order input).
  • hodirac_upmixer - An Ambisonic upmixer (1-3rd order input, 2-5th order output).

Note these plug-ins are released under their own licensing terms.

Other plug-ins

  • hades_renderer - A flexible head-worn microphone array to binaural renderer.
  • cropac_binaural - An adaptive binaural first-order ambisonic decoder based on the parametric CroPaC model, with a built-in SOFA loader and head-tracking support via OSC messages.
  • HOSIRR - An Ambisonic room impulse response (RIR) renderer for arbitrary loudspeaker setups, based on the Higher-order Spatial Impulse Response Rendering (HO-SIRR) algorithm.
  • UltrasonicSuperHearing - A plug-in for the binaural reproduction of ultrasonic sound sources captured using a 6-sensor ultrasonic microphone array, which uses direction-of-arrival estimation and pitch-down-shifting prior to binauralisation in order to provide the appropriate localisation cues.

Edit this page on GitHub

+ + + \ No newline at end of file diff --git a/research/playback-calibration/summary.md b/research/playback-calibration/summary.md new file mode 100644 index 0000000..942671f --- /dev/null +++ b/research/playback-calibration/summary.md @@ -0,0 +1,83 @@ +# Playback Calibration And Loudspeaker Rendering + +## Objective + +Account for the room's actual speaker output path when rendering the AquaSynth/Ambisonic scene, so the spatial field is not merely correct inside the DSP graph but wrong in the air. + +## Current Mechanism + +The previous Ambisonic research path ends at "decoder / renderer: binaural, stereo, OBS, speakers." That is too vague. For a speaker playback rig, the renderer has to own the actual speaker layout, output-device latency, per-speaker delay/gain, and room/loudspeaker response at the listening position. + +## Main Finding + +Ambisonics is scene-based: the B-format/HOA bus is not a speaker feed. A decoder converts that scene into a specific playback layout. ICST describes the workflow as source -> encoder -> B-format master -> decoder -> speakers; the same scene can feed different decoders without changing the scene itself [ICST]. + +IEM's AllRADecoder is explicitly for arbitrary loudspeaker layouts, and IEM configuration files can store loudspeaker directions and decoder matrices [IEM AllRAD; IEM Config]. SPARTA also includes `sparta_ambiDEC`, a frequency-dependent loudspeaker ambisonic decoder with user-specified loudspeaker directions and JSON import [SPARTA]. So the playback side should use an explicit decoder configuration, not a hard-coded stereo-ish downmix. + +Impulse response measurement matters because the physical speaker chain contributes delay, level, frequency response, and room reflections. FLUX:: MiRA summarizes impulse response measurement as a way to characterize a room/venue and amplifier/loudspeaker behavior [MiRA]. For this project, we need enough measurement to align arrivals and preserve spatial imaging, not a maximal audiophile pilgrimage with incense and regret. + +## Required Playback Model + +```mermaid +flowchart TD + A["AquaSynth Ambisonic bus"] --> B["Layout-specific ambisonic decoder"] + C["Speaker geometry: x/y/z or az/el/r"] --> B + D["Output device latency"] --> E["Render alignment"] + F["Per-speaker IR measurements"] --> G["Delay/gain/EQ correction"] + B --> G + G --> E + E --> H["Physical speakers in room"] + H --> I["Listening position / performance zone"] + I --> J["Measurement feedback"] + J --> F +``` + +## Invariants + +- The ambisonic bus remains speaker-independent until the decoder stage. +- The decoder is specific to the active speaker layout. +- Speaker positions, output channel mapping, delays, trims, and correction filters are configuration, not folklore. +- Capture-side sync and playback-side alignment are separate responsibilities. +- If speakers are also used for calibration pings, acoustic echo from playback into microphones must be modeled or gated; otherwise the sync estimator may happily synchronize to its own tail. + +## Practical Design + +1. Declare one playback profile. + Store speaker count, channel order, device name, sample rate, nominal buffer size, and speaker coordinates. + +2. Measure output latency and per-speaker impulse responses. + Send a chirp/pulse through each speaker one at a time and capture it at the reference mic/listening position. Extract direct-arrival delay, level, polarity, and a short correction response. + +3. Generate or load a decoder. + Use AllRAD/IEM or SPARTA-style layout config first. Export the decoder matrix/profile if possible. Do not invent a decoder until existing tools fail a real invariant. + +4. Apply per-speaker correction after decoding. + The Ambisonic decoder outputs ideal speaker feeds. Delay/gain/EQ correction belongs after that, because it corrects the physical playback channels. + +5. Align live monitoring latency. + The live mic field, synth outputs, Faust DSP latency, decoder latency, audio driver buffer, and speaker propagation delay all contribute to when sound arrives. Keep a latency ledger so visual/OBS output and room playback can be aligned deliberately. + +6. Prevent speaker output from corrupting mic sync. + During calibration, mute/route intentionally. During live operation, the sync estimator should distinguish external voices from playback bleed when possible. At minimum, tag known outgoing calibration signals and avoid using those windows for passive drift estimates unless deliberately measuring echo. + +## Fit With Live Adaptive Sync + +The live adaptive sync system aligns microphone streams into one capture-time bus. The playback calibration system aligns rendered speaker feeds into one room-time presentation. These two machines meet at the AquaSynth graph, but they should not share a compensator. Shared compensators are where clear systems go to become soup. + +## Open Questions + +- How many speakers are wired in, and through what output device/interface? +- Are they stereo, quad, 5.1-ish, asymmetrical room speakers, or something stranger? +- What is the intended sweet spot: the operator chair, camera zone, whole room, or a performer's position? +- Will the speakers be audible to the microphones during live use? +- Does AquaSynth host the decoder/correction itself, or should the speaker renderer live outside AquaSynth after the Faust graph? + +## Citations + +- [IEM AllRAD] IEM Plug-in Suite, "AllRADecoder Guide." Mirror: `mirrors/iem-allradecoder-guide.html`. +- [IEM Config] IEM Plug-in Suite, "Configuration Files." Mirror: `mirrors/iem-configuration-files.html`. +- [IEM Plugins] IEM Plug-in Suite, "Plug-in Descriptions." Mirror: `mirrors/iem-plugin-descriptions.html`. +- [RingBuffer] RingBuffer, "Decoding to Loudspeaker Setups." Mirror: `mirrors/ringbuffer-decoding-loudspeaker-iem.html`. +- [SPARTA] SPARTA, "Overview." Mirror: `mirrors/sparta-overview.html`. +- [ICST] ICST Ambisonics, "How it Works." Mirror: `mirrors/icst-how-it-works.html`. +- [MiRA] FLUX:: MiRA, "Impulse response measurement." Mirror: `mirrors/flux-mira-impulse-response-measurement.html`. diff --git a/research/spout-obs-bridge/bibliography.md b/research/spout-obs-bridge/bibliography.md new file mode 100644 index 0000000..ea95b60 --- /dev/null +++ b/research/spout-obs-bridge/bibliography.md @@ -0,0 +1,5 @@ +# Spout OBS Bridge Bibliography + +- Off World Live, `obs-spout2-plugin`, OBS plugin implementing Spout2 input/output. Mirror: `mirrors/obs-spout2-plugin-github.html`. +- leadedge, `Spout2`, Windows video frame sharing system and SDK. Mirror: `mirrors/leadedge-spout2-github.html`. +- leadedge, Spout project home. Mirror: `mirrors/spout-home.html`. diff --git a/research/spout-obs-bridge/mirrors/leadedge-spout2-github.html b/research/spout-obs-bridge/mirrors/leadedge-spout2-github.html new file mode 100644 index 0000000..b808bd2 --- /dev/null +++ b/research/spout-obs-bridge/mirrors/leadedge-spout2-github.html @@ -0,0 +1,1822 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + GitHub - leadedge/Spout2: A video frame sharing system for Microsoft Windows · GitHub + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+
+ + +
+ Skip to content + + + + + + + + + + + +
+
+ + + + + + + + + + + + + + + + + + + +
+ +
+ + + + + + + + +
+ + + + + +
+ + + + + + + + + +
+
+
+ + + + + + + + + + +
+ +
+ +
+ +
+ + + + / + + Spout2 + + + Public +
+ + +
+ +
+ + +
+
+ +
+
+ + + + +
+ + + + + +
+ + + + + + + + + + + + + + + + + + +
+ + + + +

leadedge/Spout2

+ + + +

Repository files navigation

Spout is a video frame sharing system for Microsoft Windows, which allows applications to share OpenGL textures in a similar way to Syphon for the Mac.

+

Spout supports DirectX 9, DirectX 11, DirectX 12 and OpenGL textures and includes a developer SDK and sample applications.

+

For questions and suggestions please contact the Spout Discourse Group. Please reserve repository Issues for code development.

+

Documentation

+ +

Further pdf documents and examples can be found in the relevant folders.
+For general questions please visit the Spout Discourse group

+

Pull requests

+

Please make all pull requests from the beta branch.

+

Donations

+

Your support will help the project to continue and develop.
+Become a GitHub Sponsor and show your support.
+Or make a donation :
+

+
+ +
+
+
+
+

About

+ +

+ A video frame sharing system for Microsoft Windows +

+ +

Topics

+ + +

Resources

+ + + +

License

+ + + + + + + + + + + + + + + +

Stars

+ + +

Watchers

+ + +

Forks

+ + + + +
+ +
+
+ + + + + + + +
+
+ + + +

+ Packages +

+ + +
+
+
 
+
+
+
+
 
+
+
+
+
 
+
+ + + + +
+
+
+ + + + + +
+
+ + +

+ Contributors

+ +
    +
  • +
    +
  • +
  • +
    +
  • +
  • +
    +
  • +
+ + +
+
+
+ + +
+
+

Languages

+
+ + + + + +
+ + +
+
+ + +
+
+
+ + + + +
+ +
+ +
+
+ +
+ +
+

Footer

+ + + + +
+
+ + + + + © 2026 GitHub, Inc. + +
+ + +
+
+ + + + + + + + + + + + + + + + + + + + +
+
+
+ + + diff --git a/research/spout-obs-bridge/mirrors/obs-spout2-plugin-github.html b/research/spout-obs-bridge/mirrors/obs-spout2-plugin-github.html new file mode 100644 index 0000000..6f88289 --- /dev/null +++ b/research/spout-obs-bridge/mirrors/obs-spout2-plugin-github.html @@ -0,0 +1,1821 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + GitHub - Off-World-Live/obs-spout2-plugin: A Plugin for OBS Studio to enable Spout2 (https://github.com/leadedge/Spout2) input / output · GitHub + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+
+ + +
+ Skip to content + + + + + + + + + + + +
+
+ + + + + + + + + + + + + + + + + + + +
+ +
+ + + + + + + + +
+ + + + + +
+ + + + + + + + + +
+
+
+ + + + + + + + + + + + + +
+ +
+ +
+ +
+ + + + / + + obs-spout2-plugin + + + Public +
+ + +
+ +
+ + +
+
+ +
+
+ + + + +
+ + + + + +
+ + + + + + + + + + + + + + + + + + +
+ + + + +

Off-World-Live/obs-spout2-plugin

+ + + +

Repository files navigation

Spout2 Plugin for OBS Studio (64bit)

+

This plugin enables the import and export of shared textures at high resolution to and from SPOUT2 compatible +programs.

+

Why

+

Previously the only way to import shared textures from SPOUT was via the DirectShow SpoutCam interface or by screen-capturing +the full-screen output of the SpoutReceiver program.

+

The SpoutCam is limited to standard webcam resolutions and capped at 1920x1080 and capturing the SpoutReceiver is both +inefficient and limited by your current screen resolution.

+

Previously, there was no way of outputting Spout video textures from OBS.

+

This plugin implements the SPOUT2 SDK, creates an OBS Source from the SPOUT shared texture and a Spout output which sends the content of the OBS canvas to Spout.

+

Please see installation and usage guide here

+

Acknowledgements

+

Thanks to the developer of OBS-OpenVR-Input-Plugin whose source +helped greatly in getting my head around the OBS API.

+

Thanks to the OBS team and their discord channel

+

Thanks to the authors of SPOUT for the library and clear documentation

+

Installation

+
    +
  • Go to the Releases Page
  • +
  • Download the windows installer: OBS_Spout2_Plugin_Installer.exe
  • +
  • Run the installer (accepting installation from untrusted source)
  • +
  • Select the OBS directory if not the default install location
  • +
+
+

N.B there are no current plans for 32bit builds, although theoretically this should be possible

+
+

Contributing / Building

+
    +
  • Clone this repo recursively
  • +
+
git clone --recursive git@github.com:off-world-live/obs-spout2-plugin
+
+
    +
  • Install CMAKE min version 3.28
  • +
  • Either configure and generate through the CMAKE Gui or through the command line for the windows-x64 architecture
  • +
  • Run Configure, Generate and then Open Project in the CMake Gui
  • +
+

Building a release locally

+
    +
  • Open git bash or similar bash terminal interpreter
  • +
  • Run ./scripts/Release.sh <version number>
  • +
  • You should find the executable (installer) and zip file in the main win-spout directory
  • +
+

Building the windows installer

+ +

Pull Requests welcome!

+

Contributors

+

Thanks to everybody that submitted bug tickets and in particular the code contributors:

+ +

Roadmap

+
    +
  • Improve CMakeLists.txt to copy Spout.dll automatically (thanks to @shugen002)
  • +
  • Spout Output
  • +
  • Spout Filter Output
  • +
+

License

+

This plugin authored by Campbell Morgan is Copyright Off World Live Ltd, 2019-2021 and licenced under the GPL V.2.

+
+ +
+
+
+
+

About

+ +

+ A Plugin for OBS Studio to enable Spout2 (https://github.com/leadedge/Spout2) input / output +

+ + +

Topics

+ + +

Resources

+ + + +

License

+ + + + + + + + + + + + + + + + +

Stars

+ + +

Watchers

+ + +

Forks

+ + + + +
+ +
+
+ + + + + + + + + + + +
+
+ + +

+ Contributors

+ +
    +
  • +
    +
  • +
  • +
    +
  • +
  • +
    +
  • +
+ + +
+
+
+ + + + + +
+
+
+ + + + +
+ +
+ +
+
+ +
+ +
+

Footer

+ + + + +
+
+ + + + + © 2026 GitHub, Inc. + +
+ + +
+
+ + + + + + + + + + + + + + + + + + + + +
+
+
+ + + diff --git a/research/spout-obs-bridge/mirrors/spout-home.html b/research/spout-obs-bridge/mirrors/spout-home.html new file mode 100644 index 0000000..050268d --- /dev/null +++ b/research/spout-obs-bridge/mirrors/spout-home.html @@ -0,0 +1,102 @@ + + + + + Spout + + + + + + +
+ + + +
+ +
+

Spout

+

+ Spout + is a system for Microsoft Windows for GPU accelerated sharing
+ of graphics between applications in a similar way to Syphon for OSX.
+ Spout supports DirectX 9, DirectX 11 and OpenGL textures.
+ A comprehensive SDK can be used for creating applications.
+ Explore the other Spout repositories for more detail.
+ Spout setup or the Spout SDK can be downloaded below. +

+ +

Download

+ +


+ +

Open source projects

+

+ As a convenient start, either for development or for practical use
+ explore a sorted list of open source projects that use Spout. +


+ +

SpoutCam

+ +

"SpoutCam" is a virtual webcam that is also a Spout receiver. A setup program provides options for frame rate and resolution and can be used to register for both 32 bit and 64 bit host programs. +

+ Download the latest release from the repository Releases page below.
+ Refer to the instructions and "readme.txt" in the archive for further details. +

+ +

Download

+
+ +

Spout To NDI

+

+ Spout to NDI is a set of programs that allow Spout senders and receivers to share video and audio by way of a network using the NDI (“Network Device Interface”) protocol, a video connectivity standard that enables multimedia systems to communicate with one another over IP. Options are available for Spout to NDI to capture system audio and send it together with image data and for NDI to Spout to receive audio from the NDI source as well as the graphics and play it through the speakers. Conversion from GPU textures is done using Openframeworks shaders by way of the ofxNDI addon. Unzip the distribution file to any convenient folder.
+ +

Download

+
+ +

Spout To OMT

+

+ Spout to OMT is a set of programs that allow Spout senders and receivers to share video by way of a network using the OMT (“Open Media Transport”) protocol. This is an open-source network protocol for high performance, low latency video over a local area network. Conversion from GPU textures is done using high speed compute shaders for minimum overhead. Unzip the distribution file to any convenient folder.
+ +

Download

+ +


+ +

Contact and support

+

+ Please visit the Spout Discourse group. +


+ +

Donations

+

Spout is open source software and there is no company or income to assist ongoing development, maintenance and support. So please consider becoming a supporter to help keep the project going. All contributions are most welcome.

+ + Show your support and become a Sponsor through GitHub

+ or

+ + Make a donation through PayPal
+ + +
+ + + + +
+ +


+ +



+ +
+ +
+ + + diff --git a/research/spout-obs-bridge/summary.md b/research/spout-obs-bridge/summary.md new file mode 100644 index 0000000..91ee38c --- /dev/null +++ b/research/spout-obs-bridge/summary.md @@ -0,0 +1,7 @@ +# Spout OBS Bridge Summary + +Spout is the Windows-local GPU texture sharing bridge for this rig. The OBS path is the Off World Live OBS Spout2 plugin, which creates an OBS source from a Spout shared texture and can also send the OBS canvas back to Spout. The Spout2 SDK is the application-side sender/receiver surface; its modern path centers on DirectX shared textures and has native DirectX/OpenGL integration options. + +Decision: Mimir should not render final pixels itself. It should publish typed render-frame packets from the fusion cache. Aquarium Engine should consume those packets, render them into a GPU render target, and publish that texture as a named Spout sender. OBS then receives that sender through the OBS Spout2 plugin. + +Latency decision: the render packet carries both source timestamps and intended presentation time. The stream compositor can buffer visual frames until they align with the ambisonic audio field's presentation timeline. Latency is allowed; unbounded drift is not. diff --git a/research/visual-spatial-map/bibliography.md b/research/visual-spatial-map/bibliography.md new file mode 100644 index 0000000..640a68b --- /dev/null +++ b/research/visual-spatial-map/bibliography.md @@ -0,0 +1,62 @@ +# Bibliography: Visual Spatial Map And Splat Scene + +Mirrored copies live under `research/visual-spatial-map/mirrors/`. + +## Camera Hardware / Capture + +- [PS3 Developer Wiki, "PlayStation Eye"](https://www.psdevwiki.com/ps3/PlayStation_Eye) + Mirror: `mirrors/ps3-playstation-eye-devwiki.html` + +- [PS2 Developer Wiki, "EyeToy"](https://www.psdevwiki.com/ps2/EyeToy) + Mirror: `mirrors/ps2-eyetoy-devwiki.html` + +- [iPiSoft Wiki, "User Guide for Multiple PlayStation Eye Cameras Configuration"](https://wiki.ipisoft.com/index.php?title=User_Guide_for_Multiple_PlayStation_Eye_Cameras_Configuration_%28printable_version%29&useskin=cleanmonobook) + Mirror: `mirrors/ipisoft-multiple-playstation-eye-cameras.html` + +- [opentrack Wiki, "PS3 Eye open driver instructions"](https://github.com/opentrack/opentrack/wiki/PS3-Eye-open-driver-instructions/54df68ead70a4069d24bc96f987feb1f57f1b25a) + Mirror: `mirrors/opentrack-ps3-eye-open-driver-instructions.html` + +- [Zadig](https://zadig.akeo.ie/) + Mirror: `mirrors/zadig-home.html` + +- [Code Laboratories, "CL Eye Platform Driver"](https://codelaboratories.com/products/eye/driver/) + Mirror: `mirrors/code-laboratories-cl-eye-driver.html` + +- [DelanClip, "PS3 Eye Camera Head Tracking - OpenTrack - Open Driver"](https://delanclip.com/manuals/ps3-eye-camera-head-tracking/) + Mirror: `mirrors/delanclip-ps3-eye-open-driver.html` + +- [Leap Motion, `leapuvc`](https://github.com/leapmotion/leapuvc) + Mirrors: `mirrors/leapuvc-github.html`, `mirrors/leapuvc-readme.md`, `mirrors/leapuvc-manual.pdf` + +- [Ultraleap, "Images"](https://docs.ultraleap.com/api-reference/tracking-api/leapc-guide/images.html) + Mirror: `mirrors/ultraleap-leapc-images.html` + +- [Ultraleap, "Images Example"](https://docs.ultraleap.com/api-reference/tracking-api/examples/images-example.html) + Mirror: `mirrors/ultraleap-images-example.html` + +## Multi-Camera Calibration / Tracking Geometry + +- [OpenCV, "Multi-camera Calibration"](https://docs.opencv.org/3.4/d2/d1c/tutorial_multi_camera_main.html) + Mirror: `mirrors/opencv-multi-camera-calibration.html` + +- [Multi-camera Calibration documentation](https://multi-camera-calibration.readthedocs.io/en/latest/) + Mirror: `mirrors/multi-camera-calibration-docs.html` + +- Florian Jacob and Reinhold Haeb-Umbach, ["Absolute Geometry Calibration of Distributed Microphone Arrays in an Audio-Visual Sensor Network"](https://arxiv.org/abs/1504.03128), arXiv:1504.03128, 2015. + Mirror: `mirrors/krekovic-2015-audio-visual-geometry-calibration.pdf` + +## Gaussian Splatting / Dynamic Scene Rendering + +- Bernhard Kerbl, Georgios Kopanas, Thomas Leimkuehler, and George Drettakis, ["3D Gaussian Splatting for Real-Time Radiance Field Rendering"](https://arxiv.org/abs/2308.04079), SIGGRAPH 2023. + Mirror: `mirrors/kerbl-2023-3d-gaussian-splatting.pdf` + +- Guanjun Wu et al., ["4D Gaussian Splatting for Real-Time Dynamic Scene Rendering"](https://openaccess.thecvf.com/content/CVPR2024/papers/Wu_4D_Gaussian_Splatting_for_Real-Time_Dynamic_Scene_Rendering_CVPR_2024_paper.pdf), CVPR 2024. + Mirror: `mirrors/wu-2024-4d-gaussian-splatting.pdf` + +- Zhexi Peng et al., ["RTG-SLAM: Real-time 3D Reconstruction at Scale using Gaussian Splatting"](https://arxiv.org/abs/2404.19706), SIGGRAPH 2024. + Mirror: `mirrors/peng-2024-rtg-slam.pdf` + +## Ambisonic View / Pose Coupling + +- [IEM Plug-in Suite, "SceneRotator Guide"](https://plugins.iem.at/docs/scenerotator/) + Mirror: `mirrors/iem-scenerotator-guide.html` diff --git a/research/visual-spatial-map/brushstroke-fusion-rendering.md b/research/visual-spatial-map/brushstroke-fusion-rendering.md new file mode 100644 index 0000000..e8bf4f9 --- /dev/null +++ b/research/visual-spatial-map/brushstroke-fusion-rendering.md @@ -0,0 +1,202 @@ +# Brushstroke Fusion Rendering Approach + +## Objective + +Fuse two PS3 Eyes, two Razer Kiyo-class RGB/mic cameras, one Leap Motion/LeapUVC stereo IR sensor, microphones, and speakers into one live spatial state, then render it as a volumetric brushstroke scene with a virtual camera/listener pose that also drives the ambisonic response. + +## Current Mechanism + +The visual-spatial map research established the required world authority: every sensor, emitter, prop, and virtual camera/listener must publish into one canonical coordinate system. The EpiphanyAquarium `fractal-domains-cache-that-bites.md` note adds the render architecture lesson: + +```text +semantic intent +-> spatial domain +-> field grammar +-> ownership tree +-> conservative summaries +-> contribution cache +-> backend packets +-> renderer +``` + +For Mimir, the equivalent is: + +```text +sensor observations +-> world-space tracks and calibration domains +-> semantic scene claims +-> confidence/ownership tree +-> conservative summaries +-> contribution cache +-> brushstroke / splat packets +-> live renderer + ambisonic pose +``` + +## Invariants + +- The world model owns truth, not the brushstroke renderer. +- Every observation is timestamped and carries sensor identity, transform, confidence, and latency. +- Camera, microphone, speaker, prop, and listener coordinates live in one frame. +- Render packets are disposable cached lowerings of scene claims. +- Audio response follows the same virtual listener/camera pose as the visual render. +- Sparse tracking must work before dense beauty rendering earns its keep. + +## Fusion Ownership + +```mermaid +flowchart TD + A["PS3 Eyes"] --> B["high-rate marker detections"] + C["LeapUVC stereo IR"] --> D["near-field hands / markers / depth hints"] + E["Kiyo / Kiyo Pro RGB"] --> F["RGB frames + feature tracks"] + G["Microphones"] --> H["audio source / sync observations"] + I["Speakers"] --> J["known playback + echo paths"] + B --> K["World Model Authority"] + D --> K + F --> K + H --> K + J --> K + K --> L["Semantic scene claims"] + L --> M["Brushstroke packet cache"] + L --> N["Ambisonic scene / listener pose"] + M --> O["Volumetric brushstroke renderer"] + N --> P["AquaSynth spatial output"] +``` + +The fusion stack should use a state-estimation model with explicit tracks: + +- `Sensor`: calibrated device, intrinsics/extrinsics, clock model, latency model. +- `Observation`: timestamped 2D/3D evidence from a sensor. +- `Track`: world-space entity estimate with covariance/confidence. +- `Claim`: semantic scene contribution, such as marker, prop, person silhouette, surface patch, speaker, microphone, or audio source. +- `Domain`: local coordinate frame that owns recurring detail or brush behavior. +- `RenderPacket`: cached brush/splat/impostor representation of claims. + +The tracker can start simple: per-marker Kalman filters or an EKF/UKF once camera geometry and nonlinear projection need it. Factor-graph/SLAM machinery can come later if the rig starts moving or the calibration is solved jointly. + +## Sensor Roles + +- PS3 Eyes: high-rate low-resolution marker observations. Best for bright balls, retroreflective dots, LEDs, wand tracking, and fast prop motion. +- LeapUVC: near-field stereo IR observations. Best for hands/control surface, close markers, and calibration events in its view. LeapUVC mode is generic UVC and should not be assumed to coexist with Ultraleap hand tracking. +- Kiyos: RGB texture, object appearance, human-readable frames, slower feature tracking, and microphone attachment transforms. +- Microphones: audio observations, source localization evidence, and sync residuals. +- Speakers: known emitted signals for transfer-path estimation and room calibration. + +## Brushstroke Render Model + +Dreams-style volumetric brushstrokes fit better as a **semantic splat/brush backend** than as the canonical map. + +Each visible scene claim lowers into brush packets: + +```text +BrushPacket { + stableKey + sourceClaim + worldFrame + envelope + strokeAxis / tangent frame + radius / anisotropy + opacity + color source + temporal confidence + motion blur / trail decay + material tags + lodRange + costTier +} +``` + +Packet sources: + +- tracked marker trails become luminous volumetric strokes +- sparse point tracks become soft surfels or wisps +- Kiyo RGB projections color nearby strokes/points +- Leap near-field depth/IR becomes dense local hand/prop strokes +- audio source estimates become optional visible aura/field strokes +- speaker/mic calibration paths can render as debug-only transfer wisps + +This is where the Aquarium cache discipline applies. The renderer should get a selected cut of brush packets, not the whole history of every sensor trace. Parent summaries render when children are stale or absent. + +## Contribution Cache + +Each track/claim carries conservative summaries: + +```text +stableKey +world bounds +last observation time +confidence +velocity bound +projected footprint +color availability +audio relevance +estimated render cost +fallback brush packet +``` + +Contribution score should include both visual and audio importance: + +```text +visualScore = projectedCoveragePx * confidence * freshness +audioScore = listenerRelevance * sourceEnergy * localizationConfidence +motionScore = velocity / trackingUncertainty +score = max(visualScore, audioScore, motionScore) / estimatedCost +``` + +That gives the renderer permission to spend detail where the virtual camera sees it, where the listener hears it, or where uncertainty/motion needs more frequent updates. + +## Volumetric Brushstroke Pipeline + +1. Calibrate all sensors into the world frame. +2. Ingest timestamped observations into ring buffers. +3. Estimate tracks and confidence. +4. Convert tracks/surfaces/sources into semantic claims. +5. Lower claims into brush packets under budget. +6. Render brush packets as anisotropic camera-facing or view-stable volumetric splats. +7. Project Kiyo RGB onto brush packets using camera poses. +8. Feed virtual camera/listener pose into the ambisonic renderer. + +The first renderer can be points/surfels with fat translucent strokes. A proper Dreams-like look can then add: + +- anisotropic ellipsoid splats +- temporal stroke trails +- stroke clustering by domain +- RGB projection and confidence blending +- procedural texture/noise in stroke-local coordinates +- LOD summaries for distant or stale tracks + +## Audio Coupling + +The visual camera pose should define or link to the audio listener pose: + +- yaw/pitch/roll rotates the ambisonic scene +- translation changes source distances, occlusion candidates, and room response estimates +- tracked props can drive synth patch positions +- visible speaker/mic locations constrain playback and capture calibration + +Do not make the ambisonic renderer read brush packets. It should read world claims: sources, listener pose, speaker/mic transforms, and confidence. Brush packets are how those claims are seen, not what they are. + +## First Slice + +Build the minimum machine that proves the spine: + +1. Enumerate the two PS3 Eyes, two Kiyos, and LeapUVC mode. +2. Calibrate camera intrinsics/extrinsics. +3. Track one visible marker in at least two cameras. +4. Triangulate a world-space point with confidence. +5. Render that point as a volumetric brushstroke with a stable key. +6. Move the virtual camera and verify visual parallax. +7. Bind that same point to a mono synth source and verify ambisonic position changes with listener pose. + +Only after that should RGB projection, dense splats, and dynamic point-cloud beauty work begin. + +## Cut Line + +Cut any design where: + +- Gaussian/brush packets become the source of truth. +- Leap, PS3 Eyes, and Kiyos each maintain separate private coordinate systems. +- camera-attached microphones are treated as synchronized because they are physically attached. +- the renderer owns calibration. +- ambisonics reads rendered geometry instead of semantic world claims. + +Pretty volumetric paint is allowed. Pretty volumetric paint with no ownership model is just smoke with a GPU budget. diff --git a/research/visual-spatial-map/mirrors/code-laboratories-cl-eye-driver.html b/research/visual-spatial-map/mirrors/code-laboratories-cl-eye-driver.html new file mode 100644 index 0000000..44861bf --- /dev/null +++ b/research/visual-spatial-map/mirrors/code-laboratories-cl-eye-driver.html @@ -0,0 +1,476 @@ + + + +CL - About CL Eye Platform - Code Laboratories + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + + +
+ + + + + + + +
+ + + + +
+
+ +
+
+ +
+
+
+ + + + + +
+
+
+
    +
  • + + + + + +
  • +
    + CL Eye Platform Driver +
    +
      + The CL Eye Platform Driver provides users a signed hardware driver which exposes supported cameras to third party applications such as Adobe Flash, Skype, MSN or Yahoo for video chat or conferencing. Two modes allow basic compatibility mode and a advanced mode for high frame rate development use. + + +
    + +
    +
    + + + + + + + +
    +
  • + + + + + + +
  • + + + +
  • +
    + Features +
    +
      +
    • Driver License
      Single camera support for personal non-commercial use.

    • +
    • Camera Controls
      Fully configurable settings such as resolution, exposure and gain.

    • +
    • Chat/Communication Usage
      Rich video chat applications (Adobe FlashAIM/Skype/MSN)

    • + +
    • Dual Operation Modes
      Offers two modes for basic and advanced usage.

    • + +
    • Testing Application
      A platform utility for running baseline audio and video tests.

    • +
    • Web Video Streaming
      Supports for web based video chat applications.

    • + + + + +
    + +
    +
    +
  • + + + +
  • + + +
  • +
    + Hardware Supported +
    + + Supported Applications +
    +
      +
    • AOL Aim

    • +
    • Adobe Flash™

    • +
    • - www.cameroid.com

    • +
    • - www.ustream.com

    • +
    • - www.youtube.com

    • +
    • MSN Messenger

    • +
    • Skype Video

    • +
    • View more...
    • + +
    + Software Modes +
    +
      +
    • Basic Mode
      Featuring frame rates optimized for compatibility with chat usage.

    • +
    • Advanced Mode
      Featuring full range of supported frame rates for research usage.

    • + + +
    +
    +
  • + + + +
  • + + +
  • +
    + + + Technical Specifications +
    +
      +
    • Microsoft Windows XP, Vista, 7
    • +
    • Support for 32/64 bit Windows
    • +
    • 15-125 FPS Capture Frame Rate
    • +
    • 320x240/640x480 Resolution
    • +
    • High Quality Uncompressed Image
    • +
    • Microsoft DirectShow™ Support
    • +
    • Signed USB 2.0 Driver


    • +
    • Windows Installer Application


    • +
    • Basic and Advanced Modes


    • +
    + +
    + +
    + + + + + +
    +
  • + + + +
+
+ +
+
+ + + + + + + + + + + + + + + diff --git a/research/visual-spatial-map/mirrors/delanclip-ps3-eye-open-driver.html b/research/visual-spatial-map/mirrors/delanclip-ps3-eye-open-driver.html new file mode 100644 index 0000000..524de27 --- /dev/null +++ b/research/visual-spatial-map/mirrors/delanclip-ps3-eye-open-driver.html @@ -0,0 +1,460 @@ + + + + + + + +PS3 Eye Camera Head Tracking - OpenTrack - Open Driver + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +

PS3 Eye Camera Head Tracking Setup with OpenTrack

No CL-Eye driver needed! Follow our guide to enjoy immersive and smooth gaming!

DELANCLiP Head Tracking Setup Manuals
+ + + +

PS3 Eye open driver Installation

This tutorial was created using a modified PS3 Camera which means the camera has an IR filter removed and floppy disk film applied for best IR tracking experience.

+

If you have unmodified PS3 Camera, the results will not be that good or even completly prevent you form using it.

+

If you purchased the camera from us, it means you have the modified camera and there is no need to attach any filters on top of the lens.

+
  • Step 1: PS3 Eye Camera preparation

      +
    1. Before anything – please remove any existing PS3 Camera drivers from your PC by following this guide
    2. +
    3. Rotate the camera lens anticlockwise to set the correct focus. (this is to zoom out view for better ir tracking)
    4. +
    5. If you decide to use unmodified camera, make sure you have already fitted the 2 pieces of the transparent coloured film enclosed with the Delanclip on the top of the lens of your camera. Just to give you an idea – In this example we used a very tiny drops of super glue and it did the job perfectly.
    6. +
    7. Plug in the camera to a free USB port of your Motherboard. It is best not to use USB hubs as they may lead to issues with driver installation. USB extension cable should be fine inf you need it.
    8. +
    9. Wait for the Windows to automatically install new device called “USB Camera-B4.(xxxx)” if you dont see this name in the device list – read this
    10. +
    11. Please note, on Windows 11 you will not see the pop up window like on the screenshot.
    12. +
    13. Attach the camera to your monitor using PS3 Monitor Clip (or if you dont have it, use a hook and loop also provided in the packgae).
    14. +
    15.  Verify that the camera is listed in device manager +
        +
      • open device manager by pressing Windows tiles and start typing: “Device”
      • +
      • You will see the Device Manager at the top of the window
      • +
      • The camera should appear in Other devices and identified itself as USB Camera-B4.(xxxx)
      • +
      • If you dont see this, then you need to uninstall the PS3 camera drivers that manmy times until you see the “USB Camera-B4.(xxxx)” in your device list.
      • +
      +
    16. +
    +
    DELANCLiP Head Tracking Setup Manuals

    Problems with installing PS3 Eye Camera in your Windows system?

    +

    Read our guide about how to troubleshoot PS3 camera driver issues.

    +
  • Step 2: Install the libusbK Development Kit

      +
    1. Important! you must see the camera in the device list identified as “USB Camera-B4.09.24.1” with exclamation mark – otherwise you will waste your time going forward.
    2. +
    3. PS3 eye camera shown in device manager with exclamation mark - which is ok for installing OpenPS Eye driver
    4. +
    5. Download the UsbK  Development Kit + +
    6. +
    7. Now very importatn instalation proceduere, make sure you will do everything exactly as we say here, otherwise it will not work.
    8. +
    9. Double click on the downloaded file and press Next button
    10. +
    11. On the following screen select “compact installation” from the dropdown list.
    12. +
    13. On the next screen check the first 5 boxes as per our screenshot
    14. +
    15. Follow with next button and finalize the installation.
    16. +
    17. You will immediately see a new window popup. Follow to the next step – number 3
    18. +
    +
    a screenshot of a computer program
  • Step 3: The USB Inf Creator / Installer

      +
    1. Select libusbK from the Kernel Driver Packages and press next
    2. +
    3. Now you should see your camera listed in the Device Selection window. If you cannot see it, that mans your USB drivers may be corrupted. You need to make sure that there is no other driver isntalled for PS3 camera in your system.
    4. +
    5. Device configuration – just press next, nothing to change here.
    6. +
    7. Select “Instal Only” and press next
    8. +
    9. Now the driver is installing, be patiend and wait until it finishes its job
    10. +
    11. Success! The driver was installed correctly and you can press finish and proceed to Opentrack installation below.
    12. +
    +

PS3 Camera Head Tracking Setup with OpenTrack

Now is the time to setup your OpenTrack Software utilising the PS3 Camera!

+
  • DelanCam Head Tracking with Opentrack

    Step 4: Opentrack Installation

      +
    1. Please note – your opentrack version needs to be at least: 2023.3.0 so please make sure you used the version we provide here to be able to use Open PS3 Driver!
    2. +
    3. Download Head Tracking Software and profiles: + +
    4. +
    5. Save these files in any easy accessable folder for you
    6. +
    7. Install opentrack (very easy – just follow the installation wizard)
    8. +
    +
  • Step 5: Custom Profiles Directory

      +
    1. Run opentrack software
    2. +
    3. Navigate to the profile dropdown menu
    4. +
    5. Open configuration directory
    6. +
    +
    a screenshot of a computer
  • Step 6: Adding an INI Profile

      +
    1. We recommend to use 7zip and double click to open the Profiles Zip file.
    2. +
    3. Now you can easily copy by using drag and drop from the downloaded ZIP to the opentrack configuration folder.
    4. +
    5. Copy both files if  you ever need to use face tracking – profile will be there for you.
    6. +
    +
  • Step 7: Selecting Profile

      +
    1. Go back to the opentrack main window
    2. +
    3. Select profile according to your product: +
        +
      • Delanclip Fusion – USB-C.ini (Latest release of our flagship product)
      • +
      • Delanclip Fusion – Micro USB.ini (Purchased before 2025)
      • +
      • Delanclip Starter – Metal Clip.ini (Also found in Gamer edition)
      • +
      +
    4. +
    5. Excellent! Move to the next point now!
    6. +
    7.  
    8. +
    +
  • Step 8a: Attaching the Delanclip Starter

      +
    1. Here is a short video explaining how to attach Delanclip on your Headphones.
    2. +
    3. It shows the Delanclip Starter, but the same principal is for the Delanclip Fusion with the difference that you do not bend it.
    4. +
    +
  • Step 8b: Attaching the Delanclip Fusion

    Delanclip Fusion Pro attachment watch this short video to become a pro. Same principle applies to the Delanclip Starter.

    +

    But here’s the secret: the placement shown is just a starting point!

    +

    Get creative, experiment with different velcro button positions, and find your own perfect fit for maximum enjoyment.

    +
  • Final Software Tweaks

    All the below operations must be done live while watching the on screen results.

    +
      +
    1. Turn on Delanclip Fusion (or plug in the USB of the Delanclip Starter) and wear it on your head.
    2. +
    3. Press “Start” on the Opentrack main window (step 1 on  the screenshot). +
        +
      1. If you experiance issues with error message: “Error occured while loading tracker PointTracker 1.1  Failed to open camera“
      2. +
      3. Make sure your Camera is not blocked in Windows – here is how: How to Unlock Camera in Windows
      4. +
      +
    4. +
    5. Point the Delanclip towards the camera, and hit the center button (default to HOME on keyboard)
    6. +
    7. On the input (step  2 on the screenshot) – click the small hammer icon to access the options window.
    8. +
    9. You should now see a very similar picture as shown in the screenshot.
    10. +
    11. Adjust the threshold to fine-tune the points. This step is only required if your tracking points are faded or barely visible.
    12. +
    13. You can also adjust the camera focus if your extracted points are faded or not bright enough by turning the lens. PS3 camera is a little tricky as it clicks to a set focus points. You can try to go somwhere in the middle – the lens will stay if you do it correctly – you may get much better results.
    14. +
    15. When you reached the stable point tracking then you are redy to move on.
    16. +
    17. Leave Opentrack running and start your game.
    18. +
    19. You are now able to look around using Delanclip.
    20. +
    21. If you wish to center the view, press the “HOME” key.
    22. +
    23. Enjoy!
    24. +
    +
    DelanCam Head Tracking with Opentrack

In depth videos

The below videos will help you to understand more in depth tweaking of the software and hardware. Highly recommended to watch!
Above video: How to fine tune movements translation (sensitivity)

For deeper insights into the mappings and advanced settings, we’ve created a comprehensive video below.

+

While these settings may not be crucial for beginners, you’re welcome to check it out if you feel like you need more information – watch the video below:

+
Above video: DelanCam1 focus adjustment for best IR point extraction. You can try to do the same with your PS3 camera, the lesn is rotatable and you may find a sweet spot for your need.

This step is not necessary step but if your camera is out of focus here is what you need to do!
+ + + +

tahts all - enjoy!

You are now ready to play!

The above steps are enough to get you into the head tracking World. +There are more advanced settings available in the opentrack software. They allow configuring more than we covered above. We do recommend to go to the advanced settings when you get familiar with the software. + +

Dont forget to check out our social media below!

There is also available more in-depth manual for setting up OpenTrack with your head tracker on the link below:

+

A complete guide to set up Head-tracking (Opentrack)

+
+ + +
+ +
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/research/visual-spatial-map/mirrors/iem-scenerotator-guide.html b/research/visual-spatial-map/mirrors/iem-scenerotator-guide.html new file mode 100644 index 0000000..3c6dccc --- /dev/null +++ b/research/visual-spatial-map/mirrors/iem-scenerotator-guide.html @@ -0,0 +1,167 @@ + + + + + + + + + SceneRotator Guide + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ +
+
+ + + + +
+
+
+
+
+

SceneRotator Guide

+

+ How to use the SceneRotator plug-in. +

+ +
+
+ + + + + + +

Introduction

+

The SceneRotator plug-in can be used to rotate an Ambisonic scene around the three axes Yaw, Pitch, and Roll. Alternatively, you can use Quaternion data to controll the rotations. In case you use the Euler angles (Yaw, Pitch, and Roll) you need to define the order of rotations.

+

MIDI Connection

+

With the MIDI Connection feature, you can directly select a MIDI device, which sends rotation data. With the scheme selector, you select the type of MIDI device, so the plug-in correctly interprets the rotation data.

+

For now, our MrHeadtracker device as well as the Supperware Head Tracker 1 is supported. If you’d like to have other headtracking devices supported, please let us know! We are happy to add them to the list.

+

Troubleshooting

+

We are aware of an issue on Windows: In case the MIDI device has already been opened in a process (e.g. DAW), the plug-in won’t be able to use it. E.g. when Reaper has enabled the device in the Midi Device options, it can’t be loaded again. This is different on macOS or linux, it works just fine. A workaround is to disable the device in the Reaper options. Sometimes also a restart of Reaper helps.

+ + + +
+
+
+
+
+ +

Website created with Hugo using the kube theme.

+
+ +
+ + + + + + + + diff --git a/research/visual-spatial-map/mirrors/inspirit-ps3eyedriver-github.html b/research/visual-spatial-map/mirrors/inspirit-ps3eyedriver-github.html new file mode 100644 index 0000000..4d634a6 --- /dev/null +++ b/research/visual-spatial-map/mirrors/inspirit-ps3eyedriver-github.html @@ -0,0 +1,1758 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + GitHub - inspirit/PS3EYEDriver: PS3EYE Camera Driver for OSX and Windows · GitHub + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+
+ + +
+ Skip to content + + + + + + + + + + + +
+
+ + + + + + + + + + + + + + + + + + + +
+ +
+ + + + + + + + +
+ + + + + +
+ + + + + + + + + +
+
+
+ + + + + + + + + + +
+ +
+ +
+ +
+ + + + / + + PS3EYEDriver + + + Public +
+ + +
+ +
+ + +
+
+ +
+
+ + + + +
+ + + + + +
+ + + + + + + + + + + + + + + + + + +
+ + + + +

inspirit/PS3EYEDriver

+ + + +

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

91 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

PS3EYE Driver

+

Some driver like routines to connect and work with Sony PS3 EYE Camera.

+

The driver is using libusb to handle USB communications.

+

Most of the code was ported from Linux kernel source.

+

You can find example code for Cinder and openFrameworks in corresponding folders.

+
+

alt text +alt text

+
+ +
+
+
+
+

About

+ +

+ PS3EYE Camera Driver for OSX and Windows +

+ + +

Resources

+ + + +

License

+ + + + + + + + + + + + + + + +

Stars

+ + +

Watchers

+ + +

Forks

+ + + + +
+ +
+
+ + +
+
+

+ Releases

+ +
No releases published
+ +
+
+ + + + +
+
+ + + +

+ Packages +

+ + +
+
+
 
+
+
+
+
 
+
+
+
+
 
+
+ + + + +
+
+
+ + + +
+
+ + +

+ Contributors

+ +
    +
  • +
    +
  • +
  • +
    +
  • +
  • +
    +
  • +
+ + +
+
+
+ + + + + +
+
+
+ + + + +
+ +
+ +
+
+ +
+ +
+

Footer

+ + + + +
+
+ + + + + © 2026 GitHub, Inc. + +
+ + +
+
+ + + + + + + + + + + + + + + + + + + + +
+
+
+ + + diff --git a/research/visual-spatial-map/mirrors/inspirit-ps3eyedriver-readme.md b/research/visual-spatial-map/mirrors/inspirit-ps3eyedriver-readme.md new file mode 100644 index 0000000..eedaa9f --- /dev/null +++ b/research/visual-spatial-map/mirrors/inspirit-ps3eyedriver-readme.md @@ -0,0 +1,15 @@ +PS3EYE Driver +======== + +Some driver like routines to connect and work with Sony PS3 EYE Camera. + +The driver is using [libusb](https://github.com/libusbx/libusbx) to handle USB communications. + +Most of the code was ported from Linux kernel source. + +You can find example code for [Cinder](https://github.com/cinder/Cinder) and [openFrameworks](https://github.com/openframeworks/openFrameworks) in corresponding folders. + +--- + +![alt text](https://raw.github.com/inspirit/PS3EYEDriver/master/shot1.png "PS3EYE Running with Cinder") +![alt text](https://raw.github.com/inspirit/PS3EYEDriver/master/shot2.png "PS3EYE Running with Cinder") diff --git a/research/visual-spatial-map/mirrors/ipisoft-multiple-playstation-eye-cameras.html b/research/visual-spatial-map/mirrors/ipisoft-multiple-playstation-eye-cameras.html new file mode 100644 index 0000000..1b8c890 --- /dev/null +++ b/research/visual-spatial-map/mirrors/ipisoft-multiple-playstation-eye-cameras.html @@ -0,0 +1,1049 @@ + + + + + + + + + User Guide for Multiple PlayStation Eye Cameras Configuration (printable version) - iPiSoft Wiki + + + + + + + + + + + + + + +
+
+
+ +
+

 NEW!  Version 3 of iPi Motion Capture is available. +iPi Motion Capture 3 delivers new features for improved workflow including support of Kinect 2, improved arms tracking, simplified calibration and others. Read more >> +

This documentation corresponds to version 2 of iPi Motion Capture. The latest documentation for version 3 can be found at https://docs.ipisoft.com/ +

+

User Guide for Multiple PlayStation Eye Cameras Configuration (printable version)

+
+

From iPiSoft Wiki

+
+
Jump to: navigation, search
+

For the impatient: +

  • Get the cameras (a minimum of 3 PlayStation Eye cameras recommended), cables, tripods, computer and a flashlight (Mini Maglite recommended). +
  • Download iPi Recorder and iPi Mocap Studio +
  • Install iPi Recorder and iPi Mocap Studio +
  • Find a suitable space (13 feet by 13 feet minimum, preferably with light walls and bright lighting). +
  • Actor should be dressed in solid-color clothing. Recommended clothing: long-sleeve shirt, blue jeans or trousers, black shoes. Deep, saturated colors are preferable. +
  • Point cameras at the actor, connect the cables and use iPi Recorder to record synchronized multi-camera video. +
  • Switch iPi Recorder to "Calibration mode" and wave the flashlight (with its reflector removed) in view of cameras to record calibration video. Don’t move the flashlight too quickly. Switch off "Calibration mode". +
  • Record a video of actor’s performance. Start from a T-pose. Then goes actor’s performance. +
  • Refer to video tutorials for instructions on how to process calibration video and actor performance video. +
  • Import your character into iPi Mocap Studio (File > Import Target Character) to adapt the animation for your character rig. Export your animation in a desired format. +
+


Contents

+ +
+

System Requirements

+

iPi Recorder

+
  • Computer (desktop or laptop): +
    • CPU: x86 compatible (Intel Pentium 4 or higher, AMD Athlon or higher), dual- or quad- core is preferable +
    • Operating system: Windows 8, 7, XP SP3, Vista (x86 or x64) +
    • USB: at least two USB 2.0 or USB 3.0 controllers +
      For more info see USB controllers +
      +
    • ExpressCard slot (for laptops) +
      Optional, but highly recommended. It allows to install external USB controller in case of compatibility issues between cameras and built-in USB controllers, or if all USB ports are in fact connected to a single USB controller +
      +
    • Storage system: HDD or SSD or RAID with write speed: +
      • For 4 cameras at 60 fps, 640 x 480 resolution: not less than 70,4 MByte/sec +
      • For 6 cameras at 60 fps, 640 x 480 resolution: not less than 105,6 MByte/sec +
      +
    +
+
Note If your write speed is lower, you can work with 320 x 240 resolution and/or lower frame rate. Alternatively, you can use compressed mode (that gives 3-5 times lower required write speed, but the CPU performance may become a bottleneck) +
+
+ +

iPi Mocap Studio

+
  • Computer (desktop or laptop): +
    • CPU: x86 compatible (Intel Pentium 4 or higher, AMD Athlon or higher), dual- or quad- core is preferable +
    • Operating system: Windows 8, 7, XP SP3, Vista (x86 or x64) +
    • Video card: Direct3D 10-capable (Shader Model 4.0) gaming-class graphics card + +
    +
+ + +
GPUz example.gif
+

Software Installation

+

iPi Recorder

+
+

Before installation: +

  • unplug all cameras from computer +
+
+


Download and run the setup package of the latest version of iPi Recorder. You will be presented with the following dialog. +

IPi Recorder 2.0 Setup.png +

  1. Select needed components +
  2. Read and accept the license agreement by checking appropriate checkbox +
  3. Press the Install button to begin installation +
+
Note. Most of the components require administrative privileges because they install device drivers or write to Program Files and other system folders. On Windows Vista/7 you will be presented with UAC prompts when appropriate during installation. If you plan to use iPi Recorder under user which has no administrative rights, you can pre-install other components separately using administrator's account. +
+
Important! +
+
  1. You can plug only one MS Kinect / ASUS Xtion / PrimeSense Carmine sensor to one USB controller. 1 USB controller bandwidth is not enough to record from 2 sensors. +
  2. You can plug not more than 2 Sony PS Eye cameras to one USB controller, otherwise you will not be able to capture at 60 fps with 640 x 480 resolution. +
+
For more info see USB controllers. +
+


Upon installation is complete, iPi Recorder will launch automatically. Continue with user's guide to get a knowledge of using the software. +

Components

+

If some of the components is already installed, it has no checkbox and is marked with ALREADY INSTALLED label. +You should not install all optional components in advance, without necessity. All of them can be installed separately at later time. Components descriptions below contain corresponding download links. +

+ + + + +
  • iPi Recorder 2.x.x.x. This is required component and cannot be unchecked.
    iPi Recorder itself. +
+

iPi Mocap Studio

+
+

Before installation: +

  • Ensure your video card supports DirectX 10. Otherwise, iPi Mocap Studio will not run on your system. +
+
+

Download and run the latest setup package of iPi Mocap Studio. You will be presented with the following dialog: +

IPi Mocap Studio 2.0 Setup.png +

  1. Read and accept the license agreement by checking corresponding checkbox. +
  2. Press the Install button to begin installation. +
+

Upon installation is complete, iPi Mocap Studio will launch automatically. +

All components are required for installation. +Please note that the installation of Microsoft .NET Framework 3.5 SP1 requires an Internet connection. If needed, you can download offline installer for Microsoft .NET separately, and run it before iPi Mocap Studio setup. +Other components are included with iPi Mocap Studio setup. +

As a result of installation, iPi Studio will be executed. You will be prompted to enter your license key or start 30-days free trial period: +
Welcome to ipistudio dlg.png
+


For more info about license protection see License. +

Recording Video with Multiple PlayStation Eye Cameras

+

Environment

+

Space

+

For a multiple PlayStation Eye configuration, you need a minimum of 13 feet by 13 feet space (4 meters by 4 meters). At smaller space, actor simply won’t fit into view of cameras. +

For 640 by 480 camera resolution, capture area can be as big as 20 feet by 20 feet (7 meters by 7 meters). That should be enough for capturing motions like running, dancing etc. +

Background

+

Light-color background (light walls and light floor) is recommended for markerless motion capture. +iPi Desktop Motion Capture is designed to work with real-life backgrounds. A multi-camera configuration (3 cameras and up) can handle certain amount of background clutter. Please keep in mind that the system can be confused if your background has large objects of the same color as actor clothes. +

Environment and clothing.jpg +

Using a green or a blue backdrop may improve results, but you are not required to use a backdrop if you have reasonable office or home environment with light-color walls and bright lighting. +

Lighting

+

For best results, your environment should have multiple light sources for uniform, ambient lighting. Typical office lighting with multiple light sources located on ceiling should be quite suitable for markeless motion capture. In a home environment, you may need to use additional light sources to achieve more uniform lighting. +

Please note that the system cannot work in direct sunlight. If you plan a motion capture session outdoors you should choose a cloudy, overcast day. +

Actor Clothing

+

Actor should be dressed in solid-color long-sleeve shirt, solid-color trousers (or jeans) and solid-color shoes. Deep, saturated colors are preferable. Casual clothes like jeans should be OK for use with markerless mocap system. iPi Desktop Motion Capture uses clothing color for separating actor from background and therefore cannot work with totally arbitrary clothing. +

Recommended shirt (torso) colors are black, blue or green. Red is not recommended because red can blend with human skin color making it difficult for the system to see hands placed over torso. Black color is useful for reducing self-shadows on torso. If you have bright uniform lighting you can get better results with a primary-color (blue or green) shirt. +

Recommended jeans/trousers color is blue. +

Recommended shoe color is black. +

iPi Desktop Motion Capture has an option of using T-shirt over long-sleeve shirt for actor clothing. However, simple long-sleeve shirt may result in more accurate motion capture. +

Special considerations

+

Don’t stand too close to a wall. Shadows on the wall may confuse the system. If standing close to a wall is unavoidable you should use additional light sources to light the wall behind actor to minimize shadows. +

System can have problems tracking push-ups and similar motions because of shadows on the floor. You can improve tracking of such motions by using additional light sources to light the floor. +

Recording Process

+

Please record a video using iPi Recorder application. It supports recording with Sony PlayStation Eye cameras, depth sensors (Kinect) and DirectShow-compatible webcams (USB and FireWire). +

iPi Recorder is a stand-alone application and does not require a powerful video card. You may choose to install it on a notebook PC for portability. Since it is free, you can install it on as many computers as you need. +

Please run iPi Recorder and complete setup and background recording steps following the instructions: +

iPi Recorder Setup +

Framerate

+

It is recommended that you record all videos at maximum available framerate. High framerate helps reduce motion blur and capture fine details of the motion. +

Maximum possible framerate for Sony PlayStation Eye camera is 60 frames per second. Sony advertises PlayStation Eye camera as capable of capturing at 120 frames per second but framerates over 60 FPS result in too much noise in PlayStation Eye camera sensor and are not usable for motion capture. +

Framerate lower than 30 frames per second is not recommended for motion capture. +

4 cameras at 320 by 240 resolution

+

A dual-core CPU should be fast enough for recording a 4-camera video at 320 by 240 resolution at 60 frames per second. +

4 cameras at 640 by 480 resolution at 60 frames per second

+

A quad-core CPU is recommended for recording at 640 by 480 resolution at 60 frames per second. If you have a dual-core CPU you may need to configure a lower framerate and/or lower compression quality to be able to record video at 640 by 480. +

6 cameras at 640 by 480 resolution at 60 frames per second

+

A quad-core CPU clocked at 2.0 GHz (or better) is recommended for recording at 640 by 480 resolution at 60 frames per second. You will also need to get additional USB controller. +

USB controllers

+

All modern computers (e.g. dual-core and better) based on Intel, AMD and Nvidia chipsets have two high-speed USB (USB 2.0) controllers on board. That should give you enough bandwidth to be able to record with 4 cameras at 640x480 (raw Bayer format) at 60 FPS, or 6 cameras at 640x480 (raw Bayer format) at 40 FPS. +

Under certain circumstances you may need to get additional USB controllers. +

Scene Set-up

+

Three Camera Configuration

+

Recommended configuration for 3-camera setup is a half-circle: +

Click to enlarge
+

Example: +

Click to enlarge
+

Virtual view of the same scene: +

Click to enlarge
+

Four Camera Configuration

+

You can set up 4 cameras in a half-circle or a full-circle configuration, depending on available space. You can improve accuracy by placing one of the cameras high over the ground (like 3 meters high). +

Recommended configuration for 4-camera setup in half circle: +

Click to enlarge
+

Example: +

Click to enlarge
+

Six Camera Configuration

+

You can set up 6 cameras in a full-circle or a half-circle configuration, depending on available space. You can improve accuracy by placing one or two cameras high over the ground (like 3 meters high). +

Recommended configuration for 6-camera full-circle setup: +

Click to enlarge
+

Camera Setup

+

Install the cameras on tripods and connect cables. +

Click to enlarge
+

Sony PlayStation Eye cameras do not have standard tripod mounting screw, so you will have to use some kind of ad hoc solution. The simplest approach is to fix the cameras to tripods using sticky tape. +

When mixing active and passive USB cables, make sure cable connection order is correct (computer->active cable->passive cable->camera). +

If you're using the PlayStation Eye camera, make sure you have the lens set to the wide setting. +

Click to enlarge
+

Calibration

+

Calibration is a process of computing accurate camera positions and orientations from a video of user waving a small glowing object called marker (for color/color+depth cameras) or a slim rectangular board called calibration board (for depth sensors). This step is essential and required for multi-camera system setup. +

Important. Once you calibrated the camera system, you should not move your cameras for subsequent video shoots. If you move at least one camera, you need to perform calibration again. +

Importance of high frame rate

+

You should record calibration video at the same resolution as your action video and at the same (or higher) frame rate. +

Calibration at a different resolution may lead to reduced accuracy because cameras usually have different minor distortions at different resolutions (caused by internal scaling algorithm). +

Calibration at low frame rate may lead to reduced accuracy because of increased synchronization errors. +

Glowing Marker

+

Mini Maglite flashlight is recommended for calibration. This is a very common flashlight in US and many other countries. Removing flashlight reflector converts it into an ideal glowing marker easily detectable by motion capture software. +

Maglite.jpg +

If you cannot get a Mini Maglite, you can use some other similar flashlight. +

Noname flashlight.jpg +

Step 1: Running iPi Recorder in calibration mode

+

Run iPi Recorder and choose one of the darkening modes in "darkening for calibration" list (for Sony PS Eye cameras) +

Calibration-darkening.png +

or set Exposure to reasonably small value (for DirectShow-compatible web cameras) +

Calibration-exposure.png +

This is important because it helps to reduce motion blur during calibration. +

Video will look dim in calibration mode. +

Important! Do not turn off the light in the room during calibration! This will not help the software but will make it harder for you to see what is happening on recorded video when you view it later. +
+

Step 2: Record calibration video

+

Start video recording. +

Move the marker slowly through your entire capture volume (front-top-right-bottom-left-back-top-right-bottom-left). Start from top and move the marker in a descending spiral motion. +

Tip. The exact trajectory of the marker is not so important, just try to cover the whole capture volume, or at least its perimeter. +

Tip. For successful calibration, several hundred frames are sufficient. Much more frames (like several thousands) will not produce more accurate results, but instead will increase the processing time and possible number of marker detection errors. So generally it is not recommended to record calibration videos of more than 1 minute length. +

Tip. You should make the marker visible to as much cameras as possible at all times. Hold the marker in the straight arm away from your body. In a circle configuration, when approaching the boundary of the capture area, position the marker inside the area, and your body outside. +

Calibration1.jpg +

Calibration2.jpg +

Calibration3.jpg +

Put the marker to the ground at each corner and at the center of capture volume. At least 4-5 ground points are needed for the correct detection of the groundplane. +

Calibration ground1.jpg +

Calibration ground2.jpg +

Step 3: stop recording and check recorded video

+
  • There is no significant motion blur (image of marker looks like a round spot rather than an ellipse or a luminescent line) +
  • Most of the time marker is visible in at least 3 cameras and not completely obscured by human body +
+

Step 4: Take note of height of your first camera over the ground.

+

Take note of height of your first camera over the ground. You will need this parameter later. If you cannot measure this height accurately, then at least make a rough estimation. +

Step 5: process calibration video in iPi Mocap Studio

+

Strictly speaking, you can postpone processing calibration video until after you finished recording other videos (e.g. your action videos). However, it is a good idea to process your calibration video as soon as it was recorded because it helps you ensure that you have good calibration. (Later on, incorrectly recorded calibration video may affect your ability to process action videos. ) +

To process calibration video please do the following: +

  • Load your calibration video to iPi Mocap Studio +
  • Important. Adjust the Region of Interest to cover the part of video that contains the glowing marker. +
  • Set the Diagonal Field of View (FOV) for your cameras on the “Scene” tab. If you use Sony PlayStation Eye or Logitech QuickCam 9000 cameras, leave the FOV value at the default 75 degrees. +
  • Go to “Calibration” tab. Check “Auto detect initial camera positions” checkbox. Click “Calibrate” button and wait while the system finishes calibration. +
  • Calibration algorithm may occasionally fail to find correct camera positions. If this happens, you should manually adjust initial camera positions to roughly match your configuration. The main thing is the correct order of the cameras around the capture area, and their approximate view directions. +
    • Reset camera positions with one of the standard half-circle or full circle configurations that best suits your configuration. +
    • For each camera that requires adjusting, switch to this camera using the toolbar button, and correct its position using controls on the “Scene” tab. +
    • Uncheck “Auto detect initial camera positions” checkbox. +
    • Click “Calibrate” button. This will rerun calibration process, without recomputing marker positions. +
    +
+

Follow video tutorial: +

+


Resulting scene should look like this: +

Calibration result1.jpg +

Green points designate correctly detected 3D marker positions. +Red points designate misdetected marker positions. +10-20% of red points should be considered normal. Calibration is good if you have at least 70% of green points. +

Tip. When doing manual camera positioning before the calibration, you may find useful to match model's position with the actor on a video: +

  • Select the frame where the actor is standing relatively straight. +
  • Show the model by checking the "View > Skin" menu item. +
  • Adjust the camera position to fit the model with the actor's image. +
+

Step 6: Define ground plane

+

You need at least 3 points in 3D space to define ground plane. For each point, click on it in 3D view and press “Mark as ground” button. +

Calibration ground3.png +

Calibration ground4.png +

WARNING: If you do not mark ground points then the ground plane is incorrect and there is no sense in using Foot tracking option and camera heights values. +

Step 7: Set scene scale using camera height as reference

+

Now cameras in your scene are properly oriented relative to other cameras and relative to ground plane. But you still need to find one more parameter: scene scale. +

Use camera #1 height over ground to set correct scene scale. +

Calibration scale1.png +

Note: Height of camera can only be used if ground plane is properly defined. If ground plane is not defined, you can use distance between camera #1 and #2 to set scene scale. +

Calibration scale2.png +

Step 8: Save calibration result into *.scene.xml file

+

Calibration xml.png +

Recording Actor's Performance

+

Recommended layout of an action video

+
  • Enter the actor. +
  • Strike a T-pose. +
  • Action +
+ + + +
T-pose.pngAction.png
+

T-pose

+

It is preferable to have actor strike a “T-pose” before the actual action. The software will need T-pose for building actor appearance model during tracking. If you make several takes of one actor you do not need to re-record T-pose before each take. +

When using the depth sensors, it is recommended to face the palms down, as it corresponds to the default orientation of the model's hand bones. +When using color cameras, it is recommended to face the palms forward, as it helps the software in determining the right color for the model's hands. +

Takes

+

Take is a concept originating from cinematography. In a nutshell, take is a single continuous recorded performance. +

Usually it is a good idea to record multiple takes of the same motion, because a lot of things can go wrong for purely artistic reasons. +

Iterations

+

A common problem with motion capture is “clipping” in resulting 3D character animation. For example, arms entering the body of animated computer-generated character. Many CG characters have various items and attachments like a bullet-proof vest, a fantasy armor or a helmet. It can be easy for an actor to forget about the shape of the CG model. +

For this reason, you may need to schedule more than one motion capture session for the same motions. Recommended approach is: +

  • Record the videos +
  • Process the videos in iPiStudio +
  • Import your target character into iPiStudio and review the resulting animation +
  • Give feedback to the actor +
  • Schedule another motion capture session if needed +
+

Ian Chisholm's hints on motion capture

+

Ian Chisholm is a machinima director and actor and the creator of critically acclaimed Clear Skies machinima series. Below are some hints from his motion capture guide based on his experience with motion capture for Clear Skies III. +

Three handy hints for acting out mocap:

+
  1. Don’t weave and bob around like you’re in a normal conversation – it looks terrible when finally onscreen. You need to be fairly (but not completely) static when acting. +
+
  1. If you are recording several lines in one go, make sure you have lead in and lead out between each one, i.e. stand still! Otherwise, the motions blend into each other and it’s hard to pick a start and end point for each take. +
+
  1. Stand a bit like a gorilla – have your arms out from your sides:
    Gorilla.jpg
    Well, obviously not quite that much. But anyway, if you don’t, you’ll find the arms clip slightly into the models and they look daft. +
+

If you have a lot of capture to do, you need to strike a balance between short and long recordings. Aim for 30 seconds to 2 minutes. Too long is a pain to work on later due to the fiddlyness of setting up takes, and too short means you are forever setting up T-poses. +

Takes

+

Because motion capture is not a perfect art, and neither is acting, it’s best to perform multiple takes. I found that three was the best amount for most motion capture. Take less if it’s a basic move, take more if it’s complex and needs to be more accurate. It will make life easier for you in the processing stage if you signal the break between takes – I did this by reaching out one arm and holding up fingers to show which take it was. +

Naming conventions

+

As it’s the same actor looking exactly the same each and every time, and there is no sound, and the capture is in lowres 320*200, you really need to name the files very clearly so that you later know which act, scene, character, and line(s) the capture is for. +

My naming convention was based on act, scene, character, page number of the scene, line number, and take number. You end up with something unpleasant to read like A3S1_JR_P2_L41_t3 but it’s essential when you’ve got 1500 actions to record. +

Processing Video from Multiple PlayStation Eye Cameras

+

Open multi-camera video in iPi Mocap Studio. Load scene configuration from file. +

Adjust the Region of Interest (ROI). (Region of Interest should cover the part of video that contains the motion). +

Go to T-pose frame. Using character Move/Rotate controls, roughly align the model with the image of actor in one view. Adjust character height and proportions (on Actor tab) to correspond to actor. +

Push the Analyze Appearance button on Actor tab to automatically adjust model's colors. For the multi-camera video, Analyze Appearance button uses only one (current) camera. This makes it easier to align model with image. You can refine actor appearance by alternating Refit Pose and Analyze Appearance. Also you can edit the colors manually or with the eyedropper tool. +

Go to the first frame of Region of Interest (ROI). Push the Refit Pose button on Tracking tab. If initial pose was recognized incorrectly, you can roughly adjust it manually and use auto-fit again. +

Push the Track Forward button on Tracking tab. +

You can save your motion capture project to a file. If your project is saved to a file, then the system will auto-save it after each processed frame during tracking. +

Refer to video tutorial for tips on video processing: +

+

Clean-up

+

Once initial tracking is performed on all (or part) of your video, you can begin cleaning out tracking errors (if any). Post-processing should be applied after clean-up. +

Cleaning up tracking gaps

+

Tracking errors usually happen in a few specific video frames and propagate to multiple subsequent frames, resulting in tracking gaps. Examples of problematic frames: +

  • Occlusion (like one hand not visible in any of the cameras) +
  • Indistinctive pose (like hands folded on chest). +
  • Very fast motion with motion blur. +
+

To clean up a sequence of incorrect frames (a tracking gap), you should use backward tracking: +

  1. Go toward the last frame of tracking gap, to a frame where actor pose is distinctive (no occlusion, no motion blur etc.). +
  2. If necessary, use Rotate, Move and IK (Inverse Kinematics) tools to edit character pose to match actor pose on video. +
  3. Turn off Trajectory Filtering (set it to zero) so that it does not interfere with your editing. +
  4. Click Refit Pose button to get a better fit of character pose. +
  5. Click Track Backward button. +
  6. Stop backward tracking as soon as it comes close to the nearest good frame. +
  7. If necessary, go back to remaining parts of tracking gap and use forward and backward tracking to clean them up. +
+

Cleaning up individual frames

+

To clean up individual frames you should use a combination of editing tools (Rotate, Move and Inverse Kinematics) and Refit Pose button. +

Note: after Refit Pose operation iPi Mocap Studio automatically applies Trajectory Filtering to produce a smooth transition between frames. As the result, pose in current frame is affected by nearby frames. This may look confusing. If you want to see exact result of Refit Pose operation in current frame you should turn off Trajectory Filtering (set it to zero), but do not forget to change it back to suitable value later.
+

Tracking errors that cannot be cleaned up using iPi Studio

+

Not all tracking errors can be cleaned up in iPi Mocap Studio using automatic tracking and Refit Pose button. +

  • Frames immediately affected by occlusion sometimes cannot be corrected. Recommended workarounds: +
    • Manually edit problematic poses (not using Refit Pose button). +
    • Record a new video of the motion and try to minimize occlusion. +
    • Record a new video of the motion using more cameras. +
    +
  • Frames immediately affected by motion blur sometimes cannot be corrected. Recommended workarounds: +
    • Manually edit problematic poses (not using Refit Pose button). +
    • Edit problematic poses in some external animation editor. +
    • Record a new video of the motion using higher framerate. +
    +
  • Frames affected by strong shadows on the floor sometimes cannot be corrected. Typical example is push-ups. This is a limitation of current version of markerless mocap technology. iPi Soft is working to improve tracking in future versions of iPi Mocap Studio. +
  • Some other poses can be recognized incorrectly by iPi Mocap Studio. This is a limitation of current version of markerless mocap technology. iPi Soft is working to improve tracking in future versions of iPi Mocap Studio. +
+

Tracking refinement

+

After the primary tracking and cleanup are complete, you can optionally run the Refine pass (see Refine Forward and Refine Backward buttons). It slightly improves accuracy of pose matching, and can automatically correct minor tracking errors. However, it takes a bit more time than the primary tracking, so it is not recommended for quick-and-dirty tests. +

Important. Refine should be applied with the same tracking parameters (e.g. feet tracking, head tracking) as the primary tracking in order not to lose previously tracked data. +

Important. Refine should be applied before motion controller data. Also, if you plan to manually edit the animation (not related to automatic cleanup with Refit Pose), then also do this after applying Refine. +

In contrast to the primary tracking, this pass does no pose prediction, and bases its computations solely on the current pose in a frame. Essentially, running Refine is equal to automatically applying Refit Pose to a range of frames which were previously tracked. +

Post-processing: Jitter Removal

+

Jitter Removal filter is a powerful post-processing filter. It should be applied after cleaning up tracking gaps and errors. +

It is recommended that you always apply Jitter Removal filter before exporting animation. +

Jitter Removal filter suppresses unwanted noise and at the same time preserves sharp, dynamic motions. By design, this filter should be applied to relatively large segments of animation (no less than 50 frames). +

Range of frames affected by Jitter Removal is controlled by current Region of Interest (ROI). +

You can configure Jitter Removal options for specific body parts. Default setting for Jitter Removal “aggressiveness” is 1 (one tick of corresponding slider). Oftentimes, you can get better results by applying a slightly more aggressive Jitter Removal for torso and legs. Alternatively, you may want to use less aggressive Jitter Removal settings for sharp motions like martial arts moves. +

Jitter Removal filter makes an internal backup of all data produced by tracking and clean up stages. Therefore, you can re-apply Jitter Removal multiple times. Each subsequent run works off original tracking/clean-up results and overrides previous runs. +

Post-processing: Trajectory Filtering

+

Trajectory Filter is a traditional digital signal filter. Its purpose is to filter out minor noise that remains after Jitter Removal filter. +

Trajectory Filter is very fast. It is applied on-the-fly to current Region of Interest (ROI). +

Default setting for Trajectory Filter is 1. Higher settings result in multiple passes of Trajectory Filter. It is recommended that you leave it at the default setting. +

Trajectory Filter can be useful for “gluing” together multiple segments of animation processed with different Jitter Removal options: change the Region of Interest (ROI) to cover all of your motion (e.g. multiple segments processed with different jitter removal setting); change Trajectory Filtering setting to 0 (zero); then change it back to 1 (or other suitable value). +

Export and Motion Transfer

+

Use File > Export Animation menu item to export all animation frames from within Region of Interest (ROI). +

To export animation for specific take, right-click on take and select Export Animation item from pop-up menu. +

Default iPi Character Rig

+

The default skeleton in iPi Mocap Studio is optimized for markerless motion capture. It may or may not be suitable as a skeleton for your character. Default iPi skeleton in T-pose has non-zero rotations for all joints. Please note that default iPi skeleton with zero rotations does not represent a meaningful pose and looks like a random pile of bones. +

+ + +
Default rig
Bone names
+

By default iPi Mocap Studio exports a T-pose (or a reasonable default pose for custom rig after motion transfer) in the first frame of animation. In case when it is not desired switch off Export T-pose in first frame checkbox. +

Motion Transfer and Custom Rigs

+

iPi Mocap Studio has integrated motion transfer technology. You can import your character into iPi Studio via File > Import Target Character menu item and your motion will be transferred to your character. You may need to assign bone mappings on the Export tab for motion transfer to work correctly. You can save your motion transfer profile to XML file for future use. iPi Mocap Studio has pre-configured motion transfer profiles for many popular rigs (see below). If you export animation to format different from format your target character was imported in, only rig will be exported. If you use the same format for export, skin will be exported as well. +

MotionBuilder

+

Use the Export Animation for MotionBuilder menu item to export your motion in MotionBuilder-friendly BVH format. MotionBuilder-friendly skeleton in T-pose has zero rotations for all joints, with bone names consistent with MotionBuilder conventions. This format may also be convenient for use with other apps like Blender. +

3D MAX Biped

+

Use the Export Animation for 3D MAX menu item to export your motion in 3D MAX-friendly BVH format. +

Create a Biped character in 3D MAX (Create > Systems > Biped). Go to Motion tab. Click Motion Capture button and import your BVH file. +

+ + +
Step 1
Step 2
+ + + +
Step 3
Step 4
+

Our user Cra0kalo created an example Valve Biped rig for use with 3D MAX. It may be useful if you work with Valve Source Engine characters. +

Maya

+

Latest versions of Maya (starting with Maya 2011) have a powerful biped animation subsystem called "HumanIK". Animations exported from iPi Mocap Studio in MotionBuilder-friendly format (the Export Animation for MotionBuilder menu item) should work fine with Maya 2011 and HumanIK. The following video tutorials can be helpful: +

+

For older versions of Maya please see the #Motion Transfer and Custom Rigs section. Recommended format for import/export with older versions of Maya is FBX. +

FBX

+

iPi Mocap Studio supports FBX format for import/export of animations and characters. By default, iPi Mocap Studio exports animations in FBX 6.0 format using FBX SDK 2012. If your target character is in FBX 7.0 or newer format, iPi Mocap Studio will export retargeted animation in FBX 2012 format. +

Some applications do not use latest FBX SDK and may have problems importing newer version FBX files. In case of problems, your can use Autodesk's free FBX Converter to convert your animation file to appropriate FBX version. +

COLLADA

+

iPi Mocap Studio supports COLLADA format for import/export of animations and characters. Current version of iPi Mocap Studio exports COLLADA animations as matrices. In case if you encounter incompatibilities with other applications' implementation of COLLADA format, we recommend using Autodesk's free FBX Converter to convert your data between FBX and COLLADA formats. FBX is known to be more universally supported in many 3D graphics packages. +

LightWave

+

Recommended format for importing target characters from LightWave to iPi Studio is FBX. Recommended format for bringing animations from iPi Mocap Studio to LightWave is BVH or FBX. +

SoftImage|XSI

+

Our user Eric Cosky published a tutorial on using iPi Mocap Studio with SoftImage|XSI: +

http://www.ipisoft.com/forum/viewtopic.php?f=13&p=9660#p9660 +

Poser

+

Export your poser character in T-pose in BVH format (File > Export). Import your Poser character skeleton into iPi Mocap Studio (File > Import Target Character). Your animation will be transferred to your Poser character. Now you can use File > Export Animation to export your animation in BVH format for Poser. +

Poser 8 has a bug with incorrect wrists animation import. The bug can be reproduced as follows: export Poser 8 character in T-pose in BVH format; import your character back into Poser 8; note how wrists are twisted unnaturally as the result. +

A workaround for wrists bug is to chop off wrists from your Poser 8 skeleton (for instance using BVHacker) before importing Poser 8 target character into iPi Mocap Studio. Missing wrists should not cause any problems during motion transfer in iPi Mocap Studio if your BVH file is edited correctly. Poser will ignore missing wrists when importing resulting motion so the resulting motion will look right in Poser (wrists in default pose as expected). +

+ + +
Step 1
Step 2
+ + + +
Step 3
Step 4
+

DAZ 3D

+

The workflow for DAZ 3D is very similar to Poser. Import your DAZ 3D character skeleton into iPi Mocap Studio (File > Import Target Character). Your animation will be transferred to your DAZ 3D character. Now you can use File > Export Animation to export your animation in BVH format for DAZ 3D. +

IMPORTANT: You can use DAZ character in COLLADA (.dae) format for preview, but it is strongly recommended that you use DAZ character in BVH format for motion transfer. DAZ3D has a problem with COLLADA (.dae) format: DAZ3D Studio does not export all bones into COLLADA (.dae). In particular, the following bones are not exported: eyeBrow, bodyMorphs. DAZ3D Studio does not use bone names when importing motions; instead, DAZ3D Studio just takes rotations from the list of angles as though it was a flat list with exactly the same positions as in DAZ3D internal skeleton. As the result, when you transfer the motion to a COLLADA character and import it back into DAZ3D, the motion will look wrong. iPi Mocap Studio displays a warning about this. To avoid this problem, import your DAZ target character in BVH format - DAZ3D Studio is known to export characters in BVH format correctly (with all bones). +

You can improve accuracy of motion transfer by doing some additional preparation of your DAZ 3D skeleton in BVH format. For DAZ 3D Michael 4.0 and similar characters, you may need to clamp thigh joint rotation to zero to avoid unnatural leg bending. For DAZ 3D Victoria 4.0, you may need to adjust foot joint rotation to change the default “high heels“ foot pose to a more natural foot pose. +

+ + +
Step 1
Step 2
+ + + +
Step 3
Step 4
+ + +
Step 5
+

iClone

+

Current version of iPi Studio can only export animation in iClone-compatible BVH format. The iMotion format is not supported as of yet. That means you will need iClone PRO to be able to import the motion into iClone. Standard and EX versions of iClone do not have BVH Converter and therefore cannot import BVH files. +

Workflow for iClone is straightforward. Export your animation using Export Animation for iClone menu item. Go to Animation tab in iClone and launch BVH Converter. Import your BVH file with Default profile, click Convert and save the resulting animation in iMotion format. Now your animation can be applied to iClone characters. +

iClone expects an animation sampled at 15 frames per seconds. For other frame rates, you may need to create a custom BVH Converter profile by copying Default profile and editing Frame Rate setting. +

BVH Converted in iClone 4 has a bug that causes distortion of legs animation. iPi Mocap Studio exports an iClone-optimized BVH correctly as can be verified by reviewing exported BVH motion in BVHacker or MotionBuilder or other third-party application. No workaround is known. We recommend that you contact iClone developers about this bug as it is out of control of iPi Soft. +

+ + +
Step 1
Step 2
+ + + +
Step 3
Step 4
+ + +
Step 5
+

Valve Source Engine SMD

+

Import .smd file for your Valve Source Engine character into iPi Mocap Studio via File > Import Target Character menu item. Your animation will be transferred to your character. Now you can use File > Export Animation to export your animation in Valve Source Engine SMD format. +

Our user Cra0kalo created an example Valve Biped rig for use with 3D MAX. It may be useful if you wish to apply more then one capture through MotionBuilder or edit the custom keyframes in MAX. +

Valve Source Filmmaker

+

DMX

+

First, you need to import your character (or its skeleton) into iPi Mocap Studio, for motion transfer. +

There are currently 3 ways of doing this: +

  1. You can import an animation DMX (in default pose) into iPi Mocap Studio. Since it has a skeleton, it should be enough for motion transfer. To create an animation DMX with default pose, you can add your character to your scene in Source Filmmaker and export DMX for corresponding animation node: +
    • open Animation Set Editor Tab; +
    • click + > Create Animation Set for New Model; +
    • choose a model and click Open; +
    • export animation for your model, in ASCII DMX format; +
      There is a checkbox named Ascii in the top area of the export dialog. +
      +
    +
  2. Alternatively, you can just import an SMD file with your character into iPi Mocap Studio. For example, SMD files for all Team Fortress 2 characters can be found in your SDK in a location similar to the following (you need to have Source SDK installed): C:\Program Files (x86)\Steam\steamapps\<your steam name>\sourcesdk_content\tf\modelsrc\player\pyro\parts\smd\pyro_model.smd). +
  3. If you created a custom character in Maya, you should be able to export it in DMX model fromat. (Please see Valve documentation on how to do this). +
+

Then you can import your model DMX into iPi Mocap Studio. Current version of iPi Mocap Studio cannot display character skin, but it should display the skeleton. Skeleton should be enough for motion transfer. +

To export animation in DMX, you should just use General... export menu item in iPi Mocap Studio and choose DMX from the list of supported formats. You may also want to uncheck Export T-pose in first frame option on the Export tab in iPi Mocap Studio. +

Now you can import your animation into Source Filmmaker. There will be some warnings about missing channels for face bones but you can safely ignore them. +

+ + +
Step 1
Step 2
+ + + +
Step 3
Step 4
+ + + +
Step 5
Step 6
+

Old way involving Maya

+

This was used until iPi Mocap Studio got DMX support. And still may be useful in case of any troubles with DMX. Please see the following video tutorial series: +

http://www.youtube.com/playlist?list=PLD4409518E1F04270 +

Blender

+

iPi Mocap Studio can export animations in Blender-friendly BVH format. For latest versions of Blender, you can use the MotionBuilder-compatible BVH export option. For Blender version 2.59 or older you can use the Blender 2.59 or older export option. +

Cinema4D

+

If you have experience with Cinema4D please help to expand this Wiki by posting Cinema4D import/export tips to Community Tutorials section of our user forum. +

Evolver

+

iPi Mocap Studio supports importing of skinned Evolver characters in COLLADA or FBX format. Import your Evolver character skeleton into iPi Mocap Studio (File > Import Target Character). Your animation will be retargeted to your Evolver character. Now you can use File > Export Animation to export your animation. +

Evolver offers several different skeletons for Evolver characters. Here is an example motion transfer profile for Evolver "Gaming" skeleton: evolver_game.profile.xml +

Evolver example.jpg +

Second Life

+

Import your Second Life character skeleton into iPi Mocap Studio (File > Import Target Character). Your animation will be transferred to your Second Life character. Now you can use File > Export Animation to export your animation in BVH format for Second Life. +

SecondLife documentation contains a link to useful SL avatar files. The ZIP file includes a BVH of the "default pose". Be sure to have that. +

See the discussion on our Forum for additional details: http://www.ipisoft.com/forum/viewtopic.php?f=2&p=7845 +

Massive

+

Please see our user forum for a discussion of animation import/export for Massive: +

http://ipisoft.com/forum/viewtopic.php?f=12&t=3233 +

IKinema WebAnimate

+

Please see the following video tutorial on how to use iPi Mocap Studio with IKinema WebAnimate: +

http://www.youtube.com/watch?v=a-yJ-O02SLU +

Jimmy|Rig Pro

+

Please see the following video tutorial on how to use iPi Mocap Studio with Jimmy|Rig Pro: +

http://www.youtube.com/watch?v=wD1keDh3fCk +

Troubleshooting

+

Installation problems

+

Potential problem: after installation, iPi Mocap Studio crashes on first start. +

Possible cause: very often, this is caused by incompatible video card. Another possible reason is broken .NET Framework installation or broken DirectX installation. +

Solution: check system requirements and make sure your operating system and .NET Framework is up to date. +

  • You can use GPU-z and CPU-z to detect exact version and features of your video card. +
+ + +
GPUz example.gif
+
  • Some systems (for example some laptops and older Mac Minis) have both integrated graphics and discrete graphics. You may need to disable integrated graphics when working with iPi Mocap Studio. To do this, please use tools offered by your discrete video card manufacturer (e.g. Nvidia or AMD). +
+

Two Kinects don't work together

+

Potential problem: two Kinect sensors do not work togethter. +

Possible cause: Most probably, both Kinects were plugged into one USB controller. In this case 1 USB controller bandwidth is not enough to handle video from 2 Kinects. +

Solution: Each Kinect should be plugged into separate USB controller. Please refer to documentation User_Guide_for_Dual_Kinect_Sensor_Configuration#Software_Installation +

How to report bugs and issues

+

When reporting bugs and issues, please specify the following info: +

  • exact version of your operating system; +
  • exact model of your video card (you can use GPU-Z to find out the model of your video card); +
  • the number and models of your cameras. +
+

You can post your bug reports on our User Forum or send them to iPiSoft tech support email. +

How to send a video to iPiSoft tech support

+

Sending your videos to iPiSoft tech support can be helpful if you experience a problem with iPiSoft's system. iPiSoft promises to use your video only for debugging and not to disclose it to third parties. +

To send a video, please upload it to some file sharing server like filefactory.com and send us the link. The other way is using peer-to-peer technology. +

If you cannot send a video because of its huge size, consider sending screenshots. Screenshots are less informative then video but still they are helpful for diagnosing various problems with tracking. +

Video Materials

+

For video materials please refer to our Gallery +


Video tutorials are available +

Cameras and accessories

+

You can order software and hardware as one package from CamVFX, our official US Reseller. +

Alternatively, you can select and order items you need separately. +

Cameras

+

Recommended camera for use with iPi Motion Capture is Sony PlayStation Eye for PS3. +


Playstation eye small2.jpg +

The following properties make it a preferred camera for motion capture: +

  • High frame rate — up to 60 frames per second (important for reducing motion blur and capturing fast motions like dancing, boxing or running). +
  • Fully automatic configuration with excellent auto-exposure and auto-gain. +
  • Very good low-light sensitivity. +
  • Low-distortion wide-angle fixed optics. +
  • Efficient protocol for image transfer (saves more than 50% of USB bandwidth when compared to other webcams). +
  • Low cost (under $40). +
+

The only downside of PlayStation Eye is its lack of tripod mounting screw. A possible solution is to use sticky tape to fix it to tripods. +

You can use other webcams with iPi Motion Capture but please be aware that most webcams are not capable of capturing video at frame rates higher than 15 frames per second. Some webcams like Logitech Webcam Pro 9000 can reliably capture video at 30 frames per second. +

Cables

+

For distances over 5m, you should use active USB 2.0 extension cables. You can daisy-chain multiple active USB 2.0 extension cables for distances longer than 10m. +

Active USB cable.jpg +

When combining active and passive cables, make sure that connection order is correct (computer->active cable->passive cable->camera). +

Tripods

+

Inexpensive 40- or 50-inch tripods like +[1] +should be fine for use with iPi Desktop Motion Capture. It is recommended to place at least one of the cameras reasonably high (2 or 3 meters high) so you may need a high (72 inch or higher) tripod like [2] or some kind of ad hoc wall mounting solution. +

Tripod.jpg +

Mini Maglite (or other flashlight) for calibration

+

You will need a small bright object for calibration. It is recommended to get Mini Maglite (or other similar flashlight) for that purpose. +

Maglite.jpg +

Video Card

+

iPi Mocap Studio relies on GPU for calculations. Therefore, it is recommended to use a fast GPU for processing recorded videos. As a minimum, you need a Shader Model 4.0-compatible (DirectX 10-compatible) GPU. Intel integrated graphics is not supported +

Please be aware that desktop GPUs are dramatically faster than laptop GPUs (due to apparent thermal constraints in laptop PCs). Laptops with switchable graphics should be configured to run iPi Mocap Studio on a discrete (high-performance) GPU. +

You can check expected tracking speed for various GPU model here. +

You can Compare and Order recommended GPU from Amazon. +

Professional GPUs like NVIDIA Quadro or ATI FireGL are not recommended due to high price and lack of performance advantage. You should understand that Quadro/FireGL video cards are designed for very specific professional OpenGL-based workloads. Their main advantage is high image quality in OpenGL rendering. iPi Desktop Motion Capture needs raw 32-bit floating point performance and does not benefit from image quality features of Quadro/FireGL video cards. For best performance, use top gaming-class NVIDIA GeForce or ATI Radeon video cards. +

USB controllers

+

All modern computers (e.g. dual-core and better) based on Intel, AMD and Nvidia chipsets have two high-speed USB (USB 2.0) controllers on board. That should give you enough bandwidth to be able to record with: +

  1. 2 depth sensors (MS Kinect, or ASUS Xtion, or PrimeSense Carmine) +
  2. or 4 cameras at 640x480 (raw Bayer format) at 60 FPS +
  3. or 6 cameras at 640x480 (raw Bayer format) at 40 FPS. +
+


You can check how many USB 2.0 controllers you have in the “Universal Serial Bus controllers” section in Device Manager. Operating system will display USB 2.0 controllers as “USB Enhanced Host Controllers” (not to be mistaken with USB 1.0 “USB Universal Host controllers”). +

USB controllers.png +

Alternatively, you can use USBView utility from Microsoft which allows to view all USB controllers and devices in a more convenient way: Download USBView +

You should make sure that you have depth sensors or cameras evenly distributed between available USB 2.0 controllers. For example, if you have 4 cameras, you should connect 2 cameras to first controller and 2 cameras to second controller. Device Manager usually shows USB cameras as “USB Composite Device” at the “Advanced” tab of USB controller properties. If you have too many cameras connected to single USB 2.0 controller, you should re-plug them to different USB ports. +

USB hubs

+

If several devices are connected on the USB hub and working in parallel, then the USB bandwidth is shared among the devices. Connecting all cameras via single USB 2.0 hub is OK only for low resolution video recording (320x240). To be able to record video at 640x480 and 60 frames per second, you should avoid connecting more than 2 cameras via single USB controller. +

Potential issues with USB controllers

+

Some USB chips have compatibility issues with specific cameras which may result in not detecting cameras, low frame rate, etc. See the next section on known issues. +

Many modern laptops have one of the integrated USB 2.0 controllers reserved for docking station. That may effectively halve your USB bandwidth. So laptops with a large number of USB ports (4 or more) are preferrable. Desktop PCs should not have such problem. +

Important. Before buying a specific laptop model it is strongly recommended to ensure it has the sufficient number of available built-in USB controllers and they are compatibile with cameras you intend to use. Find it at your local retailer, and bring cameras with you to test. Though it may be easier if your friend/neighbour has such a laptop. Or ask at our forum, maybe someone already used this model with our software or can recommend specific model for your camera configuration. +

If you plan getting an additional USB controller, please be aware of potential problems with USB 2.0 controllers available on the market. Many USB 2.0 controllers are based on defective NEC USB 2.0 chip, not capable of standard USB 2.0 speeds. That's a well-known bug specific only to NEC USB 2.0 chip. By contrast, all NEC USB 3.0 chips are good and do not have this problem. +

Some manufacturers sell what they call a "USB 2.0 ExpressCard" controller for laptops. Many of such ExpressCard controllers are in fact internal USB hubs in ExpressCard form factor. That means they do not add actual USB bandwidth to the system. All USB 3.0 ExpressCard are controllers. +

If you plan getting additional high-speed USB controller, we strongly recommend that you get a USB 3.0 controller, just to be on the safe side. +

When setting up a USB 3.0 controller, make sure you configure it to handle USB 2.0 traffic (as opposed to passing USB 2.0 to integrated USB 2.0 controllers in your chipset). There should be a corresponding setting in driver properties or in BIOS. +

Known compatibility issues

+

Combinations of camera & USB chip listed below have known compatibility issues. Unfortunately, there is no guarantee that other combinations will work for sure, but most will. If you need an additional USB controller for connecting specific camera, be sure not to get the one based on incompatible USB chip. +

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Camera +USB chip +Symptoms +Comments / possible solution +
ASUS Xtion (Live) +NEC µPD720200 (USB3) +Camera is not properly recognized by the system. +This firmware update makes device behave the same as PrimeSense Carmine 1.08 with this controller. +
ASUS Xtion (Live) +VIA VT6212L (USB2) +Frame drops of color picture in 640x480@30 (depth + color). + +
PrimeSense Carmine 1.08 +NEC µPD720200 (USB3) +Frame drops in 640x480@30 (depth + color). +Stable work in 640x480@30 (depth only) and 320x240 (depth + color). +
PrimeSense Carmine 1.08 +VIA VT6212L (USB2) +Frame drops in 640x480@30 (depth + color). +Stable work in 640x480@30 (depth only) and 320x240 (depth + color). +
PrimeSense Carmine 1.08 +VIA VL800(USB3) +Frame drops, reduced frame rate, freezes in depth + color modes. +Stable work in depth only modes. +
Sony Playstation Eye +VIA VL800 (USB3) +Great number of bad frames and frame drops. Camera is invisible to iPi Recorder. + +
Sony Playstation Eye +VIA VT6212L (USB2) +Great number of frame drop in 640x480@60 mode. +2 cameras can work smoothly at lower FPS (up to 640x480@50). +
+

You can order software and hardware as one package from CamVFX, our official US Reseller. +

Alternatively, you can select and order items you need separately. +

Cameras

+

Recommended camera for use with iPi Motion Capture is Sony PlayStation Eye for PS3. +


Playstation eye small2.jpg +

The following properties make it a preferred camera for motion capture: +

  • High frame rate — up to 60 frames per second (important for reducing motion blur and capturing fast motions like dancing, boxing or running). +
  • Fully automatic configuration with excellent auto-exposure and auto-gain. +
  • Very good low-light sensitivity. +
  • Low-distortion wide-angle fixed optics. +
  • Efficient protocol for image transfer (saves more than 50% of USB bandwidth when compared to other webcams). +
  • Low cost (under $40). +
+

The only downside of PlayStation Eye is its lack of tripod mounting screw. A possible solution is to use sticky tape to fix it to tripods. +

You can use other webcams with iPi Motion Capture but please be aware that most webcams are not capable of capturing video at frame rates higher than 15 frames per second. Some webcams like Logitech Webcam Pro 9000 can reliably capture video at 30 frames per second. +

Cables

+

For distances over 5m, you should use active USB 2.0 extension cables. You can daisy-chain multiple active USB 2.0 extension cables for distances longer than 10m. +

Active USB cable.jpg +

When combining active and passive cables, make sure that connection order is correct (computer->active cable->passive cable->camera). +

Tripods

+

Inexpensive 40- or 50-inch tripods like +[3] +should be fine for use with iPi Desktop Motion Capture. It is recommended to place at least one of the cameras reasonably high (2 or 3 meters high) so you may need a high (72 inch or higher) tripod like [4] or some kind of ad hoc wall mounting solution. +

Tripod.jpg +

Mini Maglite (or other flashlight) for calibration

+

You will need a small bright object for calibration. It is recommended to get Mini Maglite (or other similar flashlight) for that purpose. +

Maglite.jpg +

Video Card

+

iPi Mocap Studio relies on GPU for calculations. Therefore, it is recommended to use a fast GPU for processing recorded videos. As a minimum, you need a Shader Model 4.0-compatible (DirectX 10-compatible) GPU. Intel integrated graphics is not supported +

Please be aware that desktop GPUs are dramatically faster than laptop GPUs (due to apparent thermal constraints in laptop PCs). Laptops with switchable graphics should be configured to run iPi Mocap Studio on a discrete (high-performance) GPU. +

You can check expected tracking speed for various GPU model here. +

You can Compare and Order recommended GPU from Amazon. +

Professional GPUs like NVIDIA Quadro or ATI FireGL are not recommended due to high price and lack of performance advantage. You should understand that Quadro/FireGL video cards are designed for very specific professional OpenGL-based workloads. Their main advantage is high image quality in OpenGL rendering. iPi Desktop Motion Capture needs raw 32-bit floating point performance and does not benefit from image quality features of Quadro/FireGL video cards. For best performance, use top gaming-class NVIDIA GeForce or ATI Radeon video cards. +

USB controllers

+

All modern computers (e.g. dual-core and better) based on Intel, AMD and Nvidia chipsets have two high-speed USB (USB 2.0) controllers on board. That should give you enough bandwidth to be able to record with: +

  1. 2 depth sensors (MS Kinect, or ASUS Xtion, or PrimeSense Carmine) +
  2. or 4 cameras at 640x480 (raw Bayer format) at 60 FPS +
  3. or 6 cameras at 640x480 (raw Bayer format) at 40 FPS. +
+


You can check how many USB 2.0 controllers you have in the “Universal Serial Bus controllers” section in Device Manager. Operating system will display USB 2.0 controllers as “USB Enhanced Host Controllers” (not to be mistaken with USB 1.0 “USB Universal Host controllers”). +

USB controllers.png +

Alternatively, you can use USBView utility from Microsoft which allows to view all USB controllers and devices in a more convenient way: Download USBView +

You should make sure that you have depth sensors or cameras evenly distributed between available USB 2.0 controllers. For example, if you have 4 cameras, you should connect 2 cameras to first controller and 2 cameras to second controller. Device Manager usually shows USB cameras as “USB Composite Device” at the “Advanced” tab of USB controller properties. If you have too many cameras connected to single USB 2.0 controller, you should re-plug them to different USB ports. +

USB hubs

+

If several devices are connected on the USB hub and working in parallel, then the USB bandwidth is shared among the devices. Connecting all cameras via single USB 2.0 hub is OK only for low resolution video recording (320x240). To be able to record video at 640x480 and 60 frames per second, you should avoid connecting more than 2 cameras via single USB controller. +

Potential issues with USB controllers

+

Some USB chips have compatibility issues with specific cameras which may result in not detecting cameras, low frame rate, etc. See the next section on known issues. +

Many modern laptops have one of the integrated USB 2.0 controllers reserved for docking station. That may effectively halve your USB bandwidth. So laptops with a large number of USB ports (4 or more) are preferrable. Desktop PCs should not have such problem. +

Important. Before buying a specific laptop model it is strongly recommended to ensure it has the sufficient number of available built-in USB controllers and they are compatibile with cameras you intend to use. Find it at your local retailer, and bring cameras with you to test. Though it may be easier if your friend/neighbour has such a laptop. Or ask at our forum, maybe someone already used this model with our software or can recommend specific model for your camera configuration. +

If you plan getting an additional USB controller, please be aware of potential problems with USB 2.0 controllers available on the market. Many USB 2.0 controllers are based on defective NEC USB 2.0 chip, not capable of standard USB 2.0 speeds. That's a well-known bug specific only to NEC USB 2.0 chip. By contrast, all NEC USB 3.0 chips are good and do not have this problem. +

Some manufacturers sell what they call a "USB 2.0 ExpressCard" controller for laptops. Many of such ExpressCard controllers are in fact internal USB hubs in ExpressCard form factor. That means they do not add actual USB bandwidth to the system. All USB 3.0 ExpressCard are controllers. +

If you plan getting additional high-speed USB controller, we strongly recommend that you get a USB 3.0 controller, just to be on the safe side. +

When setting up a USB 3.0 controller, make sure you configure it to handle USB 2.0 traffic (as opposed to passing USB 2.0 to integrated USB 2.0 controllers in your chipset). There should be a corresponding setting in driver properties or in BIOS. +

Known compatibility issues

+

Combinations of camera & USB chip listed below have known compatibility issues. Unfortunately, there is no guarantee that other combinations will work for sure, but most will. If you need an additional USB controller for connecting specific camera, be sure not to get the one based on incompatible USB chip. +

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Camera +USB chip +Symptoms +Comments / possible solution +
ASUS Xtion (Live) +NEC µPD720200 (USB3) +Camera is not properly recognized by the system. +This firmware update makes device behave the same as PrimeSense Carmine 1.08 with this controller. +
ASUS Xtion (Live) +VIA VT6212L (USB2) +Frame drops of color picture in 640x480@30 (depth + color). + +
PrimeSense Carmine 1.08 +NEC µPD720200 (USB3) +Frame drops in 640x480@30 (depth + color). +Stable work in 640x480@30 (depth only) and 320x240 (depth + color). +
PrimeSense Carmine 1.08 +VIA VT6212L (USB2) +Frame drops in 640x480@30 (depth + color). +Stable work in 640x480@30 (depth only) and 320x240 (depth + color). +
PrimeSense Carmine 1.08 +VIA VL800(USB3) +Frame drops, reduced frame rate, freezes in depth + color modes. +Stable work in depth only modes. +
Sony Playstation Eye +VIA VL800 (USB3) +Great number of bad frames and frame drops. Camera is invisible to iPi Recorder. + +
Sony Playstation Eye +VIA VT6212L (USB2) +Great number of frame drop in 640x480@60 mode. +2 cameras can work smoothly at lower FPS (up to 640x480@50). +
+ + +
+ +
+
+
+
+
+ + + + + + + + + diff --git a/research/visual-spatial-map/mirrors/kerbl-2023-3d-gaussian-splatting.pdf b/research/visual-spatial-map/mirrors/kerbl-2023-3d-gaussian-splatting.pdf new file mode 100644 index 0000000..50f6ef5 Binary files /dev/null and b/research/visual-spatial-map/mirrors/kerbl-2023-3d-gaussian-splatting.pdf differ diff --git a/research/visual-spatial-map/mirrors/krekovic-2015-audio-visual-geometry-calibration.pdf b/research/visual-spatial-map/mirrors/krekovic-2015-audio-visual-geometry-calibration.pdf new file mode 100644 index 0000000..14846ec Binary files /dev/null and b/research/visual-spatial-map/mirrors/krekovic-2015-audio-visual-geometry-calibration.pdf differ diff --git a/research/visual-spatial-map/mirrors/leapuvc-github.html b/research/visual-spatial-map/mirrors/leapuvc-github.html new file mode 100644 index 0000000..eb9529a --- /dev/null +++ b/research/visual-spatial-map/mirrors/leapuvc-github.html @@ -0,0 +1,1737 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + GitHub - leapmotion/leapuvc: Examples in Python, Matlab, and C for interpreting data from UVC-Enabled Leap Motion Peripherals · GitHub + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+
+ + +
+ Skip to content + + + + + + + + + + + +
+
+ + + + + + + + + + + + + + + + + + + +
+ +
+ + + + + + + + +
+ + + + + +
+ + + + + + + + + +
+
+
+ + + + + + + + + + + + + +
+ +
+ +
+ +
+ + + + / + + leapuvc + + + Public +
+ + +
+ +
+ + +
+
+ +
+
+ + + + +
+ + + + + +
+ + + + + + + + + + + + + + + + + + +
+ + + + +

leapmotion/leapuvc

+ + + +

Repository files navigation

leapuvc

+

LeapUVC gives you access to the Leap Motion Controller image data through the industry standard UVC (Universal Video Class) interface. This gives you low level controls such as LED brightness, gamma, exposure, gain, resolution, and more.

+

The LeapUVC release features examples in C, Python, and Matlab, as well as OpenCV bindings that show how to stream from multiple devices, track ArUco markers, change camera settings, grab lens distortion parameters, and compute stereo depth maps from the images. Use the Leap Motion Controller to track physical objects, capture high-speed infrared footage, or see the world in new ways.

+

The bundled firmware updating binaries, which are required to unlock LeapUVC access, must be executed on a Windows machine. The included Python Examples have been tested on Windows and Linux; Mac is not currently supported. Since this is experimental software, use it at your own risk.

+

With this experimental release we hope to open up entirely new use cases for the Leap Motion Controller in education, robotics, art, academic research, and more.

+

License

+

These examples are licensed for use under the terms of the Leap Motion Developer SDK agreement.

+
+ +
+
+
+
+

About

+ +

+ Examples in Python, Matlab, and C for interpreting data from UVC-Enabled Leap Motion Peripherals +

+ + +

Resources

+ + + +

License

+ + + + + + + + + + + + + + + + +

Stars

+ + +

Watchers

+ + +

Forks

+ + + + +
+ +
+
+ + +
+
+

+ Releases

+ +
No releases published
+ +
+
+ + + + +
+
+ + + +

+ Packages +

+ + +
+
+
 
+
+
+
+
 
+
+
+
+
 
+
+ + + + +
+
+
+ + + +
+
+ + +

+ Contributors

+ +
    +
  • +
    +
  • +
  • +
    +
  • +
  • +
    +
  • +
+ + +
+
+
+ + +
+
+

Languages

+
+ + + + +
+ + +
+
+ + +
+
+
+ + + + +
+ +
+ +
+
+ +
+ +
+

Footer

+ + + + +
+
+ + + + + © 2026 GitHub, Inc. + +
+ + +
+
+ + + + + + + + + + + + + + + + + + + + +
+
+
+ + + diff --git a/research/visual-spatial-map/mirrors/leapuvc-manual.pdf b/research/visual-spatial-map/mirrors/leapuvc-manual.pdf new file mode 100644 index 0000000..6be8bab Binary files /dev/null and b/research/visual-spatial-map/mirrors/leapuvc-manual.pdf differ diff --git a/research/visual-spatial-map/mirrors/leapuvc-readme.md b/research/visual-spatial-map/mirrors/leapuvc-readme.md new file mode 100644 index 0000000..698928f --- /dev/null +++ b/research/visual-spatial-map/mirrors/leapuvc-readme.md @@ -0,0 +1,15 @@ +# leapuvc + +LeapUVC gives you access to the Leap Motion Controller image data through the industry standard UVC (Universal Video Class) interface. This gives you low level controls such as LED brightness, gamma, exposure, gain, resolution, and more. + +The LeapUVC release features examples in C, Python, and Matlab, as well as OpenCV bindings that show how to stream from multiple devices, track ArUco markers, change camera settings, grab lens distortion parameters, and compute stereo depth maps from the images. Use the Leap Motion Controller to track physical objects, capture high-speed infrared footage, or see the world in new ways. + +The bundled firmware updating binaries, which are required to unlock LeapUVC access, must be executed on a Windows machine. The included Python Examples have been tested on Windows and Linux; Mac is not currently supported. Since this is experimental software, use it at your own risk. + +With this experimental release we hope to open up entirely new use cases for the Leap Motion Controller in education, robotics, art, academic research, and more. + +## License + +These examples are licensed for use under the terms of the [Leap Motion Developer SDK agreement][sdkagreement]. + +[sdkagreement]: https://developer.leapmotion.com/sdk_agreement "Leap Motion Developer SDK Agreement" diff --git a/research/visual-spatial-map/mirrors/multi-camera-calibration-docs.html b/research/visual-spatial-map/mirrors/multi-camera-calibration-docs.html new file mode 100644 index 0000000..a38cb69 --- /dev/null +++ b/research/visual-spatial-map/mirrors/multi-camera-calibration-docs.html @@ -0,0 +1,219 @@ + + + + + + + + + + Multi-Camera Calibration — Multi-camera Calibration documentation + + + + + + + + + + + + + + + + + + +
+ + +
+ +
+
+
+ +
+
+
+
+ +
+

Multi-Camera Calibration

+

This package provides a pipeline for multi-camera calibration, as well as general geometry utilities like keypoint triangulation and reprojection. There are several other packages include their own calibration pipelines, including anipose, FreiPose and JARVIS. We created a new library for multicamera calibration because we wanted:

+
    +
  • a standalone calibration solution that can be mixed and matched with other tools

  • +
  • a modular, extendable and well-documented code base

  • +
  • a pedagogical pipeline that exposes each step of the process

  • +
  • the ability to register camera that point in opposite directions

  • +
+
+

What is calibration?

+

Calibration is the process of measuring the intrinsic and extrinsic parameters of two or more cameras that are being used to track things in 3D. Intrinsics are private to each camera and include parameters like focal length and radial distortion that determine the relationship between pixels in the camera image and lines in 3D space. Extrinsics capture the relative translations and rotations between cameras (see diagram). For more details, see Under the hood.

+_images/setup_intrinsics_extrinsics.jpg +
+

+
+
+
+

Who should use this tool?

+

Anyone using multiple cameras (or a single camera with mirrors) to track stuff in 3D can use this package. The main use case we had in mind is 3D pose tracking for animal behavior research. We include import/export functions for a variety of 3D tracking packages, including JARVIS and GIMBAL. In fact, our pipeline can function as a drop-in replacement for the calibration steps in JARVIS or anipose. If would like to integrate multicam_calibration into your pipeline, feel free to open a github issue!

+ + +
+

Under the hood

+ +
+ + +
+
+ + +
+
+ +
+
+
+
+ + + + \ No newline at end of file diff --git a/research/visual-spatial-map/mirrors/opencv-multi-camera-calibration.html b/research/visual-spatial-map/mirrors/opencv-multi-camera-calibration.html new file mode 100644 index 0000000..f5067f1 --- /dev/null +++ b/research/visual-spatial-map/mirrors/opencv-multi-camera-calibration.html @@ -0,0 +1,150 @@ + + + + + + + +OpenCV: Multi-camera Calibration + + + + + + + + + + + + + +
+
+ + + + + + + + + + + +
+
OpenCV +  3.4.20-dev +
+
Open Source Computer Vision
+
+
+ + + + + + + + +
+
+ + +
+ +
+ + +
+
+
+
Multi-camera Calibration
+
+
+

This tutorial will show how to use the multiple camera calibration toolbox. This toolbox is based on the usage of "random" pattern calibration object, so the tutorial is mainly two parts: an introduction to "random" pattern and multiple camera calibration.

+

Random Pattern Calibration Object

+

The random pattern is an image that is randomly generated. It is "random" so that it has many feature points. After generating it, one print it out and use it as a calibration object. The following two images are random pattern and a photo taken for it.

+
+random_pattern.jpg +
+image
+
+pattern_img.jpg +
+image
+

To generate a random pattern, use the class cv::randpattern::RandomPatternGenerator in ccalib module. Run it as

cv::randpattern::RandomPatternGenerator generator(width, height);
generator.generatePattern();
pattern = generator.getPattern();

Here width and height are width and height of pattern image. After getting the pattern, print it out and take some photos of it.

+

Now we can use these images to calibrate camera. First, objectPoints and imagePoints need to be detected. Use class cv::randpattern::RandomPatternCornerFinder to detect them. A sample code can be

cv::randpattern::RandomPatternCornerFinder finder(patternWidth, patternHeight, nMiniMatches);
finder.loadPattern(pattern);
finder.computeObjectImagePoints(vecImg);
vector<Mat> objectPoints = finder.getObjectPoints();
vector<Mat> imagePoints = finder.getImagePoints();

Here variable patternWidth and patternHeight are physical pattern width and height with some user defined unit. vecImg is a vector of images that stores calibration images.

+

Second, use calibration functions like cv::calibrateCamera or cv::omnidir::calibrate to calibrate camera.

+

Multiple Cameras Calibration

+

Now we move to multiple camera calibration, so far this toolbox must use random pattern object.

+

To calibrate multiple cameras, we first need to take some photos of random pattern. Of cause, to calibrate the extrinsic parameters, one pattern need to be viewed by multiple cameras (at least two) at the same time. Another thing is that to help the program know which camera and which pattern the photo is taken, the image file should be named as "cameraIdx-timestamp.*". Photos with same timestamp means that they are the same object taken by several cameras. In addition, cameraIdx should start from 0. Some examples of files names are "0-129.png", "0-187.png", "1-187", "2-129".

+

Then, we can run multiple cameras calibration as

cv::multicalib::MultiCameraCalibration multiCalib(cameraType, nCamera, inputFilename,patternWidth, patternHeight, showFeatureExtraction, nMiniMatches);
multiCalib.run();
multiCalib.writeParameters(outputFilename);

Here cameraType indicates the camera type, multicalib::MultiCameraCalibration::PINHOLE and multicalib::MultiCameraCalibration::OMNIDIRECTIONAL are supported. For omnidirectional camera, you can refer to cv::omnidir module for detail. nCamera is the number of camers. inputFilename is the name of a file generated by imagelist_creator from opencv/sample. It stores names of random pattern and calibration images, the first file name is the name of random pattern. patternWidth and patternHeight are physical width and height of pattern. showFeatureExtraction is a flags to indicate whether show feature extraction process. nMiniMatches is a minimal points that should be detected in each frame, otherwise this frame will be abandoned. outputFilename is a xml file name to store parameters.

+
+ + + + + + + diff --git a/research/visual-spatial-map/mirrors/opentrack-ps3-eye-open-driver-instructions.html b/research/visual-spatial-map/mirrors/opentrack-ps3-eye-open-driver-instructions.html new file mode 100644 index 0000000..c6b796e --- /dev/null +++ b/research/visual-spatial-map/mirrors/opentrack-ps3-eye-open-driver-instructions.html @@ -0,0 +1,3050 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + PS3 Eye open driver instructions · opentrack/opentrack Wiki · GitHub + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+
+ + +
+ Skip to content + + + + + + + + + + + +
+
+ + + + + + + + + + + + + + + + + + + +
+ +
+ + + + + + + + +
+ + + + + +
+ + + + + + + + + +
+
+
+ + + + + + + + + + + + + + + + + + + + +
+ + + + + + +
+
+

PS3 Eye open driver instructions

+ + +
+ +
+ Gildas edited this page Sep 19, 2022 + · + 23 revisions +
+ +
+
+
+
+

PS3 Eye open driver

+

opentrack now has a standalone driver for the PS3 Eye, and it's mature enough for its existence to be advertised more widely. Now you don't have to pay for (or pirate, for that matter) the CL Eye driver that's buggy, slow, unmaintained and broken with every other Windows 10 update. The crashes are gone and there's less CPU usage. It's also free as in beer.

+
    +
  • Uninstall any previous driver.
  • +
  • Use the zadig program <https://zadig.akeo.ie> to install the libusb-win32 driver onto interface 0 of the camera. Don't use the WinUSB driver!
  • +
  • If it fails to work, try connecting the camera onto a USB 2.0 or a USB 3.0 controller that you haven't used yet. Then run zadig again.
  • +
  • In Device Manager, make sure the driver is assigned to the device. If it's not, use the "update driver" button to select libusb (it's its own category).
  • +
  • If it STILL fails to work, try the procedure again, except this time installing libusbK instead of libusb-win32
  • +
  • If it doesn't work despite all that, switch to libusb-win32 again (see this comment).
  • +
+ +
+ +
+
+
+ + + +

Clone this wiki locally

+
+ + + + + + +
+
+ +
+
+
+
+ +
+
+ + +
+ +
+ + + +
+
+ +
+ +
+

Footer

+ + + + +
+
+ + + + + © 2026 GitHub, Inc. + +
+ + +
+
+ + + + + + + + + + + + + + + + + + + + +
+
+
+ + + diff --git a/research/visual-spatial-map/mirrors/peng-2024-rtg-slam.pdf b/research/visual-spatial-map/mirrors/peng-2024-rtg-slam.pdf new file mode 100644 index 0000000..1bed71b Binary files /dev/null and b/research/visual-spatial-map/mirrors/peng-2024-rtg-slam.pdf differ diff --git a/research/visual-spatial-map/mirrors/ps2-eyetoy-devwiki.html b/research/visual-spatial-map/mirrors/ps2-eyetoy-devwiki.html new file mode 100644 index 0000000..5a3d389 --- /dev/null +++ b/research/visual-spatial-map/mirrors/ps2-eyetoy-devwiki.html @@ -0,0 +1,378 @@ + + + + + +EyeToy - PS2 Developer wiki + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+
+
+

EyeToy

+
+
From PS2 Developer wiki
+
+
+ +
+ Jump to navigation + Jump to search +
+ +

Overview[edit | edit source]

+

The EyeToy is essentially just a normal USB 1.1 camera in a case that matches the PS2's design language. It was quite popular and many games either optionally support or even require it. Several variants have been made by different OEMs. +

+

Variants[edit | edit source]

+

SLEH-00030 and SLEH-00031 follow the fat PS2's design language and were only available in black. SCEH-0004 follows the slim PS2s design language and was only available in silver. +

+

SLEH-00030[edit | edit source]

+

Manufactured by Logitech. +

+

SLEH-00031[edit | edit source]

+

Manufactured by Namtai. +

+

SCEH-0004[edit | edit source]

+

Redesigned. Smaller, silver case. Manufactured by Namtai or Chicony (same model number for both manufacturers). +

+ + + + +
+
+ +
+
+
+
Cookies help us deliver our services. By using our services, you agree to our use of cookies.
+
+ +
+

Navigation menu

+
+ + + + +
+ + + + + + + + +
+
+ + + + + + + + + + + +
+
+ + + + +
+ + + + + + + + + \ No newline at end of file diff --git a/research/visual-spatial-map/mirrors/ps3-playstation-eye-devwiki.html b/research/visual-spatial-map/mirrors/ps3-playstation-eye-devwiki.html new file mode 100644 index 0000000..36d6e9f --- /dev/null +++ b/research/visual-spatial-map/mirrors/ps3-playstation-eye-devwiki.html @@ -0,0 +1,482 @@ + + + + + +PlayStation Eye - PS3 Developer wiki + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+
+
+

PlayStation Eye

+
+
From PS3 Developer wiki
+
+
+ +
+ Jump to navigation + Jump to search +
PlayStation Eye
+ + +

PlayStation Eye[edit | edit source]

+

many disassembly pictures @ http://codelaboratories.com/research/view/ps3-eye-disassembly +


+CEJH-15001 (SCPH-98047, SLEH-00448, SLEH-00201, SLEH-00203, )
+DC 5V, 500mA
+

+

Video and Audio[edit | edit source]

+

Video[edit | edit source]

+
  • 640×480 pixels @ 60 Hz
  • +
  • 320×240 pixels @ 120 Hz
+

Audio[edit | edit source]

+
  • 16-bit audio @ 48 kHz
  • +
  • SNR (sound to noise ratio) @ 90 decibles
+

Gallery[edit | edit source]

+ +

PS3Eye multicam driver[edit | edit source]

+

http://nuigroup.com/forums/viewthread/6903/ +

+

PS3 Eye Streaming with Raspberry[edit | edit source]

+

http://www.raspberrypi.org/phpBB3/viewtopic.php?f=26&t=7397 +

+

Firmware related files[edit | edit source]

+

The camera is supported using the AVC library +

+ + + + + + + + + + + + + + + + + + + + + + + + + + +
dev_flash\vsh\resource\avc\ec\cfg +
FilenameVersionSizeMD5Remarks +
ps3eyetoy.cfg1.00-1.025440e79f611e83108028a2f6b5a40cf83939content: http://pastie.org/10227111 +
micarray.cfg1.00-1.025594299e8e01a3c50961bf79e8948ee51310content: http://pastie.org/10227109 +
array_es7.mod1.00-1.0226648bf6096054995f5c6d799883d7dfdf933 +

Note: no variants and no longer present in versions above 1.02 +


+

+ + + + + + +
+
+ +
+
+
+
Cookies help us deliver our services. By using our services, you agree to our use of cookies.
+
+ +
+

Navigation menu

+
+ + + +
+ + + + + + +
+
+ + + + + + + + + +
+
+ +
+ + + + + + + + + + + + + + +
+ +
+ + + + + + + + + \ No newline at end of file diff --git a/research/visual-spatial-map/mirrors/psmoveapi-camera-docs.html b/research/visual-spatial-map/mirrors/psmoveapi-camera-docs.html new file mode 100644 index 0000000..a9eb893 --- /dev/null +++ b/research/visual-spatial-map/mirrors/psmoveapi-camera-docs.html @@ -0,0 +1,194 @@ + + + + + + + + Tracking and Camera — PS Move API 4.0.12 documentation + + + + + + + + + + + + + + +
+
+
+
+ +
+

Tracking and Camera

+

This page contains information about the camera integration, especially +as it relates to tracking and the different variants for different OSes.

+
+

PS3 Camera

+

This camera is always supported on Linux, and supported on macOS and Windows +when using the PS3EYEDriver camera control driver.

+

On Linux, the kernel includes built-in support for the camera using the +ov534 driver, so the V4L2 camera control driver also support the PS3 Eye.

+

The camera needs a USB 2.0 port.

+

On Windows, you need to install Zadig (WinUSB driver) for the +“USBCamera-B4.09.24.1” (or similar) device so that the PS3EYEDriver +can access the camera using libusb. This has been tested with the +“WinUSB (v6.1.7600.16385)” driver and Zadig 2.7.

+
    +
  • Tested models: SLEH-00448

  • +
  • No firmware upload necessary

  • +
+

Supported resolutions:

+
    +
  • 640x480 @ 60 Hz (default)

  • +
  • 320x240 @ 187 Hz

  • +
+
+
+

PS4 Camera

+

This camera is currently supported on Linux when using the V4L2 camera control +driver, and macOS when using the AVFoundation camera control driver.

+

You need to upload a firmware binary blob using the psmove camera-firmware command.

+

The camera needs a USB 3.0 port.

+

If you have a PSVR kit, you can order the Camera Adaptor (to convert +PS4 AUX to USB 3.0) from Sony for free. You only need to enter the serial +number of your PSVR kit during the ordering process, no need for a PS5. +If you only have the camera but no PSVR, you can look to buy the +adaptor online. The official Sony one is CFI-ZAA1.

+

The official PSVR camera adaptor for PS5 (CFI-ZAA1) will be detected +automatically. If you are using a homemade PS4 AUX-to-USB adapter, use +psmove camera-firmware --force-ps4 to detect any OV580 in boot mode +as PS4 (otherwise it’s assumed that the camera is a PS5 camera, as there +is no way yet to distinguish the PS4 and PS5 cameras from the USB +descriptor in boot mode only, which is probably why the CFI-ZAA1 appears +as USB parent device during enumeration).

+

After the firmware upload the camera is supported using +the uvcvideo kernel driver in OpenCV/V4L2 (Linux) +and UVCService on OpenCV/AVFoundation (macOS).

+
    +
  • Tested models: CUH-ZEY2 (“round model v2”)

  • +
  • Tested firmware SHA-1: fe86162309518a0ffe267075a2fcf728c5856b3e

  • +
  • Firmware search path: /etc/psmoveapi/camera-firmware-ps4.bin, $HOME/.psmoveapi/camera-firmware-ps4.bin

  • +
+

Supported resolutions:

+
    +
  • 1280x800 @ 60 Hz (default)

  • +
  • 640x400 @ 120 Hz

  • +
  • 320x192 @ 240 Hz

  • +
+
+
+

PS5 Camera

+

This camera is currently supported on Linux when using the V4L2 camera control +driver, and macOS when using the AVFoundation camera control driver.

+

You need to upload a firmware binary blob using the psmove camera-firmware command.

+

The camera needs a USB 3.0 port.

+

After the firmware upload the camera is supported using +the uvcvideo kernel driver in OpenCV/V4L2 (Linux) +and UVCService on OpenCV/AVFoundation (macOS).

+
    +
  • Tested models: CFI-ZEY1

  • +
  • Tested firmware SHA-1: 0fa4da31a12b662a9a80abc8b84932770df8f7e1

  • +
  • Firmware search path: /etc/psmoveapi/camera-firmware-ps5.bin, $HOME/.psmoveapi/camera-firmware-ps5.bin

  • +
+

Supported resolutions:

+
    +
  • 1920x1080 @ 30 Hz

  • +
  • 1280x800 @ 60 Hz (default)

  • +
  • 960x520 @ 60 Hz

  • +
  • 640x376 @ 120 Hz

  • +
  • 448x256 @ 120 Hz

  • +
  • 320x184 @ 240 Hz

  • +
+
+
+ + +
+
+
+
+ +
+
+ + + + \ No newline at end of file diff --git a/research/visual-spatial-map/mirrors/psmoveservice-ps3eyedriver-github.html b/research/visual-spatial-map/mirrors/psmoveservice-ps3eyedriver-github.html new file mode 100644 index 0000000..f4f4f0e --- /dev/null +++ b/research/visual-spatial-map/mirrors/psmoveservice-ps3eyedriver-github.html @@ -0,0 +1,1731 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + GitHub - psmoveservice/PS3EYEDriver: PS3EYE Camera Driver for OSX and Windows · GitHub + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+
+ + +
+ Skip to content + + + + + + + + + + + +
+
+ + + + + + + + + + + + + + + + + + + +
+ +
+ + + + + + + + +
+ + + + + +
+ + + + + + + + + +
+
+
+ + + + + + +
+ This repository was archived by the owner on May 3, 2021. It is now read-only. +
+ + + + + + + + +
+ +
+ +
+ +
+ + + + / + + PS3EYEDriver + + + Public archive +
+ + forked from inspirit/PS3EYEDriver + + + +
+ +
+ + +
+
+ +
+
+ + + + +
+ + + + + +
+ + + + + + + + + + + + + + + + + + +
+ + + + +

psmoveservice/PS3EYEDriver

+ + + +
 
 

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

39 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

PS3EYE Driver

+

Some driver like routines to connect and work with Sony PS3 EYE Camera.

+

The driver is using libusb to handle USB communications.

+

Most of the code was ported from Linux kernel source.

+

You can find example code for Cinder and openFrameworks in corresponding folders.

+
+

alt text +alt text

+
+ +
+
+
+
+

About

+ +

+ PS3EYE Camera Driver for OSX and Windows +

+ + +

Resources

+ + + + + + + + + + + + + + + + + +

Stars

+ + +

Watchers

+ + +

Forks

+ + + + +
+ +
+
+ + +
+
+

+ Releases

+ +
No releases published
+ +
+
+ + + + +
+
+ + + +

+ Packages +

+ + +
+
+
 
+
+
+
+
 
+
+
+
+
 
+
+ + + + +
+
+
+ + + +
+
+ + +

+ Contributors

+ +
    +
  • +
    +
  • +
  • +
    +
  • +
  • +
    +
  • +
+ + +
+
+
+ + +
+
+

Languages

+
+ + + + +
+
    +
  • + + + C + 49.5% + +
  • +
  • + + + C++ + 49.4% + +
  • +
  • + + + Makefile + 1.1% + +
  • +
+ +
+
+ + +
+
+
+ + + + +
+ +
+ +
+
+ +
+ +
+

Footer

+ + + + +
+
+ + + + + © 2026 GitHub, Inc. + +
+ + +
+
+ + + + + + + + + + + + + + + + + + + + +
+
+
+ + + diff --git a/research/visual-spatial-map/mirrors/psmoveservice-pseye-software-setup-windows.html b/research/visual-spatial-map/mirrors/psmoveservice-pseye-software-setup-windows.html new file mode 100644 index 0000000..f7d70d8 --- /dev/null +++ b/research/visual-spatial-map/mirrors/psmoveservice-pseye-software-setup-windows.html @@ -0,0 +1,325 @@ + + + + PSEye Software Setup (Windows) - psmoveservice/PSMoveService GitHub Wiki + + + + + + + + + + + +
+ + +
+

PSEye Software Setup (Windows) - psmoveservice/PSMoveService GitHub Wiki

+

Overview

+

There are many different ways to get the PS Eye working on a Windows PC. This is pretty confusing, so expect to be confused.

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
DriverCostNotes
IPISoft PS3EYEDriverFreeSupports multiple cameras. Applications like ex. skype cant see the cameras. The most easy PSEYE driver setup !
PS3EYEDriverFreeWraps libusb. Open source and continues to be worked on so sometimes can be flaky. Supports as many cameras as there is bandwidth for. Instructions are below.
CL Eye Platform Driver$2.99 USDExposes the camera as a DirectShow object so can be used easily by this and other software (e.g. Skype, I think, no guarantees). Only supports one camera.
CL Eye Platform SDK$4.99 USDSupports multiple cameras. Only works if you use your own (non-redistributable, developer version) CLEyeMulticam.dll. Delete the dll that comes with PSMoveService to use the DLL that gets installed with the SDK.
CL Eye redistributable + registered camera$?? USD * n_camerasApparently, after you register your cameras, the redistributable CLEyeMulticam.dll that we supply works. I don't think I ever correctly registered my camera so I don't know if it actually works.
PS3EyeDirectShowFreeExposes the camera as a DirectShow object so can be used easily by this and other software (e.g. Skype, I think, no guarantees). https://github.com/jkevin/PS3EyeDirectShow
+

Setup the camera drivers

+

Option A) IPIsoft PSEYE Driver Installation

+

NOTE: If you have already install libusb based drivers, please uninstall those drivers first in Windows device manager

+

This installation option is very simple. Just download the driver for your Windows OS architecture and all PSEYE Cameras should be recognized. IPISoft is software for Motion Capture Full Body system. This PS3Eye driver was not originally intended to be used with PSMoveService (opensource software), though it does appear to work. Therefore, please do not spam the support of IPISoft with questions about this driver. Use at your own risk.

+

Downloads:

+

64-bit OS:

+

http://files.ipisoft.com/drivers/PlayStation3_Eye_iPi-x64.msi

+

32-bit OS:

+

http://files.ipisoft.com/drivers/PlayStation3_Eye_iPi-x86.msi

+

Result in 'Device Manager', tested successfully in Windows 10 x64:

+

+

Source info: +http://docs.ipisoft.com/User_Guide_for_Multiple_PS_Eye_Cameras_Configuration

+

Option B) Libusb PSEYE Driver Installation

+

The instructions below apply to our customized version of the PS3EYEDriver wrapper around libusb.

+

Libusb Driver Pre-Install

+

Download UsbDeview from Nirsoft.net. This is a handy utility for showing all connected usb devices attached to your system and what drivers they are running. Once downloaded, launch usbdeview.exe.

+
    +
  1. After you have plugged in the PS Eye camera you should see entries for the camera and the microphone of the ps3 eye. +
      +
    • The cameras take a lot of USB bandwidth. This is especially a concern if using more than one camera. We have some tips here on how to manage your camera-connections. Also see what Oculus has to say on the matter.
    • +
    +
  2. +
  3. Under the "service name" column for the camera you should see "usbccgp" for the camera device and "usbaudio" for the audio device. +a. If you don't see "usbccgp" for the camera, right-click on the entry with the purple dot and select "uninstall selected devices", after which, unplug the ps3eye camera, and then plug it back in.
  4. +
  5. Close usbdeview
  6. +
+

images/USBCameraPreInstall.jpg

+

Libusb Driver Install

+

Download the Zadig usb utility from here. This is another handy usb utility for installing custom usb drivers. In our case we're going to use it to install the libusb driver over the top of the default Windows driver for the PS Eye camera. Once downloaded, Launch zadig_2.2.exe.

+
    +
  1. From the options menu launch select "List all devices"
  2. +
  3. From the device list drop down select "USB Camera-B4.09.24.1 (Interface 0)"
  4. +
  5. From the driver drop down select "libusb-win32 (v 1.2.6.0)"
  6. +
  7. Click the Install Driver option
  8. +
  9. There should be a few second delay and then it should say "driver installed successfully"
  10. +
+

Note: Computer reboot might be require to complete installation. +Note 2: Having an error message from the timeout expiration might not mean the driver was not install properly.

+

images/ZadigInstall.jpg

+

Libusb Driver Post-Install

+
    +
  1. Launch usbdeview.exe again.
  2. +
  3. You should now see an entry in the "service name" column for the PS3Eye camera that says "libusb0" +a. If you don't see "libusb0" for the camera, right-click on the entry with the purple dot and select "uninstall selected devices", then unplug the ps3eye camera, then plug it back in, and try reinstalling the lib usb driver.
  4. +
  5. Close usbdeview
  6. +
+

images/USBCameraPostInstall.jpg

+

Testing

+

Now that the camera drivers are installed you should be able to run test_camera.exe and see a video feed (640x480/60Hz) for each camera you have plugged in (up to 3 cameras max).

+

If test_camera.exe closes immediately, or the video feed is stuck on a solid color, don't panic. This is sadly a common occurrence, especially on a initial setup. It's quite likely that you have to do some driver shenanigans in zadig to make this work.

+

If that worked the first time, please take a moment to celebrate your good fortune. Then launch PSMoveService.exe. You should see messages about any attached cameras getting opened in the log window.

+

Then launch the PSMoveConfigTool.exe and open the "Tracker Settings" and "Test Video Feed" sub menu. If your camera is set up properly then this should show you the images from the camera at 640x480/60Hz. While you have the video feed up, it's also a good time to grab the move controller and see if the frame encompasses enough of your intended interaction area with the controller.

+

Also when the PS Eye camera is recording you should see the red LED turn on. Sometimes this light will get stuck on in the event of an application crash. Most of the time it won't matter, but you can reset the camera simply by unplugging it and then plugging it back in again.

+

images/Recording.jpg

+

Proceed to the next step

+

PSMove-Bluetooth-Pairing

+
+
+ +
+ + + + \ No newline at end of file diff --git a/research/visual-spatial-map/mirrors/psmoveserviceex-ps3-eye-setup.html b/research/visual-spatial-map/mirrors/psmoveserviceex-ps3-eye-setup.html new file mode 100644 index 0000000..b2ab4c1 --- /dev/null +++ b/research/visual-spatial-map/mirrors/psmoveserviceex-ps3-eye-setup.html @@ -0,0 +1,279 @@ + + + + Setting up the PlayStation 3 Eye Camera - Timocop/PSMoveServiceEx GitHub Wiki + + + + + + + + + + + +
+ + +
+

Setting up the PlayStation 3 Eye Camera - Timocop/PSMoveServiceEx GitHub Wiki

+
+

[!WARNING] +Before connecting or acquiring PlayStation 3 Eye cameras, first take a look at Compatible Devices and Hardware Requirements!

+
+

📷 PlayStation 3 Eye

+

ps3eye

+

🔌 Connection

+

Simply connect the PlayStation 3 Eye with any USB port on your Computer.

+

Do not exceed 5 meters of total cable length (including the 1 meter USB cable of the PlayStation 3 Eye camera), if you are using passive USB extensions cables. If you need to lay out cables over longer distances, its required to get active repeater USB cables or place USB Hub between them to repeat the data signals to avoid signal loss. Externally powered active USB repeater cables are not required as PlayStation 3 Eye cameras can still run at a very low USB voltage (~3.3v).

+

💿 Installation

+

To use the PlayStation 3 Eye camera, you will need LibUSB drivers in order for PSMoveServiceEx to receive image streams and change settings from the camera. You can use PSMoveServiceEx - Virtual Device Manager to install all necessary camera drivers.

+

Navigate to VDM - Service Management and click Install PlayStation Eye Drivers to install the drivers for the PlayStation 3 Eye camera.

+
+

[!NOTE] +Those drivers are not compatible with DirectShow and PlayStation 3 Eye cameras will not show as Webcams in Microsoft Windows. Decreasing the risk of other applications accessing them.

+
+

+

⚙️ Configuration

+

Use PSMoveServiceEx Config Tool to configure the PlayStation 3 Eye camera. You can find all camera settings when calibrating controller or head-mounted display colors in Config Tool > Tracker Settings > Color Calibration

+ + + + + + + + + + + + + + + + + +
Steps
+

🧰 Troubleshooting

+

📌 PSMoveServiceEx Config Tool stream shows 'Tracker timed out.' or has a unstable framerate.

+

This means PSMoveServiceEx did not receive any image from the camera in a while or the image could not be received in time from the camera. +This can be caused by the following reasons:

+
    +
  • USB cable too long (signal loss).
  • +
  • Too many heavy-duty devices connected (such as cameras) to one USB Host Controller causing insufficient bandwidth.
  • +
  • Device disconnected.
  • +
  • Improper configuration (such as too high framerate).
  • +
  • Bad or broken cable.
  • +
+
+

[!NOTE] +Please check Compatible Devices and Hardware Requirements to properly connect your PlayStation 3 Eye camera.

+
+

📌 PlayStation Eye 3 Cameras do not show in PSMoveServiceEx Config Tool.

+

Check if all camera drivers are installed properly as seen in 💿 Installation section of this page or restart PSMoveServiceEx. If they are installed properly and they still wont show up, check troubleshooting section 📌 PlayStation Eye 3 Cameras do not show in PSMoveServiceEx Config Tool..

+
+
+ +
+ + + + \ No newline at end of file diff --git a/research/visual-spatial-map/mirrors/ultraleap-images-example.html b/research/visual-spatial-map/mirrors/ultraleap-images-example.html new file mode 100644 index 0000000..61becb2 --- /dev/null +++ b/research/visual-spatial-map/mirrors/ultraleap-images-example.html @@ -0,0 +1,1056 @@ + + + + + + + + + Images Example - Ultraleap documentation + + + + + + + + + + + + + + + + + + Contents + + + + + + + + + Menu + + + + + + + + Expand + + + + + + Light mode + + + + + + + + + + + + + + Dark mode + + + + + + + Auto light/dark mode + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+ +
+
+ +
+ +
+
+ +
+
+
+
+
+ +
+ +
+
+
+

Images Example

+

The Ultraleap Tracking Service creates the tracking data provided in a LEAP_TRACKING_EVENT() struct by analyzing a set of stereo images taken by the Ultraleap Hand Tracking Camera.

+

You can subscribe to receive the images LeapC is analyzing by setting the eLeapPolicyFlag_Images policy flag. This will cause LeapC to start dispatching LEAP_IMAGE_EVENT messages through the usual message pump.

+

Unsubscribing is achieved by calling LeapSetPolicyFlags again with eLeapPolicyFlag_Images as the “clear” argument.

+
+

ImageSample.c

+

The ImageSample.c program uses the callback mechanism defined by the ExampleConnection described in the other examples. Instead of printing out frame information, though, ImageSample keeps track of the latest images that the LeapC API has provided.

+
+
LeapC allocates the memory for the images that it provides, and will not release the memory for a set of images until:
+
+
+

If the client wants access to the image after this point then they should copy the data into another, user-allocated, buffer.

+

The buffer size needed for a set of images is determined by the image dimensions, the image bytes-per-pixel, and the number of images in the set. Each image’s dimensions are contained in the LEAP_IMAGE_PROPERTIES structure. For a typical image from a Leap Motion Controller, this size is:

+
640 (width) * 240 (height) * 1 (byte-per-pixel) * 2 (images) = 307200 bytes
+
+
+

The full sample code is:

+
#include <stdio.h>
+#include <stdlib.h>
+#include "LeapC.h"
+#include "ExampleConnection.h"
+
+/** Callback for when the connection opens. */
+static void OnConnect(void){
+  printf("Connected.\n");
+}
+
+/** Callback for when a device is found. */
+static void OnDevice(const LEAP_DEVICE_INFO *props){
+  printf("Found device %s.\n", props->serial);
+}
+
+/** Callback for when a frame of tracking data is available. */
+static void OnFrame(const LEAP_TRACKING_EVENT *frame){
+  printf("Frame %lli with %i hands.\n", (long long int)frame->info.frame_id, frame->nHands);
+  for(uint32_t h = 0; h < frame->nHands; h++){
+    LEAP_HAND* hand = &frame->pHands[h];
+    printf("    Hand id %i is a %s hand with position (%f, %f, %f).\n",
+                hand->id,
+                (hand->type == eLeapHandType_Left ? "left" : "right"),
+                hand->palm.position.x,
+                hand->palm.position.y,
+                hand->palm.position.z);
+  }
+}
+
+/** Callback for when an image is available. */
+static void OnImage(const LEAP_IMAGE_EVENT *imageEvent){
+    printf("Received image set for frame %lli with size %lli.\n",
+           (long long int)imageEvent->info.frame_id,
+           (long long int)imageEvent->image[0].properties.width*
+           (long long int)imageEvent->image[0].properties.height*2);
+}
+
+int main(int argc, char** argv) {
+  //Set callback function pointers
+  ConnectionCallbacks.on_connection          = &OnConnect;
+  ConnectionCallbacks.on_device_found        = &OnDevice;
+  ConnectionCallbacks.on_frame               = &OnFrame;
+  ConnectionCallbacks.on_image               = &OnImage;
+
+  LEAP_CONNECTION *connection = OpenConnection();
+  LeapSetPolicyFlags(*connection, eLeapPolicyFlag_Images, 0);
+
+  printf("Press Enter to exit program.\n");
+  getchar();
+  CloseConnection();
+  DestroyConnection();
+  return 0;
+}
+//End-of-Sample
+
+
+

Note that this example does not attempt to display image data graphically.

+
+

Back to top

+
+
+ +
+
+ +
+ +
+
+ + + + + + \ No newline at end of file diff --git a/research/visual-spatial-map/mirrors/ultraleap-leapc-images.html b/research/visual-spatial-map/mirrors/ultraleap-leapc-images.html new file mode 100644 index 0000000..48d64f3 --- /dev/null +++ b/research/visual-spatial-map/mirrors/ultraleap-leapc-images.html @@ -0,0 +1,1062 @@ + + + + + + + + + Images - Ultraleap documentation + + + + + + + + + + + + + + + + + + Contents + + + + + + + + + Menu + + + + + + + + Expand + + + + + + Light mode + + + + + + + + + + + + + + Dark mode + + + + + + + Auto light/dark mode + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+ +
+
+ +
+ +
+
+ +
+
+
+
+
+ +
+ +
+
+
+

Images

+

Ultraleap Hand Tracking Cameras use infrared stereo cameras as tracking sensors. The LeapC API provides an image buffer containing the sensor brightness values and a distortion buffer containing the camera calibration map. The distortion map can be used to correct lens distortion in the image data. The function LeapRectilinearToPixel() can be used to undistort specific pixels in an image and the function LeapPixelToRectilinear() can be used to find the 3D location corresponding to a matched pair of stereo pixels.

+
+

Using Images

+

Images are sent automatically if the policy for images is set.

+

When a stereo image pair is available, a LEAP_IMAGE_EVENT() is added to the event queue and can be accessed via LeapPollConnection(). The image event contains image properties, distortion correction data, and a pointer to the buffer.

+

Both images from the stereo cameras are written to the same buffer; left image first then right image.

+

To get an image:

+
    +
  1. Open the connection

  2. +
  3. Set image policy

  4. +
  5. Poll for images

  6. +
  7. Continue polling for image frames (optionally also poll for tracking frames and status messages). When an image is available, a LEAP_IMAGE_EVENT() message is provided by LeapPollConnection(). The LEAP_IMAGE_EVENT() struct contains the images, a description of the images, and the distortion map

  8. +
+
+

Image Buffer Size

+

The image buffer size depends on the device type as well as the image type and format. The buffer size depends on the width, height, and pixel format (bytes per pixel) of the image. A single buffer must hold stereo images. For example, if the current images produced by an Ultraleap Hand Tracking Camera device 640 x 240, 1-byte pixels, then the buffer size must be: 640 x 240 x 1 x 2 = 307,200 bytes.

+
+
+
+

Image Distortion

+

When a ray of light enters an Ultraleap Hand Tracking Cameras, the lens bends the ray so that it hits the sensor. The sensor then records a greyscale brightness value at a specific pixel location. No lens is perfect, so a ray of light does not land on the sensor in the optically perfect spot. The distortion map provides data to correct this imperfection, allowing you to calculate the true angle of the original ray of light. You can use the corrected angle to generate a rectified image. Using the angles from both images in the stereo pair, you can triangulate the 3D location of a feature identified in both images.

+

For image rectification, the distortion map can be fed to a shader program that can efficiently interpolate the correction applied to rays of light to produce a texture containing the rectified image. For getting the true angle for a small set of points, you can use the LeapPixelToRectilinear() function (but this is not typically efficient enough to transform a full bitmap at a high frame rate).

+
+

Rectifying Image Points

+

To rectify individual points in an image, you can call the LeapPixelToRectilinear() function. Given a ray from the camera, this function returns the pixel coordinates in the image buffer that contain the light recorded from that direction, corrected for lens distortion.

+

The following code takes a target texture size and determines the correct pixel brightness value. For each pixel, the code computes the ray direction to the point in the scene that would illuminate the target pixel given an ideal optical system. The code then corrects for optical distortion by calling LeapRectilinearToPixel(), which provides the pixel coordinates in the image buffer that actually contains the brightness value for the target pixel. The code copies the brightness value to the target texture buffer. The result is a rectified image.

+
 //The pixel size of the textures we write the undistorted images to
+#define TEX_WIDTH 400
+#define TEX_HEIGHT 400
+//Max field of view varies by device, use 8 for the peripheral, 22 for Rigel
+#define MAX_FOV 22
+
+//image_buffer contains the image data
+for( float row = 0; row < TEX_HEIGHT; row++ ) {
+  for( float col = 0; col < TEX_WIDTH; col++ ) {
+    //Normalize from pixel xy to range [0..1]
+    LEAP_VECTOR input;
+    input.x = col/TEX_WIDTH;
+    input.y = row/TEX_HEIGHT;
+
+    //Convert from normalized [0..1] to ray slopes
+    input.x = (input.x - .5) * MAX_FOV;
+    input.y = (input.y - .5) * MAX_FOV;
+
+    LEAP_VECTOR pixel = LeapRectilinearToPixel(*connection, eLeapPerspectiveType_stereo_left, input);
+    int dindex = (int)floor(row * TEX_WIDTH + col);
+    int pindex = (int)roundf(pixel.y) * image_width + (int)roundf(pixel.x);
+    if(pixel.x >= 0 && pixel.x < image_width && pixel.y >=0 && pixel.y < image_height){
+    undistorted_image_left[dindex] = ((char*)image_buffer)[pindex];
+    } else {
+      undistorted_image_left[dindex] = 128;
+    }
+  }
+}
+
+
+

For the image on the right, remember to offset the index into the image_buffer array since the right image follows the left.

+

You can get the actual device field of view from the LEAP_DEVICE_INFO() struct. Use tan(fov/2) in the calculation above instead of MAX_FOV.

+

Note that LeapRectilinearToPixel() may not be fast enough to rectify the image to a high resolution texture in real-time. Generally, you should limit its use to individual points, image patches, or low-resolution textures. For better performance for full images, you can use a shader to rectify the image.

+
+
+

Rectifying with a Shader

+

To rectify an entire image with a shader, create a texture containing the data in the distortion buffer. The distortion buffer is contained in the distortion_matrix field of the LEAP_IMAGE_EVENT() struct. This data is set up to use a shader’s normal UV interpolation mechanism. You can use the interpolated UV coordinates to look up the corrected brightness in the image texture, as in the following fragment shader program:

+
uniform sampler2D rawTexture;
+uniform sampler2D distortionTexture;
+
+void main()
+{
+  vec2 distortionIndex = texture2D(distortionTexture, gl_TexCoord[0].st).xy;
+  float hIndex = distortionIndex.r;
+  float vIndex = distortionIndex.g;
+
+  if(vIndex > 0.0 && vIndex < 1.0 && hIndex > 0.0 && hIndex < 1.0)
+  {
+    gl_FragColor = vec4(texture2D(rawTexture, distortionIndex).rrr, 1.0);
+  }
+  else
+  {
+    gl_FragColor = vec4(0.2, 0.0, 0.0, 1.0);
+  }
+}
+
+
+

To interpolate correctly, the distortion texture must use the following filter and wrap parameters:

+
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_LINEAR);
+glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_LINEAR);
+glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, GL_CLAMP_TO_EDGE);
+glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_T, GL_CLAMP_TO_EDGE);
+
+
+

The distortion texture must use two floating point values per texel and contain 64 x 64 texels:

+
glTexImage2D(GL_TEXTURE_2D, 0, GL_RG, 64, 64, 0, GL_RG, GL_FLOAT, distortion_buffer_left);
+
+
+

The distortion map itself rarely changes – the only time it changes is if the Ultraleap Hand Tracking Camera is swapped with a different device, or if the device is rotated so that the user’s hands enter from the opposite side of the vertical field of view. In this case the Ultraleap Hand Tracking Software inverts the image, and the distortion map, to automatically maintain the proper orientation. (Auto-orientation can be turned off in the Ultraleap Tracking Service configuration.)

+
+
+

Detecting When the Distortion Map Changes

+

The distortion map contains a grid of values that can be used to rectify the images. Rectification should be performed if you are aligning 3D objects with pixels in the images or if you are performing stereo triangulation on the image pair.

+

Each image complete event contains the distortion map for that image. However, since translating the map to an interpolation array or shader texture can itself be an expensive task, you typically only want to do this when the distortion map actually changes – not once for every image request. The distortion map can only change when the images are flipped due to automatic or manual re-orientation, or if the device itself is changed. You can use the LEAP_IMAGE_EVENT::matrix_version field to detect when the distortion map changes. When the matrix_version changes between images, you know that the distortion map has changed. The matrix version number ALWAYS increases when the distortion map changes and never returns to a prior value.

+
+

Back to top

+
+
+
+ +
+
+ +
+ +
+
+ + + + + + \ No newline at end of file diff --git a/research/visual-spatial-map/mirrors/wu-2024-4d-gaussian-splatting.pdf b/research/visual-spatial-map/mirrors/wu-2024-4d-gaussian-splatting.pdf new file mode 100644 index 0000000..a13c96d Binary files /dev/null and b/research/visual-spatial-map/mirrors/wu-2024-4d-gaussian-splatting.pdf differ diff --git a/research/visual-spatial-map/mirrors/zadig-home.html b/research/visual-spatial-map/mirrors/zadig-home.html new file mode 100644 index 0000000..8821a7d --- /dev/null +++ b/research/visual-spatial-map/mirrors/zadig-home.html @@ -0,0 +1,205 @@ + + + + + Zadig - USB driver installation made easy + + + + + + + + + +
+ + + + +
+ +
+

[Zadig icon] Zadig

+
USB driver installation made easy
+ [Zadig screenshot] +

Zadig is a Windows application that installs generic USB drivers, such as WinUSB, libusb-win32/libusb0.sys or libusbK, to help you access USB devices.

+

It can be especially useful for cases where: +

    +
  • you want to access a device using a libusb-based application
  • +
  • you want to upgrade a generic USB driver
  • +
  • you want to access a device using WinUSB
  • +
+

+

Note: "libusb-based" above means an application that uses either libusb, libusb-win32 or libusbK.

+

Download

+

Updated 2024.06.13:

+

+

System Requirements:

+ Windows 7 or later.
+ Windows XP and Windows Vista are NO LONGER SUPPORTED.

+

Usage

+

Download the executable and run it — no installation is necessary.
If elevation is required, you will be prompted for it.

+

An usage guide for Zadig is available HERE.

+

The executable is digitally signed and the signature should state: "Akeo Consulting"

+ +

Frequently Asked Questions (FAQ)

+

A Zadig FAQ is available HERE.

+

To provide feedback, report a bug or request an enhancement please use the github issue tracker. Or you can send an e-mail.

+

License

+

GNU General Public License (GPL) version 3 or later.
You are free to distribute, modify or even sell the software, insofar as you respect the GPLv3 license.

+

Zadig is based on libwdi which uses an LGPL version 3 or later license. +

The executable is produced in a 100% transparent manner, from its public source, using a Visual Studio environment.

+ +

Changelog

+
    +
  • Version 2.9 (2024.06.13)
      +
    • Add libusb-win32 ARM64 driver installation support (courtesy of Peter Dons Tychsen)
      + Note: Requires Windows 10 1511 or later with x86 (32-bit) emulation support.
    • +
    • Fix UAC not being properly triggered during driver installation (courtesy of Andrew Meyer)
    • +
    • Embedded drivers: WinUSB v6.1.7600.16385, libusb-win32 v1.4.0.0, libusbK v3.1.0.0 & USBSer (native)
    • +
  • +
+

Source Code

+

  • libwdi 1.5.1 source (330 KB). Includes Zadig in the examples directory.
  • +
  • Alternatively, you can clone the git repository using: +
    $ git clone git://github.com/pbatard/libwdi
  • +
  • For more information, see the GitHub project.
+ If you are a developer, you are very much encouraged to tinker with Zadig/libwdi and submit patches.

+
+ +
+ + diff --git a/research/visual-spatial-map/summary.md b/research/visual-spatial-map/summary.md new file mode 100644 index 0000000..9222e15 --- /dev/null +++ b/research/visual-spatial-map/summary.md @@ -0,0 +1,153 @@ +# Visual Spatial Map And Splat Scene + +## Objective + +Extend the AquaSynth/Mimir concept from "one camera view plus spatial audio" into a shared live spatial scene: + +- two Razer Kiyo-class camera/mic rigs provide RGB and audio; one is Kiyo Pro +- two PlayStation Eye / PS3 Eye cameras provide cheap high-rate low-resolution tracking views +- one Leap Motion provides close-range stereo IR/hand/marker sensing +- tracked objects and speakers/mics live in one world coordinate frame +- nice-camera RGB is projected onto reconstructed/tracked geometry +- renderer exposes adjustable virtual camera position +- ambisonic render follows the same listener/camera pose + +## Hardware Sanity + +The high-rate cheap cameras in this rig are **two PlayStation Eye / PS3 Eye** units, not PS2 EyeToys. The PS3 developer wiki lists PlayStation Eye video modes at 640x480 @ 60 Hz and 320x240 @ 120 Hz [PS3 Eye]. Community tracking stacks sometimes expose higher practical modes/settings, but that needs verification against the exact driver and USB topology. The PS2 EyeToy is described as a normal USB 1.1 camera [PS2 EyeToy], so it remains only a rejected naming confusion path. + +Windows driver state matters. The opentrack wiki documents a current open-driver route for PS3 Eye using Zadig to install `libusb-win32` on interface 0, with `libusbK` as a fallback if needed [opentrack PS3 Eye; Zadig]. The old Code Laboratories CL Eye driver exists and advertised DirectShow/high-frame-rate support, but its own page targets Windows XP/Vista/7 era systems [CL Eye]. For this rig, the open-driver path is the default candidate; CL Eye is historical reference material, not a dependency to pirate or preserve. + +The beauty/RGB cameras are **two Razer Kiyos, one of them Kiyo Pro**. Their microphones can participate in the audio system, but camera attachment does not imply audio clock sync. They need the same live adaptive sync treatment as other independent USB audio devices. + +The **Leap Motion** is a separate stereo IR sensor. LeapUVC gives low-level UVC access to Leap Motion image data and controls such as LED brightness, gamma, exposure, gain, resolution, and camera calibration/depth examples [LeapUVC]. Ultraleap's LeapC image API also exposes stereo infrared image buffers and distortion/calibration maps, with functions for rectilinear/pixel mapping [Ultraleap Images]. In this rig, installing the LeapUVC driver means the device shows up as a generic UVC device and is no longer addressable through Ultraleap software in that mode. Treat LeapUVC and Ultraleap tracking as mutually exclusive operating modes unless proven otherwise on the actual machine. + +Cut line: enumerate the actual five visual devices and their capture modes before designing the capture layer. Nostalgia is not a transport protocol, and USB bandwidth is not moved by vibes. + +## Current Mechanism + +The audio research now has four separate authorities: + +- live adaptive mic sync +- feedback/system identification +- playback calibration +- ambisonic scene/rendering + +The visual system needs the same discipline. It should not be bolted onto OBS as "some extra cameras." It needs its own map authority. + +## Core Invariant + +There must be one canonical world coordinate system. + +Every sensor and emitter needs a transform into that world: + +- two PS3 Eye tracking cameras +- two Razer Kiyo-class RGB cameras +- Leap Motion stereo IR sensor +- camera-attached microphones +- standalone microphones +- speakers +- tracked props/markers +- virtual render camera/listener + +Audio and visual pipelines can run at different rates, but they must publish timestamped observations into the same spatial frame. + +## Research Findings + +OpenCV and related tools already cover the geometry spine: multi-camera calibration estimates camera intrinsics/extrinsics, while triangulation and reprojection operate over calibrated camera sets [OpenCV; Multi-camera Calibration]. The `multi-camera-calibration` project is explicitly built for multi-camera 3D tracking workflows and exposes keypoint triangulation and reprojection utilities [Multi-camera Calibration]. + +Jacob and Haeb-Umbach address the exact cross-modal trap: audio and visual self-localization can produce separate coordinate systems with unknown rotation, scale, and translation. Their paper proposes using audio-visual events localized by both microphones and cameras to align acoustic sensors into the visual coordinate system [Jacob/Haeb-Umbach 2015]. This is directly relevant for camera-attached mics and speaker/mic calibration pings that are also visually tracked. + +Gaussian splatting is plausible as a render representation, but it is not the first authority. Kerbl et al. start from sparse points produced during camera calibration and optimize 3D Gaussians for real-time novel-view rendering [Kerbl 2023]. Wu et al. extend Gaussian splatting into dynamic scenes with 4D-GS [Wu 2024]. RTG-SLAM shows real-time reconstruction using Gaussian splatting, but its published shape assumes RGB-D input [Peng 2024]. The PS3 Eyes are not RGB-D; they are marker/pose sensors. Leap can provide close-range stereo IR evidence. The Kiyos provide RGB texture/evidence. + +For audio coupling, IEM SceneRotator confirms the ordinary ambisonic operation: rotate an ambisonic scene by yaw/pitch/roll or quaternion data [IEM SceneRotator]. That covers orientation. Translation/listener movement is more involved and should be represented in the world model before pretending one rotator solves full six-degree navigation. + +## Recommended Architecture + +```mermaid +flowchart TD + A["2 PS3 Eyes: high-rate low-res"] --> B["2D marker detections"] + C["2 Razer Kiyos: RGB + mics"] --> D["RGB frames + camera poses"] + P["Leap Motion: stereo IR"] --> Q["near-field hands/markers/depth cues"] + E["Mic/speaker/audio estimators"] --> F["Audio landmarks and transfer paths"] + G["Calibration observations"] --> H["World model authority"] + B --> H + D --> H + Q --> H + F --> H + H --> I["Tracked 3D points / prop poses"] + H --> J["Sensor and emitter transforms"] + I --> K["Geometry/point cloud/splat update"] + D --> K + K --> L["Free-view renderer"] + J --> M["Ambisonic listener/render pose"] + L --> N["Virtual camera output"] + M --> O["AquaSynth spatial audio output"] +``` + +## Practical Path + +1. Build a calibration rig. + Use a ChArUco/checkerboard or wand for camera intrinsics/extrinsics, then a visible-and-audible marker event for audio-visual alignment. Ping pong balls are good tracking props if they are visually distinctive; retroreflective or lit markers are better if the cameras can be filtered/exposed for them. + +2. Separate camera roles. + PS3 Eyes own low-resolution marker tracking and temporal precision. Kiyos own RGB projection and human-readable video. Leap owns close-range stereo IR/hand/marker evidence. Do not ask the Kiyos to be high-rate trackers, do not ask PS Eyes to be beautiful, and do not ask Leap to see the whole room. + +3. Start with sparse 3D tracking. + Triangulate marker positions from calibrated camera detections. Publish timestamped 3D points/rigid-body poses. This proves the coordinate system before the splat renderer enters wearing sunglasses. + +4. Register audio devices into the same world. + Camera-attached mics inherit an initial transform from their camera rig plus measured mic offset. Standalone mics/speakers need calibration observations. Audio-visual events can align modality-specific maps [Jacob/Haeb-Umbach 2015]. + +5. Add RGB projection. + Project nice-camera RGB onto tracked geometry/point cloud using known camera intrinsics/extrinsics. Begin with colored points or surfels before live Gaussian optimization. + +6. Add splatting as a render layer, not the state owner. + A Gaussian scene can render the current world beautifully, but the canonical state should remain calibrated transforms, timestamps, tracked points, and confidence. Gaussian parameters are a view/render representation. + +7. Couple virtual camera/listener pose to ambisonics. + The adjustable visual camera pose should also define the audio listener pose. Orientation can rotate the ambisonic scene; translation requires source/listener geometry and distance/room modeling. + +## Failure Modes + +- USB bandwidth can dominate before CPU/GPU does, especially with multiple PS Eye cameras. +- LeapUVC driver mode can make the Leap unavailable to Ultraleap software; do not design a runtime that requires both raw LeapUVC frames and Ultraleap hand tracking at the same time. +- Unsynchronized cameras produce triangulation jitter unless timestamps/exposure timing are handled. +- Rolling shutter or auto-exposure can corrupt high-speed marker tracking. +- Ping pong balls are visually cheap but ambiguous if multiple identical balls cross paths. +- Gaussian splatting from sparse/noisy live markers alone will not create a rich scene; it needs RGB observations and/or depth/geometric priors. +- Dynamic 4D splatting is research-grade. A reliable v1 should render sparse points/surfels first. +- Camera-attached microphones still need audio clock/drift handling; physical attachment does not synchronize sample clocks. + +## Cut Line + +Do not begin by building a live Gaussian splat engine. Begin with the world model: + +- enumerate the two PS3 Eyes, two Razer Kiyos, and Leap Motion +- choose Leap operating mode: LeapUVC raw stereo IR or Ultraleap tracking service +- verify frame rates and USB topology +- calibrate intrinsics/extrinsics +- triangulate one marker +- attach one audio source to the same coordinate frame +- drive ambisonic scene orientation from virtual camera/listener pose + +After that spine holds, add RGB projection, then splats. + +## Citations + +- [PS3 Eye] PS3 Developer Wiki, "PlayStation Eye." Mirror: `mirrors/ps3-playstation-eye-devwiki.html`. +- [PS2 EyeToy] PS2 Developer Wiki, "EyeToy." Mirror: `mirrors/ps2-eyetoy-devwiki.html`. +- [iPiSoft] iPiSoft Wiki, "User Guide for Multiple PlayStation Eye Cameras Configuration." Mirror: `mirrors/ipisoft-multiple-playstation-eye-cameras.html`. +- [opentrack PS3 Eye] opentrack Wiki, "PS3 Eye open driver instructions." Mirror: `mirrors/opentrack-ps3-eye-open-driver-instructions.html`. +- [Zadig] Zadig. Mirror: `mirrors/zadig-home.html`. +- [CL Eye] Code Laboratories, "CL Eye Platform Driver." Mirror: `mirrors/code-laboratories-cl-eye-driver.html`. +- [DelanClip] DelanClip, "PS3 Eye Camera Head Tracking - OpenTrack - Open Driver." Mirror: `mirrors/delanclip-ps3-eye-open-driver.html`. +- [LeapUVC] Leap Motion, `leapuvc` repository and manual. Mirrors: `mirrors/leapuvc-github.html`, `mirrors/leapuvc-readme.md`, `mirrors/leapuvc-manual.pdf`. +- [Ultraleap Images] Ultraleap, "Images." Mirror: `mirrors/ultraleap-leapc-images.html`. +- [OpenCV] OpenCV, "Multi-camera Calibration." Mirror: `mirrors/opencv-multi-camera-calibration.html`. +- [Multi-camera Calibration] Multi-camera Calibration documentation. Mirror: `mirrors/multi-camera-calibration-docs.html`. +- [Jacob/Haeb-Umbach 2015] Florian Jacob and Reinhold Haeb-Umbach, "Absolute Geometry Calibration of Distributed Microphone Arrays in an Audio-Visual Sensor Network." Mirror: `mirrors/krekovic-2015-audio-visual-geometry-calibration.pdf`. +- [Kerbl 2023] Bernhard Kerbl et al., "3D Gaussian Splatting for Real-Time Radiance Field Rendering." Mirror: `mirrors/kerbl-2023-3d-gaussian-splatting.pdf`. +- [Wu 2024] Guanjun Wu et al., "4D Gaussian Splatting for Real-Time Dynamic Scene Rendering." Mirror: `mirrors/wu-2024-4d-gaussian-splatting.pdf`. +- [Peng 2024] Zhexi Peng et al., "RTG-SLAM: Real-time 3D Reconstruction at Scale using Gaussian Splatting." Mirror: `mirrors/peng-2024-rtg-slam.pdf`. +- [IEM SceneRotator] IEM Plug-in Suite, "SceneRotator Guide." Mirror: `mirrors/iem-scenerotator-guide.html`. diff --git a/state/evidence.jsonl b/state/evidence.jsonl index 424197d..0d3f72b 100644 --- a/state/evidence.jsonl +++ b/state/evidence.jsonl @@ -1,3 +1,4 @@ {"ts":"2026-05-21T17:31:13+00:00","type":"implementation","status":"ok","note":"MimirVideoFrameDescriptor carries dimensions, pixel format, stride, device timestamp, and optional native/GPU handle; MimirNativeIngestStreamSource can push LeapStereoIr or other camera frames directly into rolling buffers while process-backed ingest remains a network/diagnostic edge."} {"ts":"2026-05-21T17:40:33+00:00","type":"decision","status":"ok","note":"Mimir live camera ingest should talk directly to drivers through IMimirVideoCaptureDriver: LeapUVC/libusb, LeapC images, Media Foundation, DirectShow, or GPU shared textures."} {"ts":"2026-05-21T18:55:00+00:00","type":"implementation","status":"ok","note":"Deleted script-era packages, tests, helper tools, dependency manifests, archived research mirrors, and stale docs from the working tree. The repo's live surface is C# Mimir.Runtime, Mimir.App, native reservoir, PowerShell bridge utilities, Aquarium, and Faust/native DSP."} +{"ts":"2026-05-21T19:05:00+00:00","type":"correction","status":"ok","note":"Restored the research archive as reference-only memory. Research may preserve external examples and failed-path lessons, but live implementation must be rebuilt through direct drivers, native buffers, Aquarium GPU compute, and Faust/native DSP."} diff --git a/state/map.yaml b/state/map.yaml index f9fbafa..924405c 100644 --- a/state/map.yaml +++ b/state/map.yaml @@ -76,9 +76,10 @@ current_status: summary: > The repo owns a C# Mimir app/runtime scaffold, a native reservoir crate, OBS bridge scripts, and persistence surfaces. Script-era packages, tests, - helper tools, dependency manifests, and archived mirrors have been removed from the - working tree. The next useful cut is direct driver ingest, not reviving old - diagnostic machinery. + helper tools, dependency manifests, and stale docs have been removed from + the working tree. Research archives remain as reference-only memory for raw + native/GPU implementation shape. The next useful cut is direct driver + ingest, not reviving old diagnostic machinery. next_action: > Feed Leap stereo IR frames through `IMimirVideoCaptureDriver` into `MimirSynchronizationHub`, measure sustained cadence over the default