Every machine vision system faces a fundamental physical constraint: the ability to resolve fine spatial detail is capped by both the sensor's pixel geometry and the lens's diffraction behavior. Miscalculating either limit leads to under-resolved images, false pass/fail decisions in automated inspection, and wasted capital on optics that exceed — or fall short of — what the sensor can actually exploit.
The Nyquist Frequency & Optics Calculator eliminates guesswork by quantifying these hard physical boundaries. It determines the maximum spatial frequency a given sensor can faithfully capture, evaluates whether the lens aperture introduces diffraction blur that degrades performance below that sensor limit, and translates all results into object-space dimensions so engineers can directly relate pixel-level physics to the real-world features under inspection.
Required Project Parameters
Before performing a system-level resolution assessment, the following design parameters must be established:
- Pixel Pitch ($p$) — The center-to-center distance between adjacent sensor pixels, expressed in micrometers (μm) or mils. This is the most fundamental hardware specification governing sampling resolution. Common industrial sensors range from 2.2 μm (high-resolution area scan) to 12.5 μm (SWIR InGaAs arrays).
- F-Number ($F/\#$) — The lens aperture setting, defined as the ratio of focal length to effective aperture diameter. Lower F-numbers admit more light but may introduce optical aberrations; higher F-numbers improve depth of field but increase diffraction blur. Typical machine vision lenses operate between $f/1.4$ and $f/16$.
- Wavelength ($\lambda$) — The dominant wavelength of the illumination source in nanometers (nm). Shorter wavelengths (e.g., 450 nm blue LED) yield smaller diffraction spots and higher theoretical resolution. Longer wavelengths (e.g., 850 nm near-infrared) produce larger diffraction blur but enable specialized contrast techniques on certain materials.
- Magnification ($M$) — The optical magnification ratio, expressing how object-plane dimensions map onto the sensor plane. A magnification of $0.5\times$ means one millimeter on the target covers two millimeters on the sensor. This parameter is essential for converting all sensor-plane metrics into actionable object-plane dimensions.
Sampling Theory and Diffraction Physics in Precision Imaging
The Nyquist Sampling Criterion Applied to Pixel Arrays
The Nyquist-Shannon sampling theorem states that a signal must be sampled at a rate of at least twice its highest frequency component to avoid aliasing — the creation of spurious low-frequency artifacts. In the spatial domain of a camera sensor, the "sampling interval" is the pixel pitch $p$, and the highest spatial frequency that can be faithfully resolved is the Nyquist frequency $f_N$.
The sensor Nyquist frequency, expressed in line pairs per millimeter, is calculated as:
$$f_N = \frac{1000}{2p}$$
where $p$ is the pixel pitch in micrometers. The factor of $2$ reflects the minimum requirement of two pixels — one for the bright line and one for the dark line — to constitute a single resolved line pair (lp). The factor of $1000$ converts from micrometers to millimeters.
For a standard industrial sensor with $p = 3.45;\mu\text{m}$, this yields $f_N \approx 145;\text{lp/mm}$. Any spatial detail finer than this frequency is physically unresolvable by the sensor, regardless of lens quality or illumination optimization.
Critical consideration for color sensors: The standard formula $f_N = 1/(2p)$ fundamentally assumes a monochrome sensor where every pixel samples the full spectrum. On a color camera with a Bayer filter mosaic, the red and blue photosite sub-arrays are spaced at $2p$ intervals. This effectively halves the Nyquist limit for those individual color channels, meaning the true chromatic resolving power of a Bayer sensor is significantly lower than what the raw pixel pitch would suggest.
The Airy Disk and the Rayleigh Criterion
Even with an optically perfect lens — one free of all geometric aberrations — diffraction imposes an inescapable limit on image sharpness. When light passes through a circular aperture, it does not focus to an infinitely small point but instead forms a characteristic pattern called the Airy disk: a bright central spot surrounded by concentric dark and bright rings.
The diameter of this Airy disk $D_A$ to its first dark ring is given by:
$$D_A = 2.44 \cdot \lambda \cdot F/\#$$
where $\lambda$ is the wavelength in micrometers and $F/\#$ is the lens aperture setting.
The constant 2.44 is not arbitrary. It derives from multiplying $1.22$ by $2$: the value $1.22$ represents the location of the first zero (first dark ring) of the Bessel function of the first kind ($J_1$), which mathematically governs the intensity distribution of electromagnetic waves diffracting through a perfectly circular aperture. Multiplying by $2$ converts the radius to a full diameter. This is formally known as the Rayleigh criterion for angular resolution.
At $\lambda = 550;\text{nm}$ and $F/4.0$, the Airy disk diameter is:
$$D_A = 2.44 \times 0.550 \times 4.0 = 5.37;\mu\text{m}$$
The Absolute Optical Cutoff Frequency
Beyond the Airy disk, a complementary metric describes the lens's resolution in frequency-domain terms. The optical cutoff frequency $f_{\text{cutoff}}$ defines the spatial frequency at which diffraction reduces image contrast to exactly zero percent — no spatial modulation survives:
$$f_{\text{cutoff}} = \frac{1000}{\lambda \cdot F/\#}$$
where $\lambda$ is in micrometers and $f_{\text{cutoff}}$ is expressed in lp/mm.
A critical distinction for practicing engineers: the optical cutoff represents a theoretical boundary of 0% contrast. In production machine vision systems, engineers rarely operate anywhere near this limit. Instead, system qualification relies on Modulation Transfer Function (MTF) data, where the goal is to maintain at least 20% to 50% contrast (denoted MTF20 or MTF50) at the spatial frequency of interest. The cutoff frequency is a ceiling; the practical operating frequency is substantially lower.
Object-Space Translation Through Magnification
All preceding metrics describe performance at the sensor plane. To relate these to the actual size of features being inspected on a physical target, the magnification ratio $M$ bridges the gap:
$$p_{\text{object}} = \frac{p}{M}$$
$$f_{N(\text{object})} = f_N \cdot M$$
At $M = 0.5\times$ with a $3.45;\mu\text{m}$ pixel pitch, each pixel maps to $6.90;\mu\text{m}$ on the object. The sensor Nyquist of $145;\text{lp/mm}$ translates to $72.5;\text{lp/mm}$ in object space — meaning the finest resolvable line pair on the target is approximately $13.8;\mu\text{m}$ wide.
Diagnosing the System Performance Bottleneck
A well-designed machine vision system should be sensor-limited, not optics-limited. The diagnostic criterion is straightforward: compare the Airy disk diameter $D_A$ to the span of two pixels ($2p$).
$$\text{If } D_A > 2p \Rightarrow \textbf{Optics Limited}$$
$$\text{If } D_A \leq 2p \Rightarrow \textbf{Sensor Limited}$$
When the diffraction blur exceeds the two-pixel Nyquist sampling interval, the lens — not the sensor — becomes the resolution bottleneck. In this condition, investing in a higher-resolution (smaller pixel pitch) sensor provides no measurable gain. The corrective action is to reduce $F/\#$, shorten $\lambda$, or accept the diffraction-imposed ceiling.
The Diffraction Blur Ratio quantifies this relationship as a percentage: $(D_A / 2p) \times 100\%$. Values above 100% indicate an optics-limited configuration.
Sensor and Lens Performance Benchmarks Across Industrial Configurations
The following reference tables provide rapid lookup values for common machine vision hardware configurations. All values are computed from first-principles physics using the formulas established above.
Table 1 — Sensor Nyquist Frequency by Pixel Pitch Class
| Sensor Class | Pixel Pitch (μm) | Nyquist Frequency (lp/mm) | Typical Application Domain |
|---|---|---|---|
| High-Resolution Area Scan | 2.2 | 227 | Semiconductor wafer inspection |
| Standard Area Scan (CMOS) | 3.45 | 145 | General-purpose industrial inspection |
| Large-Format Area Scan | 4.5 | 111 | PCB and flat-panel display inspection |
| Line Scan (Standard) | 5.0 | 100 | Web, print, and continuous surface inspection |
| Scientific CMOS (sCMOS) | 6.5 | 77 | Fluorescence microscopy, life sciences |
| SWIR InGaAs Array | 12.5 | 40 | Near-infrared thermal and material sorting |
Table 2 — Airy Disk Diameter (μm) by F-Number and Wavelength
| F-Number | 450 nm (Blue) | 550 nm (Green) | 660 nm (Red) | 850 nm (NIR) |
|---|---|---|---|---|
| f/1.4 | 1.54 | 1.88 | 2.25 | 2.90 |
| f/2.8 | 3.07 | 3.76 | 4.51 | 5.81 |
| f/4.0 | 4.39 | 5.37 | 6.44 | 8.30 |
| f/5.6 | 6.15 | 7.52 | 9.02 | 11.61 |
| f/8.0 | 8.78 | 10.74 | 12.88 | 16.59 |
| f/16 | 17.57 | 21.47 | 25.77 | 33.18 |
Table 3 — Optical Cutoff Frequency (lp/mm) by F-Number and Wavelength
| F-Number | 450 nm (Blue) | 550 nm (Green) | 660 nm (Red) | 850 nm (NIR) |
|---|---|---|---|---|
| f/1.4 | 1587 | 1299 | 1082 | 840 |
| f/2.8 | 794 | 649 | 541 | 420 |
| f/4.0 | 556 | 455 | 379 | 294 |
| f/5.6 | 397 | 325 | 271 | 210 |
| f/8.0 | 278 | 227 | 189 | 147 |
| f/16 | 139 | 114 | 95 | 74 |
Cross-referencing Tables 1 and 3 reveals a critical design insight. A standard $3.45;\mu\text{m}$ sensor has a Nyquist of $145;\text{lp/mm}$. At $f/8.0$ with green illumination, the optical cutoff drops to $227;\text{lp/mm}$ — still above the sensor Nyquist, so the system remains sensor-limited. However, switching to NIR at $850;\text{nm}$ drops the optical cutoff to just $147;\text{lp/mm}$, approaching parity and signaling the onset of diffraction-dominated performance.
How Aperture, Wavelength, and Pixel Pitch Govern Resolving Power
The Aperture Trade-Off: Depth of Field vs. Diffraction
Stopping down the lens (increasing $F/\#$) is often the first instinct when an application demands greater depth of field — the range of object distances that appear acceptably sharp. However, every incremental F-stop narrows the aperture and proportionally enlarges the Airy disk.
Consider a $3.45;\mu\text{m}$ sensor under green ($550;\text{nm}$) illumination. At $f/2.8$, the Airy disk is $3.76;\mu\text{m}$ — well below the two-pixel threshold of $6.90;\mu\text{m}$, yielding a comfortable sensor-limited system. At $f/8.0$, the Airy disk grows to $10.74;\mu\text{m}$, exceeding $2p$ and pushing the system into optics-limited territory. The resolution ceiling is now set by diffraction, not by pixel density.
The practical engineering challenge is to find the optimal F-number where the depth of field requirement is met without the diffraction blur exceeding the sensor's Nyquist sampling interval.
Wavelength Selection as a Resolution Lever
Switching from broadband or red illumination to a shorter-wavelength source provides a direct, physics-based improvement in diffraction-limited resolution. At $f/4.0$, blue LED illumination ($450;\text{nm}$) produces an Airy disk of $4.39;\mu\text{m}$, whereas NIR ($850;\text{nm}$) produces $8.30;\mu\text{m}$ — nearly double the blur diameter.
This makes blue or UV illumination a powerful tool for high-resolution metrology applications where the material's spectral reflectance permits. The optical cutoff at $f/4.0$ jumps from $294;\text{lp/mm}$ with NIR to $556;\text{lp/mm}$ with blue — an 89% increase in the theoretical resolution ceiling.
Monochrome vs. Bayer Sensors in High-Precision Applications
Beyond the per-pixel diffraction considerations, the sensor's spectral architecture imposes an additional resolution constraint that is often underappreciated. On a Bayer-pattern color sensor, only 25% of the pixels capture red, 25% blue, and 50% green. The effective spatial sampling for red and blue channels occurs at a pitch of $2p$, cutting the channel-specific Nyquist frequency in half.
For machine vision applications demanding the highest possible resolving power — such as semiconductor defect inspection or precision dimensional gauging — monochrome sensors paired with narrowband illumination remain the gold standard. This combination eliminates the Bayer interpolation penalty entirely and allows the full pixel pitch to contribute to spatial resolution.
Sub-Pixel Algorithms: Exceeding the Nyquist Limit for Edge Location
While the Nyquist frequency establishes the absolute physical limit for resolving repeating periodic patterns (alternating line pairs), modern machine vision software routinely achieves measurement precision far beyond this constraint for a different class of task: isolated edge location.
Through sub-pixel interpolation — fitting a mathematical model to the intensity gradient across an edge transition — algorithms can locate the position of a single edge to an accuracy of $1/10$th to $1/50$th of a pixel. On a $3.45;\mu\text{m}$ sensor, this corresponds to edge localization precision of $0.35;\mu\text{m}$ to $0.07;\mu\text{m}$.
This capability does not violate Nyquist theory. The Nyquist limit governs the faithful reconstruction of a complete waveform from discrete samples. A single edge is not a periodic signal; it is a step function whose transition point can be estimated with arbitrary precision given sufficient signal-to-noise ratio. The distinction between resolving power (Nyquist-limited) and localization precision (SNR-limited) is essential for correctly specifying a machine vision system's measurement capability.
Frequently Asked Questions
No — in fact, the opposite is frequently the case. While $f/16$ maximizes depth of field, it also maximizes diffraction blur. On a $3.45;\mu\text{m}$ sensor with green illumination, the Airy disk at $f/16$ reaches $21.47;\mu\text{m}$, which is more than three times the two-pixel Nyquist sampling span of $6.90;\mu\text{m}$. The system is deeply optics-limited at this aperture.
The optimal approach is to calculate the minimum $F/\#$ that satisfies the depth-of-field requirement and verify that the resulting Airy disk does not exceed $2p$. For most industrial sensors with pixel pitches between $3;\mu\text{m}$ and $5;\mu\text{m}$, the diffraction crossover typically occurs between $f/4.0$ and $f/8.0$ depending on wavelength. Operating beyond this crossover sacrifices resolving power for depth-of-field margin that may not be required.
The published pixel pitch of a Bayer sensor describes the physical distance between adjacent photosites, but not all photosites sample the same spectral channel. In the standard RGGB Bayer mosaic, the red-sensitive pixels are arranged in a grid with an effective pitch of $2p$, as are the blue pixels. Only the green channel — with double the number of pixels — samples at the native pitch $p$.
This means the effective Nyquist frequency for the red and blue channels is approximately $\frac{1000}{4p}$, half the monochrome value. For a $3.45;\mu\text{m}$ Bayer sensor, the green-channel Nyquist remains $\approx 145;\text{lp/mm}$, but the red and blue channels drop to $\approx 72;\text{lp/mm}$. Demosaicing algorithms partially recover apparent resolution through interpolation, but this reconstructed detail is inferred rather than directly sampled — a critical distinction for metrology applications where measurement traceability is required.
Sub-pixel interpolation does not violate the Nyquist sampling theorem; it addresses a fundamentally different measurement problem. The Nyquist frequency defines the highest periodic spatial frequency — alternating bright and dark line pairs — that can be faithfully resolved without aliasing. It governs resolving power: the ability to distinguish two closely spaced features.
Edge localization is a separate problem. When measuring the position of a single isolated edge, the intensity transition spans multiple pixels, and a mathematical model (typically a Gaussian, sigmoid, or moment-based fit) is applied to the intensity gradient to estimate the edge center with sub-pixel precision. Achievable accuracy is typically $1/10$th to $1/50$th of one pixel, bounded primarily by signal-to-noise ratio (SNR) rather than sampling density.
In practice, this means a vision system with a pixel pitch of $5.0;\mu\text{m}$ and magnification of $1.0\times$ can resolve line pairs no finer than $100;\text{lp/mm}$ (Nyquist), yet locate a single edge to within $0.10;\mu\text{m}$ to $0.50;\mu\text{m}$ using sub-pixel algorithms — provided illumination uniformity and SNR are adequate.
Automated Optical Qualification: Replacing Guesswork with Physics
Manual estimation of sensor-optics compatibility — often based on experience, vendor datasheets, or informal rules of thumb — is prone to error in ways that directly impact production yield and measurement uncertainty. An engineer selecting $f/8.0$ for depth-of-field margin without evaluating diffraction may unknowingly sacrifice 30% or more of the sensor's resolving power. Conversely, specifying a $2.2;\mu\text{m}$ high-resolution sensor behind an NIR lens at $f/5.6$ wastes the investment, as the $11.61;\mu\text{m}$ Airy disk far exceeds the $4.4;\mu\text{m}$ two-pixel span.
Automated calculation from first-principles physics — Nyquist frequency, Airy disk diameter, optical cutoff, and system limitation diagnosis — transforms system specification from an iterative trial-and-error process into a deterministic engineering workflow. Every parameter is traceable to a physical constant or a measured hardware specification. The result is a machine vision system designed to operate precisely at the intersection of sensor capability and optical performance, with no margin wasted and no resolution left unrecovered.