Link: https://www.research-collection.ethz.ch/handle/20.500.11850/683623
Thesis title: A Scientific Event Camera: Theory, Design, and Measurements
Author: Rui Garcia
Advisor: Tobi Delbrück
This thesis explores the physical limits of neuromorphic event cameras and proposes a novel highly-sensitive event camera targeting scientific applications. Certain scientific applications, such as fluorescent imaging of neural activity (especially voltage imaging), as well as tracking of particles/objects moving at high speed, require vision sensors operating near the limits of physics.
These applications consist in the detection of low contrast changes in light intensity during time intervals of only a few milliseconds or less, in observations that can last several minutes. Currently, these applications rely on scientific image sensors capable of capturing thousands of frames per second. While the paradigm of frames is highly prevalent in computer vision, it has significant downsides. Namely, for the applications described, the acquisition of thousands of frames per second during several minutes leads to a highly redundant output, resulting in an extremely inefficient data utilization.
These applications consist in the detection of low contrast changes in light intensity during time intervals of only a few milliseconds or less, in observations that can last several minutes. Currently, these applications rely on scientific image sensors capable of capturing thousands of frames per second. While the paradigm of frames is highly prevalent in computer vision, it has significant downsides. Namely, for the applications described, the acquisition of thousands of frames per second during several minutes leads to a highly redundant output, resulting in an extremely inefficient data utilization.
Vision in animals is the product of millions of years of evolution through natural selection, resulting in visual systems orders of magnitude more efficient than frame-based cameras. Neuromorphic silicon retinas such as the Dynamic Vision Sensor (DVS) event camera draw inspiration from biological visual systems to build more efficient vision sensors.
Characteristics of the DVS such as its high-speed performance with low latency and high data efficiency, as well as its high dynamic range, make it an emerging technology with growing popularity over the last years. These characteristics make the DVS a promising candidate for the scientific applications mentioned above. However, DVS implementations proposed before this work did not demonstrate sufficient sensitivity in the light-constrained settings required by these applications.
The main purpose of the work presented in this thesis is the development of a novel DVS event camera with improved sensitivity under dim light. To achieve this goal, this thesis investigates the physical limits of the DVS technology, demonstrating that the DVS is limited to a minimum of 2x shot noise, and providing the conditions for the camera to operate near this limit. It also shows that spatial and temporal integration of light are fundamental to improve sensitivity in the dark - a result known from other visual systems, but never fully exploited in the DVS. This new knowledge, resulting from extensive measurements of DVS cameras and supported by theoretical analysis, resulted in a more realistic model of the DVS pixel, capable of reproducing measured phenomena and aligning with theory.
The results obtained are useful for DVS users, by providing optimal biasing strategies, for algorithm developers, by providing novel interpretations and insight about DVS data encoding, and for DVS designers, by defining the limits of the technology and optimization goals.
Finally, supported by an improved understanding of the DVS pixel and its limits, this thesis proposes SciDVS: A scientific event camera capable of responding to edges of 1.7 % contrast under dim light settings at 0.7 lx on-chip illuminance. SciDVS features an array of 126 × 112 pixels with a pitch of 30 μm, implemented on a 180 nm CMOS Image Sensor process. The SciDVS pixel introduces novelty such as an auto-centering high dynamic range pre-amplifier, improved bandwidth control achieving cutoff frequencies down to 3.5 Hz, and pixel binning.
Characteristics of the DVS such as its high-speed performance with low latency and high data efficiency, as well as its high dynamic range, make it an emerging technology with growing popularity over the last years. These characteristics make the DVS a promising candidate for the scientific applications mentioned above. However, DVS implementations proposed before this work did not demonstrate sufficient sensitivity in the light-constrained settings required by these applications.
The main purpose of the work presented in this thesis is the development of a novel DVS event camera with improved sensitivity under dim light. To achieve this goal, this thesis investigates the physical limits of the DVS technology, demonstrating that the DVS is limited to a minimum of 2x shot noise, and providing the conditions for the camera to operate near this limit. It also shows that spatial and temporal integration of light are fundamental to improve sensitivity in the dark - a result known from other visual systems, but never fully exploited in the DVS. This new knowledge, resulting from extensive measurements of DVS cameras and supported by theoretical analysis, resulted in a more realistic model of the DVS pixel, capable of reproducing measured phenomena and aligning with theory.
The results obtained are useful for DVS users, by providing optimal biasing strategies, for algorithm developers, by providing novel interpretations and insight about DVS data encoding, and for DVS designers, by defining the limits of the technology and optimization goals.
Finally, supported by an improved understanding of the DVS pixel and its limits, this thesis proposes SciDVS: A scientific event camera capable of responding to edges of 1.7 % contrast under dim light settings at 0.7 lx on-chip illuminance. SciDVS features an array of 126 × 112 pixels with a pitch of 30 μm, implemented on a 180 nm CMOS Image Sensor process. The SciDVS pixel introduces novelty such as an auto-centering high dynamic range pre-amplifier, improved bandwidth control achieving cutoff frequencies down to 3.5 Hz, and pixel binning.
0 Response to "PhD thesis on SciDVS event camera"
Post a Comment