Imaging and Sensing Technology

Imaging and Sensing Technology

CMOS image sensor development at Sony began in 1996 and led to the launch of our first CMOS image sensor (IMX001) in 2000. At the time, CMOS image sensors produced noisy images under low light and were also inferior to CCD image sensors in the number of pixels. However, the lower readout speed of CCD image sensors convinced us that they would be unable to support high-resolution data as the industry moved from SD to HD video. In 2004, we therefore changed course greatly by shifting our focus from CCD to CMOS image sensor development. It was a bold decision. Instead of holding the world’s number one share in CCD image sensors, we would be building on a negligible market share in CMOS image sensors.
Later in 2007 we commercialized CMOS image sensors with an original column A/D conversion circuit for fast, low-noise performance, followed in 2009 by back-illuminated CMOS image sensors with twice the sensitivity of conventional image sensors – beyond the human eye.
Further examples of technical innovation that has enabled us to constantly lead the industry include stacked CMOS image sensors in 2012 – with higher image quality and multiple functions in a smaller package, thanks to layering of the pixel and signal-processing sections – and, in 2015, the world’s first image sensors with a Cu-Cu connection, enabling smaller packages, higher performance, and greater productivity in manufacturing.

Image sensor Advance in key technology

The CMOS image sensor with a column A/D converter achieves fast and low-noise performance by the column A/D converter (advances of peripheral circuit technology). The back-illuminated CMOS image sensor achieves higher sensitivity due to the back-illuminated structure (advances in pixel technology). The stacked CMOS image sensor achieves high image quality and multiple functions in a smaller package (advances in layering technology). The stacked CMOS image sensor with Cu-Cu connections realizes smaller packages, higher performance, and greater productivity by direct connection of copper pads (advances in chip-to-chip connection).

Column-Parallel A/D conversion circuit-equipped CMOS image sensors (commercialized in 2007)

These sensors use original Column-Parallel A/D conversion technology, with an A/D converter for each vertical row of pixels, arranged in a parallel configuration.
In this arrangement, analog signals read from the vertical signal lines can be conveyed directly to each row’s ADC across a minimal distance, which reduces loss of image quality from noise entering the signal during analog transmission and accelerates signal readout. Noise is also reduced through dual noise cancellation, with high-precision cancellation applied to both analog and digital circuits.

Back-illuminated CMOS image sensors (commercialized in 2009)

These image sensors adopt an innovative back-illuminated structure that offers lower noise and nearly double the sensitivity of conventional front-illuminated CMOS image sensors.
Without interference from wiring or transistors, light is received from the back side of silicon substrate, which increases the amount of light entering each pixel and reduces loss of sensitivity relative to light entering at various angles. This enables smooth, clear images even at night or in other low-light conditions.

Stacked CMOS image sensors (commercialized in 2012)

In the stacked structure adopted by these image sensors, the pixel section where back-illuminated pixels are formed is layered over a chip (instead of the supporting substrate used in conventional back-illuminated CMOS image sensors) where signal processing circuits are formed.
One advantage is that large-scale circuits can be mounted on a small chip. Since each section is formed on a separate chip, specialized manufacturing processes can be used to produce a pixel section with high image quality and a circuit section with high performance, enabling higher resolution, multi functionality, and a compact size.

Stacked CMOS image sensor with Cu-Cu connections (commercialized in 2015)

Cu-cu connections involve direct connections between the copper pads formed on the layering surfaces of pixel chips and logic circuit chips. Without the need to provide electrical connections through pixel chips or special areas for connections, manufacturers can make smaller image sensors at a higher rate of productivity. Offering greater freedom in pin layout and higher density, the technology will help enable stacked CMOS image sensors with expanded functionality.

Sony’s technologies, expanding into sensing applications

In the image sensor market, we are also focusing on the sensing field, where applications are expected to expand. Besides the imaging technology we have honed for viewing captured images, we will be combining this technology with sensing technology for acquiring and using a various information. In this way, we are cultivating new applications and markets for image sensors.

Sensing technology expanded by image quality, information, and higher speed

ToF Image Sensors: Expanding Possibilities in 3D Object Recognition

Time-of-flight (ToF) image sensors determine the distance to objects from the time it takes emitted light to reflect off the objects and reach the sensor. Pixel technology in Sony’s back-illuminated structure produces depth maps as accurate as those of conventional sensors even at 1.5 times the distance. Gesture, object recognition, and obstacle detection are expanding applications for these sensors, which are used in augmented reality (AR) and virtual reality (VR) scenarios, and in robots and drones that require autonomous operations.

  • Image captured with a CMOS image sensor
  • A back-illuminated ToF image sensor
  • 3D model image combining a depth map and monochrome image

Automotive image sensors: Better recognition in many environments, for safety and security

Sony has commercialized the IMX490 CMOS image sensor for automotive cameras with 5.4 effective megapixels, the industry’s highest *1 in a sensor that both mitigates LED flicker (from LED signs and traffic signals) and offers a wide dynamic range. Also, IMX324 was released, which is a stacked CMOS image sensor with 7.42 effective megapixels, the industry’s highest resolution *2 for forward-sensing cameras. We expect these sensors to be used in more cars than ever, in applications including advanced driver-assistance systems (ADAS) and camera monitoring systems (CMS) that replace rearview mirrors.

  • *1As of announcement on December 18, 2018.
  • *2As of announcement on October 23, 2017.

Sample comparison movies of image sensors

  1. Industry’s highest image sensor resolution for automotive cameras

    Comparison of distant sample images

    IMX324 shooting example IMX324 (7.42 million pixels) enlarged image IMX224 (1.27 million pixels) enlarged image
  2. High sensitivity of 2,666 mV (standard value F5.6, in pixel binning mode)

    Comparison under low-light (0.1 lux) in sample images

    IMX324 (pixel addition mode) shooting example IMX224 shooting example

Automotive initiatives: Sensor Fusion

One forward-looking initiative at Sony is called Sensor Fusion. The technology under development integrates raw data from camera feeds, LiDAR, and milliwave radar to identify vehicles and other objects. As a sensor manufacturer, Sony is uniquely positioned to combine the signal processing, noise reduction, and data optimization in this technology. These examples show how effective Sensor Fusion can be. Even under challenging conditions for object recognition – such as fog, glare, or rain at night – Sensor Fusion enables accurate identification sooner than other systems.

Explanatory drawing showing sensor fusion that recognizes vehicles, etc. by fusing camera images with LiDAR data and Raw data of millimeter wave radar
  • Integrates large amounts of raw data from camera images, LiDAR, and milliwave radar
  • Highly accurate object identification even in difficult conditions
  • Scenario 1:

    Current Technology

    Sony's Sensor Fusion

    I dentifies automobiles ahead obscured
    by fog
  • Scenario 2:

    Current Technology

    Sony's Sensor Fusion

    I dentifies automobiles ahead obscured
    by backlight
  • Scenario 3:
    Night & Rain

    Current Technology

    Sony's Sensor Fusion

    I dentifies automobiles ahead obscured
    by headlights and rain

Automotive initiatives: Solid State LiDAR

Leveraging Sony’s semiconductor technologies, Solid State LiDAR uses highly accurate distance measurement to gain a precise 3D grasp of real-life spaces. This also leads to improved recognition capability for long-distance objects.

Industry’s first* Stacked Depth Sensor for Automotive LiDAR with SPAD Pixels

This is the stacked direct Time of Flight(dToF) depth sensor for automotive LiDAR using single-photon avalanche diode (SPAD) pixels, an industry first*.
By stacking SPAD pixels and distance measuring processing circuits into a single chip, distances of up to 300meters can be measured at 15cm resolution with high precision and high speed.
*Among stacked depth sensors for automotive LiDAR. According to Sony research(as of announcement on February 18, 2021)

What is a SPAD pixel?

SPAD is a pixel structure that uses avalanche multiplication to amplify electrons from a single incident photon, causing a cascade like an avalanche, and it can detect even weak light.

Illustration of avalanche multiplication

Direct Time of Flight method(dToF)

This is one of the distance measurement methods, in which the flight time (time difference) of the light emitted from the light source and reflected by the object reaches the sensor is detected, and it measures the distance to the object.
The distance measurement sensor using the dToF method uses SPAD pixels that detect a single photon, enabling long-distance and highly precise distance measurement.

Illustration of dToF method

Features of a SPAD Depth Sensor

Sony's SPAD Depth Sensor employs a back-illuminated SPAD pixel structure that uses a Cu-Cu connection to achieve conduction for each pixel between the pixel chip (top) and the logic chip equipped with distance measuring processor circuits (bottom).
By its original pixel structure, sensor can be compact yet high-resolution, also enables high-precision, high-speed measurement at 15-centimeter range resolutions up to a distance of 300 meters*

*When measuring an object with a height of 1 meter and reflectance of 10% using additive mode of 6 x 6 pixels (H x V) under cloudy daylight conditions.

  • Structural diagram and features
  • Point cloud

Polarization image sensors: Revealing the unseen

Polarizers were separate in conventional polarization cameras, but innovative CMOS image sensors from Sony incorporate the polarizer into a back-illuminated CMOS image sensor. As a one-chip solution with the polarizer on the photodiode, it enables more compact polarization cameras. Potential applications are not limited to automotive field but include a variety of other applications that involve capturing subjects obscured by glare, or capturing details of surface unevenness.

When taking a picture of a pond with a polarization image sensors, the waves are removed and the underwater koi-fish appear clear
  • Tablet filling inspection

    An ordinary image sensor

    Image of tablet and aluminum sheet

    A polarization image sensor

    Image of tablet and aluminum sheet
  • Lens distortion inspection

    An ordinary image sensor

    Image of glasses

    A polarization image sensor

    Image of glasses
  • Eliminating glare from sunlight

    An ordinary image sensor

    Image of a car

    A polarization image sensor

    Image of a car

World's First Intelligent Vision Sensor with AI Processing Functionality

Intelligent Vision Sensor is the first image sensor in the world to be equipped with AI processing functionality.* Including AI processing functionality on the image sensor itself enables high-speed edge AI processing and extraction of only the necessary data, which, when using cloud services, reduces data transmission latency, addresses privacy concerns, and reduces power consumption and communication costs.

*:Among image sensors. According to Sony research (as of announcement on May 14, 2020)

What is Edge AI?

The spread of IoT has resulted in all types of devices being connected to the cloud, making commonplace the use of information processing systems where information obtained from such devices is processed via AI on the cloud. On the other hand, the increasing volume of information handled in the cloud poses various problems : increased data transmission latency hindering real-time information processing ; security concerns from users associated with storing personally identifiable data in the cloud ; and other issues such as the increased power consumption and communication costs cloud services entail.
From these backgrounds, the need for Edge AI that enables AI processing within edge devices is increasing.

Method of Edge AI

To realize Edge AI processing, edge devices need to be equipped with AI processing functionality, but there are differences depends on where AI processing works within the edge devices.It is also possible to be equipped with an AI processor separately from an image sensor within a system configuration of the camera. On the other hand, Intelligent Vision Sensor has AI processing functionality so that enables Edge AI processing within the image sensor.

World’s first image sensor equipped with AI processing functionality

Intelligent Vision Sensor features a stacked configuration consisting of a pixel chip and logic chip. In addition to the conventional image sensor operation circuit, the logic chip is equipped with Sony’s original DSP(Digital Signal Processor) dedicated to AI signal processing, and memory for the AI model. This configuration eliminates the need for high-performance processors or external memory, making it ideal for edge AI systems.

Metadata output

Intelligent Vision Sensor can output metadata which is semantic information belonging to image data, making for reduced data volume.Ensuring that image information is not output helps to reduce security risks and address privacy concerns. In addition to the image recorded by the conventional image sensor, users can select the data output format according to their needs and uses, including ISP (Image Signal Processor) format output images (YUV/RGB) and ROI (Region of Interest) specific area extract images.

Data output format selectable to meet various needs

*:AI functionality has the characteristic of using a statistical or probabilistic method, and as a result, metadata that is erroneously recognized may be added.

High-speed AI processing

When a video is recorded using a conventional image sensor, it is necessary to send data for each individual output image frame for AI processing, resulting in increased data transmission and making it difficult to deliver real-time performance. Intelligent Vision Sensor performs ISP processing and high-speed AI processing (3.1 milliseconds processing for MobileNet V1*) on the logic chip, completing the entire process in a single video frame. This design makes it possible to deliver high-precision, real-time tracking of objects while recording video.

* MobileNetV1:An image analysis AI model for object recognition on mobile devices.

Example of real-time tracking with product and task at a register

Selectable AI model

Users can write the AI models of their choice to the embedded memory and can rewrite and update it according to its requirements or the conditions of the location where the system is being used. For example, when multiple cameras employing this product are installed in a retail location, a single type of camera can be used with versatility across different locations, circumstances, times, or purposes. When installed at the entrance to the facility it can be used to count the number of visitors entering the facility; when installed on the shelf of a store it can be used to detect stock shortages; when on the ceiling it can be used for heat mapping store visitors (detecting locations where many people gather), and the like. Furthermore, the AI model in a given camera can be rewritten from one used to detect heat maps to one for identifying consumer behavior, and so on.

Example of camera usages in a facility

Demonstration video of Worker Monitoring
*Realized in combination with various sensors

to PageTop