August 24 2021
Image sensors for automotive applications are an indispensable feature for car safety and the future. Sony Semiconductor Solutions Group (“the Group”) boasts the top market share globally in image sensors, and yet it lags behind in the areas of automotive applications. To gain competitiveness in this segment, the Group has developed an image sensor for automotive cameras which delivers both high dynamic range (HDR) and LED flicker mitigation (LFM) at high levels. The task of developing an image sensor that entailed unfamiliar technical characteristics meant various difficulties that were typical when entering a new business sector. Automotive settings present harsh conditions to overcome, such as high temperature, humidity, and vibration, while high quality must be ensured. This meant that the design, manufacturing, and quality assurance divisions had their own unique challenges to address and conquer. In this interview, we asked about the Group’s Safety Cocoon concept and behind-the-scene story about developing the automotive image sensor, a key element of self-driving automobile technology.
Azami：I was initially a design project leader (D-PL) of image sensors for security and industrial applications. Around 2012, the Group was looking into the automotive business, and I was appointed as a D-PL for IMX224, the project’s first product. I took this as a great opportunity because I was interested at the time in working with foreign companies.
In the beginning, the project started to develop automotive image sensors modelled on security image sensors. The basic idea was that the security sensors had the attributes that were useful for automotive sensors, such as extensive durability and the ability to capture images in dark places. However, once the business became established, we found out that the automotive image sensors had a whole set of different requirements. Those days, I would visit customers overseas and come back feeling deflated at their feedback, time and again.
Azami：Unlike security or industrial image sensors, automotive image sensors had the standards specifically developed for automotive applications, such as IATF 16949*1 and AEC-Q100,*2 which we already knew. We tried very hard to make our design meet these standards, but our proposal was tarnished by one customer.
The problem was high dynamic range (HDR). HDR allows to capture images over a wide range of luminance, from dark to bright, and this had to be a complete system within the sensor. For security image sensors, there is a method called DOL (digital overlap) that realizes HDR, but this method synthetizes images under different exposures outside the sensor. The Group did not have the expertise to complete the image synthesis within a sensor, and it had to be developed from scratch.
Another challenge was the LED flicker,*3 the phenomenon that is everywhere in society today. The customer requested that the LED light, such as brake lights and traffic lights, must be captured in every single frame. This proved to be a tough call due to the fact that there were no established industrial standards for LED light frequency, which posed the difficulty to ensure all LEDs were covered. I needed a help, so I asked Sakano-san from the design division to join me and we did brain storming over and over again.
*1: The international standards of quality management system specifically designed for the automotive industry.
*2: The quality standards for automotive ICs prescribed by AEC (Automotive Electronics Council).
*3: LED flickering that occurs when photographing LED indicators, traffic lights, etc.
Sakano：I have always been interested in HDR, and Azami-san’s invitation was all welcome. But there was a problem. Image sensors for automotive cameras require very high quality standards, and initial defect rate had to be extremely low. Normally, other image sensors are improved iteratively through a course of repeated production cycles. Whereas, automotive image sensors must be manufactured by mature processes, and this was an unwritten rule. For this reason, the Research Division responsible for the process development, where I belong, had a rule that the image sensors for automotive applications were out of bounds. Therefore, my involvement with the project was initially unofficial and on a volunteer basis.
Azami：Apart from the design, there were also challenges in terms of manufacturing. We did not have an information network, knowledge, and know-how in the automotive industry those days. So, it was hard work even to study the standards particular to this industry, such as IATF 16949 and ISO 26262.*4 We held study sessions again and again with Nishizuka-san and Shimozono-san.
*4: The international standards concerning the functional safety of electrical/electronic systems of road vehicles.
Nishizuka：The standards like IATF 16949 and ISO 26262 are clear enough themselves, but it was very difficult to adapt our work processes to them. To probe and understand each standard was difficult in itself, but it was even more challenging to align our project members to this understanding. It was necessary that every member understood and implemented the ways of doing things that were unfamiliar to us.
Despite these difficulties, there was a positive aspect, that all these efforts to pursue the project were for the realization of autonomous driving and other future advancement.
Shimozono：The requirements set out in those standards like IATF 16949 and ISO 26262 were very specific, different from ones we normally handled. We needed to find out the ways to apply them in manufacturing in practice. Another particularity of automotive applications is that so many customer inspections were involved. How we interpreted the standards often turned out to be unsatisfactory to our customers, who gave us feedback with their specific requests, as well. It was such hard work to meet all these requirements and requests.
Nishizuka：There are several aspects to the reasons why this issue posed a major challenge. For example, the dashboard can heat up over 100°C in summer time. So, the sensor must be able to withstand these temperatures. Secondly, driving on a road causes vibration. It takes extremely high levels of engineering to make precision devices that withstand vibration and operate without making errors.
Also, as Sakano-san mentioned it earlier, there is the zero-defects expectation, that is, to minimize the initial defect. The request was below 1 ppm, which means fewer than one defective unit in every one million produced. In order to meet the requirements of both high performance and reliability, we had to set up the product evaluation environment as strict as never before.
Sakano：In terms of HDR, they required 120 dB for the automotive sensors whereas for other applications it was about 70 dB. To raise the dynamic range from 70 to 120 dB, it is not simply increasing the signal processing capability by 120 divided by 70, that is, 1.7 times. It was in fact to increase it 320 times over. This figure is impossible to be achieved by the conventional image sensor architecture. We needed to develop a brand new architecture.
Previously, the sensors were designed to store electric charge in the photodiode. To free up a space for processing signals by several hundred folds, we decided to create a separate container for storing electric charge. At this point, we adopted the capacity used for memory such as DRAM to the image sensor, which had not been used for image sensors before. A new pixel structure and pixel control method enabled the sensor to capture multiple images under different exposures simultaneously. This enhanced the dynamic range dramatically.
Related News Release：
Sony Releases CMOS Image Sensor for Automotive Cameras with
5.4 Effective Megapixels, the Industry’s Highest in a Sensor that Delivers Simultaneous HDR Shooting and LED Flicker Mitigation
Sony Commercializes the Industry's First*1 High-Sensitivity
CMOS Image Sensor for Automotive Cameras,
Delivering Simultaneous LED Flicker Mitigation and High-Quality HDR Shooting
Shimozono：For the manufacturing division, the task was to provide a new structure for the new architecture. In order to realize the performance required for automotive applications, we adjusted each line process. The production is running with the minimum margin. The ultimate level of dust control was necessary to ensure the production lines dust-free in order to eliminate defects to achieve the goal of zero-defects.
Automotive image sensors need performance guaranteed under the temperatures ranging from -40 to 125°C, for which improvement was necessary in many aspects, such as the dark period performance in high temperature environments. For realizing the HDR function, the photodiode performance needed to be improved. Also, pixels had to be balanced out. There were so many issues to address in the development. So, we worked closely with the design team and developed process requirements.
As for the package, the challenges we faced were ensuring the reliable mounting despite the differences in expansion factors under different extreme temperatures required for automotive applications and the high-temperature reflow at 250 degrees. These were overcome by developing a special package dedicated to automotive sensors.
Azami：As with all image sensors of the Group, our image sensors for automotive applications have low noise in dark situation and low white spot defects. The technology is superior that images can be captured even under a high temperature of 105°C, which is uncontested by our competitors. What makes this possible is, I think, that we have the know-how in the plant, built on our expertise in CCDs, and also that we have both the pixel design and manufacturing sections working together in the company. We work very closely together, so we can have prompt feedback, which helps to speed up the improvement. Through discussions, we can determine if the problem can be resolved either on the process or design side. The fact that we can determine these minute details ourselves is a great asset to the Group.
Another great advantage of our image sensors is the high level integration of HDR and LFM. Our global customer support is also our stronghold. I believe that the Group has both the technology and the environment that enable us accurately to reflect customer requests in our products.
Sakano：Given that image sensors are like ultimate analog devices, it is difficult to make improvements without close collaborations between the design and process teams. So, the Group has the great advantage in having this environment that allows close communication between the design and process divisions.
Nishizuka：We have client quality inspection every so often, but their feedback on our plants and design development is always very positive. We tell our customers that they can come to us for inspection any time, though the Covid situations make it difficult at the moment, and this is because we believe that trust and confidence are key elements of automotive-related products.
Sakano：I think it was the heat tolerance in relation to the performance. The temperature range of -40 to 125°C is a functional guarantee after all. It does not mean to guarantee performance. However as long as the sensors function, there will be images produced, and these images could be a cause of complaints. The difficulty is in that there are no clear criteria. The complaint depends on customers. Work process also poses some difficulties. Apart from Sony’s own reliability test standards, there are other reliability tests prescribed by the industry’s authorities. Given this, despite the relatively long overall schedule for the development compared to other categories, the time allocated for the process development is not very long. Furthermore, the process needs to be highly complete and reliable before moving on to mass production, and this includes the yield. As a result, we the process development are under a massive pressure.
From the viewpoint of business, there is another difficulty. It is customary in this industry that, in order to make business out of a product, it must conquer many competitions after it is finalized. We are one of the late-comers in this business and have not yet managed to be in a position to negotiate and agree on specifications with a customer before starting a product development. This situation puts us, responsible for the technology planning, under considerable pressure until we see the result.
Shimozono：Client requirements were very demanding. They want various performance attributes for the image sensor. For example, it should not saturate under certain conditions that are prone to highlight blowout, such as under bright sunlight and snow on the ground, and it should maintain stable functioning in high or low temperatures, among other quality assurance requirements.
The expected levels of reliability, quality management, and customer services were much higher than those for other categories. It took a true commitment to meet all these requirements and ensure high product performance.
Nishizuka：Now, we are heading toward autonomous driving. This sets us on the tasks of realizing the zero defects as well as safety and security in a harsh environment. These highly ambitious objectives are going to be such hard work. The fact that only a few defects can lead to a significant failure keeps us on our toes.
Azami：I imagine that cars with the Group’s image sensors for automotive applications will make our Safety Cocoon concept a reality, delivering safety by cocooning the cars and the road system as a whole.
Sakano：In terms of automotive applications, I also work on the SPAD depth sensor for automotive LiDAR. In this sense, I would like to be making contributions to the realization of the Safety Cocoon concept through a combination of these different sensors. It is also my ambition to go beyond the automotive business and contribute to other categories of the Group with the technology by gaining a lead on the HDR technology.
Nishizuka：It would make me proud to have the world recognize our automotive image sensors to be indispensable for the safety and security of autonomous vehicles. It would be wonderful if our image sensors were adopted to all cars around the world and automated driving became a reality.
Imagine the future mobility, in which people enjoy Sony’s games, music, movies, and other forms of entertainment in the autonomous driving vehicles. What a fun idea that is!
Shimozono：Safety in mobility is important, of course, but I would also like to help solving social issues such as traffic accidents and the mobility vulnerable in society, so that people’s lives will be enriched through our technology.
The introduction of EVs is said to transform the automobile industry, and image sensors are also expected to play as vital a role as car batteries do. So, it would be wonderful that our efforts stimulated this growing market segment to support the Japanese industries.
Azami：Traffic accidents are life-threatening. If you choose your car simply on the basis of prices, you might end up regretting your choice in an unexpected accident. It is thus our responsibility to promote the virtues of our image sensors carefully so that customers are well informed and make the choice that they will not regret. This effort amounts to saving people from losing their loved ones in traffic accidents, too. Keeping this idea in mind, I would like to work on the zero defects and, also, the development of next-generation solutions, such as a system that self-diagnoses and alerts of any failure in image sensors.
Sakano：For me, as I have been involved in the automotive applications from the start, my ambition would be to enable the Group to make profits in this business area through our leading expertise in engineering and technology. In doing so, I would like to develop a smart architecture that, in view of the environmental issues, ensures high levels of efficiency and self-controls the system by detecting and evaluating the environment. By proposing such an architecture to the industry, I would like our Group to step up to the leading position in the industry.
April 8, 2021, marked the first ever accident-free day in the 53-year period of surveillance conducted by the Japanese National Police Agency. I wish that our efforts will eliminate traffic accidents from this world, where zero-accident will be a regular feature. It would make me so proud if our image sensors helped to bring this about.
Nishizuka：My ambition is to literally realize the zero-defects quality in the Group’s automotive image sensors. I honestly believe that, all members of the Group’s automotive application team working in unity, this would be possible to achieve.
Shimozono：The first goal for me is to be the world’s No. 1 in image sensors for automotive applications. In sensing, we should aim for enhanced safety through improved detection distances and viewing angles achieved by improved resolution through miniaturization. In this sense, I would like to pursue enhanced environmental robustness through a fusion of SPAD depth sensor into sensing technology.
As much as delivering image sensors that can enrich people’s lives, I would like to create the work environment in the Group where work brings us joy.