Making ‘Smart Headlights’ with Machine Learning

October 24, 2018
Making ‘Smart Headlights’ with Machine Learning

Duke engineers are helping to invent headlights that automatically recognize surrounding environments, selectively illuminate important objects and display information on the roadway.

ECE Professor Xin Li is helping HASCO Vision Technology create 'smart headlights' that can display information within the beam such as the directions shown on the roadway here.

It’s a common scene for anybody driving at night on a dark road. Zipping around corners and over hills, the car’s high beams are on to improve vision while the driver’s hand remains poised to turn them off at a moment’s notice, lest they blind oncoming traffic and cause an accident.

Xin Li believes there is a better solution, and he’s working with China’s largest headlight manufacturer to make it a reality.

“Modern headlights don’t have just one or two light bulbs, they can have up to a million,” said Li, professor of electrical and computer engineering at Duke University and Duke Kunshan University. “I’m working with industry partners to create a ‘smart headlight’ that can control each pixel individually and automatically light up different areas in front of the car after recognizing the surrounding environment.”

For example, the headlight could reduce the amount of light pointed at an oncoming car while simultaneously increasing the illumination of an upcoming road sign. Or it could detect nearby pedestrians and alert the driver by highlighting their body while avoiding shining light directly into their eyes.

Machine learning can teach 'smart headlights' to recognize the environment and help the driver by illuminating nearby pedestrians while avoiding blinding them in the process.

The challenge to making such a headlight isn’t necessarily creating the different patterns of light—it’s teaching the car how to automatically recognize the surrounding environment and make the patterns on its own. It’s a problem that HASCO Vision Technology—the largest automotive lamp manufacturer in China—is turning to Li to help them solve with machine learning.

Many car companies are using cameras and machine learning to help control their own versions of self-driving cars, so Li is far from alone in this regard. Machine learning algorithms, however, need vast amounts of data to learn from, and many of the datasets and algorithms already created for this purpose have focused on daytime driving.

“Our application doesn’t care about the daytime though,” said Li, who splits his time between Duke’s campuses in North Carolina and Kunshan, China. “Using this approach for smart headlights is actually more difficult because the lighting conditions are much worse. It’s a unique challenge that has not been well studied in the past.”

While his industry partners are working to collect more nighttime footage and painstakingly annotate important objects like signs, pedestrians and other cars, Li is optimizing the machine learning algorithm. Because decisions must be made in real-time, the researchers must choose the right hardware and design the algorithm to fit its architecture.

Helping Li with this task is Duke Kunshan research scientist Xin Feng. Together, Li and Feng already have a working demo that, while impressive, still needs to improve before it hits the roads.

Another option for displaying information on the roadway with 'smart headlights' is reminding drivers of the speed limit.

“Detection accuracy is very important—you can’t miss anything or anyone,” said Li. “And while that’s a very important and challenging problem, it’s only one metric. The other is real-time response. If the algorithm takes too long to respond, then it isn’t useful. Technically, those are the two most challenging issues.”

But they’re issues Li expects to resolve in the near future, along with adding a few more bells and whistles. Another aspect of the project uses the headlights to project important information such as weather and road conditions, traffic signs, navigation directions, and even driving lanes onto the roadway within the headlight beams themselves.

Other advances could include making use of alternative sensors that self-driving cars may eventually possess such as radar and lidar. But for now, the project is using only forward-facing cameras to keep costs down, as in the next few years consumers are much more likely to see smart headlights on the roads in large numbers than self-driving vehicles with added detection capabilities.

“I think we can get a first-generation product on the market in the next two years,” said Li. “Once we have those on the road, we can get even more feedback and data to further improve accuracy and response time to make future iterations even better.”