[lane detection] a lane detection method based on neural network and structural constraints: ultra fast structure aware deep lane detection (ECCV 2020)
Innovation: (1) compared with the method of returning a complete lane line, the method of describing the lane line in this scheme can flexibly describe the lane line with complex topology; Compared with the method based on segmentation and clustering, it greatly reduces the computational complexity.
SAL: active Lane learning method based on salience mask: idea Abstract: this idea proposes an active Lane learning method to select the most effective training data for lane model, so as to achieve the goal of greatly reducing the amount of data annotation. For an unmarked image, we first use two different lane line models to infer, and get the detection results and image features
Laneformer: an end-to-end lane line detection system based on transformer's environment perception:
Summary of existing lane detection algorithms
- Traditional image processing
Traditional lane detection methods rely on highly defined, manual feature extraction and heuristic methods. Usually, post-processing technology is needed to filter out false detection to form the final lane. These traditional methods can easily lead to robustness problems due to road scene changes. The traditional method is fast, but it is only suitable for line detection, and the effect of curve detection is not good.
Step 1: convert color image into gray image, and the main function is CV2.cvtColor;
Step 2: Gaussian smoothing filter, the main function is CV2. Gaussian blur;
Step 3: Canny edge detection. The main function is CV2.Canny;
Step 4: ROI selection. The main functions are CV2.fillPoly and CV2.bitwise_and；
Step 5: Hough transform line detection. The main functions are CV2.HoughLinesP and CV2.Line;
Step 6: image overlay. The main function is CV2.addWeighted.
- Deep learning method
Source code: https://github.com/MaybeShewill-CV/lanenet-lane-detection
Interpretation of the paper: https://blog.csdn.net/c20081052/article/details/80622722
LaneNet is a multi task model combining semantic segmentation and vector representation of pixels, which is responsible for instance segmentation of lane lines in the picture. Each lane line forms an example, so that end-to-end training can be realized. Before segmenting the lane line for fitting the lane, we further propose to use a learned perspective transformation to make this adjustment on the image and compare it with the fixed aerial view. By doing so, we ensure the robustness of lane line fitting under road plane change, which is different from the existing methods that rely on fixed and predefined perspective transformation matrix. In summary, we propose a fast lane detection algorithm with a frame rate of 50 FPS, which can handle most lanes and lane changes. The algorithm has been verified in tuSimple data set and achieved advantageous results.
lanenet is based on tensorflow framework, and the reference content provided is very complete. The effect is verified on tusimple. It is characterized by curve fitting, which uses deep learning network for curve fitting. Using H-net to learn the fitting matrix can take advantage of the big data training and optimization of the deep learning model to obtain a more adaptive fitting matrix than the traditional bird transform or polynomial fitting method calculate once.
Source code: https://github.com/XingangPan/SCNN
Interpretation of the paper: https://blog.csdn.net/rain2008204/article/details/80078667
Spatial CNN(CNN), which transforms the traditional connection form of convolution layer by layer into the form of slice by slice convolution in feature map, so that information can be transferred between pixel rows and columns in the graph. This is especially suitable for detecting long-distance continuous shape targets or large targets, which have strong spatial relationship but poor appearance clues, such as traffic lines, telegraph poles and walls. SCNN is developed based on torch framework. Torch's unique table structure makes it difficult to convert its model to other forms such as cafe or tensorflow, but its deep learning network architecture designed based on the morphological attribute of lane line is worthy of reference.
Source code: https://github.com/SeokjuLee/VPGNet
Interpretation of the paper: http://www.sohu.com/a/270406788_ seven hundred and fifteen thousand seven hundred and fifty-four
Effect display video: https://www.bilibili.com/video/av26202634/
It is an end-to-end trainable multi task network, which can detect and recognize road and pavement signs at the same time in extreme weather by using vanishing point information. Mainly for night and extreme weather, it has very high accuracy and robustness, and can also run in real time (20fps).
Thesis title: Key Points Estimation and Point Instance Segmentation Approach for Lane Detection
Thesis address: https://arxiv.org/pdf/2002.06604.pdf
Code address: https://github.com/koyeongmin/PINet new
ADAS/AD basic concept [you do not have edit permission]
1, Industry background:
1. Pyramid supply chain system of automobile industry:
As a century old industry, the automotive industry has a complex supply chain system. At the top of the pyramid is OEM(Main engine factory/Vehicle factory), responsible for developing, manufacturing and selling vehicles; Next is the primary supplier of auto parts( Tier1),Secondary supplier( Tier2),...wait. 1)OEM It is a well-known complete vehicle factory, such as SAIC Volkswagen, SAIC GM, FAW Volkswagen, GAC Toyota, Chang'an Ford, Geely, great wall, SAIC passenger cars, GAC passenger cars and so on; 2)Tier1 That is, it is a first-class component supplier that holds the real core technology of automobiles, such as Bosch and ZF/Trina Solar, Continental, Magna, Delphi, Denso, Aisin, BorgWarner, Valeo, Schaeffler, Autoliv, Visteon, United electronics, etc; 3)Tier2 That is, various Tier1 Suppliers: take the semiconductor chip industry in automotive electronic components as an example, mainly Texas Instruments( TI),Renesas Electronics( Renesas),NXP( NXP),Infineon( Infineon)Wait, chip manufacturers.
2. New energy and intelligence of vehicles:
At present, automobile has two general development directions, one is new energy (hybrid, plug-in hybrid, electric, fuel cell, etc.), and the other is intelligent (intelligent driving and Intelligent Networking). It is necessary for China to develop new energy sources. There are many factors to protect the environment, but the effect is not necessarily obvious. It is more about the concept of propaganda. The real main thing is to reduce the dependence on oil in the field of national economy and people's livelihood. In addition, the barriers of traditional internal combustion engines are too high, and the core technologies are abroad. China also wants to find a way and direction to create opportunities for overtaking in some corners. I think the ultimate goal of new energy should be fuel cells. However, because China's fuel cell technology is too weak, Japan is the most developed country in the world. There is a problem. If we develop fuel cells, it will be equivalent to China throwing money to chicken feet( Japanese, Japanese English pronunciation (homonym, the same below), to help support the foot pot chicken to develop the fuel cell industry chain. Of course, this kind of thing can't be done. We can only see while walking. When the domestic related technology reaches a certain critical value, we will regain this line. Hybrid? It's very fuel-efficient, but it still has the strongest technology. For example, the Toyota Prius and other hybrid Toyota cars (corolla hybrid, leiling hybrid, Camry hybrid...) still need to burn oil, which is not in line with the national energy strategy. Plug in hybrid? Only when the development of electric vehicles in the early stage can not be promoted, the market can be activated; Because of the existence of plug-in hybrid, people who buy cars have less doubts and concerns. People who build charging piles can build more charging piles because of demand to help spread the infrastructure related to electric vehicles. Finally, we can only develop electric vehicles first. After all, there is a Tesla in the United States (Eagle sauce) who has lived for so many years and can help Chinese people touch Eagle sauce to cross the river. Moreover, the technical threshold of this technical route is relatively low. You don't see the new car building forces of Yishui. Why do they all use electric vehicles? Intellectualization currently includes two sub directions, one is intelligent driving direction, and the other is intelligent networking direction. Intelligent driving direction is divided into ADAS and AD((explained in detail below). For the direction of intelligent networking, the ultimate goal should be the realization of intelligent cockpit, that is, the intelligent networking and multimedia of the whole cockpit, and all kinds of cool HMI Display, various AR Technology, all kinds of screens, all kinds of Internet facilities, all kinds of humanized and scientific design, all kinds of xiangweilai automobile NOMI That kind of artificial intelligence robot (although the current function is frustrated)... It should be noted that these two directions have not appeared recently. In fact, in the field of traditional automotive electronics, there have always been two categories of automotive electronics, one is electronic control and the other is information and entertainment. However, the direction of intelligent driving is the further development of electronic control products, and the direction of intelligent networking is the further development of information and entertainment.
ADAS(Advanced Driver Assistance Systems)，Chinese Name: advanced driving assistance system. The main purpose is to improve the safety during vehicle driving. By reminding the driver and taking over the control of the vehicle, inform the driver of the potential danger and take some protective measures to prevent collision and other accidents. ADAS It mainly includes some adaptive features, including adaptive lighting, automatic cruise, automatic braking, lane keeping and blind spot detection.
2, Explanation of concepts & Terms:
Before talking about ADAS/AD, first paste the abbreviations and meanings of various common ADAS/AD features related to it, as shown below:
a. Abbreviations and full names of various intelligent driving function points:
CC: Cruise Control;
ACC: Adaptive Cruise Control;
FSRA (or ACC stop & go): Full Speed Range ACC adaptive cruise;
FCW: Front Collision Warning;
AEB: Automatic Emergency Brake;
Aeb-v: automatic emergency brake - vehicle;
Aeb-p: automatic emergency brake - pedestrian;
Aeb-c: automatic emergency brake - cycling;
LDW: Lane Departure Warning;
LKA: lane keeping assist;
Lka-ldp: LKA - Lane Department prevention LKA lane departure correction;
LKA-LC: LKA - Lane Centering LKA's lane is centered;
Elk: emergency lane keeping;
TSR: Traffic Sign Recognition;
ISA: Intelligent Speed Adaptation;
IHBC: Intelligent High Beam Control;
DMS: Driving Monitor System;
TJA: traffic jam assist;
Hwa: Highway assist;
APA: Auto Parking Assist;
RPA: Remote Parking Assist;
TJP: Traffic Jam Pilot (technology companies often call it low-speed automatic driving function);
HWP: Highway Pilot (often referred to as high-speed automatic driving function by technology companies);
FCTA: Front Crossing Traffic Alert;
RCTA: Rear Crossing Traffic Alert;
BSD: Blind Spot Detection;
LCA: lane change assist;
Door: doors opening warning;
RCW: Rear Collision Warning;
From the above feature s, we can see the obvious trace of small step iteration, which always evolves from the technical path of detection alarm control. For example, the relationship between TSR (traffic sign recognition) and ISA (intelligent speed control) first identifies the traffic signal and displays it on HMI, and then applies some traffic signals, such as the application of speed limit sign (ISA intelligent speed control). The same pairwise relationships include FCW/AEB, LDW/LKA, etc.
There is also an evolutionary trend that can characterize the transfer of vehicle responsibility control, such as TJA to TJP, HWA to HWP, etc.
This broken function point has one advantage, which makes it convenient for OEM to select product configuration and match high and low-end products; It is also convenient for Tier1 suppliers to quote and collect money.
Also known as Advanced Driver Assistant Systems / Autonomous Driving; Among them, ADAS focuses on L1/L2/L3 intelligent driving, and AD focuses on L4/L5 intelligent driving.
c. Terminology differences of advanced driving assistance / automatic driving / intelligent driving:
According to the above contents, there are two technical routes for cars in the driverless direction.
One is based on tradition Tier1 Adopted by the supplier ADAS Route, i.e. advanced driving assistance route, is gradually developed into fully automatic driving in the way of small step iteration. The technical route refers to the mass production vehicle market of mainstream passenger cars and commercial vehicles, which belongs to the field of automotive electronics. It mainly relies on the traditional automotive electronic development process and tools, that is, in terms of hardware, to develop various electronic control unit modules that meet vehicle regulations( ECU)And peripheral sensors. The software relies on the embedded development environment provided by various chip suppliers, such as Texas Instruments( TI),Renesas electronics, Infineon, etc. develop vehicle specification level embedded software. Algorithms are generally divided into bottom detection algorithms and upper application algorithms. Bottom detection algorithms are generally sensor environment detection algorithms, such as Mobileye of EyeQ A variety of visual detection algorithms (lane line detection algorithm, vehicle and pedestrian target detection and tracking algorithm, traffic light detection algorithm, traffic sign recognition algorithm, driveable area detection algorithm, etc.) implemented by series of chips, and the bottom detection algorithm is mostly used C/C++The upper application algorithm actually refers to various algorithms, such as ACC/AEB/LKA/TJA Equal function point( features)，The upper application algorithm is also called vehicle control algorithm. The function points are closely related to the definition of functional requirements of the main engine factory and rely on MATLAB/Simulink And other visual development tools to MBD(Model Based Design，Model based design) development process, develop and manage various ADAS Function point. ADAS Product development relative to AD The characteristic of development is that the development process is decoupled into many specific discipline modules, and there are complex development roles, such as system engineer (system requirements, system architecture, system testing), electronic engineer (or hardware engineer), embedded software engineer, Algorithm Engineer (mainly refers to perceptual algorithms such as lane line detection) feature Engineer (based on MATLAB/Simulink Develop various function points), Mechanical Engineer (responsible for the shell and label of parts and products), manufacturing engineer (related to production line manufacturing), test&Verification Engineer, matching engineer, etc. these complex roles are in accordance with V-Model(V Model) development process, i.e. requirements analysis, requirements development, system architecture, system design, sub module architecture of various disciplines (software, hardware, machinery, matching, manufacturing, verification, algorithm feature And other sub modules), sub module design, sub module implementation, sub module test, sub module integration, sub module integration test, system integration, system integration test and system test (also known as demand test, function test and vehicle test) to develop automotive electronic products. ADAS The route is currently making money, so players mainly use various mature and giant routes Tier1 The company is the main force, such as Bosch, Continental, Delphi, Autoliv and ZF/Trina Solar, electric equipment, etc. they occupied the main markets and made huge profits; domestic companies could only engage in 360 look around, automatic parking and other low technical barriers ADAS System, but currently L1 and L2 Great progress has also been made in the development of advanced functions and products. Common players include Hengrun technology, Huayu automobile, Lianchuang, Yanfeng Visteon, as well as many intelligent driving start-ups. Because of the core hardware technologies (such as millimeter wave radar hardware and camera CMOS Image sensor, lens, various MCU,ARM,DSP Chips, etc.) are still huge Tier1/Tier2 Hands, so large multinational Tier1 It is still in a stable monopoly position. The common auto parts products of this route mainly include intelligent front view camera module( FCM, Forward Camera Module),Forward millimeter wave radar module( FRM, Forward Radar Module),ADAS/AD Domain controller( ADC, ADAS/AD Domain Controller),Side rear millimeter wave radar module( SRR, Side-Rear Radar),360 Look around controller+Look around camera( BEV, Bird's-Eye View)It is worth mentioning that these products are not simple single sensors, but integrated sensors ECU(Electronic control unit (ECU) module is a complete subsystem. General automotive electronics field, if the product word is emphasized module(Module), rather than emphasizing Sensor(Sensors) generally refer to those with various computing units and functions ECU. The other is based on all kinds of high-tech companies (once high-tech companies should generally refer to high-tech in the fields of computer technology, biotechnology and aerospace technology. Now the so-called high-tech companies should have specifically referred to all kinds of high-tech companies IT/internet/Software companies, especially in China), such as Google, Baidu, Alibaba rookie, etc AD Route, i.e. automatic driving route, is mainly based on ROS Based on robot system and characterized by a large number of applications of high-precision map technology, artificial intelligence technology and computer vision technology L4 Level automatic driving. At present, the technical route is mainly based on Demo Based on PC The software of computer (industrial computer) is used to optimize and polish its own software and algorithms, and has not been embedded and industrialized. At present, this technical route is very biased towards the software industry, completely does not adopt the development process and tool chain of the traditional automobile industry, and directly uses its own strong code implementation ability to realize high-precision map, global path planning, environmental perception and positioning( RTK,VSLAM),Obstacle prediction, local path planning, vehicle control and other modules to realize various vehicle functions. Most employees are software developers such as programmers, without complex developer role division. Due to L4 The technology is not yet fully mature, so high-tech companies can't cooperate with all kinds of large enterprises for the time being Tier1 Suppliers compete. At present, in addition to participating in the passenger car and commercial vehicle markets Demo In addition to the project, this robot technology is also actively used in scenes and products in closed parks such as non person flow vehicles. Strictly speaking, some driverless logistics vehicles and other products landed by domestic science and technology companies are not real auto driving technology for large-scale mass production, but consumer transportation robots, because these products do not meet strict vehicle regulations and are free of charge In addition, due to the current large-scale Tier1 Rich and powerful, worry L4 The technology path has been subverted by new entrants, and the automatic driving technology route is also actively arranged. The way of betting at both ends is adopted to occupy one side ADAS And monopolize the market AD Field and technology companies are actively preparing for war, such as traditional Tier1 Bosch, a giant, began to recruit a large number of employees in Suzhou in 2017 L4 Level developers, such as path planning engineer, high-precision map engineer, sensor fusion engineer, etc AD And technology companies give up because they lack deep experience in vehicle manufacturing ADAS Field, trying to AD Domain transcends tradition Tier1.
To sum up, in order to integrate these two technical routes and facilitate the naming of these two technical routes, the term "intelligent driving" has recently become popular to generally refer to advanced driving assistance (ADAS) and automatic driving (AD). Especially among various new car manufacturers, ADAS and ad routes are generally not distinguished internally, and relevant practitioners are used to calling them intelligent driving engineers.