In recent years, the demand for advanced driver assistant systems, and autonomous driving applications has increased tenfold. This is due to the fact, that most accidents are caused by a single factor: the human driver. If we could eliminate this one great weakness, personal transport would become safer, cheaper, and faster. Although autonomous cars do not make the same mistakes humans do – they do not drink, do not fall asleep, do not speed – on-board systems still have weaknesses.

A sensor system mounted on the autonomous vehicle, among its many advantages will always face problems, that are difficult to solve such as occlusion of objects and the large cost of precise vehicle-mountable sensors. On-board computers will always face the issue of limited computational capacity due to cost, and limited electrical power, although the latency of the external communication does not cause a problem. Additionally, the vehicle must make decisions without having up-to-date information about the states and intentions of other vehicles in its vicinity, therefore it cannot behave optimally.

In our vision, the Central System we developed can make all the decisions in the cloud. Sensor stations would recognise, and localize the vehicle, which would not need complex hardware or expensive sensors, keeping vehicle consumer prices down. Our Central System keeps a real-time digital twin of the area of interest, keeping track of all actors and their intentions digitally in the cloud. This enables very efficient and precise decision making and control that can run on powerful servers with high framerates.  The smart city/parking garage/test centre equipped with our system could calculate the necessary control decisions optimally not just for a single vehicle, but for all vehicles subscribed to the service. This would enable high level decisions about traffic flow, and emergency rerouting, making the system far more efficient than today’s individual automated vehicles. High speed communication interfaces can then send the signals to the actuators on the vehicles. This is what is called Control as a Service or (CaaS), meaning, that the infrastructure controls the vehicles that subscribe to this service.

To realize this vision, we tested the first ever example of a Cloud Control system on a real vehicle on the ZalaZONE test track this April. An experiment in which a real car is controlled from the cloud has – to the best of our knowledge – never taken place before.

In our experiment, a test vehicle was following a predefined trajectory, when a pedestrian stepped into its path. The Central System, which detected the pedestrian using our distributed sensor stations, preformed the necessary sensor fusion and object-level tracking, detected the danger by calculating the distance of the pedestrian from the vehicles path, made the decision to stop the vehicle and used the closed loop control directly running in the cloud to bring the vehicle safely to a standstill. Collision avoided!  

Why is this special? Well, the vehicle received the control signals – the throttle and braking values and steering angle – directly from our central system. This has never been done before in the public literature.

How does it work? Our sensor stations are equipped with camera and lidar sensors. An object detector recognised the pedestrian on the image, and based on the lidar sensor’s distance measurements, the pedestrian was localized on the road to create an object detection. The detections were fused in the Central System, which, based on the vehicle’s position and trajectory and the pedestrian’s location, decided whether an emergency stop is necessary. The controller calculated the necessary throttle and breaking values and steering angle based on the vehicle’s reported states to keep it on trajectory, and if necessary to bring the vehicle to a safe standstill. When the danger has passed, the vehicle can drive on.

During our experiment, we verify, that cloud-based control of a real test vehicle is indeed possible, and our system can avoid a collision. In the future, we plan to test how the vehicle can be controlled using our perception system rather than the vehicle’s position reports. These experiments are required to develop our Central System that could, in the future, be responsible for traffic safety in a smart city.