Autopilot: How the Partial Automation of Airplanes Can Inform On-Road Vehicular Automation

A picture from the 2013 Paris Air Show; picture courtesy of http://abcnews.go.com/Technology/wireStory/flies-hovers-paris-air-show-48100343

On June 19, the 52nd Paris Air Show began. This air show showcases technological innovations and acts as a place where airplane manufacturers showcase their newest planes. This year, a few key changes are being highlighted: how aircraft manufacturers are shifting away from focusing on one-time sales of aircraft to services such as using “Power-by-Hour” contracts and engine leases. One item that is not being highlighted is the automation of the aircraft itself. Airplanes are already highly automated and the civil aviation industry is continuing the process.  While on-road vehicle original equipment manufacturers (OEM) are still trying to figure out how to make vehicles autonomous, air travel has been greatly automated for decades. With all this in mind, the automotive sector would be smart to pay attention to the lessons learned by the civil aviation industry.

While it is an exaggeration to say that a plane “flies itself,” the pilot can- and often does- stop flying actively and tells the autopilot how to fly the plane after takeoff and before landing. The autopilot does not “drive” the plane necessarily; it merely absorbs the initial orders (and change in orders) of the pilot. These autonomous technologies are complemented by advance warning systems that can jolt a pilot to attention when necessary. Ground proximity warning systems (GPWS) warn pilots whether they are about to hit the ground or another obstacle. More impressively, a traffic collision avoidance system (TCAS) use vehicle-to-vehicle (V2V) communication to identify whether another aircraft is in the vicinity, if the two aircraft are destined to crash if they maintain their flight plan, communicate with the TCAS on the other plane (or planes), and then the different TCASs tell the pilots on the different planes how to avoid crashing into each other. As a result of these innovations, passenger aircraft- and other aircraft- no longer have engineers or navigators on board. This means the pilots can be content to mostly monitor the flight, take over when human interaction is necessary, and know the only two times they are always required to pay close attention is during takeoff and landing.

One should note that in-air OEMs and suppliers are looking into further automated innovation in the airspace. Boeing says it is developing its own artificial intelligence to run its computer-driven passenger planes in the future. Meanwhile the Defense Advanced Research Projects Agency (DARPA), the United States’ Defense Department’s research organization, is interested in creating robots that can co-pilot alongside a human. One should also note that airplanes created by the European OEM Airbus are already heavily automated, with each task using three different computers each that give orders and select actions based on a vote. This ensures there is at least one back up computer available in case one of the computers fails. It appears that as on-road automation is becoming a reality, airborne automation is further ahead and continuing towards an apparently natural conclusion.

A Flight Management System (FMS) Unit

While automating flight is a global phenomenon, commercial aircraft automation is regulated differently on a regional basis. In the United States, pilots are required to oversee the plane throughout the whole flight, even if the plane is on autopilot. By contrast Asian carriers require pilots to keep the autopilot on above 3,000 feet (~914 meters). Furthermore, while aircraft made by Boeing prefer that humans make the final decisions while the flight management systems (FMS) guide and assist the pilots, Airbus planes usually fly themselves unless the pilot overrides it. None of these approaches are perfect. Automating the plane can eliminate the chance of human error. If one were to extrapolate statistics from the on-road transportation industry, one should consider the National Highway Traffic Safety Administration (NHTSA) report from the mid-2000s said that 93% of accidents are due to human error. That number includes behaviors such as drinking and texting, but even if one excludes such situations, the data suggests that removing the human element from driving and by extension flying could save lives, if only the technology could get there first.

A white and red Turkish Airlines A330-300 with the undercarriages extended over a blue sky.

An Airbus A330, a heavily automated plane flown by Turkish Airlines

What can the on-road industry learn from all this? First of all, there is the debate about whether vehicles should be partially or fully automated.  Ford and Google have been arguing that one should skip the partially automated vehicle phase and go straight to full automation because people may become too trusting of an (incomplete) autopilot. As a result the “drivers” will stop paying attention to the road and will only return to active driving when they hear the advance warning system signal. The question is how much advance notice one should have. This question was discussed when a Tesla car got in trouble because one of its vehicles hit a truck while its driver was disengaged. In this case the accident escaped legal trouble because the driver had at least seven seconds to notice that he was about to crash and the autopilot gave him several warnings before he crashed. The question of when the autopilot should transfer control to a human is an important question in flight as well. Aside from the issue of how much time is needed to react, there is also the question of whether people can figure out how to react in the time allotted. Civil aviation has its own share of stories regarding warning systems and malfunctioning autopilots. For example, Air France Flight 447 crashed in 2009 partially because the pilots had grown used to highly automated airplanes and lacked the experience to fly the plane properly when the autopilot was disconnected due to weather problems. If the Tesla incident is an example of what can happen when the driver is too distracted or too trusting to heed the warnings, the Air France example suggests the need for backups for the autopilot. This may be why Ford says it will not start selling partially autonomous vehicles (AV). Instead, it says it will focus on manual vehicles (and those with driver assisted technologies) until 2021, when they promise to put their first autonomous vehicles on the market. By contrast BMW, Mercedes-Benz, and Volkswagen AG’s Audi are claiming they will sell semi-autonomous cars in 2018, with capabilities such as automated driving under a certain speed, which will warn drivers to take over with little advance notice, perhaps even 10 seconds. The Air France Flight 447 incident suggests the automotive industry should carefully consider copying the aviation industry’s reliance on multiple backup autopilot systems.

The aviation industry teaches us that regulation is imperative to safety and will remain an important issue, and there is no guarantee that countries will regulate automation the same way across the globe. If we are somewhere between fully manual and fully autonomous vehicles, that means regulators need to ascertain where the line should lie in terms of responsibility. As mentioned above Asian fliers are required to let the autopilot take over for most of the flight and the North Americans are supposed to treat their autopilot as a “flyer assist,” not a strict guide. Overall, we need to remember that autonomous vehicles are not a new, unprecedented technology. We have existing examples of autonomous technologies in aviation and there is no need to reinvent the wheel from scratch.

From The Stable to the Rest Stop and Further On

A horse stable found in modern day Poland; these stables used to be more common before vehicles replaced draft animals. 

Autonomous vehicles will greatly change the landscape. In the past, people traveled on horses and had to stop at individual stables to temporarily house their steeds, feed them, and prepare them to go back out into the world. They also served as a place for humans to rest, eat, and take care of themselves until their horses were ready to travel. Nowadays rest stops serve similar roles for people and their horseless carriages, except instead of caring for one’s draft animal the rest stops are designed to fuel and repair one’s vehicle. Just as unhorsing the carriage has removed the stable from the landscape, automating the vehicle and, in some cases, removing its occupants will also transform the environment. While the occupants of autonomous cars may need to eat, autonomous trucks will only need to stop for fuel and maintenance. If trucks are fully automated, it follows that fueling stations will change as well.

The dehumanization of vehicles will likely change how fueling takes place. At first glance it is unclear why anything must differ. Right now most vehicles require outside aid to be fueled. This will not change in the near-term; few automakers are promising vehicles that can fuel themselves as well as drive themselves. Generally fueling stations have at least one person around and this will remain for the foreseeable future. At first, fueling stations may require more staff to handle the differing fueling and payment protocols. Fueling companies may respond by automating the stations. To get there, a great deal will have to change.

A gas station near Mount Fuji

A great deal of work will need to be done to automate the fueling process. One start-up, Powerhydrant, is developing a way to automate fueling electric vehicles, which can facilitate charging vehicles without the need to have human intervention. Automating the process can make it faster. Any successful method of fueling a vehicle will require a great degree of engineering, since the vehicle will need to know when it needs to be reenergized, whether it is attached to a charger (or a pump), whether it is receiving energy, and when to detach from the fueling station. The vehicle will also have to decide when to stop receiving energy or disengage the charger. Should it charge until it is full, or should it leave “early” so it only has to pay a certain amount of money for its fuel or just enough to finish a trip so the vehicle can recharge at its “home base” for free. Payment will be tricky; if there is no human in the vehicle, the people at the fuel stations need to be trusted to service and charge the vehicle as directed, and the vehicle needs to be programmed to pay and monitor whether it has been serviced properly.

The methods of fueling the vehicle- whether the fuel is petrol, diesel, or electricity- will have to be standardized as well to ensure vehicles can be charged at any of these stations. This is already an issue in Europe regarding electric vehicles (EV); while Tesla uses a particular standard to energize its vehicles in most of the world, in the European Union (EU) it is required to use the standard Mennekes plug and allow competitors’ EV charging stations to charge their vehicles. By contrast non-Tesla vehicles such as the Chevy Bolt cannot be charged at Tesla Superchargers in the United States.

Perhaps an even bigger question is how vehicles will choose where to charge. Right now you can find fueling stations from competing companies across the street, with drivers selecting fueling stations based on (fluctuating) prices, brand, and perceived quality of the fuel. Companies will prefer to program priorities into the vehicles to avoid having to make a decision every time a vehicle needs to be refueled. This process opens up another set of questions. No matter what happens, antitrust issues will be in play. Regulators may frown on the Tesla vehicles avoiding non-Tesla branded chargers, and some regulators may worry about a company owning fueling stations, vehicles, and the fuel.

All these processes need to be well-secured. A cyber criminal could interrupt the payment process by absorbing the funds meant for the fuel station attendant, disrupt the process so the fuel station initiates the fueling process without a vehicle available to receive the fuel, or swipe personal information from the vehicle and/or the fuel station’s records. They could even paralyze transportation by preventing refueling.

The future of the fueling plays into a large change in the world of the IoC: a focus on services. Until recently the main focus in vehicular transportation has been on selling vehicles, fueling them, and maintenance. Right now vehicle sales are dropping within the United States and while the  vehicle sales in China and a few other markets increased in 2016, global sales may soon resemble those of the United States. Perhaps people will rely less on personal transportation now that they can use services such as Instacart for delivery of groceries and Amazon for delivery of many other goods and services. While the vehicle is an important part of the connected vehicle, we need to consider how services such as fueling will be effected. If the companies of today want to survive, their business models will have to change.

Computing on Four Wheels

A Google Car that is being used for mapping

What has gigabytes of storage, dozens of central processing units (CPUs), and stays in one place 95% of the time? Not a data center that is rarely moved, but a connected car. Nowadays even completely manual cars spend 95% of their time parked. This is because people only use their cars to travel to and from work and places of leisure, leaving them unused most of the day. This may change if cars become data centers. Now that people are adding computers and sensors to cars, the new “smartcar” revolution may do as much for cars as smartphones today. Just as smartphones can do more than make calls, “smartcars” can do much more than just move, even if they lack autonomous capabilities.

To start, connected vehicles will help maintain infrastructure (To see more on sensorized signs, see my previous blog). Vehicles can serve a similar function in collecting and relaying the state of infrastructure, weather information, and the flow of traffic; and compare that data to sensors in the signs and the road to ensure a near-perfect information environment. Connected vehicles can scan the infrastructure for potholes, non-sensorized signs, and other points of interest and send them to different services. Google is already patenting this technology in regards to finding potholes. previous blog

Data centers will be more mobile and flexible once they become part of the vehicle. Dr. Yuji Inoue (Chairman, Toyota InfoTechnology Center) has spoken to me before about how vehicles can be used as part of the communications infrastructure. These smartcars can act as mobile homes to web domains which are considered strategic and thus may be safer if they are not in a single place, or they can be data centers that can give extra processing power to organizations that want it, like a hospital that needs the extra processing power to get better access to information or an animation company that needs extra computational power. Dr. Inoue suggests connected vehicles can serve as emergency communication centers, a way to create a “temporary internet” when a natural disaster knocks out the main communication infrastructure.

Picture of a truck with a data center attached to it. Picture courtesy of https://www.esds.co.in/blog/the-concept-of-mobile-data-centers/.

The beauty of having all the vehicles interconnected is that they create a clear picture when viewed altogether. Each vehicle gives you a moving picture, and adding all the vehicles together can create a picture over a terrain. One can install instruments and sensors to help weather organizations better understand and predict the weather. These connected vehicles can act as mobile data centers that collect and synthesize this information. All of this information can be collected across the vehicles and the signs and analyzed to make a more full understanding of the weather.

Connected vehicles may cross the line from consuming infrastructure to becoming it. Navigation applications such as Waze use user reports to actively inform its users of traffic, blocked roads, and many other incoming obstacles. One can use the information collected by many sensorized vehicles do this automatically. Connected vehicles can make traffic safer and more predictable. Helicopters may no longer be needed. Connected vehicles can act as mobile traffic lights to spot traffic jams and accidents from afar. The vehicle known as the RINSPEED Oasis can “light up” the area in front of itself as a way to signal pedestrians it is safe to pass. Perhaps the headlights could be modified so that they emit colored light, such as red lights for “stop” and green lights for “go.” Further modifications can be made for people who are hard of hearing or who are colorblind. Little to none of this requires vehicles to be fully automated, and all of it can benefit from these sensors. As vehicles become infrastructure and data centers, businessmen and regulators will wonder how this may affect questions such as data ownership, the right to sell that data to others, and how this data must be secured.

Sensitive Signposting

As the connected vehicle becomes more commonplace, signs will have to change to improve transportation. The signs we see alongside the road serve an important purpose. They inform drivers and pedestrians about how the vehicles on the road should behave. This is all buttressed by a legal system that requires everyone on the road and crossing the road to follow the rules, to ensure that everyone acts in a predictable and safe manner. Connected vehicles offer an opportunity to take the existing system of signs and make them more efficient. They can continue to give human drivers and pedestrians information, give guidance to driving programs, and sensors in the signs can give information about the weather, state of the road, and incoming traffic jams to incoming cars. Theoretically the connected vehicle can exist with the current infrastructure, but we can take advantage of the new technology to make infrastructure more effective than ever before.

What we recognize as a stop sign is just a piece of metal to a connected vehicle unless it is programmed to recognize it. This task is more complicated than one might expect, as the computer scientist Artur Filipowicz of Princeton University described. In his paper “Using Virtual Worlds, Specifically GTA5, to Learn Distance to Stop Signs,” he collected more than 1 million images of stop signs from the videogame Grand Theft Auto V in different environments, weather conditions, and times of day to help researchers understand how to make a similar system to help “real” cars identify stop signs. Such a system will have to adjust for different possibilities, such as the sign being defaced. This does not take into account that some countries use different symbols for stop signs. For instance, the Japanese stop sign is an inverse red triangle. There is a simpler solution to this problem: use sensors to tell cars what the humans already know.

We can add sensors to the existing infrastructure so that connected vehicles can quickly recognize signs without having to compare them against millions of pictures. Las Vegas in Nevada and Frisco in Texas are going further by connecting their traffic signal networks with pilot vehicles. The two cities believe this will make traffic smoother by giving the drivers access to more information relating to traffic lights. The pedestrians will not have to alter their behavior at all.

 

Sensors can do more than just traffic control; they can collect and relay the state of the infrastructure, weather information, and the flow of traffic. Sensors can also be added to roads. Road sensors can tell whether the roads need to be repaired soon or if the ground will be slippery due to rain. Volvo tested magnetic roads that can sense and guide connected vehicles. This technique can help the platoon management of driverless cars. Sensors can also help maintain infrastructure. A group of engineers at Carlos III University in Madrid, Spain created a sensor that can measure how much salt is poured on roads to prevent freezing in real time. Excessive salt can harm the vehicles on the road, the infrastructure, and the environment at large.

Adding sensors to all signs will take time. This could lead to accidents where a connected truck passes by a “sensorless” left turn only sign and fails to turn left. Regulations may be required to “sensorize” the signs because getting it wrong can be fatal. Until sensorization is widespread, cars equipped with such technology may be restricted to sensorized roads. All of these issues can be magnified by the threat of cyber attacks which can reconfigure the signs and cause chaos. All of these sensors must be secured if they are to be trusted. However, overall adding sensors will greatly improve transportation. Technically one can have an autonomous vehicle with the current infrastructure, but if you really want to move forward, you have to remodel the road as well.

The Employment Gap and the Connected Truck

The Employment Gap and the Connected Truck

The above picture is of an autonomous truck that is operated by Uber’s self-driving company, Otto

The American Trucking Association (ATA) believes that freight volumes will increase by 29% over 2015-2026, and this requires the addition of 90,000 jobs in the United States per year to meet demand. However, issues such as pay and erratic hours have created a trucker shortage in this industry. While some companies are trying to solve this problem by making these jobs more enticing to workers, connected vehicle technologies can use Internet of Things technology to fill the gap.

There are currently several approaches to filling this gap. One method is to replace the humans with fully autonomous trucks, but this will not happen within the next few years. Ford and Volvo are promising to put self-driving vehicles on the market by 2021 and 2020 respectively, but even if they meet their objectives it is unlikely that trucking companies will immediately purchase platoons of autonomous trucks. They may eventually replace their human drivers with autonomous trucks- especially since the average age of a trucker is 49 as of 2015 and the ATA expects the number of open driver positions to increase if the economy picks up- but it is doubtful in the short term.

One solution is to use existing technology to have a single human driver drive one truck at the front and several of the trucks behind his truck. Nikola Motors, an electric truck company, is aiming to have a human driver drive the truck they are in and have up to seven trucks drive behind him. This would be scalable, since the technology could be used to drive two trucks at once to a single location and more trucks could be added to create a fleet of trucks. The main issue is that these trucks may not be able to drive to single locations, since trucks are quite large and can take generous amounts of road wherever they are.

One person who has hit on a pre-autonomous solution for vehicles in general and trucks in particular is Maarten Sierhuis, the Director of Nissan Research Center Silicon Valley: “teleoperation.” Since autonomous vehicles will not be a reality in the short term, automotive manufacturers and other companies related to the automotive industry allow humans override the vehicles from afar whenever an autonomous vehicle meets an obstacle it does not understand. Mr. Sierhuis himself believes autonomous vehicles will not be here in the next five to ten years. Instead of putting an autonomous vehicle on the road and trusting it, he and others engage in “Seamless Autonomous Mobility,” where autonomous vehicles are put on the roads while humans watch them through vehicles and other sensors from afar to ensure they drive safely and keep them out of accidents.

Nissan is not the only company that takes advantage of teleoperation. Karl Iagnemma, co-founder and CEO of the autonomous ride-sharing company nuTonomy also says his company is developing a teleoperation system. Google’s Waymo is studying the idea, Uber filed a patent for an autonomous vehicle following a human driven vehicle or being driven through remote control in 2015, autonomous vehicle start-up Zoox has a patent on it, and so does Toyota. One should note that the Nissan vehicles do not switch to “remote control” in times of crisis, only in situations it does not understand, like how nuTonomy’s vehicles had to learn about seagulls. In Nissan’s case, the vehicles are currently wired to ping Nissan’s control centers in certain situations and wait for new instructions. Nissan knows there is a big market for this- CEO Carlos Ghosn sees multiple markets for fleets of vehicles, including logistics companies (cargo).

One can see how heavy use of teleoperation is useful in a pre-automation world. While companies tend to use teleoperation for testing purposes, it can act as an intermediate step between the current stage of driving and full automation. It puts fewer people on the roads and allows companies to put drivers into shifts instead of driving on the road for days at a time. There are still a few issues with this semi-autonomous solution. For instance, the “drivers” will have to eliminate problems related to latency. Furthermore, the remote drivers must ensure the vehicles “learn” so humans can intervene less and less, such as when vehicles learn how to respond to traffic cones. Teleoperation is seen as a stage in the process of automation and should be treated as such.

Overall though, teleoperation is scalable and will help fill the trucking job gap until automation is more accepted. It will shrink the employment gap by allowing truckers to operate multiple trucks at once, rotate the operators, and/or have call centers where truckers can drive remotely or take over when the trucks cannot process what they need to do next. There will be political questions in the short term, since successful use of partial-to-full teleoperation in the short term (and full automation in the long term) could affect overall employment. It will also require creating new signals on the vehicles themselves, since inevitably one of these remote cars will come across an accident on the road and will have to “explain” its intentions to the people there, especially if the remote driver decides to intervene. There is also the issue of whether trucks may be owned by one company while operated by another. Even though teleoperation may be in the short term a stopgap measure to fully automated trucking, it will transform the trucking industry.

“Breaker 19 – Expect IIoT to Deliver a Convoy of Autonomous Trucks”

“Breaker 19 – Expect IIoT to Deliver a Convoy of Autonomous Trucks”

by Alexander Soley (on left), Connected Vehicle Research Analyst for Dell Technologies, with a foreword by Dr. Said Tabet (on right), Chief Architect for IoT Solutions, Dell EMC

ForewAMSSOLEYord:SAITABE2

As an IIC testbed lead and member of the IIC Steering Committee, I am very focused on keen insights into movements of IoT that will cause disruptions in various markets.  At the last IoT Solutions World Congress in Barcelona  my panel on Connected Vehicles featured a new voice analyzing strategies for succeeding in the connected vehicle marketplace, Alexander Soley — the video of our session is online. Mr. Soley has since published on the topic of connected vehicles in Digiworld Economic Journal, and now shares some thoughts with us on the coming disruption in freight carriage — connected trucks.

Blog: IIoT Delivers Autonomous Trucking

Most people are aware of connected cars and should consider the changes that will come as trucking is connected as well. Autonomous trucking is coming, and the trucking industry needs to think about how it may fill the current labor shortage, how automation will help the industry, and what regulations will be put into place.  Just as containerization revolutionized shipping, automation of trucks will greatly change the trucking industry.

Most people are aware of connected cars and should consider the changes that will come as trucking is connected as well. Autonomous trucking is coming, and the trucking industry needs to think about how it may fill the current labor shortage, how automation will help the industry, and what regulations will be put into place.  Just as containerization revolutionized shipping, automation of trucks will greatly change the trucking industry.

The trucking industry makes up 70.1% of the United States’ domestic freightage and should retain a majority of it for at least another decade. In fact, the American Trucking Association (ATA) expects that freight volumes will increase by 29% over 2015-2026. This will require the firms to add 90,000 jobs per year to meet demand, even as trucking will take a smaller share of overall freight transportation over time, with the ATA suggesting that trucking will only make up 62.5% of transportation by 2026. However, a confluence of different factors such as pay and erratic hours makes it hard for the trucking industry to meet that target. This is already resulting in a trucker shortage and companies prioritizing driver attraction and retention. While full automation may eliminate truckers in the long term, it is also true that it may fill jobs that are being opened but not being filled by human beings.

Mercedes-Benz is piloting autonomous freightliner trucks on public roads in Nevada. There are other players in the market such as Uber, Amazon, and Tesla. Nikola Motors is considering it, although right now they are focusing on technology that allows them to have one driver drive multiple trucks at once. Right now Mercedes-Benz advertises its pilot as a way to make the roads safer and free up the drivers by making “their job more attractive by assigning them other tasks.” Trucking is a long and arduous job where truckers may be away from their homes for days at a time. If one automates even parts of the trip, the driver will have more tools that will allow him or her to better analyze the situation or perform other tasks such as studying or data entry. In fact, a trip can be partly automated so one can learn a skill during the slower parts of the trip and reserve one’s attention for the more difficult parts of the trip. If one can fully automate the trip, trucking will become more predictable since the same programs will be “driving” each time and driver behavior will not enter the equation. It will also eliminate downtime, since autonomous trucks do not need to stop to satisfy the driver’s needs to get refreshments.

It is unclear as of right now whether it would be cheaper to hire a trucker or to automate the car entirely. The Mercedes-Benz trucks are on the public roads already, but they are not ready to leave the pilot stage, especially without a driver. If there is no one on the truck, sensors will have to be installed to ensure the safety of the cargo and the truck. There are also a great number of externalities to consider, for instance how automating trucking will affect such factors as fueling stations. These will be written about in another blog. Furthermore, all these issues bring up the crucial question of regulation.

As the trucking industry automates more and more, regulations will focus less on driver behavior and more on program behavior. As of right now many private companies say they expect the modern infrastructure of lines on the roads and signage will be enough, and most of them are planning their technologies with little to no expectation of change. There are also important issues in terms of who has access to the trucking data. Many, if not most, of the regulations listed on the Federal Motor Carrier Safety Administration’s website focus on regulations surrounding the driver, although some such as 49 CFR 383.37 and 49 CFR 387.7 can be reinterpreted so that the “employer” or “motor carrier” can be the person who “employs” the autonomous trucks. There will also be the issue of varying regulations. For example, California currently requires a driver at the wheel while Florida does not, and the vast majority of the nation which do not have laws on the subject at all. Furthermore, this all assumes that cars and trucks should be regulated in the same way, and yet the requirements of a connected car making a short run with a passenger are different from a thousand-mile trip with flammable cargo.