![Don’t Be Evil: should the tech sector prevent killer robots? Don’t Be Evil: should the tech sector prevent killer robots?](https://spectator.clingendael.org/sites/spectator/files/styles/header_image/public/2019-09/Kayser-Drone-Pixabay.jpg?h=234eeb33&itok=xod5B7Ie)
Don’t Be Evil: should the tech sector prevent killer robots?
With the emergence of the digital era, militaries have increasingly been looking for cooperation with the tech sector. The risk of involvement of tech companies in the development of lethal autonomous weapons has led to heated discussions. What purposes may their technology be used for? Are there clear red lines? For peace ngo PAX it is crucial that companies make explicit where they draw the line regarding increasing autonomy in weapon systems to avoid contributing to the development of lethal autonomous weapons.
Ten years ago no one would have expected that a company like Google would be involved in a military project. However in 2018 it became clear Google was working with the Pentagon on project Maven, which uses artificial intelligence to interpret video images, which could provide the basis for automatic targeting and autonomous weapon systems.1 This led to Google employees writing an open letter that called on Google to cancel Project Maven, stating “Google should not be in the business of war”.2 Following the staff’s letter, Google decided to not renew its contract and has since published ethical AI principles, which state that Google will not design or deploy AI in “weapons or other technologies whose principal purpose or implementation is to cause or directly facilitate injury to people”.3
![Silicon Valley from above. © Patrick Nouhailler Flickr](/sites/spectator/files/inline-images/Kayser%20-%20Silicon%20Valley%20from%20above.%20%C2%A9%20Patrick%20Nouhailler%20Flickr.jpg)
In the past few years, there has been increasing debate within the tech sector about the impact of new technologies on our societies. Concerns related to privacy, human rights and other issues have been raised. The issue of weapon systems with increasing levels of autonomy, which could lead to the development of lethal autonomous weapons, has also led to discussions within the tech sector.
The latest PAX report ‘Don’t Be Evil?’ looks at the risk of tech companies (unintentionally) becoming involved in the development of lethal autonomous weapon systems, popularly known as ‘killer robots’.4 As part of the report, fifty tech companies were surveyed in order to establish their level of concern.
The use of AI to allow weapon systems to autonomously select and attack targets is highly controversial
This is an important debate in which tech companies play a key role. To ensure that this debate is as fact-based and productive as possible, it is valuable for tech companies to articulate and publicise clear policies on their stance, clarifying where they draw the line between what AI technology they will and will not develop.
Lethal autonomous weapon systems
Lethal autonomous weapon systems are weapon systems that could select and attack targets without meaningful human control. This means that the decision to use lethal force is delegated to a machine, and that an algorithm can ‘decide’ to kill humans. The function of autonomously selecting and attacking targets could be applied to various autonomous platforms, for instance drones, tanks, fighter jets or ships.
With the emergence of the digital era, militaries have increasingly been looking for cooperation with the tech sector. The use of AI by militaries in itself is not necessarily problematic, for example when used for autonomous take-off and landing, navigation or refuelling. However the use of AI to allow weapon systems to autonomously select and attack targets is highly controversial. The development of these weapons would have an enormous effect on the way war is conducted. It has been called the third revolution in warfare, after gunpowder and the atomic bomb. Many experts warn that these weapons would violate fundamental legal and ethical principles and would destabilize international peace and security. In particular, delegating the decision over life and death to a machine is seen as deeply unethical.
Increasing autonomy in weapon systems
It has become evident in recent years that militaries are striving to increase the levels of autonomy throughout their weapon systems. Many have placed AI as a high priority area of work. There are multiple examples of weapon systems in use today that illustrate the concerns regarding increasing levels of autonomy.5
National militaries are keen to profit from the expertise held in the private sector, often outsourcing certain aspects of military projects to tech companies. Today, a main issue is that most technologies are dual-use, meaning they can have both civilian and military applications. Nowadays, even seemingly innocent tech tools can be weaponised and repurposed for the battlefield. An example is that of the HoloLens, Microsoft’s Augmented Reality Goggles. These were originally designed to be used by technicians, doctors and gamers. In November 2018 it was reported that Microsoft had secured a USD 480 million contract with US military to supply 100.000 headsets for training and combat purposes. This contract intends to “increase lethality by enhancing the ability to detect, decide and engage before the enemy”. The news of this contract also drew protests from within the company, with an open letter being addressed to both the CEO and the President of Microsoft.6
![An unmanned aerial vehicle, the UAVision Ogassa V, prepares to take off in Troia, Portugal during an unmanned systems trial in September 2019. © NATO](/sites/spectator/files/inline-images/Kayser-An%20unmanned%20aerial%20vehicle%2C%20the%20UAVision%20Ogassa%20V%2C%20prepares%20to%20take%20off%20in%20Troia%2C%20Portugal%20during%20an%20unmanned%20systems%20trial%20in%20September%202019.%20%C2%A9%20NATO.jpg)
Many emerging technologies are dual-use and have clear peaceful uses. In the context of the PAX report, the concern is with products that could potentially also be used as key-components in lethal autonomous weapons. Moreover, there is the worry that unless companies develop proper policies, some technologies not intended for battlefield use may ultimately end up being used in weapon systems.
The development of lethal autonomous weapons takes place in a wide spectrum, with levels of technology varying from simple automation to full autonomy, and being applied in different weapon systems’ functionalities. This has raised concerns of a slippery slope where the human role is gradually diminishing in the decision-making loop regarding the use of force, prompting suggestions that companies, through their research and production, must help guarantee meaningful human control over decisions to use force.
The PAX report
In the hope of contributing to the discussion, the PAX report illustrates some developments in this area, with varying levels of (proclaimed) autonomy or use of AI. A number of technologies can be relevant in the development of lethal autonomous weapons. Companies working on these technologies need to be aware of that potential in their technology and they need to have policies that make explicit how and where they draw the line regarding the military application of their technologies. The report looks at tech companies from the following perspectives: big tech, hardware, AI software and system integration, pattern recognition, autonomous (swarming) aerial systems, ground robots.
21 companies are classified as ‘high concern’
For the report fifty companies from twelve countries, all working on one or more of the technologies mentioned above, were selected and asked to participate in a short survey, asking them about their current activities and policies in the context of lethal autonomous weapons. Based on this survey and our own research PAX has ranked these companies based on three criteria:
- Is the company developing technology that could be relevant in the context of lethal autonomous weapons?
- Does the company work on relevant military projects?
- Has the company committed to not contribute to the development of lethal autonomous weapons?
Based on these criteria, seven companies are classified as showing ‘best practice’, 22 as companies of ‘medium concern’, and 21 as ‘high concern’. To be ranked as ‘best practice’ a company must have clearly committed to ensuring its technology will not be used to develop or produce autonomous weapons. Companies are ranked as high concern if they develop relevant technology, work on military projects and have not yet committed to not contributing to the development or production of these weapons.
![German Rheinmetall KZO drone being launched during NATO's Iron Wolf II exercise in 2017 in Lithuania. © NATO](/sites/spectator/files/inline-images/Kayser-German%20Rheinmetall%20KZO%20drone%20being%20launched%20during%20NATO%27s%20Iron%20Wolf%20II%20exercise%20in%202017%20in%20Lithuania.%20%C2%A9%20NATO.jpg)
Both Amazon and Microsoft are ranked as high concern. They both work on various relevant technologies including recognition software, drones and cloud infrastructures. Both companies are currently bidding for JEDI, a USD 10 billion contract that will serve as the cloud infrastructure spanning the Pentagon to soldiers on the ground. Chief Management Officer of the project has explained: “This program is truly about increasing the lethality of our department”.7 Without clear policies that make explicit what purposes their technology may be used for they risk contributing to the development of lethal autonomous weapons.
Another example of company of concern is AerialX, a Canadian company that produces the DroneBullet. It is a counter-drone system designed to take out rogue drones in the sky. The DroneBullet is a small kamikaze drone that looks like a miniature missile. It has an operational endurance of ten minutes and can cover a maximum of 3km. However, it can reach speeds of up to 200km/h and uses a machine vision target system.
The target system is an AI-led capability that enables the DroneBullet to autonomously identify, track and engage or not engage an approved target set. Although it is designed to only engage specifically approved drones based on certain characteristics, it is not difficult to imagine how such a technology could work with a different target library against other targets. In fact, AerialX is currently working to modify the weapon for a warhead-equipped loitering munition solution.
For PAX it is crucial that companies make explicit where they draw the line regarding increasing autonomy in weapon systems. They should set up clear policies to make explicit what purposes their technology may be used for and what their clear red lines are. Without such policies, companies risk contributing to the development of lethal autonomous weapons.
Google now has policy stating that it will not develop AI for weapons systems
What can tech do?
As the above examples show, tech companies have an important role to play to prevent the development of killer robots. There are concrete steps companies can take to prevent their products contributing to the development and production of lethal autonomous weapons.
- Commit publicly to not contributing to the development of lethal autonomous weapons.8
- Establish a clear policy stating that the company will not contribute to the development or production of lethal autonomous weapon systems, and including implementation measures such as:
- Ensuring each new project is assessed by an ethics committee;
- Assessing all technology the company develops and its potential uses and implications;
- Adding a clause in contracts, especially in collaborations with ministries of defence and arms producers, stating that the technology developed may not be used in lethal autonomous weapon systems.
- Ensure employees are well informed about what they work on and allow open discussions on any related concerns.
Several of the companies surveyed for the report have already taken similar steps.9 As previously mentioned, Google now has policy stating that it will not develop AI for weapons systems.
![As part of NATO's Unified Vision 2014 Trial, members of the Italian Air Force launch a surveillance drone (STRIX, a multi-purpose, man-portable, totally autonomous TUAS) over Oerland, Norway © NATO](/sites/spectator/files/inline-images/Kayser-As%20part%20of%20NATO%27s%20Unified%20Vision%202014%20Trial%2C%20members%20of%20the%20Italian%20Air%20Force%20launch%20a%20surveillance%20drone%20%28STRIX%2C%20a%20multi-purpose%2C%20man-portable%2C%20totally%20autonomous%20TUAS%29%20over%20Oerland%2C%20Norway%20%C2%A9%20NATO.jpg)
In response to our survey Google added that “since announcing our AI principles, we’ve established a formal review structure to assess new projects, products and deals. We’ve conducted more than 100 reviews so far, assessing the scale, severity, and likelihood of best- and worst-case scenarios for each product and deal”. VisionsLabs, a Russian pattern recognition company, told us that they explicitly prohibit the use of their technology for military applications and that it is part of their contracts. In addition, VisionsLabs said that they also monitor the results/final solution developed by their partners.
Current developments in tech, and military involvement of tech companies, demonstrate how it is crucial that companies take active steps to make sure that none of their technology is used in the development of lethal autonomous weapon systems. Concern among the tech sector is widespread. Indeed, more than 240 companies and organisations, as well as more than 3’200 individuals have signed a pledge to never develop, produce or use lethal autonomous weapon systems.10 This is an important voice in the debate as these experts understand the technology and the consequences of their use in autonomous weapon systems.
As the tech sector possesses the technology required for the next phase of lethal autonomous weapons development, it is now vital for all companies to take additional steps to make sure that these weapon systems remain science fiction.
- 1Campaign to Stop Killer Robots, ‘Google, other companies must endorse ban’, 16 May 2018, and Cheryl Pellerin, ‘Project Maven to Deploy Computer Algorithms to War Zone by Year’s End’, DoD News, 21 July 2017.
- 2 ‘Letter to Google CEO’, The New York Times.
- 3Google, ‘Artificial Intelligence at Google: Our Principles’.
- 4PAX, ‘Don't be evil? A survey of the tech sector’s stance on lethal autonomous weapon’, August 19 2019.
- 5PAX, ‘Where to draw the line - Increasing Autonomy in Weapon Systems – Technology and Trends’, November 12 2017.
- 6Microsoft Workers 4 Good, Twitter status, 22 February 2019.
- 7Employees of Microsoft, ‘An Open Letter to Microsoft: Don’t Bid on the US Military’s Project JEDI’, Medium, 12 October 2018.
- 8See e.g. Ariel Conn, ‘AI Companies, Researchers, Engineers, Scientists, Entrepreneurs, and Others Sign Pledge Promising Not to Develop Lethal Autonomous Weapons’, Future of Life Institute, 18 July 2018.
- 9PAX, ‘Lethal autonomous weapon systems and the tech sector: some examples of best practices’, August 19 2019.
- 10The Future of Life, ‘Lethal autonomous weapons pledge’, 2018, and Ariel Conn, ‘AI Companies, Researchers, Engineers, Scientists, Entrepreneurs, and Others Sign Pledge Promising Not to Develop Lethal Autonomous Weapons’, Future of Life Institute, 18 July 2018.
0 Comments
Add new comment