SSH Key Management

The SSH protocol is the global gold standard for remote system administration and secure file transfer. SSH (Secure Shell) is used in every data center and in every major enterprise. One of the features behind the immense popularity of the protocol is the strong authentication using SSH keys.

SSH keys are a critical access management problem

SSH keys provide the same access as user names and passwords. Furthermore, they often grant access to privileged accounts on the operating system level, giving a command line. Yet, in many cases, SSH keys have been completely overlooked in identity and access management planning, implementation, and audits. Users have been able to create and install keys without oversight and controls. This has led to violations of corporate access policies and dangerous backdoors.

Over the last few years, it has turned out that most large organizations have massive numbers of SSH keys in their environment. These keys are like passwords. They grant access to resources - production servers, databases, routers, firewalls, disaster recovery systems, financial data, payment systems, intellectual property, and patient information.

Information security starts from controlling who is given access to systems and data. If there is no control over access, there is no security, no confidentiality, no integrity, and no guarantees of continued operation.

Overview video

Intro into SSH keys and SSH key management

Link to SSH Risk Assessment

Insight from real customer cases

We have worked with many companies, including several global top-10 banks, leading retailers, and other large Fortune 500 companies. Based on our findings, most organizations:

• Have extremely large numbers of SSH keys - even several million - and their use is grossly underestimated

• Have no provisioning and termination processes in place for key based access

• Have no records of who provisioned each key and for what purpose

• Allow their system administrators to self-provision permanent key-based access - without policies, processes, or oversight.

In the case of one representative customer, we went through a quarter of their IT environment as part of a major SSH key management project. They had five million daily logins using SSH, most of them using SSH keys for automation. We analyzed 500 business applications, 15000 servers, and found three million SSH keys that granted access to live production servers. Of those, 90% were no longer used. Root access was granted by 10% of the keys.

NIST issues guidance on SSH key management

US National Instute of Standards and Technology (NIST) as issued guidance on SSH key management as NIST IR 7966. It is a good starting point for understanding how to manage access using SSH. We wrote most of the NIST guidelines, and have expanded upon them in our internal processes. We also invented SSH (Secure Shell). We are the best subject matter experts in the field.

Regulatory compliance requires SSH key management

Typical requirements for compliance include:

• Managing identities and credentials - SSH keys are access credentials

• Provisioning and termination process for access - including access based on SSH keys

• Segregation of duties - elimination of key-based access from test and development systems into production

• Disaster recovery - limiting attack spread from primary systems to disaster recovery sites and backup systems

• Privileged access controls - SSH keys are often used to bypass jump servers

• Boundary definition and documentation of connections for payment systems, financial data environments, patient data environments, or between government information systems

• Incident response and recovery - being able to change compromised SSH keys.

The analyst view on SSH keys

• A Gaping Hole in Your IAM Strategy (IDC)

• SSH Governance is Needed to Reduce Risk & Bridge the Trusted Access Gap (IDC)

• Smartly Manage Secure Shell Keys (IDC)

The risk of unmanaged SSH keys

Unmanaged access exposes organizations to significant risks that could in the worst case bring down critical information systems for months. Unmanaged keys risk systemic failure of critical infrastructure, especially in a cyberwarefare scenarios.

Stolen and misused keys

There are far more SSH keys in circulation than anyone seems to believe. In one financial sector customer case we encountered 3 million keys (750,000 distinct keypairs) from 15,000 servers. In another case there were 4.5 million authorized keys from 100,000 servers. Typical numbers for Fortune 500 companies range from hundreds of thousands to millions - many times more than they have employees or system administrators with command line operating access.

With so many unmanaged keys the likelihood of keys being misused, stolen, or used as part of an attack is high. One single key can be enough to gain undetected access to critical systems and data.

The impact of compromise is very high

We have found that in several customer cases about 10% of the discovered keys grant root access. An attacker getting root means they can do anything on the server - including inject fraudulent data, subvert encryption software, install persistent malware, or outright destroy the system. Confidentiality, integrity, and continuity of operations are all compromised. Even if the key gives non-root access, local privilege escalation vulnerabilities can often result in the attacker gaining root access.

There is substantial risk that SSH keys can be used to spread an attack to a majority of all production servers in the organization. This could cause billions of dollars of damage to shareholders. Many companies use SSH keys to push data to disaster recovery sites and backup systems, exposing them to attacks as well.

When keys are used for file transfers between business partners, they may be used to spread the attack between organizations. Improperly configured SFTP file transfer connections may be used to log into other organizations using stolen keys. The keys may be used for stealing other authentication credentials or installing data collection software for furthering the attack in other ways.

For the modern society, cyberwarfare is a relevant risk, and a coordinated attack across critical infrastructure with the intention to destroy and confuse is a real possibility. Recently leaked CIA hacking tools were collecting SSH keys.

Our solutions to SSH key management

The role of SSH, the company, in SSH key management projects is typically to provide best-of-breed software, best subject matter expertise, and help to structure and manage the project and define SSH-related policies. We typically provide 1-2 subject matter experts for larger projects to work with the customer's engineers or outsourcing partners.

SSH Risk Assessment™

We usually start with an assessment of SSH and SSH key usage, even before any formal project starts. Our Risk Assessment software, process and reporting helps understand the severity of the issue and evaluate risk and priority. Our SSH Risk Assessment service is designed to be fast and non-invasive.

Universal SSH Key Manager®

Universal SSH Key Manager is our flagship product for managing SSH keys. It has been used by numerous large and mid-sized organizations for solving their key management problems. It handles the entire lifecycle for key-based access, and integrates to leading identity and access management systems, privileged access and privilege elevation systems, as well as SIEM (Security Information and Event Management) and configuration management.

Together with consulting services, Universal SSH Key Manager makes it easy for customers to solve their key management issues. We also can run the whole project for you, working with your application teams and identity & access management, cryptography, security engineering, operations, and/or IT transformation groups as needed, globally. We've run SSH key management projects in the US, UK, Germany, and Singapore, among others.

 

 

PrivX® lean privileged access management

We also recommend looking at PrivX On-Demand Access Manager. It is a lean privileged access management solution that helps eliminate risk from passwords, vaults and unmanaged SSH keys.

 

Request more information

Please use our contact us form to get in touch with your local SSH.COM expert >