sshd – SSH server process

sshd is the OpenSSH server process. It listens to incoming connections using the SSH protocol and acts as the server for the protocol. It handles user authentication, encryption, terminal connections, file transfers, and tunneling.

How to Install SSH Server on Linux

The SSH server usually comes up as a readily installable package on most linux distributions. However, it is not always installed by default. You can try ssh localhost to test if it is running; if it respons with something like Connection refused, then it is not running.

On Debian-derived distributions, the command to install an SSH server is usually:

aptitude install openssh-server

On Red Hat derived distributions, the command would usually be:

yum install openssh-server

These commands must be run as root.

If the server does not start automatically, try using the service sshd start command, or just reboot the computer.

To get an SSH server for Windows, see Tectia SSH. To get one for IBM z/OS mainframes, see Tectia SSH for Mainframes.

Startup and Roles of Different sshd processes

The sshd process is started when the system boots. The program is usually located at /usr/sbin/sshd. It runs as root. The initial process acts as the master server that listens to incoming connections. Generally this process is the one with the lowest process id or the one that has been running the longest. It is also the parent process of all the other sshd processes. The following command can be used to display the process tree on Linux, and it is easy to see which one is the parent process.

ps axjf

For example, it is easy to see in the following output that process 2183 is the master server.

 PPID   PID  PGID   SID TTY      TPGID STAT   UID   TIME COMMAND  ...     1  2183  2183  2183 ?           -1 Ss       0   8:51 /usr/sbin/sshd -D  2183 12496 12496 12496 ?           -1 Ss       0   0:00  \_ sshd: cessu [priv] 12496 12567 12496 12496 ?           -1 S    15125  24:07  |   \_ sshd: cessu  2183 12568 12568 12568 ?           -1 Ss       0   0:00  \_ sshd: cessu [priv] 12568 12636 12568 12568 ?           -1 S    15125   0:00  |   \_ sshd: cessu@pts/2 12636 12637 12637 12637 pts/2    12637 Ss+  15125   0:00  |       \_ -zsh ...

The other sshd processes are child processes that serve a single connection. A new process is created for each new SSH session.

If the SSH server is upgraded or restarted, only the master server is generally restarted. The server has been designed so that the server processes serving existing connections continue to operate. This minimizes the disruption to users when, for example, server configuration is changed. The easiest way to restart the SSH server is usually to use service sshd restart. However, care should be taken when upgrading configurations remotely, as errors could prevent connecting to the server again (see below).

It is also possible to kill individual processes by killing the server process for a particular user, terminal, or command. This could be done, e.g., using the kill -9 <processid> command.

It is also possible to run multiple master sshd processes on the same system. This is very unusual, but we have seen enterprise customers with more than ten servers running simultaneously with different configurations. Each server would need to listen to a different port and usually would have a different configuration file.

Configuration File

The SSH server has a configuration file, usually /etc/sshd/sshd_config. The configuration file specifies encryption options, authentication options, file locations, logging, and various other parameters. For a detailed description, please the sshd_config documentation.

Logging

The SSH server uses the syslog subsystem for logging. There are many ways to configure syslog and several syslog servers. Many enterprises also collect syslog data into their centralized SIEM (Security Incident and Event Management) system.

On most systems, syslog is configured to log SSH-related messages by default into files under /var/log/. On Debian-derived systems, the default log file is usually /var/log/auth.log. On Red Hat derived systems, the default log file is usually /var/log/secure.

Both the syslog facility and logging level can be configured in the server configuration file. It is strongly advised to set the logging level to VERBOSE, so that fingerprints for SSH key access get properly logged. Newest OpenSSH versions may log them automatically, but many Linux distributions still come with versions that don't log fingerprints without this setting. See SSH key management for why this is important.

Debugging SSH Connection Problems

Sometimes logging into an SSH server just doesn't seem to work, and it can be difficult to figure out what the problem is. There are basically three tools that help diagnose connection and authentication problems:

SSH Client -v Option

The first approach is to add the -v option when calling the client on the command line. For example:

ssh -v [user@]host

This will print verbose debugging output that can usually identify what the problem is. Thigs to check include:

• Does it successfully establish the TCP connection to server? If not, it could be DNS or routing problem or the server could be down. If the output includes Connection established, then the connection was successful.

• Check the user name that it is trying to authenticate as. Look for a line containing Authenticating to <hostname> as '<username>'.

• Check that it successfully negotiates encryption. If you see a line containing SSH2_MSG_SERVICE_ACCEPT received, then encryption negotiation was successful. If not, then the server or client must be reconfigured. An outdated host key on the client could also cause this (use ssh-keygen -R <hostname> on the client to remove old host key if necessary; see ssh-keygen).

• Look at the authentication methods the server is willing to accept. Look for lines containing Authentications that can continue: <list of methods>. If the method you are trying to use is not included, you need to change the configuration of the server and restart the server. This is a fairly common cause of problems if using anything other than password or public key authentication.

• If you see a line containing Authentication succeeded, then it is not an authentication problem. If login fails after this, then it could be a problem with the user's login shell or, e.g., .bashrc.

• It is fairly common for X11 forwarding to fail. It is disabled by default in the OpenSSH server. You need to edit the sshd_config file on the server to have the line X11Forwarding yes to enable it. It often does not to be enabled on enterprise application servers, but in universities, home environments, and development servers it is usually needed. Again, remember to restart the server.

Log Files

Looking at the log files can often reveal insights into the cause of the problem. The messages sent to the client are intentionally designed to reveal quite little about the user being logged in as. This is for security reasons. For example, we don't want attacker to be able to test which user accounts exist on a target system. More information about, e.g., authentication failures can often be found in the log file.

Run the Server in Debug Mode

A system administrator can manually run the server with the -d option to get extra verbose output from the server. This is often the last resort when diagnosing connection problems. Usually the cause of authentication failures is quite clearly visible in its output.

It may be desirable to run the new server in a different port then the normal server, so as to not prevent new connections to the server (especially if it remote!). In this case, the server would be run (as root) with something like sshd -d -p 2222 and then the client would connect with ssh -p 2222 [user@]host.

Command-Line Options

It is rare to have to manually provide command options for the SSH server. Generally only people repackaging SSH or creating new linux distributions or new embedded platforms (e.g., IoT devices) would do this.

The following options are available in OpenSSH:

-4 Only use IPv4 addresses. This might be used in environments where DNS gives IPv6 addresses but routing does not work for them.

-6 Only use IPv6 addresses. This might be used for testing to make sure IPv6 connectivity works.

-C connection_spec Used for testing particular Match blocks in the configuration file, in combination with the -T option. The connection_spec is a comma-separated list of <keyword>=<value> pairs, where <keyword> can be one of: user, host, laddr, lport, addr. Multiple -C options are permitted and combined.

-c host_certificate_file Specifies the path of a file containing the host certificate for the host. The certificate is in OpenSSH's proprietary format.

-D Do not detach and become daemon. This is often used when sshd is run using systemd. This allows easier monitoring of the process in such environments. Without this option, the SSH server forks and detaches from terminal, making itself a background daemon process. The latter has been the traditional way to run the SSH server until recently. Many embedded systems would still use the latter.

-d Enables debug mode. The server does not fork, and will exit after processing a single connection. This can be used for diagnosing user authentication and other problems, and usually gives more information about the problem than is set to the client.

-E log_file Appends logs to the specified file, instead of sending them to syslog.

-e Write debug logs to standard error. This could be used for debugging.

-f config_file Specifies the path of the server configuration file. By default, /etc/ssh/sshd_config is used.

-g login_grace Specifies how quickly users must authenticate themselves after opening a connection to the SSH server. The default is 120 seconds, but this can be changed in the server configuration file. The timeout prevents permanently reserving resources on the server by opening an unauthenticated connection to it.

-h host_key_file Specifies a file from which to read a host key. The default is to use /etc/ssh/ssh_host_<algorithm>_key files. Only one host key can be specified for each algorithm.

-i This would be used if the server was run through inetd. However, nobody does it these days.

-k timeout This option is obsolete. It was used with SSH version 1. Its use is strongly discouraged.

-o option Overrides any configuration option specified in the configuration file. This could be useful for testing and running multiple servers on different ports.

-p port Specifies the port that the server listens on. The default is 22. The port can also be specified in the server configuration file.

-q Doesn't send anything to the system log. This is not recommended; the only real use of this option would be for an attacker to hide logins using a backdoor. This option really shouldn't be there.

-T Reads the server configuration file, checks its syntax, and exits. This is useful for checking that the configuration file is ok before restarting the server. Checking the configuration file is especially important if updating the configuration remotely. In fact, in such cases it is best to first test the new configuration by running a second server on a new port, and only restart the primary server after a successful login using the test server. This can be combined with the -C option to test individual Match blocks in the configuration file.

-t Checks the validity of the configuration file and referenced keys. See -T for advise on additional testing before restarting a server remotely.

-u len This obscure option has only one useful purpose: specifying -u0 causes dotted IP addresses to be stored in the utmp file (which contains information about logins to the server). This disables DNS lookups by the SSH server, if the authentication mechanism or from= patterns on authorized keys do not require then. Otherwise it would specify the size of an utmp structure on the host, the cases where it needs to be manually specified are very rare.

Getting Version Number of the Installed Server

When connecting to an SSH server, it reports a version number to the client. OpenSSH does not provide a documented means for getting the version number of the server.

If the server is given the unrecognized -v option, it however reports its version number on the second line of output. This can be very helpful in determining the version number from a shell script.