Complete Guide to Setting Up a Secure SSH Configuration in Linux (2025)

As a Linux system administrator or enthusiast, securing your remote connections should be at the top of your priority list. SSH (Secure Shell) is the backbone of remote administration for Linux systems, but an improperly configured SSH server can leave your system vulnerable to brute force attacks, unauthorized access, and other security threats.

In this comprehensive guide, I’ll walk you through the process of setting up a rock-solid SSH configuration that balances security with usability. I recently had to secure several Linux servers for a client, and I’m sharing all the lessons learned and best practices I discovered along the way.

Whether you’re managing a single home server or an enterprise fleet of Linux machines, these SSH hardening techniques will help you sleep better at night knowing your systems are protected against common attack vectors.

1. Introduction to SSH and Its Importance

SSH, or Secure Shell, is a cryptographic network protocol that allows secure system administration and file transfers over unsecured networks. Unlike its predecessors (telnet, rlogin), SSH encrypts all traffic between client and server, protecting your data and credentials from eavesdropping.

Here’s why SSH is critical for Linux systems:

• It provides encrypted communications between hosts over an insecure network
• It enables secure remote command execution and system administration
• It offers secure file transfer capabilities through SCP and SFTP
• It supports tunneling, port forwarding, and X11 connections
• It allows for automated, passwordless connections between systems

The default SSH configuration that comes with most Linux distributions is functional but often prioritizes convenience over security. As with many aspects of system administration, the default settings aren’t always the most secure.

2. Understanding SSH Security Risks

Before diving into configuration changes, it’s important to understand the security risks associated with SSH:

Brute Force Attacks

The most common attack against SSH servers is the brute force attack, where attackers attempt to guess your password by trying numerous combinations. If you check your auth.log file, you’ll likely see evidence of these attempts:

$ sudo grep "Failed password" /var/log/auth.log | head
Mar 5 14:23:15 ubuntu-server sshd[12345]: Failed password for invalid user admin from 192.168.1.100 port 39654 ssh2
Mar 5 14:23:17 ubuntu-server sshd[12346]: Failed password for invalid user administrator from 192.168.1.100 port 39700 ssh2
Mar 5 14:23:20 ubuntu-server sshd[12347]: Failed password for invalid user root from 192.168.1.100 port 39856 ssh2

Man-in-the-Middle Attacks

These attacks occur when someone positions themselves between your client and the server, intercepting and potentially altering communications. Proper key verification helps prevent this.

Unauthorized Access

If an attacker gains access to your private key (or cracks a weak password), they can access your system with the same privileges as you.

Vulnerable SSH Versions

Older versions of SSH may contain security vulnerabilities that have been patched in newer releases.

3. Setting Up SSH Key-Based Authentication

Password authentication is convenient but vulnerable to brute force attacks. SSH key-based authentication is significantly more secure and should be your default method for SSH access.

Generating SSH Keys

On your local machine (not the server), generate a new SSH key pair:

$ ssh-keygen -t ed25519 -C "your_email@example.com"

I recommend using Ed25519 keys as they provide strong security with shorter key lengths. If you need compatibility with older systems, use RSA with at least 3072 bits:

$ ssh-keygen -t rsa -b 4096 -C "your_email@example.com"

When prompted, provide a secure passphrase. This adds an extra layer of protection if your private key is ever compromised.

Transferring Your Public Key to the Server

Use the ssh-copy-id command to copy your public key to the server:

$ ssh-copy-id username@server_ip

Alternatively, you can manually copy the key:

$ cat ~/.ssh/id_ed25519.pub | ssh username@server_ip "mkdir -p ~/.ssh && chmod 700 ~/.ssh && cat >> ~/.ssh/authorized_keys && chmod 600 ~/.ssh/authorized_keys"

Testing Key-Based Authentication

Try logging in with your key:

$ ssh username@server_ip

If you set a passphrase, you’ll be prompted to enter it. Consider using ssh-agent to avoid typing your passphrase repeatedly:

$ eval "$(ssh-agent -s)"
$ ssh-add ~/.ssh/id_ed25519

4. Configuring the SSH Server for Enhanced Security

Now that key-based authentication is set up, it’s time to harden the SSH server configuration. All these changes should be made in the /etc/ssh/sshd_config file on your server.

Disable Password Authentication

Once you’ve confirmed that key-based authentication works, disable password authentication:

# /etc/ssh/sshd_config
PasswordAuthentication no
ChallengeResponseAuthentication no
UsePAM no

Disable Root Login

Prevent direct root login via SSH:

# /etc/ssh/sshd_config
PermitRootLogin no

Change the Default Port

While security through obscurity isn’t a complete solution, changing the default SSH port (22) can reduce automated attacks:

# /etc/ssh/sshd_config
Port 2222

Choose a port number above 1024 that isn’t used by another service.

Limit User Access

Specify which users are allowed to connect via SSH:

# /etc/ssh/sshd_config
AllowUsers username1 username2

Alternatively, you can use AllowGroups to permit access to members of specific groups.

Use Strong Encryption Algorithms

Configure SSH to use only strong encryption algorithms:

# /etc/ssh/sshd_config
KexAlgorithms curve25519-sha256@libssh.org,diffie-hellman-group-exchange-sha256
Ciphers chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com,aes256-ctr,aes192-ctr,aes128-ctr
MACs hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,umac-128-etm@openssh.com

Implement Idle Timeout

Automatically disconnect inactive sessions:

# /etc/ssh/sshd_config
ClientAliveInterval 300
ClientAliveCountMax 2

This configuration will disconnect clients after 10 minutes of inactivity (300 seconds × 2 intervals).

Apply the Changes

After making these changes, restart the SSH service:

$ sudo systemctl restart sshd

5. Using SSH Config Files for Easier Connections

SSH client configuration files can make your life easier by storing connection details for servers you frequently access. This is especially helpful when using non-standard ports or multiple identities.

Creating a Client Config File

On your local machine, create or edit ~/.ssh/config:

# ~/.ssh/config
Host myserver
    HostName server_ip_or_domain
    Port 2222
    User username
    IdentityFile ~/.ssh/id_ed25519
    IdentitiesOnly yes
    
Host staging
    HostName staging.example.com
    Port 2222
    User deploy
    IdentityFile ~/.ssh/staging_key
    
Host *
    ServerAliveInterval 60
    HashKnownHosts yes
    AddKeysToAgent yes

With this configuration, you can simply type ssh myserver instead of ssh -p 2222 -i ~/.ssh/id_ed25519 username@server_ip.

Understanding Config Options

• Host: A nickname for your SSH connection
• HostName: The actual hostname or IP address
• Port: The SSH port number
• User: Your username on the remote server
• IdentityFile: Path to your private key
• IdentitiesOnly: Only use the specified key for this connection
• ServerAliveInterval: Seconds between keep-alive packets
• HashKnownHosts: Hash hostnames and addresses in known_hosts file

6. Advanced SSH Hardening Techniques

For those seeking even stronger security, here are some advanced hardening techniques:

Implement Two-Factor Authentication (2FA)

Add an extra layer of security with Google Authenticator or similar 2FA tools:

$ sudo apt install libpam-google-authenticator
$ google-authenticator

Follow the prompts to set up 2FA, then modify your PAM configuration:

# /etc/pam.d/sshd
auth required pam_google_authenticator.so

And update your SSH config:

# /etc/ssh/sshd_config
ChallengeResponseAuthentication yes
AuthenticationMethods publickey,keyboard-interactive

Use Fail2ban to Protect Against Brute Force Attacks

Fail2ban temporarily bans IP addresses that show malicious signs:

$ sudo apt install fail2ban
$ sudo cp /etc/fail2ban/jail.conf /etc/fail2ban/jail.local

Edit jail.local to configure the SSH jail:

# /etc/fail2ban/jail.local
[sshd]
enabled = true
port = 2222
filter = sshd
logpath = /var/log/auth.log
maxretry = 3
bantime = 3600

Restart fail2ban to apply the changes:

$ sudo systemctl restart fail2ban

Set Up SSH Login Notifications

Get notified whenever someone logs in via SSH by adding this to /etc/pam.d/sshd:

session optional pam_mail.so dir=/var/mail standard

Use Port Knocking

Port knocking adds an extra layer of obscurity by requiring a specific sequence of connection attempts before the SSH port opens:

$ sudo apt install knockd

Configure knockd in /etc/knockd.conf:

[options]
    UseSyslog
[openSSH]
    sequence = 7000,8000,9000
    seq_timeout = 10
    command = /sbin/iptables -A INPUT -s %IP% -p tcp --dport 2222 -j ACCEPT
    tcpflags = syn
[closeSSH]
    sequence = 9000,8000,7000
    seq_timeout = 10
    command = /sbin/iptables -D INPUT -s %IP% -p tcp --dport 2222 -j ACCEPT
    tcpflags = syn

7. Troubleshooting Common SSH Issues

Even with careful configuration, you might encounter SSH issues. Here are solutions to common problems:

Connection Refused

If you get “Connection refused,” check:

• Is the SSH service running? sudo systemctl status sshd
• Is the port correct? sudo netstat -tuln | grep ssh
• Is a firewall blocking the connection? sudo iptables -L

Permission Denied

For “Permission denied” errors:

• Check that your public key is in ~/.ssh/authorized_keys on the server
• Verify permissions: ~/.ssh should be 700, ~/.ssh/authorized_keys should be 600
• Check the server logs: sudo tail -f /var/log/auth.log

Host Key Verification Failed

If you get “Host key verification failed”:

$ ssh-keygen -R server_ip_or_hostname

Then reconnect and verify the new host key.

Slow SSH Connection

For slow connections, try:

# /etc/ssh/sshd_config
UseDNS no
GSSAPIAuthentication no

8. Best Practices for SSH in Production Environments

When managing SSH in production environments, consider these best practices:

Implement a Bastion Host

Use a dedicated bastion host (jump server) as the single entry point to your infrastructure:

# ~/.ssh/config
Host bastion
    HostName bastion.example.com
    User jumpuser
    IdentityFile ~/.ssh/jump_key
Host internal-server
    HostName 10.0.0.5
    User admin
    IdentityFile ~/.ssh/internal_key
    ProxyJump bastion

Use SSH Certificates

For large environments, consider using SSH certificates instead of managing authorized_keys files:

$ ssh-keygen -t rsa -f ca_key
$ ssh-keygen -s ca_key -I user_id -n username -V +52w id_rsa.pub

Then configure the server to trust the CA:

# /etc/ssh/sshd_config
TrustedUserCAKeys /etc/ssh/ca.pub

Rotate SSH Keys Regularly

Implement a process for regularly rotating SSH keys, especially when team members leave.

Audit SSH Access

Monitor and audit SSH access with tools like auditd:

$ sudo apt install auditd
$ sudo auditctl -w /etc/ssh/sshd_config -p wa -k sshd_config

Use Configuration Management

Manage SSH configurations across multiple servers with tools like Ansible, Puppet, or Chef.

9. Conclusion and Next Steps

Securing SSH is a critical step in hardening your Linux systems. By implementing key-based authentication, disabling password login, changing the default port, and applying other security measures discussed in this guide, you’ve significantly improved your server’s security posture.

Remember that security is not a one-time task but an ongoing process. Stay informed about new vulnerabilities and best practices, and regularly audit your SSH configuration.

Next Steps

To further enhance your Linux security knowledge, consider exploring:

• Complete Guide to Disk Partitioning in Linux for Maximum Performance
• Setting up a comprehensive firewall with iptables or ufw
• Implementing intrusion detection systems like AIDE or Tripwire
• Learning about Linux security modules like SELinux or AppArmor

What SSH hardening techniques have you implemented on your systems? Have you encountered any challenges with the methods described in this guide? Share your experiences in the comments below!

Until next time, stay secure!

Alice