How to Use SSH Config File in Linux: Manage Servers Like a Pro

If you’re still typing ssh -i ~/.ssh/id_prod -p 2222 [email protected] every time you connect to a server, the Linux SSH config file is about to change your life. I’m not exaggerating. The file lives at ~/.ssh/config, and once you’ve set it up, that mile-long command becomes ssh prod. That’s it. That’s the whole pitch.

I’ve been managing a six-node homelab for years, plus a handful of cloud droplets, plus client servers. Without ~/.ssh/config, my terminal history would look like a ransom note made from IP addresses. With it, I have aliases, automatic key selection, jump host routing, and connection multiplexing — all from one tidy text file that OpenSSH reads every time I run ssh, scp, rsync, or Ansible.

By the end of this guide, you’ll know how to write Host blocks, use ProxyJump to reach servers behind a bastion, and speed up Ansible runs with ControlMaster. There’s also one debugging command that 90% of tutorials skip — I’ll cover it in the troubleshooting section, and it’ll save you hours.

Quick answer: The Linux SSH config file is ~/.ssh/config. Create it, set permissions to chmod 600, and add Host blocks like this: Host myserver / HostName 1.2.3.4 / User alexa / IdentityFile ~/.ssh/id_ed25519. Then connect with ssh myserver.

Get a VPS from as low as $11/year! WOW!

What Is the SSH Config File (And Why You’re Missing Out)

The SSH config file is a plain text file that tells your SSH client how to connect to remote hosts. Instead of memorizing IPs, ports, and key paths, you define them once under a friendly alias and forget the details forever.

It’s read every time you run any tool built on OpenSSH. That includes scp command, rsync, sftp, and Ansible. They all inherit your aliases automatically. No extra setup, no plugin gymnastics.

User Config vs System-Wide Config

There are two SSH client config files on a Linux system:

• ~/.ssh/config — your personal user config. This is what you’ll edit 99% of the time.
• /etc/ssh/ssh_config — the system-wide client config that applies to every user on the machine.

Important: don’t confuse /etc/ssh/ssh_config (client) with /etc/ssh/sshd_config (server daemon). They’re different files. If you’re tightening up the server side, that’s a separate job — see my guide on secure SSH server configuration for that.

When Does SSH Read the Config File?

OpenSSH walks through configuration sources in this priority order:

1. Command-line flags (highest priority)
2. ~/.ssh/config (your personal config)
3. /etc/ssh/ssh_config (system-wide config)

The first match for any given directive wins. So if you set Port 2222 in your user config and pass -p 22 on the command line, the flag wins. Knowing this order saves serious head-scratching later.

I learned this the hard way back when I was building out my homelab — Proxmox, Pi-hole, Gitea, Nextcloud, all on different VLANs, all with different ports. I tried managing it with shell aliases. It was awful. The day ~/.ssh/config finally clicked, my mental overhead got cut in half.

Creating the SSH Config File

The file doesn’t exist by default on most distros. Create it and lock down the permissions in two commands:

touch ~/.ssh/config
chmod 600 ~/.ssh/config

The ~/.ssh/ directory itself should be chmod 700. If you’ve never set up SSH before, you’ll also want to generate an SSH key first — a config file pointing at keys you don’t have isn’t going to do much.

Creating the File and Setting Permissions

Permissions matter more than you’d think. SSH is paranoid by design. If your config file is world-readable, SSH may silently ignore it or refuse to load certain directives. I’ve debugged this on a fresh VM more times than I’d like to admit.

SSH Config File Syntax: Host Blocks and Directives

The syntax is dead simple. Each Host keyword starts a new block. Everything indented below applies to that host pattern. Order matters — the first matching block wins for each directive.

Host Blocks — The Core Building Block

Here’s the smallest useful Host block:

Host prod
    HostName 203.0.113.42
    User deploy
    Port 2222
    IdentityFile ~/.ssh/id_prod

Now ssh prod connects you as [email protected] on port 2222 using the id_prod key. That single block replaces a 50-character command.

Global Defaults with Host *

The Host * pattern matches every host. It’s perfect for sane defaults:

Host *
    ServerAliveInterval 120
    ServerAliveCountMax 3
    AddKeysToAgent yes
    HashKnownHosts yes

Critical placement rule: put Host * at the bottom. SSH uses first-match-wins for most directives, so any specific host blocks must appear above the wildcard block.

Most-Used Directives Reference

For the full directive list — and there are dozens more — the official ssh_config man page is the canonical reference. I keep it bookmarked.

Host patterns also support wildcards. Host dev-* matches dev-web, dev-db, and dev-cache. Useful for applying common settings to a group of servers without copy-pasting.

Practical SSH Config Examples for Real-World Use

Theory is fine. Real configs are better. Here’s a setup pulled almost verbatim from my actual homelab.

Single Server Alias

The classic before-and-after:

# Before
ssh -i ~/.ssh/id_prod -p 2222 [email protected]

# After ~/.ssh/config
Host prod
    HostName 203.0.113.42
    User deploy
    Port 2222
    IdentityFile ~/.ssh/id_prod

# Now
ssh prod

Managing a Multi-Server Homelab

This is the config that runs my homelab. Names changed, IPs sanitized, but the structure is real:

Host proxmox
    HostName 10.0.0.10
    User root
    IdentityFile ~/.ssh/id_homelab

Host pihole
    HostName 10.0.0.20
    User alexa
    IdentityFile ~/.ssh/id_homelab

Host gitea
    HostName 10.0.0.30
    User git
    Port 2222
    IdentityFile ~/.ssh/id_homelab

Host nextcloud
    HostName 10.0.0.40
    User alexa
    IdentityFile ~/.ssh/id_homelab

The first time this clicked for me was setting up five DigitalOcean droplets in one weekend. I’d been typing full IPs all morning, got fed up, sat down with a fresh cup of coffee, and rewrote my workflow with aliases. By the end of the day I was connecting to anything by name. Felt like the system finally trusted me to be lazy in the right way.

This same config makes managing remote storage trivial — even spinning up a quick NFS server on a homelab node becomes ssh proxmox instead of digging up an IP.

Different Keys for Different Hosts

Don’t use the same key everywhere. I keep work, personal, and homelab keys separated:

Host work-*
    IdentityFile ~/.ssh/id_work
    IdentitiesOnly yes

Host homelab-*
    IdentityFile ~/.ssh/id_homelab
    IdentitiesOnly yes

Host github.com
    IdentityFile ~/.ssh/id_personal
    IdentitiesOnly yes

IdentitiesOnly yes is the unsung hero here. Without it, SSH offers every key in your agent to every server. With it, only the key you named gets sent. Cleaner logs, less risk of accidentally identifying yourself across contexts.

Preventing Dropped Connections with ServerAliveInterval

If your SSH sessions die after a few minutes of idle time, drop this in a Host * block:

Host *
    ServerAliveInterval 120
    ServerAliveCountMax 3

Your client now sends a tiny keepalive every 120 seconds. If three in a row fail, it disconnects gracefully instead of hanging forever. NAT routers and corporate firewalls love to kill idle connections — this fixes it.

Advanced Techniques: ProxyJump and ControlMaster

This is where the SSH config file goes from convenient to indispensable.

ProxyJump for Bastion Hosts and Private Networks

ProxyJump lets you reach a private server through a public bastion host with one command. It was introduced in OpenSSH 7.3 (back in 2016) and replaces the older, clunkier ProxyCommand.

“ProxyJump is the easiest and recommended way to jump between hosts because it ensures that traffic passing through the intermediate hosts is always encrypted end-to-end.” — Teleport Engineering Team

Here’s the setup. Define your bastion first, then reference it on private hosts:

Host bastion
    HostName bastion.example.com
    User alexa
    IdentityFile ~/.ssh/id_ed25519

Host db-internal
    HostName 192.168.10.50
    User alexa
    ProxyJump bastion
    IdentityFile ~/.ssh/id_ed25519

Now ssh db-internal automatically routes through bastion. End-to-end encryption is preserved — the bastion sees encrypted traffic only.

Need to chain through two jump hosts? Use a comma:

ProxyJump bastion1,bastion2

For deeper bastion patterns — multi-hop, dynamic targets, the works — the OpenSSH jump host cookbook is excellent. And if you also need port forwarding tunnels through that bastion, my SSH tunneling guide picks up where this one leaves off.

ControlMaster: Connection Multiplexing for Speed

This one quietly changed how I do everything. ControlMaster lets new SSH sessions reuse an existing connection to the same host. The first connection authenticates normally; every subsequent one reuses the open socket. The result is huge speedups for tools that open many SSH connections — Ansible playbooks, recursive rsync, fast scp transfers.

The setup needs a sockets directory:

mkdir -p ~/.ssh/sockets
chmod 700 ~/.ssh/sockets

Then in your config:

Host *
    ControlMaster auto
    ControlPath ~/.ssh/sockets/%r@%h-%p
    ControlPersist 10m

ControlPersist 10m keeps the master connection open for 10 minutes after the last session closes. The next ssh within that window connects instantly.

SSH Config File Security Tips

Convenience without security is just a footgun. A few rules I live by:

• chmod 600 always. Re-check after copying configs between machines. SSH ignores configs that are too permissive.
• Use Ed25519 keys. They’re compact, fast, and resistant to side-channel attacks. RSA still works but Ed25519 is the modern default.
• Don’t put ForwardAgent yes in Host *. Only enable agent forwarding on hosts you fully trust. A compromised host can use your forwarded agent to hop to your other servers.
• Never store passphrases in the config. Use ssh-agent with AddKeysToAgent yes instead.
• Set IdentitiesOnly yes when you specify an IdentityFile. Stops SSH from offering every key in your agent.

Client config is only half the story. Pair it with server-side hardening — strong sshd_config, key-only auth, and fail2ban to block brute force attacks. Defense in depth.

Troubleshooting SSH Config Issues

When things break — and they will — these are the commands I reach for first.

ssh -v for verbose output

Run ssh -v hostname to see which config file and Host block SSH is applying. Add more vs for more detail: -vv, then -vvv for the full debugging firehose.

ssh -G for the resolved config

Here’s the command most tutorials skip — and it’s the single most useful debugging tool I know:

ssh -G prod

ssh -G hostname prints the final, fully-resolved config that SSH would apply for that host. Every directive, every wildcard match, all merged. Whenever a connection acts weird, run ssh -G first. Half the time the answer jumps right out.

Common mistakes that bite everyone

• Permissions too open. Not 600 = SSH ignores the file. Run ls -l ~/.ssh/config to verify.
• Host * placed first. Specific hosts must come BEFORE the wildcard block. First match wins.
• Hostname vs HostName. Some implementations are case-sensitive. Stick to HostName as documented.
• Wrong key offered. Add IdentitiesOnly yes to your Host block.

If your connections feel sluggish even after fixing the config, that’s a separate rabbit hole — I’ve got a full breakdown in troubleshooting slow SSH connections.

Wrapping Up

The SSH config file is one of those small Linux features that quietly transforms how you work. Five minutes of setup, and you’re managing dozens of servers like they’re a single fleet. Aliases, key routing, jump hosts, multiplexing — all from a plain text file with permissions locked down at 600.

Start small. Add one Host block for the server you connect to most often. Then another. Within a week, you’ll wonder how you lived without it. BTW, I use Arch.

If you found this useful, you might also enjoy my walkthroughs on SSH tunneling for port forwarding, Ansible on Linux to automate config across all those new aliases, and the secure SSH server configuration guide to lock down the other side of the connection. Happy SSH-ing.

Alexa Velinxs

I'm Alexa Velinxs, a cryptocurrency trading expert passionate about demystifying digital assets for both beginners and seasoned investors. Through my writing, I share actionable strategies, market insights, and practical tips to help you navigate the crypto landscape with confidence. Let's explore the future of finance together.

See Full Bio