Uff, that's rough, hopefully no bad follow-on effects!
I normally don't post *this* often, but I found that my main work machine has been hacked since two weeks now.
I had a remote login user that I'd give out to others. It was a restricted shell "bash -r", so it couldn't traverse directories, but it definitely could upload/download resources to the internet and scan some directories.
I'm not too sure. I saw high usage of "kswapd" with all cores at 100% usage, and the user had copied over a ".configrc4" directory which contained a local rsync and tor binary. Tor was running quite hot, and rsync was transmitting a lot too.
No idea what exactly. I pulled the ethernet, purged the account and killed anything I didn't recognize. I wish I acted a bit more smart and actually checked what was happening. I think they were crypto mining? Not sure what would require 400% cpu and large rsync activity.
So once I'd calmed down, I took a look through the logs.
`journalctl -u sshd --since <rough date> --until <rough date>` revealed that A TON of people had tried accessing my machine. My first mistake was leaving my ssh port on 22, and not installing a fail2ban daemon.
My second mistake was using a super weak, but very memorable password for this remote login. It was super easy to crack, and since I wasn't banning IPs, they could try as many times as they needed.
My third mistake was letting anyone other than myself access the machine via ssh. I should never have given out my remote login, and it was stupid not to limit the "AllowUsers" field in my ssh config.
Lesson hopefully learned. Wiping the machine. Changing all my passwords and revoking auth keys on related machines. Reinstalling with LUKS enabled by default on the home partition. Installing ClamAV. Crying.
That's really rough. Don't blame yourself too much, or take it too hard, just learn the lesson and move on.
One other recommendation, beyond running Fail2Ban, is to disable password authentication and only use public key. It makes getting set up on new machines a little harder, but it rules out brute-force attempts on ssh.
Naive question: how do you get the public key on the machine, without ssh-copy-id?
Yeah, that's the tricky part. You either copy the public key over first /then/ disable password authentication, or you use some other means of copying it... Nextcloud, Syncthing, USB, floppy disk...