It sounds like it’s the kernel but whether it has anything to do with ssh, I really don’t know. Sometimes parts work together in surprising ways, as I learned with the recent sshd/systemd/xz exploit.

You might be fine and this was the most alarming exploit since it’s very inconvenient, but personally I’d restart just to be sure.