In my free time, I help run a small Mastodon server for roughly six hundred queer leatherfolk. When a new member signs up, we require them to write a short application—just a sentence or two. There’s a small text box in the signup form which says:
Please tell us a bit about yourself and your connection to queer leather/kink/BDSM. What kind of play or gear gets you going?
This serves a few purposes. First, it maintains community focus. Before this question, we were flooded with signups from straight, vanilla people who wandered in to the bar (so to speak), and that made things a little awkward. Second, the application establishes a baseline for people willing and able to read text. This helps in getting people to follow server policy and talk to moderators when needed. Finally, it is remarkably effective at keeping out spammers. In almost six years of operation, we’ve had only a handful of spam accounts.
I was talking about this with Erin Kissane last year, as she and Darius Kazemi conducted research for their report on Fediverse governance. We shared a fear that Large Language Models (LLMs) would lower the cost of sophisticated, automated spam and harassment campaigns against small servers like ours in ways we simply couldn’t defend against.
Besides finding better ways to positively recognize bots, we also need to invent ways that make it quicker to realize “false alarm, this user is actually legit”.
For example, users should have an option to pin posts and comments to their profile, and I suggest to provide at least two different ‘tabs’ for this in the public profile: One tab just for the usual “posts and comments you would like the world to see”, but another tab for “some recent, complex interactions between you and other (established) users that in your eyes prove quite well you’re not a bot”. The purpose is simply to save others, worried that you could be a bot, some time of going through your posts in search of signs for humanity. Yes, this can be gamed to some degree (what can’t?). However, at a technical level, the feature is little more than a copy of the “pin” feature that would be very nice to have anyways, so we can get an appreciable improvement in our ability to tell users from bots for very little programming effort.
The future is probably going to be:
- Whitelist federation models with invite-only instances.
- Anubis and similar software being a base requirement for operation.
- Getting creative with stuff like picture-based logic puzzles as a type of captcha.
- A retreat into less publicly visible spaces like chat apps that you can only get access to through networking.
Eh frankly I just see us moving to more strict reputation based systems - someone has to vouch for you.
I mean… I’ve been using forums since '93 and lies were its past, too.
At least those liars were human beings that actually showed some creativity with their made up bullshit.
These attacks do not have to be reliable to be successful. They only need to work often enough to be cost-effective, and the cost of LLM text generation is cheap and falling. Their sophistication will rise. Link-spam will be augmented by personal posts, images, video, and more subtle, influencer-style recommendations—“Oh my god, you guys, this new electro plug is incredible.” Networks of bots will positively interact with one another, throwing up chaff for moderators. I would not at all be surprised for LLM spambots to contest moderation decisions via email.
I don’t know how to run a community forum in this future. I do not have the time or emotional energy to screen out regular attacks by Large Language Models, with the knowledge that making the wrong decision costs a real human being their connection to a niche community.
Ouch. I’d never want to tell someone ‘Denied. I think you’re a bot.’ – but I really hate the number of bots already out there. I was fine with the occasional bots that would provide a wiki-link and even the ones who would reply to movie quotes with their own quotes. Those were obvious and you could easily opt to ignore/hide their accounts. As the article states, the particular bot here was also easy to spot once they got in the door, but the initial contact could easily have been human and we can expect bots to continuously seem human as AI improves.
Bots are already driving policy decisions in government by promoting/demoting particular posts and writing their own comments that can redirect conversations. They make it look like there is broad consensus for the views they’re paid to promote, and at least some people will take that as a sign that the view is a valid option (ad populum).
Sometimes it feels like the internet is a crowd of bots all shouting at one another and stifling the humans trying to get a word in. The tricky part is that I WANT actual unpaid humans to tell me what they actually: like/hate/do/avoid. I WANT to hear actual stories from real humans. I don’t want to find out the ‘Am I the A-hole?’ story getting everyone so worked up was an ‘AI-hole’ experiment in manipulating emotions.
I wish I could offer some means to successfully determine human vs. generated content, but the only solutions I’ve come up with require revealing real-world identities to sites, and that feels as awful as having bots. Otherwise, I imagine that identifying bots will be an ever escalating war akin to Search Engine Optimization wars.
Definitely something I’ve observed even here. Luckily we get few applications and there is a report button, but I share the author’s frustration and the author’s jaded view of a limited timeline on services such as ours being tenable. Eventually it will be trivially easy to flood this place with slop.
Question: what would happen if the server implemented something like Anubis on the application and/or create post pages? Would that not block most bots from completing these forms? Is that just not feasible at our scale?
I think Anubis is really focused on scraper-bots feeding AI models, rather than posting bots. It’s based on requests to non-standard endpoints in your own app, which you specify for Anubis in a couple places (e.g. leaving out of /robots.txt or /.well-known).
If you’re using e.g. a python bot that uses headless chromium executing JS to post stuff, you’re probably going to code in known-good endpoints for comments and posts, rather than hitting random ones like a scraper bot would.
Anubis is good for stopping the n-request-per-second spamming of scrapers, but not so much for just blocking non-human bots that post at normal rates.
My last employer was a Fortune 50, and we did automation detection through behavioral mapping, like posting locations, times, and even word patterns (a very cool experimental project that I got to work on, which used a database of normalized English word frequency to detect bots based on language that was too-similar across users, or even too “perfect”, though this was only used as an indicator and never considered definitive). It is extremely difficult to detect human-impersonating bots based on raw network traffic alone.
believe that i’ve seen versions of this attack where the spammer makes a few innocuous non-spam (but not contribution either) posts before going full LLM bot
A more critical weakness is that these accounts only posted obvious spam; they made no effort to build up a plausible persona. Generating plausible human posts is more difficult, but broadly feasible with current LLM technology.
i will have to screenshot next time and see what people think
obviously they could be hijacked accounts and i might just be extremely judgey of your average twitter flight account
I was going to say “but someone could post a picture with the date and their username” and then realized that can be spoofed. And then I thought what about some kind of video? Wait that could probably be spoofed too.
Maybe you could have it so only real people could get in via word of mouth? But eventually I’m sure someone with bad intentions could enter
I wonder if a personalized reputation system based on your votes of other people’s comments, and influenced by votes from folks who have earned enough upvotes from you, could be developed without turning your feed into an echo chamber like Facebook.
Sort of like PageRank, but for fediverse users instead of web pages, and with each user keeping (and seeing) their own rankings of everyone else.
