mirror of
https://github.com/blader/humanizer.git
synced 2026-09-27 18:19:51 +00:00
Blind study: the rewrite pass wins 16/16 on quality, and does not move AI detection #229
Labels
No labels
bug
documentation
duplicate
enhancement
good first issue
help wanted
invalid
question
wontfix
No milestone
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
skills/blader-humanizer#229
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Hello Mr. Chen,
First of all I want to thank you for your work on this plugin. Without it, I would not be writing this message to you now, nor would I have found some of the passions I am deeply obsessed with currently. I currently am building my own solo business in the realm of SEO, AEO and GEO which was born from a conversation I had with my mother about her brick and mortar business and her Shopify storefront. At some point during the journey I stumbled upon your humanizer plugin.
The culmination of that moment is what I'd like to convey to you here in this open issue. I ran a blind study with your tool in conjunction with my own version in great detail. If you have time, I'd love for you to dig into my project in detail. It was initially meant at some level to mask AI assisted prose, but eventually what was born from it is far more important I believe. I'll be including the details below. Again, thank you. I want you to know that this stranger out in the world wide web was truly inspired due to your work. The thing I found is that your pass is the only part of the whole study that had tangible evidence of a clear-cut win.
What worked
Blind pairwise, quality only, authorship never mentioned to the judges:
The third row is what I believe is something worth your time. My checker adds nothing measurable on top of your
pass. It emitted its input byte-identical in 11 of 18 cells, so twelve of those twenty-one
trials were judges comparing a file to itself.
Your rewrite pass carries the entire quantitative benefit. It is the sole component in the entire
study with an unambiguous positive result. I built the others on top of it.
What didn't work, which I believe matters for your README
I originally vendored this to try to beat AI detection. It does not, and I tested it three ways:
that could not resolve anything under about four points. Genuine human prose was called human
143 times out of 143.
try harder fooled 0 of 15.
Then I took it to a commercial detector. Pangram 4.0 returned 100% AI on four variants,
including one where I changed zero words and only moved where sentences joined.
The finding related to your pattern list
Judge-cited tells, before and after your pass:
The pass crushes the lexical patterns on your list and grows the rhythmic ones. Ten named
tells in my data have no rule at all, and every one of them is rhythmic rather than lexical. My
read is that a subtractive rewrite makes prose more uniformly well-shaped, and uniform good shape
is itself a tell.
The repo description says the tool "removes signs of AI-generated writing." With 3,240 forks, I
suspect that some people may read that as defeats AI detection. My data says it does not, and that the
thing it actually (perhaps subjectively) does is makes the draft better to read.
Reproduce any of it yourself
Corpora, judgment files, tally scripts and the two numbers I had to correct mid-study:
https://github.com/itsryanlenk/humanist
Caveats: every judge is a language model from one family, which is the friendliest possible version of the detection test, in the future I'd like to run it more stringently. The quality study ran 77 of 144 planned pairwise trials after infrastructure failures. The human control is matched on register and length but not on subject.
I look forward to your input, positive or negative. I'm @itsryanlenk everywhere if it's easier.
Hello - thanks for your analysis!
It is not currently possible to defeat Pangram through pure LLM generation through any skill or prompt (I have tried!)
Regardless, defeating AI detectors is not a goal of this repo - rewriting AI text to sound less AI to humans is a goal.
Hello - thanks for your analysis!
It is not currently possible to defeat Pangram through pure LLM generation through any skill or prompt (I have tried!)
Regardless, defeating AI detectors is not a goal of this repo - rewriting AI text to sound less AI to humans is a goal.
Of course, it was a pleasure to dig into the data. After running the numbers I think you've achieved that goal with this skill and that's something to be very proud of. The numbers don't lie, as you very obviously know. Looking forward to following more of your work and projects in the future, and again. Thank you.
Let me know if you have any other additional thoughts down the road. I'll be tinkering more with the skill in the future for certain.
Of course, it was a pleasure to dig into the data. After running the numbers I think you've achieved that goal with this skill and that's something to be very proud of. The numbers don't lie, as you very obviously know. Looking forward to following more of your work and projects in the future, and again. Thank you.
Let me know if you have any other additional thoughts down the road. I'll be tinkering more with the skill in the future for certain.
Then it's not a humanizer - A human or humanized text should pass all the AI detectors (except in case of a false positive)
I would be interested if you could point me to a tool that can do that today. I'd be pleasantly surprised. I don't think @blader's tools make claims that it does that, nor does mine.