Editors Should Give Concrete Evidence When Marking Reviews As Plagiarised or AI-written And not Rely 100% on AI Detectors.
complete
Nyang'or Otieno
I've had an editor mark my review as plagiarized because they say some sentences in my review "don't feel like something a human can write", and they said AI detectors confirmed their claims. I found this site for raising complaints because editors decisions are final and I'd have nowhere else to air this.
First, since when is "feeling a review isn't something a human can write" proof? And speaking of AI-detectors, don't these editors know that AI checkers aren't accurate and shouldn't be trusted without additional concrete proof?
Research has shown these detectors give false positives and negatives countless times, so why should they be used irrefutable proof here? AI detectors have even marked the US constitution as AI-written, how much more a mere review? Several sources have hugely pointed out these flaws:
Despite the emergence of AI detector tools, concerns have been raised about their authenticity and effectiveness. Two primary reasons underlie these doubts:
- The workings of many AI detector tools are not fully understood. This is raising questions about their reliability.
- Some tools produce seemingly random results, further casting doubt on their accuracy.
While AI detection tools may help identify some AI-generated content, their accuracy, and reliability are far from guaranteed.
And here's sound advice from this website:
Educational institutions, businesses, and individuals should be mindful of the risks of relying solely on AI detection tools to identify AI-generated content. Instead, a combination of such tools, human judgment, and other verification methods should be employed to ensure the authenticity of the content.
The study in https://edintegrity.biomedcentral.com/articles/10.1007/s40979-023-00146-z found that:
Although such automated detection can identify some plagiarism, previous research by Foltýnek et al. (2020) has shown that text-matching software not only do not find all plagiarism, but furthermore will also mark non-plagiarised content as plagiarism, thus providing false positive results. This is a worst-case scenario in academic settings, as an honest student can be accused of misconduct.
These sources are too many and research that flaw these AI Detectors overwhelming, and I'm confounded that I've even had to research all this after this nasty experience with an editor who seems not to know or disregard all this. Sadly, some editors will still go ahead to use AI detectors as the ONLY evidence for marking reviews as AI written like in my case. They don't give any concrete or logical proof at all other than what these flawed AI Detectors give them. Worst-case scenario would be when administrators side with such editors, which is something I'm yet to find out in my own case. I believe editors should be made aware that such serious decisions like marking reviews are AI written or plagiarized must be accompanied with concrete evidence and not just results from automatic detectors.
Scott Hughes
complete
Editors are already required to provide concrete evidence.
Editors are already advised to check at least two different automated AI detectors that have already been tested to prove they are accurate.
Keep in mind, multiple peices of evidence that wouldn't be considered "concrete" or sufficient proof on their own often would be when taken together.
For instance, if a review does not contain multiple specific examples from the book and/or the reviewer's own personal life (i.e. if the phrase "for example" and/or "for instance" does not appear multiple times in the review) then that on its own is not sufficient proof that the review was written by AI, but combined with two different AI detectors marking it as AI, then taken together would be considered proof beyond a reasonable doubt.
Nyang'or Otieno
Scott Hughes and which are these AI detectors? I've personally tested as many of the free AI detectors available online with my own writings that were written from scratch, and they've sometimes given false positives. Similarly, I've taken a clearly AI-written writeup but some of them have given false negatives. These include Quillbot, Grammarly, Copyleaks, Undetectable.ai, zerogpt, and a few others. I can record a video repeating the test to prove this. Its the reason some of these AI detectors like Quillbot clearly indicate in their Web page that you shouldn't rely on them, especially if you're making a decision that would likely negatively impact someone's career or academic standing. I'm yet to find an AI detector that is 100% accurate. If this is the case, how is it possible for these editors to ascertain beyond reasonable doubt that the result they've gotten are not false negatives or positives? I agree the results are sometimes accurate, but what about the times when they aren't? I think AI detectors work on the principle of probability, and since AI have perfect grammar and sentence flow, writings that also appear perfect are sometimes flagged whether they are AI-written or not. Similarly, if you write an utterly poor sample with multiple grammar errors and unreadable writeup, it won't be flagged even if it was generated by AI. There are certain instances AI involvement is clear, like when there is utterly wrong information about the plot of the book, the author's name being misspelled, wrong title being used multiple times, etc, all of which can be irrefutable evidence that AI could have been involved. But using AI detectors as the sole evidence is a little over the hedge. From where I stand, most of the AI detectors available online have been tested and found not to be 100% accurate, so editors should not solely rely on them.
Stephanie Runyon
Nyang'or Otieno I agree with you. I've been flagged and the editor literally put what the AI detector stated word for word. It literally said that the review was human written. Appealing with an admin has cost me 14 days so far.
Stephanie Runyon
I am still waiting. I have been since November 21st.
Mayang Bature
Nyang'or Otieno you're actually bringing up valid points and I recognise your name from the review team too. But here, it seems Scott is dismissing these points and marking such suggestions as complete despite the shallow replies they get. He doesn't want to explain who tested these AI detectors or how and just labels them as "concrete evidence."
Another baffling thing is the concept of collective evidence that becomes definite proof. Shouldn't this logic work the other way around too? That is, if two or more AI text detectors say that a review was not written by AI, would that not be sufficient "concrete evidence" and proof that that review was not written by AI?
At this point, the use of AI to solve this issue is only counterproductive and actually does more harm, in my opinion.
R
Riziki
Mayang Bature, Stephanie Runyon, Nyang'or Otieno, Scott Hughes.
I'll just leave this here in case Scott and the management cares to even give it a thought:
He explains it really well why AI detectors should not be trusted:
And here is Turnitin itself 'owning to their mistakes (on false positives)' and advising their users to take their results 'with a BIG grain of salt' and be the last judges on whether the articles are AI written or not depending on context and unique case-by-case basis:
Julia McCoy breaks it down really well here:
And then think about it: most ai detectors also have "ai removers" or "humanizers" as an additional or separate service (and most often paid). It makes sense that they may (allegedly) deliberately output false positives to get people to pay for their ai humanizing tools. If I genuinely wrote something originally from scratch and an ai detector said it's detecting AI involvement, it's only natural that I would readily want to reduce that ai score to protect my academic standing or career even if that result isn't accurate, even if it means paying for those ai humanizers. How can such be trusted at all? It's like the proverbial doctor who painted stones to look like football and the next day his/her hospital was teeming with patients to be treated for feet injuries after attempting to kick those ball-like stones.
Simply dismissing and banning a user for using AI without any other backing at all except results from AI detectors is not right. Understand that the problem isn't banning people who actually use AI but SOLELY depending on AI DETECTORS to do that with absolutely no other proof. It may hurt scammers, yes, but it hurts and scares off people who genuinely didn't use AI more - people whose reviews were inaccurately flagged by these AI detectors but the editor just went along with it. I personally know friends who've quit onlinebookclub.org forever and want nothing to do with it after their reviews were wrongly flagged and the editors just went with it without any additional backing at all, and worse, the admins, who are supposed to be in the middle ground and look at the situation more soberly and consider all and such possibilities, sided with the editors.
R
Riziki
Ps:
I think solely relying on AI detectors without additional proof is setting up this platform for failure in the long run.