AI Helps Spot 1,000 “Questionable” Journals

Scientists, you might be familiar with the feeling of opening your email inbox to find a message from a journal you have never encountered before, offering its services in publishing your research.

In today’s “publish or perish” culture, this offer might sound enticing – until you are hit with sizable (often hidden) fees and none of the services that an author might expect, such as peer review or editing.

Journals with these low editorial standards, sometimes called “predatory” journals, are increasingly posing issues for scientists. Even when their behavior isn’t necessarily predatory from a financial perspective, their unethical approach to publishing large volumes of unreviewed research can make it difficult for scientists to find high-quality information. 

Establishing whether a journal is a reputable publication or not can be a difficult task. The website for a journal with questionable editorial practices may seem entirely professional on the surface, but dig deeper, and you might uncover a web of self-citations and other poor practices.

Now, computer scientists have developed a new AI-powered tool that can automatically flag questionable journals.

In a paper published in Science Advances, the team used the AI to sift through a list of nearly 15,200 open-access journals available on the internet. From these, the tool flagged approximately 1,400 journals as being potentially problematic. Subsequent auditing from human experts determined that, while the AI did make mistakes, more than 1,000 of the flagged journals did indeed show some form of questionable practice.

To learn more about this tool and the dangers of such journals, Technology Networks spoke with the study’s lead author Daniel Acuña, associate professor of computer science at the University of Colorado Boulder.

 

Alexander Beadle (AB): How would you define a “questionable” journal?

Daniel Acuña (DA): This can be a rather touchy subject, so we wanted to be really careful with how we presented this research. There has been a lot of controversy around calling journals “predatory”, which is obviously a very loaded word.

We wanted to stick within a definition that relies on an institution that is very well-regarded, the Directory of Open Access Journals (DOAJ). They don’t call them questionable journals, but essentially, we said that if the DOAJ is removing a journal from their list and the reason they provide is because of certain questionable practices, like providing no information on their editorial board, then we will label those as “questionable”.

More broadly speaking, questionable journals are journals that do not serve the purpose that journals are supposed to – being a repository of human knowledge that has gone through some filtering and evaluation from peers, with everything being transparent and free of conflicts of interest. 

 

AB: What sorts of dangers do questionable journals pose?

DA: I am a scientist, so I search for stuff all the time. Indexing services such as Google Scholar don’t really filter where they extract their information from, so you run the risk of these journals effectively contaminating the research of others.

As the saying goes, “science is built on the shoulders of giants”. But really, it is about more than giants – it is built up on the shoulders of little people all over the world, building on what each other discover, trusting that the published research they see has gone through some kind of filter.

Of course, that is not always the case. Sometimes you might choose to rely on a pre-print publication, and so you must do a little more legwork. But you are at least assuming that there are good intentions behind what you are reading and that people are not just publishing whatever comes their way.

I should also note, you can have publications in questionable journals that are okay; it doesn’t mean that every paper is problematic, it just means that it didn’t go through what the scientific community would consider to be a proper process for publishing that knowledge.

 

AB: In your paper, you present an AI-powered tool that can help to flag questionable journals. How exactly does this tool evaluate if a journal is following best practice?

DA: Evaluating journals with AI has been tried it before, but the issue we saw with these previous approaches is that they work as “black boxes” – it isn’t clear how the AI is making these decisions.

We wanted to connect our AI to real metrics that people use, such as the best practices that the DOAJ outlines. When our AI tells us that a journal has a 99% probability of being questionable, we can see why it thinks that.

For example, the DOAJ says that journals should have an advisory body or an editorial board, the board should be listed and it should be made up of people who have some connection to that area of science in some way. We ask the AI to go to the journal’s editorial board website page and scrape it. Then we see if those board members are present in databases of publications and authors and scientists.

We take three major sets of features. Firstly, how the website looks; does it have proper content (i.e., does it have a page listing the editorial board) and how does it look aesthetically (i.e., does it look professional). We also look at the code of the website to see how it was built. There is research out there that suggests that there are companies that are using software to spawn journals and when they get caught, they will start a new one that uses the same underlying code.

The last set of features is related to what we call “bibliometric patterns”, such as frequent self-citation, lots of citations from the same institution or journal, who the authors are, the frequency of citations, etc.

 

AB: How accurately does this tool perform?

DA: Surprisingly, we thought we would need all the feature sets that we had built-in to get good results. But practically, you are never going to have all of these – website scraping is not always possible because of CAPTCHAs, blocked content, etc. Focusing on the bibliometrics and removing things that are difficult to capture, the performance wasn’t quite as good as everything together, but it was still good enough that we could take this model and apply it to larger datasets that we hadn’t analyzed before.

In the grand scheme of things, we have a false positive rate of around 24% percent. That means that out of the 15,000 journals we looked at, when the AI predicts that around 1400 will be questionable, just over 300 of them will turn out to actually not be problematic. Of course, that means some wasted time for the human auditors, but we will also then be able to catch around 1000 questionable journals.

The false positive rates are something that you can tweak, in the sense that our model tells you a probability of something being problematic and the user gets to choose how stringent they want the bar to be. Let’s say you want to be like the Transportation Security Administration (TSA) at the airport, you want to catch every single bad actor and be super stringent. Just like at the airport where you have that incredibly low threshold, you will end up falsely flagging a lot of normal people, but the advantage is that you can catch every bad guy.

 

AB: Why is it useful to have this kind of tool, as opposed to exclusively relying on those human auditors?

DA: It is a volume issue for sure, but the other issue is that these journals adapt and change strategies all the time. It’s good to have an AI that can also adapt quickly and catch these journals that are gaming the system and mimicking real websites. People understandably sometimes make errors, but some things are just very difficult to catch. Sometimes you do have cases where the website might not look that great, but if it is on a very obscure topic, it could just be a very obscure yet legitimate journal.

The other key point is that the DOAJ is a community-driven organization. This means that they can really only systemically analyze a very small number of journals every year – it is a task that simply takes a long time to do.

 

AB: Are there plans to one day make the tool more widely available? How do you see this technology being applied in the long-term?

DA: Since starting this research many years ago, I have started a company to help publishers with research integrity issues – ReviewerZero AI – and this is all disclosed in the paper as well. But yes, [widening availability] is one of the things that we would like to do. At the moment, the tool is available through the services of ReviewerZero AI for people who are professionals in the field, such as research integrity officers.

We are updating the tool with other signals and it should get better over time. Once it gets into a state where it could potentially be shared with authors, it would be an interesting application to help them filter out unsuitable journals for their work. What we see is that a lot of the authors who are published in these questionable journals are from the Global South, and so perhaps we can share this tool with universities there so that they can help their scientists vet the journals where they submit their work.

 

About the interviewee:

Daniel Acuna, PhD is an associate professor of computer science at the University of Colorado Boulder, where he is currently on entrepreneurial leave. He is the founder of ReviewerZero AI, an AI-driven platform that identifies issues like image tampering, citation manipulation and statistical anomalies to protect research integrity.

In academia, Acuna has conducted extensive research in research integrity, publishing over 70 peer-reviewed papers. His work has been supported by the U.S. Office of Research Integrity, the National Science Foundation, the Sloan Foundation and DARPA’s SCORE initiative. In 2021, he organized Computational Research Integrity Conference (cri-conf.org), which united researchers, ethicists and tech experts to address real-world issues in research integrity.

Reference: Zhuang H, Liang L, Acuna DE. Estimating the predictability of questionable open-access journals. Sci Adv. 2025;11(35):eadt2792. doi: 10.1126/sciadv.adt2792