Uncovering ChatGPT Usage In Academic Papers Through Excess Vocabulary

Frequencies of PubMed abstracts containing certain words. Black lines show counterfactual extrapolations from 2021–22 to 2023–24. The first six words are affected by ChatGPT; the last three relate to major events that influenced scientific writing and are shown for comparison. (Credit: Kobak et al., 2024)
Frequencies of PubMed abstracts containing certain words. Black lines show counterfactual extrapolations from 2021–22 to 2023–24. The first six words are affected by ChatGPT; the last three relate to major events that influenced scientific writing and are shown for comparison. (Credit: Kobak et al., 2024)

That students these days love to use ChatGPT for assistance with reports and other writing tasks is hardly a secret, but in academics it’s becoming ever more prevalent as well. This raises the question of whether ChatGPT-assisted academic writings can be distinguished somehow. According to [Dmitry Kobak] and colleagues this is the case, with a strong sign of ChatGPT use being the presence of a lot of flowery excess vocabulary in the text. As detailed in their prepublication paper, the frequency of certain style words is a remarkable change in the used vocabulary of the published works examined.

For their study they looked at over 14 million biomedical abstracts from 2010 to 2024 obtained via PubMed. These abstracts were then analyzed for word usage and frequency, which shows both natural increases in word frequency (e.g. from the SARS-CoV-2 pandemic and Ebola outbreak), as well as massive spikes in excess vocabulary that coincide with the public availability of ChatGPT and similar LLM-based tools.

In total 774 unique excess words were annotated. Here ‘excess’ means ‘outside of the norm’, following the pattern of ‘excess mortality’ where mortality during one period noticeably deviates from patterns established during previous periods. In this regard the bump in words like respiratory are logical, but the surge in style words like intricate and notably would seem to be due to LLMs having a penchant for such flowery, overly dramatized language.

The researchers have made the analysis code available for those interested in giving it a try on another corpus. The main author also addressed the question of whether ChatGPT might be influencing people to write more like an LLM. At this point it’s still an open question of whether people would be more inclined to use ChatGPT-like vocabulary or actively seek to avoid sounding like an LLM.

22 thoughts on “Uncovering ChatGPT Usage In Academic Papers Through Excess Vocabulary

      1. I find it often misinterprets or makes up stuff that wasn’t in the text, or leaves important parts out, and the summaries are sometimes vague to the point of uselessness.

        1. Sounds a lot like the majority of the news readers on any network these days.
          I’ve watched one local channels staffers stumble over the same script error for two days.

          1. A great deal of news articles around the web were already written by AI scripts prior to Chat-GPT becoming a thing.

            It’s mostly about swiping articles form reputable news sites and then re-arranging them to appear like original text.

  1. Word tells aren’t new.

    Anybody who uses ‘hullabaloo’ regularly learned English by listening to Pravda’s English language propaganda.

    Also anybody who types or says ‘late stage capitalism’ is an indoctrinated credulous moron repeating derp!

    1. In these final, frenzied days of late stage capitalism, I’m happy to be reminded of the useful word “hullabaloo”, which I shall try to remember to use use to characterize the derp I hear so very often.

  2. Well with the analysis code available people will just know how to cheat better, but consider they’re using ChatGPT to cheat in the first case, most are probably to lazy to go to tall that work.

  3. “According to [Dmitry Kobak] and colleagues this is the case, with a strong sign of ChatGPT use being the presence of a lot of flowery excess vocabulary in the text.”

    Like a love letter to the grant giver.

    1. Their research results show a lot of potential but it’s crucial that they delve into less technical areas of academia, e.g. fine and literary arts, before we accept their methodology. Seeing similar patterns in these, or any area which encourages the use of flowery language, will be crucial to proving whether their method is universally applicable or must be tuned to each one. Given word usage changes over time and the outsized influence of the culture, school curriculum and like the persons under which one studies, shadows or is trained has on one’s own language, I think they have a significant challenge to overcome with a broader dataset.

      No, this was not written using any form of AI but rather NI or PI, i.e. natural or personal. Speaking…in the AI world we find ourselves in of late, does this mean that “common sense” will soon be known as CI?

    1. Most likely. Anyone who opened many books as a kid has been accused of sounding like a dictionary/thesaurus; accusing a studious academic type of sounding like a robot seems like a no-brainer. Which is fortunate for the accusers, as that leaves them a surplus of one-half.

  4. the thing is, we’re not so different from chatgpt. we will radically restructure our language usage in the presence of a new pattern of speaking. people have made a big deal about the fact that all future AI systems will be trained on data in part generated by AI. but natural intelligence also will be trained on data in part generated by AI.

    at the margins, some people are surely avoiding chatgpt directly but nonetheless changing their vocabulary because of it.

Leave a Reply

Please be kind and respectful to help make the comments section excellent. (Comment Policy)

This site uses Akismet to reduce spam. Learn how your comment data is processed.