Peer Review

6 Articles

AI’s Existence Is All It Takes To Be Accused Of Being One

February 11, 2024 by 31 Comments

New technologies bring with them the threat of change. AI tools are one of the latest such developments. But as is often the case, when technological threats show up, they end up looking awfully human.

Recently, [E. M. Wolkovich] submitted a scientific paper for review that — to her surprise — was declared “obviously” the work of ChatGPT. No part of that was true. Like most people, [E. M. Wolkovich] finds writing a somewhat difficult process. Her paper represents a lot of time and effort. But despite zero evidence, this casual accusation of fraud in a scientific context was just sort of… accepted.

There are several reasons this is concerning. One is that, in principle, the scientific community wouldn’t dream of leveling an accusation of fraud like data manipulation without evidence. But a reviewer had no qualms about casually claiming [Wolkovich]’s writing wasn’t hers, effectively calling her a liar. Worse, at the editorial level, this baseless accusation was accepted and passed along with vague agreement instead of any sort of pushback.

Showing Your Work Isn’t Enough

Interestingly, [Wolkovich] writes everything in plain text using the LaTeX typesetting system, hosted on GitHub, complete with change commits. That means she could easily show her entire change history, from outline to finished manuscript, which should be enough to convince just about anyone that she isn’t a chatbot.

But pondering this raises a very good question: is [Wolkovich] having to prove she isn’t a chatbot a desirable outcome of this situation? We don’t think it is, nor is this an idle question. We’ve seen how even when an artist can present their full workflow to prove an AI didn’t make their art, enough doubt is sown by the accusation to poison the proceedings (not to mention greatly demoralizing the creator in the process.)

Better Standards Would Help

[Wolkovich] uses this opportunity to reflect on and share what this situation indicates about useful change. Now that AI tools exist, guidelines that acknowledge them should be created. Explicit standards about when and how AI tools can be used in the writing process, how those tools should be acknowledged if used, and a process to handle accusations of misuse would all be positive changes.

Because as it stands, it’s hard to see [Wolkovich]’s experience as anything other than an illustration of how a scientific community’s submission and review process was corrupted not by undeclared or thoughtless use of AI but by the simple fact that such tools exist. This seems like both a problem that will only get worse with time (right now, it is fairly easy to detect chatbots) and one that will not solve itself.

“Room Temperature Superconductor” LK-99, Just Maybe It Could Be Real

August 1, 2023 by 66 Comments

To have been alive over the last five decades is to have seen superconductors progress from only possible at near-absolute-zero temperatures, to around the temperature of liquid nitrogen in the 1980s and ’90s, and inching slowly higher as ever more exotic substances are made and subjected to demanding conditions. Now there’s a new kid on the block with an astounding claim of room-temperature and pressure superconductivity, something that has been a Holy Grail for physicists over many years.

LK-99 is a lead-copper-phosphate compound developed by a team from Korea University in Seoul. Its announcement was met with skepticism from the scientific community and the first attempts to replicate it proved unsuccessful, but now a team at Huazhong University of Science and Technology in China claim to have also made LK-99 samples that levitate under a magnetic field at room temperature and pressure. This is corroborated by simulation studies that back up the Korean assertions about the crystal structure of LK-99, so maybe, just maybe, room temperature and pressure superconductors might at last be with us.

Floating on a magnetic field is cool as anything, but what are the benefits of such a material? By removing electrical resistance and noise from the equation they hold the promise of lossless power generation and conversion along with higher-performance electronics both analogue and digital, which would revolutionize what we have come to expect from electronics. Of course we’re excited about them and we think you should be too, but perhaps we’ll wait for more labs to verify LK-99 before we celebrate too much. After all, if it proves over-optimistic, it wouldn’t be the first time.

Peer-Reviewed Continuity Tester

September 30, 2022 by 10 Comments

One of the core features of the scientific community is the concept of “peer review” where any claims made by a scientist are open to be analyzed and reproduced by others in the community for independent verification. This leads to either rejection of ideas which can’t be reproduced, or strengthening of those ideas when they are. In this community we typically only feature the first step of this process, the original projects from various builders, but we don’t often see someone taking those instructions and “peer reviewing” someone’s build. This is one of those rare cases.

[oxullo] came across [Leo]’s original build for the ultimate continuity tester. This design is much more sensitive than the function which is built in to most multi-meters, and when building this tool specifically some other refinements can be built in as well. [oxullo] began by starting with the original designs, but made several small modifications. Most of these were changing to surface-mount parts, and switching some components for ones already available. Even then, there was still a mistake in the PCB which was eventually corrected. The case for this build is also 3D printed instead of being made out of metal, and with the original video to work from the rest fell into place easily.

[oxullo] is getting comparable results with this continuity tester, so we can officially say that this design is peer reviewed and tested to the highest of standards. If you’re in need of a more sensitive continuity sensor, or just don’t want to shell out for a Fluke meter when you don’t need the rest of its capabilities, this is the way to go. And don’t forget to check out our original write-up for this tester if you missed it the first time around.

Peer Review In The Age Of Viral Video

November 29, 2017 by 73 Comments

Recently, a YouTube video has been making the rounds online which shows a rather astounding comparison between two printed models of the US Capitol. Starting with the line “3-D PRINTERS CAN NOW PRINT TWICE AS FAST”, the video shows that one print took four hours to complete, and the other finished in just two hours by virtue of vibration reducing algorithms developed at the University of Michigan. The excitement around this video is understandable; one of the biggest limitations of current 3D printer technology is how long it takes to produce a model of acceptable quality, and if improvements to the software that drives these machines could cut total print time in half, the ramifications would be immense.

In only a few weeks the video racked up tens of thousands of views, and glowing articles popped up with headlines such as: “How to cut 3D print times in half by the University of Michigan” and “University of Michigan professor doubles 3D printing speeds using vibration-mitigating algorithm“. Predictably, our tips line lit up with 3D printer owners who wanted to hear more about the incredible research that promised to double their print speed with nothing more than a firmware update.

The only problem is, the video shows nothing of the sort. What’s more, when pushed for details, the creators of the video are now claiming the same thing.

Continue reading “Peer Review In The Age Of Viral Video” →

HJWYDK: The Journal Our Community Has Been Awaiting

November 28, 2017 by 44 Comments

We’re excited to announce the Hackaday Journal of What You Don’t Know. This will be a peer-reviewed journal of white papers that goes well beyond “look what I did” and will provide full design, data, and everything else needed to reproduce the most interesting things the engineering world has to offer. It’s a complete description of your knowledge offered up for the benefit of all.

Topics will include original and creative research, engineering, and entertainment in the areas of interest to the Hackaday community. These papers should embody original insight, experience, or discovery in any sufficiently challenging domain knowledge. This will be the manual for the things you need to know, but probably don’t. HJWYDK makes that knowledge freely available using the Open Access model for publications. It will be a journal without paywalls or frustration. It’s the journal you will reach for whenever you need to do something that feels impossible.

Useful information doesn’t just happen. It’s won through struggle and leads to unique knowledge. Have your accomplishments recognized at a higher level, and make sure they live on and are freely available.

All papers accepted by the editorial and review process will be immediately published online. They will also be printed in the annual Proceedings of the Hackaday Superconference, with the best submissions invited to present in person at the conference. Submit your papers now!

We are currently seeking Associate Editors and Peer Reviewers. Editors should send your background info to journal@hackaday.com. Reviewers should join the team on the HJWYDK project page and mention your areas of expertise in the join request.

Become A Peer Reviewer For Citizen Scientist

June 20, 2016 by 10 Comments

One of the keys to our scientific community is the concept of Peer Review. When important discoveries are made, the work is reviewed by others accomplished in the same field to test the findings. This can verify the work, but it can also open up new questions and lead to new discoveries.

We’re adding Peer Review to the Hackaday Prize. It’s a new way to apply your skills for the benefit of all. The current challenge is Citizen Scientist; calling for projects that help make scientific research more widely available. A set of independent eyes giving constructive feedback to these entries can be a huge end run to success. After all, you don’t know what you don’t know. Having help recognizing stumbling points, or just receiving a second opinion that you’re on the right track makes a big difference when treading in unknown territory.

Becoming a Peer Reviewer is simple. Pick a project you are interested in, review it thoroughly while making notes in a respectful, positive, and constructive way. When you’re ready, submit your Peer Review using this form. We will privately share your review with the project creator.

Hackaday.io is the most vibrant hardware collaboration platform in the world. Peer Review is yet another interesting way to get more brilliant minds in our community involved in building something that matters.

The HackadayPrize2016 is Sponsored by: