Money, status, or even survival – there’s no shortage of incentives for faking results in the scientific community. What can we do to prevent it, or at least make it noticeable? One possible solution is cryptographic signing of measurement results.
Here’s a proof-of-concept from [Clement Heyd] and [Arbion Halili]. They took a ThermoFisher Scientific 7500 Fast PCR (Polymerase Chain Reaction) machine, isolated its daughter-software, and confined it into a pipeline that automatically signs each result with help of a HSM (Hardware Security Module).
A many machines do, this one has to be paired to a PC, running bespoke software. This one’s running Windows XP, at least! The software got shoved into a heavily isolated virtual machine running XP, protected by TEE (Trusted Execution Environment). The software’s output is now piped into a data diode virtual serial port out of the VM, immediately signed with the HSM, and signed data is accessible through a read-only interface. Want to verify the results’ authenticity? Check them against the system’s public key, and you’re golden – in theory.
This design is just a part of the puzzle, given a typical chain of custody for samples in medical research, but it’s a solid start – and it happens to help make the Windows XP setup more resilient, too.
Wondering what PCR testing is good for? Tons of things all over the medical field, for instance, we’ve talked about PCR in a fair bit of detail in this article about COVID-19 testing. We’ve also covered a number of hacker-built PCR and PCR-enabling machines, from deceivingly simple to reasonably complex!
Great project. There are a lot of ways to fix this in modern instrumentation without all the work arounds. It’s definitely best done at the manufacturer level and I applaud them for what they’ve done despite having that level of direct access. True hacker stuff. For slow run instruments like PCR it makes sense, for high throughput high volume instruments there could be issues with this sort of thing. Love the premise and theres been a lot of advances in this sort of thing over the years. In some cases they are even putting the key/tag into the physical samples themselves! For PCR thats a no-go.
yeah the headline item’s only part of it – I’m also impressed by the virtualization they did, that alone’s pretty impressive IMO! Even without security measures, this kind of “putting manufacturer software into a container” work is seriously worthwhle, and then they did extra stuff with it – that’s honestly way more than I could expect from a WinXP system setup for a PCR machine companion software, and it’s pretty reassuring to see things done right
Disagree on handing this off to manufacturers. Maybe having a mechanism built in, but not the actual keys, and through a standard interface, not their garbage software package.
I am not sure what signing achieves in disease screening, when you can input any sample if manipulation of the results is your goal. In my country we do ring testing by sending copies of the same sample to different laboratories. Manipulation needs to be detected already at the time the specimen is taken and bagged (sealed). If the problem is in academia, maybe have 2 universities study the same disease on the same people independently.
yeah that’s what i think too. there’s a je nes say qua of truth-seeking behavior and for better and worse, detecting that spirit in our comrades is almost the only tool we have for results we can’t verify ourselves. if someone is trying to fake their results it’s going to be very hard to establish safeguards other than recognizing a faker or an outlier in the data.
(je ne sais quoi) 🙂
That’s why reproducible results and peer review matter. Both, though. One is useless without the other. Also pretty important that both institutions have totally different sponsors, lest they both have the same conflict of interest.
It detects tampering, sure. It makes it easier to forensically spot fraud, yeah. But it also provides a crucial way of detecting and countering bit-rot.
My take: the authors thought this to be important enough to devote hours of work to it, so there might be something there? Definitely not the whole solution, but it’s a fun experiment and it could very well be applicable in other specific situations!
It’s not criticism, your article and the work is great! But the external link mentions this is only one step of many. An external lab has no incentive to return fake results, quote the opposite, they’d lose their accreditation (ISO 1702) and be liable to be sued.
I hope my follow-up comment clears up any unintentional misunderstandings. Did not mean to make anyone feel bad over my analysis how to fix the problem at the top.
Joe got it right: “reproducible results”
My idea in general was to do reproduction at the beginning, so false journal articles cannot “poison the well” for years.
Isaac Asimov wrote in the 1980’s already that people do not want to trust the experts. I am a scientist as well, to me the trust in academia is pivotal. One person can taint the good reputation of an entire institute.
Anyway, everyone a fine Sunday.
Yeah, the premise is largely mistaken, all this provides is that a specific machine was used at a specific time. Researchers are not going out of their way to fake results in any discernable number and usually fraudulent conspiracies simply don’t publish the parts they want to hide.
E where this might be actually valuable it’s cases where chain of custody is monitored, which certainly includes medical treatment, finances, etc. Notably the precursors of (70’s} digital cryptographic ledgers were called “triple entry accounting” because of the obvious use case.
Perhaps less applicable to PCR, as it’s a quality test. I.e. it’s either “is” or “isn’t”. It’s not a chromatographic test where full data about sample is recorded But even in that case all boils down to samples and test kit, not machine itself.
Really great idea. In an ideal
Future journals and conferences would require data to be collected on instruments like this. Not just PCR obviously but any instrumentation. the same way they require statements about conflict of interest etc. also there would be a huge segment of science this just can’t work for and there is an argument that cheaters gonna cheat no matter what but it would be a start. I love it.
PCR, atleast in some of the applications it has been put to in the last few years, is far too prone to false positives to be all that useful. Adding cryptographic signatures to results that could already be flawed through the use of an excessively high cycle threshold risk implying a level of accuracy which simply isn’t there in the first place. It would be like quoting a measurement of the length of a high street made with a 1 metre rule stick to an accuracy of the nearest nanometre, implying a level of confidence in the result which the measurement method just doesn’t provide in the first place.
I believe this isn’t to certify that the results are accurate per se, just that the dada hasn’t been manipulated after being collected. So in your analogy not the precision of the instrument, just that the person doing the measuring didn’t “accidentally” say 5m when it was really 4m and they knew it.
yeah that. there’s absolutely places where PCR works well and this could help for specifically data manipulation prevention purposes, “this is exactly what the machine produced”. applicability of PCR is definitely a whole another battle, a machine can’t really stop you from misapplying it, that’s why we have peer reviews&meta discussions and whatnot
So, essentially, it’s data/document signing. Why this requires so special implementation using proprietary software OS while a raspberry Nano with OSS can do that? If that original OS, Installing side software on it isn’t safe and may interfere – usually that’s not stick WinXP but customized with “real-time” elements setup.
We’re talking openly about how 40-cycle PCR is completely fraudulent and thus we need some kind of security on this stuff so we don’t go insane again? Wow the vibe shift really is here
did someone actually argue that? as far as I can see, even the OG author isn’t doing that, much less me.
I think people are kinda missing the point of this or maybe I really am missing the point. There already exist several methods of verifying results that are published. Each with pros and cons. Trust for one – the problems with that are self evident. Then there is peer review and at this point I will stop saying there are problems but none of this is perfect. After that is verifiable results- anyone should be able to replicate. This is nearly impossible in reality. Since science seems to be suffering a crisis of mistrust in the US at least, I fully support anything to combat that.