[Josip] has been playing around with race conditions on web interfaces lately, finding vulnerabilities on both Facebook and Digital Ocean. A race condition can occur when a piece of software processes multiple threads using a shared resource.
For example, [Josip] discovered that he was able to manipulate page reviews using just a single Facebook account. Normally, a user is permitted to leave just one review for any given Facebook page. This prevents a single user from being able to skew the page’s overall ranking by making a bunch of positive or negative reviews. The trick to manipulating the system was to intercept the HTTP request that submitted the page review. The request was then replayed over and over in a very short amount of time.
Facebook’s servers ended up processing some of these requests simultaneously, essentially unaware that multiple requests had come in so close together. The result was that multiple reviews were submitted, artificially changing the pages overall ranking even though only one review actually showed up on the page for this user. The user can then delete their single review, and repeat this cycle over and over. It took Facebook approximately two months to fix this vulnerability, but in the end it was fixed and [Josip] received a nice bounty.
The Digital Ocean hack was essentially the exact same process. This time instead of hacking page reviews, [Josip] went after some free money. He found that he was able to submit the same promotional code multiple times, resulting in a hefty discount at checkout time. Digital Ocean wasted no time fixing this bug, repairing it within just ten days of the disclosure.
Interesting that a billing system is not using sanitized transactions, this is (to me at least) a new attack vector but as with sql injection the blame is on the developer for not knowing how sql works. Deep SQL knowledge should become a mandatory requirement for any developer dealing with billing, accounting, medical or costumer data.
The solution tho this problem is obvious, simple and should not take “just 10 days” to fix
In the stored procedure (you are using SPs right? oh right the new hippie kids prefer object model to abstract the db silly me) that does an actions that should be validated for uniqueness, start a transaction, make the checks you need to the data (enough money for a withdraw, promotion code not used, etc), if the checks pass make the changes necessary to the tables and commit the transaction, if the checks don’t pass, rollback
Agreed.
The first step is to check if (x,y,z), then, they shall pass.
Another important note is that multiple checks should never be done at the same time. Do each check in a separate transaction, then go to the next one.
Some devs love to copy/paste (not mentioning any names) examples online without realizing that something simple will get them canned.
I think all the checks (x,y,z) should be done in the same transaction, to assure the atomicity of the checks and the effects. Unless the checks are assuring against unconexed effects, splitting the transaction into several transaction is a common source of race condition errors.
You make a lot of assumptions about their architecture. Your comment holds true for small, simple sites backed by a single SQL database. Your comment also shows a lack of experience with sites backed by micro services.
A service should make the same validations and fail if required conditions are not met.
Changes to a dataset should always be atomic if you value your data, where and how the dataset is stored/accessed should be irrelevant
Blame the project manager up through top executives. They hold the purse and set the agenda.
Facebook’s dB is (in)famously not ACID-compliant, and I’ve often wondered how they managed the transactional-style stuff that all dBs have at some level to maintain integrity. Seems they didn’t.
I wonder how much they had to fix.
10 days to fix? Some people literally have more money than brains.
http://imgur.com/r/ProgrammerHumor/oTuw29u
He also knows not to mention that he actually multiplies his repair estimates by a factor of 6