Bureaucracies generate paper, usually lots of paper. Anything you consider private — especially anything that could get you in trouble — should go in a “burn box” which is usually a locked trash can that is periodically emptied into an incinerator. However, what about a paper shredder? Who hasn’t seen a movie or TV show where the office furiously shreds papers as the FBI, SEC, or some other three-letter-agency is trying to crash the door down?
That might have been the scene in the late 1980s when Germany reunified. The East German Ministry of State Security — known as the Stasi — had records of unlawful activity and, probably, information about people of interest. The staff made a best effort to destroy these records, but they did not quite complete their task.
The collapsing East German government ordered documents destroyed, and many were pulped or burned. However, many of the documents were shredded by hand, stuffed into bags, and were awaiting final destruction. There were also some documents destroyed by the interim government in 1990. Today there are about 16,000 of these bags remaining, each with 2,500 to 3,000 pieces of pages in them.
Machine-shredded documents were too small to recover, but the hand-shredded documents should be possible to reconstruct. After all, they do it all the time in spy movies, right? With modern computers and vision systems, it should be a snap.
You’d think so, anyway.
Shield and Sword of the Party
The Stasi has been likened to the Soviet KGB. Using civilian informants, they contributed to the arrest of about a quarter of a million people between 1950 and 1990. They also had extensive files, and if they survived the destruction, people can ask to see the information the Stasi collected about them.
The agency was known for pervasive and invasive spying, with agents in every apartment building and all major companies. At the time they disbanded, the agency had over 90,000 employees and nearly 175,000 informants. This works out to one secret policeman for every 166 East Germans. By contrast, the Gestapo had one agent for every 2,000 people. They were also known for harassing enemies of the state. If you want to learn more about the Stasi, Deutsche Welle has an interesting short documentary about the agency and its spying activities that you can watch below.
So you can see why the Stasi leadership wanted to destroy files. Citizens occupied the Stasi offices, but not before about 5% of the documents — 1 billion sheets of paper — were destroyed somehow. As the German Democratic Republic fell, many citizens protested the destruction of the papers, primarily to ensure there was evidence to prosecute wrongdoing in the agency. However, some informants wanted the documents destroyed so they would not be identified.
The new government appointed an office to control the records, but there was a strong debate about what to do with them. Some wanted them sealed or destroyed. Others wanted them used for prosecution. In the end, the Unification Treaty allowed people to access their own files starting in 1992. Between 1991 and 2011, about 2.75 million people have requested to see their files. Near relatives can also request the files of deceased or missing persons. The media and schools can access documents that are redacted for personal information. The archive is now the responsibility of the Federal Archives.
Reconstruction
However, there are still these 16,000 bags of fragments — about 45 million pages worth. In some cases, the destroyed files had pages simply torn in half or quarters. Those are the easy ones, but they do not all fall into that category. The 36 archivists tasked with reconstruction processed 327 bags in 13 years, not a speedy record.
To help, the German government turned to computers and the Fraunhofer Institute. Scientists there demonstrated software known as e-Puzzler that would revolutionize the document reconstruction process. However, that turned out not to be the case. While it does work, the process is painfully slow.
In theory, it makes sense. An article in The Guardian from 2007 describes the machine. According to the article:
The machine works by scanning the document fragments into a computer image file. It treats each scrap as if it is part of a huge jigsaw puzzle. The shape, colour, font, texture and thickness of the paper is then analysed so that eventually it is possible to rebuild an electronic image of the original document.
Some marketing material from Fraunhofer itself says, “The system uses an adaptive, non-deterministic workflow to process a wide range of characteristics, such as the contour, color, writing, and lines of the fragments.” Seems plausible. You can see the system in action in the video below. That video also notes some of the possible reasons the project has been a failure.
So What Went Wrong?
It isn’t clear why this isn’t feasible. The Fraunhofer system did help Bundesbank match up damaged banknotes. However, banknotes are more uniform and have known features that the Stasi documents lack. And, as you can see in the video below, it still looks like there is some manual work required. Despite putting in about 6.5 million euros, the official word is the process didn’t scale well for this many documents.
On the one hand, you have to imagine that computers and image processing has come a long way since 2013. It is surprising you couldn’t do much better with modern hardware and techniques. Of course, if you were conspiracy-minded, you might wonder if someone doesn’t want the project to succeed.
To be fair, opening the bags is a chore. Archivists try not to disturb the order of the papers and they often contain trash which we are sure is pretty disgusting after all these years. Some papers have clips or staples and many are wrinkled. The machine needs the pages separated and flattened. To help speed up the process, each piece destined for the machine has to be about 2 cm square or larger. The downside is the documents are two-sided, which doubles the number of trips for each piece.
We get it. Building one radio is easy. Building 16,000 of them is hard. We know people can unshred documents that aren’t reduced to dust or ash. Even the crosscut shredder isn’t foolproof if you have the right open-source software. The Iranians famously employed carpet weavers to reassemble documents taken from the US embassy in 1979. You can hear more about some of these cases in Edward Robinson’s CHCon presentation in the video below.
What’s Next?
We suspect the Stasi files will remain shredded and unread for a long time, but maybe not entirely for technical reasons. We also imagine since DARPA has sponsored challenges for unshredding, that someone — maybe a lot of someones — has some great tech for this that they aren’t making public.
If you are worried the secret police are listening –and not just Google and Amazon — you need an NLJD. If you like cold war intrigue, how about the adventures of stealing a moon capsule?
Throw some HDC at the problem.
https://www.quantamagazine.org/a-new-approach-to-computation-reimagines-artificial-intelligence-20230413/
“One secret policeman for every 166 East Germans” was the last employed layer, but I’ve read that the layer below that meant 1 in 10. The DDR Stasi was really insidious.
Well a people want a government that looks out for it’s citizens, this is what you get.
No when you get a people that inherently distrust each other built up by years of their leaders social engineering, combined with a culture of rule following, combined with the inability for your government to have any amount of self-determination because you’re country was crushed in a war and dominated by another country.
That is what you get.
I wonder what the percentage was in Iraq under Saddam Hussein? How about North Korea through its history?
It’s how oppressive regimes keep control, by having a massive number of people willing to rat out anyone for anything they can say is traitorous, disloyal, spying, etc. If nobody has been reported in a town or village in a while, randomly grab some family in the middle of the night.
Masaji Ishikawa, born to a Japanese mother and Korean father, lived under that terror in North Korea from 1996 until he escaped to China in 1996. Since his father was formerly in the Korean army, he and his family were treated badly by the Japanese. He took it out on his wife, until Korean Communist agents threatened him, stop hitting your wife, or else. Masaji’s parents were enticed to relocate their family to Korea. They thought it was South Korea. Nope. The promises of a free house, jobs, and free education for the children were of course all lies. The house was a shack, the jobs were forced labor in the rice paddies, and the “education” was indoctrination on the greatness of communism and the glory of Kim Il Sung.
At one point Masaji got a job driving a tractor, but there were no maps and asking the wrong person how to get to some village could get you arrested as a spy. Why? Because all the roads were state secrets.
Two translators worked with Masaji on his book “A River in Darkness” about his early life in Japan, the move to North Korea, and all the horrible things that happened there, including being reduced to eating grass and weeds when the government would take all the crops they grew. The only reason China didn’t toss him back to North Korea when he made it across the border in 1996 was because he was half Japanese. Apparently they had a policy to repatriate to Japan any Japanese or half Japanese who made it out of North Korea.
The Stasi had many civil agents, too. That’s why East Germans did (do) never really trust anyone.
It could be your nice friendly grandma who’s secretly telling Stasi everything you said about mama and papa and your friends. No, seriously. It was very common that you were being betrayed by your own family members. And that’s just the tip of the ice berg. Many citizens had found hidden microphones in their homes or were being watched and followed by strangers. And just because they perhaps made a political comment or joke in daily life that someone noticed and wrote down. That was enough to get on a list. That’s exactly why that kind of “Aufarbeitung” is so important. These shred papers contain all the ugly details about these things. Nowadays, ex-DDR citizens are usually being informed if their old record was being restored/found. They can then have a look what the Stasi was thinking about them. It’s a form justice, even if it’s late.
mix old paint into shred
I throw our shredded documents 8nto the compost heap.
Insects can read.
What went wrong?
Many of the Stasi and SED operatives are still part of the current government in Germany.
The last chancellor of Germany had a SED / DDR background.
And just like that funding for the project was cancelled.
Linking fund cancellation with the GDR youth of former chancellor Merkel is a bit of a far stretch.
The same could be said about ex-Gestapo and ex-NSDAP members which found their way into both Germanies when they were founded.
In some ways, maybe, East Germany stayed being the Third Reich.
The Soviets didn’t change much, just renamed things.
The Hitl*rjugend (Hitl*r youth) became “Freie Deutsche Jugend” (FDJ, Free German Youth) and so on.
That’s why there is such a break between ex-West and ex-East Germany, maybe.
West Germany had re-adopted democratic elements of the Republic of Weimar,
while it also had evolved, became less “German” and more casual over the years.
In the 80/90s, this went so far that new born children got English names by their parents.
This was a time in which we all believed in a future were we would live in an international community.
East Germany and East Germans, however, didn’t change so much.
They’re still very “German” , which reflects in language and traditions, too.
There’s kind of an aversion against anglicisms and skepticism towards democracy, too, I think.
They also traditionally think that we ex-West Germans are snobby and always look down on them.
As if we had nothing better to do. In reality, I’m afraid, they just struggle to integrate in this new world.
They separate themselves or see differences everywhere. Maybe they have inferiority complexes?
Before I met ex-East Germans in person, I thought they were just like us.
Personally, I find this outcome ironic, because East Germans were totally into American things in the 80s.
Music, Jeans, Movies, American fashion etc.. They even sang “born in the USA” loudly at an open-air
concert..
And when they fled to West Germany, they seemed so grateful back then.
They said we’re such nice and helpful people. Why has this picture changed? What did “we” do wrong? *sigh*
Anyway, speaking under correction here. I also don’t mean to spread propaganda.
I wrote to the best of my knowledge, but I’m not without mistakes, sadly.
So please double check what I said. I merely mentioned this so you have a greater view on the matter.
Saying that funding of the project was cancelled is dis-ingenuous. The whole project was taken from being a specialist project that needed its own funding and made a part of their Federal Archives which has ongoing funding. At the same time their Federal Archives department is mandated by law to continue working on it. The project actually has more resources allocated to it now than it did before.
https://www.stasi-unterlagen-archiv.de/en/the-agency/future-of-the-stasi-records/
My first guess would be that this “unshredding” and puzzling together works quite well. Especially when a large part of the content of each bag is from the same set of documents.
But those who ware in the know have all intent to keep it secret and obfuscated for as long as possible.
Much in the same way that for example the breaking of Enigma was kept a secret for a long time after WWII because similar cipher machines were used a lot for diplomatic correspondence, and as long as it was kept a secret, those machines kept on being used, and were not replaced by more secure machines.
Or why the Venona Project was shut down. It caught Julius and Ethel Rosenberg but not many others. The USSR, when starting up new spies, would only use a spy’s real name once in communication to the USA, when assigning their code name. If our spies didn’t intercept that message, they wouldn’t know what person the code name was for.
Since Venona was declassified in 1995 (and proved that Joe P. McCarthy was right about deep Soviet infiltration into American government in the 1950’s) I’ve often wondered if there’s a large batch of intercepted communications that have not yet been decrypted.
I suspect the very spies Venona was attempting to catch had plenty to do with keeping the program from intercepting ALL of the communications back and forth between the Soviet intelligence agencies and their American agents. The right people in the right places in Venona could quietly ignore certain messages that would be very damaging to the Soviets plans here.
Senator McCarthy had nothing to do with the House Un-American Activities Committee. He was attempting to root out Soviet agents in the Federal Government, not Hollywood. Also, HUAC was started in 1938 and petered out in the early 1970’s under a different name. McCarthy was only a Senator from 1946 until his death in 1957, and only got on the anti-communist bandwagon circa 1950.
Yet many people blame him for everything HUAC did throughout its entire history and call it “McCarthyism”, despite his complete non-involvement with HUAC and brief 7 years going after ‘reds’ in government. I wonder if the USSR had people slip McCarthy just enough info to get him going, then had their “useful idiots” and outright sympathizers in the news media ridicule him?
“The agency was known for pervasive and invasive spying, with agents in every apartment building and all major companies. ”
Google and Facebook are eating the STASI for lunch. And they don’t even have to employ agents. People take them in their homes voluntarily and pay for it to boot.
That’s “Alphabet” and “Meta”, which are both extremely common words and mostly unsearchable, and even if you try to search, uh, guess who filters your results for you :)
The eastern concept of communism was centralized. The western concept is decentralized. Why have a secret policeman on official payroll for every 166 citizens when you can simply convince one in five of them to do it online for free? And train them to have an unconscious policeman inside their own heads?
What?
‘Real communism has never been tried’ is the endless derp of marxists everywhere. They’re wrong though, it’s been tried and always produces authoritarian socialism. Can’t be fixed, inherent in the system. Decentralized marxism has no mechanism to organize an economy. No markets. It’s just a bad idea, no matter how many times it’s renamed.
Too much centralized power, power corrupts. Stick a fork in it, put it’s adherents in a zoo.
‘Come and see the authoritarianism inherent in the system’ shouts the filth loving peasant.
Pure collectivism has never been viable except in some small tribal groups who developed on their own “organically” as their traditional way of life.
Beyond a small group, there’s always someone who seeks power over the rest of the group, and those who will take advantage of the rules to get their share without putting in their full measure of effort. Several early European colonies in North America tried it and the outcomes were 1. Everyone died, game over. Most likely what hapened to the Crotoan colony. 2. Many died then the survivors went back to Europe. 3. Many died and the survivors tossed collectivism and adopted a trading / market / capitalistic system. One of these was the Pilgrims we based Thanksgiving on.
Dictator wannabees and slackers are the dooms of collectivism. The dictators tend to be a sort who disdain “hard work” yet will go to a lot more effort to avoid a few hours of physical labor. If they’d just get out in their assigned plot and tend to the crops each day they’d have more free time and wouldn’t end up with everyone other than their in-group of flunkies hating them.
When people who fail to properly place the needs of the group ahead of their own and their families see that no matter how much work they do, they only get the same amount of food etc out of the common larder, they’ll work less. And keep cutting back. Then when the people in charge of the larder have to reduce the ration for everyone (but possibly not the leader(s) because they’re the brains of the operation) the ones slacking off get upset. So do the ones putting in their full share.
The fix is either to inspire the slackers to pick up the pace, or *force* them to work harder. But then the members appointed to be enforcers are busy standing around with the pitchforks or guns instead of farming, so there’s even less labor going into feeding the group.
It spirals out of control as the graves pile up until the people say “This communal ideal is crap! Everyone gets to do with their plots of land as they please and trade directly with other members. The common larder is closed, now lets see if we can stop starving to death.”
I don’t know. This reads like an essay about the trouble and working mentality in the US, but is it applicable on a global scale? Taking Europeans phase of colonialism as a prime example isn’t exactly appropriate, I’m afraid. It was a dark time of cruelty, suffering and identity crisis. And lack of education. And slavery. Not sure if these things apply to today in exactly same way.
And why must a leader, a manager do physical work ? Mental work can be much more demanding. Working like a robot on a field doesn’t require much thinking. Most pawns did that on a daily basis and had a happy and healthy life (within their limited horizon). Being forced to find a solution for a national or international problem -on time- can be much more of a burden. Being responsible for so many things can cause many sleepless nights, stomach cramps and heart attacks. Especially, if you know that you have so many political enemies that hide in the dark. All the stress can let you age quickly. Just look at how Mr. Obama got his gray hairs within a few years..
Also, there are cultures who have a strong feeling of togetherness.
In Japan, for example, the well being of the community is more important than that of the individual (which isn’t exactly great IMHO). Each individual knows the importance of the community. And they’re all overworked there. People literally die there because they work more than they need, could and should. They’re workaholics, so to say. Families loose their members because of work. Over there, work/labor rather turned out to be a disease rather than a heroic activity.
Please don’t get me wrong, I don’t mean to be a critic here. I really wished I wouldn’t need to write these lines.
But personally, I think that there should be a big “in the USA” sticker on every second post or comment on the web.
Because many of those strange worldviews, especially the political ones, are not applying to the rest of the world. IMHO.
“The fix is either to inspire the slackers to pick up the pace, or *force* them to work harder”
Hm. So in a free country, people must be “forced” to work? Isn’t that a form of slavery then?
And what if those “slackers” are simply people who take a break to sort their thoughts? Who are discouraged to work because they’re not okay with the situation?
What if they are actually philosophers?
Personally, I wonder if that kind of labor is an instrument to hold back citizens from thinking.
Citizens who are “forced” to work hard have no time to think, to question their fate or the situation in their community.
In concentration camps, we learnt in school, people literally worked to death. It was a cruel method to get away with them and to make them silent. And if they couldn’t work anymore, they were burnt alive in the boiler. So much to about how great labor is.
In a free country, no one should be forced to work in order to support others. Pursuing knowledge instead of doing physical labor is fine, but not if it’s at the expense of someone else being forced to do physical labor to support you. All food requires some physical labor to create it, and everyone needs food. I don’t think it’s reasonable to think that all the farmers who support us would do it for free, or that in a perfect society enough people would become farmers just because they want to do that work. Because it’s hard work.
Vernor Vinge foresaw this becoming an industrial process. Why take all the time to scan a book page-by-page when you can dump it into an industrial shredder, blow the shreds down a brightly-lit duct lined with high-speed cameras, and reconstruct it in software? (The older characters who loved and respected physical books and libraries were NOT pleased.)
I was trying to remember who wrote that book. (It’s great, by the way).
The book is “Rainbows End” (no apostrophe, it’s a statement :-) )
As well as good tale it causes one to question whether it is the books as artefacts or the knowledge that they contain that is valuable.
Is it worth destroying the books if the information in them becomes freely available to all?
Seems like an idea that would sound good to a theoretician.
I know people that do this. (Without the shredding. Did anyhow, I expect it’s mostly done by now).
They will destroy the only copy of old docs to scan them. Weird obsession, but I kind of get it. Who knows when you’ll need to use a PDP-11/23 again.
Also recovering docs from old 9 track tapes. Apparently it helps to bake them on low for a day or so.
I’m throwing shredded documents into the toilet. Helps prevent poop splash and increases security. Recovering wet, yellow-brown paper shreds would not be a very nice task, even for very dedicated agency. It’s only a handful of shreds but do it every day and it adds up. Very soon a bag full of shredded paper is empty.
I hope you’re on a septic tank. So you can personally enjoy the consequences.
The urea in urine -supplies nitrogen for bacteria to break down cellulose (in the paper).
If bacteria could effectively break down cellulose, we’d be able to ferment crop waste for fuel.
It remains a dream and a project for genetic engineers. There are a few organisms that can. Termites and their gut bacteria pull it off.
Save on TP. Wipe with shredded paper instead. Stringy clingy though.
And paper cuts where the sun don’t shine. Ouch.
The CIA did exactly this during the cold war as the Russians used classified documents as toilet paper. Never underestimate the amount of effort your enemy is willing to put in.
Several places I worked collected sensitive documents in locked bins that were then collected by a company that dumped the bin though a big shredder in the back of a truck before taking the shreds to a paper recycling plant. Good for document security and good for the planet as well.
The ‘dirty secret’ of recycled paper is that the only real recycled paper is the % listed on a product as Post Consumer. Since the invention of paper a lot of it has been recycled, but within the paper factories. All the edge and end trimmings and when the paper web breaks and the ends are trimmed to straighten them, all of that goes right back to the pulp vats at the star of the manufacturing process and called “recycled”.
So if a paper product claims %100 recycled but doesn’t say Post Consumer or break down the % that’s Post Consumer, then it’s really 0% recycled.
Same deal with plastics. All molding factories working with thermoplastics grind up the waste from sprues, runners, and failed shots then dump it back into the hoppers with fresh “virgin” pellets and often the same type of plastic from post consumer sources. There’s usually always 10% to 15% virgin resin in an item made from recycled plastic. That gets the new item to over 90% of the strength and other properties of the same item made with 100% virgin resin. Design the thing with at least a 10% “over build” and it’ll be just right with 85-90% molded once resin.
An analog is when making breads and pastries a baker will take bits of dough left over from making the loaves, doughnuts etc then mash them together to make another loaf or a few more doughnuts. It’s just like is done with paper trimmings but no bakery boasts of bread and doughnuts containing “recycled dough”.
Tater Tots could be called “recycled potato product”. Ore-Ida invented them as a product to use what had until then been thrown away as garbage. All the snips and bits left from making french fries and other cut potato products were being tossed until someone figured out how they could finely shred it all then make the shreds stick together in partially cooked cylinders then flash freeze them.
This software was probably not made to be sold on the free market to users or companies (perhaps unshredding is just a niche problem). I have seen some patterns in similar, tax funded, projects: to get the grants, much paper work is done up front to proof (or to pretend) the viability and the benefit of the project. Government agencies, often without deep technical knowledge, control the projects by meticulously checking the timesheets, the staffs CVs, their employment contracts and university degrees. Sometimes, promising concepts and prototypes are produced but seldom a great product.
I guess, this is because people working on such projects get too little feedback and too little pressure from real users that need the product and too much feedback and too much pressure from buraucrats that are mainly concerned with ticking the right boxes on their check lists.
The STASI were amateurs doing a sloppy job compared to what is going on now.
And not just in Germany. Nor just in the five eyes countries (+ germany).
I think they now collect so much data and monitor so many communication that even with a 2:1 ratio of spooks v population they could not keep track. Just the cameras alone..
And sure, they use AI and computers to sift, but that too has limitation I would hope.
I seem to recall DARPA offering a bounty for code to reassemble scans of shredded documents. Solved the problem for a cheap $50k.
When I was in the Air Force we had a machine that shredded the documents so fine they looked like dryer lint. It then compressed that lint into blocks that went to fuel a heating plant. Pretty effective.