A group of researchers have built an algorithm for finding hidden connections in artwork.
The team, comprised of computer scientists from MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL) and Microsoft, used paintings from the Metropolitan Museum of Art and Amsterdam’s Rijksmuseum to demonstrate these hidden connections, which link artwork that shares similar styles, such as Francisco de Zurbarán’s The Martyrdom of Saint Serapion (above left) and Jan Asselijn’s The Threatened Swan (above right). They were initially inspired by the “Rembrandt and Velazquez” exhibition in the Rijksmuseum, which demonstrated similarities between the artists’ work despite the former hailing from the Protestant Netherlands and the latter from Catholic Spain.
The algorithm, dubbed “MosAIc”, differs from probabilistic generative adversarial network (GAN)-based projects that generate artwork since it focuses on image retrieval instead. Rather than focusing solely on obvious factors such as color and style, the algorithm also tries to uncover meaning and theme. It does this by constructing a data structure called a conditional k-nearest neighbor (KNN) tree, which provides a tree-like structure where branches off a central image indicate similarity to the image. In order to query the data structure, these branches are followed until the closest match to an image in a dataset is found. In further iterations, it prunes unpromising branches in order to improve its time for new queries.
Some results from running the algorithm against museum collections were finding similarities between the Dutch Double Face Banyan and a Chinese ceramic figurine, traced to the flow of porcelain and iconography from the Chinese to the Dutch in the 16th to 20th centuries.
A surprising result of this study was discovering that the approach could also be applied to find problems with deep nerual networks, which are used for creating deepfakes. While GANs can often have blind spots in their models, struggling to recreate certain classes of photos, MosAIc was able to overcome these shortcomings and accurately reproduce realistic images.
While the team admits that their implementation isn’t the most optimized version of KNN, their main objective was to present a broad conditioning scheme that is simple but effective for applications. Their hope is to inspire related researchers to consider multi-disciplinary applications for algorithms.
“Connections” seems like a poor choice of wording, in my opinion. “Similarities” is more accurate. They could probably stumble across actual connections, which implies intentional similarity, but the algorithm seems to merely find similarities in color and shape. Any connection would be coincidental. I guess it could narrow down the number of candidates that humans would need to compare when looking for actual connections, but it doesn’t seem revolutionary.
The deepfake angle is interesting, but also not very impressive. “Yes, those obviously terrible AI generated images are terrible.” I guess you could automate some AI training data selection, but like [i]most[/i] things AI, it seems like the only exceptional characteristic is that it’s AI.
I’m guessing that the reason these models sometimes produce terrible pictures is because the only way to tell that it’s a terrible picture is for a human to look at it and say “that looks terrible”. If you can automate the recognition that some generated images are terrible, you can build that into the training and make it be not terrible.
When you have a hammer everything is a nail. AI is a great way to automate paradigm bias.
“They were both painted on flat surfaces”
“Aha! That proves the connection..the machine says so!”
“The algorithm seems to merely find similarities in color and shape”
I hate people that comment without reading, with all due respect the article literally states:
“Rather than focusing solely on obvious factors such as color and style, the algorithm also tries to uncover meaning and theme” and then goes into more detail how they do that, you could’ve also clicked the provided link and have gotten even more detail?
But no, instead you think b*tching that AI is useless adds value to this discussion?…. good job kid.
nah, its good old leopard sofa http://rocknrollnerd.github.io/ml/2015/05/27/leopard-sofa.html
>researchers were inspired by an unlikely, yet similar pairing: Francisco de Zurbarán’s “The Martyrdom of Saint Serapion” and Jan Asselijn’s “The Threatened Swan,” two works that portray scenes of profound altruism with an eerie visual resemblance.
aka similar vertical white texture and arms up pose
Cool! wish they provided a few more examples of ‘linked together’ art though
Lucubration is the word.