This week
The project kickoff
Before the coding period began I had previously been working on classification of the Christian icons in the painting. But in order to caption art, we needed a good source. So I had been collecting sources here.
In order to get some idea on how to proceed with the sources collected I contacted my mentor Fred.
Minutes of the meeting.
Attendees - Fred, Marcelo, Rishab, Tiago
-
I gave a brief idea about how I plan to proceed with this project in two phases. First would be the curation and subsequently the creation of a multimodal pipeline.
-
For curation, I proposed to build a simplistic module which will enable us to automatically add images(and descriptions) of paintings to a database(could be GDrive as well). Once the images are added it should automatically be assigned to a directory denoting a saint. The advantages of this would be that we could use it for both classification and tagging.
-
Since classification is one of the goals of Parth’s project I also proposed that we could possibly work together
-
Tiago asked me what are some of the challenges I could foresee. My answer was oriented mostly towards curation since I don’t possess any information on this topic.
-
The next topic in discussion was related to the types of images we would be using. Whether it would be just painting or also sculptures or what period it would be from. Marcelo suggested that it would be better to stick to paintings as vision algorithms apply relatively easier. Apart from that we decided to stick to Renaissance paintings and then expand to other strokes/styles.
-
Since I already had begun researching sources. I shared that in the meeting. While the sources are enough, we need to check the license details if we were to make the dataset public.
-
The last minutes were related to the captions of the paintings. Given a source, what I need to check is whether these images are just mere descriptions or contain richer representations.
-
An example of the prior was discussed by Tiago. The image has so many elements in it in every corner!
License.
One of the problems foreseen by Marcelo was that if our dataset was to be made public then we would have to make sure that the sources we took images from allowed us to actually use their images for research or commercial purposes. Hence I spent some time figuring out whether the sources I had collected were usable or not. Apart from that I also compiled a list of musuems that provide open access to their images. I will be updating as I find more.
| Source | Link | Captions | Filtering | Size | License |
|---|---|---|---|---|---|
| Art Institute of Chicago | https://www.artic.edu/collection?q=saint | Yes | Required | 50k | Creative Commons Zero (CC0) |
| Belvedere, Austria | https://sammlung.belvedere.at/opencontent/images | Yes(German) | Required | Unknown | Creative Commons |
| J. Paul Getty Museum, Los Angeles | https://www.getty.edu/art/collection/search?open_content=true&q=Saint | Yes | Required | Unknown | Unrestricted and commercial use |
| Metropolitan Museum of Art, New York | https://www.metmuseum.org/art/collection | Yes | Required | 375k | Open Access |
| Minneapolis Institute of Art | https://collections.artsmia.org/info/open-access | Yes | Required | 50k | CC PDM |
| National Gallery of Art, Washington, D.C. | https://www.nga.gov/open-access-images.html | Yes | Required | 50k | Open Access |
| Smithsonian Institution, Washington, D.C. | https://www.si.edu/openaccess/ | Yes | Required | 4.4M | Open Access |
| Yale University | https://artgallery.yale.edu/collection/search/saints | Yes | Required | 250k | Open Access |
Inspecting sources.
One of the points discussed in the meeting was that paintings normally have a lot of elements in them so while collecting sources we should also check whether the description correctly describes the elements of the painting.
Let’s understand this with an example.
This is a random image taken from the Cleveland Musuem of Art that provides open access to 60k images.
If i would look at this image, normally I would say that a mother is holding a baby.
That is normal so why make a painting? Are the agents in the painting someone special? These are the types of questions that our description should answer.
Now if we see the description we observe that the description gives us more insights about the painting. It tells us about the expressions of the agents and who the agents are. It also describes the technique used and it’s similarity to another painting. Hence the description feeds us with a richer understanding of the painting that our naive mind wouldn’t comprehend by just looking at it.
More on this can be found here
The Raw pipeline.
1. Data Curation
2. Tagging
Next week
Finalizing probably on what sources to take images from and what period.