Classifying Books by Genre Based on Cover

A book cover can convey a lot about the content of the book. Despite the adage to not evaluate something based on outward appearances, we apply machine learning to see if we can, in fact, judge a book by its cover, or more specifically by its cover art and text. The classification was done considering three different aspects - cover image only, cover text only and both image and text in a multimodal approach. Image classification was done using transfer learning with Inception-v3. For text detection from the cover image, images were first converted to greyscale and different thresholds were applied to detect maximum text. This text was then vectorized and used to train a Multinomial Naïve Bayes model. We also trained custom CNNs for image and text modalities. For multimodal classification, we examine late fusion model, where the modalities are combined at decision level, and early fusion model, where the modalities are combined at the feature level. Our results show that the late fusion model performs best in our setting. We also observe that text is more informative with respect to genre prediction and that significant efforts need to be devoted to solve this image-based classific