Identifying potential associations between proteins and compounds is significant and challenging in the drug discovery process. Existing deep-learning-based methods tend to treat compounds and proteins as sequences or graphs. Inspired by the rapid development of computer vision technology, we argue that more abundant characterizations can be extracted from the images of compounds than from their sequences or graphs. Therefore, we propose an interaction model named picture-word order compound protein interaction (PWO-CPI) which learns the representation from structural images of compounds and protein sequences by using convolutional neural network (CNN). The experiments show that PWO-CPI outperforms state-of-the-art CPI prediction models. We also perform drug–drug interaction (DDI) experiments to validate the strong potential of structural formula images of molecular structures as molecular features. In addition, with the aid of generative adversarial networks, the visualization of image features demonstrates PWO-CPI can learn compound structural features implicitly and automatically.