TS-GAN: Image Generation based on Text and Sketch with Generative Adversarial Networks

Abstract

There have been many studies to generate images based on text description and sketch image with Generative Adversarial Network. However, since previous studies generate images with only one single input such as text description or sketch image, there is a limitation that an unwanted image is generated when text description is not enough or the sketched image is very different from the real image. In this thesis, i propose TS-GAN, a new technique to overcome the limitations of previous studies by using sketch image resources and text description resources together. TS-GAN consists of two main steps. In the first step, it generates the main shape of the object and background at low resolution based on the sketch image. And then, the second step is to generate a more realistic image at high resolution based on the image generated from the first step and text description. Through various experiments with CUB and Oxford-102 datasets which are widely used in computer vision, we show that TS-GAN generates images very well by using text description and image sketch.