A. AI Methodology
-Generator Two generators work in the same way, take
Generator X as an example. Generator X consists of 3 parts:
encoder, transformer and decoder.
-Encoder: For simplicity, the input has been fixed as
[256,256,3]. By using convolution network, we first extract
features. A convolution network takes an image as input, then
uses filters in the image to extract features each time.
-Transformer: This part is to transform feature vectors of
image with domain X to domain Y . For this, we have used
6 layers of resnet blocks. Resnet block can be summarized as
follows, which contains a direct channel from the input to the
output and two convolution layers.
-Decoder: Decode and encode is opposite twill rebuild an
image from to each other and and image was build by using
feature vectors gained before, and this is done by applying
three de-convolution layers which use reversed parameters of
encoding step.
-Discriminator: Discriminator also needs to extract features, so it is basically a convolution network. So the next
work is to decide whether they are matching with specific area
or not. For 1D output, a final convolution layer has added.
-Loss: To meet our goal, the loss function must satisfy:
Discriminator X should be trained because it is necessary for
Fig. 3. Discriminator
Fig. 4. Full objective of CycleGAN
images from domain X close to one. So, discriminator X minimize as (DiscriminatorX(x)−1)2. Also, since discriminator
X differentiate between generated and original images, it predict zero for the images that generates by generator. Discriminator X minimize the (DiscriminatorX(GeneratorY −→
!X(y)))2. Moreover, generator X should gradually sure
the discriminator Y for authenticity of its output images.
Because the recommended discriminator Y of output images is near to 1. Thats why, generator X minimize the
(DiscriminatorY (GeneratorX!Y (x)) −→ 1)2. And the
most important cycle loss captures previous image with the
usage of different generator and the difference of images
should be as small as possible.
B. Interior Decoration Methodology
We use UE4 software. The interior decoration design system
has rich functions. Users could use this system to draw the
house type with any size and any structure. The user also could
design the house with rich furniture. The user could place the
furniture, move the furniture, and rotate the furniture. The user
could also add the light to the room.
• Material replacement of the floor wallpaper is achieved.
• Furniture items are selected from the library for placement and can be rotated and scaled.
• User could move in each room.
• User could learn the top view of the overall browsing
unit.
• Save and read design.
• User could edit the type of house by drawing units and
adding doors or windows.
• Increase the animation of furniture.
• Increase the function of analog lighting.
http://www.aleeshainstitute.com/interior-designing-course.php
In this part, we will introduce the AI module for our system.
A. Motivation
During the process of implementation, the most headscratching part comes from the texture and maps. We learned
to transform some 3D’s max models into UE4, there were
always some incompatibility errors. And as a result, the
transformed model only contains the shape information with
gray colors which means all the texture and maps were lost.
We have found that most of the models downloaded from
the internet used the V-ray material. Although V-ray is very
powerful and realistic, unfortunately, UE4 does not support
this kind of rendering method yet. Afterwards, we tried to
convert the V-ray material into the normal material. The texture
information is kind of retained, but there were still some
color deviation or other strange problems. Besides, sometimes
when we import models into UE4, we have to attach those
textures or maps to models manually, which is also very
time-consuming. Even if we convert models successfully, the
model’s appearance is fixed. If you want to change models
color or texture, you need to either prepare many variegated
models or make other textures by yourself. Both ways are very
time consuming and space-consuming.
B. Proposed Method
Here comes the idea of making textures with the help of AI,
specifically (GAN). Image transformation is a very popular
topic nowadays, especially with the help of neural networks.
Most of the image translation is about one-to-one mapping,
which in other words, used pairwise images to train the model.
However, such pairwise images are not always available. And
most of the time, we are only provided with groups of pictures
of a similar feature. CycleGANs goal is to implement an
unpaired image translation. This translation learns the mapping
G : X!Y (X, Y two groups of images of X,Y source and target
domain) such that in ideal case, the generated image G(X)
has the same distribution of images in Y. And in this way, the
generated image has some features of images in Y.
The basic idea of implementation is to use adversarial
networks. The generator network attempts to generate images
from the desired distribution, also the discriminator network
needs to distinguish whether an image is authentic from
the target domain or a not genuine generated from source
domain. And by competing with each other, finally we will
Fig. 6. Pipeline of CycleGAN
Fig. 7. Horse to Zebra
get the desired generator. Since there is huge measurements,
for mapping a network in the input images are able to
satisfy such distribution, in this way we can hardly get the
desired output images. And there are occasions when the
model collapses, and inputs maps the output images and the
optimization process fails for the making of progress. As a
result, another constraint needs to be added which is the cycleconsistency. To transform image from source distribution to
target and then vice versa, The samples has been received
by source distribution. And with the help of this, successful
transformation can be achieved. We have used CycleGAN (a
varietal version of GAN) to train some funny models. In brief,
CycleGAN takes two datasets with different styles as input,
and it will learn the transformation pattern between them.
For example, by providing a horse image set and a zebra
image set, CycleGAN will learn to transform a horse in an
image into a zebra without changing anything else, or the
other way around. The advantage of CycleGAN lies in that
you do not need to provide pairwise images to train which
means the horse dataset can contain any kind of casual horses
and the zebra dataset can contain any kind of casual zebras
and they do not have to match each other. This advantage
of CycleGAN is very suitable for texture transform because
there are textures with different UV maps and textures with
different light features or metallically which makes it difficult
to find pairwise datasets. Thus, with the help of CycleGAN,
we can prepare a standard dataset with gray colors textures
and different target datasets, such as wood material dataset or
grass material dataset. Then, we can use CycleGAN to train
different conversion models by feeding a different combination
of datasets and import these models into UE4 afterwards. The
easiest version of texture transform is just the transformation
of maps. The computational detail and the network structure is shown below:
• Domain: X, Y .
• 2 maps G : X!Y &&F : Y !X.
-Adversary Loss: An adversarial looses has been applied
together with mapping functions G : X!Y with discriminator
DY , the adversarial loss is:
ζGAN (G; DY ; X; Y ) = EY ∼ P data(y)[logDY (y)] + Ex ∼
Pdata(x)[log(1 − DY (x))]
G generates images like G(x) which is similar to images of
domain Y, and DY differentiate among the G(x) translated
images and the samples of y. G reduce the adversary D which
tries to increase it. The same adversarial loss is introduced F :
Y !X and DX for the mapping function and for discriminator
ewspectively. -Cycle Consistency Loss: Adversarial loss will
not ensure that the learned function maps input separately xi
with the final output yi. For mapping functions, Every images
x from domain X should be consistent and translation cycle
of image x backward to the original image, i.e.
x −→ G(x) −→ F(G(x)) ≈ x
Similarly
y −→ F(y) −→ G(F(y)) ≈ y
We could incentive cyclic consistency loss:
ζ−cyclic(G; F) = Ey ∼ P data(x)[ F(G(x))−x 1]+Ey ∼
Pdata(y)[ G(F(y)) − y 1].
-Full Objective:
ζ(G; F, DX, DY )
= ζGAN (G, DY , X, Y ) + ζGAN (F, DX,Y,X) + λζcyc(G, F)
where λ control 2 objectives that solved
G∗; F∗ = argmin(G, F)max(DX, DY )
mathcalL(G, F, DX, DY ).
C. What We Have Done
We have explored the possibility of combining tensor flow
and CycleGAN with UE4. Meanwhile, we have implemented
some pre trained models in UE4 to show the amazing effect GAN just as shown below. In detail, we have hung
a painting in our scene as a decoration. By selecting it and
press the F key, our program will automatically apply a
transformation model on it. After the transformation work
is done, the painting on the wall will be replaced by the
regenerated one.
Comments
Post a Comment