Styleflow concise reading: use continuous flow to complete attribute editing

The paper ：StyleFlow: Attribute-conditioned Exploration of StyleGAN-Generated Images using Conditional Continuous Normalizing Flows

• ACM TOG 2021

code The paper

Preface ： This paper is a little cumbersome and obscure , The following content only represents personal reading comprehension

Abstract

Abstract: Now you can go through uncondition gan（ for example Stylegan） Generate high quality 、 diversification 、 Realistic images , However, there are limited attribute options to control the generation process 、 At the same time, it is difficult to guarantee the quality of output , In addition, due to gan latent space Entanglement property of , Editing along one attribute can easily lead to tampering with other attributes . Under the background of conditional exploration of entangled latent space , Two sub problems of attribute conditional sampling and attribute control editing are studied .

As GAN An example of conditional continuous normalized flow in latent space , We express conditional exploration as a conditional continuous normalized flow determined by attribute characteristics , So that StyleFlow As a simple of these two sub problems 、 Effective and robust solutions . We use StyleGAN The potential space of human face and car to evaluate our method , And in real photos and StyleGAN The resulting image shows fine-grained separation editing along various attributes . for example , For faces , We change the pose of the camera , Light variation , expression , Facial hair , Gender and age . Last , Through extensive qualitative and quantitative comparison , We proved that StyleFlow Advantages over other concurrent work .

The author has carried out the following two tasks ：

（1） Attribute conditional sampling （ That is, sampling high-quality real images with target attributes ）

（2） Controllable attribute editing （ That is, only edit the target attribute , It's best to keep other information of the source image ）

among , On mission 1 in , The generated image adopts StyleGan generator , In order to calculate the attributes of the image , An attribute classifier is used

Model method

The author wrote this article in detail , Introduced many contents , But it's not easy to read .

In the fourth part , Explained NORMALIZING FLOWS, In fact, this is also the core content of this method , About Flow Introduction to the stream , But for people who know each other for the first time （ For example, I ） It's very dizzy to read , Next, we will introduce the model method of this article in a simple way .

because StyleGan The generation process of is ：

• First, a random vector conforming to Gaussian distribution is generated by random seeds z
• take z after mapping network Generate decoupled vectors w
• And then w Send in systhesis network Generate pictures

The author introduces two FLOW The way of flow ：DNF、CNF, Used to calculate the conversion between data distributions , Finally, the author chose to use CNF.（ There is no introduction here , Interested can read the paper ）

It can be understood as ：1. adopt CNF modular , Can be w Vector combination attributes The vector passes through reverse inference（ Reverse reasoning ） obtain z vector 2. adopt CNF modular , Can also be z Vector combination attributes The vector passes through forward inference（ Forward reasoning ） obtain w vector

CNF The structure of the module is shown in the figure below ：

The following term StyleFlow Model of ：

（1）Joint Reverse Encoding（JRE）： First get a w vector （ According to real images Project to get ）, utilize generator Generate pictures , Then send the picture to the classifier （ The author uses Microsoft face attribute classification api And light prediction network DPR） obtain attributes vector , The following will w and at Vector feed CNF Network get z vector

（2）Conditional Forward Editing（CFE）： edit attributes, Then combine it with z Vector feed CNF Network get w’ vector .（ Editing vectors here should be similar interfacegan The trained direction vector ）

（3）Edit Specific Subset Selection： The author found that , take w’ Load vectors into w+ Different layers of can bring good results . for example ,1. Change lighting ： Put in 7~11 layer 2. Change the age ： Put in 4~7 Layer, etc. .

StyleFlow Generate effect evaluation - FID score

About training

StyleFlow The main training is CNF blocks, This part recommends reading papers and projects issues Author's reply .

Edit the generation diagram

Finally, let's show StyleFlow Edit rendering of method ：

Epilogue

StyleFlow My thesis is very detailed , I recommend you to have a look ~