Smoothing the Generative Latent Space with Mixupbased Distance Learning
Abstract
Producing diverse and realistic images with generative models such as GANs typically requires large scale training with vast amount of images. GANs trained with extremely limited data can easily overfit to few training samples and display undesirable properties like ”stairlike” latent space where transitions in latent space suffer from discontinuity, occasionally yielding abrupt changes in outputs. In this work, we consider the situation where neither large scale dataset of our interest nor transferable source dataset is available, and seek to train existing generative models with minimal overfitting and mode collapse. We propose latent mixupbased distance regularization on the feature space of both a generator and the counterpart discriminator that encourages the two players to reason not only about the scarce observed data points but the relative distances in the feature space they reside. Qualitative and quantitative evaluation on diverse datasets demonstrates that our method is generally applicable to existing models to enhance both fidelity and diversity under the constraint of limited data. Code will be made public.
1 Introduction
Remarkable features of Generative Adversarial Networks (GANs) such as impressive sample quality and smooth latent space interpolation have drawn enormous attention from the community, but what we have enjoyed with little gratitude claim their worth in a datalimited regime. As naive training of GANs with small datasets often fails both in terms of fidelity and diversity, many have proposed novel approaches specifically designed for fewshot image synthesis. Among the most successful are those adapting a pretrained source generator to the target domain [mo2020freeze, ojha2021few, li2020few] and those seeking generalization to unseen categories through feature fusion [gu2021lofgan, hong2020matchinggan]. Despite their impressive synthesis quality, these approaches are often critically constrained in practice as they all require semantically related large source domain datasets to pretrain on. For some domains like abstract art paintings, medical images of rare symptoms and cartoon illustrations, it is very difficult or impossible to collect thousands of samples, while at the same time, finding an adequate source domain to transfer from is not straightforward either. This poses intimidating challenge for generative modeling, but we seek to find solutions in the darkest of times.
One of the biggest challenges of learning generative models with scarce data is that the model easily overfits. To directly address this issue, [zhao2020differentiable] and [karras2020training] have proposed data augmentation techniques that show promising results on lowshot generation tasks with datasets containing hundreds to thousands of training samples. Nevertheless, they show unsatisfactory performance with handful of data points (e.g., ) and generative modeling under these circumstances still remains extremely challenging.
Inspired by [ojha2021few] that proposes novel distance regularization to effectively transfer diversity information from the source to target, we recast the overfitting issue as stairlike latent space problem and suggest Mixupbased Distance Learning (MDL) for both the generator and the discriminator to effectively control it.
As diversity is already heavily constrained by the small dataset, we wish to maximally exploit the given data points by continuously exploring their semantic mixups [zhang2017mixup]. With overfitting, however, the discriminator is convinced only with the few observed samples, showing overly confident and abrupt decision boundaries. This induces the generator to map a subset of its latent space to a single mode, displaying stairlike transitions in the latent space. Hence, to smooth the feature space, we intentionally explore the generator latent space with continuous interpolation coefficient , enforcing relative semantic distances between samples to follow the mixup ratio. Simultaneously, by penalizing the discriminator, we prohibit the discriminator from embedding images to arbitrary locations for its convenience of memorizing, and guide its feature space to be aligned with semantic distances.
We further observe that models trained with our regularizations resist mode collapse surprisingly well with neither large source domain datasets nor special augmentation. We believe that our distance regularizations encourage the model to preserve inherent diversity present in early stages throughout the course of training, opening up the doors for sample diversity under rigorous constraints.
In sum, our contributions can be summarized as:

We propose a twosided distance regularization that encourages learning of smooth and modepreserved latent space through controlled latent interpolation.

We introduce a simple framework for fewshot image generation without a large source domain dataset that is compatible with existing architectures and augmentation techniques.

We evaluate our approach on a wide range of datasets and demonstrate its effectiveness in generating diverse samples with convincing quality.
2 Related Works
Oneshot image generation The most challenging case of fewshot image generation is the oneshot case where given a single image, the generator must learn its context to create diverse outcomes. SinGAN [shaham2019singan] approximates the distribution of a single image by differing its scale during the adversarial training. Training from the downscaled image introduces ambiguity to the generator and this induces diversity when the generator comes to learn in high resolution. Based on SinGAN, ConSinGAN [hinz2021improved] proposes a technique to control the tradeoff between fidelity and diversity of generated samples. OneShot GAN [sushko2021one] uses a dualbranch discriminator where each head respectively identifies real context and real layout of the generated sample. As oneshot image generation methods focus on exploiting a single image, they are not directly applicable to fewshot image generation tasks where the generator must learn the underlying distribution of a collection of images.
Lowshot image generation Given a limited amount of training data, the discriminator in conventional GAN can easily overfit. To mitigate this problem, DiffAugment [zhao2020differentiable] imposes differentiable data augmentation to both real and fake samples while ADA [karras2020training] devises nonleaking adaptive discriminator augmentation. As generating high resolution images is more challenging with limited data and budget, FastGAN [liu2020towards] suggests a skiplayer excitation module and a selfsupervised discriminator, which saves computational cost and stabilizes lowshot training. GenCo [cui2021genco] shows impressive results on lowshot image generation task by using multiple discriminators to alleviate overfitting. Despite their promising performances on lowshot benchmarks, these methods often show significant instability under stricter data constraint, namely in fewshot setting.
Transferbased fewshot generation Thus far, the fewshot image generation task where the dataset size is even more limited () mostly required training on larger dataset with similar semantics [wang2018transferring, wang2020minegan] mainly due to its inherent difficulty. A group of works [gu2021lofgan, hong2020matchinggan] learns transferable generation ability on seen categories and seek generalization into unseen categories through fusionbased methods. FreezeD [mo2020freeze] and EWC [li2020few] introduce transfer learning frameworks for GANs that selectively update the target model’s parameters to mitigate overfitting. Meanwhile, CDC [ojha2021few] computes the similarities between samples within each domain and encourages the corresponding similarity distributions to resemble each other. It aims to directly transfer the structural diversity of the source domain to the target, yielding impressive performance. In this paper, we leverage the formulation of this work and modify for the situation where no source domain is available.
Generative diversity Mode collapse has been a long standing obstacle in GAN training. [arjovsky2017wasserstein, mao2017least] introduce divergence metrics that are effective at stabilizing GAN training while [durugkar2016generative, ghosh2018multi] tackle this problem by training multiple networks. Another group of works [liu2019normalized, mao2019mode, tran2018dist, yang2019diversity] proposes regularization methods to preserve distances in the generated output space. Unlike these works, we consider the fewshot setting where the phenomenon has a far devastating impact, and introduce an interpolationbased distance regularization method as an effective remedy.
Latent mixup Since [zhang2017mixup], mixupbased methods have been actively explored in semisupervised learning literature to enforce smooth behaviors in between training samples [berthelot2019mixmatch, verma2021interpolation, berthelot2019remixmatch]. In generative models, [radford2015unsupervised] emphasizes the importance of smooth latent transition as a counterevidence for memorization, but as stateoftheart GAN models trained with sufficient data naturally possess such property [karras2020analyzing, brock2018large], it has been mainly studied with autoencoders. [berthelot2018understanding, oring2020autoencoder] regularize autoencoders to learn smooth latent space while [wertheimer2020augmentation, sainburg2018generative] explore their potential as generative models through interpolation.
3 Approach
We consider the situation where only few train examples (e.g., ) are available with no semantically similar source domain. Under harsh data constraint, overfitting greatly restricts a model’s ability to learn data distribution and produce diverse samples. We identify its byproduct unsmooth latent space as the core obstacle, as it not only indicates memorizing but also prohibits hallucination through semantic mixup. We observe that both the generator and the discriminator suffer from the problem with insufficient data, evidenced by discontinuous latent interpolation and overly confident decision boundary, respectively.
To this end, we propose mixupbased distance learning (MDL) framework that guides the two players to form soft latent space and leverage it to generate diverse samples. We further discover that our proposed regularizers effectively combat mode collapse, a problem particularly more devastating with a small dataset, by preserving diversity present in early training stages. As our formulation is inspired by [ojha2021few], we first introduce their approach in Sec. 3.1, and formally state our methods in Sec. 3.2 and Sec. 3.3. Our final learning framework and the corresponding details can be found in Sec. 3.4.
3.1 CrossDomain Correspondence
In [ojha2021few], the authors propose to transfer the relationship learned in a source domain to a target domain. They define a probability distribution from pairwise similarities of generated samples in both domains and bind the latter to the former. Formally, they define distributions as
(1)  
(2) 
where is the generator activation at the layer and are latent vectors. Note that and correspond to source and target domain generator, respectively, and , are way discrete probability distributions consisting of pairwise similarities. Then, along with adversarial objective , they impose a KLdivergencebased regularization of the following form:
(3) 
The benefits of this auxiliary objective are twofold: it prevents distance collapse in the target domain through distribution binding and transfers diversity from the source to target via onetoone correspondence.
3.2 Generator Latent Mixup
In [ojha2021few], the anchor point could be chosen arbitrarily from the prior distribution since they were transferring the rich structural diversity of the source domain to the target latent space. As this is no longer applicable in our setting, we propose to resort to diverse combinations of given samples. Hence, preserving the modes and learning interpolable latent space are our two main desiderata. To this end, we define our anchor point using Dirichlet distribution as follows:
(4) 
where . Using Eq. 4, the latent space can be navigated in a quantitatively controlled manner. Defining probability distribution of pairwise similarities as in [ojha2021few], we bind it to the interpolation coefficients instead. The proposed distance loss is defined as follows:
(5)  
(6)  
(7) 
where denotes the Dirichlet distribution with all1 parameters. This efficiently accomplishes our two desiderata. Intuitively, unlike naive generators that gradually converge to few modes, our regularization forces the generated samples to differ from each other by a controlled amount, making mode collapse very difficult. At the same time, we constantly explore our latent space with continuous coefficient vector , explicitly enforcing smooth latent interpolation. An anchor point similar to [ojha2021few] can be obtained with onehot coefficients .
3.3 Discriminator Feature Space Alignment
While the generator distance regularization alone can alleviate mode collapse and stairlike latent space problem surprisingly well, the root cause of constrained diversity still remains unresolved, i.e., discriminator overfitting. As long as the discriminator delivers overconfident gradient signals to the generator based on few examples it observes, generator outputs will be strongly pulled towards the small set of observed data. To encourage the discriminator to provide smooth signals to the generator based on reasoning about continuous semantic distances rather than simply memorizing the data points, we impose similar distance regularization on its feature space. Formally, we define our discriminator where refers to the final FC layer that outputs {real, fake}. When a set of generated samples and the interpolated sample is provided to , we construct an way distribution similar to Eq. 6 as
(8) 
where refers to a linear projection layer widely used in selfsupervised learning literature [chen2020simple, chen2021exploring, grill2020bootstrap] and . Without the linear projector, we found the constraint too rigid that it harms overall output quality. We define our distance regularization for the discriminator as
(9) 
This regularization penalizes the discriminator for storing memorized real samples in arbitrary locations in the feature space and encourages the space to be aligned with relative semantic distances. Thus it makes memorization harder while guiding discriminator to provide smoother and more semantically meaningful signals to the generator.
3.4 Final Objective
Fig. 2 shows an overall concept of our method. Our final objective for the generator and the discriminator takes the form:
(10) 
(11) 
where we generally set and .
As our method is largely independent of model architectures, we apply our method to two existing models, StyleGAN2^{1}^{1}1https://github.com/rosinality/stylegan2pytorch[karras2020analyzing] and FastGAN[liu2020towards]. We keep their objective functions as they are and simply add our regularization terms. For StyleGAN2, we interpolate in rather than , which has been shown to have better properties such as disentanglement [wang2021high, zhu2020improved, alaluf2021restyle]. Interpolation coefficients is sampled from a Dirichlet distribution of parameters all equal to one. Patchlevel discrimination [isola2017image, ojha2021few] is applied for interpolated images to encourage our generator to be creative while exploring the latent space.
4 Experiments
We validate our method on diverse domains, ranging from photorealistic images to handdrawn illustrations. We further analyze the effect of individual components, i.e., model architecture, data augmentation and proposed regularizations, through ablation on different datasets, demonstrating that our approach is compatible with existing model architectures and augmentation techniques.
Baselines We mainly apply our method to the stateoftheart unconditional GAN model, StyleGAN2 [karras2020analyzing]. Data augmentation techniques introduced by [zhao2020differentiable] and [karras2020training] show promising performance on lowshot image generation task, so we evaluate them along with ours and refer to them as DiffAug and ADA respectively. As we observed significant performance drops for DiffAug when applied to normal StyleGAN2 with 8layer mapping network, we set the number of FC layers to 2 only for DiffAug as done by the authors while leaving the base architecture untouched for others. We additionally apply our method to FastGAN [liu2020towards], which is a lightweight GAN architecture based on novel SkipLayer channelwise Excitation module (SLEmodule) that allows faster convergence with limited data.
Datasets For quantitative evaluation, we use AnimalFace Dog [si2011learning], Oxfordflowers [nilsback2006visual], FFHQbabies [karras2019style], face sketches [wang2008face], 100shot Obama and Grumpy Cat [zhao2020differentiable], anime face [liu2020towards] and Pokemon (pokemon.com, [liu2020towards]). Aforementioned datasets contain 100 to 8189 samples, so we simulate fewshot setting by randomly sampling 10 images, if not stated otherwise. For qualitative evaluation, we further experiment on face paintings of Amedeo Modigliani [yaniv2019face] and landscape drawings [ojha2021few]. All experiments are done on images. Tab. 1 summarizes the datasets and the number of shots used in each dataset. Generation results on other datasets can be found in the supplementary materials.
missingmissing 







10  10, 100  10, 100, 1000  10  10, 100  






10, 100  10  10  10  10  
missingmissing 
Evaluation Metrics One of the difficulties in fewshot generative modeling lies in the subtlety of evaluation, as the most widely used evaluation metric, Fréchet Inception Distance (FID) [heusel2017gans], does not accurately represent generation quality [li2020few] with small datasets. We measure FID for datasets containing a sufficient number () of samples along with pairwise Learned Perceptual Image Patch Similarity (LPIPS) [zhang2018unreasonable] as a measure of diversity. For simulated fewshot tasks, FID score is computed against the full dataset, as in [li2020few, ojha2021few]. We further use LPIPS as a distance metric for demonstrating interpolation smoothness and mode preservation.
missingmissing 

Method  Anime Face  AnimalFace Dog  Oxford Flowers  Face Sketches  Pokemon  
FID ()  LPIPS ()  FID ()  LPIPS ()  FID ()  LPIPS ()  FID ()  LPIPS ()  FID ()  LPIPS ()  
FastGAN  123.71  0.3410 0.0040  103.04  0.6327 0.0042  182.72  0.6669 0.0025  76.28  0.1476 0.0062  123.53  0.5780 0.0092 
StyleGAN2  166.03  0.3627 0.0085  177.54  0.5688 0.0051  177.33  0.5371 0.0031  94.16  0.4347 0.0021  257.63  0.4391 0.0051 
StyleGAN2 + DiffAug  162.02  0.2035 0.0014  136.09  0.5593 0.0073  186.97  0.6870 0.0049  43.15  0.4380 0.0087  280.12  0.1794 0.0016 
StyleGAN2 + ADA  130.24  0.2879 0.0045  236.53  0.6355 0.0058  167.76  0.7188 0.0086  62.82  0.3994 0.0028  214.28  0.4962 0.0041 
FastGAN + Ours  107.61  0.4783 0.0057  99.84  0.6251 0.0058  180.53  0.6568 0.0089  45.03  0.4159 0.0045  144.00  0.5839 0.0061 
StyleGAN2 + Ours  73.16  0.5484 0.0045  97.82  0.6812 0.0053  136.56  0.7342 0.0085  40.02  0.4788 0.0057  117.05  0.5393 0.0119 
StyleGAN2 + DiffAug + Ours  70.16  0.5507 0.0030  96.37  0.6819 0.0055  129.94  0.7053 0.0052  35.59  0.4705 0.0049  114.33  0.6070 0.0092 
StyleGAN2 + ADA + Ours  75.00  0.5709 0.0050  94.14  0.6843 0.0029  127.70  0.7627 0.0077  39.24  0.4821 0.0034  155.48  0.5440 0.0080 
missingmissing 
4.1 Qualitative Result
Fig. 3 shows generated samples of different methods trained with 10 real samples. We observe that baseline methods either collapse to few modes or severely overfit to the training data, resulting in inability to generate diverse novel samples. Ours is the only method that produces a variety of convincing samples that are not present in the training set. We can see that our method combines visual attributes such as hairstyle, beard and glasses in a natural way, producing distinctive samples under harsh data constraint.
The difference is more distinguished when we take a closer look. In Fig. 4 we display “uncurated” sets of images generated from models trained with 10shot paintings of Amedeo Modigliani that share a common training sample as nearest neighbor. Outputs of DiffAug show little differences among them, but our method generates unique samples with recognizable visual features. We believe this is because our distance regularization enforces outputs from different latent vectors to differ from each other, proportionally to the relative distances in the latent space. More generated samples from Amedeo Modigliani paintings can be found in the supplementary materials.
4.2 Quantitative Evaluation
Tab. 2 shows FID and LPIPS scores for 10shot image generation task. We can see that our method consistently outperforms the baselines, often with significant margins. Moreover, our regularizations can be applied concurrently to data augmentations to obtain further performance gains. Note that while StyleGAN2 armed with advanced data augmentations fails to converge from time to time, our method guarantees stable convergence to a better optimum across all datasets.
While pretrainingfree 10shot image synthesis task has not been studied much, several works [liu2020towards, zhao2020differentiable] have previously explored generative modeling with as little as 100 samples. We present quantitative evaluations on popular lowshot benchmarks in Tab. 3. We observe that our method consistently improves the baseline, and the margin is larger for more challenging tasks, i.e., dataset with greater diversity or fewer training samples. We discuss experiments on these benchmarks in depth in Sec. 5.
missingmissing 

Dataset  Obama  Grumpy Cat  Oxford Flowers  Obama  Grumpy Cat 
Shot  100  100  100  10  10 
LPIPS  0.6152  0.6132  0.7945  0.5978  0.5982 
StyleGAN2  63.05  43.34  192.15  174.67  76.35 
+ Ours  58.38  26.56  81.98  62.70  41.05 
+ DiffAug  46.87  27.08  91.56  66.81  45.64 
+ DiffAug + Ours  45.37  26.52  63.96  57.88  39.28 
4.3 Ablation Study
We further evaluate the effects of the two proposed regularizations, MDLG (for generator) and MDLD (for discriminator), through ablation under different settings. In Tab. 4, we observe that in general, our regularizations both contribute to better image quality and diversity, while in some special cases, only adding MDLG leads to better FID score. We conjecture that aligning discriminator’s penultimate output vectors with the interpolation coefficients can impose overly strict constraint for some datasets. We nonetheless observe consistent improvements on diversity.
Tab. 5 shows the performance of our method under a larger data setting. Since FFHQbabies contains more than 2,000 images, we randomly sample subsets of size 10, 100 and 1,000. We can see that the performance of StyleGAN2 steadily improves with more training samples, but our method can consistently push the curve upward. Hence, we believe that with limited data in general, our method can be broadly used to improve model performance. Lastly in Tab. 6, the effect of using different Dirichlet concentration parameters for mixup is illustrated. We find that setting yields the best performance, so we uniformly use this throughout the experiments.
missingmissing 

Base Method 

Dog (10shot)  Babies (100shot)  Flowers (100shot)  
FID ()  LPIPS ()  FID ()  LPIPS ()  FID ()  LPIPS ()  
StyleGAN2  177.54  0.5688  130.99  0.5744  192.15  0.7472  
✓  95.36  0.6731  71.70  0.6381  84.03  0.7801  
✓  ✓  97.82  0.6812  63.35  0.6468  81.98  0.7823  
missingmissing 
missingmissing 

Method  Metric  shot  
10  100  1000  
StyleGAN2  FID ()  184.77  130.99  44.42 
LPIPS ()  0.4756  0.5744  0.6300  
StyleGAN2 + Ours  FID ()  93.15  63.35  38.05 
LPIPS ()  0.6230  0.6468  0.6685  
missingmissing 
missingmissing 

Metric  
FID ()  76.35  73.16  80.83 
LPIPS ()  0.5359  0.5484  0.5321 
4.4 Latent Space Smoothness
Smooth latent space interpolation is an important property of generative models that disproves overfitting and allows synthesis of novel data samples. As our proposed method focuses on diversity through latent smoothing, we quantitatively evaluate this using a variant of Perceptual Path Length (PPL) proposed by [karras2019style].
PPL was originally introduced as a measure of latent space disentanglement under the assumption that a more disentangled latent space would show smoother interpolation behavior [karras2019style]. As we wish to directly quantify latent space smoothness, we slightly modify the metric by taking 10 subintervals between any two latent vectors and measure their perceptual distances. Tab. 7 reports the subinterval mean and standard deviation, and the mean for the full interval. Note that as PPL is a quadratic measure, the sum of subinterval means can be smaller than the endpoint mean. We first notice that StyleGAN2+DiffAug suffers severe mode collapse as apparent from Fig. 5, so the low mean and standard deviation values simply indicate training failures. StyleGAN2 with no augmentation, StyleGAN2+ADA and ours show similar endpoint mean, suggesting that the overall total perceptual distance is consistent, while ours displays the lowest mean PPL and standard deviation. As low PPL variance across subintervals is a direct sign of perceptually uniform latent transitions, we can verify the effectiveness of our method in smoothing the latent space. Similar insight can be found from Fig. 5 where the baselines either collapse or display stairlike latent transition while ours shows smooth semantic interpolation. More details on PPL computation can be found in the supplementary materials.
4.5 Preserving Diversity
missingmissing 

Method  Mean  Std. Dev.  Endpoint Mean 
StyleGAN2  21.910  12.657  60.901 
StyleGAN2 + ADA  25.577  9.198  63.558 
StyleGAN2 + DiffAug  0.820  0.095  22.083 
StyleGAN2 + Ours  12.818  4.187  64.280 
As opposed to [ojha2021few] that preserves diversity in the source domain, our method can be interpreted as preserving the diversity inherently present in the early stages throughout the course of training, by constantly exploring the latent space and enforcing relative similarity/difference between samples. To validate our hypothesis, we keep track of pairwise LPIPS of generated samples and the number of modes in the early iterations. Fig. 6 shows the result, where the number of modes is represented by the number of unique training samples (real images) that are the nearest neighbor to any of the generated images. In Fig. 6, we can see that vanilla StyleGAN2 and our method show similar LPIPS in the beginning, but the baseline quickly loses diversity as opposed to ours that maintain relatively high level of diversity throughout the training. Fig. 6 delivers similar implication that FastGAN trained with our method better preserves modes, thus diversity, compared to the baseline.
Combined with latent space smoothness explained in Sec. 4.4, generators equipped with MDL learn rich modepreserving latent space with smooth interpolable landscape. This naturally allows generative diversity particularly appreciated under the constraint of extremely limited data.
5 Discussion
The tradeoff between fidelity and diversity in GANs has been noted by many [brock2018large, karras2019style]. Truncation trick, a technique widely used in generative models, essentially denotes that diversity can be traded for fidelity. In fewshot generation task, it is very straightforward to obtain nearperfect fidelity at the expense of diversity as one can simply overfit the model, while generating diverse unseen data points is very challenging. This implies that with only a handful of data, the diversity should be credited no less than the fidelity.
However, we believe that the widely used lowshot benchmarks, e.g., 100shot Obama and Grumpy Cat, inherently favor faithful reconstruction over audacious exploration. The main limitations we find in these datasets are twofold: (i) the intradiversity is too limited as they contain photos of a single person or object, evidenced by low LPIPS in Tab. 3 and (ii) FID is computed based on the 100 samples that were used for training. We acknowledge that (ii) is a common practice in generative models, but the problem with these benchmarks is that the number of samples is too limited, making it possible for some models to simply memorize a large portion of them. These two combined results in benchmarks that allow relatively easy replication and reward it generously at the same time. In other words, we believe that a model’s capacity to explore continuous image manifold and be creative can potentially backfire in these benchmarks.
To address these limitations, in Tab. 3 we extend the benchmark with three additional datasets: 100shot Oxfordflowers, 10shot Obama and Grumpy Cat. The first one challenges the model with greater diversity while the last two evaluate its capacity to learn distribution in a generalizable manner, as the FID is still computed against the full 100 images. As our method mainly aims for modeling diversity, we observe marginal performance gains in the traditional benchmarks. However on the extended benchmarks, our proposed method shows significant contributions, confirming that it excels at learning diversity even under challenging situations.
6 Conclusion
Image generation with a handful of data samples has been deemed extremely challenging due to overfitting and mode collapse, thus it was mainly tackled with transfer learning based methods. However, these approaches share an inherent limitation that a semantically similar source domain dataset should be available. To overcome this limitation and broaden the potential applications for GAN models, we propose mixupbased distance regularizations that effectively alleviate mode collapse and smooth the otherwise stairlike latent space of generative models. We empirically demonstrate that our method is capable of generating diverse unseen samples of convincing quality after training on as little as 10 real samples, with no pretraining or special data augmentation. In short, our method can be directly added on top of an existing GAN model, e.g., StyleGAN2, to accommodate it for fewshot image synthesis. We hope our work facilitates future research on data efficient generative modeling, which we believe has great upside in both academics and applications.
References
A Implementation Details
StyleGAN2 We adopt the standard StyleGAN2 architecture^{2}^{2}2https://github.com/rosinality/stylegan2pytorch for resolution images, with 8 fully connected layers in the mapping network. We keep the hyperparameters such as the learning rate, regularization weights and frequency, untouched, and only add our proposed MDL.
DiffAug We essentially follow the official configuration^{3}^{3}3https://github.com/mithanlab/dataefficientgans for lowshot generation, including the twolayer mapping network and three data augmentation methods. We have also tried with a standard 8 FC layer mapping network and observed significant drops in the overall performance as shown in Tab. S1.
missingmissing 

FC layers  Obama (100shot)  Grumpy Cat (100shot) 
2  46.87  26.52 
8  71.13  38.42 
FastGAN We use the official FastGAN implementation^{4}^{4}4https://github.com/odegeasslbc/FastGANpytorch for images. As FastGAN doesn’t have a separate mapping network, we interpolate in space.
MDL For MDL, we alternate between the normal adversarial training step and the interpolation/regularization step. In the former we go through normal imagelevel discrimination and in the latter, we apply patchlevel discrimination on the mixup samples and compute losses for MDLG and MDLD. For patch discrimination, we largely adopt the implementation of Crossdomain Correspondence (CDC)^{5}^{5}5https://github.com/utkarshojha/fewshotganadaptation. Our linear projection layer for the discriminator operates on 512 dimension.
Percpetual Path Length For PPL computation, we mainly follow the implementation in StyleGAN. The difference is that we subdivide a latent interpolation path into 10 subintervals and compute the perceptual distance for each line segment. Since the original PPL computation divides the perceptual distance by the squared step size, we divide each subinterval length by . For clear demonstration, we divide the endpoint mean by as well. Note that the overall procedure is equivalent to calculating LPIPS multiplied by the factor of 100. The standard deviation is computed across the subintervals, and averaged for the interpolation paths.
Number of Modes We generate 500 samples and compute their perceptual distances to the 10 training samples. We record the index for the real sample with the smallest perceptual distance and report the unique count. It is visually apparent from Fig. S1 that our method helps the model preserve modes and maintain diversity.
B Training Snapshots
We provide training snapshots for FastGAN and StyleGAN2 for visual demonstration of diversity and interpolation smoothness. Fig. S1 clearly shows that as opposed to vanilla FastGAN that rapidly loses diversity and converges to few prototypes, MDL successfully alleviates this. Fig. S2 displays interpolation snapshots for StyleGAN2. In early training iterations, it does show relatively smooth latent transition, but the sample quality is very unsatisfactory. As the training proceeds, the sample quality improves as the model overfits, but consequently the interpolation smoothness is quickly lost. This describes the classic dilemma in fewshot generative modeling. In contrast, Fig. S3 shows that as MDL is effective at maintaining latent space smoothness, it provides a sweet spot where reasonable sample quality and smooth latent transition coexist. Note that models with MDL do inevitably overfit in the end, but we can find reasonable stopping point that produces diverse unseen samples with satisfactory visual quality.
C Additional Generated Samples
We present synthesis results from face paintings of Amedeo Modigliani, abstract paintings of Paul Klee^{6}^{6}6https://www.kaggle.com/robgonsalves/abstractpaintings/version/1, CC BY 4.0 and illustrations of Japanese animation character Totoro in Fig. S4, Fig. S5 and Fig. S6, respectively. Images of Paul Klee and Totoro were crawled from the web, and we used only 5 real samples for the latter. Additional interpolation examples are also displayed in Fig. S7 and Fig. S8.
D Sample Images from Lowshot Benchmarks
In Fig. S9, we present samples from Obama and Grumpy Cat datasets. As they contain images of a single character, the intradiversity is inherently very limited.
E Naive Application of GAN adaptation
We display results from naive application of CDC. Since it is very difficult to find a semantically similar source domain for datasets like Pokemon and abstract paintings of Paul Klee, we naively leverage the source generator trained on FFHQ. As the source and the target are semantically different, the adaptation does not yield satisfactory outcomes as expected. We can observe the dilemma here as well that in the early iterations, the face shape learned in the source domain is clearly visible while in later stages, the face shape is no longer visible but the model collapses altogether. As CDC preserves distances in the target domain through the correspondence to the source domain, it is not applicable to domains that lack an adequate source dataset to transfer from.