Search engine

CMU researchers present a content-based search engine for Modelverse, a model-sharing platform that contains a diverse set of deep generative models


The objective of content-based pattern search is introduced, which attempts to locate the most relevant deep-image generation patterns that meet a user’s input query. As shown in the graph below, a user may receive a pattern based on their ability to synthesize images that match an image query (eg, landscape photo), text query (eg, African animals), a sketch request (for example, a drawing of a standing cat), or a likeness to a provided template request. But why is a model search based on helpful content? Deep generative models are being developed to serve as the basis for content creation software and applications. They are no longer just the results of scientific studies.

The search method (1st line) allows searches with four distinct modalities – text, photos, existing drawings and models – from left to right. The first two models are displayed in the second and third rows. The color of each template icon indicates the template type. In all modalities, the technique discovers applicable models with comparable semantic notions. | Source:

Each model represents a miniature world of carefully chosen themes, which can include realistic renderings of people and landscapes, images of ancient pottery, cartoon caricatures, and aesthetic aspects of a single artist. More recently, various techniques have made it possible to creatively modify and customize existing models, whether through human-in-the-loop interfaces or by refining GANs and text-image models. Each generative model could signify the model creator’s meaningful engagement with a particular concept. It is becoming more and more impossible for a user to know every fascinating generative model, although it can be vital to select the best model for their particular need.

The ability to quickly synthesize an unlimited set of images, interpolations, or latent variable manipulations is provided by each generative model. Yet researchers have found that selecting the ideal model from a large collection can produce significantly better results than those obtained by choosing an inappropriate model. Pattern search allows users to locate a pattern that best suits their unique needs, just as information and image retrieval allows users to find the appropriate information in large collections of traditional documents. The challenge of content-based pattern finding is difficult; even the simple question of whether a single model can generate a particular image can be computationally demanding.

Unfortunately, many deep generative models do not provide an efficient or accurate method of density estimation nor do they natively measure cross-modal (e.g., text and image) similarity. A naive Monte Carlo technique can compare the input query against tens or millions of samples of each generative model and select the model whose elements most frequently match the input query. Pattern searching would be extended with such a sampling-based technique. They first provide a generic probabilistic formulation of the pattern-finding problem to solve the problems mentioned above, followed by a Monte Carlo baseline. To save time and space, they “compress” the model distribution into pre-computed 1st and 2nd order moments of the deep feature embeddings of the original samples.

Then they construct closed-form solutions for pattern retrieval using an input image, text, sketch, or pattern query. Real-time calculations can be made of their ultimate formula. On 133 deep generative models, including GANs (eg, StyleGAN family models), diffusion models (eg, DDPM), and autoregressive models, they evaluate their methods and perform ablation tests (for example, VQGAN). Their approach offers a significantly faster search (in 0.08 milliseconds, a 5x speedup) while maintaining good accuracy compared to the Monte Carlo baseline. Finally, they show how to use GAN inversion and pattern fine-tuning to a few strokes as pattern-finding applications.

Their solution, to their knowledge, is the first content-based search algorithm for machine learning models. The research method is deployed on Modelverse, an online platform for scholars, students, and artists to effectively use and share generative models, available at

This Article is written as a research summary article by Marktechpost Staff based on the research paper 'Content-Based Search for Deep Generative Models'. All Credit For This Research Goes To Researchers on This Project. Check out the paper, github link, Modelverse and project.

Please Don't Forget To Join Our ML Subreddit and Youtube Channel

Consultant intern in content writing at Marktechpost.