Collaborative Generation with Non-Parametric Sampling

Sample collections

We introduce Collaborative Generation, a system that generates a visual stream in response to users’ interests and consumption trends. Users’ collective behaviour guides what is generated, and newly generated images become reference images for later rounds.

Non-Parametric Sampling

Our goal is to generate an evolving visual collection that reflects the user’s preferences. We frame this as a sampling problem: defining a distribution over image content and generating new images from it.

We take inspiration from the work of Efros and Leung on non-parametric texture synthesis [1]: generating new visual patterns by sampling from examples while preserving their structure.

We treat the image diffusion model as a renderer conditioned on text as input parameter [2][3]. Given a reference collection, a language model constructs a shared schema—a set of visual attributes used to describe every image in the collection. The reference set therefore implicitly defines both what may vary and what should remain consistent.

Like retrieval-augmented generation (RAG) [4][5], our method uses references to guide generation. Instead of supplying them only as context, we construct new descriptions by sampling observed attribute values from the reference collection. This explicitly constrains the distribution of text captions and helps limit hallucinated details.

The two main challenges of our approach are: defining visual attributes that jointly describe an image (analogous to a pixel in [1]), and determining how to sample new combinations while preserving dependencies between attributes (the distribution over pixels in [1]).

To define these attributes, we post-train a language model F_\theta to produce a schema \mathcal S_\theta from a reference collection. We train using reconstruction: each reference image is described through the schema, rendered back into an image, and compared with the original. Different collections thus implicitly define different visual aspects for the schema to include in more detail. Given a reference collection \mathcal R, the model F_\theta produces a schema: \mathcal S_\theta = F_\theta(\mathcal R).

Applying the schema to a reference image I_k gives its visual attributes vector S_{k,\theta} = \mathcal S_\theta(I_k). The image model then generates a reconstruction: \hat I_{k,\theta} \sim G(S_{k,\theta}).

We optimize the model parameters \theta to minimize reconstruction error:

\theta^* = \arg\min_\theta \mathbb E_{\mathcal R\sim p_{\mathrm{train}}} \left[ \frac{1}{|\mathcal R|} \sum_{I_k\in\mathcal R} \mathbb E_{\hat I_{k,\theta}} \left[ d\left(I_k,\hat I_{k,\theta}\right) \right] \right]

Here, p_{\mathrm{train}} is the distribution of training reference collections, and d measures perceptual and structural differences between a reference image and its reconstruction. After training, the model produces a schema directly from a reference collection.

To sample coherent combinations, we determine which attributes can vary independently and which should be sampled together. For example, furniture dimensions and room layout should be sampled together to preserve spatial coherence, while the color palette may allow more independent variation. We use a vision-language critic post-trained on human preference judgments to guide this choice, building on prior work on image preferences and visual feedback [6][7]. A candidate dependency structure \mathcal H specifies which attributes are sampled jointly. It defines a sampler q_{\mathcal H}, that draws attribute values uniformly from their observed instances in the reference dataset \mathcal D. We sample descriptions and render a batch of M images:

\widetilde S_j\sim q_{\mathcal H}(S\mid\mathcal D), \qquad \widetilde I_j\sim G(\widetilde S_j), \qquad j=1,\ldots,M.

The critic evaluates batches produced under different dependency structures against the same reference collection \mathcal R, assessing visual coherence, consistency with the references, and diversity.

Collaborative Generation

The process can be applied repeatedly to a growing image pool \mathcal P. At each round \tau, users’ collective behaviour guides the selection of a reference collection representing a coherent interest or trend. The references are encoded using a shared schema, and sampled attribute combinations are rendered into M new images that expand the image pool:

\mathcal P_{\tau+1} = \mathcal P_\tau \cup \left\{ \widetilde I_j^{(\tau)} \right\}_{j=1}^{M}

Generated images can become references in later rounds. As users’ interests evolve, new reference collections bring together earlier and newly generated images, expanding the range of visual combinations the system can explore. We call this evolving process Collaborative Generation.

References

  1. Alexei A. Efros and Thomas K. Leung. Texture Synthesis by Non-parametric Sampling. Proceedings of the Seventh IEEE International Conference on Computer Vision, vol. 2, pp. 1033–1038. IEEE, 1999. DOI: 10.1109/ICCV.1999.790383.
  2. Eyal Gutflaish, Eliran Kachlon, Hezi Zisman, Tal Hacham, Nimrod Sarid, Alexander Visheratin, Saar Huberman, Gal Davidi, Guy Bukchin, Kfir Goldberg, and Ron Mokady. Generating an Image From 1,000 Words: Enhancing Text-to-Image With Structured Captions. arXiv:2511.06876 [cs.CV], 2025.
  3. James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, Wesam Manassra, Prafulla Dhariwal, Casey Chu, Yunxin Jiao, and Aditya Ramesh. Improving Image Generation with Better Captions. OpenAI, 2023.
  4. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. Advances in Neural Information Processing Systems, vol. 33, 2020.
  5. Andreas Blattmann, Robin Rombach, Kaan Oktay, Jonas Müller, and Björn Ommer. Retrieval-Augmented Diffusion Models. Advances in Neural Information Processing Systems, vol. 35, 2022. DOI: 10.52202/068431-1114.
  6. Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, and Omer Levy. Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generation. Advances in Neural Information Processing Systems, vol. 36, 2023. DOI: 10.52202/075280-1594.
  7. Youwei Liang, Junfeng He, Gang Li, Peizhao Li, Arseniy Klimovskiy, Nicholas Carolan, Jiao Sun, Jordi Pont-Tuset, Sarah Young, Feng Yang, Junjie Ke, Krishnamurthy Dj Dvijotham, Katherine M. Collins, Yiwen Luo, Yang Li, Kai J Kohlhoff, Deepak Ramachandran, and Vidhya Navalpakkam. Rich Human Feedback for Text-to-Image Generation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19401–19411, 2024.