Splashery
Playground

Chapter 5Pictures, screens, and your own input

The Splashes in Chapter 2 are built from shapes. Some Splashes show something that is not a shape: a page of a PDF, a photo, a GIF, a video, or something you type or draw. Three additions to a recipe make them: picture sheets, an input panel, and a few view settings. They are still Gaussian splats, all the way down, but for the sharp page faces of Splash Book, Photo Album, and Picture Lab, which are textured meshes over splat paper. The Splashes on the Pictures and Pages shelf are being tried out, so they are on the shelf only when Splashery is opened with ?labs=1.

Picture sheets: a page, a photo, a GIF, a video

A sheet is a flat place in a Splash where a picture shows: k.sheet({ id, center, width, height }). The kit draws nothing there. The site opens the picture before the build, then turns it into splats on the sheet after the build, in a background worker so the page stays smooth. A PDF page becomes a smooth sheet of large soft splats in the paper's color, with a small splat of its own color wherever the page differs from the paper, which is the ink (method: "ink"); in a photo, each sampled pixel becomes a flat splat ("pixels"); a GIF or a video becomes a screen that is redrawn on every frame ("screen"). The site picks the method from the kind of file. It rebuilds a sheet finer as you zoom in, and frees the pages you have left, so a book of 200 pages does not need 200 pages of splats. Splash Book, Photo Album, and Picture Lab draw a page's face as a textured mesh in place of its splats.

A page of an article, drawn with splats, with a sharp figure and a table

A PDF page (the Picture Lab)

A photo of a tulip field in a wooden frame, built from the recipe below

The recipe below: a photo in a frame

Both pictures are made of splats. Photo: "Tulip field" by DennisM2, CC0. The article is our own sample text.

Three parts of a recipe work together:

A whole picture Splash, in a few lines. Its input gives the Splash sheet a button that opens a photo, a GIF, or a video; the rest of the Splash is a card and four wooden bars: 

import { smoothstep } from "../src/kit.js";

// Two small picture toys for Chapter 5 of the Splash Manual: a framed photo
// that opens your own picture, and a little book whose page turns like paper.
export const RECIPES = {
  "little-frame": {
    turntable: false,
    tiltLock: true, // a drag only spins it left and right
    pictures: {
      sample: () => "assets/toys/picture-lab/photo.jpg", // shown until you open your own
      accept: ["image", "gif", "video"],
    },
    input: {
      title: "Your own picture",
      media: { accept: ["image", "gif", "video"] },
      note: "Open a photo, a GIF or a video. Files stay on this device.",
    },
    build(k) {
      // The picture: a sheet, filled with splats after the build.
      k.sheet({ id: "photo", center: [0, 0, 0.03], width: 1.5, height: 1 });
      // A gray card behind it, and four wooden bars around it.
      k.add(k.box(1.6, 1.1, 0.02), { pos: [0, 0, -0.02], color: "#9a9a9a" });
      for (const [w, h, x, y] of [
        [1.8, 0.1, 0, 0.6],
        [1.8, 0.1, 0, -0.6],
        [0.1, 1.1, -0.85, 0],
        [0.1, 1.1, 0.85, 0],
      ])
        k.add(k.box(w, h, 0.1), { pos: [x, y, 0], color: "#8a5a34" });
    },
  },

Turning a page

A page that turns like paper is a sheet with leaf: slot (a slot from 0 to 9) and a spine to turn about: k.spine({ at, axis, dir }) gives the spine's place, its direction, and the direction from it along the pages when they lie flat. Each frame, drive sets out.leaves[slot] = { angle, curl }: the angle in radians (0 lying flat, π turned all the way over) and the curl, how much the page bends along a circular arc (a negative number lets the free edge lag behind, like paper in air). Every splat of the page turns with the curve at its place, so it stays a solid sheet at every angle, which a page bent with soft regions would not.

A page of a book swinging over from the right, curled at its free edge, with the next page under it
The little book below, halfway through a turn. The next page is already there under the turning one.

The little book below uses a pulse control for the turn, the way Chapter 2's gust does. The page under the turning one is built ahead of time (a sheet is normally built only while it is visible, unless it requests ahead: 1, so a page that shows up halfway could make the turn wait), and the page number moves on when the turn ends. Keep a turning sheet on the body (part 0), because a part's own turn would replace the leaf's. The far side of the turning page shows the page in mirror image, like tracing paper, until it lands. 

  "little-book": {
    turntable: false,
    tiltLock: true,
    pictures: { sample: () => "assets/toys/picture-lab/article.pdf", accept: ["pdf"] },
    input: { title: "Your own book", media: { accept: ["pdf"] }, note: "Open a PDF." },
    controls: [{ key: "turn", label: "Turn the page", type: "pulse", ease: 1.2 }],
    action: { key: "turn", label: "Turn the page" },
    drive(t, c, out, info) {
      const pics = info.data?.pictures; // { page, count, go(n), next(), prev(), ... }
      // The page changes when the turn ends, and the turning page lands on the left.
      const turning = c.turn > 0;
      if (BOOK.turning && !turning && pics) pics.go(pics.page + 1);
      BOOK.turning = turning;
      const page = pics?.page ?? 0;
      // The pulse falls from 1 to 0: the page swings from 0 to a half turn (pi)
      // about the spine, and its free edge lags behind, like paper.
      const s = turning ? smoothstep(0, 1, 1 - c.turn) : 0;
      out.leaves = [{ angle: Math.PI * s, curl: -1.2 * Math.sin(Math.PI * s) }];
      out.sheets = {
        // Once the turn starts, the left page is the one that is about to land on it.
        left: { page: turning ? page : page - 1 }, // a page out of range shows nothing
        under: { page: page + 1 }, // built ahead, and uncovered as the page lifts
        front: { page }, // the turning page (its far side shows in mirror image)
      };
      // It is built, but hidden, until the turning page lands on it.
      out.parts.leftPage = { visible: turning ? 0 : 1 };
    },
    build(k) {
      BOOK.turning = false;
      k.spine({ at: [0, 0, 0], axis: [0, 1, 0], dir: [1, 0, 0] });
      const W = 1.4;
      const H = 1.9;
      const leftPage = k.part("leftPage");
      const page = { width: W, height: H };
      k.sheet({ id: "left", ...page, part: leftPage, center: [-W / 2, 0, 0], align: [1, 0] });
      k.sheet({ id: "under", ...page, center: [W / 2, 0, 0], align: [-1, 0] });
      // The turning page: a sheet on leaf slot 0, a hair in front of the others.
      k.sheet({ id: "front", ...page, center: [W / 2, 0, 0.008], align: [-1, 0], leaf: 0 });
      // The cover, behind the pages.
      k.add(k.box(2 * W + 0.1, H + 0.1, 0.03), { pos: [0, 0, -0.03], color: "#7a2f2f" });
    },
  },
};

// Whether the page was turning in the last frame.
const BOOK = { turning: false };

Pop out, the page view, and Draw a box

Three buttons in the stage's column belong to the picture Splashes. Pop out (on Picture Lab, Splash Book, the Storybook, and Photo Album) raises a tapped figure or photo off its page into a card that stands in front of it, and a tap puts it back. On a closed book, a figure raised off the cover shows in front of the cover, not behind it.

The page view button (on Splash Book, the Storybook, and Photo Album) switches between one page and both pages framing the open spread. The choice is kept in the Splash's link. Draw a box (on Picture Lab, Splash Book, and Photo Album) raises whatever you draw it round off the page, the way Pop out raises a figure: hold B, or use the button, and drag. While a figure is raised or a box is drawn, the tilt lock shows off, because both need the tilt free, and it eases back to the tilt it left when they are put away. Using Splashery shows where the buttons sit.

Figure to come: a photo rising out of the album with Pop out, in three frames.

Your own input: typing, drawing, opening a file

A recipe can ask for something from the visitor. Its input adds a panel to the Splash sheet (tap Splash in the bottom bar), in one of three forms:

Picture Lab's input in the Splash sheet: Open a file…, a field for a web address with Open, the file on show (article.pdf, page 1 of 2), Previous and Next, and Words on this page
The input panel of a picture Splash, in the Splash sheet. Files are read on your device; nothing is uploaded.

For typing and drawing, read(text, fileName) turns what came in into option values, or throws an Error whose message the panel shows in place; and shown() says what is showing now. Keep what goes into options small: options are saved in the Splash's link. Here is a whole Splash with a typed input, a row of bars for the numbers you give it: 

// A small toy with its own typed input, for Chapter 5 of the Splash Manual:
// type some numbers and the toy stands them up as bars.
const LIMIT = 12;
const COLORS = ["#e8384f", "#ffc233", "#1fb59a", "#4dabf7"];

export const RECIPES = {
  "little-bars": {
    options: [
      {
        key: "numbers",
        label: "Your numbers",
        type: "text",
        default: "3 1 4 1 5 9 2 6",
        hidden: true,
      },
    ],
    input: {
      title: "Your own numbers",
      placeholder: "3 1 4 1 5 9 2 6",
      button: "Show them",
      fileButton: "Open a text file…",
      accept: ".txt,.csv",
      note: `Type up to ${LIMIT} numbers from 0 to 20, separated by spaces or commas, or open a text file with them.`,
      // Turns the typed text into option values, or throws a message that the panel shows.
      async read(text) {
        const list = String(text).match(/-?\d+(\.\d+)?/g) || [];
        if (list.length === 0) throw new Error("Type some numbers, like 3 1 4 1 5.");
        if (list.length > LIMIT) throw new Error(`That is more than ${LIMIT} numbers.`);
        if (list.some((n) => n < 0 || n > 20)) throw new Error("Use numbers from 0 to 20.");
        return { numbers: list.join(" ") };
      },
      shown: () => "Your numbers",
    },
    build(k, o) {
      const values = String(o.numbers).split(" ").map(Number);
      values.forEach((v, i) => {
        const h = 0.05 + v / 10;
        k.add(k.box(0.14, h, 0.14), {
          pos: [(i - (values.length - 1) / 2) * 0.2, h / 2, 0],
          color: COLORS[i % COLORS.length],
        });
      });
    },
  },
};

The Screen and the Gaussian Splatting Splash

Two more Splashes put these to work: the Screen on the Pictures and Pages shelf, and Gaussian splatting on the AI and Computing shelf.

The Screen shows a video or a GIF on a screen in one of four styles: an old TV, a flat TV, a cinema with curtains, and a hologram. Its picture is one sheet with method: "screen"; the knobs, the curtains, and the glass are ordinary kit parts. A tap switches it on, and then plays and pauses; the Sound button in the stage's column turns the sound on. You can open your own video or GIF.

Each style opens whole, at a view of its own: the old TV 4.55 of its radii away, the flat TV 3, the cinema 3.45, and the hologram 6 (4.6, 4.3, 5.1, and 7.7 on a screen taller than wide). In the cinema, a tap on the picture plays and pauses the video, and a tap on the curtains, on the valance above them, or on the cinema's button in the Splash sheet opens and closes the curtains. The Screen's own sounds (the switching, the curtains, the click of play and pause) sit 6 LU (loudness units, about a decibel each) under its video, and the video plays with sound whenever Sound is on, from the first tap.

Figure to come: the Screen's four styles, each at its opening view.

The Gaussian Splatting Splash lets you watch how a photo becomes splats. Its Training view fits 2,400 flat (two-dimensional) Gaussians to a photo (a strawberry, or your own), by gradient descent in a background worker: the splats start as random specks and, step by step, they slide, stretch, and change color until together they are the picture. The fit uses front-to-back blending, equivalent to the renderer's back-to-front blending, with the renderer's finite bell, so what you see is the real fit. One splat lets you set the sizes, the opacity, and the color of a Gaussian illustrated by a cloud of small splats and turn it (Chapter 1's numbers); Many splats builds a small rubber duck from kit shapes and then shrinks every splat to a dot, so you see the splats that make it; and Sorting shows the splats of a ball appearing one by one, in the back-to-front order a camera draws them. You choose among the four in the View list of the Splash sheet.

A brown old television showing a frame of a video with a rabbit and a butterfly

The Screen (old TV)

A strawberry drawn by splats that are still learning it, with the photo and a falling loss curve below

Gaussian Splatting, training

Video frame from "Big Buck Bunny" © Blender Foundation, CC BY 3.0. Strawberry photo by Joselodos, CC0.
One large red splat drawn as a cloud, with three colored axes showing its sizes
The One splat view: a Gaussian illustrated by small splats, with its three sizes drawn as axes (Chapter 1's s1, s2, and s3).

Steering the view: the stage's buttons and the tilt lock

Three of the round buttons in the column at the stage's right edge control how you look at the Splash on show, and none of them changes the Splash. After 5 seconds untouched they fade with the other controls; touch the strip at the top or bottom of the screen to bring them back. Using Splashery shows the whole column and the bars around the stage.

The stage's column of buttons on a phone, numbered top to bottom: 1 Mute the sound, 2 How to play, 3 Full screen, 4 Layers, 5 Reset, 6 Turntable, 7 Detail and splats, 8 Hands-on, 9 Tilt lock, and 10 Record, with the fold button beside Record
The stage's buttons on a phone, each named by its own label. Reset, Turntable, and Tilt lock change how you look at the Splash. On a phone a fold button beside Record puts the column away.

Two fingers on a touch screen zoom. They roll the Splash only after a clear twist (about 17 degrees) and turn it only when they move together, so a normal pinch never spins what you are looking at. A two-finger drag moves the Splash across the screen instead of turning it, as do Shift or Option/Alt and a drag on a computer; Reset puts it back in the middle. A double-tap does too unless the Splash uses it for page focus.