Pixel art anti aliasing

[alex5nader]

Objective

Better anti aliasing for upscaled pixel art.

Pixel art often uses nearest-neighbor sampling when upscaling. This works well for 2D environments with an integer scale factor, but otherwise leads to aliasing artifacts.

Solution

I'm proposing an implementation of the solution described in this video: https://youtu.be/d6tp43wZqps

The video explains it well and includes graphics to better illustrate why the process works, but here's a simplified summary if you'd prefer:

Simplified explanation

1. A linear sample at a given UV can be thought of as the average color within a texel-sized box centered at that UV.
2. Ideally, pixel art would instead use a pixel-sized box at each UV.
3. It is possible to scale the pixel-sized box to texel-size while preserving its average color. This moves its center.
4. Because the average color is the same, a linear sample at the new center can provide the correct color for the original pixel.
5. If the sprite is rotated, a pixel-sized box is no longer a rectangle, so the AABB of the box is used instead.
6. The AABB is not a perfect match, but inaccuracy can be decreased by weighing the center of the box more than the edges when finding the average color.

I've implemented this in my fork here and have a sample project using it here. Because of the following questions, though, I don't think it's ready for a PR quite yet.

Unanswered questions

For the following questions, I'm not sure which approach is best. In the current implementation, I arbitrarily chose to add pbr_functions::texture_sample and use a shader define PIXEL_ART_ANTI_ALIASING. This can be easily changed depending on the discussion.

When to check if enabled

To actually implement this strategy within Bevy, all relevant instances of texture sampling must be replaced with this function. I'll call it pixel_art_texture_sample. There are a few ways to approach this:

1. Add another new function (texture_sample) that checks if this feature is enabled. Use this to replace all relevant calls to a built-in texture sample.

Pseudocode

fn texture_sample(texture, sampler, uv) {
  if enabled {
    return pixel_art_texture_sample(texture, sampler, uv);
  else {
    return textureSample(texture, sampler, uv);
  }
}

2. Every time a texture is sampled, check if this feature is enabled, either using the built-in or pixel_art_texture_sample.

Example

From pbr.wgsl :

if ((pbr_bindings::material.flags & pbr_types::STANDARD_MATERIAL_FLAGS_BASE_COLOR_TEXTURE_BIT) != 0u) {
    if enabled {
        output_color = output_color * pixel_art_texture_sample(pbr_bindings::base_color_texture, pbr_bindings::base_color_sampler, uv, view.mip_bias);
    } else {
        output_color = output_color * textureSampleBias(pbr_bindings::base_color_texture, pbr_bindings::base_color_sampler, uv, view.mip_bias);
    }
}

(Most of the builtin texture samples use a mip bias. This can be compensated for as explained here)

How to check if enabled

There are also two approaches for the enabled check itself. We can either add a new shader define, or add a new flag to the standard material's flags.

What is a "relevant texture sample"?

There's also the question of what texture samples should use the new method. Bevy's PBR shaders have many different textures, and any time one of them is sampled, we have to choose whether this new behaviour is necessary.

Obviously, base color texture should use it. Using it for the normal map and emissive textures also seems useful. The metallic/roughness and occlusion textures don't seem particularly likely to be provided with pixel art, but if they were, I think using this technique would produce more consistent results.

There are more texture samples than these in the PBR shaders, though. A grep shows the following files:

• src/environment_map/environment_map.wgsl
• src/render/parallax_mapping.wgsl
• src/render/shadow_sampling.wgsl
• src/render/shadows.wgsl
• src/ssao/gtao.wgsl

I don't really know enough about PBR to judge if these should also use this technique. But, since this technique shouldn't change the overall shape of the texture, perhaps shadows don't need to use it.

Possible issues

• Because this is implementing a sample function within the shader itself (as opposed to the builtin linear and nearest functions), shader authors have to remember this feature exists and not just use the builtin sample functions.

[dmyyy]

Relevant: #9349

[alex5nader]

I actually initially looked into both FilterMode and ImageSampler, but after looking into the WebGPU spec I'm pretty sure it's not possible to instruct the GPU to use anything other than linear or nearest (hence the need to replace calls to a built-in textureSample).

I suppose we could add a pixel art option to one of these, but then they're no longer equivalent to the wgpu types and would have to be special cased inside Bevy somehow.

[mnemotic]

Likewise relevant: #1856 (comment)

[alex5nader]

Yep, this technique solves the same issue as handmade hero and the shadertoy. It is more complicated though, to also help with 3d environments where sprites aren't just scaled by a non-integer amount, but may also be rotated away from the camera.

[mnemotic]

For what it counts, I'd love to have this in Bevy.

[dmyyy]

I want this in Bevy as well - however with 0.12 on the horizon I think people are going to be busy preparing for the new release + reviewing the mountain of in-flight rendering features.

@alex5nader I think it would make sense to ask about this in the #rendering-dev channel on discord if you don't get a response by then.

[alex5nader]

Yeah, my timing is pretty bad haha. I'm perfectly fine waiting to let people prepare for the release, but figured I'd go ahead and make my work public in the meantime. Once things have calmed down I do plan to reach out.

[agausmann]

Unanswered questions are perfectly fine for a PR! Opening a PR would be the best way to make sure your questions don't get accidentally overlooked by the project maintainers. If you don't want it merged yet, just mark as draft.

[wbpaley]

Was this ever added?

(Pardon this Bevy newbie if there's an obvious place for me to have checked instead of asking...)

Also, again pardon newbie Q, I'm bringing images into a 3D world as simple flat meshes and can scale them with the mouse wheel--but it's important to see the pixels when they're large. Seems like this might allow that?

Thank you all! (Newbie but already accomplishing a lot with my hacks... ; )

[Mattsi-Jansky]

To do this for all textures by default in Bevy 0.14 you can simply do:

App::new()
        .add_plugins(DefaultPlugins
                .set(ImagePlugin::default_nearest())

To do this in Bevy 0.14 without changing every texture in the app by default, you can set it per-texture like so:

#[derive(Resource)]
pub struct TextureHandles {
    pub my_texture: Handle<Image>,
}

fn load_material(
    mut images: ResMut<Assets<Image>>,
    texture_handles: Res<TextureHandles>,
) {
    if asset_server.load_state(texture_handles.my_texture.id()) != LoadState::Loaded {
        return;
    }

    let image = images.get_mut(&texture_handles.my_texture).unwrap();
    image.sampler = ImageSampler::Descriptor(ImageSamplerDescriptor::nearest());
}

The complication here is that this can only be done after the image is loaded, which is done in parallel. The 'handle' you get may not be in a loaded state, hence why we verify the loaded state and return after it isn't ready.

Of course, you don't want this system running on every frame so you'll want to use run conditions to ensure that it only runs when needed. I do this using a state enum that is set to Loading by default, use criteria to ensure the system only runs when when in Loading state, then after the change is applied set the state to Loaded.

[wbpaley]

Thank you both!

I ended up enabling nearest-neighbor for all images, but only for magnify, with the code below--but it's good to know how to do it only for some.

Pardon me not posting my own answer as soon as I found it, but I'm pleased to learn about the possible future 3d semi-smooth, semi-nearest since I'm operating with 2d images in a 3d world and want to show their pixels cleanly--but without those distracting jaggies--when I zoom way in. So far we're using an orthogonal camera transform so they don't seem to appear.

(Started with appreciation to #9118, then expanded the default to suit my asymmetric needs.)

Related: though I use linear as a minimize filter, very small images do get pretty noisy. I couldn't find a bicubic filter; in there one? I suppose I could do what I did in Java and downscale in a few steps (e.g., 2/3 each step until I get to the size I want), but it surely would be nice to have the GPU do something pretty, smart, and quick!

        app.add_plugins((
            DefaultPlugins.set(
                //  prevents blurring of textures only when magnifying
                ImagePlugin { default_sampler: ImageSamplerDescriptor {
                    mag_filter: ImageFilterMode::Nearest,
                    min_filter: ImageFilterMode::Linear,
                    mipmap_filter: ImageFilterMode::Linear,
                    ..Default::default()
                }},
            ),
        ));

[alex5nader]

I looked briefly and it seems like most graphics APIs only support nearest-neighbor and bilinear filtering. Bicubic would be possible to implement similarly to the proposed feature here, where an alternate texture sampling function would be implemented within the shader. I don't think this has been done yet in Bevy.

[wbpaley]

Yes, nice--and what I hoped for: in the shader.

We don't need it any time soon, so maybe we can piggy-back on your work and contribute this when we're both freer to do fine polishing.

Best of luck finishing your degree!

[agausmann]

Nearest-neighbor will make pixel art render properly in 2d

Not quite - You can have a similar issue in 2D where the sprite stays flat, but might be scaled at small, fractional scale factors. (Sometimes you can avoid this and use integer scale factors for pixel-perfect rendering, but not always, like for zoom animations.)

For example, if you have a scale factor of 4.5 - if you use nearest neighbor, about half of the texels would be 4 pixels wide, and the other half would be 5 pixels, and as the sprite moves, the size of each texel can noticeably jitter as their pixel alignment changes.

[Mattsi-Jansky]

Ah, sorry, my reply may have been somewhat missing the point. Honestly I just wanted to leave a helper here for future visitors, because this thread is one of the top Google results when searching for ways to disable texture anti-aliasing in 3d assets in Bevy.