Unknown6656.CVGLPixelShader

This is a small package wich enables the modification of OpenCV or NumPy images using OpenGL pixel shaders written in GLSL. Pixel shaders are shader programs which are executed in parallel on each pixel of a given image or texture (Note: As pixel shaders are implemented as fragement shaders in the context of this library, they are technically not executed on every pixel but rather on every fragment).

You can learn more about GLSL, as well as pixel and fragment shaders here:

• https://en.wikipedia.org/wiki/OpenGL_Shading_Language
• https://en.wikipedia.org/wiki/Shader
• https://www.khronos.org/opengl/wiki/OpenGL_Shading_Language
• https://www.khronos.org/opengl/wiki/Fragment_Shader

Download / Installation

1. Install the package using pip install Unknown6656.CVGLPixelShader or download it from https://pypi.org/project/Unknown6656.CVGLPixelShader/

2. Insert the following import statements into your Python code:

   import cv2
   from Unknown6656.CVGL import *

   If you run into any issues regarding package dependencies, try installing all dependencies as listed in requirements.txt using the command pip install -r requirements.txt.

Basic Example

Create a new shader by invoking the constructor PixelShader(...) as follows:

my_shader = PixelShader('''
    vec4 color = texture(image, coords);

    out_color = vec4(1 - color.xyz, 1);
''')

The code segment passed to the constructor is GLSL code and will be inserted into the fragment shader of the program. Have a look at the official GLSL documentation pages from Khronos for more information on how to use the shader language.

The shader can then be used as follows:

image_in = cv2.imread('.../path/to/image.png')   # read the input image from file
image_out = my_shader(image_in)                  # process the image using the pixel shader
cv2.imwrite('.../path/to/output.png', image_out) # save the processed image

my_shader.close()                                # close the shader and free all
                                                 # underlying resources

You may want to replace a call to PixelShader.close(self) with the usage of with-statement blocks:

with PixelShader('''
    vec4 color = texture(image, coords);

    out_color = vec4(1 - color.xyz, 1);
''') as my_shader:
    image_in = cv2.imread('.../path/to/image.png')
    image_out = my_shader(image_in)
    cv2.imwrite('.../path/to/output.png', image_out)

All Shaders created with PixelShader(...) expose the following variables to GLSL:

• int32 width: The image width in pixel.
• int32 height: The image height in pixels.
• vec2 coords: The current pixel coordinates, normalized to [0..1]x[0..1].
• sampler2D image: The image as a 2D texture sampler with four color channels (RGBA).
• vec4 out_color: The output color at this position as an RGBA-vector with values ranging from 0 to 1 for each color channel. If this variable is unused, the input color will be copied to the output.

Using User-Defined Variables

You may sometimes want to pass variables to the shader. This can be performed as follows:

1. Create a new shader variable:

   my_variable = ShaderVariable('time', ShaderVariableType.FLOAT)

   This code creates a new variable with the name 'time' and the type float (32-bit IEE754 floating-point value). Take a look at the enum ShaderVariableType for all supported shader variable types.

2. Create a new shader and pass all variables to the shader:

   my_shader = PixelShader('''
       out_color = texture(image, coords);
       out_color.x += sin(time);
   ''', variables = [my_variable])

   Note that the GLSL code makes usage of the variable time in the line out_color.x += sin(time);.

3. Assign a value to the variable:

   my_shader[my_variable] = time.time()

4. Use the shader:

   image_out = my_shader(image_in)

Complete example

image = cv2.imread('...../image.png')
v_time = ShaderVariable('time', ShaderVariableType.FLOAT)
time = 0

with PixelShader('''
    out_color = texture(image, coords);
    out_color.x += sin(time * 2) * .5;
''', '', [v_time]) as shader:
    while True:
        time += .01
        shader[v_time] = time
        image = shader(image)

        cv2.imshow('image', image)
        if cv2.waitKey(1) & 0xff == ord('q'):
            break

Pre-defined Shaders

Feel free to make usage of some pre-defined shaders which come with this library. These shaders include:

• Vignetting
• Hue
• Saturation
• Grayscale
• Invert
• Brightness
• Contrast
• Sepia
• Bloom
• Blur (Linear, Radial, Gaussian)
• RGB Split (Linear, Radial)
• Kernel Convolution (Single- and Double-Pass)
• Embossing
• Edge Detections (Sobel, Sobel-Feldman, Prewitt, Roberts, ...)

To use any of these shaders, add the following import-statement to your script:

from Unknown6656.CVGL.Shaders import *

You may then use the shaders as follows:

image = cv2.imread('/path/to/image.png')

with Blur(radius = 20) as blur: # "Blur" is a pre-defined shader
    processed = blur(image)

cv2.imshow('input', image)
cv2.imshow('output', processed)
cv2.waitKey(0)

The pre-defined shaders allow variables to be set after initialization as well:

image = cv2.imread('/path/to/image.png')

with Blur(radius = 10) as blur:
    processed_1 = blur(image)   # process the image with radius = 10
    blur['radius'] = 20         # change the radius to 20
    processed_2 = blur(image)   # re-process image with the new blur radius

Performance

Let me be clear. The performance of this library is dogshite. Pixel shaders usually have a great performace, however they are designed to be used inside of render pipelines of games -- not headless in some random botched-up code. Hoever, I tried to compare the performance of this library with equivalent "traditional" code (i.e. using mainly OpenCV and NumPy).

The following graph compares the runtime (in seconds) of executing x-times the "vignetting" effect on a bitmap using this library and a "traditional" method:

The following graph compares the mean runtime (in seconds) averaged over 100 iterations of the following effects using this library and "traditional" implementations:

Both graphs have been created on a machine with the following hardware specifications:

LENOVO Legion 5
Intel i7-11800H @ 2.3Ghz, 8 Cores, 16 Threads
NVIDIA RTX 3060 M (6GB VRAM)
16GB RAM

One can clearly see that effects implemented with this library have a baseline runtime of 5ms. However, this runtime does not increase significantly when implementing more complex effects using GLSL (in comparison to "traditional" implementations).

Notes

Please keep the following in mind:

1. This library is still in beta. Some bugs may occur.
2. You may declare multiple shaders and use them independently from each other. Please keep in mind that the image processing is still performed synchroniously in the background, meaning that two shaders cannot process an image at exactly the same time (at least on a per-process basis).