"Computer vision is a subfield of artificial intelligence (AI) that equips machines with the ability to process, analyze and interpret visual inputs such as images and videos" -IBM
"Computer vision enables machines to interpret, analyze, and pull meaningful data from images and videos. This field of AI uses deep learning and neural networks to recognize objects, people, and patterns with high degrees of accuracy. In other words, it replicates human sight and the cognitive ability to interpret visual data." -Microsoft
"Computer vision tasks include methods for acquiring, processing, analyzing, and understanding digital images, and extraction of high-dimensional data from the real world in order to produce numerical or symbolic information, e.g. in the form of decisions" -Wikipedia
Computer Vision is how we process pictures/video into something computers can understand, how computer handle that information, and how we extract useful conclusions from those pictures/videos.
Open source library that implements most computer vision tools we will need in both python and C++
uv init
uv add opencv-python
Library for fast optimized python math. Useful for computer vision and working with OpenCV
uv add numpy
Library of models for object detection we will use later.
uv add ultralytics
Body landmark detection model we will use later. Get it from googles website or the install script.
Computer vision is used in the medical industry to automate and improve the accuracy of medial imaging. It is often used to find things such as
The Agriculture industry makes use of Computer Vision in many ways.
A boring but necessary function. Computer Vision is the backbone for document digitization and document OCR. Converting paper documents into digital text.
Judges love computer vision projects, and they usually are not that complicated to set up once you know what tools you have at your disposal.
As we will see later, computer vision can be used to track where objects, including people, are in a room! This has been the backbone of many of our most beloved companies. We can track things such as
Computer Vision's ability to track things also makes it vital in the defense industry, powering things like
Before we can extract meaning from an image, the computer needs to understand what is happening in the image. Because computers cannot perceive an image, we need to process the raw pixels into an easier to work with form.
To accomplish this, we need to do image preprocessing, normalizing aspects of the image into a more workable form.
First we need to understand how a computer see's an image. At its core, an image is just a large array of numbers, representing the location and content of each pixel.
Images are stored in an array of (H, W, Chan_Vals)
Using code like this we can see what one of these arrays might look like
import cv2
#Read the image (Open CV loads images in B,G,R color format)
image_bgr = cv2.imread("cat.jpg")
print(f"{image_bgr}")
[[[202 196 183]
[202 196 183]
[202 196 183]
...
[186 180 167]
[186 180 167]
[186 180 167]]
We can also look at other types of image data using OpenCV, such as the size or datatype being stored in the image
import cv2
#Read the image (Open CV loads images in B,G,R color format)
image_bgr = cv2.imread("cat.jpg")
print(f"{image_bgr.shape}")
print(f"{image_bgr.dtype}")
(1000, 944, 3)
uint8
Red Color Mask
Red Mask Pixel: [ 0 0 183]
Blue Color Mask
Blue Mask Pixel: [202 0 0]
Green Color Mask
Green Mask Pixel: [0 196 0]
Emphasized Reds
Emphasized Blues
Emphasized Greens
Many machine learning models used in Computer Vision are also trained off of a specific size of image. Processing differently sized images can cause a loss of quality or a unwanted clipping of the images content. Many models clip to some square image size such as 256x256 or 640x640.
Smaller images can also be much quicker to process. If perfect quality is not needed reducing the amount of data will speed up processing. a 500*500 image is 250 thousand pixels to process while a 2000*2000 is 4 million.
Images that are not the correct size have to be either clipped, expanded, or resized into the correct format. There are many ways to do this but some of the most common include interpolation, and stretching.
Total Pixels: 944000
Total Pixels: 61696
Total Pixels: 3840
When we resize an image, the computer has to generate new pixels to fit the new aspect ratio. This is done through interpolation which estimates values in a known range. There are a few ways to handle interpolation demonstrated below
Blocky but Fast
Smooth but lost quality
Slow but smooth
Depending on the Computer Vision task, another preprocessing step we can take is the complete removal of the color channels. By passing each pixels color values through a function we can convert each pixel to a different brightness shade of gray. While this does lose data, it also shrinks the image size and can be useful for certain tasks that do not rely on color and only use brightness, such as edge detection.
Using OpenCV's cvtColor function we can quickly change the entire image
#Convert the Image to a grayscale
#Use BGR to Gray to match imread
gray_image = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
Images with lots of noise can be slightly blurred to remove that noise.
Instead of storing pixels on a scale of 0-255, we can normalize to a floating point number between -1 and 1. Helps models converge quicker.
Another step of image resizing. Adds a neutral color that is not later processed. Pads the smaller dimension out till image is a perfect square for better processing.
A feature is a piece of information in an image that implies a point of interest. Features help classical CV tools detect objects in images without knowing what an image is.
Classical processing largely focuses on finding and describing features in an image. This is typically accomplished using kernels to analyze the contents of an image.
A Kernel is a matrix of weights (typically 3x3) that slides over an image. At each pixel you sum pixel value * weight and assign that value to the center pixel. This allows you to describe changes in the image over a set of pixels.
Matrix with a Gaussian Distribution, can blur over an image reducing noise. Used to reduce noise and useful for other detection.
[1, 2, 1]
[2, 4, 2]
[1, 2, 1]
Values differences on either side of the kernel. If all values under the kernel are similar, final value trends to 0.
[1 ,1 ,1]
[0, 0, 0]
[-1 ,-1 ,-1]
Enhances pixel intensity and visibly defines objects more. Emphasis value of the center pixel.
[0, -1, 0]
[-1, 5, -1]
[0, -1, 0]
Below is the result of running Sobel edge detection on our example Cat. Notice how Cat and radiator borders start to blend in and the Cats fur borders are highlighted. Also shown is the same process after a box blur has been applied to the image.
While edge detection is one of the most commonly used types of classical computer vision processing, other models do exist that serve different purposes.
Grayscale mask over entire image, convert values below threshold to black and above to white. Useful for pulling subjects out or removing backgrounds for OCR.
Early form of object recognition that uses significant differences in pixel values to split images into blobs of similar objects.
Classic computer vision processing methods were revolutionary and had many years of fine-tuning. Processes such as HOG + SVM allowed for relatively complex detection of things like Cars and Pedestrians, but required careful fine-tuning. Many of these classical processes had to be fine-tuned by hand to work on a small subset of objects, and would break with new data.
Edge detection did a good job at telling us where something was in an image but could not tell us what the object was. This weakness was solved with...
Earlier approaches had to have pre-defined kernels that were fine tuned by hand to accurately detect where objects were. Machine Learning changed that by moving away from a single purpose kernel to a series of learned kernels working in tandem. This new model was dubbed Convolutional Neural Networks (CNN)
To produce a computer vision model we start with random kernel weights and a labeled dataset. The model runs its kernels over the picture -> Measures the error (the loss) -> Adjusts the kernels to be closer -> Repeats. Do that enough and you get accurate kernels that understand what they are detecting.
To improve accuracy, models began implementing multiple levels of convolution using differently weighted kernels. These feature maps each attempt to detect a specific feature and build off of each other into more abstract concepts. While a early feature may still only detect edges and corners, future layers may use that information to detect a body, and then a face and eyes off of the body.
This stacked feature hierarchy can be expanded on to add more functionality to these models. Once a model can tell that a body is in the picture, it can also attempt to track that bodies movement by generating and tracking landmarks on the body.
YOLO is a very popular family of object detection models provided by Ultralytics.
Google's MediaPipe is another extremely popular family of models that handle different types of body tracking.