Computer Vision Logo

Computer Vision

How to make your computer see things
Presented By: Logan
Computer Vision Logo

What is Computer Vision?

What Is Computer Vision Logo

HEADING

What is Computer Vision

"Computer vision is a subfield of artificial intelligence (AI) that equips machines with the ability to process, analyze and interpret visual inputs such as images and videos" -IBM

"Computer vision enables machines to interpret, analyze, and pull meaningful data from images and videos. This field of AI uses deep learning and neural networks to recognize objects, people, and patterns with high degrees of accuracy. In other words, it replicates human sight and the cognitive ability to interpret visual data." -Microsoft

"Computer vision tasks include methods for acquiring, processing, analyzing, and understanding digital images, and extraction of high-dimensional data from the real world in order to produce numerical or symbolic information, e.g. in the form of decisions" -Wikipedia

Normal People Description

Computer Vision is how we process pictures/video into something computers can understand, how computer handle that information, and how we extract useful conclusions from those pictures/videos.

What Is Computer Vision Logo

Tools we Will Need

Open CV

Open source library that implements most computer vision tools we will need in both python and C++

uv init
uv add opencv-python
        

NumPy

Library for fast optimized python math. Useful for computer vision and working with OpenCV

uv add numpy
        

Ultralytics

Library of models for object detection we will use later.

uv add ultralytics
        

Media Pipe

Body landmark detection model we will use later. Get it from googles website or the install script.

What Is Computer Vision Logo

Uses of Computer Vision

Healthcare

Computer vision is used in the medical industry to automate and improve the accuracy of medial imaging. It is often used to find things such as

  • Fractures
  • Tumors
  • Cardiovascular issues

Agriculture

The Agriculture industry makes use of Computer Vision in many ways.

  • Monitor Crop Health
  • Track Livestock
  • Detect Plant Disease

Document Processing

A boring but necessary function. Computer Vision is the backbone for document digitization and document OCR. Converting paper documents into digital text.

What Is Computer Vision Logo

Uses of Computer Vision

To Win Hackathons

Judges love computer vision projects, and they usually are not that complicated to set up once you know what tools you have at your disposal.

'Surveillance'

As we will see later, computer vision can be used to track where objects, including people, are in a room! This has been the backbone of many of our most beloved companies. We can track things such as

  • Cars
  • Clothing Items
  • People

'Defense'

Computer Vision's ability to track things also makes it vital in the defense industry, powering things like

  • Airplane Flying
  • Unmanned Vehicles
  • Anti-Missile Systems
  • Missile Systems
Computer Vision Logo

Image Pre-Processing

Preprocessing Logo

Image Pre-Processing

Before we can extract meaning from an image, the computer needs to understand what is happening in the image. Because computers cannot perceive an image, we need to process the raw pixels into an easier to work with form.

To accomplish this, we need to do image preprocessing, normalizing aspects of the image into a more workable form.

Note: This is for more typical non-ML and non LLM processing, LLM's are a confusing magic box im not touching
Preprocessing Logo

Color Masking

First we need to understand how a computer see's an image. At its core, an image is just a large array of numbers, representing the location and content of each pixel.

Images are stored in an array of (H, W, Chan_Vals)

Using code like this we can see what one of these arrays might look like

  import cv2

  #Read the image (Open CV loads images in B,G,R color format)
  image_bgr = cv2.imread("cat.jpg")

  print(f"{image_bgr}")
        
  [[[202 196 183]
  [202 196 183]
  [202 196 183]
  ...
  [186 180 167]
  [186 180 167]
  [186 180 167]]
        

Note: Channel values are typically [R, G B] or [R, G, B, A], printout is truncated by OpenCV
Preprocessing Logo

Color Masking

We can also look at other types of image data using OpenCV, such as the size or datatype being stored in the image

  import cv2

  #Read the image (Open CV loads images in B,G,R color format)
  image_bgr = cv2.imread("cat.jpg")

  print(f"{image_bgr.shape}")
  print(f"{image_bgr.dtype}")
        
  (1000, 944, 3)
  uint8
        
Note: uint8 = unsigned 8 bit integer. Each pixel takes 3 bytes in this picture.
Preprocessing Logo

Color Channels

Red Cat

Red Color Mask

Red Mask Pixel: [ 0 0 183]

Blue Cat

Blue Color Mask

Blue Mask Pixel: [202 0 0]

Green Cat

Green Color Mask

Green Mask Pixel: [0 196 0]

Note: All pixels taken at [0][0], original pixel is [202 196 183]
Preprocessing Logo

Color Channels

Red Cat

Emphasized Reds

Blue Cat

Emphasized Blues

Green Cat

Emphasized Greens

Note: Formula for highlighting is 2X - Y - Z
Preprocessing Logo

Resizing

Many machine learning models used in Computer Vision are also trained off of a specific size of image. Processing differently sized images can cause a loss of quality or a unwanted clipping of the images content. Many models clip to some square image size such as 256x256 or 640x640.

Smaller images can also be much quicker to process. If perfect quality is not needed reducing the amount of data will speed up processing. a 500*500 image is 250 thousand pixels to process while a 2000*2000 is 4 million.

Images that are not the correct size have to be either clipped, expanded, or resized into the correct format. There are many ways to do this but some of the most common include interpolation, and stretching.

Preprocessing Logo

Image Resizing

Original Image (944x1000)

Total Pixels: 944000

241x256

Total Pixels: 61696

60x64

Total Pixels: 3840

Note: Done with OpenCV resize. A little lie since the images are resized up
Preprocessing Logo

Interpolation Types

When we resize an image, the computer has to generate new pixels to fit the new aspect ratio. This is done through interpolation which estimates values in a known range. There are a few ways to handle interpolation demonstrated below

Nearest Neighbor

Blocky but Fast

BiLiniear

Smooth but lost quality

BiCubic

Slow but smooth

Note: Set custom interpolation with OpenCV
Preprocessing Logo

Gray Scaling

Depending on the Computer Vision task, another preprocessing step we can take is the complete removal of the color channels. By passing each pixels color values through a function we can convert each pixel to a different brightness shade of gray. While this does lose data, it also shrinks the image size and can be useful for certain tasks that do not rely on color and only use brightness, such as edge detection.

Using OpenCV's cvtColor function we can quickly change the entire image

#Convert the Image to a grayscale
#Use BGR to Gray to match imread
gray_image = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
        
Note: Function for conversion weights colors differently based on human eye sensitivity. Grayscale is 1/3 the size original
Preprocessing Logo

Other Types of Pre-Processing

Gaussian Blur / Denoising

Images with lots of noise can be slightly blurred to remove that noise.

Normalization

Instead of storing pixels on a scale of 0-255, we can normalize to a floating point number between -1 and 1. Helps models converge quicker.

Letter Boxing

Another step of image resizing. Adds a neutral color that is not later processed. Pads the smaller dimension out till image is a perfect square for better processing.

Classical Solutions Logo

Classical Processing

Classical Solutions Logo

Classical Processing

Features

A feature is a piece of information in an image that implies a point of interest. Features help classical CV tools detect objects in images without knowing what an image is.

Classical processing largely focuses on finding and describing features in an image. This is typically accomplished using kernels to analyze the contents of an image.

Kernels

A Kernel is a matrix of weights (typically 3x3) that slides over an image. At each pixel you sum pixel value * weight and assign that value to the center pixel. This allows you to describe changes in the image over a set of pixels.

Note: The process of moving a Kernel over an image is called convolution
Classical Solutions Logo

Types of Kernels

Gaussian Blur

Matrix with a Gaussian Distribution, can blur over an image reducing noise. Used to reduce noise and useful for other detection.

[1, 2, 1]
[2, 4, 2]
[1, 2, 1]
        

Edge Detection

Values differences on either side of the kernel. If all values under the kernel are similar, final value trends to 0.

[1   ,1   ,1]
[0,   0,   0]
[-1  ,-1 ,-1]
        

Sharpening

Enhances pixel intensity and visibly defines objects more. Emphasis value of the center pixel.

[0, -1,  0]
[-1, 5, -1]
[0, -1,  0]
        
Classical Solutions Logo

Edge Detection Example

Below is the result of running Sobel edge detection on our example Cat. Notice how Cat and radiator borders start to blend in and the Cats fur borders are highlighted. Also shown is the same process after a box blur has been applied to the image.

Classical Solutions Logo

Other Types of Classical Detection

While edge detection is one of the most commonly used types of classical computer vision processing, other models do exist that serve different purposes.

Threshold Detection

Grayscale mask over entire image, convert values below threshold to black and above to white. Useful for pulling subjects out or removing backgrounds for OCR.

Blob Detection

Early form of object recognition that uses significant differences in pixel values to split images into blobs of similar objects.

Classical Solutions Logo

Issues with Classical Solutions

Classic computer vision processing methods were revolutionary and had many years of fine-tuning. Processes such as HOG + SVM allowed for relatively complex detection of things like Cars and Pedestrians, but required careful fine-tuning. Many of these classical processes had to be fine-tuned by hand to work on a small subset of objects, and would break with new data.

Edge detection did a good job at telling us where something was in an image but could not tell us what the object was. This weakness was solved with...

Modern Solutions Logo

Machine Learning with Computer Vision

Modern Solutions Logo

The Start of Machine Learning

Earlier approaches had to have pre-defined kernels that were fine tuned by hand to accurately detect where objects were. Machine Learning changed that by moving away from a single purpose kernel to a series of learned kernels working in tandem. This new model was dubbed Convolutional Neural Networks (CNN)

The Training Loop

To produce a computer vision model we start with random kernel weights and a labeled dataset. The model runs its kernels over the picture -> Measures the error (the loss) -> Adjusts the kernels to be closer -> Repeats. Do that enough and you get accurate kernels that understand what they are detecting.

Modern Solutions Logo

Stacked Kernels

To improve accuracy, models began implementing multiple levels of convolution using differently weighted kernels. These feature maps each attempt to detect a specific feature and build off of each other into more abstract concepts. While a early feature may still only detect edges and corners, future layers may use that information to detect a body, and then a face and eyes off of the body.

This stacked feature hierarchy can be expanded on to add more functionality to these models. Once a model can tell that a body is in the picture, it can also attempt to track that bodies movement by generating and tracking landmarks on the body.

Modern Solutions Logo

Modern CV Tools

YOLO (You Only Look Once)

YOLO is a very popular family of object detection models provided by Ultralytics.

  • Process Images in a single pass, splits the image into regions, and each region predicts boxes, labels, and confidence scores
  • Filters out duplicate detections
  • Useful for real time object detection and tracking

Google's MediaPipe

Google's MediaPipe is another extremely popular family of models that handle different types of body tracking.

  • Different models for hands, face, or body landmarks
  • 2 step system where a body is first detected and then cropped around for landmark detection
  • Outputs landmark coordinates for body tracking
END Logo

THANK YOU

Me
info   lsenol@kent.edu
web   www.senol.dev
social   @goatmanwaffles