Photogrammetry Explained: From Multi-View Stereo to Structure from Motion Jeremy Cohen PyImageSearch

​[[{“value”:”

Table of Contents

Photogrammetry Explained: From Multi-View Stereo to Structure from Motion

In this blog post, you will learn about 3D Reconstruction.

This blog post is the 1st of a 3-part series on 3D Reconstruction:

Photogrammetry Explained: From Multi-View Stereo to Structure from Motion (this blog post)3D Reconstruction: Have NeRFs Removed the Need for Photogrammetry?3D Gaussian Splatting: The End Game of 3D Reconstruction?

To learn about 3D Reconstruction, just keep reading.

Photogrammetry Explained: From Multi-View Stereo to Structure from Motion

Should you shoot with one eye closed or both eyes open?

During the 2024 Paris Olympics, something unusual caught my eye during the shooting competition. While most athletes wore specialized glasses, eye covers, and high-tech gear to sharpen their precision, one Turkish shooter, Yusuf Dikeç, stood out. He shot with both eyes open, no elaborate equipment, and an air of effortless confidence.

I immediately wondered: “How did he shoot with both eyes open?” I always assumed you had to close one eye to focus with your dominant one. This is also what I learned when shooting for the first time. And after a quick search, I realized that many professional shooters actually train to keep both eyes open. This is not just a stylistic choice but a technique that can improve depth perception, situational awareness, and reaction time.

In the same way, when we think about 3D reconstruction and photogrammetry, we rely on a similar principle.

Whether it’s using multiple cameras or a flying drone to take pictures, we use multiple viewpoints to get a better sense of depth and even reconstruct an entire 3D scene.

Today, you may have come across many ways to do 3D reconstruction: Structure from Motion, Visual SLAM (Simultaneous Localization And Mapping), NeRFs, Stereo Vision, 3D Gaussian Splatting… In reality, there are 3 main families of algorithms, and I want to cover them in a mini-series of 3 blog posts:

This first blog post will explore Photogrammetry, which we can see as the geometric solution to 3D reconstruction.The second blog post will introduce you to NeRFs, the neural network solution.Finally, the last blog post will talk about 3D Gaussian Splatting, which combines both solutions and is currently the favored technique.

So, let’s begin with Photogrammetry!

You may have recently heard this term via Apple and Google, or you may have seen them when studying techniques to take an image to a 3D model, when learning SLAM, or when looking at 3D Computer Vision. In reality, there are two main photogrammetry methods involved:

Multi-View StereoStructure from Motion

So what is it about? Let’s take a look, and then we’ll see some examples…

Technique #1: Multi-View Stereo

The first technique that photogrammetry relies on comes from one of my favorite Computer Vision ideas: 3D Computer Vision. If you’ve never come across it, please read my complete roadmap here. The core idea? Images are not meant to stay 2D, you can create 3D models from 2D images… And the most obvious way to do this is via Stereo Vision.

When you’re using stereo cameras, you can triangulate the 3D positions of each pixel, and therefore recover a 3D point cloud.

The process looks like this:

Have you seen how we’re able to reconstruct the 3D scene using just two cameras? This is an idea many Computer Vision Engineers totally miss — because they’re so focused on image processing, Deep Learning, and OpenCV that they forget to take the time to understand cameras, geometry, calibration, and everything that really draws the line between a beginner Computer Vision Engineer, and an Intermediate one.

So how does that work? The process, explained in depth in my Pseudo-LiDAR post, looks like this:

Estimate a depth map from a stereo pairUse the camera’s parameters to turn the depth map into a 3D point cloud

Awesome! So now, we have a 3D scene, and we can actually interact with it!

So what is “Multi-View” Stereo? Well, do you notice all those white “holes” in our 3D scene? It’s because it’s only a two-view reconstruction — from two cameras.

If we add more cameras, we add more viewpoints, and thus, we can fill the holes and recover a 3D view. This is the idea of multi-view stereo. We can use 3, 4, 5, or even 10 overlapping images and retrieve a complete 360° view of a scene or physical objects:

The process is iteratively doing 2-View reconstructions. First, between cameras 1 and 2. Then, between 2 and 3. Then 3 and 4. And so on… Since we know exactly all the camera parameters, we’re just overlapping 2-View Reconstructions. In some cases, algorithms like the Iterative Closest Point (ICP) can also align the point clouds together for a better output.

Now, there are cases when you cannot do this, for example, when you don’t know the camera’s relationship to each other. We’re having images, but we don’t know how far camera 1 is from camera 2, or how oriented they are, and so on…

The problem is called Structure from Motion (SfM).

Technique #2: Structure from Motion

Now comes Structure from Motion, and as I explained, this happens when you don’t know the camera parameters or relationships to each other, and you can’t easily do the 3D Reconstruction:

If we still want to reconstruct a scene, we’ll need to re-estimate these parameters; we re-estimate the structure from the motion of cameras… And for this, we’ll rely on feature matching.

What does feature matching have to do with retrieving the relationships of the cameras?

Here is a process to understand:

From a pair of stereo images…

We’ll detect features (corners, edges, …) and match them.

From these, we’ll use heavy mathematics (singular value decomposition, RANSAC (Random sample consensus), and 8-Point Algorithm) to retrieve the motion of cameras through time and their positions.

And finally, recover the structure (3D points) from the motion previously estimated. Hence, the name structure from motion.

And so this is our Technique #2. Using it, we can work with any pair of digital images, whether we know the parameters or not… and as you guessed, we can also work with multiple consecutive images. But not just consecutives, because what I showed you here is what we could call an “incremental” structure from motion.

There is a family of Structure from Motion algorithms named “Global SfM” where the images don’t even need to belong to the same cameras or same period; it’s just random images of the same scene…

Want to understand it better? Let’s look at an example.

Example: COLMAP

COLMAP is a photogrammetry software doing Multi-View Stereo or Structure from Motion. It’s widely used in the industry to create a dense point cloud from images. I created a little bonus video to show you how it’s easy to reconstruct 3D models from 2D images.

See what we’re doing? First, we capture images, and we may or may not specify the camera location, parameters, lens distortion, or any other parameter. In this example, we’re feeding a file with everything noted.

This, and in particular Structure from Motion, is the king of 3D Reconstruction. It exists as sparse or dense, as incremental or global, and can benefit from advanced techniques such as Bundle Adjustment. Companies have used Photogrammetry to create extremely good-looking 3D models…

So, how good is Photogrammetry?

It’s actually pretty good, especially because we benefit from defined structures, we use geometry, and thus, we can do a lot of reconstruction…

But since 2020, a NeRFs “fever” has taken place online. Have you felt it? Neural Radiance Fields have been about turning an image into a 3D model… but using Deep Learning!

And in an upcoming post, we’ll dive into the 5 main steps to build neural radiance fields!

For now, let’s do a summary of what we learned in this blog.

Summary and Next Steps

There are 3 “main” ways to do reconstruction: Photogrammetry, NeRFs, and Gaussian Splatting. The most popular, used, and acknowledged is Photogrammetry, the science of reconstructing 3D objects from consecutive pairs of 2D images. It consists of two key techniques: Multi-View Stereo and Structure from Motion.Multi-View Stereo is the geometric way to reconstruct a scene from multiple pairs of images using depth triangulation and digital camera parameters. The principle relies on iterative 2-View Reconstructions based on Stereo Vision.Structure from Motion happens when we don’t know the camera parameters. In this case, we’ll recover them by matching the features of the image and estimating the motion needed from one to another.We can code the algorithms ourselves (and it’s much recommended for engineers to learn it), but at an advanced level, we prefer to use libraries such as COLMAP to do the work for us automatically.

Next Steps

If you enjoyed this blog post, I invite you to read my other 3D Computer Vision posts:

A 5-Step Guide to Build A Pseudo-LiDAR using Stereo Vision: https://www.thinkautonomous.ai/blog/pseudo-lidar-stereo-vision-for-self-driving-cars/Computer Vision: From Stereo Vision to 3D Reconstruction: https://www.thinkautonomous.ai/blog/3d-computer-vision/

How to get an Intermediate/Advanced Computer Vision Level?

I have compiled a bundle of Advanced Computer Vision resources for PyImageSearch Engineers who wish to join the intermediate/advanced area. When joining, you’ll get access to the Computer Vision Roadmap I teach to my paid members, and on-demand content on advanced topics like 3D Computer Vision, Video Processing, Intermediate Deep Learning, and more from my paid courses — all free.

Instructions here: https://www.thinkautonomous.ai/py-cv

Citation Information

Cohen, J. “Photogrammetry Explained: From Multi-View Stereo to Structure from Motion,” PyImageSearch, P. Chugh, S. Huot, and R. Raha, eds., 2024, https://pyimg.co/oa19e

@incollection{Cohen_2024_Photogrammetry_Explained,
author = {Jeremy Cohen},
title = {Photogrammetry Explained: From Multi-View Stereo to Structure from Motion},
booktitle = {PyImageSearch},
editor = {Puneet Chugh and Susan Huot and Ritwik Raha},
year = {2024},
url = {https://pyimg.co/oa19e},
}

The post Photogrammetry Explained: From Multi-View Stereo to Structure from Motion appeared first on PyImageSearch.

“}]] [[{“value”:”Table of Contents Photogrammetry Explained: From Multi-View Stereo to Structure from Motion Technique #1: Multi-View Stereo Technique #2: Structure from Motion Example: COLMAP Summary and Next Steps Next Steps Citation Information Photogrammetry Explained: From Multi-View Stereo to Structure from Motion…
The post Photogrammetry Explained: From Multi-View Stereo to Structure from Motion appeared first on PyImageSearch.”}]]  Read More 3D Computer Vision, 3D Reconstruction, Camera Calibration, Depth Estimation, Photogrammetry, Stereo Vision, Tutorial, Visual SLAM, 3d modeling, 3d reconstruction, COLMAP, computer vision, depth perception, multi-view stereo, nerf, photogrammetry, structure from motion, tutorial, visual SLAM