Platform Comparison Test Results

This document presents performance and accuracy comparison test results between PoSDK and current mainstream 3D reconstruction platforms on standard datasets.

Pose-only platform comparison framework with other platforms

Pose-only platform comparison framework with other platforms

Platform Introduction

PoSDK

Tip

PoSDK Runtime Parameter Configuration

PoSDK executes comparison tests through the globalsfm_pipeline plugin with main parameters:

  • Preprocessing Type: preprocess_type=posdk (also supports openmvg, opencv)

  • Evaluation Mode: evaluation_print_mode=comparison

  • Comparison Algorithms: compared_pipelines=openmvg,COLMAP,GLOMAP

  • Performance Analysis: enable_profiling=true

For complete parameter descriptions, refer to Plugin Configuration Documentation.

OpenMVG (Open Multiple View Geometry)

  • Project Homepage: https://github.com/openMVG/openMVG

  • Repository: https://github.com/openMVG/openMVG

  • Technical Features:

    • Open-source multi-view geometry library

    • Provides complete SfM (Structure from Motion) solution

    • Supports both incremental and global reconstruction methods

  • Development Team: Pierre Moulon and open-source community contributors

  • License: MPL2 (Mozilla Public License 2.0)

COLMAP

  • Project Homepage: https://COLMAP.github.io/

  • Repository: https://github.com/COLMAP/COLMAP

  • Technical Features:

    • Industry-leading 3D reconstruction system

    • Supports incremental SfM, dense reconstruction, and MVS

    • Provides graphical interface and command-line tools

    • High-precision pose estimation and point cloud reconstruction

  • Development Team: Johannes Schönberger, ETH team

  • License: BSD License

GLOMAP (Global Mapping)

  • Project Homepage: https://github.com/COLMAP/GLOMAP

  • Repository: https://github.com/COLMAP/GLOMAP

  • Technical Features:

    • Next-generation global SfM system

    • Focuses on rapid reconstruction of large-scale scenes

    • Pose estimation method based on global optimization

    • COLMAP-compatible data format

  • Development Team: ETH team

  • License: BSD License

Test Environment

  • Operating System: Ubuntu 24.04 LTS

  • Processor: Intel/AMD x86_64

  • Test Dataset: Strecha standard dataset

    • fountain-P11 (11 images)

    • castle-P19/P30 (19/30 images)

    • entry-P10 (10 images)

    • Herz-Jesus-P8/P25 (8/25 images)

Sparse Point Cloud Visualization

The following shows sparse point cloud reconstruction result comparisons from different platforms on the Strecha (castle-P30) dataset:

COLMAP sparse point cloud reconstruction result

COLMAP

 GLOMAP sparse point cloud reconstruction result

GLOMAP
OpenMVG sparse point cloud reconstruction result

OpenMVG

 PoSDK sparse point cloud reconstruction result

PoSDK (Ours)

Notes:

  • All point clouds are compared and displayed under the same dataset

  • Point cloud colors represent RGB information of feature points

  • GIF animations show point cloud rotation views for easy observation of reconstruction quality

Note

Table data comes from platform test results. In the tables:

  • Red bold: Best result for the same dataset

  • Black bold: Second-best result for the same dataset

Total Runtime Comparison

The following table shows total runtime (unit: milliseconds) on different datasets:

Dataset PoSDK (ms) OpenMVG (ms) COLMAP (ms) GLOMAP (ms)
Herz-Jesus-P25 14,434 26,549 91,958 83,249
Herz-Jesus-P8 3,639 6,671 22,313 19,540
castle-P19 7,278 12,011 63,818 59,280
castle-P30 14,688 23,790 121,657 105,849
entry-P10 4,187 8,727 31,482 30,002
fountain-P11 5,465 12,610 37,517 32,239

Accuracy Comparison Results

1. Global Pose Rotation Error

Global pose rotation error statistics (unit: degrees):

Dataset Algorithm Mean Median Min Max StdDev
Herz-Jesus-P25 PoSDK 0.0584 0.0508 0.0336 0.1182 0.0227
Herz-Jesus-P25 COLMAP 0.0642 0.0570 0.0299 0.1213 0.0249
Herz-Jesus-P25 GLOMAP 0.0641 0.0521 0.0329 0.1187 0.0243
Herz-Jesus-P25 OpenMVG 0.0546 0.0478 0.0329 0.1044 0.0192
Herz-Jesus-P8 PoSDK 0.0194 0.0203 0.0120 0.0261 0.0049
Herz-Jesus-P8 COLMAP 0.0623 0.0621 0.0191 0.1083 0.0298
Herz-Jesus-P8 GLOMAP 0.0583 0.0573 0.0208 0.0993 0.0268
Herz-Jesus-P8 OpenMVG 0.0228 0.0244 0.0152 0.0326 0.0060
castle-P19 PoSDK 0.0445 0.0401 0.0102 0.0884 0.0233
castle-P19 COLMAP 0.1609 0.1240 0.0345 0.8518 0.1703
castle-P19 GLOMAP 0.0473 0.0444 0.0095 0.0960 0.0237
castle-P19 OpenMVG 0.0695 0.0592 0.0227 0.1486 0.0371
castle-P30 PoSDK 0.0638 0.0598 0.0459 0.1120 0.0142
castle-P30 COLMAP 0.1791 0.1292 0.0816 1.5168 0.2498
castle-P30 GLOMAP 0.0968 0.0990 0.0485 0.1723 0.0283
castle-P30 OpenMVG 0.0843 0.0841 0.0530 0.1255 0.0200
entry-P10 PoSDK 0.0244 0.0231 0.0099 0.0461 0.0103
entry-P10 COLMAP 0.1151 0.1172 0.0643 0.1559 0.0256
entry-P10 GLOMAP 0.0780 0.0865 0.0491 0.1065 0.0181
entry-P10 OpenMVG 0.0321 0.0309 0.0112 0.0508 0.0125
fountain-P11 PoSDK 0.0294 0.0271 0.0233 0.0392 0.0056
fountain-P11 COLMAP 0.0594 0.0646 0.0225 0.0827 0.0178
fountain-P11 GLOMAP 0.0587 0.0618 0.0266 0.0788 0.0170
fountain-P11 OpenMVG 0.0297 0.0281 0.0221 0.0399 0.0054

2. Global Pose Translation Error

Global pose translation error statistics (unit: normalized distance):

Dataset Algorithm Mean Median Min Max StdDev
Herz-Jesus-P25 PoSDK 0.0053 0.0056 0.0014 0.0126 0.0024
Herz-Jesus-P25 COLMAP 0.0108 0.0112 0.0014 0.0268 0.0057
Herz-Jesus-P25 GLOMAP 0.0111 0.0117 0.0012 0.0273 0.0059
Herz-Jesus-P25 OpenMVG 0.0053 0.0059 0.0009 0.0122 0.0024
Herz-Jesus-P8 PoSDK 0.0037 0.0037 0.0016 0.0064 0.0012
Herz-Jesus-P8 COLMAP 0.0043 0.0045 0.0020 0.0060 0.0014
Herz-Jesus-P8 GLOMAP 0.0042 0.0042 0.0025 0.0061 0.0012
Herz-Jesus-P8 OpenMVG 0.0038 0.0036 0.0018 0.0067 0.0013
castle-P19 PoSDK 0.0245 0.0195 0.0053 0.0610 0.0148
castle-P19 COLMAP 0.1970 0.1033 0.0570 1.3986 0.2936
castle-P19 GLOMAP 0.0297 0.0272 0.0040 0.0593 0.0164
castle-P19 OpenMVG 0.0305 0.0224 0.0132 0.0714 0.0168
castle-P30 PoSDK 0.0213 0.0167 0.0087 0.0684 0.0119
castle-P30 COLMAP 0.2597 0.1424 0.0110 3.1922 0.5519
castle-P30 GLOMAP 0.0555 0.0565 0.0134 0.0910 0.0212
castle-P30 OpenMVG 0.0211 0.0197 0.0041 0.0616 0.0109
entry-P10 PoSDK 0.0064 0.0050 0.0038 0.0130 0.0031
entry-P10 COLMAP 0.0304 0.0244 0.0139 0.0541 0.0128
entry-P10 GLOMAP 0.0207 0.0211 0.0114 0.0294 0.0061
entry-P10 OpenMVG 0.0063 0.0052 0.0028 0.0126 0.0032
fountain-P11 PoSDK 0.0025 0.0022 0.0008 0.0041 0.0011
fountain-P11 COLMAP 0.0042 0.0040 0.0014 0.0071 0.0018
fountain-P11 GLOMAP 0.0037 0.0042 0.0011 0.0059 0.0017
fountain-P11 OpenMVG 0.0026 0.0023 0.0005 0.0043 0.0012

3. Relative Pose Rotation Error

Relative pose rotation error statistics (unit: degrees):

Dataset Algorithm Mean Median Min Max StdDev
Herz-Jesus-P25 PoSDK 0.0529 0.0441 0.0059 0.2656 0.0393
Herz-Jesus-P25 OpenMVG 0.0524 0.0470 0.0089 0.1928 0.0305
Herz-Jesus-P8 PoSDK 0.0317 0.0273 0.0065 0.0720 0.0177
Herz-Jesus-P8 OpenMVG 0.0395 0.0353 0.0130 0.1331 0.0259
castle-P19 PoSDK 0.1564 0.0932 0.0092 0.8094 0.1749
castle-P19 OpenMVG 0.1369 0.0962 0.0296 0.6888 0.1270
castle-P30 PoSDK 0.1740 0.1049 0.0051 1.2164 0.1939
castle-P30 OpenMVG 0.1832 0.1003 0.0038 2.3068 0.2555
entry-P10 PoSDK 0.0426 0.0332 0.0088 0.1519 0.0323
entry-P10 OpenMVG 0.0545 0.0398 0.0053 0.2499 0.0476
fountain-P11 PoSDK 0.0450 0.0425 0.0056 0.2625 0.0404
fountain-P11 OpenMVG 0.0502 0.0432 0.0114 0.1244 0.0302

Dataset Details

Strecha Dataset

The Strecha dataset is a classic multi-view stereo (MVS) evaluation dataset provided by École Polytechnique Fédérale de Lausanne (EPFL), containing:

  • fountain: Fountain scene with complex geometric structures and rich textures

  • castle: Castle scene with large viewing angle span and challenging occluded regions

  • entry: Entry scene of medium scale with rich building details

  • Herz-Jesus: Church scene with fine architectural details and lighting variations

Each dataset provides:

  • High-resolution color images

  • Calibrated camera intrinsic matrices

  • Ground truth camera poses

  • Ground truth 3D point clouds (for accuracy evaluation)

References:

  1. Strecha, C., et al. “On Benchmarking Camera Calibration and Multi-View Stereo for High Resolution Imagery.” CVPR 2008.

  2. Moulon, P., et al. “OpenMVG: Open Multiple View Geometry.” ICCV 2013 Workshop.

  3. Cai, Q., et al. “A pose-only solution to visual reconstruction and navigation.” TPAMI 2023.