Ground Truth and Evaluation

This document describes how to set ground truth data for algorithm evaluation in PoSDK method plugins and the core functionality of data package operations.

Quick Navigation

Core Concepts

Data Package Management Mechanism
Ground Truth Evaluation System

Function List

Required Package Operations

Function Name	Brief Description	Details Link
`SetRequiredData()`	Set single required input data	Details
`GetRequiredPackage()`	Get required input data package reference	Details
`SetRequiredPackage()`	Batch set required input data package	Details
`GetRequiredDataTypes()`	Get required input data type list	Details
`GetRequiredDataPtr<T>()`	Simplified access to required input data (recommended)	Details

Output Package Operations

Warning

Reserved Interface

The following output package operations are reserved interfaces and are not yet implemented in the current version. Algorithms should directly obtain result data through the return value of Build() or Run().

Function Name	Brief Description	Details Link	Status
`GetOutputPackage()`	Get output data package pointer	Details	Reserved
`AddOutputDataPtr()`	Add data to output package	Details	Reserved
`GetOutputDataTypes()`	Get output data type list	Details	Reserved
`ClearOutputPackage()`	Clear output data package	Details	Reserved

Ground Truth Operations

Function Name	Brief Description	Details Link
`SetGTData()`	Set ground truth data	Details
`GetGTData()`	Get ground truth data	Details

Evaluator Operations

Function Name	Brief Description	Details Link
`SetEvaluatorAlgorithm()`	Set evaluator algorithm name	Details
`GetEvaluatorAlgorithm()`	Get evaluator algorithm name	Details
`CallEvaluator()`	Manually call evaluator	Details

Data Package Management

PoSDK’s MethodPreset class provides two core data management mechanisms:

1. Required Input Package (`required_package_`)

Input data required by the algorithm
Must be provided before Build() is called
Missing data will cause the algorithm to fail
Accessed via required_package_["key"]

2. Output Package (`output_package_`)

Warning

Reserved Feature

Output package functionality is a reserved interface and is not yet implemented in the current version. Algorithms should directly obtain result data through the return value of Build() or Run().

Result data after algorithm execution (reserved)
Results added via AddOutputDataPtr() (reserved)
Can contain multiple types of output data (reserved)
Retrieved via GetOutputPackage() (reserved)

Ground Truth Evaluation System

Ground Truth Data: Used for accuracy evaluation and result verification
Typically used during development and testing phases
Does not affect actual algorithm execution
Accessed via SetGTData()/GetGTData()
Works with evaluators for automatic accuracy analysis (when enable_evaluator=true)

Key Differences Comparison

Feature	Required Input Package	Output Package (Reserved)	Ground Truth Data
Direction	Input	Output	Evaluation
Required	Required	Generated by algorithm (reserved)	Optional
Purpose	Core input for algorithm	Algorithm execution results (reserved)	Accuracy evaluation
Impact if Missing	Algorithm cannot run	No output results (reserved)	Only affects evaluation
Typical Content	Features, tracks, camera parameters	Poses, reconstruction results (reserved)	Standard answers
Access Method	`GetRequiredDataPtr<T>()`	`AddOutputDataPtr()` (reserved)	`SetGTData()`
Setting Time	Set before execution	Generated during execution (reserved)	Set before execution
Current Status	✓ Implemented	⚠️ Reserved interface, not implemented	✓ Implemented

Function Details

Required Package Functions

`SetRequiredData()`

Function Signature:

bool SetRequiredData(const DataPtr &data_ptr);

Purpose: Set a single required input data

Parameters:

data_ptr DataPtr: Data pointer to set

Return: bool - Whether setting was successful

Usage Example:

auto tracks_data = LoadTracks("tracks.bin");
method_ptr->SetRequiredData(tracks_data);

`GetRequiredPackage()`

Function Signature:

const Package& GetRequiredPackage() const;  // const version
Package& GetRequiredPackage();              // non-const version

Purpose: Get reference to required input package

Return: Package & - Reference to required package

Usage Example:

const auto& req_pkg = method_ptr->GetRequiredPackage();
for (const auto& [key, data_ptr] : req_pkg) {
    std::cout << "Required data: " << key << std::endl;
}

`SetRequiredPackage()`

Function Signature:

void SetRequiredPackage(const Package &package);

Purpose: Batch set required input package

Parameters:

package Package: Package to set

Usage Example:

Package input_package;
input_package["data_tracks"] = tracks_ptr;
input_package["data_camera_models"] = cameras_ptr;
method_ptr->SetRequiredPackage(input_package);

`GetRequiredDataTypes()`

Function Signature:

std::vector<std::string> GetRequiredDataTypes() const;

Purpose: Get list of required input data types

Return: std::vector<std::string> - List of required data type names

Usage Example:

auto required_types = method_ptr->GetRequiredDataTypes();
for (const auto& type : required_types) {
    std::cout << "Required type: " << type << std::endl;
}

`GetRequiredDataPtr<T>()` (Recommended)

Function Signature:

template <typename T>
std::shared_ptr<T> GetRequiredDataPtr(const std::string &data_name) const;

Purpose: Simplified access to required input data with automatic type conversion

Parameters:

data_name std::string: Data name in required_package_

Return: std::shared_ptr<T> - Smart pointer to the data

Usage Example:

// Usage in MethodPreset-derived class:
auto tracks = GetRequiredDataPtr<Tracks>("data_tracks");
auto features = GetRequiredDataPtr<FeaturesInfo>("data_features");

// Or through dynamic_pointer_cast:
auto method_preset = std::dynamic_pointer_cast<MethodPreset>(method_ptr);
if (method_preset) {
    auto poses = method_preset->GetRequiredDataPtr<GlobalPoses>("data_global_poses");
}

Tip

Recommended Usage

GetRequiredDataPtr<T>() provides a more concise API, automatically handles Package lookup and type conversion, reducing code redundancy.

Output Package Functions

Warning

Reserved Interface

The following output package operations are reserved interfaces and are not yet implemented in the current version. Algorithms should directly obtain result data through the return value of Build() or Run().

`GetOutputPackage()` ⚠️ Reserved Interface

Function Signature:

DataPackagePtr GetOutputPackage() const;

Purpose: Get output data package pointer (reserved)

Return: DataPackagePtr - Smart pointer to output package

Current Status: ⚠️ Reserved interface, not yet implemented

`AddOutputDataPtr()` ⚠️ Reserved Interface

Function Signature:

void AddOutputDataPtr(const DataPtr &data_ptr, const std::string &data_name = "");

Purpose: Add data to output package (reserved)

Parameters:

data_ptr DataPtr: Data pointer to add
data_name std::string: Data name (optional, uses data_ptr type if empty)

Current Status: ⚠️ Reserved interface, not yet implemented

`GetOutputDataTypes()` ⚠️ Reserved Interface

Function Signature:

std::vector<std::string> GetOutputDataTypes() const;

Purpose: Get list of output data types (reserved)

Return: std::vector<std::string> - List of output data type names

Current Status: ⚠️ Reserved interface, not yet implemented

`ClearOutputPackage()` ⚠️ Reserved Interface

Function Signature:

void ClearOutputPackage();

Purpose: Clear output data package (reserved)

Current Status: ⚠️ Reserved interface, not yet implemented

Ground Truth Functions

`SetGTData()`

Function Signature:

void SetGTData(DataPtr &gt_data);

Purpose: Set ground truth data for evaluation

Parameters:

gt_data DataPtr: Ground truth data pointer

Usage Example:

// Load ground truth data for evaluation
auto gt_poses = LoadGroundTruthPoses("ground_truth.txt");
method_ptr->SetGTData(gt_poses);

// Algorithm will automatically compare with ground truth if enable_evaluator=true
auto result = method_ptr->Build(input_data);

`GetGTData()`

Function Signature:

DataPtr GetGTData() const;

Purpose: Get currently set ground truth data

Return: DataPtr - Ground truth data pointer

Usage Example:

auto gt_data = method_ptr->GetGTData();
if (gt_data) {
    std::cout << "Ground truth data is set" << std::endl;
}

Evaluator Functions

`SetEvaluatorAlgorithm()`

Function Signature:

virtual void SetEvaluatorAlgorithm(const std::string &algorithm_name);

Purpose: Set algorithm name identifier for evaluation

Parameters:

algorithm_name std::string: Algorithm name (to distinguish different configurations)

Usage Example:

// Used to identify different algorithm configurations in evaluation results
method_ptr->SetEvaluatorAlgorithm("PoSDK_v2");

`GetEvaluatorAlgorithm()`

Function Signature:

virtual std::string GetEvaluatorAlgorithm() const;

Purpose: Get algorithm name for evaluation

Return: std::string - Algorithm name (defaults to GetType() value)

`CallEvaluator()`

Function Signature:

bool CallEvaluator(const DataPtr &result_data);

Purpose: Manually call evaluator for accuracy evaluation

Parameters:

result_data DataPtr: Result data to evaluate

Return: bool - Whether evaluation was successful

Usage Example:

// Usually no need to call manually, Build() calls automatically when enable_evaluator=true
// Usually no need to call manually, Build() calls automatically when enable_evaluator=true
auto result = method_ptr->Run();
bool eval_success = method_ptr->CallEvaluator(result);

Note

Automatic Evaluation

CallEvaluator() is automatically called by MethodPresetProfiler::Build() when:

enable_evaluator option is set to true (default: false)
Result data is not null
Ground truth data is set via SetGTData()

If enable_evaluator is false, you need to manually call CallEvaluator() after Build() or Run().

Application Examples

Example 1: Basic Package Operations

Demonstrates basic usage of required input package:

class MyPoseEstimator : public MethodPresetProfiler {
public:
    MyPoseEstimator() {
        // Declare required input data
        required_package_["data_tracks"] = nullptr;
        required_package_["data_camera_models"] = nullptr;
    }

    DataPtr Run() override {
        // Get required input data
        auto tracks = GetRequiredDataPtr<Tracks>("data_tracks");
        auto cameras = GetRequiredDataPtr<CameraModels>("data_camera_models");

        // Execute pose estimation algorithm
        auto result_poses = EstimatePoses(tracks, cameras);

        LOG_INFO_ZH << "位姿估计完成，输出" << result_poses->Size() << "个位姿";
        LOG_INFO_EN << "Pose estimation completed, output " << result_poses->Size() << " poses";

        // Directly return result data
        return result_poses;
    }
};

// Usage
auto estimator = FactoryMethod::Create("my_pose_estimator");

// Set required inputs
estimator->SetRequiredData(tracks_data);
estimator->SetRequiredData(camera_models);

// Execute algorithm, directly get return value as result
auto result = estimator->Build();
if (result) {
    auto poses = GetDataPtr<GlobalPoses>(result);
    // Use result data...
}

Example 2: Ground Truth Evaluation Flow

Demonstrates how to set ground truth data and perform automatic evaluation:

class MyBundleAdjustment : public MethodPresetProfiler {
public:
    MyBundleAdjustment() {
        required_package_["data_tracks"] = nullptr;
        required_package_["data_camera_models"] = nullptr;
        required_package_["data_initial_poses"] = nullptr;
    }

    DataPtr Run() override {
        // Get input data
        auto tracks = GetRequiredDataPtr<Tracks>("data_tracks");
        auto cameras = GetRequiredDataPtr<CameraModels>("data_camera_models");
        auto initial_poses = GetRequiredDataPtr<GlobalPoses>("data_initial_poses");

        // Execute Bundle Adjustment optimization
        auto optimized_poses = OptimizePoses(tracks, cameras, initial_poses);

        LOG_INFO_ZH << "Bundle Adjustment优化完成";
        LOG_INFO_EN << "Bundle Adjustment optimization completed";

        // Directly return result data
        return optimized_poses;
    }
};

// Complete evaluation flow
int main() {
    // 1. Create algorithm instance
    auto ba_method = FactoryMethod::Create("my_bundle_adjustment");

    // 2. Set algorithm name identifier
    ba_method->SetEvaluatorAlgorithm("PoSDK_BA_v2.1");

    // 3. Set required inputs
    auto tracks = LoadTracks("input_tracks.bin");
    auto cameras = LoadCameras("cameras.txt");
    auto initial_poses = LoadInitialPoses("initial.txt");

    ba_method->SetRequiredData(tracks);
    ba_method->SetRequiredData(cameras);
    ba_method->SetRequiredData(initial_poses);

    // 4. Set ground truth data for evaluation
    auto gt_poses = LoadGroundTruthPoses("ground_truth_poses.txt");
    auto gt_points = LoadGroundTruthPoints("ground_truth_points.txt");

    ba_method->SetGTData(gt_poses);  // Set pose ground truth

    // 5. Enable automatic evaluation (optional, default is false)
    ba_method->SetMethodOption("enable_evaluator", "true");

    // 6. Execute algorithm (will automatically call evaluator if enable_evaluator=true)
    auto result = ba_method->Build();

    // 7. View evaluation results (evaluator automatically outputs to console and files)
    LOG_INFO_ZH << "算法执行完成，评估结果已保存";
    LOG_INFO_EN << "Algorithm completed, evaluation results saved";

    // 8. Use returned result data
    if (result) {
        auto optimized_poses = GetDataPtr<GlobalPoses>(result);
        
        // Manually perform additional evaluation (optional)
        // Note: CallEvaluator parameter should be result data, evaluator automatically gets GT from GetGTData() for comparison
        // ba_method->CallEvaluator(result);
        
        // Use optimized poses...
    }

    return 0;
}

Ground Truth Data Preparation Guide

// Ground truth pose data preparation example
void PrepareGroundTruthData() {
    // 1. Create ground truth poses
    auto gt_poses = FactoryData::Create("global_poses");
    gt_poses->Init(5);  // 5 camera poses

    // Set reference camera pose (usually identity transformation)
    Matrix3d R_ref = Matrix3d::Identity();
    Vector3d t_ref = Vector3d::Zero();
    gt_poses->SetRotation(0, R_ref);
    gt_poses->SetTranslation(0, t_ref);

    // Set ground truth poses for other cameras
    for (int i = 1; i < 5; ++i) {
        Matrix3d R_gt = LoadRotationFromFile("gt_rotation_" + std::to_string(i) + ".txt");
        Vector3d t_gt = LoadTranslationFromFile("gt_translation_" + std::to_string(i) + ".txt");
        gt_poses->SetRotation(i, R_gt);
        gt_poses->SetTranslation(i, t_gt);
    }

    // 2. Create ground truth 3D points
    auto gt_points = FactoryData::Create("world_point_info");
    gt_points->resize(1000);  // 1000 3D points

    for (size_t i = 0; i < 1000; ++i) {
        Point3d gt_point = LoadPointFromFile("gt_point_" + std::to_string(i) + ".txt");
        gt_points->setPoint(i, gt_point);
        gt_points->setUsed(i, true);
    }

    // 3. Save ground truth data for algorithm use
    gt_poses->Save("ground_truth_poses.posdk");
    gt_points->Save("ground_truth_points.posdk");

    LOG_INFO_ZH << "真值数据准备完成：" << gt_poses->Size() << "个位姿，"
                << gt_points->getValidPointsCount() << "个3D点";
    LOG_INFO_EN << "Ground truth data prepared: " << gt_poses->Size() << " poses, "
                << gt_points->getValidPointsCount() << " 3D points";
}

Best Practices

1. Clear Package Management

// Recommended: Clearly distinguish input data, directly return results
class MyAlgorithm : public MethodPresetProfiler {
public:
    MyAlgorithm() {
        // Required input data declaration
        required_package_["data_tracks"] = nullptr;
        required_package_["data_cameras"] = nullptr;
    }

    DataPtr Run() override {
        // Get input data
        auto tracks = GetRequiredDataPtr<Tracks>("data_tracks");
        auto cameras = GetRequiredDataPtr<CameraModels>("data_cameras");

        // Algorithm processing
        auto result = ProcessData(tracks, cameras);

        // Directly return result data
        return result;
    }
};

// ❌ Not recommended: Confusing input/output management
// Don't dynamically add required_package_ items outside constructor
// ⚠️ Note: Current version does not support AddOutputDataPtr(), should directly return results

2. Complete Ground Truth Evaluation

// Recommended: Complete evaluation flow setup
auto algorithm = FactoryMethod::Create("my_algorithm");

// Set algorithm identifier (for evaluation result distinction)
algorithm->SetEvaluatorAlgorithm("MyAlgo_v1.2");

// Set input data
algorithm->SetRequiredData(input_tracks);
algorithm->SetRequiredData(camera_models);

// Set ground truth data
auto gt_poses = LoadGroundTruth("gt_poses.txt");
algorithm->SetGTData(gt_poses);

// Enable automatic evaluation (optional)
algorithm->SetMethodOption("enable_evaluator", "true");

// Execute and automatically evaluate
auto result = algorithm->Build();

// ❌ Not recommended: Ignore ground truth setup
// Missing SetGTData() will prevent accuracy evaluation

3. Error Handling and Validation

// Recommended: Check data validity
DataPtr Run() override {
    auto tracks = GetRequiredDataPtr<Tracks>("data_tracks");
    if (!tracks || tracks->size() == 0) {
        LOG_ERROR_ZH << "输入轨迹数据无效";
        LOG_ERROR_EN << "Invalid input tracks data";
        return nullptr;
    }

    auto result = ProcessTracks(tracks);

    // Validate output data
    if (!result || !IsValidResult(result)) {
        LOG_ERROR_ZH << "算法输出结果无效";
        LOG_ERROR_EN << "Algorithm output is invalid";
        return nullptr;
    }

    return result;
}

// ❌ Not recommended: Missing data validation
// Not checking input data validity may cause algorithm crash

4. Documentation and README Standards

The plugin’s README should clearly state:

## Data Interface

### Required Inputs
- `data_tracks`: Feature track data type - [`Tracks`](link-to-tracks-doc)
- `data_camera_models`: Camera parameters - [`CameraModels`](link-to-camera-doc)

### Output Data
- Algorithm returns result data through return value of `Build()` or `Run()`
- Return type: `DataPtr`, can get specific type via `GetDataPtr<T>()`
- ⚠️ Note: Current version does not support `AddOutputDataPtr()` and `GetOutputPackage()`, should directly use return value

### Ground Truth Support
- Supports pose ground truth evaluation (same format as output poses)
- Supports 3D point ground truth evaluation (same format as output point cloud)
- Use `SetGTData()` to set ground truth data

### Evaluation Metrics
- Pose rotation error (angle)
- Pose translation error (distance)
- 3D point reconstruction error (Euclidean distance)

Ground Truth and Evaluation

Quick Navigation

Core Concepts

Function List

Required Package Operations

Output Package Operations

Ground Truth Operations

Evaluator Operations

Function Navigation

Application Examples

Data Package Management

1. Required Input Package (required_package_)

2. Output Package (output_package_)

Ground Truth Evaluation System

Key Differences Comparison

Function Details

Required Package Functions

SetRequiredData()

GetRequiredPackage()

SetRequiredPackage()

GetRequiredDataTypes()

GetRequiredDataPtr<T>() (Recommended)

Output Package Functions

GetOutputPackage() ⚠️ Reserved Interface

AddOutputDataPtr() ⚠️ Reserved Interface

GetOutputDataTypes() ⚠️ Reserved Interface

ClearOutputPackage() ⚠️ Reserved Interface

Ground Truth Functions

SetGTData()

GetGTData()

Evaluator Functions

SetEvaluatorAlgorithm()

GetEvaluatorAlgorithm()

CallEvaluator()

Application Examples

Example 1: Basic Package Operations

Example 2: Ground Truth Evaluation Flow

Ground Truth Data Preparation Guide

Best Practices

1. Clear Package Management

2. Complete Ground Truth Evaluation

3. Error Handling and Validation

4. Documentation and README Standards

References

1. Required Input Package (`required_package_`)

2. Output Package (`output_package_`)

`SetRequiredData()`

`GetRequiredPackage()`

`SetRequiredPackage()`

`GetRequiredDataTypes()`

`GetRequiredDataPtr<T>()` (Recommended)

`GetOutputPackage()` ⚠️ Reserved Interface

`AddOutputDataPtr()` ⚠️ Reserved Interface

`GetOutputDataTypes()` ⚠️ Reserved Interface

`ClearOutputPackage()` ⚠️ Reserved Interface

`SetGTData()`

`GetGTData()`

`SetEvaluatorAlgorithm()`

`GetEvaluatorAlgorithm()`

`CallEvaluator()`