Top Datasets For Maritime AI, USV Perception & Obstacle Detection · Bifrost

Top Datasets For Maritime AI, USV Perception & Obstacle Detection

A framework for evaluating maritime datasets, plus 14 of the best open-source datasets for USV perception, obstacle detection, and ship classification, scored on ontology, diversity, and label quality.

Not All Datasets Are Equal

The hardest part of building physical AI isn't the algorithms—it's the data. High-quality, high-coverage datasets are almost impossible to acquire unless you're willing to deploy immense sensor fleets across the globe. Even when these datasets exist, their flaws are often invisible until they show up as failures in real-world deployment. Their quality often varies dramatically—a detail that is difficult to assess from a surface-level evaluation. Poorly designed datasets don't just make your performance metrics look bad—they set your system up to fail when it matters most.

Why Does It Matter?

We've seen firsthand how poorly designed datasets—with bad ontologies, limited diversity, or inconsistent labeling—can cause significant issues in real-world deployment and result in mishaps in production. By sharing this list of top open-source maritime datasets, we aim to provide a framework for evaluating and selecting datasets that meet the highest standards of quality.

Whether you're building a USV perception system or simply exploring maritime AI, this list will serve as a resource to help you find the best data for your needs.

How Do We Measure Dataset Quality?

At Bifrost, we employ a suite of tools and methodologies to rigorously quantify dataset quality. These include UMAP-based cluster analysis, which visualizes high-dimensional data distributions to detect redundancies, over-representations and gaps, ensuring datasets are well-balanced and representative. Our scenario coverage assessments systematically verify the inclusion of diverse operational contexts and edge cases, while distribution evaluations of object classes and environmental conditions ensure sufficient representation across all relevant operational conditions.

These analyses feed into a simplified scoring framework across three critical dimensions: ontology, diversity and label quality.

1. Ontology Quality

Ontology refers to how the authors have chosen to group specific maritime objects into classes. This is one of the trickiest steps to get right - we've personally seen poor ontology lead to a great deal of failure cases in the field. One of the primary reasons for this is the significant variation in object ontologies and inconsistent definitions across authors, regions, and maritime subject matter experts.

2. Diversity Coverage

A diverse and high-coverage dataset captures a high percentage of all possible operational conditions, ensuring that a physical AI system is equipped to handle the complexity and variability of real-world deployment. This includes:

Diversity is the single most important factor in ensuring a model can generalize effectively to real-world scenarios—a crucial capability for the success of USV systems.

3. Label Quality

Label quality refers to the accuracy and consistency of annotations. Inconsistent, noisy, or inaccurate annotations, such as mislabeled classes or poorly drawn bounding boxes, can lead to significant performance degradation in deployed systems. We use both automated consistency checks and manual audits to verify precision and accuracy of real world data.

We hope these categories outline a clearer framework for evaluating maritime datasets, ultimately enhancing their utility for maritime perception development.

1. LARS Dataset

LARS is a maritime panoptic obstacle detection benchmark designed for real-time marine object detection, featuring scenes from lakes, rivers and seas. It boasts the largest diversity in recording locations, scene types, obstacle classes, and acquisition conditions among the related datasets.

2. WaterScenes Dataset

WaterScenes is especially useful for training and evaluating models in tough conditions. We love that they've already tagged these images so it's easy to retrieve images with a specific condition. Overall high quality dataset, with quite a few different label types. While diverse in terms of sensor conditions, the sequences are mostly captured in the same location.

3. MODD2 (Marine Object Detection Dataset)

A large and challenging multi-modal marine obstacle detection dataset captured by a real USV. The USV was manually guided and simulated realistic navigation scenarios in which an obstacle may present a danger to the USV, allowing it to capture diverse weather conditions, extreme situations, and various small obstacles.

4. MASTR1325 Dataset

MASTR1325 is a large-scale marine dataset focused on semantic segmentation, tailored for development of obstacle detection methods in small-sized coastal USVs. Each image is labeled for various marine objects, critical for navigation systems. Do note however that the images in this dataset are segmentation only, no bounding boxes.

5. SPSCD Maritime Dataset

The SPSCD dataset was built as a reference dataset for video surveillance and ship classification in real maritime zones. The dataset allows for estimating detection and classification performance, which provides versatile ship annotations and classifications for passenger ports with a large number of small-to-medium-sized ships that were not monitored by the automatic identification system AIS and/or the vessel traffic system (VTS).

6. KOLOMVERSE

This dataset contains 100k+ 4K images with variations in illumination, viewpoint, occlusion, background, scale and proportion and is available by request. However the ontology is weak (all vessels are considered boat), but at least there's rare wind farm and lighthouse labels if they're objects you need to detect.

7. Pohang Canal Dataset

A multimodal maritime dataset for autonomous ship navigation in restricted waters, acquired in restricted waters in Pohang, South Korea. The dataset focuses on small boats and vessels in canal environments, under varied weather and lighting conditions. High quality, clean dataset that has LiDAR but not many obstacles - useful for navigation purposes.

8. Singapore Maritime Dataset (SMD)

Everyone's favorite SMD is the ImageNet of maritime object detection - good for its time, dated, but everyone still uses it. Good for general object detection with decent classes and good label quality (there's even an additional set of cleaned-up, third party labels, see SMD Plus). Lacking in diversity - only has clear, hazy and evening conditions in relatively calm waters. We recommend using SMD as baseline evaluation data for open ocean testing instead of training. Has updated labels created by a third party, Singapore-Maritime-Dataset-Plus.

9. SeaShips MCVWT Dataset

A dataset for detecting and classifying maritime vessels with around 7k maritime images containing 6 classes of vessels of varying sizes such as bulk cargo carriers, ore carriers, and fishing boats. Images lack diversity - only has clear, hazy conditions in relatively calm waters.

10. MassMIND Dataset

An exhaustive dataset of real-life Long Wave Infrared (LWIR) images geared towards development of deep learning algorithms specifically for the Maritime domain. This dataset has seasonal and temporal diversity with various scene coverage and class distribution.

11. MUSSID Dataset

The MUSSID dataset serves to fill the gap for publicly available maritime image datasets that were developed under limited environments with slight-to-moderate variations in maritime features. This dataset accomplishes this by incorporating various geographical, seasonal, and maritime features. While the dataset offers a wider variety of features, its main use case is for sea horizon line (SHL) detection - a lot of empty ocean shots, unique angles, lots of glare - which would be good for training against glare and reflections if that's what you're looking for.

12. MARVEL 2016 Dataset

The Marvel dataset serves to help with ship identification and categorization which makes it a very curated and easy dataset in the sense that there's only one ship per image and they are fairly large ships. In terms of usefulness, it's a good dataset to get started with but not really suitable and relevant for production.

13. MariShipSegHEU

The dataset mainly contains images used for ship classification and segmentation tasks, with a portion of its images from pre-existing public maritime datasets such as Singapore Maritime Dataset (SMD). As such, the images here are very curated and easy as well, similar to the MARVEL dataset. Would also recommend this as a good dataset to get started with, but not really suitable and relevant for production either.

14. VAIS Dataset

The VAIS dataset contains 2014 trained images of maritime vessels. Vessels are annotated with bounding boxes but image quality is low - has only 1 large vessel per image and classifies all vessels as boats in general.

Create your own high-quality maritime datasets

Looking to create your own high-quality maritime datasets? We'd love to show you how! Reach out to us for a live demo of our data generation platform and see it in action.