Iterative, Deep Synthetic Aperture Sonar Image Segmentation

ABSTRACT

Synthetic aperture sonar (SAS) systems produce high-resolution images of the seabed environment. Moreover, deep learning has demonstrated superior ability in finding robust features for automating imagery analysis. However, the success of deep learning is conditioned on having lots of labeled training data, but obtaining generous pixel-level annotations of SAS imagery is often practically infeasible. This challenge has thus far limited the adoption of deep learning methods for SAS segmentation. Algorithms exist to segment SAS imagery in an unsupervised manner, but they lack the benefit of state-of-the-art learning methods and the results present significant room for improvement. In view of the above, we propose a new iterative algorithm for unsupervised SAS image segmentation combining superpixel formation, deep learning, and traditional clustering methods. We call our method Iterative Deep Unsupervised Segmentation (IDUS). IDUS is an unsupervised learning framework that can be divided into four main steps: 1) A deep network estimates class assignments. 2) Low-level image features from the deep network are clustered into superpixels. 3) Superpixels are clustered into class assignments (which we call pseudo-labels) using k-means. 4) Resulting pseudo-labels are used for loss backpropagation of the deep network prediction. These four steps are performed iteratively until convergence. A comparison of IDUS to current state-of-the-art methods on a realistic benchmark dataset for SAS image segmentation demonstrates the benefits of our proposal even as the IDUS incurs a much lower computational burden during inference (actual labeling of a test image). Because our design combines merits of classical superpixel methods with deep learning, practically we demonstrate a very significant benefit in terms of reduced selection bias, i.e. IDUS shows markedly improved robustness against the choice of training images. Finally, we also develop a semi-supervised (SS) extension of IDUS called IDSS and demonstrate experimentally that it can further enhance performance while outperforming supervised alternatives that exploit the same labeled training imagery.

Motivation

SAS is a coherent imaging modality which is capable of producing high-resolution and constant-resolution images of the seafloor. The segmentation of these images plays an important role in analyzing the seabed environment. One challenge faced by SAS seabed segmentation is acquiring labeled data. Obtaining pixel-level labels is time-consuming and often requires a diver survey to obtain correct results. This lack of labeled data makes it difficult to deploy convolutional neural networks (CNNs) for SAS seabed segmentation as such methods usually require lots of labeled data to work well.

In this work, we propose a CNN-based technique that overcomes the need to train on large datasets of pixel-level labeled data to achieve success. Specifically, we note the correlation between class boundaries of superpixels and human notated segmentation of SAS. We devise a novel scheme that combines classical superpixel segmentation algorithms (which are non-differentiable) with deep learning (which is differentiable) to obtain a deep learning algorithm capable of using superpixel information of unlabeled pixels as a way to overcome the lack of abundant training data.

We evaluate our method, which we call iterative deep unsupervised segmentation (IDUS), against several comparison methods on a contemporary real-world SAS dataset. We show the compelling performance of IDUS against existing state-of-the-art methods for SAS image co-segmentation. Additionally, we show an extension of our method which can obtain even better results when semi-labeled data is present.

The above figure shows an iteration of the simplest version of IDUS. We store the superpixel segments and labels in the superpixel segments memory and superpixel labels memory, respectively. IDUS alternates between training a deep segmentation network and generating new pseudo-labels.

Dataset

The dataset consists of 113 high-resolution complex-valued SAS images acquired from a high-frequency (HF) synthetic aperture sonar (SAS) system deployed on an unmanned underwater vehicle (UUV). The images were originally 1001×1001 pixels in size which we downsample to 512×512 pixels for computational efficiency.

The above table shows the class names and corresponding sample proportions. The class distribution is highly imbalanced, especially for the Shadow class.

The above figure shows example SAS images with various textures: SH (blue), DS (green), BS (cyan), SG (red), RK (purple), SR (gold), LR (gray) and unlabeled pixels (black). Textures from left to right are: (a) large sand ripple and seagrass; (b) rock and small sand ripple; (c) bright sand, dark sand and large sand ripple; (d) seagrass, bright sand, dark sand and shadow.

Experimental Results

The above shows confusion matrices of IDUS along with comparisons against state of the art unsupervised methods.

The above shows normalized mutual information (NMI) for various choices of cluster numbers. The results demonstrate that IDUS performs better than competing alternatives, independent of the number of clusters.

The above shows the pixel accuracies (PAs) of each class along with the corresponding mean pixel accuracy (MPA) for each method evaluated on the testing set (thirty-three sonar images for testing, eighty images for training). The best results for each class and best MPA are in bold. IDUS has the highest PAs for most classes and the best MPA demonstrating its superior performance over competing methods.

We use IDSS to fine-tune the network by using different size training sets of labeled data set to explore the performance improvement of combining IDUS with supervised training. The above shows mean pixel accuracy (MPA) for different training set sizes; higher values indicate better performance. IDSS(pre-trained on IDUS) outperforms comparison methods, especially in the case of very limited training (five samples).

We examined mean pixel accuracy in a limited training data scenario whereby only 6.25% of the training data is available. As shown in the above figure, the benefits of IDUS over state of the art are even more pronounced when training is limited.

Related Publications

  1. Y. -C. Sun, I. D. Gerg and V. Monga, "Iterative, Deep Synthetic Aperture Sonar Image Segmentation," in IEEE Transactions on Geoscience and Remote Sensing, doi: 10.1109/TGRS.2022.3162420. [IEEE Xplore, arXiv]

  2. Y. -C. Sun, I. D. Gerg and V. Monga, "Iterative, Deep, and Unsupervised Synthetic Aperture Sonar Image Segmentation," OCEANS 2021: San Diego – Porto, 2021, pp. 1-5, doi: 10.23919/OCEANS44145.2021.9705927. [IEEE Xplore]

Selected References

  1. O. Ronneberger et al., “U-net: Convolutional networks for biomedical image segmentation,” in Int. Conf. on Medical Image Computing and Computer-assisted Intervention. Springer, 2015, pp. 234–241.

  2. M. Lianantonakis et al., “Sidescan sonar segmentation using texture descriptors and active contours,” IEEE Journal of Oceanic Engineering, vol. 32, no. 3, pp. 744–752, 2007.

  3. D. P. Williams, “Unsupervised seabed segmentation of synthetic aperture sonar imagery via wavelet features and spectral clustering,” in IEEE International Conference on Image Processing, 2009, pp. 557–560.

  4. A. Zare et al., “Possibilistic fuzzy local information C-Means for sonar image segmentation,” in 2017 IEEE Symposium Series on Computational Intelligence (SSCI). IEEE, 2017, pp. 1–8.

  5. M. Rahnemoonfar et al., “Semantic segmentation of underwater sonar imagery with deep learning,” in IEEE International Geoscience and Remote Sensing Symposium, 2019, pp. 9455–9458.

Email
ipal.psu@gmail.com

Address
104 Electrical Engineering East,
University Park, PA 16802, USA

Lab Phone:
814-863-7810
814-867-4564