Manual Photo-Identification

Researchers involved in the study of marine mammals frequently require the ability to identify individual animals for the study of various behavioral and ecological patterns.  The use of photo-identification of individual dolphins based upon natural markings has been used as an alternative to identification based upon artificial markings, such as tagging and freeze branding.  The presence of nicks, scars, and notches on the dorsal fin provides a sufficient set of features for identifying individual adult dolphins.  See Figure 1.

Figure 1:  Dolphin dorsal fins with distinctive natural markings. (Photo courtesy of Eckerd Dolphin Research Group)

Researchers photograph dolphins in their natural surroundings and compare new photographs of their dorsal fins against a catalog of photographs of previously identified dolphins.  The catalogs are organized into categories of distinct fin shapes and each of the shape categories are further divided into subcategories denoting regions where the predominant fin damage is present.  The manual photo-identification process, although effective, is extremely time consuming and visually stressful, particularly when large collections of known dolphins are used.  The sheer quantity of image data and the somewhat constrained properties of this class of images makes the automation of the photo-identification process an appealing and very feasible subject for content-based image retrieval.  The complications associated with such a project include the difficulty of creating a reliable and intuitive method for database query and the inherent computational intensity associated with the comparison of image data.

Automated Identification in DARWIN

DARWIN is a computer vision system that helps researchers to identify individual bottlenose dolphins, Tursiops truncatus, facilitating the comparison of digital images of the dorsal fins of new dolphins with a database of previously identified dolphin fins.  In addition to the digital dorsal fin image and an approximation of the fin outline, textual information containing sighting data is stored for each of the previously identified dolphins.  The software uses a semi-automated process to create an approximation of the fin outline.  The outline is used to formulate a sketch based query of the dolphin database.  The system utilizes a variety of image processing and computer vision algorithms to perform the matching process which is necessary to identify those previously identified fins which most closely resemble the unknown fin.  The program presents the database fin images to the researcher in rank order for comparison with the new fin image.

The initial DARWIN system used expert systems and qualitative fin descriptions to identify dolphin dorsal fins [1], similar to the method used by Mizroch [2]. As expected, different researchers frequently described the same fin using different damage classifications or descriptions of specific damage features. Resolving these differences required complex analysis, and the performance of the resulting program was less than optimal.

Figure 2:  Main window of DARWIN. A list of images and sighting data from the database are shown at left, image and outline of highlighted individual are shown at right.

The current DARWIN software takes a more quantitative approach to query the image collection. A researcher wishing to identify an unknown dolphin must import a digital image of a dolphin dorsal fin and roughly trace a general outline of the leading and trailing edges of the dorsal fin with the cursor. The center of the trace initializes the positions of a series of evenly spaced points positioned along the edge of the fin and provides a constraint space for the automated repositioning of the points. DARWIN uses active contours [3] to move the points from their initial locations to the actual edge of the fin. The movement is controlled by several energy measures. Large magnitude values in the image gradient attract the points; measures which monitor the even spacing and smoothness of the chain of points restrict their movement. Figure 3 illustrates the movement of the chain of points. Once the points have been appropriately repositioned, the user has the option to manually relocate individual points along the edge of the fin. This opportunity for user interaction is necessary since photographic artifacts such as glare spots are interpreted by the program as nicks and notches. In addition, parameters of the active contour which encourage the convergence toward a smooth outline, may cause the automatic repositioning to miss nicks or notches with extreme angles.

Figure 3:  Initial location of traced outline (left), and outline repositioned using active contours (right)

The DARWIN system currently stores the description of each fin in the database as a set of evenly spaced points which approximate the outline of the dolphin dorsal fin. A one dimensional representation of the dorsal fin outline composed of edge orientations of the piecewise approximation of the fin outline, is created during the even spacing process. This chain code type representation is used in the identification of salient feature points of the fin outline. To compute these two representations, a fixed spacing between points is specified. An imaginary arc of the specified radius is drawn from the base point of the outline to determine its intersection point along the outline. The orientation angle of the line segment between the base point and the intersection is computed and stored as the first value in the chain code representation of the dorsal fin outline. The intersection point is used as the new base point and orientation angles are likewise computed for each line segment in the approximated outline. Dorsal fin outlines contain very few consistantly identifiable features for alignment. As illustrated in Figure 4, the DARWIN system automatically locates 1) the start of the leading edge, 2) the end of the leading edge (where the leading edge angle begins to bend toward the fin tip) 3) the tip of the dorsal fin, 4) the most prominent notch on the trailing edge, and 5) the end of the trailing edge.

The general shape of the plot of a fin's absolute chain code (the orientations of the line segments of the outline) is that of a step edge (Figure 4), and the location of the tip corresponds to the point of transition from low to high. DARWIN uses the quadratic spline wavelet from [4] which produces large coefficients for step edge type features in the transformed signal. In order to identify the tip of the dorsal fin, a wavelet decomposition of the chain of angles comprising the outline is computed. At the coarsest level of analysis, the tip is identified by locating the maximum coefficient value. This initial positon is tracked back through each of the finer levels of the decomposition, to more precisely identify the position of the tip in the chain of angles [4]. The process is illustrated in Figure 5.

Figure 4:  Left: Dorsal fin outline with feature points indicated; Right: Chain code of outline.

Notches along the trailing edge of a dolphin dorsal fin provide primary features for the manual photo-identification process. Prominent notches in a fin outline appear as valleys in the plot of a fin's absolute chain code and a decomposition, with the wavelet mentioned above, will produce local minima for features of this type. To automate the identification of the most prominent notch along the trailing edge of the dorsal fin, only that portion of the outline following the fin tip is analyzed. Local minima with the largest magnitudes are identified at an intermediate transform level as candidate notches and are tracked to coarser levels. The minima that decreases most slowly in magnitude across the coarser levels is selected as the most prominent notch. As with the fin tip, the precise location of the notch is determined by back-tracking through increasingly finer levels of the decomposed signal.

Figure 5:  Finding the tip of the dorsal fin from the wavelet transform of the outline. A plot of the original chain code is shown on top, with increasingly coarse details below. The position of the tip is found on the coarsest (bottom) level, and tracked back through the finer levels to obtain a more accurate location.

To automatically identify the starting point of the dorsal fin outline at the base of the leading edge, the absolute angles between successive points along the outline are examined. A threshold angle is selected to maximize between class variance of the angles which comprise the the proper leading edge of the fin and the angles which are associated with the body of the dolphin, based on Otsu's method for unsupervised threshold selection in image data [5]. Line segments at the leftmost end of the outline are discarded if their angles diverge significantly from the predominant orientation of the leading edge, accurately identifying the point at which the dorsal fin meets the dolphin's back. If this junction is occluded or missing from the initial outline, the first point of the edge is stored as the start of the dorsal fin, indicating a truncated trace. The ending point of the leading edge is similarly identified as the point at which the angles of the line segments begin to significatly diverge from the predominant orientation of the leading edge.

Figure 6:  Automatic feature point identification works well for a variety of shape classes of fins.

Identifying feature points along the outline of the dorsal fin provides a basis for automating the registration of any two fin outlines prior to comparison. Given two sets of three non coincident, non colinear points, the transformation matrix which maps one set of points onto the other can be computed. Mean squared error is computed between the transformed outline of the unknown fin and each fin in the database to identify those fins which are most similar to the unknown fin.

Figure 7:  Although viewing angle angles for the database fin and "unknown" fin are quite different (the same fin in this case), the featrure points are consistantly identified, thus the outlines are well alighned.

References

[1] R. Stanley, DARWIN: Identifying Dolphins from Dorsal Fin Images, vol. Senior Thesis, Eckerd College, 1995.

[2] S.A. Mizroch, J.A. Beard, and M.Lynde, "Computer-assisted photo-identification of humpback whales, " Report to the International Whaling Commission, Special Issue 12, pp.~63--70, 1990.

[3] M. Kass, A. Witkin, and D. Terzopoulos, "Snakes: Active contour models," International Journal of Computer Vision, pp. 321--331, 1988.

[4] S. Mallat, "A Theory for Multiresolution Signal Decomposition: The Wavelet Representation," IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 11, No. 7, pp. 674-693, July 1989.

[5] N. Otsu, "A Threshold Selection Method from Gray-Level Histograms," IEEE Transactions on Systems, Man, and Cybernetics, Vol. SMC-9, No. 1, pp.62-66, 1979.

[6] D. Wilkin and K. Debure, "Query by Sketch in DARWIN: Digital Analysis of Whale Images on a Network", Proceedings of the SPIE Conference on Storage and Retrieval for Image and Video Databases VII, San Jose, CA, 1999.