DolphinGemma

DolphinGemma is an artificial intelligence model developed by Google to analyze the vocalizations of marine mammals. The model is based on Google’s open-source Gemma series of large language models and is specifically configured to study the vocal communications of Atlantic spotted dolphins (Stenella frontalis). With approximately 400 million parameters, DolphinGemma is an “audio-in/audio-out” model capable of processing audio inputs and generating audio outputs, focusing on deciphering structural patterns within sound sequences.

The model was developed in collaboration with the Wild Dolphin Project (WDP). Since 1985, WDP has observed a specific population of Atlantic spotted dolphins in the Bahamas, generating a comprehensive data set that includes individual dolphin life histories, behavioral observations, and labeled underwater video recordings. DolphinGemma was trained using this database. Dolphin vocalizations were converted into tokens via Google’s SoundStream audio encoder and fed into the model’s learning process.

DolphinGemma aims to analyze the sequential patterns in natural dolphin vocalizations to identify recurring structures and motifs within sound sequences. The model operates similarly to large language models used for human language: it takes previous sounds as input and predicts the most probable following sounds. This structure enables researchers to investigate whether natural dolphin sounds contain meaningful patterns and whether a form of linguistic structure exists in their communication.

^{Left Image: A mother spotted dolphin observes her calf while foraging. When finished, she will use her unique signature whistle to recall the calf. Right Image: A spectrogram visualization of the whistle (Source:} Google⁾

One system designed for field deployment of DolphinGemma is CHAT (Cetacean Hearing Augmentation Telemetry), developed in collaboration with the Georgia Institute of Technology. This system does not aim to decode the dolphins’ complex natural communications directly but instead seeks to create a simpler, shared vocabulary. CHAT is based on the assumption that dolphins can learn to imitate artificially generated whistles associated with specific objects to communicate.

The CHAT system detects imitated whistles from users, identifies which whistle was produced, provides feedback to the researcher, and prompts the researcher to present the corresponding object. This feedback loop enables reinforcement of the sound-object association. The initial version of the system used Google Pixel 6 devices; a new generation running on the Pixel 9 model will be deployed in the field during the 2025 summer season. In this new version, both deep learning models and template-matching algorithms can operate simultaneously.

DolphinGemma’s architecture of approximately 400 million parameters is compatible with portable devices for fieldwork. This reduces the need for specialized hardware and enhances system efficiency in ocean environments. Although primarily trained on Atlantic spotted dolphin vocalizations, DolphinGemma will be released as open-source software, allowing adaptation to vocalizations of other species such as bottlenose or spinner dolphins. The model’s flexible design enables researchers to retrain it using their own datasets.

The model is used in scientific research to analyze the natural acoustic communication of marine mammals. Automating sound analysis processes previously reliant solely on human labor reduces research time and facilitates more systematic detection of patterns. Furthermore, outputs from DolphinGemma are integrated with the CHAT system to create a more interactive research environment. This allows patterns identified in natural sound sequences to be translated into simple interaction models.

Google plans to release the DolphinGemma model as open-source to the research community in summer 2025. This release will facilitate access for institutions and academics worldwide conducting marine mammal research and encourage its application in studies of vocal communication across different species. This approach is expected to increase international collaboration in research on acoustic communication in marine mammals.