Automated seaweed species classification using deep learning and large language models – Scientific Reports

Dataset collection and preparation

An initial set of 800 original images was collected during field surveys conducted in the coastal waters of Hurghada, the Gulf of Suez, and Marsa Alam, Egypt (Red Sea coast). The dataset covers 43 species across three phyla: Chlorophyta, Phaeophyta, and Rhodophyta. The dataset exhibits an inherent class imbalance, with the number of images per species varying from approximately 14 to 21. This variation is attributable to ecological heterogeneity among habitats and species-specific abundance patterns (Table 1).

Taxonomic identification was performed through detailed morphological examination using standard taxonomic keys^{11, 2,12,13,14,15};. Species identification followed established taxonomic criteria for marine macroalgae. The identifications were further validated by comparing the collected field images with reference specimens preserved in the Suez Canal University Herbarium.

Unlike web-scraped datasets, which may suffer from inconsistent labeling, variable image quality, and limited ecological context, the present dataset was constructed entirely from field-collected images under natural environmental conditions. This ensures higher ecological validity and a more realistic representation of species variability, making the dataset suitable for training and evaluating machine learning models for marine biodiversity applications. The combination of multi-site sampling, taxonomic validation, and natural habitat variability ensures that the dataset is representative for real-world ecological classification tasks.

Data augmentation and splitting

The original 800 images were expanded to 3,440 images using rotation (± 30°), flipping, brightness adjustments (0.8–1.2), and scaling (20% zoom). Augmentation parameters were applied adaptively, with more intensive transformations for minority classes, to mitigate class imbalance and ensure robust representation across all species. The dataset was split into 80% training, 10% validation, and 10% testing subsets using a stratified scheme, guaranteeing proportional representation of all 43 species in each subset.

CNN model development

The CNN utilized the EfficientNet-B0 architecture with transfer learning (ImageNet pretrained weights). This architecture was selected for its efficiency and state-of-the-art performance in biological classification ^,16. The model was trained to recognize key morphological features, such as branching patterns (e.g., dichotomous vs. pinnate), thallus texture (e.g., smooth, rugose), the presence of air bladders (pneumatocysts), and overall structural complexity^17,18.

Architecture and implementation

The model was implemented in TensorFlow/Keras. The top layer of the EfficientNet-B0 base was replaced with a Global Average Pooling layer followed by a fully connected Dense layer with Softmax activation for 43-class classification. This design enabled the model to leverage high-level feature representations learned from ImageNet while adapting to the specific characteristics of the seaweed dataset.

Training protocol

The CNN model was trained for a total of 35 epochs, with early stopping applied based on validation loss (patience = 5) to prevent overfitting and ensure optimal generalization. Transfer learning was implemented using a two-stage strategy: initially, the base layers of EfficientNet-B0 were frozen to preserve general feature representations learned from large-scale ImageNet data, while only the classification head was trained. In the second stage, selected upper convolutional layers were gradually unfrozen to enable fine-tuning and adaptation to domain-specific features of seaweed morphology. This approach improved the model’s ability to capture fine-grained differences among visually similar species while maintaining stability in training. The complete training configuration, including optimizer settings, learning rate, loss function, batch size, and other hyperparameters, is summarized in Table 2.

Claude AI vision-language model (VLM) development

To explore an alternative classification paradigm beyond conventional convolutional neural networks, we implemented a vision–language model (VLM) based on the multimodal capabilities of Claude 3.5 Sonnet accessed via the Anthropic API. Unlike CNN-based models that rely primarily on pixel-level pattern recognition, VLMs integrate visual understanding with semantic reasoning, enabling the model to interpret visual features in conjunction with contextual knowledge. This paradigm aligns with recent developments in multimodal AI systems capable of performing structured reasoning over visual inputs and generating interpretable outputs^8,19.

In this framework, the model receives an input image together with a constrained classification prompt that limits the possible outputs to a predefined list of seaweed taxa. The objective of this approach was not only to assess classification performance but also to evaluate the interpretability and reasoning capabilities of a large vision–language model in a marine biodiversity monitoring context.

To ensure a fair and reproducible evaluation, a total of 100 test images were randomly selected from the held-out test set, ensuring representative coverage across the 43 seaweed species. The reported VLM performance corresponds to raw model predictions without any human intervention, reflecting the standalone capability of the model.

Technical configuration and prompt engineering

To ensure reproducibility and controlled inference behavior, the Claude VLM was queried using a structured prompt-engineering strategy. Each classification request included a system prompt designed to emulate expert taxonomic reasoning while constraining the output to the predefined species list.

The system prompt used for each query was structured as follows:

“You are an expert marine taxonomist. Analyze the following image and identify the seaweed species from this list: [List of 43 species]. Provide your answer in JSON format with the fields ‘species_name’, ‘confidence’, and ‘reasoning’.”

This structured output format ensured that the model returned standardized responses containing the predicted species name, an associated confidence score, and an explanatory reasoning component describing the morphological features influencing the decision.

The Claude API was configured with controlled inference parameters to reduce stochastic variation and improve reproducibility. Specifically, the temperature parameter was set to 0 to enforce deterministic outputs, while the top-p parameter was set to 0.9 to maintain limited probabilistic diversity within the sampling distribution. The maximum token length was limited to 512 tokens to allow sufficient space for structured reasoning without excessive response length.

Model responses were automatically parsed using a Python-based regular expression script, which extracted the predicted species label, confidence score, and explanatory reasoning from the returned JSON structure. These parsed outputs were subsequently used for evaluation and integration into the validation workflow.

Human-in-the-loop (HITL) validation system

To improve reliability and interpretability, a Human-in-the-Loop (HITL) validation framework was integrated into the Claude AI classification pipeline. Human-guided validation mechanisms are increasingly recognized as essential components of Explainable Artificial Intelligence (XAI), particularly in ecological applications where classification errors may have significant implications for biodiversity monitoring and conservation^20,21,22,23.

Within this workflow, the Claude model generates a predicted species label, confidence score, and explanatory rationale for each input image. Predictions with confidence scores below a predefined threshold (confidence < 0.7) are automatically flagged for expert review. These cases typically correspond to visually ambiguous taxa or morphologically similar or underrepresented species.

During the validation stage, a marine biologist with taxonomic expertise reviews the flagged predictions, verifies the classification, and corrects labels when necessary. The validated outputs are stored in a structured SQL database, enabling systematic recording of expert corrections.

This process establishes a continuous learning loop, where expert feedback can be used to guide future prompt refinement and potential model optimization. Such iterative human–AI collaboration is particularly valuable for addressing the long-tail distribution of rare marine species and improving robustness in fine-grained ecological classification tasks²³.

Model deployment and accessibility

To enhance the practical utility and accessibility of the proposed framework, the trained convolutional neural network (CNN) model was deployed as an interactive web application using Hugging Face Spaces. This platform was selected due to its ease of access and capability to host machine learning models without requiring local installation or specialized computational resources²⁰. The application was implemented using the Gradio library, providing an intuitive interface that enables users to upload seaweed images and obtain real-time classification results.

The system returns the top-5 predicted species along with their corresponding confidence scores, offering interpretable outputs that can facilitate marine biodiversity monitoring and environmental education applications.

For reproducibility, the deployment includes the trained Keras model file (best_seaweed_aug_model.keras), class label mappings (class_names.json), and all required dependencies.

The application is publicly accessible at: https://huggingface.co/spaces/Mahmoudsami/Seaweed_Identifier.

The Claude AI model is maintained as a “Claude AI Artifact,” representing a combination of the specific model version, optimized prompts, and the HITL protocol. It is accessible via API for integrated applications and is available at: Claude AI Artifact (Seaweed Identifier: AI-Powered Marine Algae Recognition Tool | Claude).

Evaluation metrics

The performance of both models was quantitatively assessed using standard classification metrics:

• Accuracy: The proportion of correctly classified images. It is calculated as (True Positives + True Negatives)/(Total Samples).

• Top-5 Accuracy: The proportion of images where the correct class is among the top five predictions. This metric is particularly relevant for practical applications where the model serves as a decision-support tool for experts.

• F1-Score: The harmonic mean of precision and recall. It is a robust metric for evaluating performance on imbalanced datasets. We report the macro-averaged F1-score, which calculates the metric independently for each class and then averages them.

• Confusion Matrix: A matrix used to visualize the performance of the classification model. Each row represents the instances in an actual class, while each column represents the instances in a predicted class. This is crucial for identifying specific inter-species misclassifications.