LDED-FusionNet: Audio-Visual Fusion for Defect Detection in Laser-Directed Energy Deposition (LDED)

Overview

This repository contains the implementation of audio-visual feature fusion using machine learning for defect detection and quality prediction in Laser-Directed Energy Deposition (LDED) additive manufacturing processes.

LDED-FusionNet Dataset Download: https://zenodo.org/records/15050300

Experimental Setup and Schematic Representation

Experimental Setup

Schematic Representation

---

Related Publications

1. Multisensor Fusion-Based Digital Twin for Localized Quality Prediction in Robotic Laser-Directed Energy Deposition (RCIM, 2023) Link

   • Proposes a Multisensor Fusion-Based Digital Twin, leveraging feature-level fusion of acoustic and visual data for LDED quality prediction.
   • Demonstrates significant improvements in localized quality prediction.

2. In-situ Defect Detection in Laser-Directed Energy Deposition with Machine Learning and Multi-Sensor Fusion (JMST, 2024) Link

   • Explores acoustic signals and coaxial melt pool images for defect detection.
   • Presents intra-modality and cross-modality feature correlations to identify critical audiovisual signatures in LDED process dynamics.

3. Inference of Melt Pool Visual Characteristics in Laser Additive Manufacturing Using Acoustic Signal Features and Robotic Motion Data (ICCAR, 2024) Link

   • Proposes a novel technique to infer melt pool visual characteristics in LAM by combining acoustic signal features with robotic tool-center-point (TCP) motion data.
   • Highlights the potential of microphone-based monitoring as a cost-effective alternative for melt pool tracking and closed-loop control in LAM

4. Multimodal Sensor Fusion for Real-Time Location-Dependent Defect Detection in Laser-Directed Energy Deposition (IDETC-CIE, 2023) Link

   • Utilizes a hybrid CNN to directly fuse acoustic and visual raw data.
   • Achieves high defect detection accuracy without manual feature extraction.

---

Related Code

Multisensor fusion-based digital twin framework, RCIM (2023)

Figure 1. Multisensor fusion-based digital twin framework.

Figure 2. Visualization of Audio-Visual Singal During LDED process.

Related code:

• 1_vision_audio_segmentation.ipynb
• 1b_audio_signal_preprocessing.ipynb
• 1c_audiovisual_signal_visualization.ipynb

AudioVisual Cross Modality Analysis, JMST (2024)

Cross-Modality Correlation Heatmap

Audiovisual Feature Fusion Results

Related code:

• 2_feature_extraction.ipynb
• 2b_spatiotemporal_feature_fusion.ipynb
• 2c_audiovisual_feature_analysis.ipynb

Inference of Melt Pool Visual Characteristics Using Acoustic Signal, ICCAR (2024)

Figure 3. Inference of Melt Pool Visual Characteristics Using Acoustic Signal.

Related code:

• 3_meltpool_feature_prediction.ipynb

IDETC-CIE (2023)

Figure 4. Hybrid CNN model fusing acoustic and visual sensor data.

Related code:

• 5c_audiovisual_CNN.ipynb

---

Installation

• Create and activate conda environment:

  conda create --name torch python=3.8.10
  conda activate torch

• Check CUDA and install PyTorch:

  nvcc --version
  nvidia-smi
  conda install pytorch torchvision torchaudio cudatoolkit=11.0 -c pytorch

• Verify GPU usage:

  import torch
  print(torch.cuda.is_available())

• Install additional dependencies:

  pip install -r requirements.txt

---

Citation

If you find this repository or the associated dataset useful in your research, please cite the relevant papers as follows:

1. Multisensor Fusion-Based Digital Twin for Localized Quality Prediction in Robotic Laser-Directed Energy Deposition (RCIM, 2023):

@article{chen2023digitaltwin,
  title={Multisensor Fusion-Based Digital Twin for Localized Quality Prediction in Robotic Laser-Directed Energy Deposition},
  author={Chen, Lequn and others},
  journal={Robotics and Computer-Integrated Manufacturing},
  year={2023},
  publisher={Elsevier},
  url={https://www.sciencedirect.com/science/article/abs/pii/S0736584523000571}
}

2. In-situ Defect Detection in Laser-Directed Energy Deposition with Machine Learning and Multi-Sensor Fusion (JMST, 2024):

@article{chen2024multisensorfusion,
  title={In-situ Defect Detection in Laser-Directed Energy Deposition with Machine Learning and Multi-Sensor Fusion},
  author={Chen, Lequn and others},
  journal={Journal of Mechanical Science and Technology},
  year={2024},
  publisher={Springer},
  url={https://link.springer.com/article/10.1007/s12206-024-2401-1}
}

3. Inference of Melt Pool Visual Characteristics in Laser Additive Manufacturing Using Acoustic Signal Features and Robotic Motion Data (ICCAR, 2024):

@inproceedings{chen2024inference,
  title={Inference of Melt Pool Visual Characteristics in Laser Additive Manufacturing Using Acoustic Signal Features and Robotic Motion Data},
  author={Chen, Lequn and others},
  booktitle={2024 International Conference on Control, Automation and Robotics (ICCAR)},
  year={2024},
  publisher={IEEE},
  url={https://ieeexplore.ieee.org/abstract/document/10569391}
}

4. Multimodal Sensor Fusion for Real-Time Location-Dependent Defect Detection in Laser-Directed Energy Deposition (IDETC-CIE, 2023):

@inproceedings{chen2023multimodal,
  title={Multimodal Sensor Fusion for Real-Time Location-Dependent Defect Detection in Laser-Directed Energy Deposition},
  author={Chen, Lequn and Yao, Xiling and Feng, Wenhe and Chew, Youxiang and Moon, Seung Ki},
  booktitle={Proceedings of the ASME 2023 International Design Engineering Technical Conferences and Computers and Information in Engineering Conference},
  year={2023},
  publisher={ASME},
  url={https://asmedigitalcollection.asme.org/IDETC-CIE/proceedings-abstract/IDETC-CIE2023/87295/1170490}
}

---

Feel free to let me know if you have any additional requests!

License

Released under the MIT License.