Data loss after running Bash script creating dirs and moving files

Question

For several instances now, I experienced some very weird behavior, where part of the directories and the files in a directory got deleted after running a script which creates new directories within that directory and writes files to it.

The details about my system:

Distributor ID: Ubuntu
Description:    Ubuntu 22.04.3 LTS
Release:    22.04
Codename:   jammy

Linux ThinkPad-P16s-Gen-1 6.2.0-37-generic #38~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Thu Nov  2 18:01:13 UTC 2 x86_64 x86_64 x86_64 GNU/Linux

Filesystem      Size  Used Avail Use% Mounted on
tmpfs           1,6G  2,9M  1,6G   1% /run
/dev/nvme0n1p6  144G  132G  3,9G  98% /
tmpfs           7,7G   76M  7,6G   1% /dev/shm
tmpfs           5,0M  4,0K  5,0M   1% /run/lock
/dev/nvme0n1p1  256M   58M  199M  23% /boot/efi
tmpfs           1,6G  132K  1,6G   1% /run/user/1001
/dev/nvme0n1p4  684G  429G  255G  63% /media/thomasrauter/Data1

For example, I ran a script that I usually run and were I already know that it works, and it worked the first time, then I ran it another time with another input and midway to the files it should process, the script itself, two other .sh scripts in that directory and half the other directories disappeared (they are not in the trash).

I had this problem in similar form in the past, where everything in a directory disappeared. I then ran fsck on that unmounted drive, nothing happened, but a few days later (worked on the computer and Ubuntu in these days) the files suddenly reappeared again.

Sometimes also, when a script writes several files to a directory, that directory does not show these files when running ls, and on the graphical user interface, it says it's loading but it never gets there. This stays like that even after I reboot.

Sorry, I cannot describe a concise problem, but my question is basically if anyone of you has seen anything like that before (I really think it has something to do with Nautilus or Nemo), because right now, my Ubuntu feels like a haunted house.

---

Edit (thank you of pointing out the issues of my question):

ls -al /media/thomasrauter/ returns:

total 24
drwxr-x---+ 4 root         root          4096 Dez  1 15:47 .
drwxr-xr-x  4 root         root          4096 Jun  4 15:54 ..
drwxr-xr-x  2 root         root          4096 Aug 19 10:24 Data
drwxrwxrwx  1 thomasrauter thomasrauter 12288 Dez  1 14:53 Data1

This is the code of one of the scripts that causes trouble (but it is not the only script, and it always did its job before):

#!/bin/bash

# Check if eight arguments are provided
if [ "$#" -ne 7 ]; then
    echo "Usage: bash $0 <seq_platform> <depth> <length_min> <length_max> <length_mean> <length_sd> <accuracy_mean>"
    echo "bash simulate_reads.sh ONT 30 1000 100000 20000 10000 0.85"
    echo "bash simulate_reads.sh PacBio 30 1000 60000 15000 5000 0.90"
    exit 1
fi

# Assigning arguments to variables
seq_platform="$1"
depth="$2"
length_min="$3"
length_max="$4"
length_mean="$5"
length_sd="$6"
accuracy_mean="$7"

if [ "$seq_platform" == "ONT" ]; then
  model='QSHMM-ONT.model'
elif [ "$seq_platform" == "PacBio" ]; then
  model='QSHMM-RSII.model'
else
  echo "Error: Only ONT and PacBio supported as seq platform."
  exit 1
fi


# Check if the image exists locally
if [[ "$(docker images -q thomasrauter/pbsim3:1.0 2> /dev/null)" == "" ]]; then
    # If the image doesn't exist locally, pull it from Docker Hub
    docker pull thomasrauter/pbsim3:1.0
fi


# Create an array to store filenames
file_array=()

# Loop to find files ending with .fa or .fasta and add them to the array
for file in *.fa *.fasta; do
    if [ -e "$file" ]; then  # Check if file exists
        file_array+=("$file")
    fi
done


for fasta_filename in "${file_array[@]}"; do
  echo -e "\n#####################################################################################\n\n"
  echo -e"\nProcessing $fasta_filename"

  output_dir_name="${fasta_filename%%.*}_${seq_platform}_pbsim3_output_$(date +"%Y-%m-%d_%H-%M-%S")"

  docker run --name pbsim3 \
    -v "$(pwd)/$fasta_filename":/pbsim3/input_files/input.fasta \
    thomasrauter/pbsim3:1.0 \
    pbsim --strategy wgs \
          --method qshmm \
          --qshmm data/"$model" \
          --depth "$depth" \
          --genome input_files/input.fasta
          --length-min "$3" \
          --length-max "$4" \
          --length-mean "$5" \
          --length-sd "$6" \
          --accuracy-mean "$7"

  docker cp pbsim3:"/pbsim3/output_files" "$(pwd)/$output_dir_name"
  docker rm pbsim3 > /dev/null 2>&1

  # Change to the specified directory
  cd "$output_dir_name" || exit 1  # Exit if the directory change fails

  # Create an array to store .fastq filenames
  local_fastq_files=()

  # Find all .fastq files and add them to the array
  for local_file in *.fastq; do
      if [ -e "$local_file" ]; then  # Check if file exists
          local_fastq_files+=("$local_file")
      fi
  done

  # Concatenate all files in the array into a single .fastq file
  local_combined_fastq="combined.fastq"  # Name of the output file
  cat "${local_fastq_files[@]}" > "$local_combined_fastq"

  # Write a file with the used parameters
  output_file="pbsim3_parameters.txt"

  # Write to the file
  {
      echo "Input_fasta_file: $fasta_filename"
      echo "Seq_platform: $seq_platform"
      echo "Depth: $depth"
      echo "Length_min: $length_min"
      echo "Length_max: $length_max"
      echo "Length_mean: $length_mean"
      echo "Length_sd: $length_sd"
      echo "Accuracy_mean: $accuracy_mean"
  } > "$output_file"

  cd .. || exit 1

  echo "Simulation of the reads for the file $fasta_file completed. The files are in the dir $output_dir_name in the current working dir."
done

Answer 1

I now know the solution to the problem I had back then. I had some system level corruption, because quite some time back, I made a hardshutdown when Ubuntu was frozen (because I did not know what to do otherwise). After I reinstalled Ubuntu, everything was fine again. So, my lesson out of this would be that if there is a very weird problem, for which no real solution exists, just save your data and set everything up again, without loosing too much time trying to figuring out what is the cause.