Run a shell script with While condition in an infinite loop based on conditions

Question

I need to create a shell script to place some indicator/flag files in a directory say /dir1/dir2/flag_file_directory based on the request flags received from a shell script in a directory /dir1/dir2/req_flag_file_directory and the source files present in a directory say dir1/dir2/source_file_directory. For this I need to run a script using a while condition in an infinite loop as I do not know when the source files will be made available.

So, my implementation plan is somewhat like this - Lets say JOB1 which is scheduled to run at some time in the morning will first place(touch) a request flag (eg. touch /dir1/dir2/req_flag_file_directory/req1.req), saying that this job is running, so look for the Source files of pattern file_pattern_YYYYMMDD.CSV (the file patterns are different for different jobs) present in the source file directory, if they are present, then count the number. If the count of the files is correct, then first delete the request flag for this job and then touch a indicator/flag file in the /dir1/dir2/flag_file_directory. This indicator/flag file will then be used as an indicator that the source files are all present and the job can be continued to load these files into our system.

I will have all the details related to the jobs and their flag files in a file whose structure is as shown below. Based on the request flag, the script should know what other criterias it should look for before placing the indicator file:

request_flags|source_name|job_name|file_pattern|file_count|indicator_flag_file
req1.req|Sourcename1|jobname1|file_pattern_1|3|ind1.ind
req2.req|Sourcename2|jobname2|file_pattern_2|6|ind2.ind
req3.req|Sourcename3|jobname3|file_pattern_3|1|ind3.ind
req**n**.req|Sourcename**n**|jobname**n**|file_pattern_**n**|2|ind**n**.ind

Please let me know how this can be achieved and also if you have other suggestions or solutions too

Answer 1

Still having some doubts about the contents of requests file, but I think I've come up with a rather simple solution:

#!/bin/bash

DETAILS_FILE="details.txt" 
DETAILS_LINES=$((`wc -l $DETAILS_FILE|awk '{print $1}'`-1)) # to remove banner line (first line)
DETAILS=`tail -$DETAILS_LINES $DETAILS_FILE|tr '\n\r' ' '`
PIDS=()
IFS=' '

waitall () { # PIDS...
  ## Wait for children to exit and indicate whether all exited with 0 status.
  local errors=0
  while :; do
    debug "Processes remaining: $*"
    for pid in $@; do
      echo "PID: $pid"
      shift
      if kill -0 "$pid" 2>/dev/null; then
        debug "$pid is still alive."
        set -- "$@" "$pid"
      elif wait "$pid"; then
        debug "$pid exited with zero exit status."
      else
        debug "$pid exited with non-zero exit status."
        ((++errors))
      fi
    done
    (("$#" > 0)) || break
    # TODO: how to interrupt this sleep when a child terminates?
    sleep ${WAITALL_DELAY:-1}
  done
  ((errors == 0))
}

debug () { echo "DEBUG: $*" >&2; }

#function to check for # of sourcefiles matching pattern in dir
#params: req3.req Sourcename3 jobname3 file_pattern_3 1 ind3.ind
check () {
  NOFILES=`find $2 -type f | egrep -c $4`
  if [ $NOFILES -eq "$5" ];then
    echo "Touching file $6. done."
    touch $6
  else
    echo "$NOFILES matching $4 pattern. exiting"
  fi
}

echo "parsing $DETAILS_FILE file..."
read -a lines <<< "$DETAILS"

for line in "${lines[@]}"
do 
    IFS='|'
    read -a ARRAY <<< "$line"
    echo "Line processed. Dispatching job ${ARRAY[2]}..."
    check ${ARRAY[@]} &
    IFS=' '
    PIDS="$PIDS $!"
    #echo $PIDS
done

waitall ${PIDS}
wait

Although not exactly in a infinite loop. This script is intended to run in a crontab.

First it reads details.txt file, as per your example.

After parsing all details, this script dispatches the check function, with sole purpose of counting the number of files matching file_pattern of each source_name folder, and if the number of files is equal to file_count, then touches the indicator_flag_file.

Hope that helps!

Answer 2

Rather have the service daemon script polling in an infinite loop (i.e. waking up periodically to check if it needs to do work), you could use file locking and a named pipe to create an event queue.

Outline of the service daemon, daemon.sh. This script will loop infinitely, blocking by reading from the named pipe at read line until a message arrives (i.e., some other process writes to $RequestPipe).

#!/bin/bash
#   daemon.sh

LockDir="/dir1/dir2/req_flag_file_directory"
LockFile="${LockDir}/.MultipleWriterLock"
RequestPipe="${LockDir}/.RequestQueue"

while true ; do

    if read line < "$RequestPipe" ; then
        # ... commands to be executed after message received ...
        echo "$line"            # for example
    fi

done

An outline of requestor.sh, the script that wakes up the service daemon when everything is ready. This script does all the preparation necessary, e.g. creating files in req_flag_file_directory and source_file_directory, then wakes the service daemon script by writing to the named pipe. It could even send a message that that contains more information for the service daemon, say "Job 1 ready".

#!/bin/bash
#   requestor.sh

LockDir="/dir1/dir2/req_flag_file_directory"
LockFile="${LockDir}/.MultipleWriterLock"
RequestPipe="${LockDir}/.RequestQueue"

# ... create all the necessary files ...

(
    flock --exclusive 200
    #   Unblock the service daemon/listener by sending a line of text.
    echo Wake up sleepyhead. > "$RequestPipe"
) 200>"$LockFile"       # subshell exit releases lock automatically

daemon.sh fleshed out with some error handling:

#!/bin/bash
#   daemon.sh

LockDir="/dir1/dir2/req_flag_file_directory"
LockFile="${LockDir}/.MultipleWriterLock"
RequestPipe="${LockDir}/.RequestQueue"
SharedGroup=$(echo need to put a group here 1>&2; exit 1)


#
if [[ ! -w "$RequestPipe" ]] ; then
    #    Handle 1st time. Or fix a problem.
    mkfifo --mode=775 "$RequestPipe"
    chgrp "$SharedGroup" "$RequestPipe"
    if [[ ! -w "$RequestPipe" ]] ; then
        echo "ERROR: request queue, can't write to $RequestPipe" 1>&2
        exit 1
    fi
fi

while true ; do

    if read line < "$RequestPipe" ; then
        # ... commands to be executed after message received ...
        echo "$line"        # for example
    fi

done

requestor.sh fleshed out with some error handling:

#!/bin/bash
#   requestor.sh

LockDir="/dir1/dir2/req_flag_file_directory"
LockFile="${LockDir}/.MultipleWriterLock"
RequestPipe="${LockDir}/.RequestQueue"
SharedGroup=$(echo need to put a group here 1>&2; exit 1)

# ... create all the necessary files ...

#
if [[ ! -w "$LockFile" ]] ; then
    #    Handle 1st time. Or fix a problem.
    touch "$LockFile"
    chgrp "$SharedGroup" "$LockFile"
    chmod 775 "$LockFile"
    if [[ ! -w "$LockFile" ]] ; then
        echo "ERROR: write lock, can't write to $LockFile" 1>&2
        exit 1
    fi
fi
if [[ ! -w "$RequestPipe" ]] ; then
    #    Handle 1st time. Or fix a problem.
    mkfifo --mode=775 "$RequestPipe"
    chgrp "$SharedGroup" "$RequestPipe"
    if [[ ! -w "$RequestPipe" ]] ; then
        echo "ERROR: request queue, can't write to $RequestPipe" 1>&2
        exit 1
    fi
fi

(
    flock --exclusive 200 || {
        echo "ERROR: write lock, $LockFile flock failed." 1>&2
        exit 1
    }
    #   Unblock the service daemon/listener by sending a line of text.
    echo Wake up sleepyhead. > "$RequestPipe"

) 200> "$LockFile"      # subshell exit releases lock automatically