Bash: Export functions for use in xargs

Question

When my bash scripts start getting complex, I usually break them up into functions. This applies especially to complex pipes, as a sequence of complicated pipe commands (e.g. containing while-loops) can quickly become hard to read. Even more so when parallelization is wanted, where xargs is very helpful.

I know that I can export functions to a subshell with export -f, thus in a simple case I can do

export -f myfunction 
some-command | xargs -Iline bash -c "myfunction 'line'"

but if the myfunction depends on other functions this becomes hard to maintain -- every time the function changes such that the functions needed by the subshell for executing myfunction change, the export statement would have to be changed -- that seems pretty error prone.

Is there some general way to export functions for use by subshells? I was thinking about something along the lines of an "export all defined functions" command, which would then allow a code structure like

main() { ... }
func1 () { ... }
func2 () { ... }
<export all functions>
main "$@"

Answer 1

Your question asks only about exporting functions. This is easy in bash, see below.

Your question title/subject implies using functions in xargs, as though they were a script; I don't know that xargs can "call" a bash function directly, but you can of course wrap your use of the exported function(s) in a script called by xargs, see below.

First, a function to list functions. User functions by default and -v to list all functions:

lsfns () {
   case "$1" in
      -v | v*)
         # verbose:
         set | grep '()' --color=always
         ;;
      *)
         declare -F | cut -d" " -f3 | egrep -v "^_"
         ;;
   esac
}

Next a function to export all user functions:

exportfns () { export -f $(lsfns); }

or just put export -f $(lsfns) in your .bashrc.

Example script doit.sh:

#!/bin/bash
lsfns "$@" # make use of function exported by parent shell :)

Example command line (after chmod a+rx doit.sh):

echo -v | xargs doit.sh

Compare with

echo "" | xargs doit.sh

---

EDIT 1: responding further to kdb's comment/answer below ("running into situations where exporting functions does not work at all"):

Export of shell functions is not Posix compatible - i.e. it only works with Bash and presumably other shells such as Zsh, Ksh etc.

That is, in Dash, and "standard" Posix shells not providing "export -f", we cannot export functions, AND if we export a function say in Bash, then run a script which starts with the sh-bang e.g. "#!/bin/dash", that script will NOT be able to use the "exported" functions from the parent shell, since functions exported to the "process environment" by Bash, are not recognised by Dash.

---

EDIT 2: responding further to OP comment "but if the myfunction depends on other functions this becomes hard to maintain":

This is probably a situation where one could make good use of the shell source command (alias "."), e.g.:

. ~/etc/my-functions.sh; myMain ...

And similarly, if you "live" in functions rather than script files, e.g. by calling myMain when you need to, then the first line of this function can be to source your function library;

since this would be excess overhead in the "running a script regularly" case, myMain becomes the command-line stub function, which (re)loads your function library, and calls the actuallyDoit function (which would also be called from inside your script, if you have a script file).

Enjoy
Zenaan

Answer 2

This seems to work to print all the function names. It feels fragile, so test it out

declare -f | grep -oP '^\S+(?=\s*\(\))'

Answer 3

export -f $(compgen -A function)

Answer 4

Since receiving the answer, I found many cases where a different technique proofed preferable: Making the script invoke itself. A simple example would be

# Print hash and disk usage for each argument
if [[ $1 == run ]]; then
  shift 1
  printf "%s\0" "$@" | xargs -0 -n 1 -P "$NUMBER_OF_PROCESSORS" -- "$0" printpar
elif [[ $1 == printpar ]]; then
  echo ":: $(cat "$2" | sha1sum) $(du -sh "$2")"
else
  echo "Invalid first parameter '$1'"
  exit 1
fi

In real-world examples I'd either make assumptions about the arguments (e.g. using __SUCH__ a shape for self-call keywords) or hide recursive invocations behind an undocumented --command-line-switch.

Exporting functions is generally more elegant, but for large numbers of function it can get prohibitively slow and I remember running into issues, where it failed entirely.