bash: how to update an associative array in an implicit subshell?

Question

The problem: I cannot update an array in a while loop. An illustration (not the actual problem):

declare -A wordcounts
wordcounts["sentinel"]=1000
ls *.txt | while read f; do
  # assume that that loop runs multiple times
  wordcounts[$f]=$(wc -w  $f)
  echo ${wordcounts[$f]}  # this prints actual data
done
echo ${!wordcounts[@]}  # only prints 'sentinel'

This does not work, because the loop after the pipe runs in a subshell. All the changes that the loop does to variable wordcounts are only visible inside the loop.

Saying export wordcounts does not help.

Alas, I seem to need the pipe and the while read part, so ways to rewrite the code above using for is not what I'm looking for.

Is there a legitimate way to update an associative array form within a loop, or a subshell in general?

Answer 1

Since you have a complex command pipe you're reading from, you can use the following:

while read f; do
    # Do stuff
done < <(my | complex | command | pipe)

The syntax <(command) runs a command in a subshell and opens its stdout as a temporary file. You can use it any place where you would normally use a file in a command.

Further, you can also use the syntax >(command) to open stdin as a file instead.

Answer 2

If you are using bash 4.2, you can set the lastpipe shell option to allow the while loop, as the last element in the pipeline, to run in the current shell instead of a subshell.

A simple demonstration:

$ echo foo | read word
$ echo $word

$ set +m  # Only needed in an interactive shell to disable job control
$ shopt -s lastpipe
$ echo foo | read word
$ echo $word
foo

Answer 3

Is there a legitimate way to update an associative array form within a loop, or a subshell in general?

You could avoid a subshell by saying:

while read f; do
  ...
done < *.txt

That said, you sample code has problems otherwise. The loop would read the file line by line, so saying

wordcounts[$f]=$(wc -w  $f)

wouldn't really make much sense. You probably wanted to say:

wordcounts[$f]=$(wc -w <<< $f)

EDIT:

Alas, I seem to need the pipe ...

Quoting from the manual:

Each command in a pipeline is executed in its own subshell (see Command Execution Environment).

Answer 4

Why use ls unnecessarily.

Following works fine:

declare -a wordcounts
for f in *.txt; do
    wordcounts+=$(wc -w  $f)
done
echo ${wordcounts[@]} 

Answer 5

I think the best solution is the one by Cookyt:

while read f; do
    # Do stuff
done < <(my | complex | command | pipe)

For me, that didn't work because in my environment I don't have /proc mounted because the <(cmd) construct needs /dev/fd/XXX and /dev/fd is a symlink to /proc/self/fd. In those cases, the solution by by chepner works:

shopt -s lastpipe
my | complex | command | pipe | while read f; do
    # Do stuff
done

If you also don't have bash, there is a third solution that works with POSIX shells (and thus also with bash):

set -- $(my | complex | command | pipe)
while [ -n "$1" ]; do
    f="$1"
    shift
    # Do stuff
done