How to use awk to generate fixed width columns with colored input?

Question

I am taking the output from 'ls -l' and passing it through awk to reformat it. This works:

list=$(ls --color=none -l | tail -n+2)
printf '%s' "$list" | awk '{printf "%-40s more stuff\n", $9}'

It produces something like:

env_profiles                             more stuff
ls_test.sh                               more stuff
saddfasfasfdfsafasdf                     more stuff
test                                     more stuff

But with --color=always it produces:

env_profiles                 more stuff
ls_test.sh                   more stuff
saddfasfasfdfsafasdf             more stuff
test                             more stuff
                                         more stuff

"env_profiles" is a directory, "ls_test.sh" is an executable file, so they are both colored and end up with different alignment. Also there is an extra line.

EDIT: Modified answer based on Ed Morton's post. Gets rid of extra line, handles filenames with spaces:

ls --color=always -l | tail -n+2 | awk '
{
    $1=$2=$3=$4=$5=$6=$7=$8=""
    field = substr($0,9)
    nameOnly = $0
    gsub(/\x1b[^m]+m/,"",nameOnly)
    if( length(field) - length(nameOnly) >= 0 ) {
        printf "%-*s more stuff\n", 40 + length(field) - length(nameOnly), field
    }
}'

Answer 1

The field ($9) that contains your colored file names starts and ends with control characters to produce the color on your terminal, e.g. in this case foo is colored on the screen but bar is not:

$ cat o1
-rwxr-xr-x  1 emorton Domain Users     21591 Nov 12  2011 foo
-rwxr-xr-x  1 emorton Domain Users     21591 Nov 12  2011 bar

$ cat -v o1
-rwxr-xr-x  1 emorton Domain Users     21591 Nov 12  2011 ^[[01;32mfoo^[[0m
-rwxr-xr-x  1 emorton Domain Users     21591 Nov 12  2011 bar

so when you printf that field in awk and give it a field width of N characters, the color-producing strings are counted as part of the width but then if they are non-printing or backspaces or whatever then the end result will not not show them and it'll look like it's using less space a field that did not contain those characters. Hope that makes sense.

It looks to me like the coloring strings always start with the character \x1b then some coloring instruction and end with m so try this:

$ awk '{
    nameOnly = $NF
    gsub(/\x1b[^m]+m/,"",nameOnly)
    printf "<%-*s>\n", 10 + length($NF) - length(nameOnly), $NF
}' o1
<foo       >
<bar       >

Note that your approach of using a specific field only works if there's no spaces in the file names

Answer 2

ls --color=always -l | tail -n+2 | awk '{count = gsub(/\x1b/, "\x1b"); if (count == 0) count += 40; else count += 50; printf "%-"count"s more stuff\n", $9}'

Explanation

gsub returns the number of substitutions made on a line. In this case we are substituting the escape character \x1b with itself, storing the number of times it appears.

If there were no escape sequences found (count == 0) we add 40 spaces of padding.
If there were escape sequences on the line (ie. color), we add 40 spaces of padding plus an additional 10 spaces. And of course, count is already equal to the number of escapes.

I found that on my system, if a line has color, it requires 10 more spaces of padding, plus the number of escape characters to match the uncolored lines. For example: normal padding = 40; there were 3 escape sequence on the line; padding should be 40 + 10 + 3 = 53. This may be different on your system and may require adjustment of the numbers.

Finally, we print the line, substituting count for the padding.