Shell one liner to prepend to a file

Question

This is probably a complex solution.

I am looking for a simple operator like ">>", but for prepending.

I am afraid it does not exist. I'll have to do something like

 mv myfile tmp
 cat myheader tmp > myfile

Anything smarter?

Answer 1

The hack below was a quick off-the-cuff answer which worked and received lots of upvotes. Then, as the question became more popular and more time passed, outraged people started reporting that it sorta worked but weird things could happen, or it just didn't work at all, so it was furiously downvoted for a time. Such fun.

The solution exploits the exact implementation of file descriptors on your system and, because implementation varies significantly between nixes, it's success is entirely system dependent, definitively non-portable, and should not be relied upon for anything even vaguely important.

Now, with all that out of the way the answer was:

---

Creating another file descriptor for the file (exec 3<> yourfile) thence writing to that (>&3) seems to overcome the read/write on same file dilemma. Works for me on 600K files with awk. However trying the same trick using 'cat' fails.

Passing the prependage as a variable to awk (-v TEXT="$text") overcomes the literal quotes problem which prevents doing this trick with 'sed'.

#!/bin/bash
text="Hello world
What's up?"

exec 3<> yourfile && awk -v TEXT="$text" 'BEGIN {print TEXT}{print}' yourfile >&3

Answer 2

This still uses a temp file, but at least it is on one line:

echo "text" | cat - yourfile > /tmp/out && mv /tmp/out yourfile

Credit: BASH: Prepend A Text / Lines To a File

Answer 3

echo '0a
your text here
.
w' | ed some_file

ed is the Standard Editor! http://www.gnu.org/fun/jokes/ed.msg.html

Answer 4

John Mee: your method is not guaranteed to work, and will probably fail if you prepend more than 4096 byte of stuff (at least that's what happens with gnu awk, but I suppose other implementations will have similar constraints). Not only will it fail in that case, but it will enter an endless loop where it will read its own output, thereby making the file grow until all the available space is filled.

Try it for yourself:

exec 3<>myfile && awk 'BEGIN{for(i=1;i<=1100;i++)print i}{print}' myfile >&3

(warning: kill it after a while or it will fill the filesystem)

Moreover, it's very dangerous to edit files that way, and it's very bad advice, as if something happens while the file is being edited (crash, disk full) you're almost guaranteed to be left with the file in an inconsistent state.

Answer 5

Not possible without a temp file, but here goes a oneliner

{ echo foo; cat oldfile; } > newfile && mv newfile oldfile

You can use other tools such as ed or perl to do it without temp files.

Answer 6

It may be worth noting that it often is a good idea to safely generate the temporary file using a utility like mktemp, at least if the script will ever be executed with root privileges. You could for example do the following (again in bash):

(tmpfile=`mktemp` && { echo "prepended text" | cat - yourfile > $tmpfile && mv $tmpfile yourfile; } )

Answer 7

If you need this on computers you control, install the package "moreutils" and use "sponge". Then you can do:

cat header myfile | sponge myfile

Answer 8

Using a bash heredoc you can avoid the need for a tmp file:

cat <<-EOF > myfile
  $(echo this is prepended)
  $(cat myfile)
EOF

This works because $(cat myfile) is evaluated when the bash script is evaluated, before the cat with redirect is executed.

Answer 9

assuming that the file you want to edit is my.txt

$cat my.txt    
this is the regular file

And the file you want to prepend is header

$ cat header
this is the header

Be sure to have a final blank line in the header file.
Now you can prepend it with

$cat header <(cat my.txt) > my.txt

You end up with

$ cat my.txt
this is the header
this is the regular file

As far as I know this only works in 'bash'.

Answer 10

When you start trying to do things that become difficult in shell-script, I would strongly suggest looking into rewriting the script in a "proper" scripting language (Python/Perl/Ruby/etc)

As for prepending a line to a file, it's not possible to do this via piping, as when you do anything like cat blah.txt | grep something > blah.txt, it inadvertently blanks the file. There is a small utility command called sponge you can install (you do cat blah.txt | grep something | sponge blah.txt and it buffers the contents of the file, then writes it to the file). It is similar to a temp file but you dont have to do that explicitly. but I would say that's a "worse" requirement than, say, Perl.

There may be a way to do it via awk, or similar, but if you have to use shell-script, I think a temp file is by far the easiest(/only?) way..

Answer 11

EDIT: This is broken. See Weird behavior when prepending to a file with cat and tee

The workaround to the overwrite problem is using tee:

cat header main | tee main > /dev/null

Answer 12

Like Daniel Velkov suggests, use tee.
To me, that's simple smart solution:

{ echo foo; cat bar; } | tee bar > /dev/null

Answer 13

The one which I use. This one allows you to specify order, extra chars, etc in the way you like it:

echo -e "TEXTFIRSt\n$(< header)\n$(< my.txt)" > my.txt

P.S: only it's not working if files contains text with backslash, cause it gets interpreted as escape characters

Answer 14

Mostly for fun/shell golf, but

ex -c '0r myheader|x' myfile

will do the trick, and there are no pipelines or redirections. Of course, vi/ex isn't really for noninteractive use, so vi will flash up briefly.

Answer 15

Why not simply use the ed command (as already suggested by fluffle here)?

ed reads the whole file into memory and automatically performs an in-place file edit!

So, if your file is not that huge ...

# cf. "Editing files with the ed text editor from scripts.",
# http://wiki.bash-hackers.org/doku.php?id=howto:edit-ed

prepend() {
   printf '%s\n' H 1i "${1}" . wq | ed -s "${2}"
}

echo 'Hello, world!' > myfile
prepend 'line to prepend' myfile

Yet another workaround would be using open file handles as suggested by Jürgen Hötzel in Redirect output from sed 's/c/d/' myFile to myFile

echo cat > manipulate.txt
exec 3<manipulate.txt
# Prevent open file from being truncated:
rm manipulate.txt
sed 's/cat/dog/' <&3 > manipulate.txt

All this could be put on a single line, of course.

Answer 16

A variant on cb0's solution for "no temp file" to prepend fixed text:

echo "text to prepend" | cat - file_to_be_modified | ( cat > file_to_be_modified ) 

Again this relies on sub-shell execution - the (..) - to avoid the cat refusing to have the same file for input and output.

Note: Liked this solution. However, in my Mac the original file is lost (thought it shouldn't but it does). That could be fixed by writing your solution as: echo "text to prepend" | cat - file_to_be_modified | cat > tmp_file; mv tmp_file file_to_be_modified

Answer 17

Here's what I discovered:

echo -e "header \n$(cat file)" >file

Answer 18

sed -i -e '1rmyheader' -e '1{h;d}' -e '2{x;G}' myfile

Answer 19

WARNING: this needs a bit more work to meet the OP's needs.

There should be a way to make the sed approach by @shixilun work despite his misgivings. There must be a bash command to escape whitespace when reading a file into a sed substitute string (e.g. replace newline characters with '\n'. Shell commands vis and cat can deal with nonprintable characters, but not whitespace, so this won't solve the OP's problem:

sed -i -e "1s/^/$(cat file_with_header.txt)/" file_to_be_prepended.txt

fails due to the raw newlines in the substitute script, which need to be prepended with a line continuation character () and perhaps followed by an &, to keep the shell and sed happy, like this SO answer

sed has a size limit of 40K for non-global search-replace commands (no trailing /g after the pattern) so would likely avoid the scary buffer overrun problems of awk that anonymous warned of.

Answer 20

sed -i -e "1s/^/new first line\n/" old_file.txt

Answer 21

With $( command ) you can write the output of a command into a variable. So I did it in three commands in one line and no temp file.

originalContent=$(cat targetfile) && echo "text to prepend" > targetfile && echo "$originalContent" >> targetfile

Answer 22

If you have a large file (few hundred kilobytes in my case) and access to python, this is much quicker than cat pipe solutions:

python -c 'f = "filename"; t = open(f).read(); open(f, "w").write("text to prepend " + t)'

Answer 23

A solution with printf:

new_line='the line you want to add'
target_file='/file you/want to/write to'

printf "%s\n$(cat ${target_file})" "${new_line}" > "${target_file}"

You could also do:

printf "${new_line}\n$(cat ${target_file})" > "${target_file}"

But in that case you have to be sure there aren’t any % anywhere, including the contents of target file, as that can be interpreted and screw up your results.

Answer 24

You can use perl command line:

perl -i -0777 -pe 's/^/my_header/' tmp

Where -i will create an inline replacement of the file and -0777 will slurp the whole file and make ^ match only the beginning. -pe will print all the lines

Or if my_header is a file:

perl -i -0777 -pe 's/^/`cat my_header`/e' tmp

Where the /e will allow an eval of code in the substitution.

Answer 25

current=`cat my_file` && echo 'my_string' > my_file && echo $current >> my_file

where "my_file" is the file to prepend "my_string" to.

Answer 26

I'm liking @fluffle's ed approach the best. After all, any tool's command line switches vs scripted editor commands are essentially the same thing here; not seeing a scripted editor solution "cleanliness" being any lesser or whatnot.

Here's my one-liner appended to .git/hooks/prepare-commit-msg to prepend an in-repo .gitmessage file to commit messages:

echo -e "1r $PWD/.gitmessage\n.\nw" | ed -s "$1"

Example .gitmessage:

# Commit message formatting samples:
#       runlevels: boot +consolekit -zfs-fuse
#

I'm doing 1r instead of 0r, because that will leave the empty ready-to-write line on top of the file from the original template. Don't put an empty line on top of your .gitmessage then, you will end up with two empty lines. -s suppresses diagnostic info output of ed.

In connection with going through this, I discovered that for vim-buffs it is also good to have:

[core]
        editor = vim -c ':normal gg'

Answer 27

variables, ftw?

NEWFILE=$(echo deb http://mirror.csesoc.unsw.edu.au/ubuntu/ $(lsb_release -cs) main universe restricted multiverse && cat /etc/apt/sources.list)
echo "$NEWFILE" | sudo tee /etc/apt/sources.list

Answer 28

I think this is the cleanest variation of ed:

cat myheader | { echo '0a'; cat ; echo -e ".\nw";} | ed myfile

as a function:

function prepend() { { echo '0a'; cat ; echo -e ".\nw";} | ed $1; }

cat myheader | prepend myfile

Answer 29

If you're scripting in BASH, actually, you can just issue:

cat - yourfile  /tmp/out && mv /tmp/out yourfile

That's actually in the Complex Example you yourself posted in your own question.

Answer 30

IMHO there is no shell solution (and will never be one) that would work consistently and reliably whatever the sizes of the two files myheader and myfile. The reason is that if you want to do that without recurring to a temporary file (and without letting the shell recur silently to a temporary file, e.g. through constructs like exec 3<>myfile, piping to tee, etc.

The "real" solution you are looking for needs to fiddle with the filesystem, and so it's not available in userspace and would be platform-dependent: you're asking to modify the filesystem pointer in use by myfile to the current value of the filesystem pointer for myheader and replace in the filesystem the EOF of myheader with a chained link to the current filesystem address pointed by myfile. This is not trivial and obviously can not be done by a non-superuser, and probably not by the superuser either... Play with inodes, etc.

You can more or less fake this using loop devices, though. See for instance this SO thread.