How to create an .IMG image of a disc (sd card) without including free space?

Question 1

The best thing to do is

Copy all the files from all the partitions preserving meta data

mkdir -p myimage/partition1

mkdir myimage/partition2

sudo cp -rf --preserve=all /media/mount_point_partition1/* myimage/partition1/

sudo cp -rf --preserve=all /media/mount_point_partition2/* myimage/partition2/
Extract the MBR

sudo dd if=/dev/sdX of=myimage/mbr.img bs=446 count=1

replace /dev/sdX with the corresponding device.
Partition the destination disk into partitions with sizes greater than copied data and should be of the same format and same flags using gparted. Google how to partition a disk.
Mount the freshly formatted and partitioned disk. On most computers, you just need to connect the disk and you can find the mounted partitions in /media folder.
Copy the previously copied data to destination partitions using following commands

sudo cp -rf --preserve=all myimage/partition1/* /media/mount_point_partition1/

sudo cp -rf --preserve=all myimage/partition2/* /media/mount_point_partition2/
Copy back the MBR

sudo dd if=myimage/mbr.img of=/dev/sdX bs=446 count=1

Now njoy Ur new disk!

Question 2

Pretty good and simple way to deal with this is simply pipe it via gzip, something like this:

# dd if=/dev/sdb | gzip > backup.img.gz

This way your image will be compressed and most likely unused space will be squeezed to almost nothing.

You would use this to restore such image back:

# cat backup.img.gz | gunzip | dd of=/dev/sdb

One note: if you had a lot of files which were recently deleted, image size may be still large (deleting file does not necessarily zeroes underlying sectors). You can wipe free space by creating and immediately deleting large file containing zeros:

# cd /media/flashdrive
# dd if=/dev/zero of=bigfile bs=1M     # let it run and quit by disk full error
# rm bigfile

Question 3

Using the bs and count parameters of dd, you can limit the size of the image, as seen in step 2 of answer 1665017.

You may already know what size image you want to create. If not, you can get a good idea from df:

df -H --total /

Substitute / with a space-separated list of all the mount points relating to the disk partitions.

A more accurate way might be to use fdisk or your preferred partition editor and get busy with a calculator.

$ fdisk -l /dev/mmcblk0

Disk /dev/mmcblk0: 7.4 GiB, 7948206080 bytes, 15523840 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x00057540

Device         Boot  Start     End Sectors  Size Id Type
/dev/mmcblk0p1        2048  186367  184320   90M  c W95 FAT32 (LBA)
/dev/mmcblk0p2      186368 3667967 3481600  1.7G  5 Extended
/dev/mmcblk0p5      188416 3667967 3479552  1.7G 83 Linux

Total used space in bytes = end sector of last partition X sector size (here that's 3667967 x 512).

Total used space in GB = total used space in bytes / 1024³ (here that's 1.749023 GB).

If you decide, for example, that your image should be exactly 2 GB, the following command will do that:

dd if=/dev/mmcblk0 of=/path/to/pi_updated.img bs=1M count=2048

The resulting image will also include the random noise beyond the greatest extent of the last partition.

Don't forget to sudo the above commands if your account doesn't already have sufficient privileges.

For my purposes, I don't need an image that is perfectly trimmed down to the last bit of data so when the real size is 1.75 GB then a 2 GB image is near enough for me. This cuts out the other 6 GB (or 30 GB or whatever the device has spare) of unused space that would otherwise be in the image.

I have seen advice in many places that dd should not be performed on a mounted partition and I followed that because it seems intuitively correct; it does seem rather like trying to sketch yourself making a sketch in a mirror with the sketch you're making also visible in the sketch. It's a bit sketchy.

Question 4

After trying multiple different methods, I found the following article:

https://medium.com/platformer-blog/creating-a-custom-raspbian-os-image-for-production-3fcb43ff3630

It's created to shrink and resize (on first boot) a raspberry-pi image but could be easily adjusted for any other Linux distribution. I successfully got it working with Debian 9 on a custom arm based chip.

The rc.local created by the pishrink script first uses raspi-config to resize the rootfs and then falls back to a method using parted (which I had to install ahead of time on my machine). I commented out the section of code where raspi-config.

My sd-card image was shrunk from 15 GB to 1.1 GB. I flashed the shrunk sd-card image with etcher. Took less than 5 mins as opposed to over half an hour for the full 15 GB image

Question 5

If you have a big SD card 16 GB, 32 GB etc but you want to save space with backup you can use:

sudo apt-get install gnome-disk-utility

Open disk utility to check witch letter is your usb drive actually has:

gnome-disks

In my case a 32GB SD card with Raspbian image on it recognised as: /dev/sde

So I run with /dev/sde:

sudo dd bs=4M status=progress if=/dev/sde | gzip > \
/you-selected-full-path-here/raspberry-images/`date +%Y%m%d`_rpi_image_backup.gz

status=progress gives you progress bar indication
| gzip > compresses the 32 GB total size and not writing the empty space from the 32 GB
`date +%Y%m%d` writes today date in the filename

Output: 20190529_rpi_image_backup.gz

And the size is only 3.5GB. If you want to write this image to a new SD card use:

https://www.balena.io/etcher/

Also you can write this image made from 32 GB to a 16 GB or 8 GB disk, it is not complaining that the image is too large anymore.

Question 6

Sparse (Shrunken) Image Cloning Instructions:

Plug in bootable Ubuntu persistent live USB with image file in usbdata partition into Single-Board Computer such as UP² Board. Boot to USB (default persistent live option). Launch Terminal and type in the following command (single line):

sudo dd if=/media/ubuntu/usbdata/ubuntu.img of=/dev/mmcblk0 bs=1M status=progress

Wait 15-20 minutes. When completed, shutdown and disconnect USB. Boot to internal eMMC drive

Creating the Image:

Create uncompressed sparse image file from internal eMMC and save to flash drive

sudo dd if=/dev/mmcblk0 bs=1M status=progress | cp --sparse=always /dev/stdin /media/ubuntu/usbdata/ubuntu.img

Extract uncompressed image file to internal eMMC

sudo dd if=/media/ubuntu/usbdata/ubuntu.img of=/dev/mmcblk0 bs=1M status=progress

Create compressed image file from internal eMMC and save to Desktop

sudo dd if=/dev/mmcblk0 bs=1M status=progress | gzip > /home/user/Desktop/ubuntu.img.gz

Extract compressed image file to internal eMMC

sudo gzip -cd /home/user/Desktop/ubuntu.img.gz | dd of=/dev/mmcblk0 bs=1M status=progress

Sources: http://dustymabe.com/2012/11/15/create-a-disk-image-without-enough-free-space/ https://forum.up-community.org/discussion/1197/solved-cloning-the-image-of-an-up-board

To modify an existing Linux image or drive from within Linux, plug in drive or mount the image:

sudo mkdir /media/root
sudo mount -t ext4 -o loop,offset=537919488 /media/user/usbdata/ubuntu.img  /media/root

To determine the offset for your image or drive, run:

fdisk -l /media/user/usbdata/ubuntu.img

Then, multiply block-size of 512 bytes by the start-block of the Linux partition, 1050624. 512*1050624 = 537,919,488

Also mount other system folders:

for i in /dev /dev/pts /proc /sys /run; do sudo mount -B $i /media/root$i; done

Then cd into root folder of img and chroot:

cd /media/root
sudo chroot .

Then you can update files, passwd, etc. To unmount, run:

for i in /dev /dev/pts /proc /sys /run; do sudo umount /media/root$i; done
sudo umount /media/root

Creating Ubuntu Persistent Live USB Image Follow these steps to flash a USB drive with a persistent live installation of Ubuntu: https://www.howtogeek.com/howto/14912/create-a-persistent-bootable-ubuntu-usb-flash-drive/

Question 7

My solution to make an image of a customized raspios:

DIR="./img"
DEVICE=/dev/sdb

# declare custom copy functions
copy() {
  echo "[$1] copy $2 to $3..."
  if [ "$1" == 'ext' ]; then
    sudo rsync -aHAXX \
      --no-inc-recursive \
      --numeric-ids \
      --filter='-x security.selinux' \
      --delete \
      --exclude={"/lost+found"} \
      --info=progress2 \
      $2 \
      $3
  else
    sudo rsync -rtD \
      --modify-window=1 \
      --no-inc-recursive \
      --delete \
      --exclude={"/lost+found"} \
      --info=progress2 \
      $2 \
      $3
  fi
}

# create working directories
mkdir -p $DIR/{mount,image}/{boot,rootfs}

# mount initial device partitions
sudo mount ${DEVICE}1 $DIR/mount/boot
sudo mount ${DEVICE}2 $DIR/mount/rootfs

# preserve partitions UUIDs
PTUUID=0x`sudo blkid -p $DEVICE | sed -e 's/.*PTUUID="\([0-9a-f]*\)".*/\1/'`
BOOTUUID=`sudo blkid -p ${DEVICE}1 | sed -e 's/.* UUID="\([0-9a-zA-Z\-]*\)".*/\1/' | tr -d '-'`
ROOTUUID=`sudo blkid -p ${DEVICE}2 | sed -e 's/.* UUID="\([0-9a-zA-Z\-]*\)".*/\1/'`

# copy contents from physical device to local folder
copy fat $DIR/mount/boot/   $DIR/image/boot/
copy ext $DIR/mount/rootfs/ $DIR/image/rootfs/
umount ${DEVICE}1
umount ${DEVICE}2

# show contents size
BOOTSIZE=`sudo du -hs $DIR/image/boot`
ROOTSIZE=`sudo du -hs $DIR/image/rootfs`
echo "boot is $BOOTSIZE"
echo "root is $ROOTSIZE"

# allocate the final .img file
# change size as needed
SIZE="5"
echo "allocating ${SIZE}G image..."
rm $DIR/$IMAGE
fallocate -l ${SIZE}G $DIR/$IMAGE

# create partitions
echo "formatting image..."
sed -e 's/\s*\([\+0-9a-zA-Z]*\).*/\1/' << EOF | fdisk $DIR/$IMAGE > /dev/null
o          # create MBR partition scheme
n          # new partition
p          # primary
1          # partition #1
8192       # start block
532479     # end block
t          # define type
0c         # W95 FAT32 (LBA)
n          # new partition
p          # primary
2          # partition #2
532480     # start block
           # last end block
t          # define type
2          # for partition #2
83         # linux
x          # advanced menu
i          # change disk identifier
$PTUUID    # identifier from original fstab
r          # return to normal menu
w          # write changes
EOF

# locally mount the .img
echo "mounting image..."
LOOP=`sudo losetup -fP --show $DIR/$IMAGE`
echo "mounted image under $LOOP"
ls -la $LOOP*

echo "creating partition boot with FAT32"
sudo mkfs.fat \
  -F 32 \
  -S 512 \
  -i $BOOTUUID \
  -n boot \
  ${LOOP}p1

echo "creating partition rootfs with ext4"
sudo mkfs.ext4 \
  -b 4096 \
  -U $ROOTUUID \
  -L rootfs \
  ${LOOP}p2

echo "mounting image loops"
sudo mount ${LOOP}p1 $DIR/mount/boot
sudo mount ${LOOP}p2 $DIR/mount/rootfs

echo "copying contents to image"
copy fat $DIR/image/boot/   $DIR/mount/boot/
copy ext $DIR/image/rootfs/ $DIR/mount/rootfs/

echo "unmounting image"
unmount "$LOOP+" > /dev/null
echo "removing loop"
sudo losetup -d ${LOOP}