systemd/udev dependency failure when auto mounting separate partition during startup

Question

EDIT: This occurs when I am trying to use a separate partition with ANY mount point, not just /var.

I am using Buildroot to build an embedded linux system. I am trying to use a separate partition for /var, but this occurs for ANY mount point. My init system is systemd, udev for device management. I've edited my fstab to:

# <file system> <mount pt>     <type>   <options>                <dump> <pass>
/dev/root       /              ext2     rw,noauto                0      1
/dev/mmcblk0p4  /var           ext2     defaults                 0      2
proc            /proc          proc     defaults                 0      0
devpts          /dev/pts       devpts   defaults,gid=5,mode=620  0      0
tmpfs           /dev/shm       tmpfs    mode=0777                0      0
tmpfs           /tmp           tmpfs    defaults                 0      0
sysfs           /sys           sysfs    defaults                 0      0
tmpfs           /run           tmpfs    defaults                 0      0

Once the system boots, I receive a message "Expecting device dev-mmcblk0p4.device...", eventually followed by:

"Timed out waiting for device dev-mmcblk0p4.device."

"Dependency failed for /var."

"Dependency failed for Local File Systems."

Same thing happens if I use /var2, or anything else, as the mount point instead of /var. Once the boot fails, it enters emergency mode and the mount will then successfully occur. I can then exit system maintenance and everything works as expected. What is causing this delayed mount?? Have I missed a step?

Screenshot of output:

Answer 1

I had a similar issue here using systemd-208 with kernel 3.13.1 and 3.14.4 under Exherbo (similar to Gentoo): my separate partitions (/var, /home, /boot and even /swap) were mounted under the desired root location. Everything worked fine. Then, upgrading to systemd 213 the boot process stopped at the point, where the partitions should have been mounted with the message: 'A start job is running for dev-sdxx.device' until timeout of 1 min 30 sec. Afterwards I was in Emergency Mode. Excerpt of

$ journalctl -xb

Jun 09 13:50:29 exathlon systemd[1]: Job dev-disk-by\x2dlabel-BAK_A4.device/start timed out.
Jun 09 13:50:29 exathlon systemd[1]: Timed out waiting for device dev-disk-by\x2dlabel-BAK_A4.device.
Jun 09 13:50:29 exathlon systemd[1]: Dependency failed for /mnt/BAK_A4.
Jun 09 13:50:29 exathlon systemd[1]: Dependency failed for Local File Systems.
Jun 09 13:50:29 exathlon systemd[1]: Triggering OnFailure= dependencies of local-fs.target.
Jun 09 13:50:29 exathlon systemd[1]: Dependency failed for /export/LinuxMint13_KDE_64.
Jun 09 13:50:29 exathlon systemd[1]: Dependency failed for /export/SystemRescueCD-x86.
Jun 09 13:50:29 exathlon systemd[1]: Dependency failed for File System Check on /dev/disk/by-label/BAK_A4.
Jun 09 13:50:29 exathlon systemd[1]: Job dev-sdb11.device/start timed out.
Jun 09 13:50:29 exathlon systemd[1]: Timed out waiting for device dev-sdb11.device.
Jun 09 13:50:29 exathlon systemd[1]: Dependency failed for /var.
Jun 09 13:50:29 exathlon systemd[1]: Dependency failed for Update UTMP about System Boot/Shutdown.
Jun 09 13:50:29 exathlon systemd[1]: Dependency failed for Load/Save Random Seed.
Jun 09 13:50:29 exathlon systemd[1]: Dependency failed for File System Check on /dev/sdb11.
Jun 09 13:50:29 exathlon systemd[1]: Job dev-sdb8.device/start timed out.
Jun 09 13:50:29 exathlon systemd-journal[1044]: Forwarding to syslog missed 15 messages.
Jun 09 13:50:29 exathlon systemd[1]: Timed out waiting for device dev-sdb8.device.
Jun 09 13:50:29 exathlon systemd[1]: Dependency failed for /home.
Jun 09 13:50:29 exathlon systemd[1]: Dependency failed for File System Check on /dev/sdb8.
Jun 09 13:50:29 exathlon systemd[1]: Job dev-sda6.device/start timed out.
Jun 09 13:50:29 exathlon systemd[1]: Timed out waiting for device dev-sda6.device.
Jun 09 13:50:29 exathlon systemd[1]: Dependency failed for /boot.
Jun 09 13:50:29 exathlon systemd[1]: Dependency failed for File System Check on /dev/sda6.
Jun 09 13:50:29 exathlon systemd[1]: Job dev-sdb5.device/start timed out.
Jun 09 13:50:29 exathlon systemd[1]: Timed out waiting for device dev-sdb5.device.
Jun 09 13:50:29 exathlon systemd[1]: Dependency failed for /dev/sdb5.

shows the failed attempts of mounting by systemd.

After looking up the internet I was able to find a remedy in configuring the kernel with

CONFIG_FHANDLE=y (before: n)

see 'systemd System and Service Manager' under

REQUIREMENTS:
 CONFIG_FHANDLE (libudev, mount and bind mount handling)

source: http://cgit.freedesktop.org/systemd/systemd/tree/README

Afterwards the problem was solved, all partitions mounted again.

I cannot tell, why this was not happening / or was not required with systemd 208.

Answer 2

Was able to work around this, although it is a sort of a hack. Would still love to know why this is occurring in the first place, but it appears as though udev isn't mounting mmcblk partitions until after systemd init is complete, which causes dependency errors if /etc/fstab calls for an mmcblk partition. udev checks the fstab, waits for an mmcblk device to be mounted and times out, THEN attempts to mount the device.

My Solution:

1) Create systemd service to handle mounting the partition:

#/etc/systemd/system/mount-data-partition.service
[Unit]
Description=Mount Data Partition
DefaultDependencies=no

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/bin/mount /dev/mmcblk0p4

2) Add a Wants dependency to this service within systemd-udev-trigger.service:

#/usr/lib/systemd/system/systemd-udev-trigger.service
[Unit]
Description=udev Coldplug all Devices
Documentation=man:udev(7) man:systemd-udevd.service(8)
DefaultDependencies=no
Wants=systemd-udevd.service mount-data-partition.service
After=systemd-udevd-kernel.socket systemd-udevd-control.socket
Before=sysinit.target
ConditionCapability=CAP_MKNOD

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/bin/udevadm trigger --type=subsystems --action=add ; /usr/bin/udevadm trigger

This causes mount-data-partition.service to be called by and executed before systemd-udev-trigger.service. The mount command will then look for /dev/mmcblk0p4 in /etc/fstab, and mount as specified (in my situation, /var). Since /dev/mmcblk0p4 is now mounted, udev recognizes that the device exists and no longer times out while waiting for it. System continues to boot as normal.

I hate to answer my own question, but hopefully this information helps someone out. If someone can enlighten me on why this is occurring in the first place, please do.