Why doesn't my sparsebundle report free space, with plenty to spare?

Question

I have a Time Machine created sparsebundle (HFS+) that reports the following limits when using hdiutil resize -limits:

 min     cur     max 
403186088   403186088   34359738368

That's a current size of 206 GB and a max of 17.5 TB. I draw the conclusion that I don't need to hdiutil resize the sparsebundle to allow it to expand. The host volume has an additional 200 GB free, so the bundle should therefore be able to expand about 100% -- however:

• Disk utility reports free space as 1 GB
• Time Machine reports free space as 1 GB

I expected them to detect 200 GB of free space available on the host volume. This is a problem since Time Machine will start deleting backups prematurely.

How do I make the sparsebundle expand when necessary to fill available space on the host file system?

Some additional questions that came up that could help to narrow it down:

1. Which part of the system is responsible for expanding the bundle when required? Does Time Machine do this explicitly or is it done transparently underneath it? I.e. is this a problem with Time Machine or with something more fundamental?
2. Is there a more reliable way to check whether Time Machine has/will run out of space than to look at the "free space" figure in Time Machine settings? I.e. can the numbers be trusted? I did remove and add the bundle as a backup destination with tmutil setdestination before checking free space.
3. If Time Machine is explicitly responsible for expanding a sparsebundle used as a backup target, will it only do this when it itself mounts the sparsebundle as a step of a networked backup? In other words, when taking the networked drive and plugging it in locally with USB3, and then manually mounting the sparsebundle, would Time Machine just skip the step where it usually expands the sparsebundle target since the bundle is now considered local storage (which doesn't generally use sparsebundles)?

MacOS Mojave 10.14.6

---

Update

There are several subtle reasons that combine into why I think my sparsebundle doesn't expand automatically, and why I form the null hypothesis that the free space reported by Time Machine is not erroneous. They include:

• I have no good reason to assume that the reported number in Time Machine and Disk Utility UI is erroneous. Conversely, I have been given no explicit assurances that the target volume will expand, nor to what extent.
• Time Machine has to my recollection failed previously to expand the sparsebundle target that it was backing up to, despite lots of available space on the host volume.
• Apple's track record of design suggests to me that if they only expose a single figure (remaining space available) in the Time Machine UI for me to base my decisions on with regards to what actions TM will take in turn, that it would've been removed already if it were an unreliable figure. I don't count the use of sparsebundles (i.e. networked backup) as such a small corner case as to warrant that level of unreliability to be ignored in their UI design phase.
• When the drive is used as a network target, TM seems to report ~200/400 GB available, while attached as a local backup target it will report ~200/200 GB available. A locally mounted sparsebundle is also not showing up in the "add target" UI in TM, but has to be added using tmutil. This suggest an inconsistency in how backup targets are handled and that Time Machine manages sparsebundles explicitly.

I realize this is not conclusive, so I tried it out:

1. I mounted the host volume over USB, and proceeded to resize the sparsebundle with hdiutil to be as small as I could make it, so that it appeared to have ~20 GB free according to all system GUI tools. It shrank by over 100 GB in that regard, so plenty of space to spare on the host volume.
2. I mounted the bundle as a local volume, and proceeded to copy an 11 GB Xcode.xip file to it twice.
3. The first transfer succeeded.
4. The second transfer failed before even starting -- "not enough free space". 1.54 GB of additional space was required. This was expected.
5. Conclusion: the sparsebundle did not expand transparently, despite over 100 GB of free space on the host volume. There is no reason for me to believe that the figure that Time Machine reports is erroneous, nor that the volume will expand transparently during TM operation.

Answer 1

Regarding your first question: It is the operating system that automatically resizes sparse bundles as necessary when you add data to them. This is done transparently underneath Time Machine, so it is not something that Time Machine explicitly handles.

The sparse bundle itself can have a maximum capacity that exceeds the available space on your drive, as it does not actually occupy any space before you write data to it. You can change resize this, but it doesn't actually change anything about how much space it takes up on the physical drive, but rather just imposes a limit on how much can eventually be written to it.

Even though Disk Utility reports it as having no real free space, it will automatically expand to fill up the drive. When the physical drive is filled up, it is ofcourse not possible to add more to the sparse bundle until you clear up space or move the bundle to a larger drive.

When you run hdiutil resize -limits my.sparsebundle you'll get a min, cur and max value. You can use the following calculations to get "human-readable" numbers for them:

  min * 512 / 1024 / 1024 = MB data stored in the sparsebundle

  cur * 512 / 1024 / 1024 = limit in MB of how much data can be added

  max * 512 / 1024 / 1024 = the largest limit in MB you can set

You can change the limit at any time using:

hdiutil resize -size 500MB my.sparsebundle

This would change the limit to 500 MB. Obviously it doesn't make sense to set the limit lower than the current amount of data stored. Perhaps a bit counter intuitive, the limit can be set higher than the amount of disk space you actually have available. The limit cannot however be set higher than the max value, which in your case is 16 TB. The reason for the 16 TB is that you have it setup with a block size of 4 kB. You can create sparse bundles with a larger block size to get a higher max value up to the theoretical maximum of just shy of 8 exabytes.

Note that this limit does not have anything to do with how much disk space is actually taken up on your physical disk. You can find this number running for example:

du -sh my.sparsebundle

If for example use Disk Utility to create an empty sparsebundle with a size of 500 MB formatted with APFS, you'll see that the above command tells you that it takes up only approx. 13 MB on the physical disk.

When you mount the sparsebundle and add files to it, the amount of space taken up on the physical disk increases. I.e. the operating system transparently expands the sparsebundle structure on the physical disk to accommodate the extra data you add to it. However, it won't add more than the arbitrary limit, you have set it.

If you select "Info" on the mounted sparsebundle in Finder, it will show you how much data has been added to the sparsebundle as well as the amount of "Free" space. This amount of "Free" space is not the amount of free space on your physical drive, but rather how much is left until you hit the (artificial) limit set on your sparsebundle, which you can change at any time later.

Answer 2

I have possibly mixed up the concepts of min, cur and max.

I had assumed that cur represented the space currently used on the host volume, and not the current total logical space of the image. In the latter case a resize would still be necessary in order to increase available logical size, which in hindsight turns out to have had the desired effect.

---

To clarify the problem for future readers (including myself):

The root cause of my problem was that I've gotten the impression that sparsebundles will "expand" automatically, without getting a clear picture of what exactly will expand.

Since I was never particularly concerned with the fact that something that's sparse will potentially occupy less space than its advertised internal space — because that's only ever to my benefit — I had just assumed that mentions of how sparsebundles can expand automatically were referring to a separate concept, e.g. expanding its internal volume's space. That's something that would have been useful to know since it would have meant that I wouldn't have to worry about that the bundle's volume was filling up, as long as there's space left on the host volume.

In other words, while I could differentiate between the bundle's physical size on disk and its internal volume's size, I thought the latter had the capability to grow automatically. The hdiutil output did not help with that misconception.

Unsurprisingly (in hindsight) this is not how it works. The volume size is fixed until manual intervention, and knowing that its physical size "expands" automatically is only marginally useful, at least for my use case.