How to add metadata to log archive, so that `log show --last` works?

Question

Whenever I run the log show command from the command line, I get a warning message, log: warning: The log archive contains partial or missing metadata.

This appears to prevent me from using the --last option to limit output to the most recent time interval. The failed log command returns an exit status of 65.

% log show --last 2s --style syslog                    
log: warning: The log archive contains partial or missing metadata
log: cannot use --last when archive metadata is missing
% echo $?
65

What I want to happen is something similar to what I get when I use the --start option with a timestamp in the recent past. The successful command returns an exit status of 0.

% date
Fri 10 Jan 2020 18:43:56 PST
% log show --start "2020-01-10 18:44:00" --style syslog
log: warning: The log archive contains partial or missing metadata
Skipping info and debug messages, pass --info and/or --debug to include.
Timestamp                       (process)[PID]    
2020-01-10 18:44:04.459968-0800  localhost xpcproxy[99272]: (libsystem_info.dylib) Created Activity ID: 0x1f9010, Description: Retrieve User by ID
2020-01-10 18:44:04.460147-0800  localhost xpcproxy[99273]: (libsystem_info.dylib) Created Activity ID: 0x1f9020, Description: Retrieve User by ID
…[snip, remaining output elided]…
% echo $?
0

How can I get the log archive to contain sufficient metadata so that log show --last 2s works as I want it to?

This is observed on macOS High Sierra 10.13.6. My account has administrator access, which the log command requires to return any content. The access is demonstrated by the log --start command succeeding.

Answer 1

I have a very unsatisfying answer to this question: wait 10 days and hope the problem goes away by itself.

That approach just worked for me:

% date                             
Sun 19 Jan 2020 07:26:30 PST
% log show --last 2s --style syslog
Skipping info and debug messages, pass --info and/or --debug to include.
Timestamp                       (process)[PID]    
% echo $?
0

Note that the message, log: warning: The log archive contains partial or missing metadata no longer appears, and the --last option now works.

While investigating, I came across the very useful essay, Inside the macOS log: logd and the files that it manages, by Howard Oakley. This pointed me to the directories /var/db/diagnostics/ and /var/db/uuidtext/.

I observe that subdirectories /var/db/diagnostics/Persist/ and /var/db/diagnostics/Special/ contain opaque .tracev3 data files:

% cd /var/db/diagnostics 
% ls -lF@ Persist 
total 533688
-rw-r--r--@ 1 root  admin   2730128 21 Dec 14:02 0000000000000061.tracev3
    com.apple.logd.metadata       40 
-rw-r--r--@ 1 root  admin  10477520 22 Dec 20:45 0000000000000062.tracev3
    com.apple.logd.metadata       40 
…[snip, omitted for brevity]…
-rw-r--r--@ 1 root  admin  10479128 18 Jan 22:56 000000000000007b.tracev3
    com.apple.logd.metadata       40 
-rw-r--r--@ 1 root  admin   2431808 19 Jan 06:29 000000000000007c.tracev3
    com.apple.logd.metadata       40 
% ls -lF@ Special 
total 35624
-rw-r--r--  1 root  admin     1544 10 Jan 06:35 0000000000000048.tracev3
-rw-r--r--  1 root  admin     1536 13 Jan 04:43 0000000000000049.tracev3
-rw-r--r--  1 root  admin     1416 18 Jan 04:56 000000000000004b.tracev3
-rw-r--r--  1 root  admin   258128 19 Jan 05:25 000000000000004c.tracev3
-rw-r--r--@ 1 root  admin  2082432  8 Jan 02:58 000000000000004d.tracev3
    com.apple.logd.metadata      40 
-rw-r--r--@ 1 root  admin  2087952 10 Jan 21:52 000000000000004e.tracev3
    com.apple.logd.metadata      40 
…[snip, omitted for brevity]…
-rw-r--r--@ 1 root  admin  2097344 18 Jan 23:11 0000000000000054.tracev3
    com.apple.logd.metadata      40 
-rw-r--r--@ 1 root  admin   984016 19 Jan 06:27 0000000000000055.tracev3
    com.apple.logd.metadata      40 

Oakley says that the macOS log system rotates these files: gradually prunes out old and unnecessary content, then deletes old files altogether. The date stamps back him up. Note that the oldest tracev3 file in Special/ is only 10 days old. Older files existed in this directory when I originally had the problem, but are now gone.

It is interesting that many (but not all) of these files contain an extended file attribute "com.apple.logd.metadata". Oakley, in xattr: com.apple.logd.metadata, log metadata, says "their content and function are unknown". I speculate that the "metadata" term in the error message might be a reference to the data in the "com.apple.logd.metadata" xattr.

So, a wild guess is that a file in /var/db/diagnostics/Special/ had a corrupted or missing xattr "com.apple.logd.metadata". In the course of a week, this file became old enough and was discarded. Once that file was discarded, its metadata was gone, and the so was the problem.