Efficient way to map changes in a filesystem hierarchy

Question

I'm currently working on a project that will enable file searching based on metadata found in the file.

It'll be comprised of 2 parts:

• a filesystem crawler that passively scans for changes and indexes the results in a database
• a search front-end that queries the database

The database will use a pseudo-hierarchical (ie flattened) model to create a map of the directory/file structure.

The fields stored in the index will include:

• file/folder name
• parent directory
• modified time
• metadata
  • ...

What I'm wondering is, how does modified time on files/folders typically work and how can it be used to reduce the area that the crawler will have to index?

Note: assume the crawler will regularly run on a fixed interval to keep the index up-to-date.

Answer 1

I assume you speak about unix-like OS. From Wikipedia (stat()):

stat() is a Unix system call that returns file attributes about an inode. The semantics of stat() vary between operating systems. As an example, the Unix command ls uses it to retrieve information on (among many others):

• atime: time of last access (ls -lu),
• mtime: time of last modification (ls -l), and
• ctime: time of last status change (ls -lc).

From man 2 stat:

The field st_mtime is changed by file modifications, for example, by mknod(2), truncate(2), utime(2) and write(2) (of more than zero bytes). Moreover, st_mtime of a directory is changed by the creation or deletion of files in that directory. The st_mtime field is not changed for changes in owner, group, hard link count, or mode.

That means that last modification time is set on a file when:

• file is created (mknod)
• file is truncated (truncate)
• modification time is changed explicitly (utime)
• one or more bytes of data was written into a file (write) - that includes cases when same data was written twice and nothing actually changed

I am not sure if it is a complete list.

---

As for mtime on a directory:

• It changes when you create, move or delete a file or a directory which is immediate child of this directory. Simple heuristic: mtime of a directory changes when the output of ls (just plain command, no recursion or any additional parameters) changes.
• It does not change when you only modify the contents of any files inside (no matter how deeply nested), or do anything inside its' subdirectories. No propagation.

---

I'm wondering if storing the data you've mentioned in a DB would be of any help - it is already stored in a file system, which is a database by itself, and all the tools for querying it are already there.