Quality Control: Check file names in a folder against filenames in a table

Question

I have a records system in an Oracle database. The records system is used to retrieve pdf, jpeg, video documents, etc.

The design is quite simple: there is a table called infrastructure_records that has a column with hyperlinks to files:

+------------+------------------------------------------+
| project_id |                file_path                 |
+------------+------------------------------------------+
| Project 1  | X:\Records_System\Folder A\Project 1.pdf |
| Project 2  | X:\Records_System\Folder B\Project 2.jpg |
| Project 3  | X:\Records_System\Folder C\Project 3.mpg |
+------------+------------------------------------------+

We use a front-end that creates a clickable hyperlink out of the file paths. Using the hyperlink, users can navigate to the files.

Question:

I want to check the files on the network drive against the list of files in the database table.

If I could get a list of files from the network into a database table, then I could easily query for errors.

However, I'm not sure what the best way would be to create the list of files in an Oracle table.

• There are approximately 60,000 files on the network drive, nestled in a variety of subfolders
• The files don't exist on the same server as the Oracle database (I'm not an I.T. guy, so I'm pretty clueless about this sort of thing). However, I do have the drive mapped in Windows Explorer on the computer that I would be running the query from.
• This QC operation would be run about once per month.
• Performance isn't a huge concern. The list of files doesn't need to be "live", although that would certainly be a plus.
• I can think of quick-and-dirty ways to do this using VBA scripting in MS Office or the like. But, it has occurred to me that there might be a more professional or out-of-box way to do this (perhaps using built-in Oracle functionality).

How can a I elegantly insert a list of files on a network drive into an Oracle table?

Answer 1

The following steps will need some tweaking, as you are working with multiple machines - your client machine, a file server (Windows?), a database server etc. Thus, the paths and the file locations are just examples (which have been tested, though). The steps A-B-C are performed only once.

Prerequisites / setup:

[A] Operating system: create a folder for storing the dir listings (in form of text files, see footnote 1) .

eg C:\Users\Wilson\datafiles 

[B] Oracle: you need the permissions CREATE TABLE and CREATE ANY DIRECTORY (ie read/write)

SQL> grant create any directory to wilson;
Grant succeeded.

[C] In Oracle, create a directory and an external table.

SQL> create directory external_tables as 'C:\Users\Wilson\datafiles';

Directory created.

create table xtable ( path_ varchar2(4000) ) 
organization external (
  type oracle_loader 
  default directory external_tables
  access parameters 
  ( 
    records delimited by newline
    fields terminated by ',' 
    missing field values are null
  ) 
  location 
  (
    'dir_contents.txt'
  ) 
)
/

 Table XTABLE created.

Get the file list, and use the external table:

[1] Open a command line window. Navigate to the network drive (X:). Execute the following command:

dir /s /b > C:\Users\Wilson\datafiles\dir_contents.txt

REM (this may take a minute ...)
REM Explanation: 
REM  dir /s dives into every subdirectory (recursively), 
REM  /b returns the dir listing without headers and summaries.
REM  > redirects the output into the file (absolute path provided)

Load the dir_contents.txt file into a text editor, make sure that it contains what you need. If - as you wrote - Oracle runs on a different machine, you may have to upload it (via sftp).

[2] Use the external table (see footnote2).

SQL> select count(*) from xtable;
COUNT(*)  
19362     

SQL> select * from xtable fetch first 7 rows only;
PATH_                             
C:\Windows\System32\0409          
C:\Windows\System32\12520437.cpx  
C:\Windows\System32\12520850.cpx  
C:\Windows\System32\3DAudio.ax    
C:\Windows\System32\3DES.dll      
C:\Windows\System32\8point1.wav   
C:\Windows\System32\aaclient.dll  


7 rows selected.

footnote1: This must be accessible when you are working with Oracle. If you collect the datafile(s) on your client machine first, you must "upload" the files (eg via sftp) into the directory that you have specified when executing CREATE DIRECTORY ...

footnote2: The sample file contains the output of

C:\Windows\System32>dir /s /b > C:\Users\Wilson\datafiles\dir_contents.txt

(notice the paths and the command redirection). If you are working with Linux, use "find / -type f > dir_contents.txt"