Log parsing strategy using scala

Question

So yesterday I had a logfile which had comma separated entries in a provided log.txt such as follows:

entry1.1,entry1.2,entry1.3
entry2.1,entry2,2,entry2.3
..........................

So with much happiness, I went ahead and created a case class:

case class LogEntry(
  entry1:String,
  entry2:String,
  entry3:String
)

And populated the case class while reading the case class as follows:

line.split match {
  case Array(entry1,entry2,entry3) => LogEntry(entry1,entry2,entry3)
}

Now problem arose when I ran my code today and I noticed that the LogEntry objects are not being created.

I looked at the log.txt provide to me today and realized that the entries have changed:

I now have:

entry1.1,entry1.2,entry1.3,entry1.4
entry2.1,entry2,2,entry2.3,entry2.4
...................................

I now have the fourth entry in each of the lines. Well, it seems like no big deal, simply change my case class with the fourth entry(code smell 1) and then change the pattern match(code smell 2)

Can somebody suggest how I should be writing my code to deal with this situation. I want to extend my code rather than modify it.

Thanks

Answer 1

First thing to do when you want to properly abstract the log lines is to give the individual elements names and proper types rather than pseudo-indices through names (i.e. entry1, entry2, etc.). So we'll have:

case class LogEntry(
    time: java.util.Date,
    userId: Int,
    host: String
)

The strictness of the typing depends on your exact use case (note that host here is not a java.net.InetAddress).

When a new column gets added to your log files, there are two things that you might want to preserve:

1. Functions that expected old-style LogEntries
2. Treatment of old log files even after new functionality is added.

Case 1 is easy. Just add the field to the case class:

case class LogEntry(
    time: java.util.Date,
    userId: Int,
    host: String,
    port: Int
)

Since you have names, you could even change the order of the fields and the old code would still work as the fields it expects are still here. Since you do not want to handle old log files anymore, just adapt the read-in code.

Case 2 is a bit trickier: You'll have to reflect the fact that you'll have old and new log entries in your running application. You can either:

1. Make additional fields optional and give default value:

   case class LogEntry(
       time: java.util.Date,
       userId: Int,
       host: String,
       port: Option[Int] = None
   ) 
   

2. Reflect versions of log formats in hierarchy (note: you should not inherit from case classes, so you'll have to make normal classes out of them):

   class LogEntry(
       val time: java.util.Date,
       val userId: Int,
       val host: String
   )
   
   class ExtendedLogEntry(
       time: java.util.Date,
       userId: Int,
       host: String,
       val port: Int
   ) extends LogEntry(time, userId, host)
   

Option 1 lets the code that treats log entries handle the difference between the entry types (i.e. all functions are still of type LogEntry => ?). Further also note that it might allow inconsistent log entries (add two fields, both not present in format 1, required in format 2, you could still set one of them to None).

Option 2 lets the calling code handle the difference between the entry types. If an advanced function needs an ExtendedLogEntry, it's type will be ExtendedLogEntry => _, so you cannot supply a LogEntry to that function (alternative: pattern match in the function and provide default / fallback behavior). Further, it prevents the inconsistency that can occur with option 1.

Concerning read-in: Option 1 will automatically set the default argument when reading old-style log files, in Option 2, old-style log files will result in LogEntry as before. So in both cases, old read-in code does not need to be changed, but potentially adapted in order to detect new-style or old-style logs and produce appropriate entries. (Note that this is potentially a problem with Option 2 as the style of the log is statically enforced through the typing system if you do not want to cast).