IT organizations have dealt with computer and network device logs for the past 40 years. There are precious few technologies still alive and kicking that boast such a long history. But despite this, enterprise log management software tools are still viewed by some as "new" to the market, even though Unix system administrators have managed logs for as long as Unix has existed, and they have been joined in recent years in impressive numbers by their counterparts in the Windows and Linux worlds.
While often underappreciated by IT professionals, logs are an extremely useful source of data for IT shops, particularly in the area of information security management. It is widely known that getting that crucial information takes both time and energy. Sadly, both are often in short supply inside overworked IT organizations. These organizations are typically consumed with fighting fires raging on every major platform, from ancient, creaking mainframes to today's robust, cloud-deployed applications and virtualized environments.
Log management software helps with accountability
Adding to their strategic value, logs are increasingly more than just a source of data for system administrators. Logging -- and tracking such activity through log management software or other tools -- is a primary means of IT accountability because most user and system actions can be recorded in logs. There are many other means of accountability inside an organization, but logs are the one mechanism that pervades all of IT, stretching even beyond the bounds of technology. If your IT operation is not accountable, that means your business is not accountable.
If your organization doesn't take logs seriously, it should raise flags about just how attentive you are when it comes to IT accountability. This is why logging is a perfect compliance technology, now mandated by a raft of regulations and laws including PCI DSS, FISMA, HIPAA and best practices frameworks such as ISO 2700 and COBIT.
As already mentioned, all IT users, whether they are malicious or good corporate citizens, leave behind traces of their activity in various logs. These digital fingerprints are generated by a number of IT components, such as user-owned desktops, servers and firewalls, routers, databases and business applications. Such records accumulate over time, creating mountains of different types of log data.
Avoid becoming the next T.J. Maxx
At the same time, more organizations are becoming aware of the value of collecting and analyzing such data using some sort of log management software. Such analysis can prove critical given the growing emphasis on data security, with companies wanting to avoid the catastrophic hacks suffered by The T.J.Maxx Cos. Inc. and Heartland Payment Systems Inc. A more pressing reason for doing so is that many regulations now mandate logging, log collection, log retention and (heaven forbid) periodic log review, along with log protection and logging access to logs themselves.
Simply producing and collecting the logs is barely half the battle. The other half involves the ability to intelligently review massive amounts of log data in order to investigate, detect and predict security threats, as well as staying on top of compliance requirements.
Traditionally, companies have reviewed logs based on their individual points of origin and, unfortunately, only after a major incident. This manual and ad hoc approach simply does not work in the age of data breaches and growing regulatory requirements. It is not only inefficient and complex, but it can also cost a large organization millions of dollars and take weeks to carry out, thereby negating or severely reducing the positive effects of a log review.
As more companies struggle with compliance mandates, the need for useful, comprehensive application logging will only increase.
A recent report released by Verizon showed that some 86% of breached organizations actually had evidence of a data security breach in their logs immediately after it occurred but simply didn't take the time to look. Today, the call to action is shifting from mere possession of log data to centralized data collection, analysis and in-depth reporting as a way to address information security and regulatory compliance issues. This means the main log-related goals of a company should be to enable both the creation and centralized collection of useful logs.
Adding emphasis to the need to centralize collection is the reality that data security threats are shifting toward applications. As more companies struggle with compliance mandates, the need for useful, comprehensive application logging will only increase. The next consideration is finding a way to search and review log data from disparate points of origin across system boundaries, the IT infrastructure, geographies and hundreds of distributes sites.
In essence, the need to paint a complete picture of IT infrastructure activity, and to satisfy key regulatory compliance mandates, generally means that IT professionals must find a way to pay strict attention to logs and log management.
Log management software as strategic weapon
The most important point to remember is that a lot of critical information can be extracted from log data. However, the limiting factor to how well that information can be put to good use is how quickly the log can be retrieved, searched and analyzed. If a company's IT staff can't access the logs in time, it will spend all of its time firefighting data security breaches rather than reacting quickly at the first signs of a breach before it melts down the business.
Log management software tools evolved to become weapons of strategic significance for organizations since they present the only way to unlock the insights from the flood of log data. Buying the right log management software tool for your organization is also a strategic decision.
But a word of warning: A lot of tools are very different in capabilities, scalability and price. For example, prices range from free to tens of millions of dollars (for large environments). The only way to arrive at the right tool for your organization is to consider your use cases, or the specific requirements and needs for log management. Despite its 40-year history, there is no one size that fits all in log management.
Anton Chuvakin is a recognized security expert in the field of log management and PCI DSS compliance. He is an author of SecurityWarrior and PCI Compliance, and a contributor to Know Your Enemy, Information SecurityManagement Handbook and other works.