Posts Tagged ‘elsa’
ELSA is one of the largest software projects outside BalaBit, which utilizes the patterndb pattern matching technology in syslog-ng. As an open source app, utilizes syslog-ng OSE at its heart. ELSA is used by network security professionals to gather enormous amount of information about what’s happening on their networks and search it with google like response times.
Recently somebody asked, if it’s possible to store digitally signed logs using ELSA. While it’s not possible using the mysql storage, syslog-ng PE has this functionality using its logstore technology. Of course, using the logstore has the drawback of storing logs twice, but it also has some advantages, like encryption or optional time stamping.
So once I could allocate a little time, I gave ELSA and syslog-ng PE a try. I used Ubuntu 10.04 and syslog-ng PE 4.2 in my tests, and with some minor quirks I was able to replace OSE with PE and add logstore support to the configuration. For details, check the ELSA wiki at https://code.google.com/p/enterprise-log-search-and-archive/wiki/UsingSyslog_NGPremiumEdition It should work in fairly similar way on all platforms supported by ELSA.
First of all, I’d like to thank the LOADays crew for the wonderful event. It was one of the best Linux events I have ever attended both as a speaker and a visitor. We got everything for a perfect conference: 100+ people focused on Linux system administration and also fuel for sysadmins: beer and pizza If you are interested in Linux, open source and system administration, you should definitely come next year and join Linux administrators from all around Europe.
I bet, my title is confusing: logging and Lumberjack. Do I really plan to write about cutting trees? No. Logging is about recording events on a computer. On Linux/UNIX systems it is usually done using syslog. So what is syslog-ng? The next generation of syslog server, the application collecting events (logs) from other applications and often also from other machines. It can collect events from many different sources, filter and process them, normalize them, store or forward them to many different destinations.
Centralized logging of events has been an important part of the IT infrastructure for many years. It is more convenient to browse logs in a central location rather than viewing them on individual machines. Central storage is also more secure. Even if logs stored locally are altered or removed, one can still check the logs on the central log server. Compliance with different regulations also makes central logging necessary. (This is an updated version of my previous syslog-ng web gui blog.)
This week is rich in patterndb related news. I just found the first job description which involves patterndb: http://www.freelancer.com/projects/PHP-Perl/Customizing-cactiEZ-syslog.html I gave a presentation about syslog-ng, patterndb and web GUIs today at Linux Academy in Hungary based on my blog series. And an updated version of ELSA, the first major software outside of BalaBit utilizing patterndb is now available: http://ossectools.blogspot.com/2011/07/elsa-vmware-appliance-available.html
Stay tuned, we hope to release some new patterns next week!
Central logging of events is already an important part of the IT infrastructure for many years. It is more convenient to browse logs at a central location instead of on individual machines. It is also more secure, as even if an individual machine is compromised, and local logs are altered or removed, one can still check the logs on the central log server. Compliance with different regulations also makes central logging necessary.
System administrators prefer to use the command line. Why bother with GUIs? Grep, awk & Co. are powerful tools, but for complex queries an SQL based web interface makes the job done a lot more quicker. Once there are many messages to search, it’s not just convenient, it’s a must. With thousands of incoming messages a second, the indexes of log databases still give Google like response times even for the most complex queries, while traditional text based tools don’t scale.
In this comparison of web GUIs for syslog-ng I try to cover solutions from simple scripts to browse logs through cloud logging to enterprise level applications. All of these have different strengths and weaknesses, and target different usage scenarios.
Enterprise log search and archive (ELSA) is a brand new centralized syslog framework with syslog-ng 3.1+ and patterndb at its heart. It is the first larger project outside of BalaBit utilizing the power of patterndb. Data storage and searching is based on mysql and sphinx, and there is a simple but powerful web interface, which gives quick access even to many millions of log messages.