LetsDefend Academy

Online practicing and training platform for blue team members









SIEM and Log Collection



Log Collection

It contains a basic log, time, source system and a message. For example, when we look at the content of the "/var/log/auth.log" file on an Ubuntu server, we can see the source, time and message information.




Logs are generally collected in the following 2 ways:

  • Log Agents

  • Agentless



  • Log Agents


    In order to implement this method, a log agent software is required. Agents often have parsing, log rotation, buffering, log integrity, encryption, conversion features. In other words, this agent software can take action on the logs it collects before forwarding them to the target.

    For example, with the agent software, we can divide a log with "username: LetsDefend; account: Administrator" into 2 parts and forward it as:

  • message1 = "username: LetsDefend"

  • message2 = "account: Administrator"

  • Syslog


    It is a very popular network protocol for log transfers. It can work with both UDP and TCP, and can optionally be encrypted with TLS. Some devices that support syslog: Switch, Router, IDS, Firewall, Linux, Mac, Windows devices can become syslog supported with additional software.

    If you want to forward your log with Syslog, you will need to parsing in syslog format.

    Syslog Format:

    Timestamp - Source Device - Facility - Severity - Message Number - Message Text

    https://flylib.com/books/1/297/1/html/2/images/1587051583/graphics/09fig02.gif

    Also, the maximum packet size that can be sent with Syslog UDP is 1024 bytes. For TCP it is 4096 bytes.


    3. Party Agents


    Most SIEM products have their own agent software. 3rd party agents have more capabilities than syslog because of the features they support. Some agents:

    Splunk: universal forwarder
    ArcSight: ArcSight Connectors

    These agents are easy to integrate into SIEM and have parsing features.


    Open Source Agents


    They are generally agents that provide basic needs comfortably. However, it may not be as effective as the agent of the SIEM product itself. (Ease of installation, integration, additional features etc.)


    Popular open source agents:


    Beats https://www.elastic.co/beats/
    NXLog https://nxlog.co/


    Agentless


    Agentless log sending process is sometimes preferred as there is no installation and update cost. Usually, logs are sent by connecting to the target with SSH or WMI.

    For this method, the username and password of the log server are required, therefore there is a risk of the password being stolen.
    Easier to prepare and manage than the agent method. However, it has limited capabilities and credentials are wrapped in the network.


    Manual Collection


    Sometimes there are logs that you cannot collect with existing agent software. For example, if you cannot read the logs of a cloud-based application with the agent, you may need to write your own script


    Summary


    As you can see, there are various ways to collect logs. These are agents and agentless. In cases where the agents on the market are not sufficient, you should write your own scripts.


    Questions