• Understanding the type of Linux Threats
  • Types of Malware on Linux
• PHP Malware
  • Shell Commands to find PHP malware
    • Dangerous PHP Functions to search for
    • Search for eval within PHP scripts
    • Search for system and shell web-shells and backdoors
    • Find malware scripts on your server
    • other examples
  • File search by the owner
• Investigative Techniques for Uncovering Malware
  • Log File Analysis
    • Investigate log files with grep
  • Analyzing Running Processes and Services
    • Investigate running processes with ps
      • List every process on the system
      • List a process tree:
      • Get security info:
      • To see every process running as www-data user
    • Investigate network connections with netstat
    • Investigate sockets with ss
      • List source port 80
      • Display All Established SMTP Connections
      • Display All Established HTTP Connections
      • List All The TCP Sockets and process info with source ip address
    • List open files with lsof
• Detecting with scanners
  • Detecting malware with maldet and Ansible

Linux servers are often considered to be more secure than servers based on other Operating Systems, yet they are not immune to the threat of malicious software such as malware. As the use of Linux continues to grow in the server market, so does the importance of understanding how to detect and manage malware scripts that target these systems. This guide provides a tutorial on finding malware on Linux servers.

If you are a WordPress user and want to read a great article about how to clean malware on WordPress check out: When your WordPress site gets hacked, you are locked out, call wp-cli to the rescue.

Understanding the type of Linux Threats

In later updates of this tutorial I will cover how to deal with Malware and clean them.

Types of Malware on Linux

While Linux is well known for its robust security features, it is not immune to malware. Malicious software on Linux can range from simple _bash_ scripts to complex programs designed to exploit vulnerabilities or perform unauthorized activities.

One key aspect of Linux malware is its stealth, as it often aims to remain undetected to carry out its objectives over extended periods.

Malware targeting Linux systems can be categorized based on behavior and impact. The most common are:

• Rootkits that maintain unauthorized root type access to the system,
• Trojans that disguise themselves as legitimate server service,
• Backdoors that run a service that provides remote access to attackers,
• Botnets that hijack server resources for coordinated attacks.

PHP Malware

PHP as a popular scripting language is well known for its share of malware scripts therefore this article has many examples of detecting PHP malware.

Shell Commands to find PHP malware

These commands can detect possible malicious scripts. It is up to you to closely inspect the content of these scripts.

Dangerous PHP Functions to search for

Here you can find a list of functions usually used in writing malware that might infect your server.

• eval() – Evaluate a string as PHP code including running shell function,
• system() – Execute an external program,
• exec() – Execute an external program,
• shell_exec() – Execute command via shell,
• passthru() – Execute an external program,
• popen() – Opens process file pointer,
• escapeshellcmd() – Escape shell meta-characters,
• pcntl_exec() – Executes specified program in current process space,
• backtick operator – Back-ticks in PHP is used to encapsulate any code that should be run as shell code.

Search for eval within PHP scripts

The eval() PHP function will evaluate a string as PHP code so it can be easily misused to hide malicious PHP code.

grep -Rn 'eval(base64_decode(' /var/www
grep -Rn 'eval(gzinflate(base64_decode(' /var/www
grep -Rn 'eval(gzuncompress(base64_decode(str_rot13(' /var/www
grep -Rn 'eval(str_rot13(gzinflate(base64_decode(' /var/www

Search for system and shell web-shells and backdoors

The system() and shell() PHP function executes the given command so it can be misused as a web shell hence acting as a stealth backdoor.

grep -Rn 'shell *(' /var/www
grep -Rn 'system *(' /var/www

Search for both functions with one command.

grep -RPn '(system|shell) *\(' /var/www

Find malware scripts on your server

grep -Rn 'GIF89aG' /var/www
grep -Rn 'tcp *(' /var/www

or searching for these functions with one command

grep -RPn '(system|shell|tcp) *\(' /var/www

If you do not want the content of the script to be displayed instead just need the name of the file use

grep -RlPn '(system|shell|tcp) *\(' /var/www

other examples

grep -Rn 'shell_exec *(' /var/www
grep -Rn 'base64_decode *(' /var/www
grep -Rn 'phpinfo *(' /var/www
grep -Rn 'system *(' /var/www
grep -Rn 'php_uname *(' /var/www
grep -Rn 'chmod *(' /var/www
grep -Rn 'fopen *(' /var/www
grep -Rn 'fclose *(' /var/www
grep -Rn 'readfile *(' /var/www
grep -Rn 'edoced_46esab *(' /var/www
grep -Rn 'eval *(' /var/www
grep -Rn 'passthru *(' /var/www

File search by the owner

Find out what files were created by the web server which is in this case www-data.

find /var/www -user www-data

Investigative Techniques for Uncovering Malware

Log File Analysis

Log file analysis is a critical component of detecting malware on Linux servers. Administrators can detect potential security breaches by closely examining logs of both system and application activities. It’s important to focus on anomalies that deviate from normal operations, such as:

• failed login attempts,
• unexpected software installations,
• unauthorized changes to system files.

Pay attention to anomalies like unsuccessful login attempts, unexpected software installations, or unauthorized alterations to system files.

Key log files to monitor include:

• /var/log/auth.log for authentication records,
• /var/log/syslog for system-related messages,
• /var/log/apache2/access.log for Apache web server access records,
• /var/log/apache2/error.log for Apache web server error records,
• /var/log/nginx/access.log for NginX web server access records,
• /var/log/nginx/error.log for NginX web server error records,
• /var/log/mail.log for mail server records.

Tip: Consistently check and analyze logs with automated tools to quickly identify potential threats. Creating a baseline of typical activity patterns greatly helps in spotting anything unusual.

When checking logs, search for patterns and connections between events. One unusual incident might not mean malware, but if there are repeated occurrences or a mix of suspicious events, it’s worth investigating further. Use tools like grep and other command-line options to filter and search through log data effectively. Remember, analyzing logs promptly is crucial to minimize the impact of potential security incidents.

Investigate log files with grep

Here are some examples.

1
grep 'failed\|invalid' /var/log/auth.log

Analyzing Running Processes and Services

When checking a Linux server for possible malware, it’s important to look at the processes and services currently running. Malicious software can pretend to be normal system processes, so finding any unusual things is crucial. Use the ps command to list all running processes and look for those that do not have a clear purpose or that are consuming an unusual amount of resources.

To further scrutinize the services, the systemctl command can reveal which services are active.

Note: The utilization of CPU and memory percentages as signs of potentially suspicious activity.

Rootkits are a specific worry when checking processes and services. They hide malware and can trick common system monitoring tools. Use rootkit detection tools such as chkrootkit or rkhunter for a more detailed investigation.

Tip: Always compare process lists with logs of network activity. Irregularities in one can often be linked to unusual discoveries in the other, giving a better understanding of possible security breaches.

Investigate running processes with ps

List every process on the system

• ps -ely
• ps -aux

List a process tree:

• ps -ejH
• ps axjf

Get security info:

• ps -eo euser,ruser,suser,fuser,f,comm,label
• ps axZ
• ps -eM

To see every process running as www-data user

The www-data is associated with the web server service, change the user as per your needs.

ps -U www-data -u www-data u

Investigate network connections with netstat

To install netstat.

apt install net-tools

Show open ports and services that are opening them.

netstat -tnlp

or

netstat --tcp --numeric --listening --program

Investigate sockets with ss

The ss command is provided by iproute2 package.

List source port 80

ss -t -a -n -s '( sport = :80 )'

Display All Established SMTP Connections

ss -o state established '( dport = :smtp or sport = :smtp )'

Display All Established HTTP Connections

ss -o state established '( dport = :http or sport = :http )'

List All The TCP Sockets and process info with source ip address

ss -p -o state all '( sport = :http or sport = :https )' src xx.xx.xx.xx

List open files with lsof

The following command lists all open files that are opened by the www-data user which in my case is the NginX web server.

lsof -n -u www-data

List opened files by www-data user and filter by ESTABLISHED keyword.

lsof -n -u www-data | grep ESTABLISHED

Detecting with scanners

Detecting malware with maldet and Ansible

Maldet is a great and free software that can detect and quarantine malware. For installation and setup I recommend Ansible.

To install the role: ansible-galaxy role install cloudweeb.maldet.

following is the playbook.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
---
- hosts: maldet_group
  vars:
    maldet_version: '1.6.5'
    maldet_default_monitor_mode: users
    maldet_service_enabled: false
    maldet_email_alert: true
    maldet_email_addr:
      - root@
      - [email protected]
    maldet_quarantine_hits: false
    maldet_inotify_cpunice: '9'
    inotify_ionice: '3'
    maldet_scan_ionice: '3'
    maldet_scan_cpunice: '9'
  roles:
     - role: cloudweeb.maldet

After the installation a cron script is created.

/etc/cron.daily/maldet

Trigger a manual scan execution by running maldet --scan-all /var/www/.

After the malware scan completes, you can access the results by running maldet -e.