Originally published at Eforensics magazine:
Today when you look at the market of malware you will find an overwhelming domination of malicious windows software. So looking for security products you can find many that exist for Windows, and write up after write up on windows malware. This makes sense as the vast majority of malware is created for the Windows OS. Here’s an example of uploads to VirusTotal over the past 7 days:
Windows exe’s number over a million and you have to go down 14 file types until you reach the ELF binaries used more commonly on Linux systems and 26 slots down you reach PHP files. On web servers the vast majority of threats tend to be written in PHP. These threats tend to be webshells, spam mailers, and phishing kits. While working at a web hosting company we used both a Open Source malware scanner called maldet, as well as a another internal tool.
What I was finding that far too many times was obvious malware being missed. Basic techniques used often in a standard digital forensics and incident response(DFIR) playbooks were not being utilized. Investigations were often piecemeal. I began to try and resolve this issue by creating new signatures utilizing yara and clamAV to increase our detection rates, but before long found this unwieldy and began work on a tool to make it easier to use these custom rules. After a little while what started as a side project to use yara malware signatures became a full featured Linux DFIR tool set incorporating the things that I found lacking in our standard investigative tools.
So lets take a look at the tool and its use cases.
Blazescan
https://github.com/Hestat/blazescan
First of all this tool looks to take advantage of existing tools when they are present. Those included here include yara, clamAV, wpcli, maldet, and many standard linux utilities.
Yara – Open source tool created and maintained by VirusTotal(Google) that allows you to create rules to match against patterns, often used for malware signatures, or threat-hunting.
https://virustotal.github.io/yara/
ClamAV – Open source antivirus scanner around since the early 2000’s currently maintained by the Cisco Talos
WP-CLI – Command line interface for managing WordPress CMS Software
Maldet – Open source malware scanner and web server specific malware rule set
https://github.com/rfxn/linux-malware-detect
As well in the environment this tool was built in we catered to many small/medium businesses and platform resellers. The environment was heavily built around servers running the cPanel server management software. CPanel’s software allow you to easily multi-home a server for dozens of sites and provide additional services like email, and statistics gathering.
To aide investigations one of the first integration’s was to enable compatibility in cPanel environments, allowing for quick and easy scans on individual cPanel accounts or entire servers.
After that came the real table steaks of a good DFIR tool. Let’s do a walk through. At current writing the tool has the following options available to the user:
-a will scan all cpanel accounts
-A will use Agressive mode to scan all cpanel accounts
uses clamd to run multicore scans, can increase load
-u will scan the specified cpanel user
-l will show the results of the last scan
-t will display ctime of the hits in the last scan
-d scan a directory of your choosing
-w will run a scan on the directory of your choosing with wordpress checks included
-f will run search for all files in the directory given and record ctime of all files
-i provide a file to pull vital stats about the file
-m will email the list of hits from the last scan, set email in blazscand.conf Mailtoaddress
-n will provide an overview of logged in users and network traffic
-N will run a tcpdump for a specified time period and write the data to a file for later analysis
-U will check for updates, and allow you to perform any available updates
-R will allow you to report a malicious file back to add a signature
use this if you encounter new malicious code that is not detected
-h will display the help menuOur candidate for the demo will be a VM I’ve used in the past I call Malware.lan. It’s a Ubuntu server with a standard LAMP setup hosting a wordpress site.
Demo
Let’s set a scenario for our investigator. The client’s web site has been acting strange. They report the site is loading slow and a user has reported their AV is warning them about traveling to the site due to a phishing alert. The responder logs into the server to investigate. First let’s take a snapshot of the site as it exists to preserve vital timestamp evidence that may be altered. Blazescan allows us to do so with the following command.
root@ubuntu-malware:~# blazescan -f Already up-to-date. What path would you like to collect forensic change timestamps? /home/webuser/public_html Provide your username: responder1 Log can be found here: /usr/local/scan/forensic-responder1-2018-08-09-1950.log
Now let’s check to see what users may be logged in and current traffic.
Here we can confirm only a single remote user, in this case us. However we know this is a small server with a single CPU core, and we and note the load on the server is unusually high. Next we can note the pattern of previously logged in users, nothing standing out here yet.
Reviewing the listening services similar story, nothing out of the ordinary for this server yet.
Active connections though, that’s a different story here we see an anomaly.
The traffic making connection’s to port 6666 and the user phpViT7U_4xie5 do not match what we expect to see on a web server. Our responder decides to capture some traffic from the anomaly.
Now our responder can look at this or keep it for later review. Our responder now decides its time to scan, and knows the site is using wordpress so uses the -w flag to run a signature based scan which includes a check of the wordpress core files using the wpcli feature.
Updated:
Now wordpress checks will also report all Administrative users for review to make sure no additional users have been added to the site.
Original:
Updated version:
Here we can see that we received both malware signature hits and wordpress core files that do not match the core file hash check.
Our responder decides they want to look more closely at the phpoViT7U_4xie5u7jkmyvugag file that looked to be active based on the earlier network traffic. Using the -i flag they pull up the vital stats on the file and can choose to search the hash against VirusTotal.
root@ubuntu-malware:~# blazescan -i /home/webuser/public_html/phpoViT7U_4xie5u7jkmyvugag
Next we will want to determine when the files arrived to try and discover the vector that was exploited.
Here we can see roughly 2 different time periods. This likely means that the malware placed in the first compromise was likely reused in the following compromise, either by the same actors or just found by others in the wild.
Our responder decides to use the time stamps associated with the phpoViT7U_4xie5u7jkmyvugag file, again based on the earlier network traffic. They start with the Apache web server logs and are able to quickly find logs matching the timestamps of the phpoViT7U_4xie5u7jkmyvugag file.
The responder can now see traffic directed at the z.php file that was also identified earlier in the malware scan. With malicious intent all but confirmed our responder kills the process earlier identified and the load on the server begins to drop immediately. After that the responder confirms the z.php is a webshell with upload and command execution functionality.
Tracking the other files in the apache web logs the responder finds the following phishing kit being run on the server impersonating the European Telecom company Orange.
This kit was not flagged in the malware scan so the responder decides to report the file back for further analysis and signature creation.
And then emails the results of the malware scan to the responsible owner for cleanup of the site.
Should the analyst need to refer to any data captured from the investigation, in this case, malware scan results, network traffic captures, or the file timestamps they are all stored should they need to be re-examined.
Update from original article:
Now there is a feature to whitelist malware signatures if they trigger on legitimate files. This takes advantage of the clamAV signature file known as:
local_whitelist.ign2
You can find this in the following directory:
/usr/local/scan/lw-yara
So lets use an example, here we are going to whitelist a file that triggers the yertle malware signature. (bonus tip using the $(pwd) as a command line argument will tell blazescan to scan your current directory)
Now let’s add the signature to the whitelist:
And run the scan again:
And now the file is no longer triggered in the scan.
Future
In the future I am planning to continue updates to the signature database as well as look into ways to make it easier for the analyst to link together file timestamps and log events. As well as other quality of life features. Pull requests and issues can be opened at the project github:
https://github.com/Hestat/blazescan