When I heard about the incident of compromised binaries, I did all my best to ensure that will not happen anymore. For that, I have tried to contact fluffypony - as known as Riccardo Spagni, the lead mantainer of the Monero project - to help information security engineers with my analysis.
After contacting fluffypony, one of his engineers made himself available to discuss with me about the possible ways. We had examinated logs and we've seen that logs were cleaned and tampered. Apparently there was no solution; however it was possible to discover the root cause of the compromission of binaries. In this short blog post, I will explain how the attackers were able to enter into the protected.
Think without logs
Log extraction is a fundamental part of the forensic science, especially in the case of an attack. Generally, logs can give you a very specific insight into what happened and who accessed the system. From timestamp to IP address, almost every action performed can be tracked, if well configured. Marking malicious and weird connections based on log were prioritary for us.
Let's start from the beginning of what happened: the attackers replaced the original binaries with the malicious ones. To do this, it is essential to have an open remote shell with maximum administrative privileges. Two main methods of attack are therefore assumed: exploitation of an exploit or stealing private user key.
We were able to speculate this because of the limited information we had. SSH enabled with authentication based on private keys and we also know that NGINX with PHP were used. In this box, postregsql was installed and also had been initiated numerous instances of the Monero daemon. That seems pretty reasonable.
Our investigation then started immediately from the logs of the ssh daemon, the accesses of NGINX and any PHP errors. We went back in the logs, but unfortunately we had the following problems:
- The logs timestamp started from the 2019-11-07;
- The logs could be ALWAYS tampered and modified - we have to remember the “red herring” concept - attackers may have put informations that could be inserted into logs that lead to a false conclusion;
We assumed that someone might have appropriated the key through exploits. We had a lot of suspects, but I used my membership status on 0day.today to investigate better. The first investigated was PHP and in fact, as I discovered days after the compromise, a new attack that takes advantage a old bug had been well described with a very simple exploit which is very simple to use.
The attack is named “PHP-FPM + Nginx - Remote Code Execution Exploit” and allows ANY remote users to execute any commands in the webserver, also called RCE attack. The pre-requisite for this attack is the NGINX configuration, and - as we discovered - all the NGINX configurations set in that server were exploitable. The vulnerability was originally discovered by d90pwn during a Capture the Flag Competition. Emil Lerner created and developed the custom exploit script which could be found here.
In PHP versions 7.1.x below 7.1.33, 7.2.x below 7.2.24 and 7.3.x below 7.3.11 in certain configurations of FPM setup it is possible to cause FPM module to write past allocated buffers into the space reserved for FCGI protocol data, thus opening the possibility of remote code execution.
I would like to thank Andhr.es for its beautiful illustration