The Evolution of Malware Encryption Part I: Basic Malware Encryption

By Christopher Elisan, Prinicipal Malware Scientist, RSA FirstWatch

We, the RSA FirstWatch team, are always at the forefront of solving the latest malware problems –one of those is malware encryption. Malware encryption is not new. It has been around since the DOS days, but has simply evolved to address the antivirus solutions designed to beat it.

In this multi-part blog, I will discuss how malware encryption has evolved from the simple application of an encryption/decryption engine to the more complicated metamorphic engine.

Malware’s main weakness is  its source code. If the source code is revealed through decompiling or disassembling, anything about the malware is laid bare. It’s darkest secret becomes exposed and solving it becomes much easier. This is why protecting the source code is one of malware’s important directives, especially if it is designed for persistence.

To better discuss the evolution of malware encryption, I will be borrowing from my book, “Malware, Rootkits and Botnets: A Beginner’s Guide.”

Malware encryption is designed to protect the malware code itself. There are three major developments in malware encryption technology:

1.)   Basic malware encryption

2.)   Polymorphism

3.)   Metamorphism.

We will tackle these three one by one but for this part of the blog, we will concentrate on basic malware encryption.

In the early days of malware most were file infectors. So to better understand basic malware encryption, I will discuss them in the context of file infectors.

An encrypted malware has three major components: the encryption/decryption engine, the encrypted malware code, and the decryption key. When the malware is executed, the encryption/decryption engine decrypts the encrypted malware code using the decryption key and then control is passed to the decrypted malware code in memory for it to do its intended purpose. Upon infection, the decrypted malware code is re-encrypted using a different key before it attaches itself to the newly infected host program. The key can be a series of bytes from a specific location in the host program. The location is constant, but the bytes found in that location differ for every target file. This makes the keys different in every infection. Because of this, each malware code that is attached to different host programs differ — no two infections are exactly alike.

Although this method was cutting edge when it was first introduced, the antivirus industry was able to catch up pretty quickly because one out of the three components remained constant. The decryption key was always different, the encrypted malware code was always different, but the encryption/decryption engine remained constant. Using the encryption/decryption engine codes, the antivirus products were able to create a signature to catch this basic form of malware encryption.

Because of this, the malware writers had to come up with a new way of encrypting malware. And so they did. They came up with polymorphism. We will discuss this further in part 2 of The Evolution of Malware Encryption. Stay tuned!


Christopher Elisan is a seasoned reverse engineer and malware researcher. He frequently speaks at various security conferences across the US and provides expert opinion about malware, botnets and advance persistent threats for leading industry and mainstream publications. He is currently the Prinicipal Malware Scientist at RSA NetWitness. Elisan is also the author of “Malware, Rootkits and Botnets: A Beginner’s Guide.”

No Comments