Reverse Engineering To Know .NET Better
Brian Long (www.blong.com)
If you find this article useful then please consider making a donation. It will be appreciated however big or small it might be and will encourage Brian to continue researching and writing about interesting subjects in the future.
The Microsoft .NET Framework is a new, exciting programming platform for Windows application developers (and potentially for developers on other Operating Systems, as we shall see). Right from the outset .NET has offered a large volume of functionality, both in terms of the underlying CLR and also the accompanying class library that is used by all .NET applications.
Becoming familiar with .NET can be achieved by making use of the mass of documentation in the .NET Framework SDK. Of course there is also a mass of third party documentation covering various aspects of the .NET Framework available in books and online on the Internet.
However it is sometimes said that a programmer can become most familiar with a system if they know exactly how it works. This can perhaps be best achieved if you have access to the underlying source of the system. Take, for example, C and C++ compilers and the Delphi compiler. These 3GL programming languages come supplied with the full source to their entire run-time library (RTL) as well as any class libraries they may use. Having full source code means any question as to the behaviour or implementation of any library feature can be readily resolved by looking at the pertinent source files.
Of course .NET does not ship with its source, but there are various tricks we can use in an attempt to overcome that hurdle, and get the same results as if we did have access to it. This is the remit of reverse engineering and this paper looks at various approaches that we can take in reverse engineering aspects of .NET, simply to understand its behaviour and operation better.
Most reverse engineering options that are open to us stem from one of the key features of managed executable files: metadata. We'll look at the list of reverse engineering options and then look in detail at each of them. But first we'll go off on a slight tangent.
Note that I used the phrase managed executable files, rather than .NET executable files? This was an intentional choice and was intended to emphasise the fact that the Microsoft .NET Framework is not the only platform that can execute managed executables. You should be aware that the Microsoft .NET is one particular implementation of an ECMA (European Computer Manufacturers Association) standard (see Reference 1).
ECMA 335 is the standard for the Common Language Infrastructure, or CLI (see Reference 2). The Microsoft .NET Framework is one example of an implementation of the CLI (it implements the CLI as well as a whole host of additional tools and classes).
The CLI defines the fact that managed executables use the Portable Executable (PE) File Format, as used by Win32 executables. The PE files contain a standard header and then contain metadata and IL (Intermediate Language) code in special sections of the file. The IL code represents the functionality in the file, which will be compiled into native machine instructions prior to execution (usually) with the JIT (just in time) compiler. A module is an example of such a file. An assembly is one or more modules combined with additional metadata called a manifest, which names and describes the assembly, and lists assemblies it depends upon.
It therefore follows that any implementation of the CLI supports assemblies in the PE format.
At the time of writing there are five implementations of the CLI in existence or under development:
Microsoft .NET Framework (see Reference 3). This was the initial CLI implementation and it supports Windows platforms (Windows 98, Windows Me, Windows 2000, Windows XP, Windows Server 2003). It is freely downloadable in binary form (source code is not supplied). You can get just the redistributable version (suitable for deploying to machines to execute managed executables against) or the SDK (which includes additional tools, documentation and developer support). At the time of writing the current version of the .NET Framework is v1.1 SP1, which supersedes version v1.0 SP2.
Microsoft .NET Compact Framework (see Reference 4). This is a CLI implementation for running on small devices that run Windows CE .NET (see Reference 5). The implementation is much more lightweight than the full desktop .NET Framework and is tuned for the limited memory and storage of small devices. At the time of writing the current version of the .NET Compact Framework is v1.0, which can be used to develop application in conjunction with Visual Studio .NET 2003, although v2.0 has just been signed off.
Microsoft Shared Source CLI or SSCLI, codenamed Rotor (see Reference 6). This is an implementation of the CLI (along with other parts of the Microsoft .NET Framework) that can run on multiple platforms. The supported platforms are Windows XP, FreeBSD 4.7 and Mac OS X 10.2, but it should also work fine on Windows 2000 and earlier versions of FreeBSD. You can freely download the entire source for SSCLI (over 3,000,000 lines of code) for non-commercial purposes. After .NET v2.0 is released we should expect a new release of SSCLI to match, according to Joel Pobar's blog.
Mono, which is a project sponsored by and mainly developed by Ximian (see Reference 7) and runs on Linux, Windows, Macintosh and Solaris. This is an implementation of the CLI, but which also endeavours to implement various other parts of Microsoft's .NET Framework such as WinForms, ASP.NET, ADO.NET and VB.NET (called Basic.NET). At the time of writing the current version is Mono 220.127.116.11.
Portable.NET from the DotGNU project (see Reference 8). While the initial target platform was GNU/Linux, it is also known to run under Windows, Solaris, NetBSD, FreeBSD, and MacOS X. The runtime engine has been tested on the x86, PowerPC, ARM, Sparc, PARISC, s309, Alpha, and IA-64 processors. At the time of writing Portable.NET 0.7.2 was the current version.
Getting back to the point made at the start of the previous section, assemblies are rich with metadata, making it easy to peer into them and identify the structure of what's inside. You can identify all the namespaces, classes, their methods, fields and properties, structures, enumerations and so on using a mechanism in .NET called reflection.
There are a variety of utilities that use reflection to show the structure and content of .NET assemblies and we'll bump into them as we look at the various reverse engineering options.
The only thing not directly exposed through metadata is the IL code itself. However this hurdle is easily surmountable with existing code and utilities available to read and display IL opcodes directly from a PE file.
As well as being able to look at the low level IL code, which may not be palatable to many programmers, there are various tools which will decompile this IL to a high level language, such as C#, VB.NET or C++ with Managed Extensions. These tools examine the IL and use heuristic analysis techniques to identify patterns that allow them to substitute high level statements in their place.
The various non-commercial implementations of the CLI all ship with source code. The most interesting example is Microsoft's own Shared Source CLI. This project started from the commercial .NET code base, although there have been multitudes of changes and some subsystems have been entirely reworked.
That notwithstanding, these source bases often offer a fascinating and educational insight into the working (or potential working) of the Microsoft .NET Framework. In many cases, the use of a high level language decompiler in conjunction with the SSCLI source can be a very productive pairing.
Let's start off with (almost) the simplest C# application we can build, the classic Hello World application.
After looking at results we get with this simple example, we'll turn our attention to looking at an already implemented method in the .NET Framework Class Library, namely the System.Collections.ArrayList class's InsertRange method. This method has been selected reasonably arbitrarily, although it is an example of a method that performs a reasonable amount of work.
The .NET Framework comes with a utility that takes source files containing IL code and metadata directives and compiles them into either modules or assemblies. The utility, ilasm.exe, is called the IL Assembler.
When you install the .NET Framework SDK (or Microsoft Visual Studio .NET or Borland C#Builder, which both include the Framework SDK) you get another utility called ildasm.exe, the IL Disassembler. As the name suggests this tool takes a compiled module/assembly and shows you the IL/metadata that constitutes it.
So rich is the IL code and metadata combination that you can use this utility to produce IL source code files that can be run through ilasm.exe to recreate a fully functioning binary. This process is called round tripping and emphasises the lack of ambiguity in the IL/metadata found in every managed executable.
The IL Disassembler operates in two modes, GUI and console. You can tell it which assembly to disassemble by passing the name on the command-line. If you tell it to generate IL source files using the /OUT command-line parameter it operates as a console application. It also runs as a console application if you pass the /TEXT parameter.
If you just pass the assembly name (or no parameters at all) it launches as a GUI app, which is often more convenient for browsing. When launched with no parameters you can use the File | Open menu (or Ctrl+O) to choose an assembly to disassemble, or alternatively drag a file onto the UI from Windows Explorer.
Running ildasm.exe on the simple Hello World application produces this.
As you can see, the tree view shows the assembly manifest and the simple namespace at the top level. In the namespace is our single class and within the class you can see its single static method, Main, as well as a reference to some other internal class elements. These include the instance constructor (.ctor), which is never used in this simple case as we do not construct an instance of HelloClass. The first item is some metadata to indicate the class does not require a class constructor (sometimes called a type initialiser), which is a method that automatically executes before the class is used, to initialise static fields.Note that by default ILDasm will display all members of a class. If you only want, for example, public and protected (family) members displayed you can use the appropriate items on the View menu or invoke ildasm.exe with the /VIS=PUB+FAM command-line switch.
Double-clicking on any item in the tree view (or selecting it and pressing Enter) shows you the implementation of the item. The Main method shows up like this:
Since the original C# code was trivial, this should be readily understandable. The Hello world string is loaded onto the stack and the System.Console.WriteLine method is called, using the string on the stack as its parameter.
So we see that simple code is evidently readable enough. However a nice touch we can add is to get the disassembly to include the corresponding source code lines as comments just before the disassembled IL, assuming the assembly was compiled with debug information and the source is available. The View | Source menu item does this or you can use the /SOU command-line switch to do the job.
Now what about something a little more taxing, such as the InsertRange method of the ArrayList class? You can locate this method by running ildasm.exe on the mscorlib.dll assembly. You can find this file in the %windir%\Microsoft.NET\Framework\<version> directory, where windir is the standard environment variable that points to your Windows installation directory and <version> represents the relevant .NET version directory, being v1.0.3705 for .NET 1.0 and v1.1.4322 for .NET 1.1.
This screenshot shows how to locate the right class within the namespaces. First look in the System namespace, then in the nested System.Collections namespace. Then scroll down the methods until you bump into InsertRange.
Because of the fact that there are several statements in this method you will find the IL code quite verbose. The following listing shows the code:
Of course it's much less intelligible now we have more code, but with the metadata and IL documentation, which can be found in Partitions II and III of the CLI specification respectively (see Reference 2) or Inside Microsoft .NET IL Assembler (see Reference 9) we could still work it out. However it would take rather longer than most of us would be prepared for.
Lutz Roeder is a developer working at Microsoft and he has his own personal Web site (see Reference 10). There you can find a popular tool called Reflector, which is at version 18.104.22.168 at the time of writing. You can load assemblies into Reflector either using the File | Open... menu (or Ctrl+O) or by dragging them onto the UI from Windows Explorer.
Reflector does a similar job to the IL Disassembler in that it displays information about an assembly by reflecting across the metadata. You can control which levels of visibility Reflector displays members from in the options: View | Options...).
When you select a method it will display the method signature at the bottom of the main window in C# syntax by default, although you can switch it to show Visual Basic, IL or even Borland Delphi syntax using the combobox embedded in the toolbar, or the Languages combobox in the options dialog. You can disassemble a method by ensuring IL is the selected language and then choosing Tools | Disassembler (or by pressing Space). Since the IL code in a given assembly is fixed, you will get much the same results from any tool:
However as you might be able to see, there are additional facilities in Reflector over ildasm.exe, such as the lists of base types and descendant types, a list of dependencies for the assembly, the ability to search for types in an assembly and a call tree. Also, the disassembly window explains the IL instructions using tooltips when you pause your mouse over them. These all make Reflector a useful tool to have available.
Borland's .NET-aware development tools also includes a reflection browser and IL disassembler called Reflection.exe. By default the tool appears to simply be a reflection browser but it can be enticed into offering IL disassembly by adding a simple registry entry. Depending on which tool you have you need to add in a different registry entry under HKEY_CURRENT_USER\Software\Borland as listed here (note the intentional difference in the C#Builder entry):
|Product||Registry key under HKCU\Software\Borland||Registry value|
|Delphi 8 for the Microsoft .NET Framework||BDS\2.0\Globals||ShowILDisassembly="1"|
Setting the value as shown will show an extra Code page when looking at a method.
The Microsoft SSCLI download is supplied as a massive source tree. It contains the source to the class libraries, the C# compiler the IL assembler and the IL disassembler and many other interesting bits and pieces. The IL disassembler is functionally identical to the commercial .NET Framework version except that it does not offer a GUI interface (SSCLI has no GUI support; it is just for building console applications).
If the SSCLI installation directory is referred to as <ROTOR> then the ildasm.exe source is located in <ROTOR>\clr\src\ildasm. You can peruse and learn from this source base to see how IL disassembly can be performed. If you need to build a custom IL viewer and can read C++ this would be a good place to start perusing.
You can use the Visual Studio .NET debugger to step through the SSCLI IL disassembler in order to understand how it operates, assuming you have built the SSCLI source base appropriately. If not, you will need to follow the SSCLI instructions in order to get a checked (full debug, optimisations on) or fast checked (full debug info, optimisations off) build of SSCLI first. Your best bet for debugging would be the checked build.
In principle, building SSCLI is actually straightforward thanks to the excellent configuration tools and files used in the source tree. You need to first install a Perl 5.6 implementation, for example ActivePerl (see Reference 11). You also need Visual Studio.NET installed before you can proceed. Installing SSCLI itself is simply a matter of extracting the files from the compressed tarball (i.e. a gzip compressed tar file) they are supplied in. WinZip or WinRar should do the trick.
In a command prompt window change to the SSCLI root installation directory and setup the SSCLI environment. The env.bat batch file will do this, and it takes parameters to set up for checked, fast checked or free mode. Once set up, your command prompt environment will have a variety of environment variables pointing to parts of the SSCLI directory tree, including ROTOR, which does indeed point to the main installation directory.
To set up the environment for checked mode execute:
Next you invoke the build process by executing:
As you might expect, the build process can be rather lengthy, depending on your hardware, but it should get there in the end.
With SSCLI built you will find ildasm.exe in %TARGETCOMPLUSSDK%\bin. TARGETCOMPLUSSDK is another environment variable, which is equivalent to <ROTOR>\build\v1.x86chk.rotor\sdk in the checked environment.
You can set up a Visual Studio.NET solution for ildasm.exe as follows:
Choose File | Open Solution...
Change the Files of type: entry to Executable files (*.exe)
Locate the SSCLI ildasm.exe in %TARGETCOMPLUSSDK%\bin
Select the Solution Explorer with View | Solution Explorer or (Ctrl+Alt+L)
Right-click the solution node and choose Properties
Select Common Properties node, then the Debug Source Files node and add in the path to the ildasm source and also to the common SSCLI include files:
View the project properties by right-clicking the .exe file in the Solution Explorer and choosing Properties, or with Project | Properties
In the Configuration Properties, Debugging node set use the Command Arguments entry to set up the ildasm.exe command-line arguments to refer to the assembly you wish to disassemble
Optionally set the Working Directory option to point to the directory housing the specified assembly
Save the solution using File | Save ildasm.sln (or Ctrl+S), after ensuring the solution is selected in the Solution Explorer
Close Visual Studio.NET
Create a batch file in the directory containing the ildasm.sln file that looks like this, substituting your SSCLI directory path as appropriate:
Now you can double-click the batch file and Visual Studio.NET will be launched in an SSCLI checked mode environment and will load your solution. You can then start debugging it as you would with your normal applications:
Another option for writing your own IL displaying utility would be to use a helper class made available by Lutz Roeder (see Reference 10). The ILReader class and its selection of helper classes are supplied in a C# source file accompanied by an example program that shows them in use.
The ILReader class relies on the calling program having access to the type whose methods require disassembling. The constructor takes two parameters: the target type's module and a class implementing the locally defined IAssemblyLoader interface. This interface defines two methods that the ILReader calls in order to load the assembly that implements the target type, Load and LoadFrom. The example program defines the trivial AssemblyLoader class with simple implementations.
Having constructed an ILReader you call its GetMethodBody method for any method you need to disassemble. GetMethodBody takes a MethodBase descendant (such as MethodInfo), which can be accessed through Type.GetMethod or Type.GetMethods, and returns a MethodBody object whose members provide all the information you require.
The sample program disassembles the System.Object class.
There is an argument that suggests that programmers entering into the .NET world are opening their applications up to the eyes of the world. The rich metadata along with the type rich IL code allows any managed code in an assembly to be disassembled back to readable IL and there is the worry about intellectual copyright issues with your algorithms on public display.
IL is easier to disassemble because of the richness of the metadata in the assembly, which is there because it makes various internal operations much easier, not because it has to be. The general advice if this is an issue to you is to make use of an obfuscator.
Microsoft Visual Studio.NET 2003 ships with an obfuscator called Dotfuscator from PreEmptive Solutions (installed into PreEmptive Solutions\Dotfuscator Community Edition directory under the main Visual Studio installation directory). Dotfuscator ships in two versions, the Professional Edition and the Community Edition. The latter is the cut-down version that ships with Visual Studio.NET 2003 and basically renames your classes and methods with unhelpful choices of names, whilst the former has many additional features and comes with a price tag. You can find out more about Dotfuscator from the links at Reference 12.
Borland Delphi for .NET and Borland C#Builder both ship with a version of another obfuscator called Demeanor for .NET from Wise Owl. This product holds a good reputation in obfuscation circles and you can find out more from the link at Reference 13. Demeanor is available in an Enterprise Edition, which you pay money for, or the Personal Edition, as supplied with C#Builder. The Personal Edition does simple renaming of methods and classes whilst the Enterprise Edition has additional features that make it an attractive purchase.
Viewing IL is all well and good, but despite its completeness, it is still not very readable. Various tools have surfaced which allow you to reconstitute high level statements from the IL in the assembly. They analyse the IL instruction sequences and use this information to build up a representation of the original source code in a given high-level language (some tools support this decompiling process with various high level language targets).
One of the first products to offer decompilation support was Exemplar by Jay Freeman (aka saurik). Exemplar is a command-line tool, which has been superseded by Anakrino, his GUI version (anakrino is a Greek word meaning to examine). You can find these tools at the URL listed in Reference 14. The current version at the time of writing is 22.214.171.124.
Anakrino doesn't (currently) support drag and drop from Windows Explorer so you must open assemblies with the File | Open... menu item or by pressing Ctrl+O. Drilling down to a method allows you to see its high-level representation in either C# (by default) or managed C++ (if you select Dialect | MC++). Here is the Main method in our trivial test case displayed in C#.
By contrast, this is what it looks like in Managed C++:
This utility is very helpful, but has a number of "quirks". If you switch language (dialect) whilst a method is displayed, the decompiled version is not updated. You must select a method then reselect the previous method to see it decompiled into the new language. Also, as you can in the screenshots above, the bottom pane (coloured light yellow) seems to be unused at present. The equivalent panel in Reflector displays the methods signature/prototype.
When you download Anakrino, it includes the command-line Exemplar tool. You can use this to disassemble our test case assembly as shown here.
Moving onto the ArrayList method, we get much more interesting results here. Anakrino produces this listing as the C# source for the InsertRange method:
As you can see this is much more meaningful that the pure IL code behind it. All we have lacking here is sensible names for the local variables along with any comments that might have helped us understand the intent of the code. However even without those you can see this is substantially easier to work with now.
We have already looked at Reflector in the context of an IL disassembler but from version 3 it also offers high-level language decompilation support. The currently supported languages are C#, Visual Basic and Delphi (selectable in the combobox embedded in the toolbar. Decompiling a selected method is just the same as disassembling it, being invoked by Tools | Disassembler (or Space).
The results when decompiling the Hello World method are the same as with Anakrino. The following listing is what we get when decompiling ArrayList.InsertRange.
Again, much the same as Anakrino; the only differences are subtle and include:
a different local variable name (num1 instead of local0)
different brace layout conventions; Anakroni puts the opening brace at the end of a line of code whilst Reflector puts them on their own line, in the same column as the previous line of code started
explicit brackets around each condition, such as index < 0
As detailed earlier there are various implementations of the CLI of which Microsoft's .NET Framework is the most well known and successful. However Microsoft's Shared Source CLI is worth paying some close attention to. This implementation started its life from the same code base that .NET has grown from and, whilst parts of it have been changed for various reasons (such as to aid porting to other platforms or to help keep certain algorithms in the domain of the Microsoft engineers), much of it is still a direct representation of code in the commercial platform.
This is, of course, not true of the other non-Microsoft implementations, but it can still be very fruitful delving into their source trees.
Assuming the SSCLI has been installed into a directory referred to as <ROTOR> the source for the ArrayList class can be found in <ROTOR>\clr\src\bcl\system\collections\arraylist.cs. The method we are looking at is InsertRange, and the source code for it looks like this:
Some key things to note about this code are:
it is exactly the same logic as decompiled by Anakrino and Reflector
the local variable has a descriptive name
there are comments describing what goes on in the code
In cases where the SSCLI still uses the same code as .NET, any effort in working out the gaps left by the decompilers is completely avoided. It is usually worth decompiling with one of the tools, then comparing with the equivalent source in SSCLI to see if the shared source file can be used as a reference to the .NET implementation.
The Mono implementation is based on reading the CLI specification and implementing the described behaviour. Therefore you don't learn anything about the inner workings of .NET by browsing Mono source, but it does give you an option to see alternative ways to implement the behaviour as outlined in the ECMA CLI specification.
If the Mono 126.96.36.199 source files are installed in <MONO> then the ArrayList source will be found in <MONO>\mono-188.8.131.52\mcs\class\corlib\System.Collections\ArrayList.cs. The InsertRange method implementation looks like this:
As you can see, this is somewhat different in its approach, but does the same thing as far as the CLI class library specification goes.
The situation with Portable.NET is much the same as with Mono with regard to its implementation. If Portable.NET 0.7.2 is installed in a directory referred to as <PNET> then you can find the ArrayList source in <PNET>\pnetlib-0.7.2\runtime\System\Collections\ArrayList.cs. The InsertRange method looks like this:
The European Computer Manufacturer's Association (ECMA) at http://msdn.microsoft.com/net/ecma.
Link to ECMA-335, Common Language Infrastructure (CLI) in PDF and PostScript format at http://www.ecma-international.org/publications/standards/ecma-335.htm and in HTML format at http://dotnet.di.unipi.it/EcmaSpecs.aspx.
Note that this documentation is also available as Word documents in the .NET Framework SDK installation directory tree. Many developers have the SDK installed implicitly by Visual Studio.NET and so in those cases they will be in the Visual Studio.NET installation directory tree. You should be able to locate a directory called Tool Developers Guides (use Windows Explorer to find it if necessary), which contains a subdirectory called docs. This contains each specification partition as a separate Word document.
A published version of this specification is available with many annotations from the people who put it together as The Common Language Infrastructure Annotated Standard by James S Miller & Susann Ragsdale, published by Addison Wesley
Microsoft .NET home page at http://www.microsoft.com/net, the download page (for version 1.0 and 1.1, both the redistributable and the Framework SDK) at http://msdn.microsoft.com/netframework/downloads/updates/default.aspx.
Microsoft .NET Compact Framework 1.0 at http://msdn.microsoft.com/smartclient/understanding/netcf/.
Microsoft Windows CE .NET at http://msdn.microsoft.com/embedded/windowsce.
Microsoft Shared Source CLI home page at http://msdn.microsoft.com/net/sscli.
Also worth checking out is Shared Source CLI Essentials by Stutz, Neward & Shilling, published by O'Reilly, which as the title suggests is a book on the SSCLI code base.
Ximian Mono home page at http://www.mono-project.com the download page is at: http://www.go-mono.com/download.html.
DotGNU Portable.NET home/download page at http://www.southern-storm.com.au/portable_net.html.
Inside Microsoft .NET IL Assembler by Serge Lidin
(of Microsoft), Microsoft Press.
This book describes the CIL (Common Intermediate Language) in detail. The author was responsible for developing the IL Disassembler and Assembler and various other aspects of the .NET Framework.
Lutz Roeder's site, home of Reflector and the IL Reader class at http://www.aisto.com/roeder/dotnet. Reflector is a reflection browser, IL disassembler and high-level language decompiler.
ActivePerl is an example of a Perl implementation, which is required to build the SSCLI source base. You can download it from http://aspn.activestate.com/ASPN/Downloads/ActivePerl.
PreEmptive Solutions Dotfuscator home page at http://www.preemptive.com/dotfuscator. You can get an evaluation copy of Dotfuscator at http://www.preemptive.com/downloads/RequestEvaluation.html.
Wise Owl Demeanor home page at http://www.wiseowl.com/products/products.aspx.
Anakrino home page at http://www.saurik.com/net/exemplar. Anakrino is a reflection browser and high-level language decompiler.
Brian Long used to work at Borland UK, performing a number of duties including Technical Support on all the programming tools. Since leaving in 1995, Brian has spent the intervening years as a trainer, trouble-shooter and mentor focusing on the use of the C#, Delphi and C++ languages, and of the Win32 and .NET platforms. In his spare time Brian actively researches and employs strategies for the convenient identification, isolation and removal of malware. If you need training in these areas or need solutions to problems you have with them, please get in touch or visit Brian's Web site.
Brian authored a Borland Pascal problem-solving book in 1994 and occasionally acts as a Technical Editor for Wiley (previously Sybex); he was the Technical Editor for Mastering Delphi 7 and Mastering Delphi 2005 and also contributed a chapter to Delphi for .NET Developer Guide. Brian is a regular columnist in The Delphi Magazine and has had numerous articles published in Developer's Review, Computing, Delphi Developer's Journal and EXE Magazine. He was nominated for the Spirit of Delphi award in 2000.