Fun with dynamic - how to build a dynamic object

The project: build a redis client

The dynamic feature in C# 4 is powerful when used appropriately. I thought it was about time I wrote a piece on how to do that. And … because I like redis so much I thought it would make a perfect example.

Now, the observant among you may be thinking “but Marc, you’ve already written a redis client” – in which case I agree (and incidentally I congratulate you on getting hyperlinks into speech – no mean feat); but that isn’t the point! This toy client isn’t meant to compete: BookSleeve is heavily optimised to allow really fast and efficient usage. This one is just for fun. If you like it and it works for you, great! All the code for this is available on google-code.

What we want

I want to be able to do things like:

// increment and fetch new
int hitCount = client.incr("hits");
// fetch next, if any (else null)
string nextWorkItem = client.lpop("pending");

but without the client knowing anything about redis except the binary wire protocol – those commands are entirely dynamic. Actually a nice advantage of this is that the client doesn’t need to be updated as new redis features are released… but I digress!

Getting started

The key in implementing a dynamic API is implementing IDynamicMetaObjectProvider – although frankly I don’t propose doing that; I’m just going to subclass DynamicObject which does a lot of the work for us. So here's our first step:

public sealed class RedisClient : DynamicObject, IDisposable {...}

This gives us the start of a client that will respond to dynamic; although it doesn't actually do anything yet - we have to tell our object to handle dynamic method calls, which we do by overriding TryInvokeMember. Again, keep in mind that this is only a toy, and we’ll do this by simply writing the command name and parameters (in redis format) down the wire, and reading one result in redis format (note that this means that we’ll pay the full latency price per operation, which isn’t ideal – and that we can’t act as a redis pub/sub subscriber – that would simply not work, since replies don’t match neatly to commands then):

public override bool TryInvokeMember(
    InvokeMemberBinder binder,
    object[] args, out object result)
{
    WriteCommand(binder.Name, args);
    result = ReadResult();
    var err = result as RedisExceptionResult;
    if (err != null) throw err.GetException();
    return true;
}

In particular, note that the name of the method requested is available as binder.Name, and the parameters are just an object[].

Writing the command

I won't dwell on the redis protocol details (feel free to read the specification), but basically we need to write the number of arguments for the command (where the command-name itself counts as an argument), followed by the command-name, followed by each of the parameters. To avoid packet-fragmentation, we’ll use some buffering into a BufferedStream which we hold in outStream, which in turn writes to a NetworkStream which we hold in netStream - and obviously after each command we need to flush those to ensure they get to the server, so we get:

private void WriteCommand(string name, object[] args)
{
    WriteRaw(outStream, '*');
    WriteRaw(outStream, 1 + args.Length);
    WriteEndLine();
    WriteArg(name);
    for (int i = 0; i < args.Length; i++)
    {
        WriteArg(args[i]);
    }
    // and make sure we aren't holding onto any data...
    outStream.Flush(); // flushes to netStream
    netStream.Flush(); // just to be sure! (although this is a no-op, IIRC)
}

And for each argument, we need to write the length of the data, followed by the data itself. I won't detail each the various formats for different types of data, but: to avoid having to test "what type of data is this?", we'll cheat by having a few overloads of a method we'll call WriteRaw, and use dynamic to get the runtime to pick between the overloads for us... sneaky:

private void WriteArg(object value)
{
    // need to know the length, so: write to our memory-stream
    // first
    buffer.SetLength(0);
    WriteRaw(buffer, (dynamic)value);
    // now write that to the (bufferred) output
    WriteRaw(outStream, '$');
    WriteRaw(outStream, (int)buffer.Length);
    WriteEndLine();
    WriteRaw(outStream, new ArraySegment(buffer.GetBuffer(), 0, (int)buffer.Length));
    WriteEndLine();
}

Did you spot the cheeky dynamic in there? Since we're already in dynamic-land, it is hard to say that this is going to have any negative impact... so; why not? It means that if I need to support a new data-type, I just add a new WriteRaw to match, and: job done - and frankly, that's about it for sending the data.

Reading the response

After that, we need to refer to the specification again to see what replies look like - it turns out that they're basically the same format as the outbound data. But we have some ambiguity - does the user want their data as a byte[]? or as a decoded string? or maybe they want int? The nice thing is: we can let them tell us, by providing a result that supports conversions via dynamic - so when they type:

byte[] blob = client.get("my_image");
string name = client.get("name");

they get the binary and text correctly. So we can subclass DynamicObject again for a class that holds the raw result, and override another method - TryConvert this time. We get passed in a different binder, this time with access to the requested type in binder.Type - which we can then use to unscramble the data accordingly. This implementation is a less interesting and more tedious (testing different matched types), so I’ll leave that out of the blog. The only thing left to do (as shown in the TryInvokeMember) is to check if the response is an error-message from the server, and turn that into a thrown Exception, so that it feels intuitively .NET. The reason I can’t do this directly when reading the reply comes down to some implementation details – some replies are themselves composed of multiple nested replies, and I want to re-use the code internally for reading those. We can’t do that if it throws when hitting an error a few levels down – the stream could be left in an incomplete state (i.e. we might not have finished reading the outer-most reply).

Summary

And: that's it! A redis client written from scratch in a little over an hour; but more importantly, a complete dynamic API illustration. Well, maybe not complete, but as you can imagine: the other available operations (properties, indexers, operators, etc) work very similarly. I hope it is illustrative. Again, all the code for this is available on google-code.

How many ways can you mess up IO?

There are so many interesting ways that people find to make a mess of reading binary data, but actually it doesn’t need to be hard. This post is intended to describe some of the more common misunderstandings people have.

Text encodings: what they are not

Text is simple, right?

Text encodings are great; you use them (whether you know it or not) all the time, whenever your computer opens any kind of text file, or downloads text (such as the ubiquitous html and json) from the internet. If you don’t know what an encoding is, then first: go and read Joel’s treatise on the subject. No really, go read that and then come back.

So now you know: an encoding is a way of working with character data (you know... words etc) over a binary medium (such as a file on disk, or a network socket). Common encodings are things like UTF-8, UTF-7, UTF-16 (in either endian-ness), UTF-32, or “Windows-1252” - however, there is a wide range of encodings available. Some (in particular the UTF-* encodings) can handle most-any unicode character, but many are restricted to the characters used most commonly in a particular locale. Ultimately, an encoding defines a map between characters and bytes, for example defining that “abc” should be represented as the bytes 00-61-00-62-00-63 (and equally, that the bytes 00-61-00-62-00-63 can be interpreted as the text “abc”).

Here’s the thing an encoding is not: it is not a way to turn arbitrary bytes into text for storage. You would be amazed how often people try this, but: that simply doesn’t work. If an encoding tries to read something that isn’t text data, then at best: it will throw an error. At worst, it will silently corrupt the data without realizing it has made a mistake. If you want to store arbitrary binary data as text, then there are other tools for that: primarily, things like base-n. In this case, it is the text that is specially formatted. For example, we might need to convey the bytes 07-A2-00-B2-EE-02 using only text.The observant will notice that I’ve just done exactly that using hexadecimal (base-16), but that it took 17 characters (12 without the dashes) to represent just 6 bytes of actual data. A common alternative to reduce this overhead is base-64, which uses characters that avoid the “control characters” likely to cause problems, while also staying in the first range 0-127, which is the most reliable region. Our 6 bytes become “B6IAsu4C” when stored as a base-64 string. Most platforms have utility methods that provide translation to and from base-64.

The key in choosing between a text-encoding and something like base-64 is simple: all you need to figure out is: which of these two things can have arbitrary contents, and which is required to follow rules (otherwise it is nonsensical)?

arbitrary* text / rules-based binary: use a text encoding
arbitrary binary / rules-based text: use base-64 (or similar)

*=at least, for the characters that the encoding supports, since not every encoding supports every character.

One last thing to say about encodings: don’t use the “system default” encoding - ever. This goes back to the old days of 8-bit systems where your text console needed a code-page that fitted into a single byte but reached at least most of the characters the user was likely to need. For me, this is code-page 1252, but yours may be different. It is exceptionally rare that this is what you want; if in doubt, explicitly specify something like UTF-8. Sometimes you can use byte-order-mark detection, but this is also pretty unreliable - many files omit a BOM. The best answer, obviously, is: define and document what encoding you are planning to use. Then use it.

Network packets: what you send is not (usually) what you get

TCP usually works as a stream of data. When you send things, it is usually not guaranteed that it will arrive at the destination in exactly the same chunks that you send. For example, let’s say that
you try to “send” 3 messages over the same socket using your chosen network library- one with 10 bytes, one with 4 bytes, and one with 8 bytes. You might think that the receiving client can call “read” 3 times and get 10 bytes, 4 bytes and 8 bytes - but it is much more interesting than that. Because it is just a stream, this is simply very unlikely to be what happens. The client could find they get 1 message with 22 bytes. Or 22 messages each with 1 byte. Or any other combination. They could get 10 bytes, 10 bytes, and the last 2 bytes never arrive (because of some network issue). All that TCP guarantees is that whatever portion of the data does make it to the receiver, it will be the correct bytes in the correct order.

Because of this, if you want to send multiple messages over the same socket it is necessary to implement some kind of “framing” protocol - i.e. some way of splitting the data back into logical pieces at the other end. How you do this is up to you, and often depends on the data being sent. For example, text-based protocols often detect a “sentinel” value to split messages (possibly a carriage return, line-feed, or some other “control character” rarely seen in regular text; the characters with values 0 (“nul”) and 3 (“etx”) are also popular choices.

If your messages are binary then it is more challenging, as usually there aren’t really any safe “sentinel” bytes to choose from - so commonly some header information is sent before each message that includes (perhaps only includes) the length of the message that follows. This could be as simple as dumping the 4-byte (32-bit) or 8-byte (64-bit) native representation of the length onto the stream (although you also need to decide whether it is “big endian” or “little endian”, obviously). There are also a range of variable-length representations, for when most messages are expected to be small, but the protocol needs to allow for much longer messages. The key thing here is that you must decide how you are going to do this, and clearly document it for consumers. This of course has the unfortunate requirement that you must actually know the length of the data you want to write before you write it - not always convenient. To help with this, some protocols (for example, the more recent web-sockets protocols) allow you to further break down a single message into multiple fragments that the client must stitch back together into a single logical piece - with an additional marker (perhaps a trailing zero-length message, or a special bit set in the header message) to indicate the end of the logical message.

Learning to read

Most frameworks have a “read” API that looks something like:

    int read(byte[] buffer, int offset, int count)

(or something appropriately similar for asynchronous access). The point being that you supply a buffer (somewhere for it to put the data), tell it where in that buffer to start writing (often, but not always, 0, and how much data you want it to read. It then goes away and fetches some data for you. Here’s the rub: in most cases, the last parameter is just a maximum - it is not “get this much data”; it is “get at most this much data”. The return value tells you what it could do: this could be non-positive if no more data will ever be available (the stream has ended), it could be count - which is to say: it could fetch all the data you wanted, or it could be any other value greater than zero and less than count. Because of this, reading often involves a loop; for example, let’s say we want to write a method to read an exact number of bytes; me might do that as:

    void readExact(byte[] buffer, int offset, int count) {
        int bytesRead;
        if(count < 0) {
            throw new ArgumentOutOfRangeException(“count”);
        }
        while(count != 0 && 
          (bytesRead = source.read(buffer, offset, count)) > 0) {
            offset += bytesRead;
            count -= bytesRead;
        }
        if(count != 0) throw new EndOfStreamException();
    }

Dissecting that:

• first we check that we aren’t requesting a negative amount of data, which is clearly silly
• then in a loop, we:
  • check to see if we want more data, then try to read at most count more data
  • if the stream ends, we break out
  • otherwise, we increment out offset, so that if we need to do another read, we don’t over-stamp the data we just fetched) - and decrement count, because we now have less work required
• finally, we check whether we still have data outstanding, because the stream terminated before we expected to - perhaps raising an error to indicate this

This is a fairly classic pattern for reading data that shows how to process the number of bytes obtained in each iteration.

Gotchas when buffering data in memory

Sometimes it isn’t possible to process all the data in a single buffer; in many cases it becomes necessary to use an in-memory store of data that you have received but still need to process (perhaps because you need yet more data before you can make proper sense of it). A common way to do this is to make use of a “memory-stream” - an in-memory object that acts like a stream, but that simply uses local memory rather than a disk or network socket. You basically “write” to the memory-stream until you think you have something you can process, then “read” from it. But did you spot the deliberate mistake? Most memory-streams acts like an old-fashioned VCR: if you record a show, hit “stop”, and then hit “play” - you will find that you are unexpectedly either watching a black screen, or something that you recorded 3 weeks ago. We forgot to rewind it. Essentially, the memory-stream has a single cursor position; after writing the data, the cursor is at the end of the data: what follows is either all zeros, or garbage from the last thing that happened to be in that space. Fortunately, rewinding a memory-stream is usually trivial:

    stream.Position = 0;

There’s a second very common confusion with memory-streams; sometimes, after writing to it you want to get the contents, but as a raw buffer (byte[]) rather than as a stream. Often, there are two different APIs for this:

• one which gives you a brand new buffer of the current logical length of the stream, copying the data into the new buffer
• one which hands you the oversized buffer that it is using internally - oversized so that it doesn’t need to allocate a new backing buffer every time you write to it

They both have uses; the second is often more efficient, as it avoids an allocation - but it is only useful if you also track the logical length of the data (often via stream.Length or stream.getLength()). To go back to our VCR analogy: one creates a brand new cassette of exactly the 37.5 minutes we recorded, and copies the video - the other simply hands us the existing 180 minute cassette: this is only useful if we also say “the bit you want is the first 37.5 minutes”.

Conclusion

This isn’t actually all that complex, but people tend to bring a lot of incorrect preconceptions to the table, which results in the same bugs – over and over and over. So let’s all save some time, and do it right. Please?

Multiplexed transactions in BookSleeve

Short version, for the impatient

BookSleeve now has support for transactions using WATCH and pre-conditions. If you don’t know what BookSleeve or redis is, this won’t be of interest to you.

Background

We use redis extensively as part of our caching (and a few other things) technologies at Stack Exchange, and as part of doing that efficiently, we put together BookSleeve, which is a high-performance multiplexer over a redis connection. In English, what that means is: you can share a single redis connection from lots of different threads, and it will worry about delivering the commands efficiently and shipping results back to the appropriate callers, without 200 commands having to pay the penalty of 200 latency. Redis even includes transaction support for complex operations, however, previously I’ve always been a bit stumped how to fit this into BookSleeve…

How do transactions work in redis?

You see, redis transactions are not like SQL transactions. A transaction is defined by a MULTI … EXEC block, with everything between the MULTI and the EXEC being processed as a single atomic unit (meaning: redis doesn’t process any requests from other connections during that time - don’t worry if that sounds draconian, redis manages extraordinarily well on a single thread anyway). That’s pretty easy to fit into a multiplexer, as we can just have a structure to buffer up commands, and send them when complete – so that already exists in BookSleeve:

using (var tran = conn.CreateTransaction()) 
{ 
    tran.Strings.Set(0, "foo", "abc"); 
    tran.Strings.Set(0, "bar", "def"); 
    tran.Execute(); 
}

But here’s the gotcha: between the MULTI and the EXEC, you don’t get results – you only get results when you commit the transaction. This means that you can’t easily request information in the middle of a transaction and then make a decision on that for what you do next – which makes a lot of sense really, since while you’re doing that thinking and network-IO, every other connection would be stalled. A sane decision really.

To get around this limitation, redis transactions actually allow a bit more cleverness… you can issue a WATCH command against keys of your choice, and then if somebody changes that key, your transaction automatically gets killed. So, a typical cycle might be:

WATCH SomeKeyName
GET SomeKeyName
{some test on the value back in your client code, then}
MULTI
SET SomeKeyName SomeNewValue
EXEC
{or if you decided you didn't like it after all}
UNWATCH

And importantly, because of the WATCH, if another connection comes along and changes SomeKeyName, then the EXEC does nothing, and returns a reply that robustly indicates the cancellation. This actually allows a lot of very subtle usage, while allowing everything to happen at the server as a single unit of work.

So what is the problem? And what did you do?

The painful bit here is simply: WATCH doesn’t work in a multiplexer. As soon as one caller issues a WATCH, either you need to stop accepting work from other callers, or you now have no idea what each caller thinks it is watching (which is very important). Another approach would be to somehow have the user write callbacks that happen as part of a transaction, but what I absolutely don’t want to do is expose anything in BookSleeve that inadvertently allow one caller to break everybody, by being too slow – so callbacks: not really an option. So for a long time, BookSleeve only made use of WATCH etc internally, and didn’t expose it to the caller. But that has finally changed!

I’ve just committed my first stab at improving this, basically by implementing a number of pre-canned pre-conditions that can be enforced on a transaction. The “pre-canned” is there to avoid the issue of opening up massive performance issues, but in reality they are very simple. For example, a previous internal usage of WATCH was to take an exclusive lock (by marking a key as in-use). Until today, that was about 100 lines of complex code that needed to know about the full gory details of the redis protocol – where-as now it is just:

TaskCompletionSource<bool> result = new TaskCompletionSource<bool>();
            
using (var tran = CreateTransaction())
{
    tran.AddCondition(Condition.KeyNotExists(db, key));
    tran.Strings.Set(db, key, value, expirySeconds);
    tran.Execute().ContinueWith(takeLockContinuation);
}
return result.Task;

where takeLockContinuation is just a static delegate instance that take the Task<bool> result from the Execute call, and sets that into result (plus handling task faults, cancellation, etc). Crucially, the condition (Condition.KeyNotExists in this case) takes care of both the WATCH steps, and the actual validation / enforcement. Basically, if there are pre-conditions (there can be multiple), then they are all sent to the server, and only if they all return the expected results is the MULTI / EXEC block sent – otherwise an UNWATCH is sent instead. Simple, elegant (IMO), and efficient.

The available pre-conditions are pretty limited at the moment, but I’ll try to add all the obvious ones I can think of (exists/not-exists for keys/hashes, equals/not-equals for keys/hashes, maybe a few inequalities, maybe something involving ttl). But finally I have a WATCH implementation that isn’t entirely sucky for a multiplexer. At the moment this is in the code repository only; I'll push to NuGet when I have a few more pre-conditions implemented.

Iterator blocks, missing methods, and .NET 4.5

We had some fun a while ago where we had a flurry of MissingMethodException occurring on some boxes, in particular, complaining that Environment.CurrentManagedThreadId was missing. And sure enough, this is a 4.5 method, and the affected servers were 4.0 – so it was kinda valid, but completely unexpected. At the time, our most pragmatic option was just: deploy 4.5, but we’ve finally had time to go back and understand it!

This happened after we upgraded our build server to 4.5, for some initial 4.5 usage – but all our core projects were still 4.0 at the time.

Fix first:

MAKE SURE YOUR BUILD SERVER HAS THE REFERENCE ASSEMBLIES FOR 4.0!

These are typically somewhere like:

C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.0

Now, the full story:

fact 1: 4.5 is an in-place over-the-top install on top of 4.0, in the GAC; once you have installed 4.5, 4.0 runs with the 4.5 assemblies

fact 2: if your build server doesn’t have the reference assemblies, it looks in the GAC – so… once you’ve installed 4.5, it will get 4.5 even if you asked (in your project) for 4.0

fact 3: the c# compiler has a detail relating to iterator blocks (aka "yield return" and "yield break") that involves threading, i.e. "which thread am I?"

fact 4: the implementation for this changes between 4.0 and 4.5; in 4.0, it uses (in the generated iterator-type’s constructor) Thread.CurrentThread.ManagedThreadId – but in 4.5 it uses Environment.CurrentManagedThreadId; it makes this decision based on the framework version it finds it has loaded (taking into account targeting etc)

Put these pieces together, and you can get a scenario where a 4.0 project that uses 4.0-only methods, seems to build successfully, but builds in a way that will only work on 4.5; it will never work on 4.0.

Fun fun fun!

Taking meta-programming beyond crazy

Firstly, a brief note that protobuf-net is up to r580 now, on both nuget and google-code; mainly small tweaks while I build up enough energy to tackle a few larger pieces (some knotty interface / dynamic / base-class improvements are next on the list)

Pushing both ends

Aimed to time with the .NET 4.5 release (and perhaps more notably, the .NETCore and .NETPortable profiles), I’ve recently spent a lot of time on meta-programming, culminating in the new precompiler that allows these slimmed down and highly restrictive frameworks to still have fast serialization (static IL, etc).

While there, I set myself a silly stretch goal; for the main purpose simply of to see if I could do it – which was: to get the whole shabang working on .NET 1.1 too. This would give me a fair claim to supporting the entire .NET framework. So, for a bit of reminiscing – what does that need?

Generics

Generics were introduced in .NET 2. v1 of protobuf-net made massive usage of generics; so much so that it actually killed the runtime on some platforms (see here, here and here). So removing most of the generics was already a primary design goal in v2.

Perhaps the most significant problem I hit here was trying to decide on a core collection type for the internal state. As it turns out, there’s no free lunch here; there is no collection type that is common to all frameworks – some don’t have ArrayList. In the end, I wrote my own simple collection – not just for this, but also because I wanted a collection that was thread-safe for iterations competing with appends (iterators only read what existed when it started iterators).

Language

You’d be amazed what you miss when you try to design something to compile on down-level compilers. For a dare, go into project-properties, the “Build” tab, and click “Advanced…” – and change the “Language Version” to something like ISO-1 (C# 1.2) or ISO-2 (C# 2.0) – see what breaks. Obviously you expect generics to disappear, but you also lose partial methods, partial classes, iterator blocks, lambdas, extension methods, null-coalescing, static classes, etc – and just some technically legal syntax that the early compilers simply struggle with. protobuf-net is configured to build in ISO-2 in the IDE, but with #if-regions to rip out the last few generics in the .NET 1.1 build. Writing iterator blocks… not fun.

Framework

There’s a silly number of variances in the core BCL between different frameworks; even things like string.IsNullOrEmpty or StringBuilder.AppendLine() aren’t all-encompassing. I ended up with a utility class with a decent number of methods to hide the differences (behind yet more #if-regions). But by far the craziest problem: reflection. And protobuf-net, at least in “Full” mode (see here for an overview of “CoreOnly” vs “Full”), uses plenty of reflection. Oddly enough, the reflection in .NET 1.1 isn’t bad – sure, it would be nice to have DynamicMethod, but I can live without it. Getting this working on .NET 1.1 was painless compared to .NETCore.

Aside / rant: how much do I hate “.GetTypeInfo()” on .NETCore? With the fiery rage of 2 stars slowly crashing into each-other. Oh, I’m sure that the differences to Type / TypeInfo make perfect sense for application-developers in .NETCore, who probably should be limiting their use of reflection, but for library authors: this change really, really hurts. The one things that lets me keep civil about this change is that in “CoreOnly” + “precompiler” we do all the reflection work up-front using the regular reflection API, so for me at least most of this ugly is just a cruel artefact. But still: grrrrrrrrrrrrrr.

Opcodes

There are a number of opcodes that simply don’t exist back on 1.1; if I’ve done my compare correctly, this is: Unbox_Any, Readonly, Constrained, Ldelem and Stelem. The good news is that most of these exist only to support generics, and are pretty easy to substitute if you know that you aren’t dealing with generics.

Metadata Version

.NET 1.1 uses an earlier version of the metadata packaging format than all the others use. This is yet another thing that the inbuilt Reflection.Emit can’t help, but - my new favorite metaprogramming tool to the rescue: IKVM.Reflection supports this. I have to offer yet another thanks to Jeroen Frijters who showed me the correct incantations to make things happy: beyond the basics of IKVM, the key part here is a voodoo call to IKVM’s implementation of AssemblyBuilder:

asm.__SetImageRuntimeVersion("v1.1.4322", 0x10000);

The 0x10000 here is a magic number that specifies the .NET 1.1 metadata format. For reference, 0x20000 is the version you want the rest of the time. As always, IKVM.Reflection seems to have considered everything; it is the gold standard of assembly writing tools. Awesome job, Jeroen. I jokingly half-expect to find that Roslyn has a a reference to IKVM.Reflection ;p

Putting the pieces together

But! Once you’ve dealt with all those trivial problems; it works. I’m happy to say that protobuf-net now has “CoreOnly” and “Full” builds, and support from “precompiler”. So if you still have .NET 1.1 applications (and I promise not to judge you… much), you can now use protobuf-net with as many optimizations as it is capable of. Which is cute:

C:\SomePath>AnotherPath\precompile.exe Net11_Poco.dll -o:MyCrazy.dll -t:MySerializer

protobuf-net pre-compiler
Detected framework: C:\Windows\Microsoft.NET\Framework\v1.1.4322
Resolved C:\Windows\Microsoft.NET\Framework\v1.1.4322\mscorlib.dll
Resolved C:\Windows\Microsoft.NET\Framework\v1.1.4322\System.dll
Resolved protobuf-net.dll
Adding DAL.DatabaseCompat...
Resolved C:\Windows\Microsoft.NET\Framework\v1.1.4322\System.Xml.dll
Adding DAL.DatabaseCompatRem...
Adding DAL.OrderCompat...
Adding DAL.OrderLineCompat...
Adding DAL.VariousFieldTypes...
Compiling MySerializer to MyCrazy.dll...
All done

C:\SomePath>peverify MyCrazy.dll

Microsoft (R) .NET Framework PE Verifier  Version 1.1.4322.573
Copyright (C) Microsoft Corporation 1998-2002. All rights reserved.

All Classes and Methods in MyCrazy.dll Verified

In case it isn’t obvious, “Net11_Poco.dll” is a .NET 1.1 dll created in Visual Studio 2003; “precompiler” has then detected the 1.1-ness, bound IKVM to the .NET 1.1 framework, and compiled a protobuf-net custom serializer for that model, as a legal .NET 1.1 dll.

Questionable sanity

Another way of reading all this is: I’ve possibly now crossed the line between “eccentric” and “batshit crazy”. I don’t have a need to use .NET 1.1, but I would be overjoyed if someone else gets some genuine usage out of this. Mainly, I just wanted to learn some things, challenge myself, and take a bit of professional pride in doing something fully and properly – just because: I can.

Introducing the protobuf-net precompiler

Over the last few posts (here and here) I’ve given a few hints as to compiling for other platforms. Basically, this all relates to how well protobuf-net works on platforms like iOS, WinRT/Metro, Silverlight, Phone 7, etc. These heavily restricted runtimes don’t allow much meta-programming, and they might be running on low-power CPUs, so reflection (even if possible) is not ideal.

I’ve played with assembly generation before, but with mixed results. For example, here for Phone 7. This could just about work for some frameworks, but was patchy on some, and won’t work at all for others.

Well, all that IKVM shininess has opened up a whole new set of tools. The small beauty that I’m ridiculously pleased with is a new utility exe in the SVN trunk (will be part of a proper release soon): precompile.

This unassuming little tool works a bit like “sgen”, but with the ability to target multiple frameworks. What it does is:

• inspect the input assembly (or assemblies) to resolve (if it can) the target framework
• initialize an IKVM universe targeting that framework
• load the core framework libraries, protobuf-net, and the input assemblies into the IKVM universe
• scan the input assemblies for types marked [ProtoContract]
• add those to a protobuf-net model
• compile the model to a type / assembly of your choosing

To use that in a project you might:

• create a new DTO project in your chosen framework, and compile it
• execute precompile to generate to a serializer assembly
• from your application project, reference the DTO and serializer assemblies
• use the type you created

For example, say I create a new Phone 7 DTO assembly, called PhoneDto (because it is late and I lack imagination). I can then create a serialization assembly via:

precompile {some path}\PhoneDto.dll –o:PhoneSerializer.dll –t:MySerializer

This will generate a library called PhoneSerializer.dll, which you can reference from your main project (in addition to the DTO and the protobuf-net core).

Then, just use MySerializer:

var ser = new MySerializer();
ser.Serialize(output, obj);

I hope this finally solves a number of tooling issues. I’m pretty pleased with it. I’ve tested it against a range of different frameworks, and it has worked well – but if you get problems, just let me know (comment here, or email me, or log an issue on the protobuf-net project site).

Enter the IKVM

aka Meta-programming and Metro for .NET

In my previous blog entry I gave an overview of how protobuf-net is arranged internally, and hinted that it all falls apart for Metro. So: what goes wrong? Currently, protobuf-net has a lot (and I do mean a lot) of code built on top of ILGenerator, the reflection class that underpins meta-programming. This class is great for hardcore library builders: you can build individual methods (via DynamicMethod) or entire assemblies (via AssemblyBuilder). It is very low level, since you are writing raw IL – but I’m not completely crazy, so I wrap that up in some utility methods. There are a few problem., though:

• ILGenerator usually expects be be working on the current framework
• Even though you can use a “reflection only load”, this still fails horribly for System.Runtime.dll (aka Metro for .NET)
• In early versions of .NET (and I try to support .NET 2 and upwards), the ability to properly inspect attribute data against reflection-only types/members (GetCustomAttributesData) is simply missing

So… basically, it all falls to pieces for generating as assembly targeting Metro for .NET, based on a Metro for .NET input assembly.

What I would need is some kind of utility that is:

• Broadly similar to Reflection.Emit, so I don’t have to rewrite a few thousand lines of code (since I want to keep that code for the runtime meta-programming on full .NET)
• Able to load and work with alternative frameworks without getting confused
• Able to give full attribute information, even on .NET 2
• Inbuilt or free, to match the existing license

Who would go to the trouble of writing such a thing?

Maybe somebody who is already writing compiler-like tools that aren’t tied to a specific framework, and who has run into the existing limitations of Reflection.Emit? Maybe somebody writing a Java/.NET bridge? Like IKVM.NET ?

Actually, for my purposes I don’t need most of IKVM. I just need one tiny piece of it: IKVM.Reflection. This library was specifically written to mimic the existing Reflection.Emit API, but without all the problems. In particular, it has a Universe class that is a bit like an AppDomain; you load assemblies into a Universe, and that is what is used for resolution etc after that. Here’s a simple example – which will immediately make sense to anyone used to Reflection.Emit – or another example specifically targeting Metro for .NET. The nice thing about the API is that mostly I can change between IKVM and Reflection.Emit via a simple:

#if FEAT_IKVM
using Type = IKVM.Reflection.Type;
using IKVM.Reflection;
#else
using System.Reflection;
#endif

OK, I would be grossly exaggerating if I claimed that was the only change I had to make, but that’s the most demonstrable bit. What this means is that just for my cross-platform compiler, everything in the Model, Strategy and Compiler modules (see previous blog) switches to IKVM terminology. The Core still uses System terminology, and the Runtime doesn’t apply (I obviously can’t instantiate/execute types/methods from another framework, even if I can inspect them).

With this technique, I can now successfully load a Metro for .NET assembly (such as a DTO), and generate a fully static-compiled assembly (“Standalone” on the diagram) targeting Metro for .NET. No reflection at runtime, no nasty hacks – just: an assembly that wasn’t generated by the MS tools.

I’m still working on turning this into a slicker tool (comparable to, say, SGEN), but a working illustration is in the repo; in particular, see MetroDTO (some sample DTOs), TestIkvm (my proof-of-concept compiler hard-coded to MetroDTO), and Metro_DevRig (which shows the generated assembly working in a Metro for .NET application, including performance comparisons to XmlSerializer and DataContractSerializer).

Additionally, it seems extremely likely that the same tool should be able to write platform-targeted assemblies for Silverlight, Phone 7, XNA, CF, etc. Which is nice.

Thanks and Acknowledgements

I’m hugely indebted to Jeroen Frijters in all of this; both for providing IKVM.Reflection, and for his direct and very prompt assistance when playing with all the above. Mainly for correcting my own brain-dead mistakes, but also for a very quick bug-fix when needed. The library is awesome, thanks.

(as a small caveat, note that protobuf-net is currently exposing a locally built version of IKVM.Reflection; this will be rectified after the next dev build of IKVM)