Thursday, 19 May 2011

protobuf-net v2, beta

It has been a long time in the coming, I know. I make no excuses, but various other things have meant that it didn’t get much input for a while…

But! I’m happy to say that I’ve just pushed a beta download up onto the project site

So… this v2… what is it?

Basically, it is still the core protobuf stream, but against all the safe advice of my employer I rewrote the entire core. For many reasons:

  • the original design grew organically, and ideas that seemed good turned to make for some problematic code later
  • the vast overuse of generics (and in particular, generics at runtime via reflection) was really hurting some platforms
  • there were many things the design could not readily support – structs, runtime models, pre-built serialization dlls, usage on mobile devices, etc
  • the code path at runtime was just not as minimal as I would like
  • and a myriad of other things

So… I rewrote it. Fortunately I had a barrage of integration tests, which grew yet further during this exercise. The way I figured, I had 2 choices here:

  • go with a richer code-generator to use as a build task (not runtime)
  • go crazy with meta-programming

After some thought, I opted for the latter. In particular, this would (in my reasoning) give me lots of flexibility for keeping things runtime based (where you are able to, at least), and would be a good opportunity to have some fun and learn IL emit to dangerous levels (which, incidentally, turns out to be very valuable – hence mine and Sam’s work on dapper-dot-net).

So what is new?

I don’t claim this list is exhaustive:

  • the ability to define models at runtime (also useful for working with types you don’t control) (example)
  • the ability to have separate models for the same types
  • pre-generation to dll
  • runtime performance improvements
  • support for structs
  • support for immutable objects via surrogates (example)
  • the ability to run on mobile platforms (unity, wp7 – and perhaps moot for a while: MonoDroid, MonoTouch – but presumably Xamarin)
  • the ability to serialize object graphs preserving references (example)
  • the ability to work with types not known in advance (same)
  • probably about 40 other things that slip my mind

I’ll try to go into each of these in detail when I can

And what isn’t there yet?

This is beta; at some point I had to make a milestone cut – but some things aren’t quite complete yet; these are not usually needed (hint: I’m happy to use protobuf-net v2 on my employer’s code-base, so I have confidence in it, and confidence that the core paths have been smoke-tested)

But things that definitely aren’t there in the beta

  • WCF hooks
  • VS tooling (it is just the core runtime dlls)
  • round-trip safe / “extension” fields
  • extraction of .proto (schema) files from a model
  • my build / deploy script needs an overhaul
  • tons of documentation / examples
  • tooling to make the “mobile” story easier (but; it should work – it is being used in a few, for example)

But; have at it

If you want to play with it, go ahead. I only ask that if something behaves unexpectedly, drop me a line before declaring loudly “it sux, dude!”. It might just need some guidance, or maybe even a code fix (I’m far from perfect).

Likewise, any feature suggestions, etc; let me know.

47 comments:

Livingston said...

A long time ago, in a galaxy far far away.....

Livingston said...

You just know that R2D2 is speaking protobufs with all those beeps.

Anonymous said...

Missing the NonGeneric serialization implementations:(

Marc Gravell said...

@Anonymous that is fixed in the code, and will be in the next code drop. Actually, it is just the *forwarding* API that was missed - if you use RuntimeTypeModel.Default, that ***is*** the same non-generic API and works fine.

Anonymous said...

Have used your protobuf-net libraries with .NET CF and have cut sync and serialisation times down to half compared to XML. Just wanted to say thanks and that I am looking forward to give v2 beta a go.

Marc Gravell said...

@anon - cool; which version of CF? I need to get the CF versions pre-built in the next beta...

Anonymous said...

We are using CF3.5. Noticed they were not in the current beta, but no worries, more than happy to wait till a later one. r282 is working great in production as is.

Thanks again.

Marc Gravell said...

@Anonymous - I'll get a VM set up so I can build that ;p VS2010 hates CF :(

And I'm too nervous to install VS2008 *after* VS2010 to get the CF support

James K said...

Thanks, that would be awesome.
Understand with the VS order, wouldn't risk it myself. Will be sure to give you some feedback after you get a chance to put in in a future beta.

Cheers.

Anonymous said...

I hope the library would expose more ability to query the meta-model. For example, the MetaType class should expose method like FindMemeberValueByName, etc.

Marc Gravell said...

What would be the use-case of FindMemberValueByName ? How would it be different from regular reflection or `dynamic` ?

Sridharan Srinivasan said...

I would be very greatfull if it possible to Improve the Performance of the ReadTypedObject Method. I tested using EQATEC profiler and this method is consuming about 800 millseconds for deserializing a complex topic map (1003ms).
The StartSubItem (30ms) and EndSubTime (3ms). I suspect that it is the TryDeserializeAuxiliaryType method which is slowing it down.

Marc Gravell said...

I would need a concrete example to work from there. In most cases the "aux" methods are not needed. Happy to investigate, but needs an example.

Alexander said...

Thank you for "protobuf-net v2, beta".
It completely eliminate our problem with MissingMethodException on invoking a generic method. And it is much faster than v1. I have tested it on our solution and could not found any bugs so far.

lesderid said...

Hello.

I tried to use your project to store and compress maps for my game. Sadly, it didn't work: "Nested or jagged lists and arrays are not supported".

Is this not possible with the current ProtoBuf base, or did you just not implement it?

Would it be possible to implement it?

Marc Gravell said...

@lesderid the protobuf spec itself doesn't have any direct representation of a nested/jagged array/list/etc, but the library could spoof this if we needed (it would need some tweaks). The workaround is to store a list/array of objects that themselves *have* a list/array. The library could essentially pretend you had done that.

I would need a more specific example to consider it for implementation, though.

lesderid said...

@Marc Well, for my maps, I made a 2D List. (a wrapper for List>)

lesderid said...

Comment got messed up.

I made a wrapper for List{List{T}}.
(replace curly braces)

danielkzu said...

I wish all the WCF stuff wasn't even there in the main assembly altogether. (i.e. ProtoBuf.ServiceModel.*, Serializer.CreateFormatter, etc.).

Ideally, it should just be a very very very small core that just reads/writes from streams, and then everything else in separate assemblies (use extension methods, etc.).

Making the serializer non-static would enable nice extensibility via extension methods whenever you added a referece to the ProtoBuf.ServiceModel.dll for example.

Marc Gravell said...

The v2 serializer *is* non-static. The legacy static API is retained for compatibility and is just an alias to RuntimeTypeModel.Default.

Re the WCF stuff - sure: use the 2.0 build instead of 3.0; WCF is the main difference.

Anonymous said...

Can you see having any problems with different versions of protobuf on both sides of the wire? Has the wire format changed? It has been working fine for us but we just noticed a ProtoException when we upgraded the client protobuf assembly (without changing the protos that are used by v2 server side)

stack trace:
ProtoBuf.ProtoException: Exception of type 'ProtoBuf.ProtoException' was thrown.
at ProtoBuf.ProtoReader.StartSubItem(ProtoReader reader) in C:\Dev\protobuf-net\protobuf-net\ProtoReader.cs:line 599
at ProtoBuf.ProtoReader.ReadTypedObject(Object value, Int32 key, ProtoReader reader, Type type) in C:\Dev\protobuf-net\protobuf-net\ProtoReader.cs:line 531
at proto_36(Object , ProtoReader )...

Marc Gravell said...

It *should* be the same - if you help me repro I'll happy help

marc m. said...

Hello,

Any ideas when extensions will be fully supported in v2? We use extensions `extensively` and do a lot of TryGetValue to get the correct extension as we don't know them in advance..

Anyway thanks for this implementation of protobuf!

Marc Gravell said...

it is simply a case of grabbing the time - lots of different plates in the air at all times. Hard to commit to a specific deadline, but will try to get this done ASAP

rstrazz said...

Hi Marc,

You have an awesome library here, I love it! It's working great in my development environment but it's not loading at my web hosting company. They have medium trust but they also have Reflection enabled. I made a Stack Overflow post that lays it out in detail, perhaps you could take a peek if you have a moment? You must be really busy, thank you for looking if you have the chance:

http://stackoverflow.com/questions/7060292/protobuf-net-getting-permission-error-on-web-host-reflection-is-enabled-on-web-h

Dave Healy said...

Brilliant! The AsReference feature in V2 is spectacular. In about 15 minutes I've updated our serialization to use both DataContract and Protobuf. This means we get a large, unwieldy, but human readable XML doc and a fast efficient, but illegible binary format.

One question though - it looks like you have to use the ProtoContract attributes (as opposed to pigging back off the DataContract attributes) if you want to serialize something as a reference.

Any plans to hook into the DataContract(IsReference = true) attribute to indicate a ProtoMemeber should be serialized as reference?

Anonymous said...

Hi there,

I just wanted to drop a line to say thank you for your great library. I just ported an app to your V2 and it works like a charm (will probably benchmark it against the old one, since I'm a huge fan of performance). Your idea of emitting code is great and feels very natural, in fact. I eager to see updates on this topic!

Cheers.

יעקב דייויס - Yaakov Davis said...

I wonder, why IL emit? Why not Expressions?

I found them very useful for generating dynamic methods at runtime; I think they can be a good replacement for Emit in most cases; they're much easier to work with and readable.

יעקב דייויס - Yaakov Davis said...

See:
http://blogs.msdn.com/b/csharpfaq/archive/2009/09/14/generating-dynamic-methods-with-expression-trees-in-visual-studio-2010.aspx

יעקב דייויס - Yaakov Davis said...
This comment has been removed by the author.
Marc Gravell said...

@Yaakov Davis - I am very familiar with Expressions too (see http://www.infoq.com/articles/expression-compiler); reasons:

1: I want to support .NET 2.0 (actually, even 1.1 is possible with v2)
2: even in .NET 3.5, expression trees are very limited
3: in 4.0, they *still* can't do some of the crazy stuff I want, at least: certainly not conveniently

to emphasise, I'm no stranger there, but they simply aren't versatile for what I am doing here - and they make for a more cumbersome model (when my approach sits very will with "just do your bit of IL here")

Marc Gravell said...

@Yaakov Davis oh, additionally, I still hope to migrate to the IKVM emitter to allow better cross-platform emit; that *certainly* won't work with expressions.

יעקב דייויס - Yaakov Davis said...

Makes sense, thanks for the explanation and the additional info.

avagam said...

Hi,

I am new to protobuf and starting with v2 library for my C# 4.0 project. After downloading zip, I could not find protogen.exe.

What should I use to generate C# classes from existing proto files?

thx.

Marc Gravell said...

There have been no changes in protogen for v2; the v1 protogen (available on google-code) should be fine. Note that .proto is not necessary for protobuf-net - it can work "code first". However, a .proto approach is useful if you are planning to interop.

avagam said...

thanks Marc, got protogen.exe

I am taking proto file defined in java and use it with protogen to create cs file.

getting error message:
C:\dev\protobuf\old\NET30>protogen.exe -i:C:\temp\protobuf\SubscriptionService.p
roto -o:C:\temp\protobuf\SubscriptionService.cs -q
C:\temp\protobuf\SubscriptionService.proto: File does not reside within any path
specified using --proto_path (or -I). You must specify a --proto_path which en
compasses this file.

do I need to specify system path?

avagam said...

just thought maybe it's not possible at all. I mean taking a proto file defined in java and then creating cs file out of it...

Marc Gravell said...

It sound like the problem there is some import/include paths. I'll have to check what ProtoMember exposes there, but - can you simplify those paths at all?

avagam said...

added directory into PATH and PROTO_PATH variables, still getting error below. not sure what to do next...

C:\dev\protobuf\old\NET30>protogen.exe -i:SubscriptionService2.proto -o:SubscriptionService2.cs
protobuf-net:protogen - code generator for .proto
C:\dev\protobuf\old\NET30\SubscriptionService2.proto: No such file or directory
An error occurred parsing SubscriptionService2.proto

Here is a sample of .proto file taken from java project:

package com.da.buffers.subscription;

option java_package = "com.da.buffers.subscription";
option java_outer_classname = "SubscriptionService";

//---------------------------------------------//
//---------- Overview -------------------------//
//---------------------------------------------//

// Subscription service maintains and run users
// subscriptions to da reports.

//---------------------------------------------//
//---------- Query Definitions ----------------//
//---------------------------------------------//

message ListSubscriptionsRequest {
//required by services .proto standard
required uint32 version = 1;

//required by services .proto standard
required string partitionId = 2;

// id of the user performing the operation (operatorId)
required string userId = 3;
}

avagam said...

actually found solution, protogen expects proto file to be in the same dir as exe.

avagam said...

Using following proto file generates empty C# class... is it by design, or a bug in C# protogen?

package com.da.gpb;

option java_package = "com.da.gpb";

import "descriptor.proto";

extend google.protobuf.MessageOptions {
optional string mappingEnum = 50001;
}

extend google.protobuf.FieldOptions {
optional string mappedTo = 50002;
}

extend google.protobuf.FieldOptions {
optional string mapping = 50003;
}

extend google.protobuf.MessageOptions {
//used to list all request that uses this message as query parameter
optional string requestNames = 50004;

//used to list all request that uses this message as query response
optional string responseNames = 50005;

//used to define which field is holding the userId of the user issuing the query
optional string userIdFieldName = 50006;
}

avagam said...

Hi Marc,

For some reason protogen does NOT recognize imports... like this one in proto file from prev post

import "descriptor.proto";

I tried everything - setting PATH, copying all proto files in the same folder, and so on. Nothing seems to work.

What should I do to make protogen work with imports? please help

thanks!

P.S. Issue is easy to replicate, take proto file from prev post, save it as proto file in dir where protogen and descriptor.proto are located, run protogen.

avagam said...

any suggestions, please?

Marc Gravell said...

@avagam I'll have to take a look - I haven't touched protogen in the v2 release, so no upcoming changes yet.

For now, one workaround is to use google's protoc to compile to binary, then use that as the input to protogen (protogen allows either text or binary input).

avagam said...

thanks!

Alleycat said...

I'm currently writing a WinRT Metro app, and I'm wondering what it would take to utilize this library in a WinRT application?

Marc Gravell said...

@AlleyCat I have not yet had time to complete the WinRT port/testing. It does not currently compile for WinRT