Cysharp/MemoryPack
Zero encoding extreme performance binary serializer for C# and Unity. - Cysharp/MemoryPack
4.6k stars · 308 forks · C#
README
MemoryPack
Zero encoding extreme performance binary serializer for C# and Unity.
Compared with System.Text.Json, protobuf-net, MessagePack for C#, Orleans.Serialization. Measured by .NET 7 / Ryzen 9 5950X machine. These serializers have IBufferWriter<byte> method, serialized using ArrayBufferWriter<byte> and reused to avoid measure buffer copy.
For standard objects, MemoryPack is x10 faster and x2 ~ x5 faster than other binary serializers. For struct array, MemoryPack is even more powerful, with speeds up to x50 ~ x200 greater than other serializers. MemoryPack is my 4th serializer, previously I've created well known serializers, ZeroFormatter, Utf8Json, MessagePack for C#. The reason for MemoryPack's speed is due to its C#-specific, C#-optimized binary format and a well tuned implementation based on my past experience. It is also a completely new design utilizing .NET 7 and C# 11 and the Incremental Source Generator (.NET Standard 2.1 (.NET 5, 6) and there is also Unity support). Other serializers perform many encoding operations such as VarInt encoding, tag, string, etc. MemoryPack format uses a zero-encoding design that copies as much C# memory as possible. Zero-encoding is similar to FlatBuffers, but it doesn't need a special type, MemoryPack's serialization target is POCO. Other than performance, MemoryPack has these features.
Support modern I/O APIs (IBufferWriter<byte>, ReadOnlySpan<byte>, ReadOnlySequence<byte>) Native AOT friendly Source Generator based code generation, no Dynamic CodeGen (IL.Emit) Reflectionless non-generics APIs Deserialize into existing instance Polymorphism (Union) serialization Limited version-tolerant (fast/default) and full version-tolerant support Circular reference serialization PipeWriter/Reader based streaming serialization TypeScript code generation and ASP.NET Core Formatter Unity (2021.3) IL2CPP Support via .NET Source Generator
Installation This library is distributed via NuGet. For best performance, recommend to use .NET 7. Minimum requirement is .NET Standard 2.1.
PM> Install-Package MemoryPack
And also a code editor requires Roslyn 4.3.1 support, for example Visual Studio 2022 version 17.3, .NET SDK 6.0.401. For details, see the Roslyn Version Support document. For Unity, the requirements and installation process are completely different. See the Unity section for details. Quick Start Define a struct or class to be serialized and annotate it with the [MemoryPackable] attribute and the partial keyword. using MemoryPack;
[MemoryPackable] public partial class Person { public int Age { get; set; } public string Name { get; set; } } Serialization code is generated by the C# source generator feature which implements the IMemoryPackable<T> interface. In Visual Studio you can check a generated code by using a shortcut Ctrl+K, R on the class name and select *.MemoryPackFormatter.g.cs. Call MemoryPackSerializer.Serialize<T>/Deserialize<T> to serialize/deserialize an object instance. var v = new Person { Age = 40, Name = "John" };
var bin = MemoryPackSerializer.Serialize(v); var val = MemoryPackSerializer.Deserialize<Person>(bin); Serialize method supports a return type of byte[] as well as it can serialize to IBufferWriter<byte> or Stream. Deserialize method supports ReadOnlySpan<byte>, ReadOnlySequence<byte> and Stream. And there are alse non-generics versions. Built-in supported types These types can be serialized by default:
.NET primitives (byte, int, bool, char, double, etc.) Unmanaged types (Any enum, Any user-defined struct which doesn't contain reference types) string, decimal, Half, Int128, UInt128, Guid, Rune, BigInteger TimeSpan, DateTime, DateTimeOffset, TimeOnly, DateOnly, TimeZoneInfo Complex, Plane, Quaternion Matrix3x2, Matrix4x4, Vector2, Vector3, Vector4 Uri, Version, StringBuilder, Type, BitArray, CultureInfo T[], T[,], T[,,], T[,,,], Memory<>, ReadOnlyMemory<>, ArraySegment<>, ReadOnlySequence<> Nullable<>, Lazy<>, KeyValuePair<,>, Tuple<,...>, ValueTuple<,...> List<>, LinkedList<>, Queue<>, Stack<>, HashSet<>, SortedSet<>, PriorityQueue<,> Dictionary<,>, SortedList<,>, SortedDictionary<,>, ReadOnlyDictionary<,> Collection<>, ReadOnlyCollection<>, ObservableCollection<>, ReadOnlyObservableCollection<> IEnumerable<>, ICollection<>, IList<>, IReadOnlyCollection<>, IReadOnlyList<>, ISet<> IDictionary<,>, IReadOnlyDictionary<,>, ILookup<,>, IGrouping<,>, ConcurrentBag<>, ConcurrentQueue<>, ConcurrentStack<>, ConcurrentDictionary<,>, BlockingCollection<> Immutable collections (ImmutableList<>, etc.) and interfaces (IImmutableList<>, etc.)
Define [MemoryPackable] class / struct / record / record struct [MemoryPackable] can annotate to any class, struct, record, record struct and interface. If a type is struct or record struct which contains no reference types (C# Unmanaged types) any additional annotation (ignore, include, constructor, callbacks) is not used, that serialize/deserialize directly from the memory. Otherwise, by default, [MemoryPackable] serializes public instance properties or fields. You can use [MemoryPackIgnore] to remove serialization target, [MemoryPackInclude] promotes a private member to serialization target. [MemoryPackable] public partial class Sample { // these types are serialized by default public int PublicField; public readonly int PublicReadOnlyField; public int PublicProperty { get; set; } public int PrivateSetPublicProperty { get; private set; } public int ReadOnlyPublicProperty { get; } public int InitProperty { get; init; } public required int RequiredInitProperty { get; init; }
// these types are not serialized by default int privateProperty { get; set; } int privateField; readonly int privateReadOnlyField;
// use [MemoryPackIgnore] to remove target of a public member [MemoryPackIgnore] public int PublicProperty2 => PublicProperty + PublicField;
// use [MemoryPackInclude] to promote a private member to serialization target [MemoryPackInclude] int privateField2; [MemoryPackInclude] int privateProperty2 { get; set; } } MemoryPack's code generator adds information about what members are serialized to the <remarks /> section. This can be viewed by hovering over the type with Intellisense.
All members must be memorypack-serializable, if not the code generator will emit an error.
MemoryPack has 35 diagnostics rules (MEMPACK001 to MEMPACK035) to be defined comfortably. If target type is defined MemoryPack serialization externally and registered, use [MemoryPackAllowSerialize] to silent diagnostics. [MemoryPackable] public partial class Sample2 { [MemoryPackAllowSerialize] public NotSerializableType? NotSerializableProperty { get; set; } } Member order is important, MemoryPack does not serialize the member-name or other information, instead serializing fields in the order they are declared. If a type is inherited, serialization is performed in the order of parent → child. The order of members can not change for the deserialization. For the schema evolution, see the Version tolerant section. The default order is sequential, but you can choose the explicit layout with [MemoryPackable(SerializeLayout.Explicit)] and [MemoryPackOrder()]. // serialize Prop0 -> Prop1 [MemoryPackable(SerializeLayout.Explicit)] public partial class SampleExplicitOrder { [MemoryPackOrder(1)] public int Prop1 { get; set; } [MemoryPackOrder(0)] public int Prop0 { get; set; } } Constructor selection MemoryPack supports both parameterized and parameterless constructors. The selection of the constructor follows these rules. (Applies to classes and structs).
If there is [MemoryPackConstructor], use it. If there is no explicit constructor (including private), use a parameterless one. If there is one parameterless/parameterized constructor (including private), use it. If there are multiple constructors, then the [MemoryPackConstructor] attribute must be applied to the desired constructor (the generator will not automatically choose one), otherwise the generator will emit an error. If using a parameterized constructor, all parameter names must match corresponding member names (case-insensitive).
[MemoryPackable] public partial class Person { public readonly int Age; public readonly string Name;
// You can use a parameterized constructor - parameter names must match corresponding members name (case-insensitive) public Person(int age, string name) { this.Age = age; this.Name = name; } }
// also supports record primary constructor [MemoryPackable] public partial record Person2(int Age, string Name);
public partial class Person3 { public int Age { get; set; } public string Name { get; set; }
public Person3() { }
// If there are multiple constructors, then [MemoryPackConstructor] should be used [MemoryPackConstructor] public Person3(int age, string name) { this.Age = age; this.Name = name; } } Serialization callbacks When serializing/deserializing, MemoryPack can invoke a before/after event using the [MemoryPackOnSerializing], [MemoryPackOnSerialized], [MemoryPackOnDeserializing], [MemoryPackOnDeserialized] attributes. It can annotate both static and instance (non-static) methods, and public and private methods. [MemoryPackable] public partial class MethodCallSample { // method call order is static -> instance [MemoryPackOnSerializing] public static void OnSerializing1() { Console.WriteLine(nameof(OnSerializing1)); }
// also allows private method [MemoryPackOnSerializing] void OnSerializing2() { Console.WriteLine(nameof(OnSerializing2)); }
// serializing -> / serialize / -> serialized [MemoryPackOnSerialized] static void OnSerialized1() { Console.WriteLine(nameof(OnSerialized1)); }
[MemoryPackOnSerialized] public void OnSerialized2() { Console.WriteLine(nameof(OnSerialized2)); }
[MemoryPackOnDeserializing] public static void OnDeserializing1() { Console.WriteLine(nameof(OnDeserializing1)); }
// Note: instance method with MemoryPackOnDeserializing, that not called if instance is not passed by `ref` [MemoryPackOnDeserializing] public void OnDeserializing2() { Console.WriteLine(nameof(OnDeserializing2)); }
[MemoryPackOnDeserialized] public static void OnDeserialized1() { Console.WriteLine(nameof(OnDeserialized1)); }
[MemoryPackOnDeserialized] public void OnDeserialized2() { Console.WriteLine(nameof(OnDeserialized2)); } } Callbacks allows parameterless method and ref reader/writer, ref T value method. For example, ref callbacks can write/read custom header before serialization process. [MemoryPackable] public partial class EmitIdData { public int MyProperty { get; set; }
[MemoryPackOnSerializing] static void WriteId<TBufferWriter>(ref MemoryPackWriter<TBufferWriter> writer, ref EmitIdData? value) where TBufferWriter : IBufferWriter<byte> // .NET Standard 2.1, use where TBufferWriter : class, IBufferWriter<byte> { writer.WriteUnmanaged(Guid.NewGuid()); // emit GUID in header. }
[MemoryPackOnDeserializing] static void ReadId(ref MemoryPackReader reader, ref EmitIdData? value) { // read custom header before deserialize var guid = reader.ReadUnmanaged<Guid>(); Console.WriteLine(guid); } } If set a value to ref value, you can change the value used for serialization/deserialization. For example, instantiate from ServiceProvider. // before using this formatter, set ServiceProvider // var options = MemoryPackSerializerOptions.Default with { ServiceProvider = provider }; // MemoryPackSerializer.Deserialize(value, options);
[MemoryPackable] public partial class InstantiateFromServiceProvider { static IServiceProvider serviceProvider = default!;
public int MyProperty { get; private set; }
[MemoryPackOnDeserializing] static void OnDeserializing(ref MemoryPackReader reader, ref InstantiateFromServiceProvider value) { if (value != null) return; value = reader.Options.ServiceProvider!.GetRequiredService<InstantiateFromServiceProvider>(); } } Define custom collection By default, annotated [MemoryPackObject] type try to serialize its members. However, if a type is a collection (ICollection<>, ISet<>, IDictionary<,>), use GenerateType.Collection to serialize it correctly. [MemoryPackable(GenerateType.Collection)] public partial class MyList<T> : List<T> { }
[MemoryPackable(GenerateType.Collection)] public partial class MyStringDictionary<TValue> : Dictionary<string, TValue> {
} Static constructor MemoryPackable class can not define static constructor because generated partial class uses it. Instead, you can define a static partial void StaticConstructor() to do the same thing. [MemoryPackable] public partial class CctorSample { static partial void StaticConstructor() { } } Polymorphism (Union) MemoryPack supports serializing interface and abstract class objects for polymorphism serialization. In MemoryPack this feature is called Union. Only interfaces and abstracts classes are allowed to be annotated with [MemoryPackUnion] attributes. Unique union tags are required. // Annotate [MemoryPackable] and inheritance types with [MemoryPackUnion] // Union also supports abstract class [MemoryPackable] [MemoryPackUnion(0, typeof(FooClass))] [MemoryPackUnion(1, typeof(BarClass))] public partial interface IUnionSample { }
[MemoryPackable] public partial class FooClass : IUnionSample { public int XYZ { get; set; } }
[MemoryPackable] public partial class BarClass : IUnionSample { public string? OPQ { get; set; } } // ---
IUnionSample data = new FooClass() { XYZ = 999 };
// Serialize as interface type. var bin = MemoryPackSerializer.Serialize(data);
// Deserialize as interface type. var reData = MemoryPackSerializer.Deserialize<IUnionSample>(bin);
switch (reData) { case FooClass x: Console.WriteLine(x.XYZ); break; case BarClass x: Console.WriteLine(x.OPQ); break; default: break; } tag allows 0 ~ 65535, it is especially efficient for less than 250. If an interface and derived types are in different assemblies, you can use MemoryPackUnionFormatterAttribute instead. Formatters are generated the way that they are automatically registered via ModuleInitializer in C# 9.0 and above.
Note that ModuleInitializer is not supported in Unity, so the formatter must be manually registered. To register your union formatter invoke {name of your union formatter}Initializer.RegisterFormatter() manually in Startup. For example UnionSampleFormatterInitializer.RegisterFormatter().
// AssemblyA [MemoryPackable(GenerateType.NoGenerate)] public partial interface IUnionSample { }
// AssemblyB define definition outside of target type [MemoryPackUnionFormatter(typeof(IUnionSample))] [MemoryPackUnion(0, typeof(FooClass))] [MemoryPackUnion(1, typeof(BarClass))] public partial class UnionSampleFormatter { } Union can be assembled in code via DynamicUnionFormatter<T>. // (ushort, Type)[] var formatter = new DynamicUnionFormatter<IFooBarBaz>( (0, typeof(Foo)), (1, typeof(Bar)), (2, typeof(Baz)) );
MemoryPackFormatterProvider.Register(formatter); Serialize API Serialize has three overloads. // Non generic API also available, these version is first argument is Type and value is object? byte[] Serialize<T>(in T? value, MemoryPackSerializerOptions? options = default) void Serialize<T, TBufferWriter>(in TBufferWriter bufferWriter, in T? value, MemoryPackSerializerOptions? options = default) async ValueTask SerializeAsync<T>(Stream stream, T? value, MemoryPackSerializerOptions? options = default, CancellationToken cancellationToken = default) For performance, the recommended API uses BufferWriter. This serializes directly into the buffer. It can be applied to PipeWriter in System.IO.Pipelines, BodyWriter in ASP .NET Core, etc. If a byte[] is required (e.g. RedisValue in StackExchange.Redis), the return byte[] API is simple and almost as fast. Note that SerializeAsync for Stream is asynchronous only for Flush; it serializes everything once into MemoryPack's internal pool buffer and then writes using WriteAsync. Therefore, the BufferWriter overload, which separates and controls buffer and flush, is better. If you want to do a complete streaming write, see the Streaming Serialization section. MemoryPackSerializerOptions MemoryPackSerializerOptions configures whether strings are serialized as UTF16 or UTF8. This can be configured by passing MemoryPackSerializerOptions.Utf8 for UTF8 encoding, MemoryPackSerializerOptions.Utf16 for UTF16 encoding or MemoryPackSerializerOptions.Default which defaults to UTF8. Passing null or using the default parameter results in UTF8 encoding. Since C#'s internal string representation is UTF16, UTF16 performs better. However, the payload tends to be larger; in UTF8, an ASCII string is one byte, while in UTF16 it is two bytes. Because the difference in size of this payload is so large, UTF8 is set by default. If the data is non-ASCII (e.g. Japanese, which can be more than 3 bytes, and UTF8 is larger), or if you have to compress it separately, UTF16 may give better results. While UTF8 or UTF16 can be selected during serialization, it is not necessary to specify it during deserialization. It will be automatically detected and deserialized normally. Additionaly you can get/set IServiceProvider? ServiceProvider { get; init; } from options. It is useful to get DI object(such as ILogger<T>) from serialization process(MemoryPackReader/MemoryPackWriter has .Options property). Deserialize API Deserialize has ReadOnlySpan<byte> and ReadOnlySequence<byte>, Stream overload and ref support. T? Deserialize<T>(ReadOnlySpan<byte> buffer) int Deserialize<T>(ReadOnlySpan<byte> buffer, ref T? value) T? Deserialize<T>(in ReadOnlySequence<byte> buffer) int Deserialize<T>(in ReadOnlySequence<byte> buffer, ref T? value) async ValueTask<T?> DeserializeAsync<T>(Stream stream) ref overload overwrites an existing instance, for details see the Overwrite section. DeserializeAsync(Stream) is not a complete streaming read operation, first it reads into MemoryPack's internal pool up to the end-of-stream, then it deserializes. If you want to do a complete streaming read operation, see the Streaming Serialization section. Overwrite To reduce allocations, MemoryPack supports deserializing to an existing instance, overwriting it. This can be used with the Deserialize(ref T? value) overload. var person = new Person(); var bin = MemoryPackSerializer.Serialize(person);
// overwrite data to existing instance. MemoryPackSerializer.Deserialize(bin, ref person); MemoryPack will attempt to overwrite as much as possible, but if the following conditions do not match, it will create a new instance (as in normal deserialization).
ref value (includes members in object graph) is null, set new instance only allows parameterless constructor, if parameterized constructor is used, create new instance if value is T[], reuse only if the length is the same, otherwise create new instance if value is collection that has .Clear() method(List<>, Stack<>, Queue<>, LinkedList<>, HashSet<>, PriorityQueue<,>, ObservableCollection, Collection, ConcurrentQueue<>, ConcurrentStack<>, ConcurrentBag<>, Dictionary<,>, SortedDictionary<,>, SortedList<,>, ConcurrentDictionary<,>) call Clear() and reuse it, otherwise create new instance
Version tolerant In default(GenerateType.Object), MemoryPack supports limited schema evolution.
unmanaged struct can't be changed anymore members can be added, but can not be deleted can change member name can't change member order can't change member type
[MemoryPackable] public partial class VersionCheck {