[Dev News] Capcom Open Source RE Engine “Data Engine” RE:Dox

[xSicKxBot]

Capcom have just open sourced a key component of their AAA game engine RE Engine, the game engine powering most of their modern titles including Resident Evil, Street Fighter, Devil May Cry, Monster Hunter, Dragon’s Dogma and more. The open sourced component is RE:Dox, a C# powered “data engine”. While not as flashy as the renderer or interactive as the physics engine, data is a huge part of just about every large game so a high performance, easy to use, modular JSON driven data engine should be incredibly useful to the wider game development community. Best of all it is released under the permissive Apache 2 open source license.

First.. What is RE:Dox?

RE:Dox is a high-performance, token-based structured data engine for .NET, developed as one of the core components of ‘REX’ Technology for CAPCOM’s next-generation game engine.

RE:Dox is not just a JSON serializer. It parses structured data into a compact, fixed-size token DOM / IR, then uses that structural index for reading, editing, format conversion, serialization, deserialization, and automatic parallel deserialization.

Second… why use RE:Dox?

RE:Dox combines three goals that usually conflict:

• the parse speed of a tape DOM (
  Code:

  System.Text.Json.JsonDocument
  ),
  
  
• the editability of a node DOM (
  Code:

  System.Text.Json.Nodes.JsonNode
  ),
  
  
• the flexibility of
  Code:

  Newtonsoft.Json
  (position-independent
  Code:

  $type
  ,
  Code:

  $id
  /
  Code:

  $ref
  , out-of-order constructor binding).
  
  

What you get:

• High-performance serialization and deserialization — on the benchmark datasets below, up to ~1.8x faster sequential deserialization, up to ~2.8x faster automatic parallel deserialization, and up to ~1.6x faster serialization than System.Text.Json.
  
  
• Lower allocation on tested workloads — for example,
  Code:

  canada.json
  deserializes with about 2.56 MB allocated by RE:Dox versus about 8.53 MB by System.Text.Json. JSON, JSON5, CBOR, MessagePack, INI, and DOX are integrated with the core model; TOML, XML, HTML, and CSV are currently preview components.
  
  
• One converter model across formats —
  Code:

  DataConverter<T>
  targets the format-agnostic
  Code:

  DataReader
  /
  Code:

  DataWriter
  abstractions instead of a specific wire format.
  
  
• Mutable token DOM — edit objects and arrays without replacing the document with a heavyweight managed object tree.
  
  
• Trivia-preserving JSON5 editing — comments and other preserved trivia can survive document edits and re-encoding.
  
  
• Compatibility layers for System.Text.Json, Newtonsoft.Json, and DataContractJsonSerializer.
  
  
• Apache-2.0 licensed.
  
  

Key Links

RE:Dox GitHub Repository

More About RE Engine

You can learn more about the RE Engine and the underyling, now open sourced RE:Dox data engine in the below. Special thanks to Synty Pass for sponsoring this video.