03-28-2019, 11:12 PM
Re-reading ASP.Net Core request bodies with EnableBuffering()
<div style="margin: 5px 5% 10px 5%;"><img src="http://www.sickgaming.net/blog/wp-content/uploads/2019/03/re-reading-asp-net-core-request-bodies-with-enablebuffering.jpg" width="58" height="58" title="" alt="" /></div><div><div class="row justify-content-center">
<div class="col-md-2">
<div><img src="http://www.sickgaming.net/blog/wp-content/uploads/2019/03/re-reading-asp-net-core-request-bodies-with-enablebuffering.jpg" width="58" height="58" alt="Avatar" class="avatar avatar-58 wp-user-avatar wp-user-avatar-58 photo avatar-default"></p>
<p>Jeremy</p>
</div>
</div>
</div>
<div class="entry-meta">
<p>March 27th, 2019</p>
<p> <!–<span class="posted-on">Posted on <a href="https://devblogs.microsoft.com/aspnet/re-reading-asp-net-core-request-bodies-with-enablebuffering/" rel="bookmark"><time class="entry-date published updated" datetime="2019-03-27T13:51:50+00:00">March 27, 2019</time></a></span><span class="byline"> by <span class="author vcard"><a class="url fn n" href="https://devblogs.microsoft.com/aspnet/author/yumengmicrosoft-com/">Jeremy Meng</a></span></span>–> </div>
<p><!-- .entry-meta --> </p>
<p>In some scenarios there’s a need to read the request body multiple times. Some examples include</p>
<ul>
<li>Logging the raw requests to replay in load test environment</li>
<li>Middleware that read the request body multiple times to process it</li>
</ul>
<p>Usually <code>Request.Body</code> does not support rewinding, so it can only be read once. A straightforward solution is to save a copy of the stream in another stream that supports seeking so the content can be read multiple times from the copy.</p>
<p>In ASP.NET framework it was possible to read the body of an HTTP request multiple times using <a href="https://docs.microsoft.com/en-us/dotnet/api/system.web.httprequest.getbufferedinputstream?view=netframework-4.7.2"><code>HttpRequest.GetBufferedInputStream</code> method</a>. However, in ASP.NET Core a different approach must be used.</p>
<p>In ASP.NET Core 2.1 we added <a href="https://docs.microsoft.com/en-us/dotnet/api/microsoft.aspnetcore.http.httprequestrewindextensions.enablebuffering?view=aspnetcore-2.1">an extension method <code>EnableBuffering()</code></a> for <code>HttpRequest</code>. This is the suggested way to enable request body for multiple reads. Here is an example usage in the <code>InvokeAsync()</code> method of a custom ASP.NET middleware:</p>
<pre><code class="csharp">public async Task InvokeAsync(HttpContext context, RequestDelegate next)
{ context.Request.EnableBuffering(); // Leave the body open so the next middleware can read it. using (var reader = new StreamReader( context.Request.Body, encoding: Encoding.UTF8, detectEncodingFromByteOrderMarks: false, bufferSize: bufferSize, leaveOpen: true)) { var body = await reader.ReadToEndAsync(); // Do some processing with body… // Reset the request body stream position so the next middleware can read it context.Request.Body.Position = 0; } // Call the next delegate/middleware in the pipeline await next(context);
}
</code></pre>
<p>The backing <code>FileBufferingReadStream</code> uses memory stream of a certain size first then falls back to a temporary file stream. By default the size of the memory stream is 30KB. There are also other <code>EnableBuffering()</code> overloads that allow specifying a different threshold, and/or a limit for the total size:</p>
<pre><code class="csharp">public static void EnableBuffering(this HttpRequest request, int bufferThreshold) public static void EnableBuffering(this HttpRequest request, long bufferLimit) public static void EnableBuffering(this HttpRequest request, int bufferThreshold, long bufferLimit)
</code></pre>
<p>For example, a call of</p>
<pre><code class="csharp">context.Request.EnableBuffering(bufferThreshold: 1024 * 45, bufferLimit: 1024 * 100);
</code></pre>
<p>enables a read buffer with limit of 100KB. Data is buffered in memory until the content exceeds 45KB, then it’s moved to a temporary file. By default there’s no limit on the buffer size but if there’s one specified and the content of request body exceeds the limit, an <code>System.IOException</code> will be thrown.</p>
<p>These overloads offer flexibility if there’s a need to fine-tune the buffering behaviors. Just keep in mind that:</p>
<ul>
<li>Even though the memory stream is rented from a pool, it still has memory cost associated with it.</li>
<li>After the read is over the <code>bufferThreshold</code> the performance will be slower since a file stream will be used.</li>
</ul>
<div class="authorinfoarea">
<div><img src="http://www.sickgaming.net/blog/wp-content/uploads/2019/03/re-reading-asp-net-core-request-bodies-with-enablebuffering-1.jpg" width="96" height="96" alt="Avatar" class="avatar avatar-96 wp-user-avatar wp-user-avatar-96 photo avatar-default"></div>
<div>
<h5><a class="no-underline" aria-label="Jeremy Meng" target="_blank" href="https://devblogs.microsoft.com/aspnet/author/yumengmicrosoft-com/" rel="noopener noreferrer">Jeremy Meng</a></h5>
<p>Software Development Engineer</p>
<p><strong>Follow Jeremy</strong> <a class="no-underline stayinformed hvr-pop" aria-label="Jeremy Meng RSS Feed" target="_blank" href="https://devblogs.microsoft.com/aspnet/author/yumengmicrosoft-com/feed/" rel="noopener noreferrer"></a></p>
<p> <!–</p>
<hr class="authorarea">–> </div>
</p></div>
</div>
<div style="margin: 5px 5% 10px 5%;"><img src="http://www.sickgaming.net/blog/wp-content/uploads/2019/03/re-reading-asp-net-core-request-bodies-with-enablebuffering.jpg" width="58" height="58" title="" alt="" /></div><div><div class="row justify-content-center">
<div class="col-md-2">
<div><img src="http://www.sickgaming.net/blog/wp-content/uploads/2019/03/re-reading-asp-net-core-request-bodies-with-enablebuffering.jpg" width="58" height="58" alt="Avatar" class="avatar avatar-58 wp-user-avatar wp-user-avatar-58 photo avatar-default"></p>
<p>Jeremy</p>
</div>
</div>
</div>
<div class="entry-meta">
<p>March 27th, 2019</p>
<p> <!–<span class="posted-on">Posted on <a href="https://devblogs.microsoft.com/aspnet/re-reading-asp-net-core-request-bodies-with-enablebuffering/" rel="bookmark"><time class="entry-date published updated" datetime="2019-03-27T13:51:50+00:00">March 27, 2019</time></a></span><span class="byline"> by <span class="author vcard"><a class="url fn n" href="https://devblogs.microsoft.com/aspnet/author/yumengmicrosoft-com/">Jeremy Meng</a></span></span>–> </div>
<p><!-- .entry-meta --> </p>
<p>In some scenarios there’s a need to read the request body multiple times. Some examples include</p>
<ul>
<li>Logging the raw requests to replay in load test environment</li>
<li>Middleware that read the request body multiple times to process it</li>
</ul>
<p>Usually <code>Request.Body</code> does not support rewinding, so it can only be read once. A straightforward solution is to save a copy of the stream in another stream that supports seeking so the content can be read multiple times from the copy.</p>
<p>In ASP.NET framework it was possible to read the body of an HTTP request multiple times using <a href="https://docs.microsoft.com/en-us/dotnet/api/system.web.httprequest.getbufferedinputstream?view=netframework-4.7.2"><code>HttpRequest.GetBufferedInputStream</code> method</a>. However, in ASP.NET Core a different approach must be used.</p>
<p>In ASP.NET Core 2.1 we added <a href="https://docs.microsoft.com/en-us/dotnet/api/microsoft.aspnetcore.http.httprequestrewindextensions.enablebuffering?view=aspnetcore-2.1">an extension method <code>EnableBuffering()</code></a> for <code>HttpRequest</code>. This is the suggested way to enable request body for multiple reads. Here is an example usage in the <code>InvokeAsync()</code> method of a custom ASP.NET middleware:</p>
<pre><code class="csharp">public async Task InvokeAsync(HttpContext context, RequestDelegate next)
{ context.Request.EnableBuffering(); // Leave the body open so the next middleware can read it. using (var reader = new StreamReader( context.Request.Body, encoding: Encoding.UTF8, detectEncodingFromByteOrderMarks: false, bufferSize: bufferSize, leaveOpen: true)) { var body = await reader.ReadToEndAsync(); // Do some processing with body… // Reset the request body stream position so the next middleware can read it context.Request.Body.Position = 0; } // Call the next delegate/middleware in the pipeline await next(context);
}
</code></pre>
<p>The backing <code>FileBufferingReadStream</code> uses memory stream of a certain size first then falls back to a temporary file stream. By default the size of the memory stream is 30KB. There are also other <code>EnableBuffering()</code> overloads that allow specifying a different threshold, and/or a limit for the total size:</p>
<pre><code class="csharp">public static void EnableBuffering(this HttpRequest request, int bufferThreshold) public static void EnableBuffering(this HttpRequest request, long bufferLimit) public static void EnableBuffering(this HttpRequest request, int bufferThreshold, long bufferLimit)
</code></pre>
<p>For example, a call of</p>
<pre><code class="csharp">context.Request.EnableBuffering(bufferThreshold: 1024 * 45, bufferLimit: 1024 * 100);
</code></pre>
<p>enables a read buffer with limit of 100KB. Data is buffered in memory until the content exceeds 45KB, then it’s moved to a temporary file. By default there’s no limit on the buffer size but if there’s one specified and the content of request body exceeds the limit, an <code>System.IOException</code> will be thrown.</p>
<p>These overloads offer flexibility if there’s a need to fine-tune the buffering behaviors. Just keep in mind that:</p>
<ul>
<li>Even though the memory stream is rented from a pool, it still has memory cost associated with it.</li>
<li>After the read is over the <code>bufferThreshold</code> the performance will be slower since a file stream will be used.</li>
</ul>
<div class="authorinfoarea">
<div><img src="http://www.sickgaming.net/blog/wp-content/uploads/2019/03/re-reading-asp-net-core-request-bodies-with-enablebuffering-1.jpg" width="96" height="96" alt="Avatar" class="avatar avatar-96 wp-user-avatar wp-user-avatar-96 photo avatar-default"></div>
<div>
<h5><a class="no-underline" aria-label="Jeremy Meng" target="_blank" href="https://devblogs.microsoft.com/aspnet/author/yumengmicrosoft-com/" rel="noopener noreferrer">Jeremy Meng</a></h5>
<p>Software Development Engineer</p>
<p><strong>Follow Jeremy</strong> <a class="no-underline stayinformed hvr-pop" aria-label="Jeremy Meng RSS Feed" target="_blank" href="https://devblogs.microsoft.com/aspnet/author/yumengmicrosoft-com/feed/" rel="noopener noreferrer"></a></p>
<p> <!–</p>
<hr class="authorarea">–> </div>
</p></div>
</div>