And how Age of Ascent helped shape ASP.NET's high performance webserver
Some excerpts from the panel interview
David, those TechEmpower results, pretty impressive, I was reading that article and pretty impressive.
It was good. We spent the last two years doing ASP.NET Core; even before it was cross platform; and one day Damian Edwards sat down and said, why do we suck? I think I was there, and I kinda said, because we don’t care.
You weren’t even on the map right?
It wasn’t because we didn’t care, it was because people would always say “Going that fast is impossible. Who cares, it’s not realistic?” But the whole point is it kinda gave us a reason to drive through that performance.
So it was a culture change for our team - it was a big thing that we did. Where we had everyone caring about allocations, caring about performance. We had a rig called the Smurf Lab that had a bunch of hardware set up to do perf testing.
So every day we’d have some new change that would go on the Smurf Lab; go check-in your code, run some stuff and we’d go “Oh we’ve got 10 more requests per second” - we’d have a count down. Today we got 5%, today we got 10%. We had a graph of us going from a couple hundred rps to 1.1 million.
Then we had Ben Adams; who is a guy from the community who appeared out of nowhere, pretty much, and he started using ASP.NET Core and he loves performance. He began to send tons of PRs to Kestrel, our web server, and we ended up having really good performance, after that whole thing.
So you mentioned someone from the community, ASP.NET Core is all Open Source?
All Open Source. The entire thing is open source.
How does it feel in the team when you are working open source?
It is, both amazing and awesome at the same time. You get the constant feedback that you want; and don’t want sometimes; and at the same time we get great contributions from people that are just interested in coding. We get great feedback from people.
The other thing I want to point out, because we always talk about the community, but one thing open source has done; I mean ASP.NET is open source for much longer than the rest of the stack was, but now that the rest of the stack is open source from the runtime, to the framework, to ASP.NET.
People like Ben Adams; they basically look at what they do and they go “right, where am I losing performance?” So for example Ben Adams recently submitted a PR that changes our threading primitives because the locking took too long. He found a way to make that smarter and faster.
So the nice thing is; not only do we have a community outside, but also the community helps to tie the teams on the other end more together by forcing PRs that span multiple layers and do these cross-cutting optimizations. That no matter what organisation set up you have; you have some organisational thinking where you only look at your own layers to a certain extent and that helps glue the whole thing together; by saying “No no, I want great perfomance from the request, all the way down the networking stack, to the runtime; the GC and many other things”.
That makes a lot of sense, I didn’t really think of open source that way. They don’t know our organisational structure and are really thinking about the whole project at once.
I think what tends to happen is when you are working on a thing for so long; you get blinders, so you see your code and you know how it works and then someone like Ben comes in and says “Oh, you can do it 4 different ways” and you go “Hmmm, you’re right - send a PR” and he sends 4 PRs; and we go “that one’s good” and we merge it and we go “Oh my gosh, we didn’t know we could do this.” Its kinda dope.
These things come out of the entire perfomance culture and we are doing it across the stack now.
And that’s actually an interesting point as well. We are not just optimizing code paths, we are actually at some points introducing new features to allow you to scale, for example a website. So pooling is something that probably wouldn’t be something that is on by default, becuase there are some caveats to using it; but its a feature that you can opt into if scale is super important to you.
Actually scale and Kestrel make me think of the Age of Ascent game.
Yea. I think Ben had a video of a GC pause - but in space. Like, all the [space]ships disappeared and it was an actual GC pause visual, and I was like “Wow! That’s super cool”
It was pretty amazing. I kinda remember it. It’s that massively multiplayer realtime game. So that’s all based on Kestrel?
Yeah now. So, before… he wrote his own webserver and was doing his own thing and he came here and we had lunch, and I said “Dude, just use Kestrel” and he went “Alright”. So he threw his stuff away and started helping with performance with Kestrel and now he uses it in his game.
You can watch the panel interview below or watch the whole panel on Channel 9