Tech Talk: RIFT Performance in 2.2

[Chantillary]

VOLAN!! Haven't tried it with my new upgraded rig, but back when I had the Phenom II x955 Black Edition CPU, once I target Volan my FPS went from 18 to (if there was a negative..) 1fps. Sometimes not moving unitl after the entire event was over. Now, as soon as I untargeted Volan, my frames shot back up to 18 ish. If you wanna test out a zone event, there is no better "Test" Than Volan, imo.

yeah, Volan can drop me from 20fps to 3 or 4 spf. That's seconds per frame. Even though my latency is only at about 90ms

[Pacifyer]

I think multicore optimization is a primary target as currently every dog haas a quad-core processor in its booth. Ignoring that much processing power is just getting old.

You have mentioned how much processor is used for models, cloaks. The calculation of these objects is independent and imho should be the first target for multithreading. You can also separate different layers into different threads - such as network layer, addon system, interface etc. There are so many opportunities and they surely should have been designed in the first place. Now rewriting code completely to support threads is a total pain.

For you to point out ideal interface customization (not fully an addon system but adding some hooks would make it such) I am encouraging to look at Everquest 2. It's interface was XML-based from day one and made possible clear and easy customization right away. They extended it's functionality after release and you could replace base interface elements with custom ones. I'm not saying their system is a way to go but surely you can learn some tricks to adopt to your system.

BTW looking at addon cpu load somethich tells me it is possible to move addon system to a separate thread.

[Misun]

Hopefully this will finally fix the Conquest/lowfps issues.

It has been terrible since CQ was first released.

[ZorbaTHut]

You have mentioned how much processor is used for models, cloaks. The calculation of these objects is independent and imho should be the first target for multithreading. You can also separate different layers into different threads - such as network layer, addon system, interface etc.

I suppose it's worth noting that the cloak physics are the only thing in the game right now that does take full advantage of multiple cores. If you have a 6-core processor with hyperthreading you can probably ramp cloak detail all the way up to max with little performance reduction. In some ways, the cloak system has been quite a nice acid test for our thread pools

The addon system, however, cannot be realistically multithreaded at all. It's intimately tied to the game state and game logic. Attempting to multithread the addon system would be begging for race conditions and deadlocks - that one's not even on the table.

Everything else exists somewhere on the spectrum from "mostly works already" to "not yet implemented, but definitely worth considering". It's going to be a long project though.

[Igneel]

Really glad to see posts like this. I enjoyed the tech talk on eve developer blogs, but to be able to read them for rift is super awesome. Keep em coming .

Great job again.

[Rounded]

I suppose it's worth noting that the cloak physics are the only thing in the game right now that does take full advantage of multiple cores. If you have a 6-core processor with hyperthreading you can probably ramp cloak detail all the way up to max with little performance reduction. In some ways, the cloak system has been quite a nice acid test for our thread pools

Problem with cloacks is that it's suspectible to the exact same texture issues as the blinking rocks in ashora et al, only difference is that the cloak texture stretches through your screen causing effects which might lead one to believe that their craphics card is on the fritz: http://i.imgur.com/NaCEniQ.jpg

The cloak gets stretched out and fills part of the screen for one frame. I just took off UI in Meridian and started hitting the dummy and then morphed the distorted texture thinking my GPU burned or runs hot, but no, it was the cloak. Turning off cloaks fixed this issue, but alas I cannot even see my cloaks as they will just randomly spazz out and always in combat.

[Beringer]

Zorba! I'm very happily surprised to see this detailed tech talk in the General Discussion forum! It's much appreciated! I may help you CPU-profiling, though I usually do profiling on game consoles (which has awesome tools). (If I give you callstacks I'd assume you'd be able to get proper symbols from your PDBs.)

Just some ideas on profiling:

1. Check L2 cache misses. Accessing memory is pretty much the slowest thing a CPU can do, and these can indicate generally badly written functions.
2. Try sampling profiling that captures callstacks for every sample. Check it with a hierarchical thread view. It was a long time since I did this on PC, perhaps some VTune successor has it. I assume you already use this - it's a great complement to manually instrumenting code.
3. Try Intel's GPA for GPU profiling, when you get to it. I like it.

[Akana]

Oh yes!!! I'm happy to hear you're working on the performance.

[Dissident]

I suppose it's worth noting that the cloak physics are the only thing in the game right now that does take full advantage of multiple cores. If you have a 6-core processor with hyperthreading you can probably ramp cloak detail all the way up to max with little performance reduction. In some ways, the cloak system has been quite a nice acid test for our thread pools

I have a hyperthreading hex core i7 in this rig I just finished building and yes, I can ramp cloak detail all the way up with absolutely no noticeable change in performance.

It makes me sad to see 1 thread doing almost all the work while the other 11 sit nearly idle, so these are really exciting changes on the horizon. It's nice that dual core systems can run Rift without issues, but some of us have higher end gaming machines and would love to see Rift at least attempt to make them work a little.

[Antigonos]

Hi Zorba,

how do you evaluate the usefullness of a 64bit client?

[Pacifyer]

The addon system, however, cannot be realistically multithreaded at all. It's intimately tied to the game state and game logic. Attempting to multithread the addon system would be begging for race conditions and deadlocks - that one's not even on the table.

The addon system as per my knowledge does primarily read game data and very rarely writes anything directly. If you worry about read locks affecting performance then you can store a copy of data necessary for addons. Yes you'll have to duplicate some writes to have a current copy all the way but there is not that much data I suppose. However I can be mistaken.
All other functionality can use the event system to interact with other subsystems quite well.

If you are telling that addons will race for writes to user interface than I would say that interface should be integrated into addon system as a single system as they generally access same data. Also if both interface and addons will use totally same API it would generally simplify developement and open new doors in addon system for interface modding.

Hope you got some useful thoughts from my trash talk.

P.S. I am not telling you to make the addon system itself multithreaded. But it certainly can be in a separate thread from the main process which would be certainly nice seeing how much cpu it currently eats with all my addons enabled.

[Crankedqceez]

I just wanted to let you know Zorba, that this is a wonderful idea you had to open a thread to discust with the players. Continue like this its a great move !

Sincerely !

[Stanley Pain]

How about moving to a DX10+ render path? That would solve a lot of your performance woes.

[ZorbaTHut]

Hi Zorba,

how do you evaluate the usefullness of a 64bit client?

How about moving to a DX10+ render path? That would solve a lot of your performance woes.

These both fall under the category of "nice to have, but a real pain to add". As much as 64-bit conversion should be easy, there are always unexpected issues especially with third-party libraries, and switching to DX10 would require rewriting a bunch of rendering code. In both cases, the gains would be minimal - RIFT doesn't overflow a 32-bit memory space as is, and since we don't want to break compatibility with our DX9 players, we wouldn't be able to use DX10-only features. In both cases, we'd add a lot of extra work for our QA department, which would have to test an extra build or an extra render path.

If we had unlimited staff working on Rift we'd probably be doing both of those things, but as it is, we'd have to sacrifice very important and desired new features. Such as all the optimization work I'm doing.

[Antigonos]

Thanks a lot for clarification! I agree, working on actualy performance boosts and optimization has to have higher prio then 64bit or DX10 when you already know that it will not boost performance significantly.