+ Reply to Thread
Page 1 of 5 1 2 3 4 5 LastLast
Results 1 to 15 of 63
Like Tree18Likes

  Click here to go to the first Rift Team post in this thread.   Thread: Watchdog puzzles

  1. #1
    Sword of Telara Semele's Avatar
    Join Date
    Mar 2011
    Posts
    872

    Default Watchdog puzzles

    So, just requiring some clarification on how user defined events fired in an event call are handled.

    My perception of these events is that they fire an event and proceed (ie, not a glorified function call) and other addons linked to these events would run that event concurrently. Because, if that's not the case and my plug-ins are now forcing KBM to wait on them returning, this isn't going to work out too well.

    I do like the idea of having a watchdog, but I feel I'm lacking true tools to make it trivial. I looked in to co-routines and, well, they're ok but a long-long way from threading.

    For example, in an ideal world my mechanic timers would be single thread jobs which would start and update via their own entry in to an event table, or at the very least the timer engine would be on its own thread and fire from its own event trigger.

    What I'm really tempted to do, is place each individual engine in it's own addon and all trigger from their own hook to System.Update.Begin. Would this make any difference to the watchdog, and/or performance?
    Rank 76 Guardian Mage

  2. #2
    Sword of Telara Semele's Avatar
    Join Date
    Mar 2011
    Posts
    872

    Default

    I just want to quickly expand on this to try and clear up what I mean.

    a) If I have a library that caches and dispatches Unit.Availability.* and pulls the data with Inspect.Unit.Detail, then it has to wait on every addon that uses this event to return, then odds are the Watchdog will be angry. This is what I'm hoping doesn't happen.

    b) If the above is true, then you'd defeat the watchdog by actually multiplying the amount of work done and have each addon pull it's own Inspect.Unit.Detail within its own hook to the event rather than pull it from a single dispatching library.

    I'm just hoping I'm wrong, but can't really proceed with a design until I know exactly how things are handled by the watchdog with regards to events and at what scope addons are identified.

    I'm just worried that if the above is true then libraries become obsolete and it proves better to just duplicate handlers entirely no matter how many addons are using the exact same process.
    Rank 76 Guardian Mage

  3. #3
    Plane Touched
    Join Date
    Feb 2012
    Posts
    228

    Default

    I believe the order the events are fired is "first subscribed, first served" (sequentially, not concurrently), and, if at some point of that chain an addon depletes the watchdog time, the rest of the chain won't be able to execute their own handlers.

    In that scenario, doing the same processing for each addon would increase the chance of a performance warning/error.

  4. #4
    Sword of Telara DoomSprout's Avatar
    Join Date
    Apr 2011
    Posts
    876

    Default

    Quote Originally Posted by Semele View Post
    a) If I have a library that caches and dispatches Unit.Availability.* and pulls the data with Inspect.Unit.Detail, then it has to wait on every addon that uses this event to return, then odds are the Watchdog will be angry. This is what I'm hoping doesn't happen.
    Sorry, that's exactly what happens.

    Quote Originally Posted by Semele View Post
    b) If the above is true, then you'd defeat the watchdog by actually multiplying the amount of work done and have each addon pull it's own Inspect.Unit.Detail within its own hook to the event rather than pull it from a single dispatching library.
    Even this won't help. The watchdog runs a timer for the entire event, it doesn't care whether the event is handled by 1 or 100 different addons.

    I'm not sure what the criteria for picking the addon to cause the error in is, but my best guess is that the first addon to make an API call after an event has taken 0.1s is going to get the chop.

    Quote Originally Posted by Semele View Post
    I'm just hoping I'm wrong, but can't really proceed with a design until I know exactly how things are handled by the watchdog with regards to events and at what scope addons are identified.

    I'm just worried that if the above is true then libraries become obsolete and it proves better to just duplicate handlers entirely no matter how many addons are using the exact same process.
    I *think* event handlers work like this...

    1. RIFT fires an event (e.g. unit available).
    2. The watchdog starts it's stopwatch.
    3. The addon system looks in the event table {Event.Unit.Availability.Full}, and calls each registered event handler in turn, making a note of which addon owns the handler.
    4. The watchdog's stopwatch hits 0.1 seconds, it looks up the last event handler that was called, and fires up the 'Your Addon Sucks' dialog of doom.
    5. No other event handlers get called.

    Given this, using libraries or not is irrelevant - RIFT only care about how long the overall event took to run.

    Gadgets: Unit Frames and Other Stuff for RIFT

  5. #5
    Champion
    Join Date
    Jun 2011
    Posts
    561

    Default

    The intention of the Watchdog is good, the implementation a PITA. I even get the feeling that the players PC is a factor cause some people came back with me saying they get performance warnings when openeing the UI of one of my addons while I don't get any at that point. I have a fairly average PC (I5-750 non OC, SSD, 5850 and 16 GB ram, 64bit). Some of the performance warning reports I got, I can only attribute to lower performance PC. A second of API CPU is not the same for everybody. A slower CPU will be able to do much less than a faster PC.

    Now reading that the watchdog might even not being able to attribute processing time to the individual addons ...

    So far the watchdog has done nothing but bash on the reputation of addon authors and confuse players.

    Cheers
    N.
    Last edited by Naifu; 07-01-2012 at 05:27 AM.

  6. #6
    Sword of Telara Semele's Avatar
    Join Date
    Mar 2011
    Posts
    872

    Default

    Oh dear, this is even worse than I thought then. There is absolutely no way to tell which addon in a chain really is the performance issue. It seems very odd that the Rift API events even bother to wait for an addon to return at all - especially System.Update.Begin. I'm sure in this day and age firing addons on their own thread would be more efficient. Isn't that the whole idea behind events anyhow?

    I can understand some events waiting, but I'm totally shocked they wait for a chain rather than each thread to finish with only the culprits of exceeding the watchdog being given errors.

    So, if I'm understanding this correctly. One addon which happens to stall out could cause the following addon in the chain to throw the error and look like it's responsible.

    AddonA Hiccups and takes 0.09s
    AddonB Runs as normal but thanks to the hiccup in A now only has 0.01s to fire it's share, it fails by 0.001s and the End user is now told AddonB is a pile of crap.
    AddonC doesn't get a look-in and breaks entirely due to now being completely inconsistent and also gets the author strange error reports which have nothing to do with the code itself.

    I'm not sure about anyone else, but this seems like a major design flaw in how the API works. I'm still hoping everyone here is wrong, and that addons are purely responsible for themselves. Otherwise this would practically make it impossible to debug certain performance warnings without authors turning on each other and saying - stop running every other addon or stop running mine... that is the solution.

    As it stands, I'm close to sidelining KBM entirely. It will now be placed on hold until I know whats what from Zorba.

    Even though I've never seen a performance warning from all 4 of my addons running (after some performance tweaks), and I've yet to hear anyone in my guild say it's happening -I'm still seeing some fire in the Daily Addon Error Reports. And from what has been discussed here, it could be a case of those reports are because an end user is running every addon known to man, and some of those could just be so far out of date and not using any of the new API.
    Rank 76 Guardian Mage

  7. #7
    Champion
    Join Date
    Jun 2011
    Posts
    561

    Default

    Quote Originally Posted by Semele View Post
    As it stands, I'm close to sidelining KBM entirely. It will now be placed on hold until I know whats what from Zorba.
    Two authors of the most popular addons out there which have spent an enormous amount of time working with the API and for the community are not happy with the current system. If nothing else, that would really make me think twice about the current situation.

    With the work due to the lastest and the upcoming patches I'm also very close to leave the whole addon stuff myself.

    Cheers
    N.
    Last edited by Naifu; 07-01-2012 at 12:00 PM.

  8.   Click here to go to the next Rift Team post in this thread.   #8
    Rift Team
    Join Date
    Oct 2010
    Posts
    927

    Default

    Quote Originally Posted by DoomSprout View Post
    I *think* event handlers work like this...

    1. RIFT fires an event (e.g. unit available).
    2. The watchdog starts it's stopwatch.
    3. The addon system looks in the event table {Event.Unit.Availability.Full}, and calls each registered event handler in turn, making a note of which addon owns the handler.
    4. The watchdog's stopwatch hits 0.1 seconds, it looks up the last event handler that was called, and fires up the 'Your Addon Sucks' dialog of doom.
    5. No other event handlers get called.
    This is close, but it isn't quite true. Keep in mind that the watchdog has two thresholds - a warning threshold, and a hard kill threshold. The warning threshold is 0.1 seconds and all it does is pop up a warning dialog, it doesn't interrupt execution in any way and any future event handlers will keep being called. The error threshold is 3.0 seconds. At that point, yeah, it'll stop further event handlers from being called, but that really shouldn't happen anyway.

    I'll probably be rigging it up soon so that it considers each event handler to have its own block of time, since it's clear that the one-block-of-time-per-event model isn't working.

  9. #9
    Sword of Telara Semele's Avatar
    Join Date
    Mar 2011
    Posts
    872

    Default

    Well, I'm hoping something happens soon. I'm now getting cheesed off royally with posts regarding "performance warnings" which I can't ever reproduce. And even if I did optimize it further, any other addon could cause KBM to still pop a warning message.

    I'm not walking away just yet, but I don't like my skill as a programmer coming in to question over something not entirely within my control. It's extremely frustrating.

    I'm all for a watchdog, but every addon should be accountable for itself, not that of others. Not only for the reputation of the author, but also to ensure authors know exactly if their code is to blame or not. Currently it's a lottery.

    And as a side note, I mean reputation talent wise, not people skills, those I don't have :P
    Rank 76 Guardian Mage

  10. #10
    Sword of Telara DoomSprout's Avatar
    Join Date
    Apr 2011
    Posts
    876

    Default

    Thanks for clarifying Zorba

    I think what the watchdog is trying to do is a really good idea. Addons can (and do) slow RIFT down, which in turn gives people the impression that RIFT itself runs slowly. This is bad, and it's totally understandable that Trion want to prevent this from happening, or at least ensure players know that it's a particular addon causing the slowdown.

    However, I think it's going about it in the wrong way. Using a 0.1s warning is catching spikes, but not a steady ongoing reduction of a player's FPS (which I believe is the only measure anyone actually cares about). Changing it to allow each individual addon it's own timeslice would help individual authors resolve performance warnings in their own addons, but I don't think it's addressing the real problem.

    As an example, take BananAH (an auction house addon). BananAH will never cause a general slowdown of RIFT. Everything it does is under the user's control, and during general gameplay (e.g. during a raid), it will do absolutely nothing. The watchdog should never trigger for BananAH, but it's probably the addon that needs the most work to get around the timer.

    The watchdog needs to look for on-going performance issues over a period of time. A single spike should never cause a warning, otherwise Trion's own Soul Tree window should be triggering warnings every time it's opened! I just think watching for large spikes in event times is fundamentally the wrong thing to do.

    While Trion do need to protect RIFT, I don't think upsetting the small but dedicated community of volunteer developers is the right thing to do either, and the performance warnings are doing that; it's a slap in the face to someone who has invested days of their own time to have the client telling players "this addon is slow".

    My personal belief is that the watchdog, while a good idea in theory, needs to be turned off until an alternative algorithm can be implemented. I don't think it's fair to make volunteer developers invest a vast amount of time and effort working around a system that is fundamentally flawed and (hopefully) likely to change in the future.

    BTW, I'm not saying this because I'm lazy and don't want to have to rewrite most of Gadgets. My addon is actually one that does need to be optimised for ongoing FPS, and should be subject to the watchdog's on-going monitoring. If the watchdog is working correctly, it is addons like Gadgets/KBM/nkThings/ClickBoxHealer/Mere's/etc/etc that it should be keeping an eye on, while having no interest at all in things like BananAH/Mail addons/RP addons etc.

    Just my 2c

    - Wildtide
    Last edited by DoomSprout; 07-02-2012 at 12:41 AM.

    Gadgets: Unit Frames and Other Stuff for RIFT

  11. #11
    Sword of Telara DoomSprout's Avatar
    Join Date
    Apr 2011
    Posts
    876

    Default

    Another thought...

    As a quick fix, why not put Pooky the Watchdog to sleep while you're not in combat? That would keep him away from addons that shouldn't need to worry about performance.







    Edit: Yes, I call him Pooky. Anthropomorphizing him makes him less scary somehow ;)
    Last edited by DoomSprout; 07-02-2012 at 01:03 AM.

    Gadgets: Unit Frames and Other Stuff for RIFT

  12. #12
    Plane Walker Imhothar's Avatar
    Join Date
    Feb 2012
    Posts
    439

    Default

    I'd be happy with the performance warning not raise a visible error message and just be included in the error report mail until the WATCHDOG tracks each addon individually.

  13. #13
    Champion
    Join Date
    Jun 2011
    Posts
    561

    Default

    Quote Originally Posted by DoomSprout View Post
    While Trion do need to protect RIFT, I don't think upsetting the small but dedicated community of volunteer developers is the right thing to do either, and the performance warnings are doing that; it's a slap in the face to someone who has invested days of their own time to have the client telling players "this addon is slow".

    My personal belief is that the watchdog, while a good idea in theory, needs to be turned off until an alternative algorithm can be implemented. I don't think it's fair to make volunteer developers invest a vast amount of time and effort working around a system that is fundamentally flawed and (hopefully) likely to change in the future.
    This! I am very happy to learn that I'm not alone with my negative opinion. Of course I'm fully aware that I am a very critic person and sometimes I feel that I'm just complaining. While my overall complaining about the API business as whole might sometimes be too much, I really feel this watchdog deseaster should really show Trion to sit down and rethink their handling of the addon API.

    We addon authors do not get paid, we invest tons of our free time to make a good product somewhat better. There are a lot of people who couldn't and wouldn't live anymore without addons like Gadgets, BananaAH, KBM, my range of addons and many more of the great addons out there. We do not get anything out of this addon stuff but the feeling of having contributed to the whole Rift community. We handle a lot of feedback, track down bugs and talk to the community.

    Let's face it: Without us addon authors Rift would be a less good game.

    If it wasn't so Trion would have shut down the API development long ago.

    In the end we don't get anything for this effort but a good feeling and the pride in our work. Trion doesn't give us any free gaming time, no ingame title, no fluffy shiny ingame stuff they give us nothing but new work with every API patch.

    They don't give us any tools so that it would be easier to export the ingame item database, to test group / raid frames, to test boss mechanics so that we would have the right tools to develop the addons easier and better. However tried to improve an addon while their guild is raiding knows what I mean.

    And now here comes the watchdog which kicks my butt as an addon author, which tramples on my reputation, which slaps me in the face. People are sending me reports saying "your addon is **** cause it's slow ... you know the game tells me that your addon is ****ty slow".

    This whole watchdog thing really pisses me off majorly.

    For all the free work me and my fellow addon authors have invested in your product of which you earn money, we all deserve some better treatment.

    Cheers
    N.
    Last edited by Naifu; 07-02-2012 at 01:51 AM.

  14.   Click here to go to the next Rift Team post in this thread.   #14
    Rift Team
    Join Date
    Oct 2010
    Posts
    927

    Default

    Quote Originally Posted by DoomSprout View Post
    However, I think it's going about it in the wrong way. Using a 0.1s warning is catching spikes, but not a steady ongoing reduction of a player's FPS (which I believe is the only measure anyone actually cares about). Changing it to allow each individual addon it's own timeslice would help individual authors resolve performance warnings in their own addons, but I don't think it's addressing the real problem.

    The watchdog needs to look for on-going performance issues over a period of time. A single spike should never cause a warning, otherwise Trion's own Soul Tree window should be triggering warnings every time it's opened!
    I agree! . . . partially.

    First, as tricky as it is to get spike detection working reliably without false positives, it's *far* harder to get on-going performance issue detection working that well. In many ways this is Version 1, where Version 2 will start paying attention to CPU usage over long periods of time.

    (That definitely won't happen until after Storm Legion so don't panic)

    But spikes are important also - the entire client locking up for a visible period of time, which can be as little as a tenth of a second, contributes rather heavily to feeling like there are performance problems. We've spent a bunch of effort to squash those - the Soul Tree is probably one of the biggest remaining issues, trust me, we're aware of it - and those turn out to be some of the more subtle and hard to find errors.

    Example time! Like I imagine many addon authors do, I've got a few personalized single-purpose addons I put together for my own purposes. One of them started triggering the watchdog, so I spent a little time to fix it. It turns out that a framerate glitch that I'd thought was in the Rift UI was actually in my addon. If I can make that mistake - an addon I wrote, running on an addon framework I wrote, compared to a core Rift UI element that I've probably spent a week or two working on - Rift players can certainly make the exact same mistake.

    And while I agree that addons like BananAH should get more leeway with performance, they actually have it already - they can use Command.System.Watchdog.Quiet() to get up to a full second of CPU at a time.

    I'll be working on changing the watchdog performance monitoring into a per-event-handler thing instead of a per-event thing - I agree, that needs to be done - and I've been considering changing the current "soft" Watchdog cutoff to 0.2 seconds, and introducing a "super-soft" cutoff at 0.1 seconds that doesn't show up for the user but does show up in the error reports. That'd be a lot of work and may not actually occur, however, we'll see how things go after the per-event-handler change goes in. Also, while texture loading is proving a major bottleneck and real asynchronous support needs to be added, Imhothar's LibAsyncTexture library is an excellent interim solution until I can provide something a bit more built-in.

    All that said, performance really is kind of a tough thing to deal with, and while it's quite clear some tweaks and added tools are needed, the general concept of the current behavior of the watchdog is probably going to stick around. I totally agree that it needs to deal with long-term CPU usage, but that's going to be in addition to, not in place of, the current behavior.

    (Eventually.)

  15. #15
    Sword of Telara DoomSprout's Avatar
    Join Date
    Apr 2011
    Posts
    876

    Default

    OK, having very briefly chatted with Zorba, I can now see the need for spike avoidance as well. Trion are working hard to remove instances of the client locking up (e.g. the Soul Tree), so it would be silly to have this effort wasted by addons doing it instead.

    While I still disagree with the severity of the warning dialog and think it should be hidden from end users, at least for a while so that we can get our shops in order, I withdraw my objection to addons not being able to spike-freeze the client

    We do still need better tools for managing performance, and the ability to hand off expensive processes like texture loading to a background thread, but hopefully those things will follow.

    Gadgets: Unit Frames and Other Stuff for RIFT

+ Reply to Thread
Page 1 of 5 1 2 3 4 5 LastLast

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts