Watchdog puzzles

[Semele]

Ok, so after much searching around my logs it is registering and isn't broken. The issue for me was all my handlers were falling under the watchdogs radar 99.9% of the time. So was very hard to see the remaining time move ever, even with a real-time record.

Moving along.

Nice chart btw Doom

[DoomSprout]

The issue for me was all my handlers were falling under the watchdogs radar 99.9% of the time

Show off ;)

Nice chart btw Doom

Google Docs FTW!

In all seriousness, the benchmarks I've been running, and a bit of reading up on the UI technology RIFT uses, is a little depressing (for me at least). Trying to build complex graphical elements (such as Unit Frames) out of a great big pile of frames is apparently always going to hit frame rates.

I'll do what I can to simplify things, but I'm quietly cursing Trion for not using a closer-to-the-wire UI system, where 1000 coloured squares (surely fewer triangles than a single NPC mesh) wouldn't bring frame rates crashing down.

- Wildtide

[Semele]

I agree on the UI thing. Because, I'm getting random reports of my pre-fetch system for Add-Watch triggering the Watchdog.

The odd thing is, all it does is build a single Add-Watch Unit frame then cache it. And, the reports actually range from slow to fast machines, but.. not every slow machine is getting the warning, and obviously not every fast machine is.

But, this is ok. As Add-Watch is my test case for moving stuff around and ensuring warnings stay consistent and move with each version.

Next step, it to pre-fetch during Event.Addon.Load.End if this is indeed where the Watchdog is still sleeping.

This will, however seem to have the adverse effect of increasing load times for Rift slightly.

[Semele]

OK, so I feel the current Watchdog is doing about as much as it can to avoid blaming the wrong addon. But, I'm not sure we'll ever be in a place where the Watchdog will stop unfairly blaming Addons for Client performance.

With this is mind, and after thinking a little more about a players option to disable the Watchdog entirely, I believe this would be the logical next step.

I don't think the current watchdog timer is working for anyone.

The add-on writers are frustrated, which is not good.

End-users like me are frustrated because Rift keeps giving add-on error message even when the add-on is working perfectly fine and not causing any lag of any kind.

At the end of the day, as the owner of the client system I should get to decide if I'm experiencing lag, not some arbitrary watchdog timer. I've never had any lag problems with any add-on I run (including wonderful ones like KBM). Instead I have a watchdog timer that pops up useless error messages in the middle of a bossfight. Why can't I disable the watchdog timer?

There has to be a better solution.

In the Addon options page, you have the user be able to have 3 options.
1: Allow the Watchdog to throw Warnings (Dev Setting)
2: Allow the Watchdog, but hide Warnings (Default - Devs still get data)
3: Disable the Watchdog entirely.

I know you've put a lot of work in to this Zorba, and in theory it's a good idea. I do however feel it shouldn't be forced on the end-user if they personally feel their addon choices are having little to no impact to warrant even a single warning pop-up. And please, remove the "Error" part of the warning Pop-up if at all possible, this is misleading.

I'm having reports from a small section of Guild members running KBM and saying everything is running smooth, but now I can't see anything because of a stupid window (their words). I explain you can ignore all warnings from this version but, to be perfectly honest, I WANT real errors to still pop that window, just not Watchdog ones - my guild run Alpha versions often, so real errors are a must.

My main beef with this at the moment, is that poor performance spikes in the Client can cause false positives for addons. Where an addon isn't even given enough time to perform the most simple of handlers.

[Semele]

I got my first 3's hard crash report today. On an encounter initialization sequence. lol

I'm now officially ignoring all Watchdog Performance errors, whether they're warnings or hard crashes. Whole system is a joke and I really don't know whats legitimate anymore.

I'm done with anything Watchdog related. As far as I'm concerned it no longer exists.

[Semele]

Decided to revisit optimization the best I could.

function PI.CycleLoad()
	if PI.Queue.Count > 0 then
		--print("Items in Queue: loading")
		local TimeStart = Inspect.Time.Real()
		local Count = 0
		repeat
			PI.Queue.First.Texture:SetTexture(PI.Queue.First.Location, PI.Queue.First.File)
			PI.Queue:Pop(PI.Queue.First)
			Count = Count + 1
			local Duration = Inspect.Time.Real() - TimeStart
			local Average = Duration / Count
			if (Average * (Count + 1)) > 0.05 then
				break
			end
		until Inspect.System.Watchdog() < 0.05 or PI.Queue.Count == 0
	end
end

At first the above was a simple "until" but, I was still getting errors, so I added extra checking with a possible break condition as I was still getting Performance Warnings from people.

Anyhow, after I implemented the above, the result was as follows. Which, may I point out adds extra CPU cycles to ensure the handler conforms to performance standards is more than a little ironic.

KBM Texture Handler 0.0.3
    This addon has caused an internal error. It may not work properly in the future and may cause performance or stability issues.
    We recommend disabling this addon, finding an updated version, or reporting the error to the addon's developer.
 
    Error: performance error, execution halted
        In KBMTextureHandler / Stage 2 Load Cycle
    stack traceback:
         KBMTextureHandler/TextureHandler.lua:74: in function <KBMTextureHandler/TextureHandler.lua:67>

The Watchdog killed the Texture Handler directly after the Texture had loaded. So, basically, the Performance Warning is pointing at the Addon API itself being horribly optimized not my code.

You really need to dump this Watchdog now. All it's doing is highlighting how bad the API itself is, and not actually helping Devs at all. The fact the Watchdog can blame the simplest of Handlers for Performance means that no matter what us Addon devs do, we'll get blamed regardless.

I stand by my original thoughts that the Watchdog is not practical, poorly implemented and not thought through, you need another solution. Because all you're doing at the moment is providing evidence to your employers that the API is horribly optimized itself. The fact we can use an API command and throw an error blaming us is laughable.

The more I play with this, the more I get the feeling the Trion Addon API Devs are pushing the blame on the real Addon Devs to hide their own performance issues, which is totally unacceptable.

[Semele]

Ahh, my wife just reminded me of something.

If a player Alt+Tabs out of Rift... guess what the Watchdog blames the performance hit on. That's right... Addons.

Jesus Zorba, just admit this was a failed idea already and nuke the Watchdog. IT DOES NOT WORK. And to be perfectly honest, never will. If this wasn't your idea, then at least have the person who thought this would be a good idea explain how it could possibly work for programmers writing under an API.

[Imhothar]

So, I'm away for two weeks and what do I see when I come back?

Performance kills (not warnings!) in the next line after calling Command.Item.Standard.Right().

What the hell?

This is a handler of Event.RightUp and all it does is read the itemid from the button, call a public event handler returned from Utility.Event.Create() (maybe that's the real problem?) followed by Command.Item.Standard.Right().

[Semele]

So, I'm away for two weeks and what do I see when I come back?

Performance kills (not warnings!) in the next line after calling Command.Item.Standard.Right().

What the hell?

This is a handler of Event.RightUp and all it does is read the itemid from the button, call a public event handler returned from Utility.Event.Create() (maybe that's the real problem?) followed by Command.Item.Standard.Right().

I feel your pain. The only Performance kills I get are from a MouseIn event, which does nothing more than change the alpha value of a frame than return. Oh well.

[DoomSprout]

Are these kills from before the latest change to the watchdog? (I really hope so)

I think the error mails are built from all errors from the last 7 days, and I've really noticed the number of performance warnings/kills dropping off dramatically in my mails.

I was hoping that they'd all have gone by now, but I'm still seeing a few from older versions of the addon. If they're still there tomorrow, I'll have to do some investigating as it'll be well over a week since the change went in.

[Semele]

Are these kills from before the latest change to the watchdog? (I really hope so)

I think the error mails are built from all errors from the last 7 days, and I've really noticed the number of performance warnings/kills dropping off dramatically in my mails.

I was hoping that they'd all have gone by now, but I'm still seeing a few from older versions of the addon. If they're still there tomorrow, I'll have to do some investigating as it'll be well over a week since the change went in.

Fairly sure mine are from before the changes. I'm reserving judgment until everything has fully filtered through.

Although, at first glance, I have zero errors from current versions.

[ZorbaTHut]

Are these kills from before the latest change to the watchdog? (I really hope so)

I went and checked in the raw database, and sadly, it's still happening recently, within the last few days.

I'm trying to track down what's going on, without, currently, much luck. If anyone can reproduce it I'd be quite grateful, even if it only triggers with a 1% reliability or whatever. Attach your test to Event.System.Update.Begin() so it runs every frame :V

I have found a glitch that can result in Inspect.System.Watchdog() over-reporting the amount of time remaining by up to 0.015s. That'll be fixed. But that wouldn't explain kills, since a kill requires 3 seconds of usage.

[Turrant]

I got my first 3's hard crash report today. On an encounter initialization sequence. lol

Off topic but, crashing when I begin an encounter has been happening to me a lot lately even on my computer that does not have any add-ons. I use an ability like Riftwalk, it puts me in combat and I crash almost imediatly or it will sometimes lock up instead for 15 seconds or so before unfreezeing.

I never even thought they could be related until I read your post. It does not happen every encounter but enough it has been happening at least once a day since the patch two wensdays ago.

[ZorbaTHut]

Off topic but, crashing when I begin an encounter has been happening to me a lot lately even on my computer that does not have any add-ons. I use an ability like Riftwalk, it puts me in combat and I crash almost imediatly or it will sometimes lock up instead for 15 seconds or so before unfreezeing.

That doesn't sound off-topic at all! In fact, that's exactly what I'd expect this bug to look like from the enduser's perspective.

Is that activating abilities through the default UI, or are you using an addon to cast that ability?

[Semele]

Off topic but, crashing when I begin an encounter has been happening to me a lot lately even on my computer that does not have any add-ons. I use an ability like Riftwalk, it puts me in combat and I crash almost imediatly or it will sometimes lock up instead for 15 seconds or so before unfreezeing.

I never even thought they could be related until I read your post. It does not happen every encounter but enough it has been happening at least once a day since the patch two wensdays ago.

This would explain a lot with regards to me getting error reports based on my encounter start code. The code does little to nothing in comparison with more complex and strenuous code.

That doesn't sound off-topic at all! In fact, that's exactly what I'd expect this bug to look like from the enduser's perspective.

Is that activating abilities through the default UI, or are you using an addon to cast that ability?

Since he stated it also happens on a machine with no addons installed I would imagine what is happening as far as the Watchdog is concerned is that when the freeze happens and an addon is installed the addon would automatically get the blame. But, when no addons are installed, it would obviously be a flat out crash/freeze with nothing to blame other than the Rift client itself.