43 replies to this topic

[ComradeShepard7]

TL/DR: Check the conclusions section at the bottom.

A few days ago, thewalrusx posted a thread asking people to collect data on how much reload canceling increases your dps. This led to a rather heated discussion about experimental methods and a bit of a disagreement about the best way of going about doing this.

Here are my methods and results for testing reload canceling's impact on fire rate in game as well as seeing if the fire rate increase evolution of Hunter Mode has any impact on fire rate with single shot weapons. The variables being tested were the only ones that were purposefully changed during testing with other variations coming from human error. The testing was not done in combat as this would lead to the presence of other variations.

Setup
-Difficulty: Gold
-Enemy: Cerberus
-Map: Giant
-Character: Geth Infiltrator with linked build

Data Collection
-Baldere and I killed all wave one enemies except one nemisis which he kept occupied while I recorded.
-Video was recorded of firing without reload canceling with Hunter Mode active, firing without reload canceling without Hunter Mode active, and firing with reload canceling without Hunter Mode active.
-my computer was acting up earlier today when I did this which led to my frame rate averaging 35 while I did this. I may take data at 60 fps another time to improve the resolution.
-Video (a really boring 6 minutes if you really want to watch it)

Data Analysis
-Video was advanced frame by frame and the times when shots were fired were recorded which could then be used to determine the time between shots.
-The time of firing refrence was the ammo indicator ticking from full to empty. For two or three of the shots when reload canceling the ammo indicator did not even show as full in the video so I used the muzzle flash as an indicator in those cases.
-A digital signal analogy for better understanding.
--Consider the ammo indicator state to be stored in a rectangular wave signal with HIGH being full and LOW being empty.
--Times are recorded at the falling edges of the signal

Results
The Following Data was produced from these tests:

-no hunter Mode, no reload canceling
--mean time between shots: 3.33 seconds
--standard deviation: 0.04 seconds
--fire rate: 18.03 shots per minute



-no hunter mode, reload canceling
--mean time between shots: 2.03 seconds
--standard deviation: 0.15 seconds
--fire rate: 29.57 shots per minute



-no hunter mode, reload canceling, mess up removed
--mean time between shots: 2.01 seconds
--standard deviation: 0.05 seconds
--fire rate: 29.92 shots per minute



-hunter mode, no reload canceling
--mean time between shots: 3.24 seconds
--standard deviation: 0.07 seconds
--fire rate: 18.52 shots per minute






Difference between HM and no HM without reload canceling:
-95% difference of means confidence interval: [0.0622,0.1160] (seconds)
-converted to rate of fire increase: [1.90,3.61] (% increase)

Conclusions
-Reload canceling improves fire rate by 64% on average including messing up, or 66% if you don't mess up. However, dps will only increase by this much if you fire imediately upon reload canceling and you must still hit your target.
-The fire rate evolution of HM seems to give a small increase in effective fire rate but the amount of variation in the data makes it difficult to say just how much.
-Even if you screw up a reload cancel, it is still faster than the normal reload time so long as you don't mess up the second reload cancel (based on single data point).

Edited by ComradeShepard7, 29 June 2013 - 10:28 PM.

[Shampoohorn]

Nice. 64% is close to the 77% that Cyonan pointed out.

There does seem to be a small difference in the distributions -/+ Hunter Mode. It's my understanding that it would affect the refire delay even on single fire weapons. There could be some inherent bias in your finger, heh. The fourth condition -- HM and RC -- might shed some light on that.

Not may replies yet. Math posts tend to get the best response on Mondays, when most of the engineers have had their coffee and become bored with actual work. Saturdays, not so much.

Edited by Shampoohorn, 29 June 2013 - 11:11 AM.

[Dunvi]

correct.

even single shot weapons have a rate of fire

this duration (1/RoF seconds) must pass before you can do anything with the gun again. this includes reloading. the reload will not begin until this time passes.

this is why weapons like the wraith with long interruptions between shots drive me crazy. this is also why some weapons's reloads can be hidden inside powers or animations much more easily than others.

i'm doing a 24hr hackathon

Edited by Dunvi, 29 June 2013 - 11:04 AM.

[Ziegrif]

Dunvi wrote...

correct.

even single shot weapons have a rate of fire

this duration (1/RoF seconds) must pass before you can do anything with the gun again. this includes reloading. the reload will not begin until this time passes.

this is why weapons like the wraith with long interruptions between shots drive me crazy. this is also why some weapons's reloads can be hidden inside powers or animations much more easily than others.

i'm doing a 24hr hackathon

That's something I want removed in ME4. Annoying.

Edited by Ziegrif, 29 June 2013 - 12:29 PM.

[OniGanon]

So to clarify: the goal of this test was to see how much HM ROF bonus affects Claymore DPS?

According to Cyonan's sheet, the theoretical perfect RC ROF is 30.34RPM, which you got pretty close to so nice work.

No test for HM + RC?

You conclude that HM ROF increase doesn't affect single shot weapons. But your test data showed a small difference between HM and no HM. Not a huge difference, but then it's not a huge ROF increase so that's to be expected.

If you meant to say that the increase is not terribly significant, that's a judgement call and should be made clear as such...

Perhaps a fully ROF evolved Marksman would be a better test to show, conclusively, whether ROF affects single shot weapons.

Edited by OniGanon, 29 June 2013 - 12:41 PM.

[Credit2team]

awesome!
thank you for doing this even though its not exactly what I asked for, still gives some useful information.

Cheers!

[someN7orother]

Shampoohorn wrote...


There does seem to be a small difference in the distributions -/+ Hunter Mode. It's my understanding that it would affect the refire delay even on single fire weapons. There could be some inherent bias in your finger, heh. The fourth condition -- HM and RC -- might shed some light on that.

Yeah, it was actually tested frame-by-frame back then when the GI came out that RoF HM evo actually speeds up reload cancelling. Possibly because the 1/RoF timespan mentioned by Dunvi decreases, but who knows. It's likely also the reason why HM+RC wasn't included in the tests?

Edited by someN7orother, 29 June 2013 - 12:51 PM.

[Jeremiah12LGeek]

Good on you for taking the time to do this.

[ComradeShepard7]

Shampoohorn wrote...

Nice. 64% is close to the 77% that Cyonan pointed out.

There does seem to be a small difference in the distributions -/+ Hunter Mode. It's my understanding that it would affect the refire delay even on single fire weapons. There could be some inherent bias in your finger, heh. The fourth condition -- HM and RC -- might shed some light on that.

Not may replies yet. Math posts tend to get the best response on Mondays, when most of the engineers have had their coffee and become bored with actual work. Saturdays, not so much.

It certainly looks like there is a difference with hunter mode but based on the means it is only a 2.5% improvement in fire rate which is no where near the 15% listed in the evolution. Maybe that is in fact from hunter mode but with the amount of overlap in the distributions (see the boxplots) there isn't a lot of confidence in saying that based on this data (at least from what I remember from my statistics course).

I am planning on redoing this at 60 fps at some point which will give better a better resolution for the data and I intend to take larger samples at that point which may reduce the overlap. I will also take data with Hunter Mode active while reload canceling to see what that yields.

[RoZh2400]

ComradeShepard7 wrote...

Shampoohorn wrote...

Nice. 64% is close to the 77% that Cyonan pointed out.

There does seem to be a small difference in the distributions -/+ Hunter Mode. It's my understanding that it would affect the refire delay even on single fire weapons. There could be some inherent bias in your finger, heh. The fourth condition -- HM and RC -- might shed some light on that.

Not may replies yet. Math posts tend to get the best response on Mondays, when most of the engineers have had their coffee and become bored with actual work. Saturdays, not so much.

It certainly looks like there is a difference with hunter mode but based on the means it is only a 2.5% improvement in fire rate which is no where near the 15% listed in the evolution. Maybe that is in fact from hunter mode but with the amount of overlap in the distributions (see the boxplots) there isn't a lot of confidence in saying that based on this data (at least from what I remember from my statistics course).

I am planning on redoing this at 60 fps at some point which will give better a better resolution for the data and I intend to take larger samples at that point which may reduce the overlap. I will also take data with Hunter Mode active while reload canceling to see what that yields.

Agreed. The 2.5% improvement doesn't really mean anything, especially considering the boxplots you created. that small "difference" is likely variation due to testing error, more than anything (error,\\ which was small, so nice job!). Maybe if you made a confidence interval of difference of means between HM/no HM, we could get a better picture. However, it seems pretty clear from a quick scan of your results that the CI would allow us to conclude that it is extremely unlikely that HM actually increases your RoF by 15%.

[jakenou]

There's another reload factor I think might be interesting to add to this. With GI, when you cloak> fire shot> deploy proxy in quick succession, you get a free reload with zero animation (for single shot weapons or if you have an empty clip).

This is common for all kits I think - fire your last shot and quickly deploy power = free reload with zero animation. It seems to me this method is a little slower perhaps with single-shotters like Javelin and Claymore, and there also seems to be a delay in the time when you can actually fire the shot afterward - i.e. once the clip appears, I find myself still hitting my gel button or cloaking to make it available right away. I'd like to see this with and without Hunter Mode Active and compared against the other reload cancel/ no cancel numbers, since it is in essence a variation of reload canceling. What do ya think ComradeShepard??

[RedJohn]

But we knew that 1 year ago.

[jakenou]

RedJohn wrote...

But we knew that 1 year ago.

Knew what?

[ComradeShepard7]

RoZh2400 wrote...

ComradeShepard7 wrote...

Shampoohorn wrote...

Nice. 64% is close to the 77% that Cyonan pointed out.

There does seem to be a small difference in the distributions -/+ Hunter Mode. It's my understanding that it would affect the refire delay even on single fire weapons. There could be some inherent bias in your finger, heh. The fourth condition -- HM and RC -- might shed some light on that.

Not may replies yet. Math posts tend to get the best response on Mondays, when most of the engineers have had their coffee and become bored with actual work. Saturdays, not so much.

It certainly looks like there is a difference with hunter mode but based on the means it is only a 2.5% improvement in fire rate which is no where near the 15% listed in the evolution. Maybe that is in fact from hunter mode but with the amount of overlap in the distributions (see the boxplots) there isn't a lot of confidence in saying that based on this data (at least from what I remember from my statistics course).

I am planning on redoing this at 60 fps at some point which will give better a better resolution for the data and I intend to take larger samples at that point which may reduce the overlap. I will also take data with Hunter Mode active while reload canceling to see what that yields.

Agreed. The 2.5% improvement doesn't really mean anything, especially considering the boxplots you created. that small "difference" is likely variation due to testing error, more than anything (error, which was small, so nice job!). Maybe if you made a confidence interval of difference of means between HM/no HM, we could get a better picture. However, it seems pretty clear from a quick scan of your results that the CI would allow us to conclude that it is extremely unlikely that HM actually increases your RoF by 15%.

The 95% confidence interval I got for the difference of the means of time between shots is [0.0622,0.1160] so we can say with a high degree of confidence that hunter mode improves fire rate with the claymore by between 1.90% and 3.61%. Certainly no where near a 15% increase.

[ComradeShepard7]

someN7orother wrote...

Shampoohorn wrote...


There does seem to be a small difference in the distributions -/+ Hunter Mode. It's my understanding that it would affect the refire delay even on single fire weapons. There could be some inherent bias in your finger, heh. The fourth condition -- HM and RC -- might shed some light on that.

Yeah, it was actually tested frame-by-frame back then when the GI came out that RoF HM evo actually speeds up reload cancelling. Possibly because the 1/RoF timespan mentioned by Dunvi decreases, but who knows. It's likely also the reason why HM+RC wasn't included in the tests?

Actually I didn't do it because I was initially trying to get a general figure for all characters. I am going to include it when I retake data at 60 fps with larger sample sizes and I think I'll test other fire rate increasing powers as well.

[OniGanon]

Why on earth would you expect the 15% ROF increase to result in 15% faster reload? ROF boost doesn't boost reload time, it only reduces the delay BEFORE the reload.

Claymore has a ROF of 64RPM, adding 0.9375s delay before reload.

Boosting that by 15% means a ROF of 73.6, which lowers the delay to 0.8152 (rounded to 4dp)

Difference between the two is roughly 0.1223s

Also... Something strange is going on here... Your RC numbers look fine, but your non RC numbers are faster than expected.

Theoretically, with a listed ROF of 64RPM and reload time of 2.57s, without RC the Claymore should fire once every 3.5075s (vs your recorded 3.33s)

[ComradeShepard7]

jkthunder wrote...

There's another reload factor I think might be interesting to add to this. With GI, when you cloak> fire shot> deploy proxy in quick succession, you get a free reload with zero animation (for single shot weapons or if you have an empty clip).

This is common for all kits I think - fire your last shot and quickly deploy power = free reload with zero animation. It seems to me this method is a little slower perhaps with single-shotters like Javelin and Claymore, and there also seems to be a delay in the time when you can actually fire the shot afterward - i.e. once the clip appears, I find myself still hitting my gel button or cloaking to make it available right away. I'd like to see this with and without Hunter Mode Active and compared against the other reload cancel/ no cancel numbers, since it is in essence a variation of reload canceling. What do ya think ComradeShepard??

This sounds like you are talking about reload animation masking with power use. I have never actually used it but from what I understand it doesn't actually change reload times, it just allows you to do other things while reloading. If I can get it too work I'll probably take data for that as well.

[ComradeShepard7]

OniGanon wrote...

Why on earth would you expect the 15% ROF increase to result in 15% faster reload? ROF boost doesn't boost reload time, it only reduces the delay BEFORE the reload.

Claymore has a ROF of 64RPM, adding 0.9375s delay before reload.

Boosting that by 15% means a ROF of 73.6, which lowers the delay to 0.8152 (rounded to 4dp)

Difference between the two is roughly 0.1223s

Also... Something strange is going on here... Your RC numbers look fine, but your non RC numbers are faster than expected.

Theoretically, with a listed ROF of 64RPM and reload time of 2.57s, without RC the Claymore should fire once every 3.5075s (vs your recorded 3.33s)

The HM evolution affecting reload time with single shot weapons was something I had heard along time ago so I decided to give it a test.

As far as my numbers, all that I did was click super fast the entire time. I don't know what would have caused that discrepency.

[ComradeShepard7]

OniGanon wrote...

So to clarify: the goal of this test was to see how much HM ROF bonus affects Claymore DPS?

According to Cyonan's sheet, the theoretical perfect RC ROF is 30.34RPM, which you got pretty close to so nice work.

No test for HM + RC?

You conclude that HM ROF increase doesn't affect single shot weapons. But your test data showed a small difference between HM and no HM. Not a huge difference, but then it's not a huge ROF increase so that's to be expected.

If you meant to say that the increase is not terribly significant, that's a judgement call and should be made clear as such...

Perhaps a fully ROF evolved Marksman would be a better test to show, conclusively, whether ROF affects single shot weapons.

The goal of this was to test how much reload canceling improved DPS but I threw in the hunter mode test as well.

There does seem to be a small difference in fire rate with and without HM but with the amount of overlap between the distributions it is problematic to really say that there is an increase. Hopefully when I take more data with better resolution there won't be so much overlap and then I will be comfortable with really reaching a conclusion.

[RoZh2400]

ComradeShepard7 wrote...

OniGanon wrote...

So to clarify: the goal of this test was to see how much HM ROF bonus affects Claymore DPS?

According to Cyonan's sheet, the theoretical perfect RC ROF is 30.34RPM, which you got pretty close to so nice work.

No test for HM + RC?

You conclude that HM ROF increase doesn't affect single shot weapons. But your test data showed a small difference between HM and no HM. Not a huge difference, but then it's not a huge ROF increase so that's to be expected.

If you meant to say that the increase is not terribly significant, that's a judgement call and should be made clear as such...

Perhaps a fully ROF evolved Marksman would be a better test to show, conclusively, whether ROF affects single shot weapons.

The goal of this was to test how much reload canceling improved DPS but I threw in the hunter mode test as well.

There does seem to be a small difference in fire rate with and without HM but with the amount of overlap between the distributions it is problematic to really say that there is an increase. Hopefully when I take more data with better resolution there won't be so much overlap and then I will be comfortable with really reaching a conclusion.

Experiment with Marksman. The results will be much more clear where the discrepancies appear, which should just be REFIRE time, not RATE OF FIRE.

[Kirchroa Alaaf]

64-66% is quite some advantage. But then again, you have to have multiple enemies and hit the targets. So it means you get 8-12 relevant shots more in one round. That means an significant increase in dps.

Cool tests.

Edited by Kirchroa Alaaf, 29 June 2013 - 05:13 PM.

[ComradeShepard7]

RoZh2400 wrote...

ComradeShepard7 wrote...

OniGanon wrote...

So to clarify: the goal of this test was to see how much HM ROF bonus affects Claymore DPS?

According to Cyonan's sheet, the theoretical perfect RC ROF is 30.34RPM, which you got pretty close to so nice work.

No test for HM + RC?

You conclude that HM ROF increase doesn't affect single shot weapons. But your test data showed a small difference between HM and no HM. Not a huge difference, but then it's not a huge ROF increase so that's to be expected.

If you meant to say that the increase is not terribly significant, that's a judgement call and should be made clear as such...

Perhaps a fully ROF evolved Marksman would be a better test to show, conclusively, whether ROF affects single shot weapons.

The goal of this was to test how much reload canceling improved DPS but I threw in the hunter mode test as well.

There does seem to be a small difference in fire rate with and without HM but with the amount of overlap between the distributions it is problematic to really say that there is an increase. Hopefully when I take more data with better resolution there won't be so much overlap and then I will be comfortable with really reaching a conclusion.

Experiment with Marksman. The results will be much more clear where the discrepancies appear, which should just be REFIRE time, not RATE OF FIRE.

I get the destinction that BSN uses, however I have been around firearms all my life so I use terms in that context. What you call rate of fire is analagous to cyclic rate of fire with actual firearms while what I am talking about is effective rate of fire. The context in which all of this is being talked about makes which I am talking about quite clear.

[k1ngl1ves]

Wait...

So shooting faster = shooting faster?

*mind blown*

[ComradeShepard7]

k1ngl1ves wrote...

Wait...

So shooting faster = shooting faster?

*mind blown*

Everyone knew that going in. The point of this was to QUANTIFY how much faster it actually is in practice.

[k1ngl1ves]

ComradeShepard7 wrote...

k1ngl1ves wrote...

Wait...

So shooting faster = shooting faster?

*mind blown*

Everyone knew that going in. The point of this was to QUANTIFY how much faster it actually is in practice.

It's faster.  A lot faster.

You're welcome.