NotoriousREV wrote: Sat Mar 23, 2019 10:48 am
I’ve been in this game for a long time (20 years) and I’ve never seen a site fail where load testing says it will.
Anyway, it's not about what is possible with the bleeding edge of load balancing and CDN (which I'll hold my hands up and say I don't tend to work with much - i manage my domains through cloudflare and have my static sites CDN'd up but that's about it because that's what's appropriate for expected load and if I tried to spend tens of thousands a month on global scale caching I'd get sacked... ), it's about how easy it is to game the system.
The protections in place to prevent mass gaming of the petitions site appear 'good enough' given that nothing on it is legally binding in any sense, and the protections have been known about publicly (to a greater or lesser extent - certainly a useful extent) for a few years now - so for people to smugly and authoritatively claim that you can just register on the same email address because some talking head said so rather annoys me when it's so very, very easily disprovable.
As for Clarkson, I can't imagine why a program with a truly global audience might gain a million signatures in a short time, compared to what is effectlvely a political spat focussed entirely on one country and why the two might be slightly different in context.
My more serious point about load testing is this: you build your infrastructure based on your understood load. If you’re doing TV advertising, for example, you estimate the response rate and scale for that plus a contingency, because you have a budget limitation somewhere along the line. Commercial reality is that I don’t really want to have a bunch of compute/memory/storage sitting around doing nothing, hence the growth of PAYG cloud services.
For example, my current site will quadruple its peak load between Christmas Eve and the 1st Monday in January. Generally Monday nights are 30-40% busier than Thursday nights, except for pay day. I’ll pre-provision for those loads because I know scaling isn’t instant.
If I suddenly get a workload that hasn’t been considered before for whatever reason, chances are I haven’t tested that high anyway. I might know where my stack breaks, but that limit may well have been considered high enough to cope with expected peak load comfortably. This is why I work with Marketing to understand their plans so I hopefully won’t get an unexpected load (giggity) but even so, if they send a push notification to every iOS app user at the same time, instead of in batches as planned, during peak traffic and traffic to our API increases by 500% above the previous record in 2 minutes, I rely on the app being well written and being tolerant of slow downs and request queuing in a graceful way. Also I shout at them and make sure they don’t DDoS us again, or if they do want to be able to that, we have a plan and budget to cope with it.
And load testing isn’t as easy as just replaying customer data. To be accurate, I have to point it at my production environment. But if I do that, I can’t place real orders because those are tied to 3rd party systems. So either I decouple those and accept a flawed test, or I place real orders and take the cost hit (not going to happen). I also need to pick a time to do it, but I’m doing other stuff overnight (eg batch jobs) and I also have real traffic even in the dead of night which I’ll affect. The load test needs to last long enough to consider degradations that may appear over time. Can I simulate my 3rd party payment gateway slowing down? 3rd party JavaScript slowing down? A DB suddenly rewriting its execution plan? Weird garbage collection behaviour in your app?
Sites don’t go down for straightforward reasons because they get engineered out. Sites go down as a chain reaction of several things happening that under normal circumstances each thing would have been non-event. The example I always use was a major incident at one of the big data centres at Canary Wharf about 10 years ago. The power went out so they needed to switch to generator power. They had 3 generators that were n+2 when installed. Gen 1 was out for planned maintenance. Gen 2 compressed air system failed so they couldn’t start it. Gen 3 started and carried full load for 15 mins, but as the required load had crept up between install and the time of the event, eventually the rectifier burnt out and the place went dark. Load testing hadn’t shown this up because they’d always done the tests at night when power draw was lower due to more idle equipment. But doing a gen test at peak load for extended periods was considered too risky in those days.
Oh, and the claim that using real log data as the basis of a load test ensures the validity of the test is false. By definition, you can only replay the logs you have which at best would replicate your existing peak load, which you already know how your stack reacts to.
To increase load over that, what do you do? Play the logs twice to double the load? Well that isn’t how real world traffic works. You’ll hit internal data caches because you’re requesting exactly the same data twice. So maybe you randomise the search parameters on your site? Well, that’s not how real traffic works either.
NotoriousREV wrote: Sat Mar 23, 2019 2:59 pm
My more serious point about load testing is this: you build your infrastructure based on your understood load. If you’re doing TV advertising, for example, you estimate the response rate and scale for that plus a contingency, because you have a budget limitation somewhere along the line. Commercial reality is that I don’t really want to have a bunch of compute/memory/storage sitting around doing nothing, hence the growth of PAYG cloud services.
For example, my current site will quadruple its peak load between Christmas Eve and the 1st Monday in January. Generally Monday nights are 30-40% busier than Thursday nights, except for pay day. I’ll pre-provision for those loads because I know scaling isn’t instant.
If I suddenly get a workload that hasn’t been considered before for whatever reason, chances are I haven’t tested that high anyway. I might know where my stack breaks, but that limit may well have been considered high enough to cope with expected peak load comfortably. This is why I work with Marketing to understand their plans so I hopefully won’t get an unexpected load (giggity) but even so, if they send a push notification to every iOS app user at the same time, instead of in batches as planned, during peak traffic and traffic to our API increases by 500% above the previous record in 2 minutes, I rely on the app being well written and being tolerant of slow downs and request queuing in a graceful way. Also I shout at them and make sure they don’t DDoS us again, or if they do want to be able to that, we have a plan and budget to cope with it.
And load testing isn’t as easy as just replaying customer data. To be accurate, I have to point it at my production environment. But if I do that, I can’t place real orders because those are tied to 3rd party systems. So either I decouple those and accept a flawed test, or I place real orders and take the cost hit (not going to happen). I also need to pick a time to do it, but I’m doing other stuff overnight (eg batch jobs) and I also have real traffic even in the dead of night which I’ll affect. The load test needs to last long enough to consider degradations that may appear over time. Can I simulate my 3rd party payment gateway slowing down? 3rd party JavaScript slowing down? A DB suddenly rewriting its execution plan? Weird garbage collection behaviour in your app?
Sites don’t go down for straightforward reasons because they get engineered out. Sites go down as a chain reaction of several things happening that under normal circumstances each thing would have been non-event. The example I always use was a major incident at one of the big data centres at Canary Wharf about 10 years ago. The power went out so they needed to switch to generator power. They had 3 generators that were n+2 when installed. Gen 1 was out for planned maintenance. Gen 2 compressed air system failed so they couldn’t start it. Gen 3 started and carried full load for 15 mins, but as the required load had crept up between install and the time of the event, eventually the rectifier burnt out and the place went dark. Load testing hadn’t shown this up because they’d always done the tests at night when power draw was lower due to more idle equipment. But doing a gen test at peak load for extended periods was considered too risky in those days.
This is more than an iPhone XS Max screens worth of post
NotoriousREV wrote: Sat Mar 23, 2019 10:48 am
I’ve been in this game for a long time (20 years) and I’ve never seen a site fail where load testing says it will.
Then the load testing wasn't done right
Even when we used Akamai to do it?
When was that Rev? We bought Soasta a couple of years ago, and with that came the load testing and real user monitoring products we use today. Who did you work with? If you let me know I can give them a kick up the arse.
On your more recent post I can agree with everything you say, but I do know that all the card providers (Visa, Mastercard, Amex etc) have furnished us with well known 'dummy' numbers that allow us to go all the way through a basket checkout etc, without fulfilling the actual orders.
Obviously when you're interfacing with plane seat booking (is it Galileo or something?) things can get a little more complicated, but I'd be surprised if there's no mechanism to do a 'fake' booking with those providers?
NotoriousREV wrote: Sat Mar 23, 2019 3:22 pm
Oh, and the claim that using real log data as the basis of a load test ensures the validity of the test is false. By definition, you can only replay the logs you have which at best would replicate your existing peak load, which you already know how your stack reacts to.
To increase load over that, what do you do? Play the logs twice to double the load? Well that isn’t how real world traffic works. You’ll hit internal data caches because you’re requesting exactly the same data twice. So maybe you randomise the search parameters on your site? Well, that’s not how real traffic works either.
I'm gonna respond the last time here, because we're massively hijacking this thread, but we don't replay the logs at all. We capture the real user journeys from the client side (not server side logs) capturing beacons using a small JS snippet injected at the edge. Once we have the real journeys we can parameterise just about any aspect of it. How long did user A wait on one page before proceeding? OK mix that up. This user went to look at the black trench coat after looking in the men's coats category. OK we can change that around too so they look at a different item, only this time pretending that the user is on a shitty mobile phone connection rather than a home BB.
Basically we can take all the user journeys, lay them out in a 'Garage Band' style format, and mix it up just about however you like.
Sure, there is the potential to do bad tests, but that's more down to the operator who set it up than the tool itself. Speaking with the guy who certified me this week, he says it took 2 years of just building load tests before he considered that he had even a basic level of competency at it. Load testing, as you identify, can be affected by a huge number of factors and understanding how to do it 'properly' is not something that can be learnt in 5 minutes, or by watching a YouTube video.
The march was great and the turnout rather amazing. Ended up carrying the children (not all at once) for a good couple of miles in total. Hopefully it'll make some difference.
Unfortunately, Richard will most likely be correct.
Look at the kids march on climate change. AFAIK, this government made no pledges at all after the protest. Not even a single gesture of good will. Theresa May's comments on it (criticising the kids being off school) shows how out of touch they all are, and how little they care. They're ignoring the voters of the future FFS.
Our parliament/leadership system doesn't need reform, it needs chucking in the bin in favour of some sort of revolution.
It’s hard to say, if course. And I think it’s a great way to voice public opinion. Our right to demonstrate like that is very important too. I think people who peacefully demonstrate like that should be hugely proud of themselves.
Orange Cola wrote: Sun Mar 24, 2019 8:46 pm
Reports growing that the cabinet want May out. Lead by Chris Grayling so we can all expect the next PM to be Theresa May, who will receive both payment for her role as PM, but also compensation for the disruption of being made PM.
