OV9

drcarlos wrote: Mon Oct 05, 2020 8:09 pm A large IT system developed in a short time probably with limited testing has a glitch, non-shocker. Compounded by some IT illiterate politician point scoring when she probably has sod all idea of the actual system design and heard someone mention excel.

Ordinarily I'd agree, but using .xls in 2020 is pretty unforgivable. They haven't fixed it now, they are just sending in smaller batches (<65k rows).

I bet all his actions have lost him precisely zero voters and reinforced his god-like status amongst his followers.

drcarlos wrote: Mon Oct 05, 2020 8:09 pm A large IT system developed in a short time probably with limited testing has a glitch, non-shocker. Compounded by some IT illiterate politician point scoring when she probably has sod all idea of the actual system design and heard someone mention excel.

I don't agree... It was such a basic error which indicates they have done zero testing at any stage. Or if they did, they didn't do it very well.

Its like the government awarding all these contracts willy nilly during their covid response. I've mainly worked for banks and the amount of due diligence we need to do before issuing contracts for technology work is mind-blowing. The amount of robust process around any technical change, from design through to go-live is even worse. I get that banks are regulated, but I'd hope that government had at least the same level of governance in place for these things but it appears that they can just do what they want.

Nah, I reckon he had it. But bear in mind going from the military hospital to the Whitehouse is like going from intensive care to the regular wards.

He's still getting world class medical care around the clock that most of us couldn't dream of, which hilariously, by all accounts, he's not even paid for.

American politics are weird and hilarious.

drcarlos wrote: Mon Oct 05, 2020 8:09 pm A large IT system developed in a short time probably with limited testing has a glitch, non-shocker. Compounded by some IT illiterate politician point scoring when she probably has sod all idea of the actual system design and heard someone mention excel.

No, no, no.

Excel is well known to have hard limits in its data handling which can be worked around by using alternatives, such as custom code to generate the data infill for the database or just CSV/spread sheet handling software that isn't Excel.

I'm a fucking idiot, and even I know to *not* use Excel if I suspect it's hard limits might be gotten to within 50% of. I have alternate spreadsheet programs that can handle millions of rows in a single sheet with ease.

This is probably sheer bloody mindedness (people know how to use Excel) rather than outright incompetence, but it fully deserves to be called out for what it is - a massive mistake.

Source - I work in telecoms, often handling tens if millions of rows of data at a time pre and post infill/exfil for analysis of fraud and errors, where it's often faster to work in a spreadsheet with some tasty vlookups than it is to write custom SQL. This is really, really basic stuff.

mr_jon wrote: Mon Oct 05, 2020 8:45 pm Ordinarily I'd agree, but using .xls in 2020 is pretty unforgivable. They haven't fixed it now, they are just sending in smaller batches (<65k rows).

Really ? They’re using a version of Excel that old ? It’s been 1,048,576 rows since Excel 2007.

I’ve been having conversations about exactly this for the last couple of weeks as we’re about to run into the million-ish row limit.

More likely they're compiling a larger list, adding it from a CSV, there's a VB front end that fails silently because they'll never reach a million rows.... And boom, data goes walkies.

I mean, they could at least use Access. Good to see Dido had spent that £45m well, eh?

Sundayjumper wrote: Mon Oct 05, 2020 9:06 pm

mr_jon wrote: Mon Oct 05, 2020 8:45 pm Ordinarily I'd agree, but using .xls in 2020 is pretty unforgivable. They haven't fixed it now, they are just sending in smaller batches (<65k rows).

Really ? They’re using a version of Excel that old ? It’s been 1,048,576 rows since Excel 2007.

I don't think it's that they've been running an ancient version of Excel.

You can still save a file as .xls in versions newer than 2007. The. xls format has the row limit.

If they would have saved it as .xlsx it have been fine...perhaps for a little while longer.

Ascender wrote: Mon Oct 05, 2020 8:48 pm

drcarlos wrote: Mon Oct 05, 2020 8:09 pm A large IT system developed in a short time probably with limited testing has a glitch, non-shocker. Compounded by some IT illiterate politician point scoring when she probably has sod all idea of the actual system design and heard someone mention excel.

I don't agree... It was such a basic error which indicates they have done zero testing at any stage. Or if they did, they didn't do it very well.

Its like the government awarding all these contracts willy nilly during their covid response. I've mainly worked for banks and the amount of due diligence we need to do before issuing contracts for technology work is mind-blowing. The amount of robust process around any technical change, from design through to go-live is even worse. I get that banks are regulated, but I'd hope that government had at least the same level of governance in place for these things but it appears that they can just do what they want.

Word. Being on the other side of the procurement fence I've seen it all. Banks are usually very needy. Ordinarily Government work is too. I've worked with various government departments including the NHS over the years and they normally do the required leg work. But it's not just that due-diligence has gone out of the window with C19, but any sort of common sense has gone with it. It's pure incompetence.

mr_jon wrote: Mon Oct 05, 2020 8:45 pm

drcarlos wrote: Mon Oct 05, 2020 8:09 pm A large IT system developed in a short time probably with limited testing has a glitch, non-shocker. Compounded by some IT illiterate politician point scoring when she probably has sod all idea of the actual system design and heard someone mention excel.

Ordinarily I'd agree, but using .xls in 2020 is pretty unforgivable. They haven't fixed it now, they are just sending in smaller batches (<65k rows).

This. Even us non-techies have known for donkeys years about the limits of xls.

It's the column limit they hit, not row afaik.

Beany wrote: Mon Oct 05, 2020 8:58 pm

drcarlos wrote: Mon Oct 05, 2020 8:09 pm A large IT system developed in a short time probably with limited testing has a glitch, non-shocker. Compounded by some IT illiterate politician point scoring when she probably has sod all idea of the actual system design and heard someone mention excel.

No, no, no.

Excel is well known to have hard limits in its data handling which can be worked around by using alternatives, such as custom code to generate the data infill for the database or just CSV/spread sheet handling software that isn't Excel.

I'm a fucking idiot, and even I know to *not* use Excel if I suspect it's hard limits might be gotten to within 50% of. I have alternate spreadsheet programs that can handle millions of rows in a single sheet with ease.

This is probably sheer bloody mindedness (people know how to use Excel) rather than outright incompetence, but it fully deserves to be called out for what it is - a massive mistake.

Source - I work in telecoms, often handling tens if millions of rows of data at a time pre and post infill/exfil for analysis of fraud and errors, where it's often faster to work in a spreadsheet with some tasty vlookups than it is to write custom SQL. This is really, really basic stuff.

Have you any inkling of the time it usually takes to gather requirements, develop, unit test, system test and uat this kind of system? The answer is usually a couple of years requirements gathering and sign off usually takes longer than they’ve had to deploy this. Unless you do it using the agile methodology, then you put in temporary tactical solutions (probably why they were using excel because it was quick, easy and supported the known requirements at the time and it will be replaced with a correct piece of proprietary code later down the line), accept defects and fix forwards, but then you end up with shit like this and morons who’ve never worked on anything the size of this or with this kind of complexity spouting shite on Twitter.

People think complex it systems are done and work in 5 minutes and have no idea how these projects work.

Source - I work for people that actually do this kind of thing at the same scale using similar methods and have had to take shit from morons like this before. Mainly because they never know what they want and move the fucking goalposts all this time.

Ascender wrote: Mon Oct 05, 2020 8:48 pm

drcarlos wrote: Mon Oct 05, 2020 8:09 pm A large IT system developed in a short time probably with limited testing has a glitch, non-shocker. Compounded by some IT illiterate politician point scoring when she probably has sod all idea of the actual system design and heard someone mention excel.

I don't agree... It was such a basic error which indicates they have done zero testing at any stage. Or if they did, they didn't do it very well.

Its like the government awarding all these contracts willy nilly during their covid response. I've mainly worked for banks and the amount of due diligence we need to do before issuing contracts for technology work is mind-blowing. The amount of robust process around any technical change, from design through to go-live is even worse. I get that banks are regulated, but I'd hope that government had at least the same level of governance in place for these things but it appears that they can just do what they want.

Less than 6 months from contract signing to a delivery, remind me in the banking industry how long a system of this size would take to set up from start to finish?

I would suggest a minimum of 2 years.

unzippy wrote: Mon Oct 05, 2020 11:17 pm It's the column limit they hit, not row afaik.

Dunno, the bit I read suggested they had three rows of data per positive test, so 'only' ~22k unique people per upload was the limit. Csv's fed into an .xls-based parser.

Saw a Merc EQC. Looks quite good for an SUV and a lot smaller than I expected!

I know we are full of IT superstars but since 1998 I’ve never known a newly installed or bit of IT software that has simply worked.

I think apart from some auto calculating excel forms (which invariably have some limitation with date or access etc) everything has been version 2,3,4 etc.
This includes software, things like Concur, WinTid, Zalaris, our own work portals etc etc.

Dave!

DeskJockey wrote: Tue Oct 06, 2020 8:24 am Saw a Merc EQC. Looks quite good for an SUV and a lot smaller than I expected!

I too believe all electric cars are riddled with the virus and must be stopped!!

Dave!

V8Granite wrote: Tue Oct 06, 2020 8:31 am

DeskJockey wrote: Tue Oct 06, 2020 8:24 am Saw a Merc EQC. Looks quite good for an SUV and a lot smaller than I expected!

I too believe all electric cars are riddled with the virus and must be stopped!!

Dave!

D'oh. Wrong thread!

https://www.bbc.co.uk/news/technology-54423988

So yes, a row-based problem, though 1400 cases per spreadsheet before they dropped off, so quite a number of rows per case (not just a few as per the article). Presumably a row per contact or something?

Pretty much everything I do works first time, it's called being rigorous!

drcarlos wrote: Mon Oct 05, 2020 11:38 pm

Ascender wrote: Mon Oct 05, 2020 8:48 pm

drcarlos wrote: Mon Oct 05, 2020 8:09 pm A large IT system developed in a short time probably with limited testing has a glitch, non-shocker. Compounded by some IT illiterate politician point scoring when she probably has sod all idea of the actual system design and heard someone mention excel.

I don't agree... It was such a basic error which indicates they have done zero testing at any stage. Or if they did, they didn't do it very well.

Its like the government awarding all these contracts willy nilly during their covid response. I've mainly worked for banks and the amount of due diligence we need to do before issuing contracts for technology work is mind-blowing. The amount of robust process around any technical change, from design through to go-live is even worse. I get that banks are regulated, but I'd hope that government had at least the same level of governance in place for these things but it appears that they can just do what they want.

Less than 6 months from contract signing to a delivery, remind me in the banking industry how long a system of this size would take to set up from start to finish?

I would suggest a minimum of 2 years.

I've been amazed by what we've achieved with this Bank during the covid crisis to enable them to totally pivot large sections of their business to work in different ways and deliver stuff quickly while preserving service. The thing that stood out to me about this thing, if its as-reported, is that at the outset, while building a system to support data from potentially the majority of the English/Welsh populous was that someone suggested Excel and didn't consider any limitations. Or at least how to manage limitations via process? Or have it as a scenario in load/performance testing?

I get that its not easy and everybody is under pressure to deliver, but this just appears to be really, really basic stuff?