Beta
×

Welcome to the Slashdot Beta site -- learn more here. Use the link in the footer or click here to return to the Classic version of Slashdot.

Thank you!

We are sorry to see you leave - Beta is different and we value the time you took to try it out. Before you decide to go, please take a look at some value-adds for Beta and learn more about it. Thank you for reading Slashdot, and for making the site better!

Toyota Acceleration and Embedded System Bugs

Soulskill posted more than 4 years ago | from the bugs-that-lead-to-crashes dept.

Bug 499

An anonymous reader writes "David Cummings, a programmer who worked on the Mars Pathfinder project, has written an interesting editorial in the L.A. Times encouraging Toyota to drop claims of software infallibility in their recent acceleration problems. He argues that embedded systems developers must program more defensively, and that companies should stop relying on software for safety. Quoting: 'If Toyota has indeed tested its software as thoroughly as it says without finding any bugs, my response is simple: Keep trying. Find new ways to instrument the software, and come up with more creative tests. The odds are that there are still bugs in the code, which may or may not be related to unintended acceleration. Until these bugs are identified, how can you be certain they are not related to sudden acceleration?'"

Sorry! There are no comments related to the filter you selected.

Toyota: (5, Funny)

dushkin (965522) | more than 4 years ago | (#31464824)

Always going forward.

Re:Toyota: (0, Redundant)

RyanCheeseman (1180119) | more than 4 years ago | (#31464838)

sometimes just a bit too fast.

Re:Toyota: (0)

Anonymous Coward | more than 4 years ago | (#31464954)

Re:Toyota: (-1, Flamebait)

Anonymous Coward | more than 4 years ago | (#31465054)

Always going forward.

Niggers: always going backward.**


**Unless high crime rates and high rates of illegitimate children are your idea of "progress".

Re:Toyota: (1)

TheLink (130905) | more than 4 years ago | (#31465062)

The car in front is a Toyota. And guess why ;).

Re:Toyota: (2, Funny)

Afforess (1310263) | more than 4 years ago | (#31465070)

"The race for quality has no finish line- so technically, it's more like a death march."

Not my quote, so I'll give them props: http://www.despair.com/quality.html [despair.com]

Re:Toyota: (-1, Troll)

Warped-Reality (125140) | more than 4 years ago | (#31465158)

At least 3 mod points will be wasted on this post.

Dumbasses.

Re:Toyota: (1)

Anonymous Coward | more than 4 years ago | (#31465198)

At least 3 mod points will be wasted on this post.

Dumbasses.

His or yours? At least his contributes.

Impossible to test (4, Informative)

Darkness404 (1287218) | more than 4 years ago | (#31464840)

Most software is nearly -impossible- to test under flawless conditions. Especially embedded systems with small amounts of CPU power and memory.

Plus, all this hype around these Toyota acceleration problems is just that, hype.

Re:Impossible to test (5, Insightful)

Anonymous Coward | more than 4 years ago | (#31464870)

Right, just hype. Except for those families were killed by the Toyota acceleration problems.

Re:Impossible to test (5, Informative)

Darkness404 (1287218) | more than 4 years ago | (#31464958)

...And if you look at the facts, you can see that all of the symptoms could easily be caused by driver error. Look at this http://www.nytimes.com/2010/03/11/opinion/11schmidt.html?scp=1&sq=driver%20error&st=cse [nytimes.com] (currently the page doesn't need registration, your results may change in the coming days/hours).

Re:Impossible to test (1, Insightful)

Anonymous Coward | more than 4 years ago | (#31465020)

Right. Driver error. You mean it wasn't the floormats (first excuse), brake pedals (second excuse) and now something else? Why are you apologizing for Toyota? They have killed a number of people now, including children. Unacceptable. And yes, other manufacturers have issues too.

Re:Impossible to test (5, Insightful)

The End Of Days (1243248) | more than 4 years ago | (#31465100)

"Unacceptable" is strong. Sad, yes, but this is real life. There is no such thing as zero risk. Taking the attitude that it is somehow achievable despite the utter impossibility is something that makes for a good trial lawyer but a terrible human.

Re:Impossible to test (1, Insightful)

Anonymous Coward | more than 4 years ago | (#31465188)

What Toyota is doing is unacceptable. They have had this issue since 1986. 1986! In 2003 this problem surfaced in the Sienna, but they hid it for 5 years. They now are blaming the aftermarket floormats, the drivers, the pedals, whatever. UNACCEPTABLE. And then ridiculous people are defending them.

Get this people: we are in a economic war with Asia, which we are currently losing. It affects you and your future in a very real way. Supporting/defending crappy foreign companies doesn't do you any good, so stop it.

Re:Impossible to test (4, Insightful)

causality (777677) | more than 4 years ago | (#31465266)

[Some action] doesn't do you any good, so stop it.

Since when did that ever prevent anyone from doing anything? You must have us confused with some society that generally considers the full implications and long-term repercussions of our decisions...

Re:Impossible to test (0)

Anonymous Coward | more than 4 years ago | (#31465216)

And yes, other manufacturers have issues too.

The prevailing form of idiocy is that if you don't specifically disclaim something, then you must be suggesting it. So I'm glad you threw that in there. Otherwise there'd probably be a whole new thread created by some well-meaning individual who knows little or nothing about argumentation. The thread would be all about manufacturer errors and defects found in vehicles made by Ford, Chrysler, GM, and other manufacturers, as though that had any relevance in a discussion about Toyota's problems, as though it could possibly excuse or further condemn Toyota. That's the only reason you felt a need to add that line.

Such uselessness that your line there is intended to prevent would be another instance of fanboyism. Like Thoreau said, if you're familiar with the concept and the principles, why the need for a myriad of instances and iterations? "I like Toyota and identify with them in some way or another, so your constructive criticism about them is like insulting my TEAM man! Saving face is the only thing that matters, so now I have to insult some of the OTHER teams so that they're no better! Hey, I can do that by bringing up irrelevant crap about other manufactuers!"

The same principle applies to most discussions around here about Microsoft, Apple, and Linux. It's why you can't discuss the fact that most spam-spewing zombies are infected Windows machines, and look at options that might realistically address this problem, without some fool saying "but if Linux had the same marketshare it'd be a virus-ridden spam-factory too!" A, people who are skilled with both Linux and Windows (and therefore, can make educated comparisons) generally do not say this. B, the potential future compromisability of Linux if it had a hypothetical position might be interesting but doesn't address the growing problems we have RIGHT NOW with Windows. C, it's a tired old argument that is always rehashed in those discussions, page-from-playbook style, and contributes absolutely nothing. It's just Windows fanboys who feel slighted and want to save face for "their team" by equating it with others.

If we could collectively get over this and realize what silly shit it is, maybe we could find productive solutions for previously insoluable problems. Imagine what it would do to politics if people cared about rational solutions that realistically address known problems, based on evidence, and stopped caring about what the other guy is doing or what he will think. Crazy, isn't it?

Re:Impossible to test (5, Interesting)

zappepcs (820751) | more than 4 years ago | (#31465108)

There are a couple of things that should be mentioned here. NASA has shown what it takes to make very small, very good code. Sure, they too have failures, but 'nearly' bug free code is quite expensive. Second, writing code is not quite like trying to create a hand crafted dashboard, if the dashboard fades, no one dies. Embedded software is quite a different beast from your normal desktop applications. When you add motion control and interaction with the code, it difference between them gets even more complex. Software in vehicles should be two things:

Open - let lots of folk see what could be wrong
Audited - audited to meet specific standards of safety and operation. Not quite the self-defeating government regulations, but more of a case by case issue: if the software has control or input to the control mechanism for the engine, braking system, suspension etc. it must meet minimum standard testing requirements. Any action that _could_ arbitrarily apply mechanical action must be tested and controlled beyond all reasonable testing/doubt. Everything should be tested, down to a pet chewing on the control cable harness.

Consumers are encouraged to think the vehicles they buy are safe and require no special knowledge of engineering or mechanics to operate. As long as they are given to think that, then passenger vehicles should be made to be just this way.

The problem for Toyota now is multifaceted. One, they have a PR shitstorm to deal with. Two, there is a dollar effect of this problem. Three, it's now on the shoulders of Toyota to get this part right for the rest of the passenger vehicle making industry.

It's possible that they might walk away from this fire with only minor long term burns and the reputation for building the safest vehicles. BUT, reading the article of this post and paying attention while doing so is necessary... IMO

Re:Impossible to test (0)

Anonymous Coward | more than 4 years ago | (#31465238)

One, they have a PR shitstorm to deal with.

More accurately, they are fighting Government's propaganda.

Re:Impossible to test (4, Insightful)

DeadPixels (1391907) | more than 4 years ago | (#31464872)

By and large, it would seem that Toyota should probably be looking for exceptional conditions rather than typical ones. Correct me if I'm wrong, but if a relatively small number of vehicles have actually exhibited the acceleration issue, it would seem like any bugs related to that would be in conditions that may not occur very often during typical driving. Seems to me that "outlier" cases or unusual methods of testing would be the best way to start; testing with typical driving conditions might not show anything.

Re:Impossible to test (0)

Anonymous Coward | more than 4 years ago | (#31464900)

By and large, it would seem that Toyota should probably be looking for exceptional conditions rather than typical ones. Correct me if I'm wrong, but if a relatively small number of vehicles have actually exhibited the acceleration issue, it would seem like any bugs related to that would be in conditions that may not occur very often during typical driving. Seems to me that "outlier" cases or unusual methods of testing would be the best way to start; testing with typical driving conditions might not show anything.

*DING*DING*DING*DING* These problems are the outliers, hence nobody should expect to find them under "typical" driving conditions.

Re:Impossible to test (1, Insightful)

Anonymous Coward | more than 4 years ago | (#31464878)

Most software is nearly -impossible- to test under flawless conditions. Especially embedded systems with small amounts of CPU power and memory.

Plus, all this hype around these Toyota acceleration problems is just that, hype.

Its not just software. This is an embedded system, the software is highly dependent on the hardware involved. When I say hardware, I mean the entire system of boards, wires, and connectors in the vehicle. Embedded systems have a tight couple of software and hardware unlike your average computer application. A slight error in the hardware systems would also send the software in some unknown behavior if not tested correctly/fully. Its a fairly difficult feat to ensure complete software/hardware compliance in an embedded system.

Re:Impossible to test (4, Interesting)

WrongSizeGlass (838941) | more than 4 years ago | (#31464960)

Exactly. Even a minor revision in a FPGA could result in unforeseen consequences. Who knows, maybe a chip manufacture failed to document a very small change to a product line (or had a typo in the docs). The problem may not be in Toyota's code, just in their cars.

Re:Impossible to test (1)

twisteddk (201366) | more than 4 years ago | (#31464996)

Nothing is impossible. It's simply a matter of cost vs. benefit. Coding is many more lines today than it was 15 years ago. And a lot of copy paste of libraries you haven't coded or tested yourself. So ofcourse problems occur. But testing SHOULD be more than simply compiling the code.

Back in the day when I took my masters in computer science, we were forced to show how we tested EVERY loop, if, while or orhter "choice" or iteration of a possiblity. Simply to ensure that the code was 100% stable. Using closed libraries, like say a predefined definition of the type "string" or "integer" that was at your own peril. Programmers today just give up too easily, and companies cut costs by scriping on testing, ducumentation and verification.

Re:Impossible to test (2, Insightful)

The End Of Days (1243248) | more than 4 years ago | (#31465168)

Programmers today just give up too easily, and companies cut costs by scriping on testing, ducumentation and verification.

I'll start with the cheap joke - apparently you didn't learn the lesson too well.

And then the actual point - this is the real world, sir. Academics may have the luxury of taking as much time as they like to never produce result. That's not how things actually work for the rest of us.

Boeing versus Airbus (5, Insightful)

goombah99 (560566) | more than 4 years ago | (#31465044)

I'm loving this conversation here because I've gotten crucified in slashdot before for making simmilar comments to the whole thread here. I grew up in a family of top managers of Boeing systems engineers. They hated computers. My dad never even learned how to turn one on. He hired other monkey to use the computers. As A child I was regailed with wonderful stories of every hard lesson in safety my dad had learned over his lifetime. He loved world war II because they got to use cutting edge designs for balls out performance yet at the same time learned how to make things reliable by disecting the accident. He would tell me about the accident that taught them that the engine pumps need to be at full speed but flow stalled on take off so that there's no lag when you hot swap after a pump fails. He told me of the accident where they learned not to route 100% of the control system wiring through any one junction box. etc...

Probably because of all these hard won lessons boeing for years insisted on fully mechanical or hydraulic flight surface controls. Whereas Airbus and other jumped on the fly-by-wire concept early. My dad would spit after hearing some youg person tout all the advantages of fly by wire. He knew them perfectly well. He was big on accepting new innovations to reduce fuel costs and increas performance. He was not a luddite. But he had a safety background that told him these electonic systems were hard as hell to validate and hard as hell to make truly independent from each other.

For example they often used triple redundant computers and if one of them disagreed the other two would vote it off the island and stop listening to it. From what I've read it's now suspected that the latest airbus crash in the pacific had one of it's root problem in the voting nexus where a superior computer over ruled a more primitive safety system.

While we all know that computer software validation is hard if not impossible. It's not something we readily admit here on slash dot. It's because for years people like my dad would throttle the innovations the computer engineeers wanted to implement. I think as a result there became this culture of computer engineers that presented the case that embedded computing could be made safer than it really could be to offset that.

So now we come full circle and have to admit there is this middle ground. Just because a computer can improve perfromance does not mean it's reliable and safe. The old guys had a point after all when it came to safety.

Next week I'll tell you about how the ancient shocking lesson of the British Commet aluminum aircraft wings falling off led to the unanticipated discovery of metal fatigue and probably was the reason Boeing was slow to move to composite materials in commercial aircraft (but not in military aircraft). In hind sight we have heard of many tales of the composite tails of plane falling off as the reason for the loss of control before a crash. Conversely, composite wings on UAVs allow them to absorb a lot of bullet holes with no loss of control and to operate under higher perfromance conditions.

The point is that safety and performance are trade offs when both are pushed to the limit. The old guys know a lot more about safety than you might expect. The young guys are all about performance.

Re:Boeing versus Airbus (1)

DeadPixels (1391907) | more than 4 years ago | (#31465152)

For example they often used triple redundant computers and if one of them disagreed the other two would vote it off the island and stop listening to it.

Sounds a little like Minority Report [wikipedia.org] , doesn't it?

Each of the three precogs generates its own report or prediction. The reports of all the precogs are analyzed by a computer and, if these reports differ from one another, the computer identifies the two reports with the greatest overlap and produces a majority report, taking this as the accurate prediction of the future.

Re:Impossible to test (1)

david.emery (127135) | more than 4 years ago | (#31465094)

"Impossible to test", but that does not mean that it's impossible to write bug-free software. It requires a substantially different approach to specification and construction than most people/companies currently use. Model Checking (http://en.wikipedia.org/wiki/Model_checking) and SPARK (http://en.wikipedia.org/wiki/SPARK_(programming_language) ) are two approaches that work. It's worth looking at what the commercial avionics industry requires for its embedded software, where 10 ^ -9 is the requirement for safety-critical avionics (http://en.wikipedia.org/wiki/DO-178B )

Note though, that no amount of 'construction by correctness' approach for software will make up in deficiencies in specifications. See the work of Nancy Leveson (http://en.wikipedia.org/wiki/Nancy_Leveson or http://sunnyday.mit.edu/ [mit.edu] ) and John Knight (http://www.cs.virginia.edu/~jck/ ) for both discussions and analysis of safety-critical software approaches and analysis of how some of these approaches have not worked as well as expected (e.g. Leveson's critique of N-version programming.)

Infallible fail. (5, Insightful)

jeckled (1716002) | more than 4 years ago | (#31464856)

Drive by wire is great and all, but I'd feel much better with a physical fail-safe than their "infallible" software. I am aware of the physical remedies for the issue, but I'd like to see the brake pedal override the accelerator.

I agree on non-software fail-safes (1)

mrflash818 (226638) | more than 4 years ago | (#31464948)

My thought is that the car should have either:

1. A hand brake, and the handbrake have a physical cutout switch (relay) that shuts off the fuel pump and disconnects the motor controller from the motor if pulled, and the car is in any but 'park' or 'neutral.'

2. A force sensor on the brake pedal, and if pushed hard (panic stop), would also trip the breaker for fuel pump and motor controller.

The concept is not new, in terms of overrides. For example: Many cars require the clutch pedal be depressed to start the car, or foot on the brake pedal.

There may be better ways to do it, but seems like an 'intuitive' way for the driver, and reasonably easy for an automotive engineer to add into the design.

Re:I agree on non-software fail-safes (2, Informative)

peragrin (659227) | more than 4 years ago | (#31465010)

technically that is what part of the update does. It forces the computer to always choose the brake over the accelerator when both pedals are registering. So if the car does accelerate a tap on the brakes should disengage it.

Re:I agree on non-software fail-safes (4, Insightful)

KarmaMB84 (743001) | more than 4 years ago | (#31465172)

Unfortunately the update assumes the computer will actually respond to the brake being pressed or any input for that matter. Toyota doesn't know for certain what is causing all of these sudden acceleration problems in which fiddling with the gas pedal, brake and even putting the vehicle in neutral won't stop the vehicle. The software update, while a sensible modification that should've been in the software all along, is sort of a hail mary toward preventing any new cases in updated vehicles.

Re:I agree on non-software fail-safes (1)

mat128 (735121) | more than 4 years ago | (#31465076)

Cutting the fuel pump isnt the right solution, what if you need to accelerate right after you braked hard? You're doomed, you just killed the engine!
They should just implement a huge red panic button to shut everything off :)

Re:I agree on non-software fail-safes (3, Funny)

WrongSizeGlass (838941) | more than 4 years ago | (#31465154)

They should just implement a huge red panic button to shut everything off :)

They could even buy them at Staples [staples.com] . Need to stop your out of control Toyota? That's easy.

Re:I agree on non-software fail-safes (1)

Vegeta99 (219501) | more than 4 years ago | (#31465192)

My VW doesn't care about the clutch being in OR the transmission being in neutral - you can use the starter to smash out the car in front of you in the parking lot.

Now the drive-by-wire? Any funkyness in input or measured output (throttle angle), and bam, I'm on the side of the road.

Re:Infallible fail. (4, Informative)

shrimppesto (766285) | more than 4 years ago | (#31465176)

i'd feel much better with drivers who know they should pop the car into NEUTRAL if it starts accelerating out of control for any reason, rather than trying to stand on the brake pedals while dialing 911 ...

oh hey... (0)

bmecoli (963615) | more than 4 years ago | (#31464866)

Don't mind me, just testing my new Toyota keyboaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa

Another interesting statistic (5, Interesting)

homer_s (799572) | more than 4 years ago | (#31464868)

From here [washingtonexaminer.com] :

In the 24 cases where driver age was reported or readily inferred, the drivers included those of the ages 60, 61, 63, 66, 68, 71, 72, 72, 77, 79, 83, 85, 89—and I’m leaving out the son whose age wasn’t identified, but whose 94-year-old father died as a passenger.

These “electronic defects” apparently discriminate against the elderly, just as the sudden acceleration of Audis and GM autos did before them. (If computers are going to discriminate against anyone, they should be picking on the young, who are more likely to take up arms against the rise of the machines and future Terminators).

Some more data here [theatlantic.com]

Re:Another interesting statistic (4, Insightful)

maxume (22995) | more than 4 years ago | (#31464890)

Be careful to note that the 24 cases discussed there are only the ones that have led to serious incidents.

Re:Another interesting statistic (1)

DeadCatX2 (950953) | more than 4 years ago | (#31465184)

Would that suggest a potential correlation between reaction time and general seriousness of the possible incidents?

Re:Another interesting statistic (5, Informative)

maxume (22995) | more than 4 years ago | (#31465260)

To me it suggests that older drivers are having more difficulty coping with the situation once it arises.

Forbes says that the guy who got himself plastered all over cable last week was 'afraid' to put the vehicle into neutral, or to turn off the engine:

http://www.forbes.com/2010/03/12/toyota-autos-hoax-media-opinions-contributors-michael-fumento.html?boxes=financechannelforbes [forbes.com]

(They link the 911 recording:

http://www.thetruthaboutcars.com/the-jim-sikes-911-call-23-minutes-of-unintended-acceleration/ [thetruthaboutcars.com]

)

So apparently being an idiot is also a likely factor in the failing to cope with the incident before it becomes lethal.

But they key observation is that the higher number of fatalities among older drivers doesn't really point to the source of the problem being driver error (rather, the driver error is in failing to deal with the situation once it arises).

Re:Another interesting statistic (1)

jeckled (1716002) | more than 4 years ago | (#31464950)

I have a feeling that if you adjusted those numbers for the distribution of Toyota owner's ages, the numbers would look more reasonable. I'm not trying to say age is not a factor, just mentioning there are other variables influencing that research.

Re:Another interesting statistic (3, Funny)

WrongSizeGlass (838941) | more than 4 years ago | (#31464994)

Now, come on. We can't blame age for this. I plan to be that old one day ... as long as I don't buy a Toyota, that is.

Re:Another interesting statistic (2, Informative)

f16c (13581) | more than 4 years ago | (#31465022)

Something to understand about those statistics: This is a self selected group based largely on income. Camrys may be everywhere but Prius' tend to be expensive.

Re:Another interesting statistic (1)

Lehk228 (705449) | more than 4 years ago | (#31465040)

also, since this is the group of serious accidents, older people are less likely to react correctly and in a timely manner to the problem.

Re:Another interesting statistic (1)

Skapare (16644) | more than 4 years ago | (#31465240)

Exactly. I can sense what my car is doing while driving it, such as gear changes in the automatic transmission, and when the tires are on the edge of small narrow roads. But my father, 25 years older, can't. He also has much slower reaction times. I'm glad he's not driving a Toyota, but I'm still not comfortable with him driving either car we do have.

Re:Another interesting statistic (1)

mister_playboy (1474163) | more than 4 years ago | (#31465160)

2010 Camry base price: $19,595 - most expensive trim starts at $26,400
2010 Prius base price: $22,800 - most expensive trim starts at $28,070

Even a fully optioned Prius couldn't be described as expensive by new car standards. The price difference isn't enough to expect it would impact the type of owners very much... they fall in the same price range.

Re:Another interesting statistic (0)

Anonymous Coward | more than 4 years ago | (#31465182)

Yeah, the past generations of elderly had problems with Audis, the actual generation of elders has problems with the poorman's car, the Toyotas, and our generation, if we become elders, will have problems with the chains of our India made bicycles, as our economy keeps falling down and our 401ks are just evaporating...

Testing. (4, Insightful)

Ihlosi (895663) | more than 4 years ago | (#31464884)

Testing only confirms the absence of known bugs. Never forget that.

Re:Testing. (-1)

Anonymous Coward | more than 4 years ago | (#31464914)

Never forget that.

Forget what?

Re:Testing. (0)

Anonymous Coward | more than 4 years ago | (#31465072)

Before this goes too far forward--I'd like to object. If the hardware or system I deploy on doesn't meet spec, it isn't a bug--it's user error. If my compiler, or the microcode it runs on top of have a failing--it's not a bug, it's a vendor failure. Admittedly, the user will never know the difference. But the purchaser should.

Now--I'll admit, *I* can never detect the absence of all bugs in something the size of an operating system. But on an embedded piece of hardware--if people supply the power they said they would supply, if they run it in the temperature they said they would run it at, if my chip and compiler run according to spec (if they even HAVE a spec). Then yes, I can build a system and run a test of all possible legal inputs. You go and throw 5 amps over a 10 milliamp input device, and there'll be magic smoke. You throw 20 over it, and maybe it will start lying to you as it de-calibrates... but those are hardware failings, and also out of spec.

But when I've got 1024 inputs, and I'm just reprocessing and presenting them--I *can* guarantee an absence of bugs if you agreed to the specification.

When I've got a "mere" 50,000 lines of code, I *can* write something without bugs--although most nobody wants to pay for it. If the behavior was properly identified. Of course, a huge chunk of that will probably be error detection and a big array of messages. Specification doesn't mean "compute the area of a circle"--it means "compute the area of a circle, whereby a circle is specified as a radius in units of [UNITS], you shall be accurate to within [MARGIN]. User input shall be a positive, real number in american ascii via stdin with a total representation of length no greater than [LEN]. If it is not, you will return message [MESSAGE] and exit with status code 1. The program will be executed on the bourne shell via command "areacircle" and be installed in /usr/local/bin", with any dependencies installed [...] and will run on kernel [version] with libc [version]"

Unfortunately, most people never write or agree to a specification in their entire life--and when they do, they come back in three weeks and say "this isn't what I needed." Software free of bugs is possible--it's just very difficult and costly.

But speaking as a programmer who does test on his own and other projects--it's time to stop calling most bugs bugs, and time to start calling them "specification faults" If people actually indicate what they wanted, there'd be a lot less bugs. Unfortunately, people lie--they say "I just want the area of a circle". What they think they mean is "I just want an easy way to get the area of a circle, and I want it to [just work] like Apple products. Don't give me any errors, and don't make me think about it. Just do the right thing already, you can usually tell anyway". And then the programmer sits down, comes up with the 55 different ways this can obviously fail, comes back with a list--and the user complains the project is over budget and adds that they'd only ever want to represent at most 5 digits anyway, why did we even ask? (Until next Wednesday, when they forgot about this). And why the heck didn't you cross-compile it for Win98, and make sure that you statically linked it against the requisite libraries--this thing breaks my spywaretoolbar...?

Spec. Fault.

Not. The Programmers. Fault.

Call this crap what it is: "specification fault"

Of course, there is an alternative theory... (1, Insightful)

Anonymous Coward | more than 4 years ago | (#31464888)

Driver error, much like the Audi "unexplained acceleration" problem.

When Audi came up with a new automatic transmission interlock that forced drivers to put their foot on the brake before they could shift out of park, the problem went away.

The US government and its friends in the not-so-big three are using this as an excuse to pick on Toyota. No other car recall has had anything close to this level of government intervention.

Of course, my car has a fail-safe mechanism to disconnect the engine from the wheels - it's called a clutch. You should get one on your next car.

Maybe it's not a bug, maybe it's an Easter Egg. (0)

kurt555gs (309278) | more than 4 years ago | (#31464902)

I haven't heard much about this, but disgruntled employee(s) that work for some blood sucking temp agency writing this control code could have easily hidden an Easter Egg that activates on some obscure set of triggers like changing the temperature while the left turn signal is on, etc.

I am thinking this is actually the most likely scenario to why these problems are happening with Toyota.

Toyota: Treat your code monkeys better!

 

pay for QA and don't over work coders that makes b (1)

Joe The Dragon (967727) | more than 4 years ago | (#31464990)

pay for QA and don't over work coders that makes bugs. When people are working 80+ hours they make more bugs then people doing 40. Also pay for QA and don't cut back there. Look at the xbox 360 to see what that left them with.

Re:Maybe it's not a bug, maybe it's an Easter Egg. (2, Insightful)

Anonymous Coward | more than 4 years ago | (#31465042)

The type of people that purposely hide bugs that will likely kill several people are the same type of people you can't really "appease" no matter what you do.

Re:Maybe it's not a bug, maybe it's an Easter Egg. (1)

mat128 (735121) | more than 4 years ago | (#31465092)

I can't believe they dont do code reviewing by peers and stuff like that. They have a pretty good mentality about things and I guess that applies to software coders as well.

Logic of Testing (4, Insightful)

Renegade Lisp (315687) | more than 4 years ago | (#31464908)

David Cummings does seem to know what he's talking about, but as it is written, there is some strange logic in the article.

If Toyota has indeed tested its software as thoroughly as it says without finding any bugs, my response is simple: Keep trying.

Testing cannot prove the absence of bugs, only their presence. There are two things that do not follow from this:

  • If you don't find any bugs, then your software doesn't have any.
  • If you don't find any bugs, then there must be some left in your software.

It sounds to me as if Toyota is saying the former, while Cummings says the latter. Neither is a correct conclusion.

Re:Logic of Testing (4, Insightful)

marcansoft (727665) | more than 4 years ago | (#31464944)

Given practical software engineering conditions though, a) is highly unlikely while b) is highly likely.

Re:Logic of Testing (3, Insightful)

Renegade Lisp (315687) | more than 4 years ago | (#31465018)

I'll grant you that, but what I don't understand is this:

If you test, and do find some bugs, does that allow you to put any more trust in your software than if you tested and didn't find any?

Re:Logic of Testing (1)

marcansoft (727665) | more than 4 years ago | (#31465166)

That's a tricky question. Testing and finding bugs ought to allow you to put more trust in your testing methodology, which subsequently can increase your trust in your software once you stop finding bugs. Testing and finding no bugs, hard as you try, quite likely means you aren't trying hard enough. Very rarely will software reach a ceiling of reasonable test-proofness before being shipped that cannot be improved with subsequent, more dedicated, more specific testing after some issues are detected in the field.

The more (and better) you test without finding bugs (or the bugs you want), the higher your chances of the target bugs not existing. Finding other, unrelated bugs helps ascertain that you are, in fact, performing a useful level of testing. If Toyota had found, say, three or four controlled panic style bugs (caused by defensive programming) that would cause engine shutdown (which is probably safer than unintended acceleration) but nothing related to acceleration, then in my mind it would be better evidence against the acceleration bug existing in software. The types of bugs do matter though - finding flimsy memory corruption or race condition type bugs might instead mean that the code has been written without proper programming practices and might lower my trust on it.

Ultimately though, it's true that no level of testing can conclusively prove the absence of bugs.

Re:Logic of Testing (1)

$RANDOMLUSER (804576) | more than 4 years ago | (#31465052)

How about this howler:

The only viable theory we could come up with was that an interrupt (an external hardware stimulus such as a timer going off) had occurred at just the right microsecond within the execution of Stolper's software. Furthermore, we theorized, the operating system (the equivalent of Windows on the flight computer) had a bug that caused it to misremember whether an arithmetic carry had occurred just before the interrupt. Although highly unlikely, it was the only credible explanation we could come up with. Because this was a new version of the operating system built for Pathfinder, still not yet fully tested itself, this theory had some credibility.

So they speculate that code they wrote had an interrupt routine that was not bracketed with PUSHF/POPF instructions!!! Which is like Assembly 101.

Re:Logic of Testing (1)

Renegade Lisp (315687) | more than 4 years ago | (#31465112)

So they speculate that code they wrote had an interrupt routine that was not bracketed with PUSHF/POPF instructions!!! Which is like Assembly 101.

I didn't read it that way. He's talking about an arithmetic carry condition being misremembered across an interrupt. This sounds to me like a CPU-internal hardware condition that might not have been included in the regular set of data that you save across an interrupt. Maybe the wrong type of PUSHF/POPF was used. It certainly doesn't sound like Assembly 101.

Re:Logic of Testing (0)

Anonymous Coward | more than 4 years ago | (#31465088)

Every sufficiently complex piece of software has at least one bug. Every sufficiently complex piece of software can be reduced by at least one line. It follows that every sufficiently complex piece of software can be reduced to a single line that doesn't work.

Re:Logic of Testing (1)

bughunter (10093) | more than 4 years ago | (#31465200)

If you don't find any bugs, then there must be some left in your software.

No, Cummings is not saying that.

Cummings is saying "There are always bugs. Therefore, if you don't find any, then you're not looking hard enough."

My SQA days for NASA manned space subcontractors taught me that no software is bug-free, unless it's trivially simple ("Hello World!"). The best you can hope for is to remove the critical ones. And the more you do to try and secure the software, make it safer and more error-tolerant, the more bugs you introduce. So you can never be 100% positive that there's not another critical bug hiding somewhere, even when you think you've found the "last one."

Especially when you think you've found the last one.

The horror scenario for Toyota (0)

Anonymous Coward | more than 4 years ago | (#31464916)

Notice that they always emphasise that 'there is no evidence that embedded software contains flaws'. Actually, Cummings notes it as well.

Remember also the statement of president Toyoda, mentioning that focus has been on speed more than quality.

There is a theoretical possibility that software development has exceeded the bounds of where you can have total overview of the code at all times (as in military and life critical applications). For example, if you develop 50 different software components controlling each of 50 components in car 1, it is very attractive to reuse 45 of those in car 2 and only rewrite the software for the 5 components that have changed. There is no need to test a module that has worked in all your previous cars, right?

And when you plug these together at the shipping stage and discover that it works 100% of the time (rounded up to the nearest thousand tests), you are set to go.

If the problem WAS in software, Toyota would have to either 1) invest a lot very quickly in bug testing which may or may not be successful, 2) rebuild the software from the ground up, 3) scrap all the cars. In the meantime they would have to absorb the costs of A) providing full refunds for the purchase to all customers, or B) providing rental cars to every owner of a Toyota for the months it takes to do these things.

Both of those options are unimaginably costly.

Therefore, if you so much as start going down the path that the problems COULD lie in the embedded software, it's pretty much total game over for Toyota.

Help me benefit from media hype (1)

Kohath (38547) | more than 4 years ago | (#31464918)

I'm in the market for a car and everyone is picking on Toyota now.

I don't believe in stupid media hype. I don't believe cars rewire themselves. And I know how to hit the breaks, shift into neutral, and/or turn off the key when I want the car to go slower (so far, hitting the breaks has always proven adequate).

Are there any really good deals on Toyotas available?

Re:Help me benefit from media hype (1)

thewils (463314) | more than 4 years ago | (#31465006)

Be careful about switching off the ignition while you're still moving. It could cause loss of steering and/or braking ability. That's b-r-a-k-i-n-g, not b-r-e-a-k-i-n-g.

Re:Help me benefit from media hype (1)

sir_eccles (1235902) | more than 4 years ago | (#31465074)

Not quite true. You may lose "power assisted" braking and steering but the wheel will still steer the car and the brakes will still work, it will just take a little more effort. Those old enough will remember a time before power assisted steering and braking.

Re:Help me benefit from media hype (1)

WrongSizeGlass (838941) | more than 4 years ago | (#31465196)

If they turn the key all the way off the steering wheel can lock. At least it does on my car.

Re:Help me benefit from media hype (1)

Kohath (38547) | more than 4 years ago | (#31465078)

Yeah, it has never actually been necessary to turn off the ignition. But my engine has killed while moving before. Steering and brakes still function, not as well, but good enough for a safe stop.

Software failed to fix my spelling of "brakes"! Someone call David Cummings!

Re:Help me benefit from media hype (1)

WrongSizeGlass (838941) | more than 4 years ago | (#31465082)

That's b-r-a-k-i-n-g, not b-r-e-a-k-i-n-g.

Lack of one leads to the other. I'm sure he'll figure it out ... I hope.

Since the actual problem hasn't been identified yet, who knows if all Toyotas have it or just a select few? Maybe it's a date/time bug? Integer overflow bug? I don't know what the problem is, but I know where it is and I'm not willing to put my family at risk in order to secure a deal on a car that may be more dangerous than most.

Re:Help me benefit from media hype (1)

simp (25997) | more than 4 years ago | (#31465214)

careful with that line of reasoning. The gear shift has no mechanical linkage to the actual gearbox. So when you shift to neutral you just send an electronic command to the gearbox computer to change to N. And the sometimes the gearbox will ignore that and NOT switch to neutral. This was originally done by design to protect the engine and gearbox under specific circumstances (full load and high rpms).

Re:Help me benefit from media hype (3, Informative)

John Hasler (414242) | more than 4 years ago | (#31465262)

> And I know how to hit the brakes...

With the engine past the redline there is very little vacuum to operate the power brakes. Without power assist the brakes may not be able to overcome the engine (this is, IMHO, a fundamental design defect).

> ...shift into neutral...

The computer may not let you do that with the car moving and the engine at high rpm. After all, the engine and/or transmission might be damaged (another design defect).

> ...and/or turn off the key...

Some of these vehicles don't have keys: just a radio remote. The emergency shutdown procedure is to hold a button down for three seconds (another design defect).

Exactly (-1, Offtopic)

Reality Master 101 (179095) | more than 4 years ago | (#31464924)

I have to say, it's this specific reason that makes me wary of anthropogenic climate change. A lot of the evidence is based on computer models, and anyone who has programmed computers knows how difficult it is to get anything computer related right. The most shocking thing about "climategate" to me wasn't the content of the emails, it was the fact that the computer models used were NOT openly available to all, and were not published with the research papers. And when you combine that with the insane complexity of the climate and the limitations of our computers, I can't help but feel that computer models are a blunt, crude tool at best, and at worst a tool of misinformation (even if the wielders have "good intentions").

I'm willing to accept that human activity is causing significant climate change, but don't use computer models to prove it to me. By all means, use computer models to gather clues about what to test. But computer models are manufactured evidence that can be made to say anything. Anyone using computer models to argue that seismic shifts in human culture are needed should be taken out and flogged until their real agenda is admitted. No one with any clue about computers would argue that they are infallible oracles of truth (or even infallible controllers of accelerator pedals).

Re:Exactly (1)

peragrin (659227) | more than 4 years ago | (#31465066)

you want just plain silly the computer models they use are different from the ones in use by the weather dept. The weather Dept is only good for about 3 days ahead. If our current setups are good for just days in advance why do they think that their models are accurate over thousands of years?

Re:Exactly (1)

Reality Master 101 (179095) | more than 4 years ago | (#31465204)

you want just plain silly the computer models they use are different from the ones in use by the weather dept. The weather Dept is only good for about 3 days ahead. If our current setups are good for just days in advance why do they think that their models are accurate over thousands of years?

Playing devil's advocate...

Apples and oranges. Predicting local weather 3 days in advance is like throwing a coin and predicting whether it's going to land heads up or tails up by calculating the air stress on the coin, factoring in the landing zone, and all other factors. That's difficult and inaccurate. Predicting weather a thousand years from now is like throwing a million coins and predicting how many heads, how many tails, (roughly 50/50) and what shape the pile is going to be (roughly a circular mound). That's easy and predictable.

Of course, climates are much more complex than a pile of coins, so even the thousand year model is undeterministically complex.

Re:Exactly (0)

Anonymous Coward | more than 4 years ago | (#31465254)

Think about this: even thousands of years ago primitive humans figured out that in certain places they had a winter and a summer every year.

Re:Exactly (1)

The End Of Days (1243248) | more than 4 years ago | (#31465256)

Denier! How dare you question the truth.

And off topic, at that. You, sir, disgust me. Our lord Al Gore will consume your soul.

There is only so much you can do.. (0)

Anonymous Coward | more than 4 years ago | (#31464938)

Well you can always assume that your real time code is buggy and write checks to ensure exact conditions before a critical assignment or operation. But given the limited amount of CPU there arent too many checks you can build into firmware.

Obama's solution (1)

ebonum (830686) | more than 4 years ago | (#31464976)

It seems he wants more software. This time to check for both pedals being depressed at the same time. One more thing to break.

The less these embedded systems have to do, the better.

Re:Obama's solution (1)

KarmaMB84 (743001) | more than 4 years ago | (#31465206)

Toyota is already doing this. It's rather odd that the computer will willingly burn the brakes completely off a vehicle that has both pedals down to start with.

Also a witch (0)

Anonymous Coward | more than 4 years ago | (#31464984)

Toyota may also have witches on staff, which may cause the unintended acceleration. If Toyota has indeed searched its personnel as it says without finding any witches, my response is simple: Keep trying. Find new ways to find witches, and come up with more creative tests, possibly involving a duck and a scale. Until these witches are identified, how can you be certain they are not related to sudden acceleration?

Failsafe software (0)

Anonymous Coward | more than 4 years ago | (#31465000)

All practically sized software has bugs, but that doesn't mean it has to be dangerous. Mechanical devices fail too and therefore the engineers add failsafe-mechanisms. If something breaks, something else must absorb the problem. Consistency checks and redundancy are indispensable even when the code is perfect, because hardware is never infallible.

Smaller Systems Solution? (1)

Manip (656104) | more than 4 years ago | (#31465016)

Didn't we already "solve" this problem in the airline industry by breaking down larger more complex software into separate physical components that can be more easily verified? Can we do the same with cars?

Just split one computer into half a dozen much smaller, simpler, little units and set the valid IO conditions for them, then have the components around them sanity check their output.

Are the brakes totally drive-by-wire as well? (1)

onionman (975962) | more than 4 years ago | (#31465024)

Pardon my laziness for not investigating this myself, but doesn't the Prius have a mechanical (hydraulic) link to the brakes that engages when the pedal is pushed down far enough? I realize that the first portion of the braking is done electronically (for the regenerative braking system), but in an emergency wouldn't a full application of the brakes slow down the vehicle?

Re:Are the brakes totally drive-by-wire as well? (0)

KarmaMB84 (743001) | more than 4 years ago | (#31465222)

It may temporarily prevent the vehicle from reaching it's top speed but I doubt there's an automobile on the planet that is designed to have the brakes applied while the gas is at full throttle without just burning the brakes down to the bare metal then continuing to accelerate.

From the days of "winmodems" (2, Interesting)

A_Non_Moose (413034) | more than 4 years ago | (#31465026)

I've said time and time again, "Never replace hardware with software" because
something dedicated to the task will always work better, or be less failure
prone (more often than not).

Would Toyota be having these problems with an accelerator cable vs electronic?

99% sure the answer is "no"...heck the solution is add some grease, make sure
it isn't pinched/looped too tightly and/or add tension to the pedal side.

Or, replace the damn cable with a new one...a 20 to 30 minute task.
(less than 10min on a motorcycle)

Oh, well, what do I know? I'm just a CS major with real world experience, pay
no attention to the man behind the keyboard!!!

Re:From the days of "winmodems" (1)

goldaryn (834427) | more than 4 years ago | (#31465228)

"Never replace hardware with software"

That's what she said

heck the solution is add some grease

That's what I said

God damn legal system (1, Insightful)

oldhack (1037484) | more than 4 years ago | (#31465060)

On a side note, our legal system forces us all to lie.

Re:God damn legal system (1)

maxume (22995) | more than 4 years ago | (#31465156)

Liar.

Re:God damn legal system (1)

oldhack (1037484) | more than 4 years ago | (#31465212)

Nuh-uh.

Metric unit (1)

Sepiraph (1162995) | more than 4 years ago | (#31465086)

Did Toyota forget to convert imperial to metric unit like those Mars Pathfinder?

prove a negative? (2, Insightful)

sjs132 (631745) | more than 4 years ago | (#31465096)

Isn't this like proving God doesn't exist?

They can test and test and not get a result that said this is the bug, so they assume that it doesn't exist.

Embedded Bugs can be a BITCH to find (1, Interesting)

Anonymous Coward | more than 4 years ago | (#31465116)

So I work on a robotics team on all of the firmware which interfaces the main computer with the actuators and sensors it needs to move. It drove me crazy because I had a bug that I couldn't replicate outside the robot.

If you were driving the robot around under normal operating circumstances, suddenly, the motor driver board would stop working. I had a simple state machine on the motor board which would update the motors one by one, but I had a bug which I hadn't noticed because it seemed innocuous at the time:

I updated the state machine immediately *after* I initiated an action which would eventually start interrupt code. Now, what ended up happening was that message about new speeds coming from the main board were also interrupts, and so there was this very odd cascading failure, where the motor board would initiate the action, the message from the main board would come in, we would jump into the message receive interrupt, and then immediately we would jump into the motor control routine. Since the state machine hadn't been updated yet, we would then execute the wrong part of the state machine, which would cause an error. The motor control routine would initiate a "stop" command and then go into an error state. Because this was a time-critical operation, I didn't care if we had failed to send a single packet, so I ignored the error, and jumped to the next motor. Unfortunately, the "stop" command would finish right as I tried to reset the state machine, which would put it in a different error state. And then when I tried to communicate with the next motor, I would do the same thing, over, and over, and over.

The whole system would lock up, but by itself, all the code was completely reasonable. If you took that segment of code out of the system to verify that it worked, it would work 100% of the time. Put it into the system, and it generally took something like 10 to 20 minutes to suddenly die.

I wonder if (1)

thephydes (727739) | more than 4 years ago | (#31465130)

the acceleration was caused by some external EMI event? Why?. It is well known amongst ham radio operators that some cars do not have sufficient EMI protection so that an otherwise correctly installed transceiver will interfere with the car electronics when used. Pure speculation I know......

Mechanical or Siftware bug (1)

hufter (542690) | more than 4 years ago | (#31465132)

This is what I've seen in the news: Toyota says there is a mechanical fault in the accelerator pedal, that may cause it to get physically stuck down. This is extremely rare, but possible. To fix it, they add a little extra piece to the pedal. Sound like a duct-tape solution, but if it works, then okay.
Did Toyota lie to us, and actually added a patch to the software instead?

fault without fault warning in computer? (1)

cenobyte40k (831687) | more than 4 years ago | (#31465148)

Is it just me or did everyone else miss the one where a guy introduced a fault into the Toyota system that caused sudden acceleration without producing an error code? I admit he had to introduce the fault artificially but if you can do that and have no fault come up on the computer how can you say that your system works 100% of the time?

Sol'n: fly by half-wire (1)

robert bitchin' (765408) | more than 4 years ago | (#31465150)

Most throttle systems I've worked with employed two cables to control the throttle in a push-pull arrangement. If setup in a single-wire arrangement then only one wire is used to pull the throttle open and a spring is used to ensure it closes when the tension is removed. In some motorcycles where the operators have a literal grip on the throttle then, if the spring is not strong enough to close the throttle (or one of the cables jams), the operator can always physically force its return by turning the throttle back manually. Car operators using pedal-based accelerators, unless they are barefoot, cannot wrap their toes around the pedal to pull it back into position.

My solution blends the two strategies: retain the pedal position sensor that captures the operators intention while retaining use of a 'return' cable. The ECU can remain in control of advancing the throttle to retain their interest in fuel economy measures while removing its control over retarding it. Pushing the pedal down merely provides enough physical free play for the ECU to work with while retaining any veto control if it gets out of hand.

The final part would encourage the adoption of accelerator pedals that users slip their feet into rather than just on top of. This would provide the same ability to positively influence the pedal return rather than expect the spring to do so.

both feet (0)

Anonymous Coward | more than 4 years ago | (#31465210)

My mother drives using both feet, as in one foot on the brake and the other on the gas. She has always driven this way. Does anyone know if this is common or unsafe?

Pedal was stuck physically to the floor. (1)

andydread (758754) | more than 4 years ago | (#31465236)

If you look at the latest 2 incidents the accelerator pedal was stuck to the floor. The one man even said he felt something funny has he pressed the pedal down to pass a car and it was stuck to the floor. It did not return. He said he even tried to pull it back to its original position. So how in the world it this an electronic problem? How does electronics cause a mechanical pedal to be stuck to the floor? Someone? Anyone?
Load More Comments
Slashdot Login

Need an Account?

Forgot your password?