Beta
×

Welcome to the Slashdot Beta site -- learn more here. Use the link in the footer or click here to return to the Classic version of Slashdot.

Thank you!

Before you choose to head back to the Classic look of the site, we'd appreciate it if you share your thoughts on the Beta; your feedback is what drives our ongoing development.

Beta is different and we value you taking the time to try it out. Please take a look at the changes we've made in Beta and  learn more about it. Thanks for reading, and for making the site better!

Why Software Builds Fail

Soulskill posted about 3 months ago | from the failure-to-bribe-the-hamster dept.

Bug 279

itwbennett writes: A group of researchers from Google, the Hong Kong University of Science and Technology and the University of Nebraska undertook a study of over 26 million builds by 18,000 Google engineers from November 2012 through July 2013 to better understand what causes software builds to fail and, by extension, to improve developer productivity. And, while Google isn't representative of every developer everywhere, there are a few findings that stand out: Build frequency and developer (in)experience don't affect failure rates, most build errors are dependency-related, and C++ generates more build errors than Java (but they're easier to fix).

cancel ×

279 comments

Sorry! There are no comments related to the filter you selected.

Because I'm lazy (5, Informative)

OzPeter (195038) | about 3 months ago | (#47317117)

Half the time when I'm working on any sort of non-trivial program (that is too large to hold in my head all at once) and I need to make a breaking code change (and one that is not easily managed with refactoring tools), I'll make the change where it is obvious to me and then let the compiler tell me where it broke and hence where I need to make my fixes.

Re:Because I'm lazy (2)

Z00L00K (682162) | about 3 months ago | (#47317239)

The most important thing is not to avoid that the build fails but to avoid distributing software packages that can't be built.

However if something can't be built due to a mistake it's often easy to find and correct. The big problems are often not that visible and it can take a while to figure them out.

What really grinds my gears is that people release source code that is possible to build, but so full of compiler warnings that you can't be certain that it's going to work as intended.

Re:Because I'm lazy (3, Interesting)

mlts (1038732) | about 3 months ago | (#47317381)

When in CS, I had a prof that had one rule that for release (not beta/alpha/dev) code, if the code had even a single warning, it was unshippable unless there was an extremely good reason (which would be part of the README) of why it happened. Yes, this was a PITA, but he was trying to teach something that seems to have been lost.

Re:Because I'm lazy (5, Funny)

0123456 (636235) | about 3 months ago | (#47317481)

He's the reason compiler writes invented pragmas to turn off warnings...

Re:Because I'm lazy (0)

Anonymous Coward | about 3 months ago | (#47318027)

So what you are saying is that you are better than the thousands of folks working on gcc?
If your compiler issues a warning; chances are you should fix it as opposed to suppressing it.

Re:Because I'm lazy (5, Insightful)

EvanED (569694) | about 3 months ago | (#47318345)

I'm a fan of warnings as much as the next guy, but there are plenty of times that's not practical. Even if you accept nothing else I say, there are plenty of times where third party code (say, for example, Boost...) has warning-producing stuff in it. Do you fix it and maintain your own branch? Submit it upstream, hope it gets accepted, then wait a month for the new release, then demand everyone upgrade to the latest bleeding-edge? That's often (maybe usually) not feasible, which means it should probably just disable it. Fortunately, GCC finally got around to adding #pragmas that sometimes let you disable and re-enable warnings for just the offending headers.

But beyond that, there's also a reason that compilers have most warnings off by default. And why -Wall doesn't turn on anything close to all warnings. And why even -Wall -Wextra isn't all warnings. Because there's a gradation of false positive/false negative tradeoffs, and what's appropriate for you isn't appropriate for everyone. Do you really compile with all warnings, or do you suppress some? Because I can almost guarantee it's the latter, even if you're suppressing them through inaction.

Re:Because I'm lazy (1)

geekoid (135745) | about 3 months ago | (#47318167)

And turning off compiler pragmas are why software is such an amateur engineering field.

Re:Because I'm lazy (1)

UnknownSoldier (67820) | about 3 months ago | (#47318327)

Why do you assume all warnings are useful?? Some of the compiler warnings are just pedantic and are "noise" such as "variable declared but not used", etc.

There is a balance between no warnings and pedantic warnings, namely the useful ones.

Re:Because I'm lazy (2)

UnknownSoldier (67820) | about 3 months ago | (#47318361)

Generally I recommend leaving most warnings on. But sometimes compiler writers go completely over board.

When you use MSVC you have to do stupid stuff like this

      #define _CRT_SECURE_NO_WARNINGS // WIN32:MSVC disable warning C4996: This function or variable may be unsafe.

The following compiler specific header suffices to compile code using without warnings, at highest warning level.

#pragma warning( disable: 4061 ) // enum value is not *explicitly* handled in switch
#pragma warning( disable: 4099 ) // first seen using 'struct' now seen using 'class'
#pragma warning( disable: 4127 ) // conditional expression is constant
#pragma warning( disable: 4217 ) // member template isn't copy constructor
#pragma warning( disable: 4250 ) // inherits (implements) some member via dominance
#pragma warning( disable: 4251 ) // needs to have dll-interface to be used by clients
#pragma warning( disable: 4275 ) // exported class derived from non-exported class
#pragma warning( disable: 4347 ) // "behavior change", function called instead of template
#pragma warning( disable: 4355 ) // "'this': used in member initializer list
#pragma warning( disable: 4505 ) // unreferenced function has been removed
#pragma warning( disable: 4510 ) // default constructor could not be generated
#pragma warning( disable: 4511 ) // copy constructor could not be generated
#pragma warning( disable: 4512 ) // assignment operator could not be generated
#pragma warning( disable: 4513 ) // destructor could not be generated
#pragma warning( disable: 4610 ) // can never be instantiated user defined constructor required
#pragma warning( disable: 4623 ) // default constructor could not be generated
#pragma warning( disable: 4624 ) // destructor could not be generated
#pragma warning( disable: 4625 ) // copy constructor could not be generated
#pragma warning( disable: 4626 ) // assignment operator could not be generated
#pragma warning( disable: 4640 ) // a local static object is not thread-safe
#pragma warning( disable: 4661 ) // a member of the template class is not defined.
#pragma warning( disable: 4670 ) // a base class of an exception class is inaccessible for catch
#pragma warning( disable: 4672 ) // a base class of an exception class is ambiguous for catch
#pragma warning( disable: 4673 ) // a base class of an exception class is inaccessible for catch
#pragma warning( disable: 4675 ) // resolved overload was found by argument-dependent lookup
#pragma warning( disable: 4702 ) // unreachable code, e.g. in header.
#pragma warning( disable: 4710 ) // call was not inlined
#pragma warning( disable: 4711 ) // call was inlined
#pragma warning( disable: 4820 ) // some padding was added
#pragma warning( disable: 4917 ) // a GUID can only be associated with a class, interface or namespace

Reference:

* http://alfps.wordpress.com/201... [wordpress.com]

Re:Because I'm lazy (2)

turgid (580780) | about 3 months ago | (#47317559)

When in CS, I had a prof that had one rule that for release (not beta/alpha/dev) code, if the code had even a single warning, it was unshippable unless there was an extremely good reason (which would be part of the README) of why it happened. Yes, this was a PITA, but he was trying to teach something that seems to have been lost.

You should be compiling with warnings as errors as soon as you start coding, and you should fix each one as they occur before you move on to write the next line of code.

Putting off fixing these problems leads to bloated and fragile code and wastes much more time debugging and fixing later.

Re:Because I'm lazy (3, Insightful)

RabidReindeer (2625839) | about 3 months ago | (#47318353)

You should be compiling with warnings as errors as soon as you start coding, and you should fix each one as they occur before you move on to write the next line of code.

Putting off fixing these problems leads to bloated and fragile code and wastes much more time debugging and fixing later.

What you should be doing outside the CS class and in the so-called "Real World" is "being productive". That usually means screw the warnings, it has to be completed ASAP or we'll find someone "more productive" than you are.

Re:Because I'm lazy (1)

Obfuscant (592200) | about 3 months ago | (#47317955)

Yes, this was a PITA, but he was trying to teach something that seems to have been lost.

Yes, he was teaching you to turn off warnings for certain operations so that when the warning was really significant it wouldn't happen.

Maybe if professor think your code is functional but not elegant maybe you should suggest professor write login page himself? Be crazy AND proud.

There is a reason why they are warnings and not fatal errors.

Re:Because I'm lazy (1)

EvanED (569694) | about 3 months ago | (#47318031)

Yes, he was teaching you to turn off warnings for certain operations so that when the warning was really significant it wouldn't happen.

Is that better or worse than teaching someone to leave warnings on for those operations and then not noticing when a really significant one appears because it's buried in an avalanche of other output?

Re:Because I'm lazy (1)

geekoid (135745) | about 3 months ago | (#47318205)

And it assume you can see the future and predict what will be really serious.
There are many instances of failure becasue it didn't seem like a big deal at the time.

Re:Because I'm lazy (1)

Obfuscant (592200) | about 3 months ago | (#47318265)

Is that better or worse than teaching someone to leave warnings on for those operations and then not noticing when a really significant one appears because it's buried in an avalanche of other output?

Worse, because when the significant one appears the code will be non-functional and there will be no compiler warning to help figure out why. It is harder to remember to go back and change the makefile (or other build script) to turn the warnings back on and then see a large number of irrelevant warnings appear (which you may think are relevant and waste time fixing without actually fixing the broken code). If you inherited the build from someone else (not a good idea in a programming class necessarily) you might not KNOW you need to go turn the warnings back on, you might just have a broken program and no warnings to tell you where to look.

By the way, I define "significant warning" to mean "actually reports something that is going to make the program malfunction". For example, "incompatible declaration of function printf" when I forget to include stdio.h is not a significant warning because I don't care what printf returns and nothing depends on it. Should I decide to turn that one off, and then do something that actually does depend on the return value and it doesn't work, it becomes a significant warning and I want to see it. I'm reasonably competent, so I can recognize a warning that refers to a line of code I've just changed, even if there are other warnings.

I get to deal with lots of code that has lots of warnings. I can handle it.

Re:Because I'm lazy (4, Insightful)

geekoid (135745) | about 3 months ago | (#47318189)

NO, he was teaching engineering practices, and a good one.

People like you is why software is in such a terrible state as an industry.

Re:Because I'm lazy (1)

Z00L00K (682162) | about 3 months ago | (#47318305)

It's not a question of turning off warnings, it's a question of correcting the code to get rid of the warnings.

If you turn off the warnings you turn off the warnings for all occurrences in the code, and that is really a dangerous thing to do. In most cases warnings are harmless but in some cases the warning is an important explanation to why something behaves in an erratic way.

Ignoring compiler warnings is stupid, dangerous and can cause serious problems to become hidden.

Re:Because I'm lazy (2)

ShanghaiBill (739463) | about 3 months ago | (#47317735)

What really grinds my gears is that people release source code that is possible to build, but so full of compiler warnings that you can't be certain that it's going to work as intended.

Where I work, all builds are run with -Wall -Wextra -Werror. So if you check in code that produces a warning, the compiler turns it into an error, and you broke the build. Which means you get to be the build babysitter until someone else breaks it.

Re:Because I'm lazy (0)

geekoid (135745) | about 3 months ago | (#47318225)

"..build babysitter until someone else breaks it."
Your QA/Build process is seriously broken if you don't have one person or one team in charge of the build at all times.

Re:Because I'm lazy (1)

Z00L00K (682162) | about 3 months ago | (#47318341)

There are even a few warnings that at least with gcc won't show up unless you use the plain "-W" flag, and even cases where they won't show up at the "-O0" level but only at "-O2". And there are a few that you have to enable explicitly.

Add a run of "splint" and/or cppcheck to make sure that the code is as good as it can be. Then execute the binary under Valgrind to make sure that there are no memory leaks. The remaining errors should be those caused by a bad system design rather than plain coding errors. In a few cases the errors might be in third party runtime libraries, but that's hard to protect against and outside the scope here.

Re:Because I'm lazy (0)

Anonymous Coward | about 3 months ago | (#47317339)

Half the time when I'm working on any sort of non-trivial program (that is too large to hold in my head all at once) and I need to make a breaking code change (and one that is not easily managed with refactoring tools), I'll make the change where it is obvious to me and then let the compiler tell me where it broke and hence where I need to make my fixes.

That works until you throw in Spring or JSF or somebody's custom reflection code. Then there are developers who insist on passing everything around as strings or untyped arrays. Or XML. I've tagged other developers' code for that and been told, "That's what testers are for!" Ugh.

It's usually a computer problem (1)

Russ1642 (1087959) | about 3 months ago | (#47317121)

We've found that builds fail for three reasons: coding errors, dependency issues, command line argument mistakes. If you're a developer you should check these three things when your builds fail and you'll likely find the issue.

Re:It's usually a computer problem (4, Insightful)

matthiasvegh (1800634) | about 3 months ago | (#47317183)

Oh coding error? Well thats helpful. Misplace a semicolon in a non-trivial meta-program or dsl in C++, and just watch the errors that the compiler spits back at you. None of which, will have anything to do with semicolons. I suppose this is why the C++ errors are considered to be easy to fix. Mistype a word, and you get 15000 lines of errors. I suppose it's easy to fix all those errors too. Yes, but figuring out what exactly the coding error was is kind of the point.

Re:It's usually a computer problem (4, Insightful)

aliensexfiend (656910) | about 3 months ago | (#47317447)

When I see the error avalanche the first place I check are the first few error messages and that is usually enough to spot the problem. Typos still make c++ compilers barf way too much crap.

Re:It's usually a computer problem (5, Insightful)

Anonymous Coward | about 3 months ago | (#47317505)

Please, give up the C++ slander.

Like any compiler output, read the first error. If you are a developer of any calibre, having a few pages of errors shouldn't phase you and it's not unique to C++ to generate a few erroneous errors. All it requires is a basic level of competence and if you don't possess that then any programming
language that facilitates you generating anything that compiles is doing noone any favours.

Re:It's usually a computer problem (0)

Anonymous Coward | about 3 months ago | (#47318055)

while I agree with almost everything you say; I have reservations about the following claim

it's not unique to C++ to generate a few erroneous errors

Re:It's usually a computer problem (0, Troll)

frank_adrian314159 (469671) | about 3 months ago | (#47318093)

Please, give up the C++ slander.

Why? It deserves it. And I speak as one who has been using it since cfront 1.0 and used it for 15 years or so professionally. Basically, its two reasons to exist - its performance and its small (-ish) runtime library are dwindling in importance. You can now get as good performance writing C primitives in a decent high-level languages. And RT minimalism has only been an issue in the embedded world for the last 20 years or so.

Yes, it was once the mainstay of the engineering software world. Let it go and let it drift away to its end as the COBOL of the embedded world.

Re:It's usually a computer problem (0, Troll)

geekoid (135745) | about 3 months ago | (#47318253)

c++ is great. I'm sorry you're not very good and never mastered it, but that isn't the languages problem.

Re:It's usually a computer problem (0)

Anonymous Coward | about 3 months ago | (#47317811)

GCC is better then LLVM/Clang...

Re:It's usually a computer problem (1)

geekoid (135745) | about 3 months ago | (#47318251)

what? You can't tell a missing semi colon from the error messages? or a misspelled word? you're not very good, are you?

Easy fix... Offshore it (-1)

Anonymous Coward | about 3 months ago | (#47317171)

Easy fix. Let the pros at Tata or Infosys handle it offshore or get a H-1B to do the work locally.

They actually are contracted and work is guaranteed.

Re:Easy fix... Offshore it (0, Funny)

Anonymous Coward | about 3 months ago | (#47317289)

Deng Zhang commit untested code to repository again. Now compiler error, build box idle, that not good. I need you write script recompile code until it work. You have five minutes finish job now before you laid off.

I don't get it.... (0)

Anonymous Coward | about 3 months ago | (#47317175)

How is C++ / Java build error comparison related to the actual study in any way?

But then again.... I'm dumb.

LaTeX never fails (2)

Extremus (1043274) | about 3 months ago | (#47317181)

My LaTeX builds rarely fail in MiKTeX. The compiler itself seems to be able to download packages and classes from a common repository (CTAN and its many mirrors).

Re:LaTeX never fails (0)

Anonymous Coward | about 3 months ago | (#47318097)

Wait until you run into the hilarious error 87. A compiler error so insidious no one even knows what causes it for sure.

Here's a concept to prevent this crap - UNIT TESTS (2)

SirGeek (120712) | about 3 months ago | (#47317189)

If GOOD/Complete unit tests for code exist and this change would break it, How freaking tough is it to run the unit test before committing your change to source code control ?

Re:Here's a concept to prevent this crap - UNIT TE (1)

Anonymous Coward | about 3 months ago | (#47317241)

How does a unit test prevent a build error?

Re:Here's a concept to prevent this crap - UNIT TE (1)

turgid (580780) | about 3 months ago | (#47317627)

How does a unit test prevent a build error?

Because you have to build your code to run the unit test. The two are (should be) intimately linked. And you should always be building against the latest official working source from the upstream repo.

Re:Here's a concept to prevent this crap - UNIT TE (2)

EvanED (569694) | about 3 months ago | (#47317769)

And of course everyone always builds with the same configuration, same compiler, on the same platform.

(We have CI servers in our environment. They break not infrequently. Why? Because someone commits a change that builds fine on Linux, and when MSVC gets ahold of it, it produces a warning that GCC doesn't catch and so the build fails. Or MSVC accepts some piece of code that is not actually legal C++ because it's too loose, so when the Linux buildbots get ahold of it, they complain.)

Re:Here's a concept to prevent this crap - UNIT TE (1)

turgid (580780) | about 3 months ago | (#47317911)

And of course everyone always builds with the same configuration, same compiler, on the same platform.

Yes, they should, and it should be scripted so that it's trivial so that there's no excuse for not doing it every single time.

Multiple compilers, multiple OSes and multiple binary architectures should all be used and they should all be available to every developer on the network. There should be enough network, storage and CPU capacity to make the builds quick so that there is no excuse for not doing them.

Re:Here's a concept to prevent this crap - UNIT TE (2, Insightful)

Anonymous Coward | about 3 months ago | (#47317823)

Why not just say that you should always build against the latest official working source before checkin? It has nothing to do with unit testing.

Re:Here's a concept to prevent this crap - UNIT TE (0)

Anonymous Coward | about 3 months ago | (#47318071)

From the study:
Developers may build with the IDE’s own compiler during development, but must ensure the code successfully builds with the centralized build system before committing.

They are building before they commit and that is what is being measured.

Re:Here's a concept to prevent this crap - UNIT TE (1)

turgid (580780) | about 3 months ago | (#47318127)

Why not just say that you should always build against the latest official working source before checkin? It has nothing to do with unit testing.

It does. You shouldn't be putting in new logic bugs with your deliveries. The code should compile cleanly and you should do due diligence to avoid putting in new bugs.

Bugs are expensive to fix, in terms of debugging time and refactoring of broken code to change additional changes built on top of the broken code.

It's amazing how much real progress you can make if you follow these simple rules.

Re:Here's a concept to prevent this crap - UNIT TE (0)

Anonymous Coward | about 3 months ago | (#47317995)

I don't think you understand the study or c++ or unit testing.
You need to build to run a unit test, and that would have counted as a build error. In fact, with this study unit tests would have increased the build failures.

Re:Here's a concept to prevent this crap - UNIT TE (1)

turgid (580780) | about 3 months ago | (#47318191)

I don't think you understand the study

I didn't RTFA because I could tell from the summary that it was a lot of nonsense.

or c++

I understand enough about C++ to know to avoid it wherever possible. I freely admit that my brain isn't big enough to fight against C++ with enough perseverance to get it to do anything useful correctly. I'll stick to C and scripting languages thank you very much.

or unit testing.

*cough* Test Driven Development *cough*

To become half way good at C++ you either need to devote you entire life to it over decades (somewhat like a monk) or have an IQ of at least 200. Life's too short and my IQ is miles too low.

Re:Here's a concept to prevent this crap - UNIT TE (1)

Anonymous Coward | about 3 months ago | (#47317287)

It's hard to run a unit test on a program that doesn't build.

Re:Here's a concept to prevent this crap - UNIT TE (0)

Anonymous Coward | about 3 months ago | (#47317471)

Unit testing! The silver bullet for everything.

If your parents had unit testing you never would've been born.

Re:Here's a concept to prevent this crap - UNIT TE (2)

OakDragon (885217) | about 3 months ago | (#47317713)

If your parents had unit testing you never would've been born.

I would still have been born, just not so buggy.

Re:Here's a concept to prevent this crap - UNIT TE (0)

Anonymous Coward | about 3 months ago | (#47318195)

It is fair to say there might have been a "unit" being tested and that is why he was conceived.

Re:Here's a concept to prevent this crap - UNIT TE (1)

Anonymous Coward | about 3 months ago | (#47317529)

TFA seems to be referring to local builds. I HOPE nobody is watching my local builds, because it would be embarrassing at times.

That said, any developer who commits broken code to source control should be flogged. Unit tests help a lot, but it's rare that anybody wants to pay us to write unit tests.

Re:Here's a concept to prevent this crap - UNIT TE (0)

Anonymous Coward | about 3 months ago | (#47317845)

Your argument is a tautology: If I can detect the problem, then I should be able to detect the problem. Please say something of value instead.

Detecting missing dependencies (0)

Anonymous Coward | about 3 months ago | (#47317201)

most build errors are dependency-related

If *only* there was a way to tell which dependencies were missing before I start an hour-long build process... perhaps a ./configure script, or a README file...

But more seriously, I'm currently trying to compile various source packages for MinGW and/or MSYS. MinGW is missing various unix-based headers (no surprise), while MSYS is missing ANSI-C headers (including stdint.h). They also have a habit of breaking well into the compilation process, even after ./configure seems to indicate that everything is okay. While trivially fixed, it's something that needs to be fixed in the source repository rather than on a per install basis.

I'm considering forking MinGW/MSYS on that alone.

Re:Detecting missing dependencies (0)

Anonymous Coward | about 3 months ago | (#47317411)

Try Cygwin.

Re:Detecting missing dependencies (0)

Anonymous Coward | about 3 months ago | (#47317581)

Cygwin is licensed under the GPL, which makes it impossible to do distribute anything other than GPL software if compiled using Cygwin. (e.g. any attempt at commcercially-sold software, needed if you want food in your belly if you can't be otherwise sponsored.)

Also, Cygwin is yet again different from MinGW/MSYS, in that it's more strictly designed for Unix-like systems. If you want to be able to use windows, you need to install X11 (not installed by default or in core), or manually import Windows header files.

And switching to Cygwin doesn't fix the issue that MinGW/MSYS is still broken.

Re:Detecting missing dependencies (0)

Anonymous Coward | about 3 months ago | (#47317653)

(e.g. any attempt at commcercially-sold software, needed if you want food in your belly if you can't be otherwise sponsored.)

No worries, brah, Obamacare will fix everything! You can live to program and pay for your food with welfare checks, like Stallman intended.

Re:Detecting missing dependencies (1)

Anonymous Coward | about 3 months ago | (#47317809)

http://scp-wiki.wikidot.com/scp-504

Simple answer (1)

slapout (93640) | about 3 months ago | (#47317209)

Complexity

Surprise, surprise (0)

Anonymous Coward | about 3 months ago | (#47317211)

Strongly typed languages catch more errors during compilation.

Dependencies? (3, Insightful)

Anonymous Coward | about 3 months ago | (#47317249)

Dependencies just magnify all other problems. If your code depends on nothing then it won't break unless the compiler changes. Unfortunately such programs don't exist because you can never depend on nothing and do anything useful. In reality if you depended on nothing you'd end up writing your own console, your own I/O, pretty much your own CRT. This sounds great until you realize your dependency is now the hardware itself and it's likely your code won't be portable in any useful sense. That's why we have kernels.

The problem with C++ is that dependency management is usually file-level and developers 'rarely' care about any file-level constructs (and nor should they, it's an abstract packaging concept). As a result you try to drag in one enum and end up with 100 #includes and 500 new classes you don't care about. This causes bigger object files to be emitted, vastly slower linkage and lots of dependencies you don't expect. All it takes now is for one of those includes to #define something unexpected and BOOM...the house of cards comes crashing down.

Also, did I mention? The C preprocessor causes a lot of grief when it's abused.

Re:Dependencies? (0)

Anonymous Coward | about 3 months ago | (#47317649)

A huge contributor to this problem is that for a long time including header files in header files was considered acceptable practice. I personally didn't change this practice until I read about how to do it better in the latest version of Effective C++

Re:Dependencies? (1)

EvanED (569694) | about 3 months ago | (#47317867)

Errr... what?

Care to give a quick summary of what the alternative is?

Dependencies? (0)

Anonymous Coward | about 3 months ago | (#47317835)

At any moment you can do:
1) Take the enum definition out of its source file
2) Add an include in the original, including the enum.
3) Include just the enum in your code, do not include the whole mess.

This would be easier if you would tell the compiler

#include

Of course if those symbols weren't in the include an error would occur, what would ensure that those symbols exist AFTER the include.

Questionable (0)

Anonymous Coward | about 3 months ago | (#47317297)

I agree it's usually about dependencies, and also with their C++/Java conclusions. However, I find it hard to accept that there is no correlation between developer experience and build failures.

Maybe it's because even experienced devs often suck completely at proper build structure and dependency checks.... I certainly get that where I work - the "experienced" guys are just as likely to break builds, but some of the "experienced" guys are also fucking useless and counter-productive and are only considered "experienced" because of how many years they've been writing useless code.

I'm the "build guy" at my shop, because few are OCD enough to do it properly.

Dependencies Problems = "It builds on my machine" (5, Insightful)

Virtucon (127420) | about 3 months ago | (#47317301)

Once code is checked in and goes through the standard build process, that's where this is expected to occur because in my experience it's the local environment where the developer does the coding that's the root problem. Why? Developers don't refresh their build environment because of the potential for other problems it may create. I had one gig to unfuck some code at a company a couple of years ago and found out that in order to set up a Dev environment in this place could take two weeks or more depending on what team you were on. You had to go through a script, download this, install that, change this.. A nightmare. Updating dependencies on a local desktop created panics amongst the developers who were reluctant to ever change anything they had which "was working" because you could spend days trying to fix what was broken. Naturally any time they migrated code into test or production (there was no build system) things failed there because of dependency related issues. Also depending on who the developer was, they naturally felt that bypassing the Test/QA cycle was a job perk.

I found dozens of dependencies on desktops that were out of date, deprecated or had major vulnerabilities and that went for the production systems as well. It was bad all the way around from a best practices perspective. Daily production crashes were the norm, the VP of Dev had a monitor on his desk so he could "troubleshoot" production problems it was that bad.

Yes there's shops like this that are still out there.

Re:Dependencies Problems = "It builds on my machin (2)

gstoddart (321705) | about 3 months ago | (#47317455)

I had an experience which was somewhat opposite (though, in a lot of ways pretty much the same).

At one point, the company went with a big giant universal build system.

Every piece of software, every module, every final build ... was recompiled from scratch on a nightly basis. It took a massive server farm many hours to do this. Even if no changes had been made.

What would happen would be someone would break a component. The build of that component, and every downstream dependency broke. The system had no concept of "this is a beta build, not for everybody" and "this is a release, and stable".

The result was that sometimes you'd have literally dozens of things which were now suddenly broken. It was too stupid of a build system to use the last known good.

So, all of a sudden you get one trivial change in some module about 4 steps removed from your stuff. But, it was all broken, and your stuff couldn't be properly built until someone fixed their stuff, and the build system went through at least one more cycle, often two.

Sometimes, companies get themselves into such a borked state with their build system (or lack thereof) that it makes doing any work impossible.

Some of us started keeping our own local copies, and writing local build scripts, because we couldn't rely on the company wide one to actually work much of the time.

Re:Dependencies Problems = "It builds on my machin (1)

Sowelu (713889) | about 3 months ago | (#47317475)

This, this, this. Dependency issues on a build server are so often things like "we added this dependency on a library locally, but didn't update the build server", or even "we checked in a change that makes it work on the dev box, but didn't make a change needed for the server". That plus forgetting to merge in stuff is really the only way I ever see build breaks (where checked-in code fails to compile, as opposed to compiles failing locally).

Re:Dependencies Problems = "It builds on my machin (1)

Anonymous Coward | about 3 months ago | (#47317493)

And I've worked in many of them. I call this effect "build pollution" and a common cause is simply not checking in files to SVN that were added.

The #includes are referencing something on the local HDD that no other developer would receive if they resync the repo. End result: everyone else is now broken and you have to go poke the dev with a stick. The other problem is that far too few places have a presubmit test that can check if your code will build by the auto-builder *BEFORE* you submit your code. This means you have to break the build to see if the build would break which is bad practice and doesn't scale well to very large teams.

The other problem is that many devs don't practice basic HDD cleanliness as you describe. When the instructions for building your code reach a certain size people stop repeating them because they're too complicated. This means bad builds and local tools can be polluting the local build environment in ways that will affect future builds (which usually results in panic, flapping and comedy). Telling people to tear down their machines to clean state won't work if the process to restore to build state is too complex - they'll ignore you and this circus just keeps coming back into town. This also ties in to build reproducibility: if there is no automation to set up the build state (a preconfigured clean HDD image or VM image) then it becomes really hard to know what the build dependencies are. People forget they installed "crapware bag-o-bolts toolkit" and forget to tell everyone else. The IT department won't know. Possibly the build engineer will know because they set up the build machine nodes but then a new dev joins with a fresh OS and BANG. They should probably ask the auto-build build engineer what's on that build box, but most new devs wouldn't know this detail to ask the right person. :)

You need to know what's on your machine at all times, prune out, clean out, keep the build machine squeaky clean and pristine. Everything that goes onto the build box should be audited and maintained in fresh-boot-state ideally. A VM is extremely helpful in doing this but creating Windows VMs with the appropriate packages can be a painful manual clickfest process. In Linux this practice is rendered trivial and it's par for the course. ...and then there's that time I found build files included via network file share from a random guy's desktop. NOPE.

Re:Dependencies Problems = "It builds on my machin (1)

Anonymous Coward | about 3 months ago | (#47317871)

A couple of years back I worked on a contract project where the original developers had clearly not actually tried a clean build from scratch with someone who didn't know the system. The project had python, postgres, java, and ruby (at least) and each of those had dependencies - some of which were specific versions of packages, some of which could just use the latest. None of it was documented, of course, because the packages were all just there in the original developers' system.

It was a nightmare to build and took maybe three weeks to build cleanly - half of this was because I was working on an old slow machine with only 500MB of memory - but then the company bought a new machine and a couple of packages that took an hour or two to build got down to only a few minutes.

Once the build worked I had a shell script that would run on a clean OS install and required only about three prompts to the user. It was at least minimally documented :) Then I got to start on understanding enough of the code to make the required changes (kind of a problem in itself as the requirements were, at best, foggy and at worst contradictory). Just about the time I had the first real code for them, the contract ended.

why greed fear ego based 'societies' fail? (-1)

Anonymous Coward | about 3 months ago | (#47317321)

no spirit... totally deception based,, re-enter momkind new clear options... no bomb us more mom us... feed the millions of innocent starving infants (1000s dying daily still by our neglect) unchosens... free the innocent stem cells... no drone us no bone us,,, give until it stops hurting/lighten our load initiative & many other native spirit based free for all awakenings... see you there..

Re:why greed fear ego based 'societies' fail? (0)

Anonymous Coward | about 3 months ago | (#47317375)

re-enter mom

incest is best

Re:why greed fear ego based 'societies' fail? (2)

HeckRuler (1369601) | about 3 months ago | (#47317501)

Is this some sort of cry for help? Are you ok? Do you need us to call someone?

Go and dependencies (1)

Anonymous Coward | about 3 months ago | (#47317371)

The designers of Go talk a lot about dependency management (Go bans cyclic or unused dependencies, and in another sense of "dependency" the compiler links in all the runtime/stdlib it needs, so no external libraries need to be installed on the target machine). I found it interesting because I'd never thought of dependencies as an important focus for a language; maybe it's because of this experience at Google.

Re:Go and dependencies (1)

BitZtream (692029) | about 3 months ago | (#47317639)

Static linking isn't unique to Go. It's telling that you just discovered it, and further more don't realize what it is that you've discovered.

so true (1)

slashmydots (2189826) | about 3 months ago | (#47317377)

As soon as you rely on someone else's 3rd party code module, you're screwed. You're in for 10 years of constant nightmarish problems. So yes, decencies are the problem.

Re:so true (1)

mlts (1038732) | about 3 months ago | (#47317439)

Of course, said code module can change, so even if your deps are right now... it might be that the black box code library that does some essential functions might not work the same after it gets updated. In some ways, this is easy to find (if you do a library upgrade and things break, with no other changes.) However, if there are other confounding variables, this might be a fairly difficult task.

Re:so true (0)

Anonymous Coward | about 3 months ago | (#47317467)

So yes, decencies are the problem.

Fuck yeah your damn right motherfucker. Nigga keepin it real! Real indecent and shit.

Re:so true (0)

Anonymous Coward | about 3 months ago | (#47317575)

your

GULDAM! Keep that shit on the low-down son. This here's a fucking CLEAN establishment! Got no time for that sorta shit-storm ya be brewin' right-there.

Re:so true (0)

Anonymous Coward | about 3 months ago | (#47317577)

+1

I'm responsible for compiling 3rd party modules from source . . . try doing a full compile on something that keeps updating dependencies.

Then some asshat here say "well, just use maven/git/jenkins/my mom" --- nobody bothers testing the build on a 'clean' machine.

"I TODE you Bobby Bourchier . . . dependencies is tha devil!"

bad study (1)

Anonymous Coward | about 3 months ago | (#47317407)

from the article : " A build was defined as “a single request from a programmer which executes one or more compiles” and deemed a failure if any compile in the build failed."
this is stupid .
When I code , I often do a first or second compile only to see all the trivial error I have made and correct them fast. I do not expect zero error .

Re:bad study (1)

HeckRuler (1369601) | about 3 months ago | (#47317601)

This. I'm inherently suspicious if it actually compiles the first time if I make any sort of significant change.

Re:bad study (0)

Anonymous Coward | about 3 months ago | (#47317855)

Obviously the answer is to turn warnings all the way to 0. ;) You laugh but I have seen it.

Eighteen THOUSAND engineers?! (1)

scottbomb (1290580) | about 3 months ago | (#47317491)

Maybe that's part of the problem. Too many cooks spoiling the broth? Perhaps I'm naive, but 18k seems a bit much for what they produce.

Re:Eighteen THOUSAND engineers?! (2)

rolfwind (528248) | about 3 months ago | (#47317631)

Well if they won't make the beta programs today that will get discontinued tomorrow, who will?

Re:Eighteen THOUSAND engineers?! (1)

HeckRuler (1369601) | about 3 months ago | (#47317635)

At all of Google?
Perhaps that'd be true if they were all trying to make the same pot of soup, but in this case it's more like all the cooks in the city with each group in their own kitchen sometimes serving entirely different ecosystems of consumer.

Re:Eighteen THOUSAND engineers?! (0)

Anonymous Coward | about 3 months ago | (#47317737)

Not only that but with "over 26 million" builds over a period of 9 months is approximately 1 build per working hour per dev. Sounds kind of excessive.
FWIW I hear from a friend who knows a few ex-google employees that claim google actively scoops up/hires talent and makes them spin a hamster wheel "so they don't compete with them", essentially burying all the codes/patents unless it's something they want to use for themselves.
The more you know.

hamster wheel (0)

Anonymous Coward | about 3 months ago | (#47317939)

It sounds like like are borrowing Hollywood's playbook. The entertainment industry always wants to keep a buffer of talent available, so they maintain stables of screenwriters whose scripts will never be filmed, an uncountable number of actors waiting to make it big, and then there are all those doomed pilots and straight-to-video movies that keep entire production crews happy and employed.

Re:Eighteen THOUSAND engineers?! (1)

EvanED (569694) | about 3 months ago | (#47318159)

Not only that but with "over 26 million" builds over a period of 9 months is approximately 1 build per working hour per dev. Sounds kind of excessive.

Really?! Not to me.

Let's say you're only actually coding half the time you're at work. Now you're talking 2 builds per hour, or one every 30 minutes. I can't speak for everyone, but I start getting pretty antsy if I haven't at least built, let alone tested, what I've been working on for that long. And we haven't even gotten into things like dealing with the failures. It's easy to go through multiple builds in almost no time at all as you see a silly syntax error or whatever, fix, and rebuild. Or even in testing: hit an assertion, look into it for 2 minutes, see what's wrong, fix, rebuild, rerun, repeat.

Re:Eighteen THOUSAND engineers?! (1)

geekoid (135745) | about 3 months ago | (#47318285)

they produce a lot of stuff. I mean 1000's of things.
I'm not sure you know what they produce.

Build System? (1)

aliensexfiend (656910) | about 3 months ago | (#47317507)

The article doesn't mention what build system was used for the C++ code. GNU Autotools? CMake? Plain old makefiles? A build system can help you modularize your build so that a complete recompile does not occur after a syntax or linker error.

Oh oh! I got another one!! (0)

Anonymous Coward | about 3 months ago | (#47317545)

People not checking this:

Installation A + Patch B = Installation B

That is: the result of patching a known fresh installation results in 'exactly' another known fresh installation. I've seen this a lot with badly maintained Linux kernels in hardward development. People patching up development hardware sometimes results in chaff files on the HDD which interact and cause strange "works-for-me" bugs. This usually results in the hilarious "Downgrade from version 2 to version 1 then upgrade to version 3" debacle. Or you could just...y'know...fresh install version 3 (which no-one does because "it takes too long").

Also: Things taking too long causes developers to get clever and shortcut things that are actually important. Try ninja-style machine maintenance out of office hours - e.g. cleaner bots, re-install bots, stuff like that. They'll never know you're cleaning house for them.

Hands down, Simplicity of dependencies (1)

EmperorOfCanada (1332175) | about 3 months ago | (#47317613)

Simplicity of dependencies are the only way I survive C++ development. For instance I was just playing with OgreSDK, and it depended on my having a specific version of CMake installed in a specific way, in a specific directory. That's fine, oddly enough I can live with that. But how long before I combine Ogre with something that requires something different about the CMake installation?

So, for instance I like the Crypto++ library because I can cheat and just slam it into my multi-platform codebase without worrying about keeping it separate. And seeing that over 50% of my code needs to run on 4 platforms I have become very sensitive to this kind of crap. I consider a platform specific #ifdef's to be a personal failure (even though they are lightly sprinkled through my code).

I somewhat like boost with its mostly header only stuff but that is not without some consequences. POCO is cool but I find that having a single codebase that is multiplatform is an IDE configuration nightmare.

So what is my favourite solution? I would say that libraries have many virtues but that in the end any fights with them can be horrific. So I am going to go with my present favourite for multiplatform dependency goodness; and that is Cocos2d-x; it isn't perfect but the nice combination of it being a library and part of your code base results in some of the fewest compilation problems that I have ever had.

I would say that its primary cost is when you go to upgrade to a much newer version of the library. Basically you have a fight on your hands. But that fight is once in a blue moon. Whereas I find some other libraries are a non-stop fight, sometimes with no resolution.

Re:Hands down, Simplicity of dependencies (1)

Anonymous Coward | about 3 months ago | (#47317797)

I've seen nightmares in C++ APIs with something as simple as the max() function. Sometimes it's as simple as someone re-"#define"-ing the symbol to something else to fix a build issue, or putting it in a namespace. Since preprocessor directives don't live in namespaces they have global effects - those things can be dangerous because they're totally unexpected and pop up all over the place. It goes against orthogonality and encapsulation.

Also: using namespace clauses in headers. LOL. NOPE.

Re:Hands down, Simplicity of dependencies (1)

EvanED (569694) | about 3 months ago | (#47318241)

I've seen nightmares in C++ APIs with something as simple as the max() function. Sometimes it's as simple as someone re-"#define"-ing the symbol...

*cough*<windows.h>*cough*

I blame... (1)

ThatsDrDangerToYou (3480047) | about 3 months ago | (#47317655)

.. Jones, the guy we let go last month. He's pretty much to blame for all build fails for the next few weeks. Then we can start blaming the marketing dept.

Build failure is just the beginning. (1)

140Mandak262Jamuna (970587) | about 3 months ago | (#47317819)

Large distributed development. Multiple point check ins. (Our divisional build is around 100 execuables and 300 dlls. Corporate level build would easily exceed 1000 executables). Our build group will launch builds on every library every hour and immediately report compilation failures and link failures. But that is just the beginning

Then comes the installation and packaging failures. The dynamic libs get out of synch, wrong dll gets packaged in, etc

Then the build is good, it does not crash on every project. But it fails daily validation suite. After clearing that hurdle the build fails certification (same as validation but more detailed comparison with golden results).

So typically our last daily build is about one day old, last validated build is two or may be three days old. Last certified build could be three to five days old.

And this is a great improvement compared to the past.

Biggest advancement that helped us were dirt cheap prices for storage. So we are able to keep multiple older working builds for the developers. Not all of them need the latest build. Second biggest thing was the multicore machines with enough horsepower to launch all the library builds simultaneously. Third was the drop in network bandwidth prices, we are able to consolidate and synch the source code repo with very small (for a developer I mean, not for a video watcher) latencies.

Gentoo (0)

aliensexfiend (656910) | about 3 months ago | (#47318023)

I really admire how Gentoo manages the build of so many packages from ebuild. I think the authors of this paper should study this project.

Re:Gentoo (-1)

Anonymous Coward | about 3 months ago | (#47318365)

Why study a blatant ripoff of BSD ports?

Thinking about the problem not the syntax (0)

Anonymous Coward | about 3 months ago | (#47318101)

When I code, I am thinking at a high level about the problem I'm solving. My builds fail because I make some trivial mistake, like using the wrong method name for the language I'm coding in, or forgetting a semicolon, or forgetting an import statement. Yes, the compiler catches it for me. I'd rather do that than switch from my high-level mental state in the problem domain to a low-level state of compiler and syntax trivialities. I'm solving a problem.

This excerpt says it all... (0)

Anonymous Coward | about 3 months ago | (#47318211)

This excerpt says it all in my opinion: "Almost 65% of all Java build errors were classified as dependency-related, such as cases where the compiler couldnÃ(TM)t find a symbol (the most common one, 43% of all build errors), a package didnÃ(TM)t exist or Google-specific dependency check failed."

Their failure rate is absurdly high! (0)

Anonymous Coward | about 3 months ago | (#47318357)

"The study found that the median build failure rate for C++ code was 38.4%, while the median for Java was 28.5%."

Load More Comments
Slashdot Login

Need an Account?

Forgot your password?

Submission Text Formatting Tips

We support a small subset of HTML, namely these tags:

  • b
  • i
  • p
  • br
  • a
  • ol
  • ul
  • li
  • dl
  • dt
  • dd
  • em
  • strong
  • tt
  • blockquote
  • div
  • quote
  • ecode

"ecode" can be used for code snippets, for example:

<ecode>    while(1) { do_something(); } </ecode>