Tom Gilb & Kai Gilb's blog

Software Engineering Requirements

By TomGilb on Sun 31 of Jan., 2010 05:29 GMT+01:00

Tom's position on real software engineering: It is Not about programming!

Software Engineering Requirements

http://bit.ly/axtT9A (external link)

www.semat.org (external link)

My Position Paper for March Zurich Conference. Too many 'coders' involved!

A select group of 'wise old men' (at least 'old') including me, is going to meet 17-19 March at ETH Zurich, to tackle the problem of making real software engineering happen. I am also going to hold some courses public in town that week too.

We have failed miserably, and we have been the methods leaders, book writers, conference speakers for decades.

Reading some other's position papers, I am already worried, as I initially was, that some of these people still do not see a distinction between programming, and software engineering - between a craft of building, and an intellectual/practical discipline of designing and architecture.

They consistently fail to mention anything about quality and cost, or design or architecture to meet performance Requirements within constraints.! They consistently refer to code and coding processes. I know these guys, they are actually intelligent people - but it amazes me they have no sense at all of what engineering is. Lack of the culture.

Failure to engineer, while "improving Scrum Burn Rates" is arguably one major reason for large software system failure. I find that most large projects do not Quantify their top few critical (quality, improvement objectives. I have seen at least 3 $100 million software projects totally fail for this reason alone. Lots of smart programmers there though!

My personal prediction is that real major change in teaching, and doing, real software engineering, will only come about as a result of very major catastrophes, caused by bad software. In my imagination 300 aircraft will kill all their passengers, on a single day, due to an air traffic control 'glitch'. Caused by poor engineering practices. Will that wake people up?

Until then we can try to prepare the ground by working out ideas of what we should be doing - but nothing will happen until we are massively motivated - and we are really not motivated yet.

Even events like Nine Eleven (11 Sept, attacks), and Credit Crunch Financial Collapse are not big enough for us to change anything effectively. So something much more painful is required. I hope our ideas survive, the '2012' catastrophe (see this film - it makes you think).

Or wait a minute, our ideas on doing software are so bad now, that maybe if they all got wiped out, we might get a fresh start based on common sense? Yeah, wipe out all those dumb software ideas, including agile, then use our heads:

One simple survivor: deliver real measurable value for money to stakeholders.

Is that so difficult to understand?

on Sun 31 of Jan., 2010 18:27 GMT+01:00, by pschwarz

Hi Tom.

Last year, I bought a copy of 'What Every Engineer Should Know about Software Engineering' (see http://bit.ly/7DOs6M) (external link)

, because I was impressed by the fact that even though its purpose was simply to help engineers better understand the nature of software engineering, it touched on relatively advanced concepts such as the SOLID principles, TDD, refactoring, and Fowler's Code Smells.

I am going to list some excerpts regarding the role of programming. I would be interested in hearing what you think about them.

"SW engineers probably spend less than 10% of their time writing code; the other 90% of their time is involved with other activities that are more important than writing code"

"Someone who spends the majority of his or her time GENERATING code is more aptly called a "programmer"

"Just as wiring a circuit designed by an electrical engineer is not engineering, writing code designed by a sw engineer is not an engineering activity"

"There are many other ancillary activities to sw engineering, one of which is the programming of the code"

"the writing of code can be the least engineering-like activity that a software engineer can undertake"

"the design is then converted into code"

on Mon 01 of Feb., 2010 12:57 GMT+01:00, by Tom Gilb

Philip,
I agree with the distinction made. My quibble is that when developed do other activities especially Requirements and design, they d them so atrociously badly. No quantification of either - intuition, badly guided by fuzzy concepts.

on Mon 01 of Feb., 2010 00:54 GMT+01:00, by pschwarz

Hi Tom,

Right at the very end of your SEMAT position paper, you say that in software engineering 'The emphasis should be on control of the outcome – the value delivered to
Stakeholders, not of the performance of a craft.'

As blog post 'The SEMAT Charter – A proposal' says, when asking the question "what is software engineering", one could begin by (amongst other things) looking at Alistair Cockburn's thoughts on Software Engineering.

What do you think of the following ideas expressed by Cockburn in Agile Software Development (2nd Ed)?:

The dictionary is clear as to what "engineering" is: "The application of science and mathematics by which the properties of matter and the sources of energy in nature are made useful to people" (Merriam-Webster's Collegiate Dictionary, Eleventh Edition, 2003).

That definition does not explain what doing engineering is about. In my experience, "doing engineering" involves creating a trade-off solution in the face of conflicting demands. Another person, though, wrote to me and said, "A basic concept of engineering is to address problems in a repeatable and consistent manner." This confusing the act of doing engineering work with the outcome of doing engineering work is a common mistake.

The outcome of doing engineering work is the factory, which is run while specific people watch carefully for variations in quantity and quality of the items being manufactured.

The act of doing engineering work is the ill-defined creative process the industrial engineer goes through to invent the manufacturing plant design. That process is not run with statistical controls, measuring quantity and quality of output. Like software development, it runs as a cooperative game of invention and communication, with individual people of different backgrounds huddling together to come up with a workable design.

When people say, "Make software development more like engineering," they often mean, "Make it more like running a plant, with statistical quality controls." But as we have seen, running the plant is not the act of doing engineering.

on Mon 01 of Feb., 2010 13:13 GMT+01:00, by Tom Gilb

Philip,
I heartily disagree with most everything my friend Alistair says. As usual (see our quality quantification debate in IEEE Software about a year ago). His dictionary definition is obviously outdated by software. He confuses the intellectual output of a design engineer, with the implementation construction of it (the factory). And further confuses the plant implementation with the operation of the plant. And puts words in people mouth that they did not say or mean. Strawmen.
His "engineering work is the ill-defined creative process" just shows he does not understand the well defined act of being creative. He mixes up the well defined process, with the unexpected feedback turns it produces. I am currently reading 'Change by Design" (Tim Brown, IDEO) which describes this, and have many other sources. He certainly has not read my CE book, or at least never comments on the content one way or another. His fine mind is closed to real engineering, and I have given up on our many conference meetings trying to convince him of anything. He is not interested. Too bad, a fine communicator with funny content.
Tom

on Mon 01 of Feb., 2010 01:27 GMT+01:00, by pschwarz

Hi Tom.

Not long ago, Martin Fowler tweeted: "Code is the mathematics of our profession; it is how we communicate when we want to be concrete".

When I read the following definitions, I can't help thinking that programming plays more than an ancillary role in software engineering:

The application of a systematic, disciplined, quantifiable
approach to the development, operation, and maintenance of
software; that is, the application of engineering to software.
“IEEE Standard Glossary of Software Engineering Terminology,” IEEE
std 610.12-1990, 1990.

The application of science and mathematics by which the properties of matter and the sources of energy in nature are made useful to people" (Merriam-Webster's Collegiate Dictionary, Eleventh Edition, 2003).

Engineering is the application of scientific and mathematical principles towards practical ends. Steve McConnel? - Professional Software Development.

on Mon 01 of Feb., 2010 13:42 GMT+01:00, by Tom Gilb

Hi Again Philip!
Well Fowler has a strange view of mathematics, he completely ignores (could be ignorance) any higher level view of software engineering, such as the quantified engineering of the ilities, such as reliability. Fowler has shown no interest in my work. But he is on of the breed of agilistas that happily lead weak-minded programmers into bad practices, which lead I assume, to our high project failure.

The application of a systematic, disciplined, quantifiable
approach to the development, operation, and maintenance of
software; that is, the application of engineering to software.
IEEE OF COURSE GETS IT RIGHT!

Engineering is the application of scientific and mathematical principles towards practical ends. Steve McConnell? -
PRETTY GOOD, STEVE IS AN EXAMPLE OF A FRIEND OF MINE WITH A WILL TO RISE ABOVE HIS PROGRAMMER ('CODE COMPLETE') ORIGINS.

TO QUOTE FROM MY SEMAT POSITION (SEE BLOG)
SOME BASIC CONCEPTS *
A software engineer is an engineer with specialty in software.

Software engineering is the discipline of making software systems deliver the required value to all stakeholders.

Engineering is an Evolutionary Process, using practical Principles, in order to determine, and identify the Means to deliver, the best achievable Performance and Cost levels balance, for optimal Stakeholder satisfaction, in a complex risk-filled environment.

Software refers to the ‘non-hardware’ aspects or components of a system.

A Softcrafter is a person who practices the craft of programming software for computers

http://www.gilb.com/tiki-download_file.php?fileId=25

Source: Planguage Glossary.

Unsurprising I like my definitions best!

Tom

on Mon 01 of Feb., 2010 01:36 GMT+01:00, by pschwarz

Hi Tom,

You say:

A Softcrafter is a person who practices the craft of programming software for computers. This type of person is better known as a ‘programmer’ (or even ‘coder’). Sometimes they call themselves software engineers, without any engineering competence or qualifications. This is rather like a good carpenter calling himself a structural engineer or skyscraper architect.

Good points, but I can't help thinking of what Kent Beck says in Smalltalk Best Practice Patterns:

Developer castes appeared with the advent of commercial software development. At the top were the systems analysts, lofty intellects, too important to actually use computers. Below them were the programmers, who knew how to turn the vague mutterings of the analysts into something the assembler or (later) compiler could recognize. At the bottom were coders, who took the sheets, filled out by the programmers, and created punched cards.
...
Better interface technology has long rendered the coder-as-typist obsolete. The word has come to mean all the low-level activities of programming. The stigma remains, however. When I became a consultant, I was told, "Don't let them know you can code, or you'll never be able to charge premium rates".

I think the time has come for a redefinition of coding. Not really redefinition, I suppose, but rehabilitation. I like the definition just fine. I think the stigma attached to it is short-sighted and dangerous.

To me, development consists of two processes that feed each other. First, you figure out what you want the computer to do. Then you instruct the computer to do it. Trying to write those instructions inevitably changes what you want the computer to do, and so it goes.

In this model, coding isn't the poor handmaiden of design or analysis. Coding is when your fuzzy, comfortable ideas awaken in the harsh dawn of reality. It is where you learn what your computer can do. If you stop coding, you stop learning.

We aren't always good at guessing where responsibilities should go. Coding is where our design guesses are tested. Being prepared to be flexible about making design changes during coding results in programs that get better and better over time. Insisting that early design ideas be carried through is short sighted.

on Tue 02 of Feb., 2010 13:41 GMT+01:00, by allankellynet

While I agree with the sentiment of the post I think you might underestimate the amount of skill (engineering or otherwise) that goes into programming. Its more than a Simple Matter of Programming.

Still, that is no excuse for missing out the essential engineering aspects which you correctly point out. As a whole I think the software development community doesn't really know what "engineering" is and what an "engineer" does.

(If software development is engineering then I am a third generation engineer, if its not then I've broken the family tradition.)

To develop successfull software systems requires a well rounded approach: understanding need, good coding, quanitication and more. Individuals need to be well rounded to have a mix of these skills.

But, the big but, our industry has broken the process down into a number of roles and encourages people to become extremely specialised in their role rather than being more general. So, you won't get hired as a Project Manager if you don't have Prince2, PMI, kept your qualifications current and have managed lots of similar projects. You won't get hired as a C# programmer if you haven't spent the last five years coding C# and nothing but C#. And so on.

When recruiters demand such specialisation then there is no time, or incentive, to be more well rounded and pay attention to other issues.

ps like the new site design