Modeling How Programmers Read Code 115
An anonymous reader writes "Following up on an experiment from December, Michael Hansen has recorded video of programmers of varying skill levels as the read and evaluate short programs written in Python. An eye tracker checks 300 times per second to show what they look at as they mentally digest the script. You can see some interesting differences between experts and beginners: 'First, Eric's eye movements are precise and directed from the beginning. He quickly finds the first print statement and jumps back to comprehend the between function. The novice, on the other hand, spends time skimming the whole program first before tackling the first print. This is in line with expectations, of course, but it's cool to see it come out in the data. Another thing that stands out is the pronounced effect of learning in both videos. As Eric pointed out, it appears that he "compiled" the between function in his head, since his second encounter with it doesn't require a lengthy stop back at the definition. The novice received an inline version of the same program, where the functions were not present. Nevertheless, we can see a sharp transition in reading style around 1:30 when the pattern has been recognized.'"
Re: (Score:2)
Re: (Score:1)
That just confirms that Soulskill is an expert editor who processes the meaning of an entire paragraph at a time, instead of scanning each word like a beginner would.
Different code == invalid results (Score:5, Informative)
This article is complete garbage. They tested 2 people with different code that produces the same results and then make up a narrative of how novice and expert coders think in different ways. Use the same code to test a much larger pool of programers and then the results might actually be interesting.
Re: (Score:3)
It might help if you actually read the articles linked to. It's an ongoing study, and results are "very preliminary".
It looks like they have various code samples. The same functionality is coded in different styles. They're studying both how novice vs. experts read code, as well as how coding styles/language features affect comprehension.
Re: (Score:2)
What "garbage results"? It's something posted by one of the subjects reflecting on his experience. The other is by one of the researchers, but isn't given as a result of anything, just an example of what they're doing. It's also from a couple months ago, and there's a link to an actual paper as well. You can't complain about garbage results, especially citing specifics that aren't even true ("only two subjects, reading different pieces of code") when there aren't any results in the article.
Re: (Score:2)
It looks like they have various code samples. The same functionality is coded in different styles.
More like it is coded in good and bad styles. The video at the top is the worst, full of bad style. Just to go through it all:
* Bad naming of parameters. Both collections and iteration variables feel like the same.
* No methods and data of program wide priority (x, y) declared at different locations.
* Using "between" instead of "in interval. There are only really one reason to use exclusive intervals, and that is when the algorithm naturally ends up on it (for example when looping over zero-bound arrays). Ot
Re: (Score:2)
with the looping variable properly renamed from x_n to i.
'x_n' is not an index, proving that you still havent comprehended the code....
The variable 'x_n' is a member of the list, not an index into the list. While 'x_n' might be a bad name, 'i' is a horrible name.
Re: (Score:2)
Yeah, somehow I think their expert Eric compiling a function in his head is somehow affected by the fact that he actually HAD functions in his version of the code to compile. If you were to run any type of code metrics on those fragments, the inline version would be rated as much more complex.
Re: (Score:1)
Yes, complete [and utter] ...
I've been programming for 40+ years and how I analyze code [and hence, my "eye track"] varies, depending upon what I'm looking for. If I need to know one thing, I'll tend to zero in quickly. If I'm going to make changes, I'll scan over everything [at least] once to try to glean the overall style so I'll know what I'm up against when/before I start to make the changes.
Not only that, but the article's last line is about "joining the experiment" if you're in the Bloomington, IN a
Re: (Score:2)
Also, I suppose the code was written either by a Python novice or a FORTRAN expert :D /.):
I'm more of a Ruby guy, but I learned Python a few months ago, and one nice thing about it are list comprehensions (the syntax will probably get messed up by
x_n = [2,8,7,9,-5,0,2]
y_n = [1,-3,10,0,8,9,1]
def between(numbers,low,high):
return [x for x in numbers if x>low and xhigh]
def common(list1,list2):
return [x for x in list1 if x in list2]
x_btwn = between(x_n,2,10)
y_btwn = between(y_n,-
Re: (Score:2)
Re: (Score:2)
Re: (Score:1)
You mean unlike every other programming language with its special characters?
How many special characters does COBOL have?
Re: (Score:1)
You mean unlike every other programming language with its special characters?
How many special characters does COBOL have?
If you'd ever been zapped in COBOL by a mis-placed period, you'd know that there are 1 too many.
Re:Different code == invalid results (Score:4, Funny)
You may have Python confused with Perl.
Re: (Score:2)
For actual cryptology I wouldn't use anything on a VM. There's very sophisticated attacks on cryptology implementations that measure things like battery power differences and time to calculate that can even tell if there's an extra if statement in there and use that to find a higher probability of a bit being a 1 or 0. If I was writing a real encryption library I wouldn't want anything that could add unseen time dependencies, like garbage collecting things at different times. Its pretty much C or asm.
Re: (Score:1)
For actual cryptology I wouldn't use anything on a VM. There's very sophisticated attacks on cryptology implementations that measure things like battery power differences and time to calculate that can even tell if there's an extra if statement in there and use that to find a higher probability of a bit being a 1 or 0. If I was writing a real encryption library I wouldn't want anything that could add unseen time dependencies, like garbage collecting things at different times. Its pretty much C or asm.
don't you have it completely backwards? wouldn't you rather use a system that has unpredictability in it, like garbage collecting(depending on free mem on the machine it might not kick in at the same time) than something that is easier to predict.
besides you're talking about a local attack when everything is lost anyhow so fuck it...
Re: (Score:2)
No, you want to make all code paths take the exact same amount of time and power. If the VM does things in a predictable way, even if your code doesn't, there's a possible attack vector there. You don't want the processor to be doing anything you can't guess in cryptography.
(Of course VM languages can still use encryption- just call out to a native part to do the actual work).
Re: Different code == invalid results (Score:3, Informative)
Your stick man needs to see a chiropractor STAT!
Re: (Score:1)
Re: (Score:1)
the point of programming an cryptological algorithm is that the code executes an encryption algorithm, not that the source code IS the encryption algorithm...... ;)
Re: (Score:2)
Interestingly the Perl interpreter seems to solve it quite well. Maybe I should try if it can also solve the halting problem? ;-)
Re: (Score:2, Troll)
The novice was probably looking for anything that looked like an actual English word and instead found only a mass of special characters.
Python is not a language, it's cryptology.
What are you, a COBOL programmer? That's nice, we'll get off your lawn now.
Video Speed? (Score:5, Interesting)
Is that video real time (adjusted for the 300Hz sample rate)? I ask because I'm not a Python programmer (I do know C, C++, asm) but about 10 seconds into the video I knew what the program would print and yet the video went on for 3 minutes. Something does not add up.
Re: (Score:2)
I just watched Eric's video. Which is different code (wtf?). Even then I wouldn't read it like he does. They definitely need to refine their experiment's methodology a little bit (ok, a lot) and get a bigger sample group.
Re: (Score:1)
This might be it:
Expert programmer + Phython programmer = NaN
Re:Video Speed? (Score:4, Insightful)
Although I don't necessarily agree with your Python joke (pretty funny, though) your comment does provoke me to wonder if Eric really is an "expert" programmer. His eyes go all over the place even when they don't have to. There's only 2 conditions to remember and one "function" (set intersection). His eyes skip all over the place. Further, after the two sets are formed (he's written them in the box and he's already determined what common() does) why is he even looking at the function common() again? Maybe it's to double check, but surely remembering two conditions and a set intersection (3 operations) is well within the grasp of human short-term memory and surely you'd only look at the function name (what it does was verified earlier). Personally after I'd glanced at between() and common() to confirm they did what they suggest they do, I'd never look at them again. But his "expert" eyes keep going back to them while he is forming his output.
Re: (Score:2)
Adding to my own comment. There's only one condition to remember; I was getting confused with the inline version of the novice. Why he keeps looking back over their code/implementation is bizarre to me... it kind of defies the point of functions in the first place. I'm not an expert but I reckon he's not an expert either.
Re: (Score:2)
His eyes go all over the place even when they don't have to. There's only 2 conditions to remember and one "function" (set intersection). His eyes skip all over the place.
My eyes often go all over the place because I'm restless a lot of the time. It doesn't necessarily need to correspond to my mental processes, which is completely lost on the eye tracker. I really don't think that it's possible to make far-reaching conclusions from strictly local stuff like this. The skills in comprehending large programs seem much more important to me (just as it's precisely those large programs that need those skills in the first place), and an eye tracker alone won't help you with researc
Re: (Score:1)
Re: (Score:2)
Yeah I agree, but watch Eric's (the expert) video again. He doesn't just glance at the function definition; he steps through it.
Re: (Score:2)
In C, for example, do you step through printf() every time you call it? Of course not... you probably don't even have the source code. The only thing that matters is input, output, pre-conditions and post-conditions. But, yeah, maybe expert Python programmers think differently.
Re: (Score:2)
Perhaps the novice is expecting the code to follow a straightforward algorithm, and the expert just assumes Python will be full of counter-intuitive operations and hidden side-effects buried in spaghetti code.
Re: (Score:2)
Hard to tell when we're only comparing two people :-)
Re: (Score:2)
I work with eye trackers on an almost weekly basis. These videos are severely slowed down for us to see the interaction. And yes, us humans have the tendency to read an entire page over and over again (subconsciously) even if entirely unnecessary. The saccades may have been scanning for more information or indeed to confirm something (our memory is incredibly short and prone to error, programmers definitely would know that they can't trust what they remember about a function or input data) or simply thinkin
read code? (Score:1)
I don't even read it, I close my eyes, which are a hindrance, and use my inner eye to feel the code. I become the code. Most times I look like a bowl of spaghetti
Re: (Score:3)
"With the blast shield down, I can't even see!"
"Your eyes can deceive you - don't trust them."
Bad link (Score:5, Informative)
They should link to the follow up post that talks about the experiment with 162 programmers http://synesthesiam.com/posts/what-makes-code-hard-to-understand.html. It also links to the paper that has even more information.
Invalid results? (Score:4, Insightful)
The code between these two individuals is completely different, even if it produces the same results. How do you discern any meaningful results out of two people reading two different sets of code?
other factors have alot to do with this?? (Score:1)
the style of the code might also make a difference (as well as the specific languages use of form)
Code of more than average complexity also shifts the reading patterns, as would non trival agorthms that require more study to figure out what it actually does
I have my own code style that assists in fast scanning and placements of specific language features for when Im working on a project of 100000 lines of actual code (30 years a programmer and I also dont care for having 100s/1000s of tiny files spreading
don't give PHB's any ideas we don't need metrics (Score:3)
Some call centers / help desks suck with BS metrics and scripts.
We need less BS metrics as people just game them and people who do a good job have poor metric scores and some with lines and lines of bloated code gets a good score.
Re:don't give PHB's any ideas we don't need metric (Score:5, Insightful)
Unfortunately, this thing seems to be en vogue in the computer fashion industry. I just attended a conference where this phrase could some up a bunch of the presentations:
"We are modeling, tapping into the power of social networks, and doing visual analytics!"
I happen to be reading The Psychology of Computer Programming, Silver Anniversary Edition" right now. An interesting quote:
The only thing that's changed here in twenty-five years is the fact that the funds dedicated by executive to eliminating programmers from their payrolls have become far more staggering than I imagined back then. And, now, I finally recognize in this executive desire a pattern so strong, so emotional, that it has blinded these executives to two facts:
1. None of these schemes has succeeded in eliminating programmers . (We have now at least ten times as many as we did then.)
2. Every one of these schemes has been concocted by programmers themselves, the very people the executives want so passionately to eliminate.
So, although people say that programmers lack interpersonal skills, they evidently have a skill at persuasion that surpasses that of the late, great P:T: Barum, famous for his theory: "There's a sucker born every minute."
I guess if I need some money for something from executive, I'll tell them that I need it to model, tap into the power of social networks and do visual analytics. That ought to get me my funds.
300 Hz (Score:2)
Re: (Score:2)
What makes you think he's sampling 30fps video? There's even a link to the eye-tracking hardware.
http://www.tobii.com/en/eye-tracking-research/global/products/hardware/tobii-tx300-eye-tracker/ [tobii.com]
Re: (Score:2)
Re:300 Hz (Score:5, Informative)
Human eyes "flicker" when they look at something. They will remain stationary for a time, then move quickly to another position. (See Saccade [wikipedia.org].)
The time for one of the fast movements between positions is in the order of 20 ms when reading, giving us a frequency of about 25 Hz. (It's only half a sine wave, so the period is 40 ms.)
Using the Nyquist-Shannon sampling theorem [wikipedia.org], we get that we must sample the eye movement with a frequency of at least 50 Hz, otherwise we'll get aliasing. Now, bring in the engineering rules of thumb, which say that it's no good riding on the Nyquist limit, but you'll need to oversample the signal a bit in order to get a useable result (It of course all depends on what you'll be using the signal for. In feedback control you usually oversample by a factor of 8-20, and in signal processing in the neighbourhood of 2-8) and you end up with a samping frequency of 100 - 400 Hz.
So, in summary, 300 Hz sounds like a perfectly good sampling frequency, perhaps even a bit in the low end, depending on what you'll use the sampled signal for.
Re: (Score:3)
And _that_ is why so many A/D systems fail miserably: because people have been very, very confused by sampling theory.
Just because it takes 100 msec to respond does not mean that the eye motion takes anywhere that time, and the motion is not "clocked" or linked to some discrete frequency. It's analog, and to measure its impulse driven movements properly you need to oversample temendously, or use some sort of triggered sensor that can record its triggers very accurately.
Re: (Score:2)
It seems perfectly reasonable to me, so I have to ask: What frequency do you recommend?
Re: (Score:2)
Re: (Score:1)
Dupe? (Score:3, Interesting)
I'm not sure this topic requires another article:
http://developers.slashdot.org/story/12/12/19/1711225/how-experienced-and-novice-programmers-see-code [slashdot.org]
Re: (Score:2)
I'm not sure this topic requires another article:
http://developers.slashdot.org/story/12/12/19/1711225/how-experienced-and-novice-programmers-see-code [slashdot.org]
well it would if they had done an actual study in the meantime.. it was shit then and it is shit now.
Re: (Score:1)
Re: (Score:2)
And are you sure the code isn't malicious? Not a good idea.
Ignoring that possibility, I very rarely need to know what a program does- I need to know how it does it in order to fix it or add features. You don't get that from running it.
Re: (Score:2)
I need to know how it does it in order to fix it or add features. You don't get that from running it.
Surely once you verify a function does what it is supposed to do you don't keep going back and stepping through it again, though? (referring to Eric the expert's eye movements here). What's the point in that?
Re: (Score:2)
Not unless I think I missed something, or its a tightly coupled implementation with another function. My point was just that "just running" code isn't a good idea and doesn't generally get you what you need.
Re: (Score:2)
By the time I'm working on a program I'm already told what it does. I don't think I've ever gone into a situation that blind. What I don't know is how it does it.
Re: (Score:2)
You can tell it's not malicious from a glance. To be malicious it would need to access the filesystem, networking or interprocess subsystems.
Re: (Score:1)
It could output a text which, when read, causes harm to you. Like the funniest joke of the world.
Re: (Score:2)
Then it would be a good death.
Re: (Score:2)
Re: (Score:1)
> (has Dilbert thought you nothing?).
I suppose he doesn't think about many people except himself, thats why he doesnt teach anymore :)
Just evaluate it (Score:2)
If you want to know what a code prints, just evaluate it, it will tell you instantly without having to worry whether you made a mistake in evaluating the code in your head.
Now if you want to understand the code, that's something else, but it's not what's asked here. The code is sufficiently straightforward that the only explanation you could give is the code itself.
Re: (Score:1)
How would a 9 end up in the last line if the second line has no 9 in it?
As far as I can see, the third line should only have an 8, nothing else.
Re: (Score:1)
OH, BTW, I just tried to print an array with Python, and it *did* output brackets and commas. So the novice was right and the expert wrong in that regard.
Point of focus (Score:1)
When reading the lines
In this last 30 seconds or so of the novice video above, you can see her back-and-forth comparison of the x and y lists. If you look carefully, however, the red dot (her gaze point) is often undershooting the numbers on both lists. Why is this? While it could be a miscalibration of the eye-tracker, the participant may also have been using her parafoveal (the region outside the fovea) to read the numbers. This and the fact that foveation and visual attention are not necessarily always t
Re: (Score:2)
x = [2, 8, 7, 9, -5, 0, 2] print [xn for xn in x if 2 < xn < 10]
y = [1, -3, 10, 0, 8, 9, 1] print [yn for yn in y if -2 < yn < 9]
print [xn for xn in x if xn in y]
Funny thing is, I find that easier to read and understand than the original. It's like "make a list of the elements in this range and print it", twice over, and finally "make a list of the stuff in x that's also in y and print it".
Flash? (Score:2)
It's funny how there's huge amounts of people criticizing flash on slashdot every day, and then we have an article with a flash video (I can only assume it's a video, since I don't run flash) attached to it.
Do these guys even know how their target audience is?
More stuff (Score:1)
Skill level? (Score:1)
Re: (Score:1)
Re: (Score:2)
Obviously, you haven't progressed past the 'beginner' stage...
Re: (Score:1)
Re: (Score:1)
One data point in a multidimensional space does not a dataset make.
Of course it does. It doesn't make a very useful dataset, and certainly not a statistically significant one, but it is a dataset nonetheless.
Re: (Score:2)
I've been programming for a long time, and I still tend to scan over the code, looking at defs. Or if I'm in a proper IDE, looking at the class outline first. Then I go to the entry point, if there is one, or the init if there isn't, and read through that line by line.
Actually, the FIRST thing I try to do when I get a non-trivial program/system dumped on me is attempt to find and review the technical documentation.
THEN I start looking for/at code, based on the documentation.
In cases where the sole documentation consists of oral folklore, I spend a brief interval meditating on the best way to hunt down and kill those responsible.
Re:This must be a taxpayer-funded experiment (Score:4, Insightful)
when confronted with a situation falling within their specialized field, experts can process information in large chunks. Whereas laymen and novices tend to process things one small piece at a time; and on top of that, they flail around a lot.
Actually, that's a load of nonsense:
1) With programs that are *actually* large, you won't find "experts" that consume them in "large" chunks, unless they use very small fonts.
2) With programs that are new to the readers, you might have to read in toto them anyway. There's no guarantee that the *actual* dependencies in the code will allow you to read it in a limited or strictly hierarchical fashion. You gotta read what you gotta read. It's not like people will only shove neat and pleasurable code on you in real life. If the code is messy, your reading of it will most likely be messy, too. Especially if you hit duplications and have "wait a minute, didn't I see this somewhere else? Lemme check" moments.
Re: (Score:1)