The actual study...Link

FreeSurfer is a popular software package to measure cortical thickness and volume of neuroanatomical structures. However, little if any is known about measurement reliability across various data processing conditions. Using a set of 30 anatomical T1-weighted 3T MRI scans, we investigated the effects of data processing variables such as FreeSurfer version (v4.3.1, v4.5.0, and v5.0.0), workstation (Macintosh and Hewlett-Packard), and Macintosh operating system version (OSX 10.5 and OSX 10.6). Significant differences were revealed between FreeSurfer version v5.0.0 and the two earlier versions. These differences were on average 8.8±6.6% (range 1.3-64.0%) (volume) and 2.8±1.3% (1.1-7.7%) (cortical thickness). About a factor two smaller differences were detected between Macintosh and Hewlett-Packard workstations and between OSX 10.5 and OSX 10.6. The observed differences are similar in magnitude as effect sizes reported in accuracy evaluations and neurodegenerative studies.The main conclusion is that in the context of an ongoing study, users are discouraged to update to a new major release of either FreeSurfer or operating system or to switch to a different type of workstation without repeating the analysis; results thus give a quantitative support to successive recommendations stated by FreeSurfer developers over the years. Moreover, in view of the large and significant cross-version differences, it is concluded that formal assessment of the accuracy of FreeSurfer is desirable.

Basically it says you can't put your trust in this particular software if you've recently switched your OS platform, upgraded your OS version or updated your software.

Demetrius: TFA doesn't say if the results are consistent from PC to PC, or PC OS to PC OS. If they're consistent there, then I'm pretty inclined to put my trust in the PC.

The article has a link to the research article, which is published in PLoS ONE and is open access so you can read it yourself. The PC is an HP machine running CentOS. They controlled for replicated run differences on the same machines as a baseline. Differences in calculations between versions of the same software were the largest, and that isn't that surprising. The differences between Linux and OS X (everything here was run from command-line) is a little more worrying, and between different versions of OS X as well. Although not mixing and matching versions and OSes in your analysis pipeline should be something they drill in to every scientists head.

The real issues is consistency of results. If you are doing comparisons in your study using the same program, on the same OS, you shouldn't have a problem. Chances are significant results will be significant regardless. My guess off the top of my head would be differences in the OS specific subroutine being called by the program to do the actual volume calculations.

Nichiren: Basically it says you can't put your trust in this particular software if you've recently switched your OS platform, upgraded your OS version or updated your software.

Sort of. It is saying you can't put trust in results if you have performed different stages of the analyses on different platforms. But frankly that should be a given for any sort of data analysis. You always sue the same version of the software for consistency and should probably do all of the analysis on the same version of an operating system as well.

Headline looks different when interpreted by subby.

TFA: "Your Brain Scan Looks Different on Mac and PC"

Subby: different on Mac.

There are so many ways to spoil the proper brightness, contrast and gamma settings between the original material and the display...

Professional graphic designers use monitor calibration hardware to avoid this very problem.

Demetrius: TFA doesn't say if the results are consistent from PC to PC, or PC OS to PC OS. If they're consistent there, then I'm pretty inclined to put my trust in the PC.

According to the paper, they used 5 Macs (2 iMac, 3 Mac Pro) and just one HP system running CentOS.

Here's the chart of systems used.

And from a cursory glance, the software they used is only available on Linux and OSX.

Hmmm. Freesurfer is an open source app targeted at Linux and MacOS. The only way to run it on PCs is thru virtualization. Its operation can also depend on the capabilities of the particular graphic card. Seems like a number of areas that could be responsible for differences.

I presume that Freesurfer isn't FDA-approved and that would account for why nobody has compared its output on different platforms up to now. I also presume that would mean that it can't be used for diagnostic purposes, so is this really a problem?

entropic_existence: The real issues is consistency of results. If you are doing comparisons in your study using the same program, on the same OS, you shouldn't have a problem. Chances are significant results will be significant regardless. My guess off the top of my head would be differences in the OS specific subroutine being called by the program to do the actual volume calculations.

Yup. Too add to this, the most important line of this paper is probably "For the same OS version, all Mac workstations produced identical results." This implies any PC architecture running the same version of Linux should produce the same results as well, and pretty much ensures that the observed differences are a software "feature". The potential for differences in software based calculations is VERY well known to the people who develop these kinds of programs, and products like matlab exist simply because they're better than most individual programmers at handling obnoxiously large numbers.

The differences can be a bit discomforting, but the reality is that if the doc sees a questionable spot, he's likely just going to do a higher resolution scan in the suspect area(s). Our CT over here (aerospace use) usually does 1440 slices and can look at regions much smaller than a human head. Top of the line MRI capability should be comparable. Assuming your doc is capable and willing, the risk of completely missing something significant because of this is pretty small.

Stavr0: There are so many ways to spoil the proper brightness, contrast and gamma settings between the original material and the display...

Professional graphic designers use monitor calibration hardware to avoid this very problem.

Yeah. If Freesurfer does its measurements based on the output to a rendered display or frame buffer (post-color temperature and gamma adjustment) rather than on the raw data file, then that could account for the differences. It would be an odd way to do it, but it also make it image-format agnostic.

Just going out on a limb here, but shouldn't any medical professional using such a program be trained in measuring the studied areas to begin with? Seems like they should be using this more as a tool than as a definitive analysis.

FreeSurfer is open source and based on MATLAB.

'nuff said.

/Maybe someone forgot to set the FPU control register and it is causing rounding differences? I know it is different between OS versions, it was changed at one Windows version too.

But let me guess: the Mac said the users had to delete all the results off of the machines before it would let them download another app.

jjorsett: I presume that Freesurfer isn't FDA-approved and that would account for why nobody has compared its output on different platforms up to now. I also presume that would mean that it can't be used for diagnostic purposes, so is this really a problem?

Its a research tool, and a widely used one at that. But this sort of thing is a known issue within computing circles anyway. The issue though is that the tools are getting easy enough to use, and the data so massive, that people with very little training in computing or even doing rigorous statistics are now doing very large analyses of large data sets in medical research.

Flt209er: The differences can be a bit discomforting, but the reality is that if the doc sees a questionable spot, he's likely just going to do a higher resolution scan in the suspect area(s). Our CT over here (aerospace use) usually does 1440 slices and can look at regions much smaller than a human head. Top of the line MRI capability should be comparable. Assuming your doc is capable and willing, the risk of completely missing something significant because of this is pretty small.

I'm guessing this program isn't used in a clinical setting. There are usually very different rules and requirements for everything when it comes to clinical diagnostics. This is used in research.

entropic_existence: I'm guessing this program isn't used in a clinical setting.

That too.

StrangeQ: Just going out on a limb here, but shouldn't any medical professional using such a program be trained in measuring the studied areas to begin with? Seems like they should be using this more as a tool than as a definitive analysis.

Why would we want to make a doctor get out an actual tool like a caliper and do something so mundane as actually "measure" the structure on the image? He has 40 other patients to see today, nurses to grope, and golf courses to get drunk upon.

This isn't a real medical app, this is "gradware." The fact that the results are garbage should be unsurprising.

Big-endian versus Little-endian version 2.0?

Several years ago I had an MRI machine shutdown and loose the scan data after I was in the machine for almost an hour because the console computer, then running Windows NT 4.0, decided to blue-screen during a brain scan. They could not get the system to reboot and claimed they could not recover what they had already done, Had to go back four days later.

/farking sucked to have one's head strapped tightly in an "Immobilization and Signal Enhancement Kit."

jjorsett: Hmmm. Freesurfer is an open source app targeted at Linux and MacOS. The only way to run it on PCs is thru virtualization.
Not sure if serious...

fluffy2097: This is pretty much what I expect from most any open source software.

Wildly differing results between every single version and needs to be tweaked to run correctly on ANY computer at all.

Even the expensive open source software that big companies produce?

fluffy2097: Benjimin_Dover: Even the expensive open source software that big companies produce?

Big and Expensive software in general suffers from this affliction, regardless of open or closed source.

Corporate are morons and the programmers know that by producing a steady stream of failure they ensure future contracts.

Ah.

This is pretty much what I expect from most any open source software.

Fixed then.

fluffy2097: Benjimin_Dover: Even the expensive open source software that big companies produce?

Big and Expensive software in general suffers from this affliction, regardless of open or closed source.

Corporate are morons and the programmers know that by producing a steady stream of failure they ensure future contracts.

keep us developers out of this. We hate building shiat software. We despise our bosses for the incentives (aka we may get to go home one day a month) they give us for building shiat software.