Data Analysis Standards (WP6)

News

All the deliverables of the work-package have been published, and are made available to the neutron/muon community. This work package is now completed.

Report on existing software for data analysis (D6.1 Task1) ;
Report on solutions for developing a common software infrastructure (D6.2 Task2)
Report on the development of prototype software (D6.3 Task3). The source code can be obtained here
Report on the evaluation of the prototype software (D6.4 Task 4).

Overview

High resolution detectors and higher data rates, among others, make new demands on software that require professional software solutions. In particular, new instruments constitute a considerable challenge for software provision and failure to address this issue leads to a delay in the scientific impact of new investment.

The importance of data analysis has now become clear and is becoming a focal point of efforts to optimise scientific production.

The aim of this networking activity is to determine how to develop, deploy and operate a common data analysis software infrastructure to facilitate joint software development in the neutron scattering community.

Workpackage 6 FP7/NMI3-II project number 283883 tasks (started June 2012):

Task 1 : Review existing data analysis software and practices of software developers (March 2013)
Task 2 : Review existing solutions for a common data analysis infrastructure (March 2014)
Task 3 : Develop prototype software in chosen solution for representative applications (Sept 2014) ; Source code
Task 4 : Evaluate prototype software (Feb 2015)

Partners: ILL (lead), STFC/ISIS, TUM and JCNS (FRM2), PSI, HZB, CEA, HZG/Hamburg, ESS Lund/Copenhagen

Look at our Meetings here with their minutes.

Tasks and Reports

Work package 6 "Data Analysis Standards" description, tasks and reports

WP6 description and tasks (544 kB)

WP6 Task1: Report on current software and practices (198 kB)

We have reviewed some of the current software for neutron data treatment, analyse their coding, features and practices.

WP6 Task 2: Report on common infrastructures (192 kB)

We discuss on aspects of the infrastructure currently used for neutron/muon software development. We derive guide lines for a future common development framework.

NMI3-II WP6 1st periodic report (Feb 2012-July 2013) (62 kB)

We summarize the objectives of the workpackage, and report on the status of the work-package (progress and achievements, deliverables)

NMI3-II WP6 mid term review (March 2014) (1 MB)

Slides to report on the Project highlights, impact, and outcome.

NMI3-II WP6 Task3 report (prototype code Data Analysis) D6.3 (517 kB)

This report documents the Task 3 of the Data Analysis Standards work- package (NMI3-II/WP6). It details the software that was produced during this project, with code for Mantid and other projects.

NMI3-II WP6 Task4 report (prototype evaluation) D6.4 (185 kB)

This report documents the Task 4 of the Data Analysis Standards work-package (NMI3-II/WP6). It deals with the evaluation of the prototype software that was produced during this project.

NMI3-II WP6 2nd periodic report (Aug 2013-Jan 2015) (173 kB)

We summarize the objectives of the workpackage 6, and report on the status of the work-package (progress and achievements, deliverables). This is the final report on the work-package, which is now completed.

LiveDVD for testing/installing software (July 2013)

Live DVD NMI3/ILL cover (July 2013) The Live DVD NMI3/ILL based upon Ubuntu 12.04. Ready to run, with all software pre-installed.

release July 2013

The Data Analysis workpackage and the Computing for Science group at the ILL, provide a ready to run LiveDVD (bootable) which includes a set of neutron scattering software. It runs out of the box, does not affect your current installed system, and can optionally be installed on a hard disk (create an empty partition/area so that the installer use it).
This is especially suited for testing software, schools/workshop/tutorials, and system deployment.

The current Live DVD 'July 2013' is based upon a Ubuntu 12.04 system with the following additional applications:

LAMP, DAVE, Mantid, Frida, Isaw, nMoldyn, iFit, qtiKWS
McStas, Vitess, SimRes
MFit/MView, vTAS, ResTrax
GRASP, SANSView, SASFit, GenX
FullProf, PDFgui, F.O.X., OpenMX, GSAS/EXPGUI

as well as:

C, C++ and Fortran compilers, Java, Python, Perl, Tcl/Tk, MPICH,
Python libs: Matplotlib, numpy, scientific, tk, numeric
AbInit, Chemtool, Gelemental, Ghemical, Gperiodic,
Gromacs, mopac, MPQC, PyMol, Rasmol, Viewmol,
Avogadro, DrawXtl, CBFlib DX, Fityk, Gabedit, Garlic, Gcrystal,
Scilab, Octave, PerlDL, PGPLOT, netcdf, ncview, hdf4, hdf5, nexus, hdfview
GeomView, OpenBabel, ParaView, Maxima, Java3d, Mayavi, qtiplot
gimp, virtualbox, openoffice, freewrl, partimage, gparted

Get the LiveDVD here (3.2 Gb ISO). Burn ISO, reboot, login with nmi3 no password. More information at the ILL Computing for Science .

History/releases:

March 2011: initial release based on Ubuntu 10.04
May 2012: release based on Ubuntu 12.04 , adding Frida, Isaw, SANSview, GenX, PDFgui, Vitess. nMoldyn and FreewWRL are buggy.
June 2012: fixed FreeWRL crash, and nMoldyn. Update your May 2012 installation by installing the following U12.04 amd64 deb packages python-netcdf, python-scientific, libmozjs2d, freewrl).
July 2013: updated packages, fixed some applications

Live DVD Release Notes

LiveDVD Release Notes (257 kB)

Neutron-Muon software Debian repository

Software sources: add http://packages.mccode.org/ Debian repository

We now provide an experimental Debian repository with selected data reduction/analysis software. The packages are available for a 64 bits Debian based linux system (e.g. Debian, Ubuntu, Mint, …).

Included packages: bornagain, dave, flatcone, frida, fullprof, genx, grasp, gsas, horace, ifit / reslibcal, isaw, lamp, mcstas-suite, mcxtrace-suite, mfit, neutrons4science, nmoldyn, restrax, root, sasfit, simres, vitess, vtas

To install the packages:

sudo apt-add-repository 'deb http://packages.mccode.org/debian stable main'
sudo apt-get update
and then use your package manager (e.g. Aptitude, apt-get, Synaptic, …).

Produced code and software

The following code has been produced:

Live data reduction server a simple web server which allows to send data files and computational requests to a server, which then dispatches it to computation services, which then report status and results.
AllToMantid a simple Algorithm that can read/write from any client (e.g. LAMP, iFit, …) and generate Mantid workspaces.
A set of Mantid loaders and algorithms to be used for ILL instruments (and others), with example data files, scripts, documentation, mainly in Algorithms LoadILL, LoadILLSANS, LoadLLB, LoadSINQFocus, DetectorEfficiencyCorUser, SetupILLD33Reduction

Download software

Generic, simple Reduction Server (113 kB)

Live data reduction server, python based.

AllToMantid (10 kB)

a simple Algorithm that interfaces Mantid with any other external software

NMI3-II WP6 Contributed code to Mantid (74 kB)

Some of the Algorithms written and contributed to the Mantid project during this WP6 (Leal, Raoul).

NMI3-II WP6 Prototype code Final (Task3) (429 kB)

The final status of the prototype code produced during the WP6 Data Analysis (Task3), sorted by categories. It includes code contributed to Mantid, instruent definition files, python script tools, the AllToMantid Algorithm, and the reduction server.