DUNE LAr Software Releases
This page describes recent releases of DUNE software for simulation and reconstruction of event data from the LAr TPCs and associated photon and muon detectors. The software is released in the dunetpc package. Earlier releases in the lbnecode package are described on this other page.
Finding and downloading DUNE releases
Releases of dunetpc (and other DUNE products) are installed at FNAL at
and on CVMFS at
A release and all its dependencies may be installed on SLF and Mac machines using the pullProducts command:
./pullProducts ~/ups slf6 dune-v06_41_00 e14 prof # Scientific Linux 6 ./pullProducts ~/ups slf7 dune-v06_41_00 e14 prof # Scientific Linux 7 ./pullProducts ~/ups d13 dune-v06_13_00 e10 prof # mac OS X 10.9 Mavericks ./pullProducts ~/ups d14 dune-v06_41_00 e14 prof # mac OS X 10.10 Yosemite ./pullProducts ~/ups d15 dune-v06_41_00 e14 prof # mac OS X 10.11 El Capitan ./pullProducts ~/ups d16 dune-v06_41_00 e14 prof # macOS 10.12 Sierra ./pullProducts ~/ups u14 dune-v06_06_01 e10 prof # Ubuntu 14
The first three arguments are the directory where the products are to be installed, the OS specifier and the product name and version. The last two are the UPS qualifiers. The last is the compiler option (prof is nearly optimized but retains the code line numbers) and the first is a colon-separated list of all other qualifiers (e.g. e7:s15, not e7-s15). The available OS and qualifiers for a given release may be deduced from the list of manifests found in the manifest directory for the desired dune release in the dune bundle listing.
Note that versions above are the latest available at the time of writing (15nov2016). Mac Mavericks and Ubuntu 14 are no longer supported.
The above command will copy many tar files to the current directory and so it is best to run it in a dedicated directory and not in the product installation area. E.g. something like:
Before first installation of UPS products, checkPrerequisites can be run to see if all prerequisite packages are installed.
mkdir ~/upstars cd ~/upstars wget http://scisoft.fnal.gov/scisoft/bundles/tools/pullProducts chmod +x ./pullProducts ./pullProducts ~/ups slf6 dune-v06_41_00 e14 prof
Tar files for individual product versions (including dunetpc) may also be downloaded directly from the SciSoft product area. To install a product, simply untar these in the local products area:
cd ~/ups tar xf ~/Downloads/lbnecode-04.20.00-slf6-x86_64-e7-prof.tar.bz2
If ups is not installed on your machine, then download and unpack the ups product tar file in this manner. Follow the link to the latest version from the soft/packages/ups SciSoft ups package list and then choose the version appropriate to your machine, e.g. ups-5.2.0-Darwin64bit+14.tar.bz2 for mac Yosemite.
UPS can be very picky about the OS version for Ubuntu. To install and use the above Ubuntu 14 releases on an Ubuntu 16 or different Ubuntu 14 sub-version machine, first set the UPS version overrride in one of the following ways:
- Set the UPS_OVERRIDE environmental variable:
export UPS_OVERRIDE="-H Linux64bit+3.19-2.19"
- Add an override file to the product directory:
echo "-H Linux64bit+3.19-2.19" > <product_directory>/ups_OVERRIDE.`hostname`
This is specific to each host. If you have a shared volume, you will need to do this for each machine.
Using DUNE releases
Scripts to set up the FNAL and CVMFS releases are provided. To set up to use release 4.21 from the FNAL installation:
source /grid/fermiapp/products/dune/setup_dune.sh setup dunetpc v06_04_00 -q e10:prof
To use the CVMFS installation of the same release:
source /cvmfs/dune.opensciencegrid.org/products/dune/setup_dune.sh setup dunetpc v06_04_00 -q e10:prof
For local installation of all products in ~/ups, set up with something like the following:
source ~/ups/setup # Or source ~/ups/setups setup git setup gitflow setup mrb setup dunetpc v06_04_00 -q e10:prof
Note that ~/ups/setups will not work with a UPS override file (env override is OK). In that case, use ~/ups/setup.
In all cases, the ups command can be used to find out which versions of dunetpc are available:
fnal:lbne> ups list -aK+ dunetpc "dunetpc" "v04_20_00" "Linux64bit+2.6-2.12" "e7:prof" "" "dunetpc" "v04_20_00" "Linux64bit+2.6-2.12" "debug:e7" "" "dunetpc" "v04_21_00" "Linux64bit+2.6-2.12" "e7:prof" "" "dunetpc" "v04_21_00" "Linux64bit+2.6-2.12" "debug:e7" ""
lar -c someconfig.fcl # Use art event loop to process data according to the instructions in someconfig.fcl fcldump someconfig.fcl 5 # Display the highest five levels of configuration in someconfig.fcl listlibs type pattern # Display the full path for all libraries of type "type" (e.g. service or module) whose full path name includes "pattern"
If you want to develop code, i.e. modify the code in dunetpc, larsoft, etc. or add code from any other repositories, you can use the dune-dev package. Instructions may be found at https://github.com/dladams/dune-dev/blob/master/README.md.
Building and installing dune releases
Build and installation instruction may be found on the Redmine CVMFS page. In particular, this page has the instructions for synchronizing the CVMFS releases with those in the FNAL grid area.
The code I (D. Adams) use to build releases is dune-larrel. See README.md in that package or use "
larrel -H" for instructions. The (optional) products of the build include a release tag of dunetpc (in the repository) and a tar file and manifest for each dune release combination of OS, version, qualifier, etc. The latter can be installed in SciSoft as described below.
Once the release tag is made, then the product must be built for each combination of platform (SLF6, OSX-10.10, ...) and optimization level (prof, debug). An alternative for supported platforms (at presnt SLF6, OSX-10.9 and OSX-10.10) is to use Jenkins: load a KCA certificate in your browser (e.g. with get-cert and import), go to https://buildmaster.fnal.gov/view/LBNE/job/dune-release-build/, click on "Rebuild Last", change the release and submit. The script ~dladams/bin/copyFromJenkins or modified ~dladams/bin/jenkinscp can then be used to copy the product tar files and manifests to the (empty) current directory.
One or more builds often fail on Jenkins. To retry the failed builds, click on the build date to get to the page for that build and then click on "Matrix Reloaded" and then "Rebuild".
The above link to Jenkins will only work from inside the FNAL domain. I have not been able to get things to work properly with VPN or an ssh tunnel. I start a browser on an FNAL machine and send the window over the network. The commands I use:
or use ~dladams/bin/jenkins.
A third option for building the product tar file and manifest is to use the script larsoft/larutils/buildScripts/buildDUNE.sh.
To install the SLF6 products in the FNAL grid area:
ssh firstname.lastname@example.org cd /grid/fermiapp/products/dune tar xf ~dladams/reltars/dunetpc-05.05.00-slf6-x86_64-e9-prof.tar.bz2
To synchronize the CVMFS area with the grid area:
ssh -l cvmfsdune oasiscfs.fnal.gov dune-sync dune/dunetpc
Replace dunetpc with dune_pardata, duneutil or lbne_raw_data to update any of those products. The script dune-sync is installed on the oasis server and may also be found at https://github.com/dladams/dune-larrel/blob/master/dune-sync.
Each synchronization generates a log in $HOME/logs and a CVMFS server tag. Periodically remove these.
The tags may be listed with
cvmfs_server tag dune.opensciencegrid.org
and tags removed with
cvmfs_server tag -r TAG dune.opensciencegrid.org
To copy each product tar file to SciSoft and install the corresponding manifest:
copyToSciSoft /afs/fnal.gov/files/home/room1/dladams/reltars/dunecode-04.20.00-slf6-x86_64-e6-prof.tar.bz2 copyToSciSoft workdir/manifests/dune-04.21.00-Linux64bit+2.6-2.12-e7-prof_MANIFEST.txt
The command script may be found at /afs/fnal.gov/files/home/room1/dladams/bin/copyToSciSoft or http://scisoft.fnal.gov/scisoft/bundles/tools/copyToSciSoft.
In case of error, log in to scisoftportal.fnal.gov and (very carefully) remove invalid files from
After installation, the release should be announced on dune-computing-news AT fnal.gov.
- dunetpc wiki
- dunetpc issues
- SciSoft dune bundles
- SciSoft packages
- dunetpc browser
- Units are cm and ns (Tom, January 2015).
- LBNE/DUNE job submission (talk by Tingjjun)
- Debugging DUNE far detector SW
- LArSoft wiki
- Getting LArSoft
- SciSoft bundles
- Release notes
- LArSoft mailing list.
- Git guidelines
- SeedService (See also the April 2016 talk by Gianluca)
- LArSoft algorithm documentation
- LArSoft documents in INSPIRE: Instructions, Log in, Search
- Redmine code browser
- art workbook
- August 2015 art/LArSoft course
- Message facility
- Services wiki
- Art tools
- cetbuildtools defines many useful CMake commands
- mrb reference guide
- LBNE tutorial
- My package to simplify building is dune-dev
- art instructions for cmake. Up-to-date instructions may be found at the top of $ART_DIR/Modules/ArtMake.cmake and friends.
- Building with cetbuildtools but without mrb.
- Instructions for any product.
- Instructions for getting a KCA certificate.
- Build qualifiers.
- mrb reference guide
35 ton prototype:
- Splitting and stitching data
- Trigger counter indices and locations
- ZS and compression algorithms
- List of 35-ton data files. Get the URL for any of these at FNAL with "~dladams/bin/find35t NAME".
- Files in run control
- Channel map
- Gains and shaping times
- Example pedestal URL
- Approved plots
LBNE jobs may be submitted using the package lbneutil described in a talk by Tingjun.
LBNE geometries are summarized here.
The reconstruction used in the 35-ton simulation is specified in lbnecode/source/fcl/lbne35t/reco/standard_reco_lbne35t.fcl.
GENIE event generator