Abstract:

Developments in hardware resources, external computing devices, third party libraries and languages capacities must be considered if big data applications aim for cutting-edge performance. Legacy code structure can make such considerations difficult or impossible. For addressing these issues, LMT (‘linear models toolbox’) has been developed as a successor for DMU. LMT is a stand-alone, single-executable, instruction file driven program for genetic evaluation and variance component estimation in multivariate data sets with hundreds of millions of equations. LMT supports a wide range of models including ssSNPBLUP random regression test-day models, metafounders, and explicitly modelled genetic groups. Tested on large scale industry data, LMT decreased the necessary processing time by 93% when compared to DMU, and needed 1.7 hours to process a random regression test-day model with 350 million equations and 250 million observations.