Activity budgets provide important insights into energy allocation of animals, thus with monitoring over time, potential changes occurring within a population can be identified. However, constructing activity budgets of marine animals that cannot be directly observed can be challenging. Accelerometers have the potential to identify a wide range of behaviours of animals based on their movement patterns. This study aimed to validate the use of accelerometery as a tool to construct activity budgets for fur seals and sea lions, and to evaluate several machine learning techniques for automatically classifying behaviour. We conducted controlled captive experiments with two Australian fur seals, three New Zealand fur seals and four Australian sea lions, where seals were videoed while wearing accelerometers. Behaviours were classified manually from video into four categories (foraging, resting, travelling and grooming) representing the behaviours that constitute most energy use for these species. Data were used to train four predictive classification models (stochastic gradient boosting, penalised logistic regression, random forests and support vector machines). Seal behaviour could be classified with high accuracy (>80%) from the classification methods, with random forests performing the best. This method developed using captive animals has potential for the interpretation of accelerometry data collected from wild animals.