Variable weather continues to be the major source of vulnerability to chronic hunger and poverty in many developing countries due to the strong dependency of livelihood strategies on rainfed farming. Quantifying the effect of climatic parameters on agricultural households is, therefore, necessary to help policymakers understand the benefits of climate policies, improve the allocation of the scarce resources dedicated to adaptation and prioritize among adaptation strategies. This article investigates the empirical relationship between the welfare of rural households and rainfall variability in the semi-arid tropics, using household panel data and high-resolution remotely sensed rainfall data from Niger. We find that a standard deviation increase in rainfall variability is associated with a reduction of real food consumption by 11.13%. Results also indicate that a standard deviation increase in rainfall variability reduces expenditures for cereal-based products, animal-based products and processed foods by 11.96%, 21.31%, and 16.23%, respectively. Our results are consistent across a battery of robustness checks. Finally, we find geographical-based differences in terms of the effect and that access to cereal banks cushions the negative effect of rainfall variability. Policy interventions aiming at improving the well-being of rural households should therefore emphasize improving climate adaptation strategies.