Fears over housing market fragility as Brexit hits construction