Since they aren't manned as well as Castle Black home of the nights watch shirt, hoodie, sweater and v-neck t-shirt, they seem the obvious choices to attack. Mance Rayder served at the Shadow Tower before he abandoned the Night's watch, it would seem logical for him to attack via the Tower. Mance sends a small force to the Tower to divert attention from the main wildling force. Bowen Marsh, the castellan of Castle Black, takes the bait and with the majority of the men at Castle Black, sets off after the raiding party that is attacking the Shadow Tower.
