Preheat oven to 375°F. Pulse flour, brown sugar, granulated sugar, cinnamon, salt, ginger, and baking powder in a food processor until combined, about 2 times. Add butter, and process until flour mixture goes past the crumbly stage and starts to clump together and look moist, about 30 seconds. Add oats, and pulse to combine, about 3 times. Transfer to a medium bowl, break up any large clumps, and chill, covered, until ready to use.

Heat a 9-inch cast-iron skillet over medium-high. Add butter, and melt until sizzling. Stir in sugar to moisten in spots. Cook, stirring occasionally, until sugar is a deep amber color, about 5 minutes. Add apples (the pan is going to be overflowing, but they will cook down quickly), lemon juice, and salt. Cook, stirring often, until apples have softened and liquid is slightly reduced, 5 to 6 minutes. Stir in flour and apple brandy until combined. Scatter oat topping evenly over apple mixture, and top evenly with ginger candy. Bake in preheated oven until oat topping is golden brown, 30 to 35 minutes. Let cool slightly, about 10 minutes. Spoon into bowls, and serve with vanilla ice cream or whipped cream.