Preheat the oven to 325 degrees. Grease and flour a 9×5 inch loaf pan. (Full disclosure - we used a non-stick loaf pan and didn't grease it, it came out fine.)
In a medium bowl, cream together the butter and sugar until smooth. Remember that the lavender sugar starts as turbinado sugar, so it will act differently than white granulated sugar. It also is not white at the base so it will be darker in color. It will eventually cream together, just be patient.
Beat in the egg until the mixture is light and fluffy; add in vanilla.
Combine the flour, baking powder, and salt; stir into the creamed mixture alternately with the milk until just blended.
Pour into the prepared pan.
Bake for 45-50 minutes in the preheated oven. Ours sometimes takes longer to bake - you might need to check it by inserting a cake tester, toothpick or knife and making sure it comes out clean.