Line a 9" x 5" x 2¾" loaf pan with parchment paper.
Prep and measure all ingredients.
Mince lemon zest to a fine paste. You will have about 2 tablespoons after mincing.
Combine zest with juice in a small bowl and mix. Set aside for later.
Whisk rice flour, potato starch, tapioca flour, xanthan gum, almond flour, baking powder, and salt in a large bowl. Set aside the flour mixture.
Combine lemon juice mixture, buttermilk, and vanilla in a medium-size bowl.
In a small bowl, gently whisk eggs.
Using a stand mixer fitted with the paddle, beat room temperature butter and sugar on medium-high speed until fluffy. Reduce speed to low and add the egg mixture. Turn off the mixer and scrape down the batter from the sides of the bowl. Beat batter at medium speed for a minute.
Reduce speed to low and add ½ the flour mixture. Next, add ½ the buttermilk mixture, scraping down the bowl as needed. Repeat with the final ½ buttermilk and ½ flour mixture.
Increase speed and beat cake batter until smooth and blended.
Scrape batter into a parchment-lined pan and smooth the top with a rubber spatula.
Place in oven, and after 15 minutes, cover lightly with tin foil. Continue to bake for 50-55 minutes. Bake until a toothpick comes out clean.
Let the cake cool in the pan on a wire rack. After about 20 minutes, grab the parchment and lift the warm cake out of the pan. Set the cake on the baking rack to cool.