Preheat the oven to 325F. Line the bottom of a 9-inch springform pan with parchment paper. Lightly grease with olive oil. Place on a baking sheet (this will help catch any olive oil that drips out).
In a large bowl, whisk together the oil and eggs until smooth. Whisk in the milk, lemon zest + juice, orange zest, and sugar until smooth. Add the flour, salt, baking soda, and baking powder and use a rubber spatula to stir and fold until just combined (try not to overmix). It’s okay if there are lumps!
Transfer to the prepared springform pan. Bake for 60-70 minutes, or until a toothpick comes out clean (times vary a lot depending on ovens and baking vessels, mine always takes 60 minutes). Let cool completely before serving.
To make the maple whipped crème fraîche, add the heavy cream to a medium bowl. Use an electric beater to beat until soft and fluffy with stiff peaks (about 2-4 minutes, don’t overbeat!). Add the crème fraîche, maple syrup, vanilla paste or seeds (if using), and a generous pinch of kosher salt and beat until combined. Taste and add more sweetener, if needed.
Spoon the whip onto the center of the cake and use the back of a large spoon to swoosh it all around the top. Sprinkle pomegranate on top. I like a light drizzle of olive oil and sprinkling of flaky salt on top, but that’s just me!