Preheat the oven to 350°F. Combine the graham cracker crumbs, brown sugar, melted butter, and salt in a food processor. Pulse until the mixture resembles slightly wet sand.
Pour the crumbs into the bottom of a 9- or 10-inch pie plate. Press the crumbs firmly onto the bottom and up the sides using a dry measuring cup to pack. Bake the crust for 10 minutes or until the crust is golden and set. Remove from the oven and let cool completely.
Meanwhile, whisk together the sweetened condensed milk, sour cream, egg yolks, lime zest, and lime juice in a medium bowl.
Pour the filling into the cooled crust.
Bake for 15 minutes or until the filling is set. (The filling will still be jiggly in the center once you lightly tap the pan.) Remove the pie from the oven and let cool completely. Refrigerate for 4 hours or up to overnight.
Remove the pie from the refrigerator and cut into slices. Garnish the pie with whipped cream, key lime slices, and lime zest, if desired.