Nelson County is located in Central Virginia midway between the metropolitan areas of Charlottesville and Lynchburg. The county has a total area of 471 square miles. It is bordered on the south and east by the James River and on the north and west by the Blue Ridge Mountains. A large portion of the western section of the county is in the George Washington National Forest.

Located in the geographic center of the state, Nelson County is approximately two hours from Richmond, three hours from Washington, D.C. and four hours from Charlotte, North Carolina. The elevation ranges from about 500 feet above sea level near the James River to more than 4,000 feet in the Blue Ridge. The climate is moderately warm in the summers, with temperatures averaging about 77 degrees in July. Winters are moderately cool, with temperatures in January averaging about 38 degrees. Average annual precipitation is about 42 inches.