The mansion, which was intended as a living memorial to George Washington, was owned and constructed by the first president's adopted grandson, George Washington Parke Custis, upon an 1,100-acre tract of land which he had inherited. Custis hired George Hadfield, an English architect who came to Washington in 1785 to help construct the U.S. Capitol, to design his estate. The Greek revival structure which Hadfield designed took Custis 16 years to complete.

The north wing was the first structure completed in 1802. It was in this building that Custis made his home, with a significant portion of it used to store George Washington memorabilia Custis was acquiring with regularity. Among the items purchased and stored in the north wing were portraits, Washington's personal papers and clothes, and the command tent which the president had used at Yorktown.

Even after the completion of the south wing in 1804, Arlington House was still only a set of detached buildings. With the completion of the central section in 1818, the house stretched 140 feet from the north to the south wing. The central section contained a formal dining room and sitting room, a large hall and a parlor. One of the most recognizable of the section's features are the eight columns of the exterior portico, each 5 feet in diameter at the base.

George Washington Parke Custis and his wife, Mary Lee Fitzhugh (whom he had married in 1804) lived in Arlington House for the rest of their lives and were buried together on the property after their deaths in 1857 and 1853, respectively. They are buried in their original graves in Section 13, at map grid N-30.

On June 30, 1831, Custis' only child, Mary Anna, married her childhood friend and distant cousin, Robert E. Lee. Lee was the son of former three-term Virginia Governor Henry ("Light Horse Harry") Lee and was himself a graduate of West Point.

Between 1841 and 1857, Lee was away from Arlington House for several extended periods. In 1846 he served in the Mexican war under Gen. Winfield Scott, and in 1852 he was appointed superintendent of the U.S. Military Academy at West Point, his alma mater. After his father-in-law died in 1857, Lee returned to Arlington to join his family and to serve as executor of the estate.

Under the terms of her father's will, Mary Anna Custis Lee was given the right to inhabit and control the house for the rest of her life. Custis' will also stipulated that upon Mary Anna's death, full title would pass to her eldest son, George Washington Custis Lee. Contrary to popular belief, Robert E. Lee never owned the Arlington estate. Lee did serve as custodian of the property, which had fallen into disrepair by the time he returned to execute his father-in-law's will. By 1859, Lee had returned the property and its holdings to profitability and good order.

Robert E. Lee and his wife, Mary Anna, lived at Arlington House until 1861, when Virginia ratified an alliance with the Confederacy and seceded from the Union. Lee, who had been named a major general for the Virginia military forces in April 1861, feared for his wife's safety and anticipated the loss of their family inheritance. In May 1861, Lee wrote to Mary Anna saying:

• "War is inevitable, and there is not telling when it will burst around you . . . You have to move and make arrangements to go to some point of safety which you must select. The Mount Vernon plate and pictures ought to be secured. Keep quiet while you remain, and in your preparations . . . May God keep and preserve you and have mercy on all our people."
• Following the ratification of secession by Virginia, federal troops crossed the Potomac and, under Brig. Gen. Irvin McDowell, took up positions around Arlington. Following the occupation, military installations were erected at several locations around the 1,100-acre estate, including Fort Whipple (now Fort Myer) and Fort McPherson (now Section 11).
• Gen. Lee deeply regretted the loss of his home at Arlington. During the early stages of the war, foreseeing the probable loss of his home and belongings, Lee wrote to his wife about Arlington:
  • "It is better to make up our minds to a general loss. They cannot take away the remembrance of the spot, and the memories of those that to us rendered it sacred. That will remain to us as long as life will last, and that we can preserve."
  • Lee continued to feel responsible for the estate and earnestly hoped that the slaves who were left behind would be educated and freed, according to the provisions of George Washington Parke Custis' will.
  • The property was confiscated by the federal government when property taxes levied against Arlington estate were not paid in person by Mrs. Lee. The property was offered for public sale Jan. 11, 1864, and was purchased by a tax commissioner for "government use, for war, military, charitable and educational purposes."
  • Arlington National Cemetery was established by Brig. Gen. Montgomery C. Meigs, who commanded the garrison at Arlington House, appropriated the grounds June 15, 1864, for use as a military cemetery. His intention was to render the house uninhabitable should the Lee family ever attempt to return. A stone and masonry burial vault in the rose garden, 20 feet wide and 10 feet deep, and containing the remains of 1,800 Bull Run casualties, was among the first monuments to Union dead erected under Meigs' orders. Meigs himself was later buried within 100 yards of Arlington House with his wife, father and son; the final statement to his original order.
  • Neither Robert E. Lee, nor his wife, as title holder, ever attempted to publicly recover control of Arlington House. They were buried at Washington University (later renamed Washington and Lee University) where Lee had served as president. The couple never returned to the home George Washington Parke Custis had built and treasured.

  • After Gen. Lee's death in 1870, George Washington Custis Lee brought an action for ejectment in the Circuit Court of Alexandria (today Arlington) County, Va. Custis Lee, as eldest son of Gen. and Mrs. Lee, claimed that the land had been illegally confiscated and that, according to his grandfather's will, he was the legal owner. In December 1882, the U.S. Supreme Court, in a 5-4 decision, returned the property to Custis Lee, stating that it had been confiscated without due process. On March 3, 1883, the Congress purchased the property from Lee for $150,000.