In the churchyard are the graves of Nancy, Unity, Diana and Pamela Mitford. With the exception of Pamela, each of them in different ways left their mark on 20th century public life. The Mitford family is a minor aristocratic English family with origins in Redesdale in Northumberland. The main family line had seats at Mitford Castle, Mitford Old Manor House, and from 1828 the then-newly-built Mitford Hall. Several heads of the family served as High Sheriff of Northumberland.

A junior line, with seats at Newton Park, Northumberland, and Exbury House, Hampshire descended via the historian William Mitford to his great-great-great-granddaughters, the “Mitford sisters”. This branch of the family had estates in Oxfordshire and Gloucestershire. The Mitford family was twice elevated to the British peerage, in 1802 and 1902, under the title Baron Redesdale. David Freeman-Mitford, 2nd Baron Redesdale was the father of the renowned sisters. The family seat was in Batsford, an inherited estate near Moreton-in-Marsh, but since he was a poor manager of finances, with seven children to feed and five servants to pay, he could not maintain the expense of this large house. So he bought and extended Asthall Manor in Swinbrook. He then built a new large house, named after the village, which appears as the family home in the books of his daughters Nancy and Jessica. This is located 1.5 miles (2.4 km) north of Swinbrook.

He was an irascible, distant figure, which may account for the various eccentricities of some of his daughters, caricatured by the journalist Ben Macintyre, as “Diana the Fascist, Jessica the Communist, Unity the Hitler-lover; Nancy the Novelist; Deborah the Duchess and Pamela the unobtrusive poultry connoisseur”. According to Jessica, Lord Redesdale wouldn’t receive any “outsiders” such as “Huns”, “Frogs”, Americans, Africans, and any other “foreigners”, which included other people’s children, most friends of the girls, and almost all young men. Exceptions were made for some relatives and selected neighbors.

In brief: Diana married the British fascist leader, Sir Oswald Moseley, who was interned during World War II and eventually moved to Paris. Their children became eminent in various fields from literature to Formula One motor racing.

Jessica, an avowed Communist, moved to the USA and was the author of ‘Hons and Rebels’, an account of her upbringing. She became an acclaimed investigative journalist. After her first marriage, disapproved of by her parents, she never spoke to her father again. Unity was the oddest one. At the end of her debutante season, she defied her parents and visited her sister Diana, who was not on speaking terms with her family on account of having left her first husband for Moseley. Unity met Oswald Mosley and became a devoted fascist. In 1933 she went to Germany, where she befriended Hitler. She lived on and off in Germany from 1933 to the outbreak of World War II. When Britain declared war on Germany and all Britons were expelled, Unity attempted suicide but succeeded only in inflicting serious brain damage. She was returned to England, all German hospital bills paid for by Hitler, where doctors decided it was too dangerous to remove the bullet, and she eventually died at the age of 33 of meningitis.

Nancy wrote successful novels [The Pursuit of Love (1945), Love in a Cold Climate (1949), and The Blessing (1951)], featuring characters based on her family, notably her father, which have been serialized on television. She also co-authored Nobless Oblige, which lampoons the English class system.

Deborah became the Duchess of Devonshire through her marriage to Lord Andrew Cavendish, second son of the 10th Duke of Devonshire. At the time, he was not expected to inherit the title but did so following the death in World War II of his brother. Pamela married a wealthy research spectroscopist and Oxford professor and settled in  Gloucestershire in comparative obscurity.

There was one son, Tom, who was killed in action in World War II.