The tower may have been built later than the house, possibly in the 1840s, and is an unusual Shetland example of a folly-type structure, even though it occupies the site of an actual Iron Age broch. The tower originally had upper storeys which were joined to steps by a footbridge which has now disintegrated. It was used at one stage as an astronomical observatory, which contained a large telescope, only the lens of which remains and which can be seen at Fetlar Interpretive Centre.

During the time of this first Fetlar-based Sir Arthur Nicolson changes on the estate, known throughout Scotland as the "Clearances", were to affect the whole social and economic structure of the island, and lay the foundation for a general decline in Fetlar's population which continued unbroken until the late 1980s. Sir Arthur enclosed much of his estate land for sheep and evicted the tenants from those parts of the island.

Probably around the 1840s, Sir Arthur built another folly-type structure in the island - the Round House at Gruting. It was originally intended as a summer house, but Sir Arthur is reputed to have once spent the night in there and heard inexplicable noises, so that ever after this the building was only used as an office for the collection of estate rents. Legend says he had used the stones from evicted tenants' houses to build the Round House and that the noises were the curses of these souls.

This first Fetlar-based Sir Arthur is also reputed to have had a son (also called Arthur) who died young, and local tradition tells a strange tale of him. Apparently, Sir Arthur had a vision in which he saw the legendary King Arthur who asked what he most wished for. When he replied that he wished for a son, King Arthur promised him this on condition that the son be named Arthur, and that he build a `praying house' dedicated to King Arthur. This is the legendary story behind the building of the tower at Brough. It must also be the most northerly Arthurian story in Britain.

When Sir Arthur died childless in 1863, the baronetcy passed to Arthur Bolt Nicolson, son of Captain James Nicolson of Aith in the Shetland Mainland, a cousin. He had spent much of his young life in Australia where he held a commission in the 4th King's Own Royal Regiment serving in New South Wales around 1831. He sold his commission and returned to Britain some years after that, but in 1853 returned to Victoria working for a time as Commissioner of Goldfields. He died in 1879 in Australia, and it is unlikely that he was ever in Fetlar. Although he inherited the title, he did not himself inherit the estate land in Fetlar. The first Sir Arthur had ensured a life-rent for his widow, Lady Eliza Jane Nicolson, and she continued to derive income from it until her death in 1891.

At this point, Captain James Nicolson's grandson Arthur T.B.R. Nicolson, who had inherited the title after his father died in 1879, inherited the use of the estate land, and became resident in Fetlar by 1893. When Sir Arthur T.B.R. Nicolson took over the estate, he complained of the appalling condition in which he found both house and stock. He states that the house had been virtually roofless since the death of Sir Arthur (Lady Eliza Nicolson, the first Sir Arthur's wife, had been living in Cheltenham). He wrote that "the tenants complained most bitterly to me of the way they had been treated by Lady Nicolson, saying that she would not do anything for their comfort. Some of them also gave this as the reason why they had not paid their rents."

The new laird seems to have made repairs to Brough Lodge so that he could reside there. He was brought up in Australia where he was born in 1842, and was educated at Melbourne College. His family was heavily involved in public life and he himself became a Justice of the Peace for Victoria and later on in Shetland.

He married Annie Rutherford, whose family have a long and colourful history in Australia and Scotland. Her father, John Rutherford, and his brothers, left Kildonan in Sutherland in the mid 19th century, and gradually amassed a fortune in land and stock in Victoria. Annie and her brothers and sisters grew up on various sheep stations and also at a town house called Belle Vue in Geelong, Victoria. Fetlar Interpretive Centre has original photographs of a house which is probably Belle Vue, and these are currently being digitised. Annie Nicolson wrote a daily diary of her life at Brough Lodge from when she went to live there to when she died in 1936, and has left us with an accurate record not only of life at Brough during that period, but also of events in the island generally.

On one very special occasion, the Coronation of King Edward VII, a party was held at Brough for the schoolchildren of the island. The one thing no-one knew until the next day, however, was that the Coronation had been postponed so that the King could be operated on. The celebration at Brough had been too early. A similar party was held for the Coronation of George V in 1911, but had itself to be postponed for several days because of bad weather. The following is the entry in Lady Nicolson's diary for the day the party finally took place:

"Helen and Barbara made 200 little cakes in the afternoon - after supper we made up 100 bags of sweets (5oz in each) and 80 bags with 1 orange, 1 bun, 1 piece of baker's bun cake and 1 piece of cake Helen made. When the children finally arrived on 24th June Arthur and Arthur J. put up goals in the Lower Park for football, and I took out "Aunt Sally", also the hobby horse. The children came at 3 with flags &c, and singing and in great glee. We met them at the Courtyard door, and when they were all in they sang the National Anthem, then got their mugs & cakes &c. Then they came into the hall and sang, then went to the Lower Park to have games. Then they came in and had another tea &c. (no fruit the 2nd time), and as they were leaving, Arthur gave each a medal and they got a bag with cake and an orange in it, and a smaller bag of sweets, and about 8 they left all very happy. We had a "Welcome" outside the Court Yard door and "God Save the King" on the wall inside, and flags &c."

Sir Arthur T.B.R. Nicolson died in 1917, leaving behind five children: Arthur J., Stanley, Lionel, Vera and Eardley. Eardley was born with brain damage, and a succession of local nurses were employed to help look after him, although Lady Nicolson seems to have taken full control when he had one of his "turns", which could be frequent. Eardley died in his twenties, only outliving his father by a few years.

All the other boys were educated at Merchiston, and later at Edinburgh University where they studied law. They came back to Fetlar most holidays, but occasionally spent time with relatives (generally on the Rutherford side) in England. Arthur J. Nicolson left a diary of one of these holidays in 1899. It is interesting for some of the events he witnessed in London at the start of the Boer War when troops were leaving for southern Africa.

During World War One, the three eldest brothers all held commissions in the Royal Naval Reserve. Arthur J. Nicolson served on HMS Caroline, one of the ships in the British fleet present at the Battle of Jutland. HMS Caroline still exists in Belfast and is the oldest surviving British battleship from that era.

Arthur J. Nicolson took over the running of the estate after his father's death. The estate was entailed to the baronetcy at the time, meaning that whoever inherited the baronetcy also inherited the estate. Sir Arthur J. Nicolson disentailed the estate, something commonly done by the British aristocracy after World War 1 in order to keep property in families by allowing inheritance outside the rules of the Baronetcy. In the case of Brough, however, this had ironic consequences for the future.

In 1927 Sir Arthur married Dolores Elaine Cubbon, a New York debutante who had been raised by her aunts. Her association with Sir Arthur seems to have happened quite suddenly, so suddenly that her own close friends were taken completely by surprise when she announced her forthcoming marriage, and most of them had yet to meet Sir Arthur. After a high society wedding in New York, Dolores' trip to Fetlar ended in London, and the marriage was annulled soon afterwards.

Sir Arthur J. Nicolson continued to run the estate in Fetlar, however, until he died of a stroke on 25th April 1952. He left the estate to his brother Stanley. Sir Stanley and his wife Jean Landles had been living at Kinglassie House in Fife before he inherited the Brough Estate. When Stanley died in 1961, the full implications of Arthur's disentailing of the estate began to unfold. Stanley was able to leave the estate to his wife although she could not inherit the baronetcy. On her death in 1988 the estate and house finally passed out of the family altogether when Lady Nicolson left it to her own niece, who is now transferring the property to Brough Lodge Trust.

The third brother of the family, Lionel, moved to London and worked for various large companies, including Marconi. He married Kathleen Mary Moon, and their daughter Anne, who lives in the south of England, is one of the only two surviving members of the family.

The only sister of the family, Vera, married Lord Herschell in 1919. The wedding took place at Brough Lodge, and the Earl of Zetland appeared off the island decked in bunting. Lord Herschell was Lord in Waiting to the King, and had been present at the coronations of Edward VII and George V. Vera and Lord Herschell took a house in the Isle of Wight for a few summers in the 1920s, and named it Fetlar House. They received regular visits from Queen Mary during Cowes Week. Their son, the current Lord Herschell, is the only other surviving member of the family. He has one daughter.

Lady Jean Nicolson, Sir Stanley's wife, lived on in Brough Lodge until the 1970s when she was too ill to manage on her own. She transferred first to a sheltered house in the island, and then to a home in Lerwick where she died in 1987. Since she left the house, it has remained empty, although a large number of artefacts from the house can be seen at Fetlar Interpretive Centre, and the large volume of family papers is now kept at the Shetland Archives in Lerwick.