The first reference to Bexhill, or Bexelei as it was originally called, was in a charter granted by King Offa of Mercia in 772AD. It is recorded that King Offa had 'defeated the men of Hastings' in 771AD.
At this time, the term Hastings would have referred to this whole area rather than the town itself as it does today. In the charter, King Offa established a church and religious community in Bexhill.
During the Norman Conquest of 1066 it appears that Bexhill was largely destroyed. The Domesday survey of 1086 records that the manor was worth £20 before the conquest, was 'waste' in 1066 and was worth £18 10s in 1086. King William I used the lands he had conquered to reward his knights and gave Bexhill manor to Robert, Count of Eu, with most of the Hastings area. Robert's grandson, John, Count of Eu, gave back the manor to the bishops of Chichester in 1148 and it is probable that the first manor house was built by the bishops at this time. The later manor house, the ruins of which can still be seen at the Manor Gardens in Bexhill Old Town, was built about 1250, probably on the instructions of St. Richard, Bishop of Chichester.The Manor House was the easternmost residence owned by the bishops and would have been used as a place to stay while travelling around or through the eastern part of their diocese. There were often disputes between the Bishops of Chichester and the Abbots of Battle Abbey, usually about land ownership in this area. In 1276 a large portion of Bexhill was made into a park for hunting and in 1447 Bishop Adam de Moleyns was given permission to fortify the Manor House.

In 1561 Queen Elizabeth I took possession of Bexhill Manor and three years later she gave it to Sir Thomas Sackville, Earl of Dorset. The Earls, later Dukes, of Dorset owned Bexhill until the mid 19th century. Their main residences were Buckhurst Place in Sussex and Knole House in Kent.