The actual manuscript from Rayleigh [1] explains it better: the area is the entire area of the vessel the oil was placed in, and the thing actually being measured was how much oil was required for it cover the whole area.
He used a fixed area (a 33 inch diameter bowl) and measured the weight of oil required to just about calm the entire water. That turned out to be 0.81 milligrams.
Some powder is added to the water, which covers the surface of the water but not the oil patch (which is circular). Then the oil patch diameter is measured.
When we did it in high school (70's) we just used compound that had a long chain (soap?) and only one end dissolved in the water. It was very easy to measure and calculate the size of the molecule . We had a series of these simple experiments. Another I recall was measure the speed at which certain volatile compounds moved through the air.
I definitely learned that all science doesn't have to involve complex equipment.
The original way was to cover the surface of a round bowl with oil. It certainly makes a lot more sense to me than trying to measure a floating disk of oil.
