Remotely operated vehicles (ROVs) are often used for inspection in aquaculture net pens which serves the important purpose of localizing holes in the net and reporting potential irregularities and damages. Manual control of the vehicle inside a net pen, while simultaneously inspecting the net structure, is difficult and puts a lot of stress on the vehicle operators. Adaptation of new solutions that enables autonomous traversal of net pens where the vehicle maintains a fixed distance, heading, and velocity relative to the net is considered essential. One of the main challenges of such autonomous solutions is a robust and tight control of the vehicle's velocities. To target this challenge, this paper presents adaptive speed controllers for the surge and sway speeds of a remotely operated vehicle with unknown parameters and under the influence of unknown external disturbances. The stability properties of the controllers are proven through Lyapunov theory, and both simulations and field experiments demonstrate their ability to track the desired speeds through the use of a net following scheme.