The Baths with young kids?

[FOURunderFOUR]

We are a family of 7 (5 kids ages 15 mos to 6 1/2 years) plus my mom and dad. We will be in BVI from 7am - 2pm, so from what I have read we should be able to make the 9am Speedy's ferry over to the Virgin Gorda. I am wondering if the area is doable with young kids? My kids are used to "long" (2-3miles for them) hikes and rock scrambles. We would probably have to carry the little one in a backpack carrier as I can't imagine a stroller would work too well, right? Would that be doable? Is it worth the effort, or will we be happier (smarter/saner) to try something else/ Any other suggestions for a family with young kids?

Thanks