I have no idea what's happening as in the product (the answer is (a)) there is a new ring being formed on the left benzene. But as I studied after leaves, a carbocation is then formed on stabilized by the right ring.
Can someone please explain the rest of the steps of this reaction? The thing I don't get is how does the three-membered ring opening happens. I attempted the question, but the answer that I came up with isn't the one matching in the book. Where did I go wrong with the mechanism?
Your first and third structures are good but the arrow pushing in the second structure doesn't get the job done. As you have noted, diol 1 forms carbocation 2a the more stable of the two options. Now the lefthand cyclopropane bond with its pair of electrons migrates to the cationic center to form resonance-stabilized species 2b and on to ketone 3. This type of structure is a candidate for a Nazarov cyclization. The arrows in 4 may be drawn in the other direction as well. Rearomatization of 5 by loss of a proton and tautomerization leads to the product 6. [Note: Structures 2b and 4 are identical. Just a reprotonation of 3.]
ADDENDUM: To answer your original question, yes it is a pinacol rearrangement. The bond that migrates (2a --> 2b) just happens to be attached via the cyclopropane ring to the site to which it is migrating. So, 1 --> 3 is a pinacol rearrangement.
The first generated intermediate, an α-hydroxycarbenium ion, rearranges through a 1,2-alkyl shift to produce the carbonyl compound. If two of the substituents form a ring, the Pinacol Rearrangement can constitute a ring-expansion or ring-contraction reaction.