This depends on the value you use for thickness. If you use the combined thickness of both rectangles, then the area must also be including both rectangles. If you use the thickness of one rectangle, use the area of one rectangle. Both methods should yield the same answer.
Should we consider the area above/below b to include both of the vertical rectangles or just the rectangle it is on when solving for shear stress?
This depends on the value you use for thickness. If you use the combined thickness of both rectangles, then the area must also be including both rectangles. If you use the thickness of one rectangle, use the area of one rectangle. Both methods should yield the same answer.