I think the thing that makes Antique Bakery considered yaoi are the hardcore explicit 14 (15 now? not sure) yaoi doujin that Yoshinaga did for the series. xD Though I really like the series itself for portraying gay characters without it being BL or yaoi. :)

But I loved Chikage and Kanda too! They were so cute! I wanted to see them more as well (I wanted more about Chikage's daughter and wife. That was perhaps my favorite side story. :D).