Very nice reflection,

It's always good to experiment with new approaches and ideas.

However, When Bert was created by Google, they make the choice to only encoder layers, as on the other side GPT is based only on decoder layers. Basically, this choice depends on learning objectives that do not need a decoder. So, did you really need a decoder in your case? I don't think.