Very interesting,

This is similar to the LIME (Local Interpretable Model-Agnostic Explanations) technique used on NLP, which tries to figure out which word most impacts the tagging result.