The M.B. quotes the Shelah saying to remove Tefillin shel Rosh with the weaker hand, to show קשה עלי פרידתכם. (Not Tefillin shel Yad.)
On the other hand, mitzvos are generally done with the stronger hand, and if one is saying the Bracha of לשמור חוקיו upon removing Tefillin at the end of the day, how is this any different from taking down the Sukkah or bringing in dishes to the Sukkah?
So, is it about showing respect, as in leaving the Mikdash slowly, or is it about guilt feelings?