No, having an adapter connected to your phone shouldn't affect your phone's performance. You should reboot the phone and, if still necessary, clear the cache partition too.
Assuming the adapter is for wired earphones, do any apps open, such as a music player app, when you plug in the adapter? If so, that app is most likely causing the problem. You could reboot into safe mode, which prevents any third-party apps from running, and see if your problem still occurs. If not, something you installed on your phone is causing the problem, which you would need to identify and uninstall.
