There is dialog configuration file named tag.config. You can find it in CuteSoft_Client/Cuteeditor/Dialogs folder.

 
The following code specifies which file will be used when the user click the insert links button.

<add pattern="A" tab="A" text="[[LINK]]" control="Tag_A.ascx" />

You can change its value and and point it to your own custom dialog.

Hope it helps.