This issue was caused by wrong use of importlib, turns out this was wrong from the very beginning and that I never studied how reimporting of libs woks.
Now the speed of script reload seems not to be influenced by BlenderKit anymore.