I think that I got some answers, after testing both the release candidate and the stable one.
I'm guessing that that the kernel has something to do with this, since I never seen this behavior in other linux flavors with 2.6 kernel + (but I have not tested the non-dsl flavor).
Ok, here goes - when using a usb mouse or keyboard, you must have (in my case. might be different for you) uhci usb disabled.
Now the differens is this way, some motherboards let's you choose is you want usb legacy enabled or disabled.
The differens is that on way you can have usb boot and keyboard, mouse support enabled, the other way you can only choose one.
As stated, I found a workaround - disabled uhci support, which worked (but it resulted in painfully slow usb stick)