Should I be worried about ambient light?

Yes! Ambient light will have a very noticeable effect on your image no matter how good the projector, lens or rest of your kit. Try to keep ambient light to an absolute minimum to maximize the impact of the projected image. Remember that your screen reflects light from any source so the darkest it can get is with the ambient light subdued as much as possible. This "black level" has a direct effect on perceived contrast ratio.