From the UnrealEd Reference Collection
(bold text emphasis mine)
<blockquote><font size="1" face="Verdana, Arial">quote:</font><hr>Creating and importing textures
Unreal textures are palettized, 256-color textures. Each texture in Unreal may have its own palette. The result is that you can achieve a lot of color variety by using textures with different palettes, while benefitting from the space savings of 8-bit textures.
The Unreal team creates textures in Fractal Painter and Photoshop, always working with the source art in 24-bit color. When a texture is complete, it is converted into a 256-color, palettized .pcx or .bmp file and imported into UnrealEd. To import a texture into UnrealEd, go into the texture browser (on the right side of the screen), and click on the "Import" button. When importing a texture, you can specify its name, the texture group it should appear in, and the package (.utx file) it should be saved in.
Unreal uses a texture mapping feature called "masking", where color zero is treated as if it were transparent. We use this for textures like vines, grating, and sprites. To create a masked texture, you will need to use a 256-color paint program to edit your image and apply color zero to the transparent portions of the texture. Then, import the texture and click on the "Masked" checkbox in the texture importing dialog. If you don't click on the "Masked" checkbox, the texture's mipmaps will not be properly masked.
When you import a texture, UnrealEd will automatically create mipmaps for the texture (mipmaps are lower resolution versions of the texture, used for antialiasing far-away polygons).
When you import two textures which use the same palette, UnrealEd makes note of this fact and takes advantage of it for optimization. In hardware rendering, the frame rate is partly dependent on the number of palettes visible in a scene, so there is a performance advantage to forcing textures to have the same palette. Programs like Image Alchemy can convert groups of 24-bit color images to 8-bit and force a common palette.<hr></blockquote>
Ok, so basically color zero in your pallette is your mask color. Try making it hot pink or something, so it won't be mistaken for anything else.
Later, when you use UnrealEd to import the texture, make sure the "Masked texture" box is checked.
--Nut