The increasing complexity of wireless communication conditions presents noteworthy obstacles to the adaptability and flexibility of cognitive radio systems. The latter can alleviate the problem of spectrum shortages through proper spectrum management. However, cognitive radio has difficulty distinguishing between signals of interest and interference, which can be a general enemy since it is undesirable because it influences the signal's quality. Efficient detection and characterization of interference in wireless communication networks is critical for ensuring strong security. However, in this research, we use the Mask R-CNN methodology to present a new concept for automatic modulation recognition and radio frequency interference detection. Moreover, this algorithm can segment, detect, and recognize several types of interference that can affect wireless communication systems, such as chirp interference (CI), continuous wave interference (CWI), and multiple continuous wave interference (MCWI) within the signal of the interest (SOI), as well as the modulation kind present in the SoI. Overall, the combination of these distinct techniques can be very valuable in the field of signal processing, especially in anti-jamming strategies in wireless networks. Moreover, the proposed approach showcases exceptional performance in the validation dataset. For radio frequency interference detection, it achieves a mean average precision (mAP) of 0.946 and a mean average recall (mAR) of 0.954. For automatic modulation recognition, it attains a mAP of 0.898 and a mAR of 0.916