Deep Convolutional Neural Networks (CNNs) have achieved human-level performance in edge detection. However, there have not been enough studies on how to efficiently utilize the parameters of the neural network in edge detection applications. Therefore, the associated memory and energy costs remain high. In this paper, inspired by Depthwise Separable Convolutions and deformable convolutional networks (Deformable-ConvNet), we aim to address current inefficiencies in edge detection applications. To this end, we propose a new architecture, which we refer to as Lightweight Edge Detection Network (LEON ). The proposed approach is designed to integrate the advantages of the deformable unit and DepthWise Separable convolutions architecture to create a lightweight backbone employed for efficient feature extraction. As we show, we achieve state-of-the-art accuracy while significantly reducing the complexity by carefully choosing proper components for edge detection purposes. Our results on BSDS500 and NYUDv2 demonstrate that LEON outperforms the current lightweight edge detectors while requiring only 500k parameters. It is worth mentioning that we train the network from scratch without using pre- trained weights.