The number of LTE (Long-Term Evolution) users and their applications has increased significantly in the last decade, which increased the demand on the mobile network. LTE-Advanced (LTE-A) comes with many features that can support this increasing demand. LTE-A supports Heterogeneous Networks (HetNets) deployment, in which it consists of a mix of macro-cells, remote radio heads, and low power nodes such as Pico-cells, and Femto-cells. Embedding this mix of base-stations in a macro-cellular network allows for achieving significant gains in coverage, throughput and system capacity compared to the use of macrocells only. These base-stations can operate on the same wireless channel as the macro-cellular network, which will provide higher spatial reuse via cell splitting. Also, it allows network operators to support higher data traffic by offloading it to smaller cells, such as Femto-cells. Hence, it enables network operators to provide their growing number of users with the required Quality of Service (QoS) that meets with their service demands. In-order for the network operators to make the best out of the heterogeneous LTE-A network, they need to use QoS-based packet scheduling algorithms that can efficiently manage the spectrum resources in the HetNets deployment. In this paper, we survey Quality of Service (QoS) based packet scheduling algorithms that were proposed in the literature for the use of packet scheduling in Heterogeneous LTE-A Networks. We start by explaining the concepts of QoS in LTE, heterogeneous LTE-A networks, and how traffic is classified within a packet scheduling architecture for heterogeneous LTE-A networks. Then, by summarising the proposed QoS-based packet scheduling algorithms in the literature for Heterogeneous LTE-A Networks, and for Femtocells LTE-A Networks. And finally, we provide some concluding remarks in the last section.