Just-in-time defect prediction (JIT-DP) is used to predict the defect-proneness of a commit and just-in-time defect localization (JIT-DL) is used to locate the exact buggy positions (defective lines) in a commit. Recently, various JIT-DP and JIT-DL techniques have been proposed, while most of them use a post-mortem way (e.g., code entropy, attention weight, LIME) to achieve the JIT-DL goal based on the prediction results in JIT-DP. These methods do not utilize the label information of the defective code lines during model building. In this paper, we propose a unified model JIT-Smart, which makes the training process of just-in-time defect prediction and localization tasks a mutually reinforcing multi-task learning process. Specifically, we design a novel defect localization network (DLN), which explicitly introduces the label information of defective code lines for supervised learning in JIT-DL with considering the class imbalance issue. To further investigate the accuracy and cost-effectiveness of JIT-Smart, we compare JIT-Smart with 7 state-of-the-art baselines under 5 commit-level and 5 line-level evaluation metrics in JIT-DP and JIT-DL. The results demonstrate that JIT-Smart is statistically better than all the state-of-the-art baselines in JIT-DP and JIT-DL. In JIT-DP, at the median value, JIT-Smart achieves F1-Score of 0.502, AUC of 0.899, Recall@20%Effort of 0.823, Effort@20%Recall of 0.009 and Popt of 0.944. In JIT-DL, at the median value, JIT-Smart achieves Top-5 Accuracy of 0.54 and Top-10 Accuracy of 0.393, Recall@20%Effort${line}$ of 0.712, Effort@20%Recall${line}$ of 0.098 and IFA$_{line}$ of 0.079, which improves the best performing baseline by 154.72%, 84.64%, 242.31%, 69.18% and 99.27%, respectively. Besides, JIT-Smart also achieves the best performance compared with the state-of-the-art baselines in cross-project evaluation.