Quantitative imaging is an emerging field that may allow prediction of oncological outcomes. We investigate whether radiomics and deep learning can predict outcomes in metastatic non-small cell lung cancer utilizing randomized trials of PD-1 inhibitors + /- stereotactic ablative body radiotherapy: PEMBRO-RT(NCT02492568), NIVORAD(ACTRN12616000352404) and MDACC(NCT02444741). A random forest model developed on PEMBRO-RT using radiomics features had an AUC of 0.57 for prediction of per-lesion progressive disease on immunotherapy compared to an AUC of 0.92 for a deep learning model. A random forest survival model using radiomics features for overall survival (progression free survival) had a concordance index of 0.63(0.59) and improved to 0.67(0.65) by adding clinical features, including PD-L1 and treatment arm. Validation on NIVORAD and MDACC revealed reduced AUCs. Overall, a deep learning compared to a radiomics model demonstrated excellent predictive value for per-lesion progressive disease for patients on immunotherapy. Models had reduced performance on external validation. Research improving generalizability is required for clinical translation.