Abstract Background Physical activity (PA) is an established protective factor for colorectal cancer (CRC), but it is unclear if genetic variants modify this effect. To investigate this possibility, we conducted a genome-wide gene–PA interaction analysis. Methods Using logistic regression and two-step and joint tests, we analyzed interactions between common genetic variants across the genome and PA in relation to CRC risk. Self-reported PA levels were categorized as active (≥ 8.75 MET-h/wk) vs. inactive (< 8.75 MET-h/wk) and as study- and sex-specific quartiles of activity. Results PA had an overall protective effect on CRC (OR [active vs. inactive] = 0.85; 95%CI = 0.81–0.90). The two-step GxE method identified an interaction between rs4779584, an intergenic variant near the GREM1 and SCG5 genes, and PA for CRC risk (p-interaction = 2.6×10 − 8 ). Stratification by genotype at this locus showed a significant reduction in CRC risk by 20% in active vs. inactive participants with the CC genotype (OR = 0.80; 95%CI = 0.75–0.85), but no significant PA–CRC association among CT or TT carriers. When PA was modeled as quartiles, the 1-d.f. GxE test identified that rs56906466, an intergenic variant near the KCNG1 gene, modified the association between PA and CRC (p-interaction = 3.5×10 − 8 ). Stratification at this locus showed that increase in PA (highest vs. lowest quartile) was associated with a lower CRC risk solely among TT carriers (OR = 0.77; 95%CI = 0.72–0.82). Conclusions In summary, we identified two genetic variants that modified the association between PA and CRC risk. One of them, related to GREM1 and SCG5 , suggests that the bone morphogenetic protein (BMP)-related, inflammatory, and/or insulin signaling pathways may be associated with the protective influence of PA on colorectal carcinogenesis.