Background: The dysregulation of gene expression is one of the key molecular features of colorectal cancer (CRC) development. This study aimed to investigate whether such dysregulation is reflected in rectal swab specimens of CRC patients and to evaluate its potential as a non-invasive approach for screening. Methods: We compared the expression level of 14 CRC-associated genes in tumor and adjacent non-tumor tissue of CRC patients and examined the correlation of their levels in tissue with paired rectal swab specimens. The level of these 14 genes in rectal swab specimens was compared among patients with CRC or polyp and control subjects, and the diagnostic potential of each dysregulated gene and the gene panel were evaluated. Results: The expression of CXCR2, SAA, COX1, PPARδ, PPARγ, Groγ, IL8, p21, c-myc, CD44 and CSF1 was significantly higher in CRC, and there was a significant correlation in the levels of most of them between the CRC and rectal swab specimens. In the training study, we showed that CD44, IL8, CXCR2 and c-myc levels were significantly higher in the rectal swab specimens of the CRC patients. Such result was confirmed in the validation study. A panel of these four genes was developed, and ROC analysis showed that this four-gene panel could identify CRC patients with an AUC value of 0.83 and identify overall polyp and precancerous adenoma patients with AUC values of 0.6522 and 0.7322, respectively. Finally, the predictive study showed that the four-gene panel demonstrated sensitivities of 63.6%, 76.9% and 88.9% in identifying overall polyp, precancerous adenoma and CRC patients, respectively, whereas the specificity for normal subjects was 72.2%. Conclusion: The expression of CRC-associated genes in rectal swab specimens reflects the dysregulation status in colorectal tissue, and the four-gene panel is a potential non-invasive biomarker for early precancerous adenoma and CRC screening.