The highly dynamic nature of agro-ecosystems in space and time usually leads to high intra-class variance and low inter-class separability in the fine spatial resolution (FSR) remotely sensed imagery. This makes traditional classifiers essentially relying on spectral information for crop mapping from FSR imagery an extremely challenging task. To mine effectively the rich spectral and spatial information in FSR imagery, this paper proposed a Scale Sequence Object-based Convolutional Neural Network (SS-OCNN) that classifies images at the object level by taking segmented objects (crop parcels) as basic units of analysis, thus, ensuring that the boundaries between crop parcels are delineated precisely. These segmented objects were subsequently classified using a CNN model integrated with an automatically generated scale sequence of input patch sizes. This scale sequence can fuse effectively the features learned at different scales by transforming progressively the information extracted at small scales to larger scales. The effectiveness of the SS-OCNN was investigated using two heterogeneous agricultural areas with FSR SAR and optical imagery, respectively. Experimental results revealed that the SS-OCNN consistently achieved the most accurate classification results. The SS-OCNN, thus, provides a new paradigm for crop classification over heterogeneous areas using FSR imagery, and has a wide application prospect.