The agricultural landscape can be interpreted at different semantic levels, such as fine low-level crop (LLC) classes (e.g., Wheat, Almond, and Alfalfa) and broad high-level crop (HLC) classes (e.g., Winter crops, Tree crops, and Forage). The LLC and HLC are hierarchically correlated with each other, but such intrinsically hierarchical relationships have been overlooked in previous crop classification studies in remote sensing. In this research, a novel Iterative Deep Learning (IDL) framework was proposed for the classification of complex agricultural landscapes using remotely sensed imagery. The IDL adopts an object-based convolutional neural network (OCNN) as the basic classifier for both the LLC and HLC classifications, which has the advantage of maintaining precise crop parcel boundaries. In IDL, the HLC classification implemented by the OCNN is conditional upon the LLC classification probabilities, whereas the HLC probabilities combined with the original imagery are, in turn, re-used as inputs to the OCNN to enhance the LLC classification. Such an iterative updating procedure forms a Markov process, where both the LLC and HLC classifications are refined and evolve collaboratively. The effectiveness of the IDL was tested on two heterogeneous agricultural fields using fine spatial resolution (FSR) SAR and optical imagery. The experimental results demonstrate that the iterative process of IDL helps to resolve contradictions within the class hierarchies. The new proposed IDL consistently increased the accuracies of both the LLC and HLC classifications with iteration, and achieved the highest accuracies for each at four iterations. The average overall accuracies were 88.4% for LLC and 91.2% for HLC, for both study sites, far greater than the accuracies of the state-of-the-art benchmarks, including the pixel-wise CNN (81.7% and 85.9%), object-based image analysis (OBIA) (84.0% and 85.8%), and OCNN (84.0% and 88.4%). To the best of our knowledge, the proposed model is the first to identify and use the relationship between the class levels in an ontological hierarchy in a remote sensing classification process. It is applied here to increase progressively the accuracy of classification at two levels for a complex agricultural landscape. As such IDL represents an entirely new paradigm for remote sensing image classification. Moreover, the promising results demonstrate the great potential of the proposed IDL with wide application prospect.