Affect recognition is currently an active research area for machine learning researchers. Researchers are paying more and more attention to the assessment and the classification of emotions in people with mental disorders in order to propose monitoring healthcare systems and then to better assist patients. In this paper, a supervised learning method to classify Autism Spectrum Disorder (ASD) and to evaluate affect levels among autistic children is proposed. The use of methodologies for clinical patient monitoring and management is the focus of our research. YouTube video frames acquired in unconstrained environments and conditions of autistic children demonstrating typical autistic behaviors, as well as images of neurotypical people are used to evaluate the performance of the proposed approach. This paper also proposes an extended version of a dataset, including the additional affect labels corresponding to the affect levels of autistic children. Finally, experiments using different models were conducted to determine which architecture performed best. These experiments have shown very promising results for continuous affect recognition of children with ASD. Results show also that using a model that has only been trained on neurotypical subjects does not perform as great when the subjects are ASD children.