Estou com problemas na execução do script na step do cluster EMR. Segue log:

Exception in thread "main" org.apache.spark.SparkException: Application application_1745163121497_0001 finished with failed status
    at org.apache.spark.deploy.yarn.Client.run(Client.scala:1342)
    at org.apache.spark.deploy.yarn.YarnClusterApplication.start(Client.scala:1775)
    at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:1112)
    at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:200)
    at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:223)
    at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:92)
    at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1204)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1213)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
25/04/20 15:33:48 INFO ShutdownHookManager: Shutdown hook called
25/04/20 15:33:48 INFO ShutdownHookManager: Deleting directory /mnt/tmp/spark-42fd5de2-5d3c-4fbe-87e3-aa86c4d62f6f
25/04/20 15:33:48 INFO ShutdownHookManager: Deleting directory /mnt/tmp/spark-291ea646-2c43-491d-9b3c-b39fd8b14d1d
Command exiting with ret '1'

Vi que já teve um tópico sobre esse mesmo problema e não vi solução.