Erro ao rodar etapa. | AWS Data Lake: processando dados com AWS EMR

Estou com problema para rodar a etapa da camada gold, tentei ler o log e entender, e me parece q ele nao consegue pegar o script... Os acessos estao certo e o codigo esta no bucket correto.

24/06/06 18:16:47 WARN Client: Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME.
24/06/06 18:17:02 INFO Client: Uploading resource file:/mnt/tmp/spark-15d0560f-055c-4733-a261-e637526dac7e/__spark_libs__13323943119012994685.zip -> hdfs://ip-172-31-41-17.ec2.internal:8020/user/hadoop/.sparkStaging/application_1717697735841_0001/__spark_libs__13323943119012994685.zip
24/06/06 18:17:03 INFO Client: Uploading resource file:/etc/spark/conf.dist/hive-site.xml -> hdfs://ip-172-31-41-17.ec2.internal:8020/user/hadoop/.sparkStaging/application_1717697735841_0001/hive-site.xml
24/06/06 18:17:03 INFO Client: Uploading resource file:/etc/hudi/conf.dist/hudi-defaults.conf -> hdfs://ip-172-31-41-17.ec2.internal:8020/user/hadoop/.sparkStaging/application_1717697735841_0001/hudi-defaults.conf
24/06/06 18:17:04 INFO ClientConfigurationFactory: Set initial getObject socket timeout to 2000 ms.
24/06/06 18:17:04 INFO Client: Uploading resource s3://ruano-datalake-emr/main.py -> hdfs://ip-172-31-41-17.ec2.internal:8020/user/hadoop/.sparkStaging/application_1717697735841_0001/main.py
24/06/06 18:17:05 INFO S3NativeFileSystem: Opening 's3://ruano-datalake-emr/main.py' for reading

[2024-06-06 19:04:21.499]Container exited with a non-zero exit code 13. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
Last 4096 bytes of stderr :
24/06/06 19:04:19 INFO SignalUtils: Registering signal handler for TERM
24/06/06 19:04:19 INFO SignalUtils: Registering signal handler for HUP
24/06/06 19:04:19 INFO SignalUtils: Registering signal handler for INT
24/06/06 19:04:19 INFO ApplicationMaster: ApplicationAttemptId: appattempt_1717700147634_0002_000002
24/06/06 19:04:20 INFO ApplicationMaster: Starting the user application in a separate Thread
24/06/06 19:04:20 INFO ApplicationMaster: Waiting for spark context initialization...
24/06/06 19:04:21 ERROR ApplicationMaster: User application exited with status 1
24/06/06 19:04:21 INFO ApplicationMaster: Final app status: FAILED, exitCode: 13, (reason: User application exited with status 1)
24/06/06 19:04:21 ERROR ApplicationMaster: Uncaught exception: 
org.apache.spark.SparkException: Exception thrown in awaitResult: 
    at org.apache.spark.util.SparkThreadUtils$.awaitResult(SparkThreadUtils.scala:56) ~[spark-common-utils_2.12-3.5.0-amzn-0.jar:3.5.0-amzn-0]
    at org.apache.spark.util.ThreadUtils$.awaitResult(ThreadUtils.scala:310) ~[spark-core_2.12-3.5.0-amzn-0.jar:3.5.0-amzn-0]
    at org.apache.spark.deploy.yarn.ApplicationMaster.runDriver(ApplicationMaster.scala:509) ~[spark-yarn_2.12-3.5.0-amzn-0.jar:3.5.0-amzn-0]
    at org.apache.spark.deploy.yarn.ApplicationMaster.run(ApplicationMaster.scala:268) ~[spark-yarn_2.12-3.5.0-amzn-0.jar:3.5.0-amzn-0]
    at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$3.run(ApplicationMaster.scala:937) ~[spark-yarn_2.12-3.5.0-amzn-0.jar:3.5.0-amzn-0]
    at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$3.run(ApplicationMaster.scala:936) ~[spark-yarn_2.12-3.5.0-amzn-0.jar:3.5.0-amzn-0]
    at java.security.AccessController.doPrivileged(AccessController.java:712) [?:?]
    at javax.security.auth.Subject.doAs(Subject.java:439) [?:?]
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1899) [hadoop-client-api-3.3.6-amzn-2.jar:?]
    at org.apache.spark.deploy.yarn.ApplicationMaster$.main(ApplicationMaster.scala:936) [spark-yarn_2.12-3.5.0-amzn-0.jar:3.5.0-amzn-0]
    at org.apache.spark.deploy.yarn.ApplicationMaster.main(ApplicationMaster.scala) [spark-yarn_2.12-3.5.0-amzn-0.jar:3.5.0-amzn-0]
Caused by: org.apache.spark.SparkUserAppException: User application exited with 1
    at org.apache.spark.deploy.PythonRunner$.main(PythonRunner.scala:112) ~[spark-core_2.12-3.5.0-amzn-0.jar:3.5.0-amzn-0]
    at org.apache.spark.deploy.PythonRunner.main(PythonRunner.scala) ~[spark-core_2.12-3.5.0-amzn-0.jar:3.5.0-amzn-0]
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) ~[?:?]
    at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
    at java.lang.reflect.Method.invoke(Method.java:568) ~[?:?]
    at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:741) ~[spark-yarn_2.12-3.5.0-amzn-0.jar:3.5.0-amzn-0]
24/06/06 19:04:21 INFO ApplicationMaster: Deleting staging directory hdfs://ip-172-31-45-126.ec2.internal:8020/user/hadoop/.sparkStaging/application_1717700147634_0002
24/06/06 19:04:21 INFO ShutdownHookManager: Shutdown hook called

Mergulhe em programação