Access AWS athena through JPA spring for download

Question

I am trying to use AWS athena using spring boot jpa datasource. I tried to configure the data source with the given properties.

spring.datasource.driver-class-name=com.amazonaws.athena.jdbc.AthenaDriver spring.datasource.url=jdbc:awsathena://athena.us-east-1.amazonaws.com:443/default spring.datasource.username=***** spring.datasource.password=*** spring.datasource.tomcat.connectionProperties=s3_staging_dir=***** 

I get below exception

  Caused by:org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.orm.jpa.JpaVendorAdapter]: Factory method 'jpaVendorAdapter' threw exception; nested exception is java.lang.IllegalArgumentException: URL must start with 'jdbc' at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:189) ~[spring-beans-4.3.7.RELEASE.jar:4.3.7.RELEASE] at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(Constructo`enter code here`rResolver.java:588) ~[spring-beans-4.3.7.RELEASE.jar:4.3.7.RELEASE] ... 51 common frames omitted Caused by: java.lang.IllegalArgumentException: URL must start with 'jdbc' at org.springframework.util.Assert.isTrue(Assert.java:92) ~[spring-core-4.3.7.RELEASE.jar:4.3.7.RELEASE] at org.springframework.boot.jdbc.DatabaseDriver.fromJdbcUrl(DatabaseDriver.java:268) ~[spring-boot-1.5.2.RELEASE.jar:1.5.2.RELEASE] at org.springframework.boot.autoconfigure.orm.jpa.DatabaseLookup.getDatabase(DatabaseLookup.java:73) ~[spring-boot-autoconfigure-1.5.2.RELEASE.jar:1.5.2.RELEASE] at org.springframework.boot.autoconfigure.orm.jpa.JpaProperties.determineDatabase(JpaProperties.java:139) ~[spring-boot-autoconfigure-1.5.2.RELEASE.jar:1.5.2.RELEASE] at org.springframework.boot.autoconfigure.orm.jpa.JpaBaseConfiguration.jpaVendorAdapter(JpaBaseConfiguration.java:105) ~[spring-boot-autoconfigure-1.5.2.RELEASE.jar:1.5.2.RELEASE] at org.springframework.boot.autoconfigure.orm.jpa.HibernateJpaAutoConfiguration$$EnhancerBySpringCGLIB$$720f8624.CGLIB$jpaVendorAdapter$4(<generated>) ~[spring-boot-autoconfigure-1.5.2.RELEASE.jar:1.5.2.RELEASE] at org.springframework.boot.autoconfigure.orm.jpa.HibernateJpaAutoConfiguration$$EnhancerBySpringCGLIB$$720f8624$$FastClassBySpringCGLIB$$9766cf.invoke(<generated>) ~[spring-boot-autoconfigure-1.5.2.RELEASE.jar:1.5.2.RELEASE] at org.springframework.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228) ~[spring-core-4.3.7.RELEASE.jar:4.3.7.RELEASE] 

Answer 1

Has anyone got a solution for this? I tried with the following config:

 @Configuration 

public class MyDataSourceConfiguration {

 @Bean @ConfigurationProperties(prefix = "blaze") @Qualifier("myDataSource") public DataSource myDataSource() { return DataSourceBuilder.create().build(); } @Bean @Qualifier("myEntityManager") public LocalContainerEntityManagerFactoryBean myEntityManagerFactory( EntityManagerFactoryBuilder builder) { Map<String, Object> properties = new HashMap(); properties.put("S3OutputLocation", "s3://mybucketdetails"); properties.put(AvailableSettings.DIALECT, MySQLDialect.class); properties.put("ProxyHost","myproxy"); properties.put("ProxyPort","1111"); properties.put("AwsCredentialsProviderClass","com.simba.athena.amazonaws.auth.DefaultAWSCredentialsProviderChain"); return builder .dataSource(myDataSource()) .packages(MyClass.class) .persistenceUnit("myUnit") .properties(properties) .build(); } 

}

My AWS credentials are available for DefaultAWSCredentialsProviderChain, but when I try to start the Spring Boot application, I get the following exception:

 java.sql.SQLNonTransientConnectionException: [Simba][JDBC](10100) Connection Refused: [Simba][JDBC](11640) Required Connection Key(s): S3OutputLocation; [Simba][JDBC](11480) Optional Connection Key(s): AwsCredentialsProviderArguments, AwsCredentialsProviderClass, BinaryColumnLength, ComplexTypeColumnLength, ConnectTimeout, MaxCatalogNameLength, MaxColumnNameLength, MaxErrorRetry, MaxSchemaNameLength, MaxStreamErrorRetry, MaxTableNameLength, ProxyHost, ProxyPort, ProxyPWD, ProxyUID, PWD, RowsToFetchPerBlock, S3OutputEncOption, Schema, SocketTimeout, StringColumnLength, UID, UseArraySupport, UseAwsLogger, UseResultsetStreaming, Workgroup at com.simba.athena.exceptions.ExceptionConverter.toSQLException(Unknown Source) at com.simba.athena.jdbc.common.BaseConnectionFactory.checkResponseMap(Unknown Source) at com.simba.athena.jdbc.common.BaseConnectionFactory.doConnect(Unknown Source) at com.simba.athena.jdbc.common.AbstractDriver.connect(Unknown Source) at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:119) at com.zaxxer.hikari.pool.PoolBase.newConnection(PoolBase.java:369) at com.zaxxer.hikari.pool.PoolBase.newPoolEntry(PoolBase.java:198) at com.zaxxer.hikari.pool.HikariPool.createPoolEntry(HikariPool.java:467) at com.zaxxer.hikari.pool.HikariPool.checkFailFast(HikariPool.java:541) at com.zaxxer.hikari.pool.HikariPool.<init>(HikariPool.java:115) at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:112) at org.hibernate.engine.jdbc.connections.internal.DatasourceConnectionProviderImpl.getConnection(DatasourceConnectionProviderImpl.java:122) at org.hibernate.engine.jdbc.env.internal.JdbcEnvironmentInitiator$ConnectionProviderJdbcConnectionAccess.obtainConnection(JdbcEnvironmentInitiator.java:180) at org.hibernate.engine.jdbc.env.internal.JdbcEnvironmentInitiator.initiateService(JdbcEnvironmentInitiator.java:68) at org.hibernate.engine.jdbc.env.internal.JdbcEnvironmentInitiator.initiateService(JdbcEnvironmentInitiator.java:35) at org.hibernate.boot.registry.internal.StandardServiceRegistryImpl.initiateService(StandardServiceRegistryImpl.java:94) at org.hibernate.service.internal.AbstractServiceRegistryImpl.createService(AbstractServiceRegistryImpl.java:263) at org.hibernate.service.internal.AbstractServiceRegistryImpl.initializeService(AbstractServiceRegistryImpl.java:237) at org.hibernate.service.internal.AbstractServiceRegistryImpl.getService(AbstractServiceRegistryImpl.java:214) at org.hibernate.id.factory.internal.DefaultIdentifierGeneratorFactory.injectServices(DefaultIdentifierGeneratorFactory.java:152) at org.hibernate.service.internal.AbstractServiceRegistryImpl.injectDependencies(AbstractServiceRegistryImpl.java:286) at org.hibernate.service.internal.AbstractServiceRegistryImpl.initializeService(AbstractServiceRegistryImpl.java:243) at org.hibernate.service.internal.AbstractServiceRegistryImpl.getService(AbstractServiceRegistryImpl.java:214) at org.hibernate.boot.internal.InFlightMetadataCollectorImpl.<init>(InFlightMetadataCollectorImpl.java:179) at org.hibernate.boot.model.process.spi.MetadataBuildingProcess.complete(MetadataBuildingProcess.java:119) at org.hibernate.jpa.boot.internal.EntityManagerFactoryBuilderImpl.metadata(EntityManagerFactoryBuilderImpl.java:904) at org.hibernate.jpa.boot.internal.EntityManagerFactoryBuilderImpl.build(EntityManagerFactoryBuilderImpl.java:935) at org.springframework.orm.jpa.vendor.SpringHibernateJpaPersistenceProvider.createContainerEntityManagerFactory(SpringHibernateJpaPersistenceProvider.java:57) at org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean.createNativeEntityManagerFactory(LocalContainerEntityManagerFactoryBean.java:365) at org.springframework.orm.jpa.AbstractEntityManagerFactoryBean.buildNativeEntityManagerFactory(AbstractEntityManagerFactoryBean.java:390) at org.springframework.orm.jpa.AbstractEntityManagerFactoryBean.afterPropertiesSet(AbstractEntityManagerFactoryBean.java:377) at org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean.afterPropertiesSet(LocalContainerEntityManagerFactoryBean.java:341) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1837) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1774) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:593) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:515) at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:320) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222) at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:318) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199) at org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1105) at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:867) at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:549) at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:142) at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:775) at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:397) at org.springframework.boot.SpringApplication.run(SpringApplication.java:316) at org.springframework.boot.SpringApplication.run(SpringApplication.java:1260) at org.springframework.boot.SpringApplication.run(SpringApplication.java:1248) AwsCredentialsProviderArguments, AwsCredentialsProviderClass, BinaryColumnLength, ComplexTypeColumnLength, ConnectTimeout, MaxCatalogNameLength, MaxColumnNameLength, MaxErrorRetry, MaxSchemaNameLength, MaxStreamErrorRetry, MaxTableNameLength, ProxyHost, ProxyPort, ProxyPWD, ProxyUID, PWD, RowsToFetchPerBlock, java.sql.SQLNonTransientConnectionException: [Simba][JDBC](10100) Connection Refused: [Simba][JDBC](11640) Required Connection Key(s): S3OutputLocation; [Simba][JDBC](11480) Optional Connection Key(s): AwsCredentialsProviderArguments, AwsCredentialsProviderClass, BinaryColumnLength, ComplexTypeColumnLength, ConnectTimeout, MaxCatalogNameLength, MaxColumnNameLength, MaxErrorRetry, MaxSchemaNameLength, MaxStreamErrorRetry, MaxTableNameLength, ProxyHost, ProxyPort, ProxyPWD, ProxyUID, PWD, RowsToFetchPerBlock, S3OutputEncOption, Schema, SocketTimeout, StringColumnLength, UID, UseArraySupport, UseAwsLogger, UseResultsetStreaming, Workgroup at com.simba.athena.exceptions.ExceptionConverter.toSQLException(Unknown Source) at com.simba.athena.jdbc.common.BaseConnectionFactory.checkResponseMap(Unknown Source) at com.simba.athena.jdbc.common.BaseConnectionFactory.doConnect(Unknown Source) at com.simba.athena.jdbc.common.AbstractDriver.connect(Unknown Source) at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:119) at com.zaxxer.hikari.pool.PoolBase.newConnection(PoolBase.java:369) at com.zaxxer.hikari.pool.PoolBase.newPoolEntry(PoolBase.java:198) at com.zaxxer.hikari.pool.HikariPool.createPoolEntry(HikariPool.java:467) at com.zaxxer.hikari.pool.HikariPool.checkFailFast(HikariPool.java:541) at com.zaxxer.hikari.pool.HikariPool.<init>(HikariPool.java:115) at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:112) at org.hibernate.engine.jdbc.connections.internal.DatasourceConnectionProviderImpl.getConnection(DatasourceConnectionProviderImpl.java:122) at org.hibernate.engine.jdbc.env.internal.JdbcEnvironmentInitiator$ConnectionProviderJdbcConnectionAccess.obtainConnection(JdbcEnvironmentInitiator.java:180) at org.hibernate.engine.jdbc.env.internal.JdbcEnvironmentInitiator.initiateService(JdbcEnvironmentInitiator.java:68) at org.hibernate.engine.jdbc.env.internal.JdbcEnvironmentInitiator.initiateService(JdbcEnvironmentInitiator.java:35) at org.hibernate.boot.registry.internal.StandardServiceRegistryImpl.initiateService(StandardServiceRegistryImpl.java:94) at org.hibernate.service.internal.AbstractServiceRegistryImpl.createService(AbstractServiceRegistryImpl.java:263) at org.hibernate.service.internal.AbstractServiceRegistryImpl.initializeService(AbstractServiceRegistryImpl.java:237) at org.hibernate.service.internal.AbstractServiceRegistryImpl.getService(AbstractServiceRegistryImpl.java:214) at org.hibernate.id.factory.internal.DefaultIdentifierGeneratorFactory.injectServices(DefaultIdentifierGeneratorFactory.java:152) at org.hibernate.service.internal.AbstractServiceRegistryImpl.injectDependencies(AbstractServiceRegistryImpl.java:286) at org.hibernate.service.internal.AbstractServiceRegistryImpl.initializeService(AbstractServiceRegistryImpl.java:243) at org.hibernate.service.internal.AbstractServiceRegistryImpl.getService(AbstractServiceRegistryImpl.java:214) at org.hibernate.boot.internal.InFlightMetadataCollectorImpl.<init>(InFlightMetadataCollectorImpl.java:179) at org.hibernate.boot.model.process.spi.MetadataBuildingProcess.complete(MetadataBuildingProcess.java:119) at org.hibernate.jpa.boot.internal.EntityManagerFactoryBuilderImpl.metadata(EntityManagerFactoryBuilderImpl.java:904) at org.hibernate.jpa.boot.internal.EntityManagerFactoryBuilderImpl.build(EntityManagerFactoryBuilderImpl.java:935) at org.springframework.orm.jpa.vendor.SpringHibernateJpaPersistenceProvider.createContainerEntityManagerFactory(SpringHibernateJpaPersistenceProvider.java:57) at org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean.createNativeEntityManagerFactory(LocalContainerEntityManagerFactoryBean.java:365) at org.springframework.orm.jpa.AbstractEntityManagerFactoryBean.buildNativeEntityManagerFactory(AbstractEntityManagerFactoryBean.java:390) at org.springframework.orm.jpa.AbstractEntityManagerFactoryBean.afterPropertiesSet(AbstractEntityManagerFactoryBean.java:377) at org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean.afterPropertiesSet(LocalContainerEntityManagerFactoryBean.java:341) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1837) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1774) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:593) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:515) at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:320) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222) at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:318) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199) at org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1105) at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:867) at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:549) at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:142) at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:775) at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:397) at org.springframework.boot.SpringApplication.run(SpringApplication.java:316) at org.springframework.boot.SpringApplication.run(SpringApplication.java:1260) at org.springframework.boot.SpringApplication.run(SpringApplication.java:1248) 

I would think that connecting to Athena through a Spring DataSource would be commonplace. Can someone help with why I see this error?

Answer 2

AWS Athena documentation says port 444 is open for outbound traffic. According to the connection documentation here https://docs.aws.amazon.com/athena/latest/ug/connect-with-jdbc.html

Not sure if this is the right decision, so I rummaged a bit, as this question is very old.

Answer 3

By default, a Spring boot automatically determines which Hibernate Dialect dialect to use based on the metadata of the data source obtained from the JDBC driver.

This error is due to the fact that the Athena JDBC driver cannot retrieve the appropriate metadata. Not sure why this fails, but you can work around this detection and explicitly declare which Hibernate Dialect to use on its own by declaring the JpaVendorAdapter component:

 @Bean public JpaVendorAdapter jpaVendorAdapter(JpaProperties properties) { AbstractJpaVendorAdapter adapter = new HibernateJpaVendorAdapter(); adapter.setShowSql(properties.isShowSql()); adapter.setGenerateDdl(properties.isGenerateDdl()); return adapter; } 

^{I keep all the default behavior but just disable that auto-detecting dialect}

And define the dialect in application.properties :

 spring.jpa.properties.hibernate.dialect=org.hibernate.dialect.XXXXXX 

But the fact is, I doubt Athena can be used with Hibernate, because I cannot find that there is an existing dialect for Athena. So what I suggest:

1. Try using a dialect from another database whose syntax is similar to Athena. Unsurprisingly, in some cases this will not work .

2. Realize the dialect for Athena on your own.