How to make an asynchronous Hive call in Java?

Question

I would like to execute a Hive request on the server asynchronously. The catch request will probably take a long time, so I would prefer not to block the call. I am currently using Thirft to make a blocking call (blocks on client.execute ()), but I have not seen an example of how to make a non-blocking call. Here is the lock code:

        TSocket transport = new TSocket("hive.example.com", 10000);
        transport.setTimeout(999999999);
        TBinaryProtocol protocol = new TBinaryProtocol(transport);
        Client client = new ThriftHive.Client(protocol);
        transport.open();
        client.execute(hql);  // Omitted HQL

        List<String> rows;
        while ((rows = client.fetchN(1000)) != null) {
            for (String row : rows) {
                // Do stuff with row
            }
        }

        transport.close();

There is no try / catch blocks in the above code to make it short.

Does anyone have any ideas how to make an asynchronous call? Can Hive / Thrift support it? Is there a better way?

Thank!

Answer 1

Hive, Hive Thirft.

Answer 2

AFAIK, Thrift . , ( "" ) , Thrift , , .

, , , , , , ! , Cassandra Java, .

: , , , .

Answer 3

Java , : https://issues.apache.org/jira/browse/THRIFT-768

java-, , :

TNonblockingTransport transport = new TNonblockingSocket("127.0.0.1", 9160);
TAsyncClientManager clientManager = new TAsyncClientManager();
TProtocolFactory protocolFactory = new TBinaryProtocol.Factory();
Hive.AsyncClient client = new Hive.AsyncClient(protocolFactory, clientManager, transport);

, . , .

Answer 4

Hive, , , Java concurrency:

 Callable<SomeResult> c = new Callable<SomeResult>(){public SomeResult call(){

    // your Hive code here

 }};

 Future<SomeResult> result = executorService.submit(c);

 // when you need the result, this will block
 result.get();

, , Runnable Callable.

Answer 5

Hive, , , . java.util.concurrent.FutureTask, , .

Answer 6

AWS Elastic MapReduce. AWS MapReduce adoop/hive Amazon - AWS MapReduce.

S3 .

Because web service calls are asynchronous, we never block other operations. We continue to monitor the status of our assignments in a separate thread and capture the results when the work is completed.