All geek questions in one place

TotalBytesBilled is different from totalBytesProcessed

I use the BigQuery developer tool for the strip and noticed some strange behavior in the "Request Cost". upon transition to details I found strange behavior in the properties totalBytesBilled and totalBytesProcessed. but I had some problems understanding this ...

From BigQuery Resource:

  • statistics.query.totalBytesBilled: total number of bytes set for the job.
  • statistics.query.totalBytesProcessed: total number of bytes processed for the job.

The description of these two properties is rather vague ...

Based on my past experience, I expect these 2 to be the same as soon as I take advantage of the free part of my quota.

Example query in a sample dataset

SELECT word,    word_count 
FROM [publicdata:samples.shakespeare] S
LIMIT 1000

returned:

   "totalBytesProcessed": "2650191",
   "totalBytesBilled": "10485760",

+4
2

: https://cloud.google.com/bigquery/pricing#on_demand

:

  • totalBytesProcessed , () . totalBytesBilled , . , , , (. ) " " (. ).

  • 10 10 , . ( ) , . , API. totalBytesBilled . ( , , , : 5 ./ 10 0,000005 . 100 000 10 , 5 . .)

  • , , .

  • , , . , , . , LIMIT , , / !

  • , , , . (, , -) , , , , . UDF , ( , ) . , , , UDF, , , 1.

    , , totalBytesProcessed. , UDF, , , High Compute, .

    , totalBytesBilled billingTier, , . , . , .

+5

-

SELECT MAX(cost) as totalBytesBilled FROM 
(SELECT 10485760 as cost) as min_billed_10MB_bytes, 
(SELECT INTEGER(1024*1024*CEIL(totalBytesProcessed/1024/1024)) as cost) as processed_rounded_up_to_MB_bytes
+1

Source: https://habr.com/ru/post/1607784/

More articles:


All Articles