I am running a few tests on the storage formats available with Hive and using Parquet and ORC as major options. I included ORC once with default compression and once with Snappy.

I have read many a documents that state Parquet to be better in time/space complexity as compared to ORC but my tests are opposite to the documents I went through.

Follows some details of my data.

    Table A- Text File Format- 2.5GB
    
    Table B - ORC - 652MB
    
    Table C - ORC with Snappy - 802MB
    
    Table D - Parquet - 1.9 GB

Parquet was worst as far as compression for my table is concerned.

My tests with the above tables yielded following results.

**Row count operation**

    Text Format Cumulative CPU - 123.33 sec
    
    Parquet Format Cumulative CPU - 204.92 sec
    
    ORC Format Cumulative CPU - 119.99 sec 
    
    ORC with SNAPPY Cumulative CPU - 107.05 sec


**Sum of a column operation**

    Text Format Cumulative CPU - 127.85 sec   
    
    Parquet Format Cumulative CPU - 255.2 sec   
    
    ORC Format Cumulative CPU - 120.48 sec   
    
    ORC with SNAPPY Cumulative CPU - 98.27 sec

   

**Average of a column operation**

    Text Format Cumulative CPU - 128.79 sec
    
    Parquet Format Cumulative CPU - 211.73 sec    
    
    ORC Format Cumulative CPU - 165.5 sec   
    
    ORC with SNAPPY Cumulative CPU - 135.45 sec 

  
**Selecting 4 columns from a given range using where clause**

    Text Format Cumulative CPU -  72.48 sec 
    
    Parquet Format Cumulative CPU - 136.4 sec       
    
    ORC Format Cumulative CPU - 96.63 sec 
    
    ORC with SNAPPY Cumulative CPU - 82.05 sec 



Does that mean ORC is faster then Parquet? Or there is something that I can do to make it work better with query response time and compression ratio?

Thanks!   



All Orc Solutions on Gang of Coders

Total of 1 Orc Solutions

Parquet vs ORC vs ORC with Snappy