noctua now supports caching. This was originally inspired by
pyathenato reduce the cost of using
noctua however has a different caching method and utilities local caching in
R environments instead of using AWS
list_query_executions. This is down to
AWS Athena output when caching is disabled.
By caching queries the performance of repeat queries is significantly improved. This is because the query is no longer sent to
AWS Athena. Instead the query ID, of the repeating query, is taken from the R environment and the result is returned from
library(DBI) library(noctua) con = dbConnect(athena()) # Start caching queries noctua_options(cache_size = 10) # Upload Data to AWS Athena dbWriteTable(con, "iris", iris, partition = c("Partition" = "01")) # initial query to AWS Athena system.time(df1 = dbGetQuery(con, "select * from iris")) # Info: (Data scanned: 3.63 KB) # user system elapsed # 0.105 0.004 3.397 # repeat query to AWS Athena system.time(df2 = dbGetQuery(con, "select * from iris")) # Info: (Data scanned: 3.63 KB) # user system elapsed # 0.072 0.000 0.348
Here we can see a performance increase of x10 with repeat query execution.
The weakness in caching occurs when the underlying data is updated. The cache will still only retrieve the previous query ID. This means that the new updated data won’t be return when the caching is enabled:
Sadly the cached query didn’t pick up the new data from
The caching method in
noctua will remember previous query ids within each R session, even if you stop and start caching in
# Start caching noctua_options(cache_size = 10) res1 = dbExecute(con, "select * from iris") # Stop caching noctua_options() res2 = dbExecute(con, "select * from iris") # Start caching noctua_options(cache_size = 10) res3 = dbExecute(con, "select * from iris") # Compare Query ID's res1@info$QueryExecutionId # 9a9272f5-0632-4774-9aa9-d07f151dabc5 res2@info$QueryExecutionId # be12fe0-3ec0-4595-b3e6-b3bf67efa266 res3@info$QueryExecutionId # 9a9272f5-0632-4774-9aa9-d07f151dabc5
We can see that
res3 utilise the same QueryID, even tho caching was stopped and started.
To clear down the cache, just set the parameter: