Only available when logged in.

Mike Spencer @mikerspencer@mastodon.scot

A reaffirming bit of academic support today: got their code run time down from 2.5 days to 2 mins. Good times

(two main improvements - shortening the ID column, and moving from csv to hive-partitioned parquet)

**Mike Spencer** @mikerspencer · Apr 4

Mike Spencer @mikerspencer

Basically very similar to this example:

**Barry Rowlingson** @geospacedman@mastodon.social · Apr 4

Barry Rowlingson @geospacedman@mastodon.social

@mikerspencer eeek! how big was the data?

**Mike Spencer** @mikerspencer · Apr 4

Mike Spencer @mikerspencer

@geospacedman
Not especially, tens of GB. I'm still amazed it took that long.

**Olivier Leroy** @defuneste@fosstodon.org · Apr 4

Olivier Leroy @defuneste@fosstodon.org

@mikerspencer @geospacedman moving to parquet is not just a change in format it is also a change in schema/structure do not undersold yourself:)

Drag & drop to upload

Back