Use native formats when storing data

Or: don't use pickles if you don't want to be in a pickle!

Hello fellow datanistas!

I've just penned a blog post that discusses why we shouldn’t use pickles to store data. The post is a reflection on a recent experience I had, which illuminated the pitfalls of relying on pickle files, especially when it comes to the sustainability and security of our projects.

In this post, I share a story from my work where a decision made two years ago came back to haunt me, all due to the choice of data storage format. It's a tale that underscores the importance of considering the future of our code and data, and how the evolving landscape of dependencies can turn the simple task of loading a pickle file into a pickly situation 🥁. But fear not, for I also discuss the lessons learned and the strategies that can help us avoid such pitfalls in the future.

I’d love to invite you to read the full story here: Use Native Formats When Storing Data. And I hope it gives you inspiration for your own work!

If you find the insights valuable, please share the post with others in your network who might also benefit from it!

Cheers,
Eric