Historically, there was no direct monetization involved in contributing to TDS (unless authors chose to join the partner program at our former hosting platform). As we establish TDS as an independent, self-sustaining publication, we’ve decided to change course, as it was important for us to reward the articles that help us reach our business goals in proportion to their impact.

How it works

The TDS Author Payment Program is structured around a 30-day window. Articles are eligible for payment based on the number of readers who engage with them in the first 30 days after publication.

Authors are paid based on three earning tiers:

• 25,000+ Views: The article will earn $0.1 per view within 30 days of publication: a minimum of $2,500, and up to $7,500, which is the cap for earnings per article.
• 10,000-24,999 Views: The article will earn $0.05 per view within 30 days of publication: a minimum of $500, and up to $1,249.
• 5,000-9,999 Views: The article will earn $0.025 per view within 30 days of publication: a minimum of $125, and up to $249.

A few important points to keep in mind: 

• Views are counted only if a reader stays on the page for at least 30 seconds, ensuring that the payouts reflect real engagement, not clicks.
• Articles with fewer than 5,000 views in 30 days will not qualify for payment.
• During these 30 days, articles must remain exclusive to Towards Data Science. After that, authors are free to republish or remove their articles.

Who can participate?

This program is available to every current TDS contributor, and to any new author who becomes eligible once an article reaches the first earning tier.

Participation in the program is subject to approval to ensure authentic traffic. We reserve the right to pause or decline participation if we detect unusual spikes or fraudulent activity. Additionally, payments are only available to authors who live in countries supported by Stripe.

Authors can submit up to four articles per month for paid participation.

Why we’re doing this

We built this program to create a transparent and sustainable system that pays contributors for the time and effort required to write great articles that attract a wide audience of data science and machine learning professionals. By tracking genuine engagement, we ensure that the best work gets recognized and rewarded while keeping the system simple and transparent.

We’re excited to offer this opportunity and look forward to supporting our contributors who keep Towards Data Science the leading destination in the data science community. 

How to contribute work?

We’re working swiftly to roll out an author portal that will streamline article pitches and feedback.

In the meantime, please send your upcoming article directly to our team using this form.

If you’re having an issue with our online form, please let us know via email (publication@towardsdatascience.com) so we can help you complete the process. Please do not email us an article that you have already sent via our form.

The post Announcing the Towards Data Science Author Payment Program appeared first on Towards Data Science.

Since founding Towards Data Science in 2016, we’ve built the largest publication on Medium with a dedicated community of readers and contributors focused on data science, machine learning, and AI. Medium built a fantastic platform, and we wouldn’t have been able to reach our audience without its help.

As of Monday, February 3, 2025, Towards Data Science will become an independent publication, and we believe this transition will help us better serve our contributors and readers.

By moving off Medium we can maintain complete control over our editorial vision, protect the domain authority we’ve built, and guide our future growth on our own terms. 

Today, we’re launching our new WordPress site, though some parts of the migration are still a work in progress. We’re in close communication with our contributors to ensure a smooth transition. Despite the move, our mission remains unchanged: We’ll continue to provide the same in-depth coverage of data science, machine learning, and AI – only now on a platform where we can further support authors by utilizing exciting new promotional tools, advanced analytics, and dynamic community features that open up fresh opportunities for engagement and growth. 

Why Independence Makes Sense for TDS

We’ve long recognized that Medium provides valuable hosting, distribution and monetization tools – resources that have helped many writers and publishers (like us!) reach a wide audience. However,we believe our long term-term goals – as well as those of our readers and contributors – are best supported on an independent platform.

Here’s why we believe an independent platform is the right move:

Improved Accessibility for Readers – We believe in open access to all articles, not in paywalls. Moving to our own platform ensures that all of our content is available without charge to readers.

Ongoing Support for Authors – Our independent platform gives authors greater flexibility to promote their work. This means authors can amplify their voices and engage with new audiences.

Ownership of our Domain Authority – Operating independently allows us to preserve and grow Towards Data Science by not only retaining our search visibility, but also allowing us to experiment with novel content formats and create deeper connections with readers.

For current TDS authors, expect an email soon with details on how to join the new platform.

For new contributors, we’re opening up submissions and would love to feature your work on our new site. More details will follow in the coming days.

Content Rights Note

We want to be clear that we will only migrate content to our new site if we have the appropriate rights to do so in accordance with Towards Data Science’s terms and conditions. Your work remains under your control, and we’re committed to honoring any prior agreements regarding its use. Only content for which we have the necessary permissions will be transferred, ensuring that your intellectual property is respected throughout this transition.

If you have any questions or concerns about how your content will be handled during this migration, or wish to remove it from our new website, please don’t hesitate to reach out to our editorial team – you can email us here: publication@towardsdatascience.com. We’re here to provide any additional clarification you might need and to address any issues that arise.

How To Submit Your Work

We’re currently updating our submissions workflow, which we plan to launch in the near future. In the meantime, please fill out this form so we can already consider your article.

Contribution Submission Portal

FAQ: Transition for Authors & Contributors

We understand many TDS contributors have questions about how our move off Medium affects your work. Thank you for sticking with us through this change.

In the next few days, current and past authors will receive an email with instructions to set up a new account on our platform. This will let you update existing articles and submit new work for consideration.

Below are answers to some of the most pressing questions we’ve received so far, along with details about new features coming to Towards Data Science.

1. Why did Towards Data Science leave Medium?

Medium has been instrumental for our growth, but recent changes in Medium policies showed us that our priorities have diverged. By moving to our own platform, we gain full control over our editorial direction and policies. This independence allows us to provide free access to our articles, and better support you, our contributors, with a full set of publishing tools–including advanced analytics, enhanced social media capabilities, and more flexible content management–to help you reach a wider audience.

2. Why now?

At the end of January 2025, Medium began redirecting content from our custom domain, which immediately impacted our site traffic and search rankings. To protect our long-term vision and maintain the site’s hard-earned visibility, we fast-tracked a plan to launch an independent WordPress site.

3. When is the new platform launching?

Our new site launched on Monday, February 3, 2025. While we’re still refining some features on the backend, the site is fully operational with the paywall removed from all content. We’re working closely with contributors to ensure the transition is as smooth as possible. 

4. What do I need to do as a contributor during this transition?

For now, simply email your article to us in a PDF or Google Doc. We’ll handle the rest by importing your work into the new CMS and publishing it on the site. You don’t need to set up a new account right away–though current and past contributors will soon receive an email with instructions to create an account for managing and updating your content.

5. How will my existing content be handled?

We want to be clear: We only migrated content to our new site if we had the appropriate rights to do so in accordance with Towards Data Science’s terms and conditions. Your work remains under your control, and we’re committed to honoring any prior agreements regarding its use. Only content for which we have the necessary permissions will be transferred, ensuring that your intellectual property is respected throughout this transition.

If you have any questions or concerns about how your content will be handled during this migration, please don’t hesitate to reach out to our editorial team – you can email us here: publication@towardsdatascience.com.

We’re here to provide any additional clarification you might need and to address any issues that arise.

6. Can I continue to publish on Medium during this transition?

Yes, absolutely. Towards Data Science leaving Medium doesn’t mean you have to leave Medium. 

Please keep in mind that to publish on TDS, you’ll need to send us your article directly. We will no longer accept submissions through, or published on, Medium.

7. What are the benefits of publishing on the new TDS?

Moving to our own platform opens up several exciting opportunities for our contributors, including

• Unlimited Reach: Your articles will be completely free to read, free from Medium’s paywall. This means your work can reach a wider, more diverse audience.
• Advanced Promotional Tools: Our new site comes with enhanced features to empower authors to better promote their work. This flexibility will make it easier to use TDS to build your personal brand and connect with new readers. 
• Greater Editorial Control: We’re excited to launch new article formats that will allow authors greater control over how their content is presented to the reader.
• Better Visibility & Control: Content on TowardsDataScience.com will maintain its search rankings without being dependent on Medium’s policies.

8. Who do I contact if I have questions?

We’re here to help. If you have any additional questions or need further clarification, please email our editorial team at publication@towardsdatascience.com.

Your feedback is important as we finalize our migration to the new platform.

We hope these answers clarify our transition process. We’re thrilled about the future of Towards Data Science, and we’re committed to making this transition a smooth and positive experience for our authors and readers. Thank you for being a vital part of Towards Data Science as we take the publication to the next level!

– TDS Editorial Team

The post Towards Data Science is Launching as an Independent Publication appeared first on Towards Data Science.

The Variable is moving soon—sign up here to ensure you receive all future newsletters.

Our prolific authors delivered some excellent work this past month, channeling all the renewed energy and excitement we’ve come to expect from January on TDS. From career advice to core programming and data-processing tasks, our most-read and -shared articles in the past month cover the topics that data professionals care about the most as they plan their next move and aim to expand their skill set.

We invite you to explore this month’s must-reads with an open mind: from the ever-shifting terrain of job descriptions to the rise of small language models (alongside large ones), they tackle well-covered areas in Data Science and machine learning from a fresh, actionable, and pragmatic perspective. Let’s get started.

• How to Pick Between Data Science, Data Analytics, Data Engineering, ML Engineering, and SW Engineering"When the job titles sound so similar and the roles have a good amount of overlap" it can be difficult to choose the right path for your own interests and priorities as a data practitioner. Marina Wyss – Gratitude Driven‘s clear and detailed overview will help you make an informed decision.
• Your Company Needs Small Language ModelsIs it time to reassess the axiom that in AI, bigger is always better? Sergei Savvov makes a compelling case for the growing footprint of small language models in industry contexts, outlining the ways "they can reduce costs, improve accuracy, and maintain control of your data," and urges us to stay mindful of these models’ current limitations.
• The Large Language Model CourseFor anyone whose new year’s resolutions included expanding their knowledge of (and practical experience with) LLMs, Maxime Labonne‘s comprehensive course is the one-stop resource you’ll need to get started—it offers a well-structured curriculum that assumes no advanced knowledge, and comes full of recommended articles, tutorials, and tools.

• 5 Simple Projects to Start Today: A Learning Roadmap for Data EngineeringFor all the aspiring data engineers out there, Sarah Lea outlines a realistic, four-month plan that covers all the essentials. Most importantly, it guides you through several project ideas so that you not only grow your theoretical knowledge, but also get to practice the concepts and workflows you’re learning about.
• How to Build a Knowledge Graph in Minutes (And Make It Enterprise-Ready)Looking to apply a similar hands-on approach to knowledge graphs? After facing major setbacks in the past, Thuwarakesh Murallie shows how you can build one on your own by leveraging the power of LLMs.
• Top 12 Skills Data Scientists Need to Succeed in 2025"Many things are changing, but other things are not. Understanding which changes require your attention is the key to success." Benjamin Bodner breaks down the skills that remain essential for data professionals amid the disruptive pressure of AI tools.
• Deep Dive into Multithreading, Multiprocessing, and AsyncioWhether you’re taking your first steps in Python or are already a seasoned programmer, there’s always more to learn; Clara Chong‘s recent post zooms in on concurrency models—approaches to handle multiple tasks simultaneously—and unpacks the stakes of choosing the right one, depending on your project’s needs.
• How to Run Jupyter Notebooks and Generate HTML Reports with Python ScriptsFor another Python-focused tutorial that foregrounds the ability to streamline tedious workflows with code, don’t miss Amanda Iglesias Moreno‘s guide to automating Jupyter Notebook execution and report generation (with a helpful detour through synthetic-data creation).
• My Experience Switching From Power BI to Looker (as a Senior Data Analyst)It’s often difficult to assess the tradeoffs and gains you’ll be making by adopting a new tool without first trying different options—or hearing firsthand accounts from professionals in a similar situation to yours. Tomas Jancovic (It’s AI Thomas) walks us through the journey of moving to Looker for his data analytics workflows, and presents a balanced, concrete account of its pros and cons.
• Three Important Pandas Functions You Need to KnowNew models and generative-AI apps come and go, but Pandas is still here with us, playing an important role in data scientists’ day-to-day work. Jiayan Yin recently shared a helpful guide for early-stage learners, zooming in on three essential functions that you’ll likely turn to again and again as you process and analyze datasets.

Our latest cohort of new authors

Every month, we’re thrilled to see a fresh group of authors join TDS, each sharing their own unique voice, knowledge, and experience with our community. If you’re looking for new writers to explore and follow, just browse the work of our latest additions from the past couple of months, including Ramsha Ali, Derick Ruiz, Dr. Marcel Müller, Rodrigo M Carrillo Larco, MD, PhD, Ilona Hetsevich, Federico Zabeo, Vladyslav Fliahin, Jérôme DIAZ, Mandeep Kular, Glenn Kong, Vladimir Kukushkin, Viktor Malyi, Ruben Broekx, Iqbal Hamdi, Richa Gadgil, Piotr Gruszecki, Jonathan Fürst, Sirine Bhouri, Kyoosik Kim, Sunghyun Ahn, Afjal Chowdhury, Tim Wibiral, Kunal Santosh Sawant, Aman Agrawal, Abdelkader HASSINE, Florian Trautweiler, Mohammed AbuSadeh, Loic Merckel, Lukasz Gatarek, Zombor Varnagy-Toth, Marc Matterson, Manelle Nouar, Paula LC, Shitanshu Bhushan, Matthew Senick, Lewis James | Data Science, Clara Chong, Bilal Ahmed, Pavel Krautsou, Erol Çıtak, Cristovao Cordeiro, Vladimir Zhyvov, Yuval Gorchover, Zach Flynn, Allon Korem | CEO, Bell Statistics, Tony Albanese, Sandra E.G., Miguel Cardona Polo, James Thorn, Vineet Upadhya, Kaushik Rajan, Mahmoud Abdelaziz, PhD, Benjamin Assel, Shirley Li, Marina Wyss – Gratitude Driven, Michal Davidson, Rémy Garnier, Uladzimir Yancharuk, David Lindelöf, Ricardo Ribas, Hunjae Timothy Lee, Ashley Peacock, Rohit Ramaprasad, Alejandro Alvarez Pérez, David Martin, Ben Tengelsen, César Ortega Quintero, Jaemin Han, Max Surkiz, Massimo Capobianco, Tobias Cabanski, Jimin Kang, Felix Schmidt, Paolo Molignini, PhD, Sayali Kulkarni, Alan Nekhom, and Chris Lettieri, among others.

Thank you for supporting the work of our authors! We love publishing articles from new authors, so if you’ve recently written an interesting project walkthrough, tutorial, or theoretical reflection on any of our core topics, don’t hesitate to share it with us.

Until the next Variable,

TDS Team

The post Data Roles, Small Language Models, Knowledge Graphs, and More: Our January Must-Reads appeared first on Towards Data Science.

Whether you’re an independent practitioner or part of a larger team trying to make sense of this emerging technology, you’ll find concrete and actionable insights in the lineup of articles we’ve selected this week. They each tackle the nuts and bolts of building AI apps and leveraging their potential for well-defined goals, while avoiding common pain points.

While these posts zoom in on specific topics and business problems, they all offer a pragmatic, accessible approach, making them useful for readers across a wide spectrum of backgrounds and experience levels. Let’s dive in.

• What Did I Learn from Building LLM Applications in 2024? – Part 2So you’ve built a prototype of an LLM-based app that looks promising. What’s next? Satwiki De offers a comprehensive roadmap based on the takeaways she’s accumulated over a year of experimentation—the main thrust being to "develop your AI-enabled application keeping the business objectives in mind."
• Why Generative-AI Apps’ Quality Often Sucks and What to Do About itHaving observed countless enterprise AI initiatives deliver underwhelming results (if not fail altogether) Dr. Marcel Müller focuses on the importance of robust assessment, and shows "how we can qualitatively and quantitatively evaluate generative AI applications in the context of concrete business processes."
• Designing, Building & Deploying an AI Chat App from Scratch (Part 1)If you’re ready to roll up your sleeves (real or proverbial) and start implementing an AI chat app, Joris Baan‘s patient guide is a great resource to help you stay on the right track. Part 1 outlines the microservices architecture and local development needs you’ll want to think about, while part 2 moves on to a detailed discussion of cloud deployment and scaling.

• Learn to Build Advanced AI Image ApplicationsMoving beyond chatbots, Ida Silfverskiöld explores the possibilities that visual generative-AI tools open up for real-world businesses and professionals—in this case, an interior design app based entirely on open-source models and frameworks.
• How to Build an AI Agent for Data Analytics Without Writing SQLWhy not harness the power of AI to streamline common data-analytics workflows that would typically require numerous SQL queries? Chengzhi Zhao aims to accomplish just that with the aid of an AI agent, and shows how you can build one yourself with LangChain and DuckDB.
• AI-Powered Information Extraction and MatchmakingFor another promising use case where an AI app can help you and your team save time and become more efficient, don’t miss Umair Ali Khan‘s patient tutorial on building AI-based match-making tools (in this case, helping job seekers find positions that fit their skills and interests).

Branching out into the world beyond AI apps, we’ve selected a few more recommended reads we thought you’d enjoy—from a beginner-friendly intro to LLMs to an in-depth analysis of data strategies.

• If your less tech-savvy colleagues could use a clear and accessible primer on what LLMs are and how they work, just send them Carolina Bento‘s top-notch introduction.
• For a more advanced exploration of LLMs’ inner workings, head right over to Jaemin Han‘s fascinating look at models’ shortcomings in generating and interpreting ASCII art—and the surprising security risks the latter might suggest.
• Quick and focused, Clara Chong‘s guide to variable scoping explains why this particular aspect of your code can have far-reaching consequences for your Data Science workflows.
• What rules should we have in place to protect humans from unknowingly interacting with AI profiles on social platforms? James Barney unpacks the fallout from the recent controversy around Meta’s (now-suspended) experiment.
• The rise of code-generating chatbots leads Murtaza Ali to wonder if the age of human-written programming tutorials is coming to an end. (Spoiler alert / sigh of relief: not just yet!)
• If you’d like to make sense of the gap you observe between between predictions and real-world outcomes, Hennie de Harder‘s new post walks us through the basics of prediction probabilities, calibration, and how to interpret these numbers in a practical context.
• In the mood for a deep dive? Jens Linden, PhD just shared the latest installment in his Demystify Data Strategy series, focusing this time on the common pitfalls of data and AI strategies—and how organizations can avoid them.
• To round out this week’s lineup, we invite you to learn about MicroPython and its potential relevance for data scientists; Sarah Lea‘s concise guide will get you up to speed on all the essential details.

Thank you for supporting the work of our authors! As we mentioned above, we love publishing articles from new authors, so if you’ve recently written an interesting project walkthrough, tutorial, or theoretical reflection on any of our core topics, don’t hesitate to share it with us.

Until the next Variable,

TDS Team

The post Building Successful AI Apps: The Dos and Don’ts appeared first on Towards Data Science.

Feeling inspired to write your first TDS post? We’re always open to contributions from new authors.

Buzzwords and trends come and go, but the core task of telling compelling stories with data remains one of the main pillars in data scientists’ daily workflow. For practitioners who’d like to up their visualization game, this week we’re highlighting some of our best recent articles on creating powerful, effective, and sleek deliverables.

Our selection tackles the topic from multiple angles, so whether you’re interested in chart optimization, geospatial aids, or interactive dashboards, we’re sure you’ll find something here to inspire you and help you expand your current skill set. Happy tinkering!

• How to Make Extremely Beautiful Charts with PythonWhy settle for "meh…" when "wow!" is within reach? Ari Joury, PhD challenges us to stop settling for mediocre visualizations, shows how a few smart and focused tweaks can transform your average chart into something you can truly be proud of, and offers more than a few concrete examples (and code snippets) to make these suggestions both concrete and actionable.
• 5 Essential Tips to Build Business Dashboards Stakeholders Love"You might be wondering, isn’t dashboarding just creating a bunch of charts? Well, Yes and No." Yu Dong zooms in on an often-undervalued aspect of data professionals’ work, and explains why it’s time to reevaluate and streamline your approach to dashboard creation.
• Awesome Plotly with Code Series (Part 8): How to Balance Dominant Bar Chart CategoriesFor the past few months, Jose Parreño‘s excellent series has addressed some of the main pain points of designing functional, easy-to-digest charts. His latest installment takes a close, pragmatic look at a common scenario where one category threatens to throw an entire bar chart off-balance.

• Step-by-Step Guide for Building Bump Charts in PlotlyReady to move on from bar charts into more advanced and custom formats? Don’t miss Amanda Iglesias Moreno‘s Plotly-based tutorial, which introduces a complete workflow for creating a bump chart, a more specialized visualization that is "designed to explore changes in a ranking over time" and allows us to "quickly identify trends and detect elements at the top or bottom of the ranking."
• Easy Map Boundary Extraction with GeoPandasWorking with geospatial data can be very rewarding—not to mention essential in many industries—but it can also get tricky and occasionally unwieldy. Lee Vaughan‘s latest Python guide brings clarity and practicality to a very common use case: extracting, measuring, and plotting country borders.

A new year often brings with it a rush of excellent new writing, and so far 2025 has not disappointed on that front. Here are several recent standouts on a wide range of topics, from hands-on AI projects to the history of GPT models.

• What would it take for a supervised-learning model to solve a Rubik’s Cube? Daniel Warfield‘s latest deep dive walks us through a fun and enlightening project.
• As more and more AI-generated content floods contemporary media and online spaces, Stephanie Kirmer wonders how we should assess its cultural impact—and navigate its (many) pitfalls.
• For a solid understanding of cutting-edge LLMs, a trip down memory lane (all the way back to the prehistoric era of 2019–2020) might be a good idea—and Shirley Li‘s thorough and accessible retrospective of GPT-2 and GPT-3 offers us precisely that.
• If you’re in the mood for something more hands-on this week, we highly recommend Maeda Hanafi, PhD‘s new guide to customizing your fine-tuning code with Hugging Face’s Transformers library.
• Are we approaching the point where AI might be able to reliably determine if something is funny? Vineet Upadhya‘s latest article reviews recent research into this fascinating topic, including a new framework for computational humor detection.
• Establishing causality is a common goal for data practitioners, but that doesn’t make it any less difficult to pull off. For his debut TDS post, Rémy Garnier presents a versatile method that you can apply across a wide range of use cases.
• After sharing a well-received list of AI projects you can experiment with over a weekend, Shaw Talebi fleshes out the details of one of his proposed ideas: an AI-powered resume optimizer.

Thank you for supporting the work of our authors! As we mentioned above, we love publishing articles from new authors, so if you’ve recently written an interesting project walkthrough, tutorial, or theoretical reflection on any of our core topics, don’t hesitate to share it with us.

Until the next Variable,

TDS Team

The post Charts, Dashboards, Maps, and More: Data Visualization in the Spotlight appeared first on Towards Data Science.

Feeling inspired to write your first TDS post? We’re always open to contributions from new authors.

As we all settle into the sometimes hectic rhythm of a new year, we hope you’ve been enjoying the excitement of kicking off projects, learning about new topics, and exploring your next career moves. We’re definitely seeing a flurry of activity among our authors—both longstanding contributors and recent additions—and are thrilled to share all the great work they’ve been cooking up over the holidays.

Our lineup of top-notch reads this week has a distinctly actionable, hands-on flavor to it—after all, what better way to harness all this energy than by tinkering with some datasets, models, and code? Whether you’re interested in learning more about cutting-edge evaluation methods or building agentic-AI tools, we’ve got you covered with a diverse selection of tutorials and practical overviews. Ready to dive in?

• Paradigm Shifts of Eval in the Age of LLMs Is it time to reevaluate the way we approach evaluations? Lili Jiang believes it is: "I’ve come to recognize that LLMs requires some subtle, conceptually simple, yet important changes in the way we think about evaluation." Her latest article offers high-level insights into what a new paradigm might look like.

• The Next Frontier in LLM Accuracy Staying thematically close to LLM optimization, Mariya Mansurova‘s new deep dive unpacks in great detail several methods we can use to increase models’ accuracy, and zooms in on advanced fine-tuning techniques.

• How to Build a Graph RAG App Ready to roll up your sleeves and dig deep into some code? Steve Hedden‘s thorough tutorial on creating your first graph RAG app is a great option for anyone who’s interested in this trending topic but needs guidance and context to ensure they’re starting off on the right foot.

• Multi-Agentic RAG with Hugging Face Code Agents Agent-based systems gained enormous steam (and buzz) last year, and it doesn’t seem like that’s about to change in 2025. Curious to learn more about them? Gabriele Sgroi, PhD‘s patient, step-by-step guide may be long, but it remains accessible and clear as it outlines the process of leveraging a "small" LLM to power a multi-agentic system—and produce good results, even on consumer-grade hardware.

• Demand Forecasting with Darts: A TutorialLLMs may be grabbing much of our collective attention these days, but business-focused workflows remain the bread and butter of industry data scientists. Sandra E.G.‘s debut TDS article provides a robust, hands-on introduction to one such essential task: demand forecasting in the context of retail sales.

• Distributed Parallel Computing Made Easy with Ray It’s crucial for data and ML practitioners to experiment with new tools and frameworks, as seemingly small improvements can accumulate into major cost and efficiency benefits. Betty LD walks us through her recent foray into the AI-focused Ray library for distributed data processing, and demonstrates its power through the use case of scalable offline batch inference.

If you’re ready to branch out into other topics this week, we’re here to help—whether your interests lie at the intersection of music and AI, quantum computing, or linear algebra (among others), we hope you explore some of these excellent articles:

• Presenting her team’s cutting-edge research, Tula Masterman explains how we can leverage the hidden state from an intermediate Transformer layer for efficient and robust content safety and prompt injection classification.
• Music-centered AI tools continue to make strides; Max Hilsdorf devotes his latest exploration of the topic to mono-to-stereo upmixing, a technique for enriching or improving our music-listening experience.
• With seven first-author publications under her belt, Malak Sadek is well-positioned to offer concrete insights for other researchers who’d like to grow their publishing footprint.
• Continuing his ongoing series on key linear algebra concepts, Rohit Pandey recently shared a new, comprehensive explainer that looks at the inner workings of orthonormal matrices.
• Quantum computing has been a hot topic for some time now, though discussions of its promise can sometimes feel almost sci-fi-adjacent. Here to help us make sense of the field and where things stand at the moment is Sara A. Metwalli, whose primer comes right as the UN has declared 2025 the international year of quantum science and technology.
• While we’re on the topic, let’s round out this week’s selection with Benjamin Assel‘s nuanced discussion of error-rate measurement in IBM quantum processors, which also includes code examples, using Qiskit.

Thank you for supporting the work of our authors! As we mentioned above, we love publishing articles from new authors, so if you’ve recently written an interesting project walkthrough, tutorial, or theoretical reflection on any of our core topics, don’t hesitate to share it with us.

Until the next Variable,

TDS Team

The post LLM Evaluation, Parallel Computing, Demand Forecasting, and Other Hands-On Data Science Approaches appeared first on Towards Data Science.

Feeling inspired to write your first TDS post? We’re always open to contributions from new authors.

Happy new year! Welcome back to the Variable!

The ink has barely dried on our 2024 highlights roundup (it’s never too late to browse it, of course), and here we are, ready to dive headfirst into a fresh year of learning, growth, and exploration.

We have a cherished tradition of devoting the first edition of the year to our most inspiring—and accessible—resources for early-stage Data Science and machine learning professionals (we really do!). We continue it this year with a selection of top-notch recent articles geared at beginner-level learners and job seekers. For the rest of our readers, we’re thrilled to kick things off with a trio of excellent posts from industry veterans who reflect on the current state of data science and AI, and share their opinionated, bold predictions for what the year ahead might look like. Let’s get started!

2025: Ready, Set, Go!

• Top 10 Data & AI Trends for 2025As the co-founder and CEO of Monte Carlo, few people are as well-positioned as Barr Moses to observe industry shifts and emerging trends—so her annual forecast should definitely be on your to-read list.
• 2024 in Review: What I Got Right, Where I Was Wrong, and Bolder Predictions for 2025After a year dominated by RAG, AI agents, and knowledge graphs, what tools and workflows should we keep track of in the coming year? Leonie Monigatti shares a thorough recap of 2024—and a sneak peek at her well-tuned crystal ball for the year to come.
• Ten Predictions for Data Science and AI in 2025From Jason Tamara Widjaja, here’s another set of insightful reflections: "My predictions for 2025 attempt to provide a view of the tensions of AI, taking an unpopular but balanced view as someone whose work depends not on selling AI, but on implementing AI well – and living through the consequences of our decisions."

Data science and machine learning, step by step by step

• The Essential Guide to R and Python Libraries for Data VisualizationWith or without AI, charts and plots aren’t going anywhere anytime soon. Sarah Lea maps out the key libraries in which current and aspiring data scientists should gain fluency.
• Roadmap to Becoming a Data Scientist, Part 2: Software EngineeringProgramming isn’t going anywhere in 2025, either. Vyacheslav Efimov‘s guide outlines the coding essentials that will lead you to data science success.
• Missing Data in Time-Series: Machine Learning TechniquesOne constant trait of real-world data: it’s messy! Learn how to navigate the chaos by following along Sara Nóbrega‘s primer on handling missing data.
• Causality – Mental Hygiene for Data ScienceTaking a few steps back from the more nitty-gritty aspects of data science work, Eyal Kazin‘s recent deep dive constitutes a "gentle intro" to the intricate art of detecting, interpreting, and applying causality.
• Master Machine Learning: 4 Classification Models Made SimpleFor anyone who enjoys structure and clarity above all else, Leo Anello‘s (extremely) thorough, 15-step tutorial on classification models would be a perfect starting point from which to expand your ML know-how.
• 2024 Survival Guide for Machine Learning Engineer InterviewsWhether you’re already applying for your first MLE job or contemplating it as one of your goals for the year, don’t miss Mengliu Zhao‘s "survival guide," aimed specifically at junior-level practitioners.
• Machine Learning Basics I Look for in Data Scientist InterviewsTackling the occasionally opaque hiring process from the other end of the table, Farzad Nobar created a helpful resource to help job applicants zoom in on the topics that really matter to employers.
• 100 Years of (eXplainable) AIBeyond the whats and hows of day-to-day work, there are also the whys: why did this model produce these outputs? Sofya Lipnitskaya‘s explainer unpacks the history of AI explainability in the context of the recent rise of LLMs.
• How to Build a General-Purpose LLM AgentTo end on a more hands-on note—and to satisfy the curiosity of all of you who’ve heard the buzz around AI agents—we highly recommend Maya Murad‘s step-by-step guide, which can form "the groundwork for designing your own custom agentic architecture" down the line.

Thank you for supporting the work of our authors! As we mentioned above, we love publishing articles from new authors; if contributing to TDS in 2025 is one of your new year’s resolutions—or even if you’ve just recently written an interesting project walkthrough, tutorial, or theoretical reflection on any of our core topics—don’t hesitate to share it with us.

Until the next Variable,

TDS Team

The post Start a New Year of Learning on the Right Foot appeared first on Towards Data Science.

Feeling inspired to write your first TDS post before the end of 2024? We’re always open to contributions from new authors.

And just like that, 2024 is (almost) in the books. It was a year of exciting transitions – both for the TDS team and, in many meaningful ways, for the data science, machine learning, and AI communities at large. We’d like to thank all of you—readers, authors, and followers—for your support, and for keeping us busy and engaged with your excellent contributions and comments.

Unlike in 2023, when a single event (ChatGPT’s launch just weeks before the beginning of the year) stopped everyone in their tracks and shaped conversations for months on end, this year we experienced a more cumulative and fragmented sense of transformation. Practitioners across industry and academia experimented with new tools and worked hard to find innovative ways to benefit from the rapid rise of LLMs; at the same time, they also had to navigate a challenging job market and a world where AI’s footprint inches ever closer to their own everyday workflows.

To help you make sense of these developments, we published more than 3,500 articles this past year, including hundreds from first-time contributors. Our authors have an incredible knack for injecting their unique perspective into any topic they cover—from big questions and timely topics to more focused technical challenges—and we’re proud of every post we published in 2024.

Within this massive creative output, some articles manage to resonate particularly well with our readers, and we’re dedicating our final Variable edition to these: our most-read, -discussed, and -shared posts of the year. As you might expect, they cover a lot of ground, so we’ve decided to arrange them following the major themes we’ve detected this year: learning and building from scratch, RAG and AI agents, career growth, and breakthroughs and innovation.

We hope you enjoy exploring our 2024 highlights, and we wish you a relaxing end of the year – see you in January!

Learning and Building from Scratch

The most reliably popular type of TDS post is the one that teaches readers how to do or study something interesting and productive on their own, and with minimal prerequisites. This year is no exception—our three most-read articles of 2024 fall under this category.

• 5 AI Projects You Can Build This Weekend (with Python)From beginner-friendly to advanced project ideas, Shaw Talebi demonstrates that anyone can get hands-on with AI.
• Understanding LLMs from Scratch Using Middle School MathHow do LLMs work? Rohit Patel offered one of the most accessible and engaging explainers you’ll ever find on this topic.
• How to Learn AI on Your Own (A Self-Study Guide)For the self-starters out there, Thu Vu put together a streamlined roadmap for studying the fundamental building blocks of AI.
• The Math Behind Neural NetworksTo understand neural networks, "the backbone of modern AI," Cristian Leo guides us deep into their underlying mathematical principles.
• Text Embeddings: Comprehensive GuideEmbeddings make the magic of LLMs possible, and Mariya Mansurova‘s thorough introduction makes it clear how and why they’ve become so important.
• How I Studied LLMs in Two Weeks: A Comprehensive RoadmapAnother excellent learning resource came to us from Hesam Sheikh, who walked us through an intensive—but accessible— curriculum to master the basics (and then some) of large language models.

RAG and AI Agents

Once the initial excitement surrounding LLMs settled (a bit), data and ML professionals realized that these powerful models aren’t all that useful out of the box. Retrieval-augmented generation and agentic AI rose to prominence in the past year as the two leading approaches that bridge the gap between the models’ potential and real-world value; they also ended up being our most covered technical topics in recent months.

• Intro to LLM Agents with LangChain: When RAG Is Not EnoughBack in March—and quite ahead of the curve—Alex Honchar published the definitive beginners’ guide to working with agents.
• Using LangChain ReAct Agents for Answering Multi-Hop Questions in RAG SystemsShowing us how agents and RAG can complement each other, Dr. Varshita Sher‘s tutorial addresses the common need of answering complex queries on internal documents.
• 17 (Advanced) RAG Techniques to Turn Your LLM App Prototype into a Production-Ready SolutionBuilding a rudimentary RAG pipeline is one thing; optimizing it so that it can actually work in a business context is another. Dominik Polzer put together a comprehensive guide to the methods you can leverage to achieving that lofty goal.
• 12 RAG Pain Points and Proposed SolutionsOn a similar troubleshooting beat, Wenqi Glantz outlines a dozen streamlined approaches for tackling some of the most common challenges practitioners face when implementing RAG.
• Choosing Between LLM Agent FrameworksIt can be tough to make informed choices in an ecosystem where both major and emerging players release new tools every day. Aparna Dhinakaran is here to help with sharp insights on the tradeoffs to keep in mind.

Career Growth

Data Science and machine learning career paths continue to evolve, and the need to adapt to this changing terrain can generate nontrivial amounts of stress for many professionals, whether they’re deep into their career or are just starting out. We love publishing personal reflections on this topic when they also offer readers pragmatic advice—here are four that stood out to us (and to our readers).

• What 10 Years at Uber, Meta, and Startups Taught Me About Data AnalyticsFrom the importance of storytelling and business acumen to the limitations of metrics, Torsten Walbaum generously consolidated lessons based on a decade of work into actionable insights.
• What I Learned in my First 3 Months as a Freelance Data ScientistCareer switches are always tricky, and moving away from the structure of working at a company to the world of self-employment comes with its own set of challenges—and, as CJ Sullivan shows, with great opportunities for learning and growth.
• How I Became A Data Scientist – No CS Degree, No BootcampFor anyone just taking their first steps in the field, Egor Howell‘s candid account of his path into data science is a must-read.
• I Spent $96k To Become a Data Scientist. Here Are 5 Crucial Lessons All Beginners Must KnowOffering a different perspective on entering the discipline, Khouloud El Alami offers practical tips on managing your data science education so that you set yourself on the right path.

Breakthroughs and Innovation

Staying up-to-date with cutting-edge research and new tools can feel overwhelming at times, which is why we have a particular soft spot for top-notch paper walkthroughs and primers on emerging libraries and models. Here are three such articles that particularly resonated with our audience.

• A New Coefficient of Correlation"What if you were told there exists a new way to measure the relationship between two variables just like correlation except possibly better"? So starts Tim Sumner‘s explainer on a groundbreaking 2020 paper.
• Intro to DSPy: Goodbye Prompting, Hello Programming!In another exciting year for open-source tools, one of the standout new arrivals was DSPy, which aims to open up LLMs for programmers and make it easier to build modular AI solutions. Leonie Monigatti‘s hands-on introduction is the perfect place to start exploring its possibilities.
• Kolmogorov-Arnold Networks: The Latest Advance in Neural Networks, Simply ExplainedKANs, "promising alternatives of Multi-Layer Perceptrons (MLPs)," made a splashy entrance last spring; Theo Wolf made their ramifications and potential benefits for ML practitioners evident with this accessible primer.

Thank you for supporting the work of our authors in 2024! If writing for TDS is one of your goals for 2025, why not get started now? Don’t hesitate to share your work with us.

Until the next Variable, coming your way in the first week of January,

TDS Team

The post 2024 Highlights: The AI and Data Science Articles That Made a Splash appeared first on Towards Data Science.

Feeling inspired to write your first TDS post before the end of 2024? We’re always open to contributions from new authors.

Our guiding principle is that it’s never a bad time to learn new things, but we also know that different moments call for different types of learning. Here at TDS, we’ve traditionally published lots of hands-on, roll-up-your-sleeves guides and tutorials as soon as we kick off a new year—and we’re sure that will be the case come January 2025, too.

For now, as we enter the peak of the holiday season, we wanted to highlight some of our best recent articles that call for a bit more reflection and a slower pace of processing: stories you can savor as you lounge on a comfy armchair, say, rather than while typing code away on your laptop (though you can do that too, of course; we won’t hold it against you!).

From the cultural impact of AI-generated content to a Bayesian analysis of dogs’ pooping habits (yes, you’ve read that right), we hope you enjoy this lineup of thought-provoking, engaging articles. And stay tuned: we can’t wait to share our 2024 highlights with you in next week’s final-edition-of-the-year Variable.

• The Economics of Artificial Intelligence – What Does Automation Mean for Workers? In his comprehensive analysis of AI’s effect on the workforce, Isaac Tham introduces a powerful framework: "AI augments or automates labor based on its performance relative to workers in a given task. If AI is better than labour, labour is automated, but if labour is better than AI, AI augments labour." He goes on to unpack the stakes, risks, and potential benefits of AI’s rapidly growing footprint.

• The Cultural Impact of AI Generated Content: Part 1 Business implications take up much of the space in conversations around AI, but as Stephanie Kirmer stresses, we shouldn’t ignore the potentially seismic shifts AI-generated content causes in the cultural sphere, too: "It would be silly to expect our ways of thinking to not change as a result of these experiences, and I worry very much that the change we’re undergoing is not for the better."

• ChatGPT: Two Years Later November 2022, when OpenAI launched the chatbot that would change everything (or at least… a lot of things), feels at once like two days and two decades ago. To help us make sense of our post-ChatGPT world, Julián Peller presents a panoramic overview of the past two years, a period of monumental transition within the "generative-AI revolution."

• The Name That Broke ChatGPT: Who is David Mayer? For anyone who enjoys their explorations of AI’s inner workings with a generous dose of intrigue and mystery, Cassie Kozyrkov‘s latest article fits the bill: it tackles some of the thorniest questions around LLM-based tools (privacy, bias, and prompt hacking, to name a few) through the example of one elusive name.

• Overcoming Security Challenges in Protecting Shared Generative AI Environments Approaching the problem of security in AI products from a different angle, Han HELOIR, Ph.D. zooms in on the particular challenges of multi-tenancy—the increasingly common situation when different groups of users (like multiple teams within a company) rely on the same data and LLM resources.

• Understanding DDPG: The Algorithm That Solves Continuous Action Control Challenges Why not take the time this holiday season to expand your knowledge of deep reinforcement learning algorithms? Sirine Bhouri‘s debut TDS article walks us through the theory and architecture behind the Deep Deterministic Policy Gradient (DDPG) algorithm, tests its performance, and examines its potential applications in bioengineering.

• LLM Routing – Intuitively and Exhaustively ExplainedWith thousands of large language models to choose from, how should practitioners decide which ones to choose for a given task? Daniel Warfield‘s accessible deep dive into LLM routing explains how this "advanced inferencing technique" streamlines this process and how the different components it relies on complement each other.

• The Intuition behind Concordance Index – Survival Analysis Understanding and preventing churn remains one of the most common goals for industry-embedded data scientists. Antonieta Mastrogiuseppe provides a thorough primer on the underlying math of survival analysis, and the key role the concordance index plays in assessing a model’s accuracy.

• Dog Poop Compass Can a 5-year-old Cavalier King Charles Spaniel teach us important lessons in Bayesian statistics? It turns out the answer is yes – as long as you follow along Dima Sergeev‘s gripping account of his attempts to detect patterns in his dog’s "bathroom" rituals.

• Causality – Mental Hygiene for Data ScienceTo round out our lineup this week, we invite you to dig into Eyal Kazin‘s thoughtful reflection on causal tools—and when (and whether) to use them. Based on his recent PyData Global conference lecture, this article balances a big-picture analysis of causal inference with the nitty-gritty factors that shape the ways we apply causal thinking in day-to-day workflows.

Thank you for supporting the work of our authors! As we mentioned above, we love publishing articles from new authors, so if you’ve recently written an interesting project walkthrough, tutorial, or theoretical reflection on any of our core topics, don’t hesitate to share it with us.

Until the next Variable,

TDS Team

The post The Economics of Artificial Intelligence, Causal Tools, ChatGPT’s Impact, and Other Holiday Reads appeared first on Towards Data Science.