Practical Lessons from Machine Learning: Logging, Notebooks, and Overnight Experiments

Machine learning (ML) often feels like a repetitive cycle: coding, waiting for results, interpreting them, and then diving back into code. While the routine may seem monotonous, there’s much to learn within this structured environment. Reflecting on my daily habit of documenting lessons over the past few years, I’ve identified three crucial insights that enhance our ML practices: keeping logging simple, using an experimental notebook, and leveraging overnight runs.

Keep Logging Simple

One of the most insightful lessons I learned was the importance of simplicity in logging experiments. For a long time, I relied on Weights & Biases (W&B) as my primary experiment logger. The platform boasts impressive features, including beautiful dashboards ideal for team collaboration. My statistics show that I trained nearly 25,000 models and dedicated around 5,000 hours of compute time—all thanks to W&B.

However, as I examined my workflow, I realized that W&B often felt like overkill for my individual projects. Many times, once I wrapped up a project, the logs accumulated dust without ever being revisited. Consequently, I made the conscious decision to simplify my approach.

Now, I record selected metrics directly into CSV and text files, using a straightforward setup that suits my needs. For hyperparameter searches, I employ Optuna, but I do so locally, saving study states to a pickle file. In the event of a crash, I can easily reload and continue my work. The key takeaway here is that logging is merely a support tool for research. Spending excessive time deciding what metrics to log can divert focus away from actual research. Embracing a more straightforward logging strategy allows me to concentrate on the essential tasks at hand.

Maintain Experimental Lab Notebooks

Though the concept of a lab notebook may conjure images of chemical experiments, in the realm of machine learning, our computers serve as our laboratories. The portability of our work allows for experiments on high-performance computing clusters, making it easier than ever to conduct multiple trials.

Inspired by a former colleague, I began to maintain a digital lab notebook—an idea that I find invaluable. Before initiating long-running experiments, I jot down what I’m testing and the rationale behind it. When I return to my results, I can quickly identify which experiments are complete and what I sought to learn from them.

This simple practice transforms the workflow. Rather than mindlessly "rerunning until it works," experiments become part of a documented feedback loop. Failures can be interpreted with greater ease, and successful outcomes become easier to replicate. This approach fosters a more intentional and thoughtful experimentation process.

Run Experiments Overnight

One small but impactful lesson I relearned recently underscores the importance of running experiments while we sleep. After discovering a bug on a Friday, I quickly patched it and reran the relevant experiments. By the following morning, I realized I had neglected to include a critical element—which meant waiting an entire day for the next results.

In machine learning, the overnight hours are precious. While we rest, our experiments should be diligently working. Not leveraging these time intervals can waste valuable compute cycles. This doesn’t imply that we should launch experiments frivolously. However, when a meaningful experiment is ready, starting it in the evening ensures that we have results to analyze the next day.

Planning plays a crucial role in effectively utilizing this time. As highlighted by Cal Newport in "Deep Work," effective workdays often commence the night before. By preparing tomorrow’s tasks today, we can set up our experiments in advance, maximizing productivity.

By embracing these lessons—keeping logging simple, maintaining a lab notebook, and running experiments overnight—machine learning practitioners can streamline their workflows, enhance productivity, and cultivate a more thoughtful approach to experimentation.