The most important code structuring advice

2023-02-24

After 5 months on the CQ4DS discord [click to join], I realised that every single time I am giving the same advice to anyone who turns up at the #roast_my_code channel.

(Thanks to the ones who do, it is a huge learning experience for the community, and it takes a lot of courage to share your work.)

❗So here is the advice:

"Wrap your hard decisions into a class." (I don't know who said that, but one excellent quote)

The main problem you face when experimenting in Data Science is that you would like to try things speculatively but don't want to lose your previous progress.

If you have ever panicked about recovering the state of your notebooks to a point where it was generating the "right" numbers, you know what I am talking about.

So how do you keep your old code and try things that might not work?

🛠 Dependency Injection:

• Create a class

• Locate the part of your code that you want to change

• Wrap the code into a function

• Move the function to the class

• Construct this class together with your experiment class

• Pass the instance to the experiment class as a parameter

• Call the function at the place where you removed the code

At this point, you decoupled the part you want to change from your old implementation. There are some details that you need to think through, like what are the arguments of the function and what are the parameters of the class, but these are straightforward. These steps also help to clean the code and make it more readable.

Once you figure this out, you can write a new version of the class and pass that at the main construction.

At this point, you can also clarify the above argument vs parameter question. Both classes must have the same interface for the run() function but can have different parameters for the class. But the class (instance) variables are fixed in case you call run() multiple times with different arguments.

Once you have the results of the new experiment, you can decide whether you want to switch back or create a new component to experiment with.

More content like this:

• How can a Data Scientist refactor Jupyter notebooks?

• Pydata London 2022 slides: "Clean Architecture: How to structure your ML projects to reduce technical debt"

Or: