Technology has accomplished many wonderful things, but one thing I'd like to focus on is the value of automation. Whenever we manage to automate something we can deploy it at scale and move onto using our human brains to work on more creative and thoughtful things.
A relevant example of this is the monitoring of wildlife with camera traps. At first scientists had to spend quite a lot of time going through photos, many of which had nothing of interest in them, looking for and then identifying wildlife. This obviously limited the scale of the camera trap surveys and took valuable time away from actually interpreting and reporting on the data. Nowadays, computer vision is used to comb through the many tens of thousands of images produced in any particular camera trap survey. This both frees up the scientists to focus on using their creative, thinking capacities while allowing for far grander scale and continuity within the survey itself. Indeed, entirely new applications of camera traps become possible with machines in the loop. For example camera traps can be used as continuous monitoring and alerting systems if there's a machine brain watching at all times.
This example rather nicely illustrates a standard part of any scientific process - transforming data from raw into more meaningful forms. Data is almost always captured in the "raw" because doing so is cost effective. Taking pictures is easy; having someone sit out in the field and count wildlife over days or weeks is not. But while raw data is cost effective it usually lacks the context and meaning required to weave narrative with. So the data has to be "enriched" into more valuable forms.
When you think about it, though, this enrichment is really just a transformation of some kind. A big vector of data X goes in and a vector of more meaningful information Y comes out. And the function F which does the transforming is the menial task that, if automated, would allow our stewards to spend more time exercising their humanity while making deploying F at scale possible. So therein is our question - how do we go about creating machine based transformations really easy?
Enter Machine Learning
The first thing we must ask of any of these transformations is whether there are a clear set of rules that can be followed. If that's the case, then all we need is some code and we're done. But if the rules are not clear or at least not clearly coded we need a different tool - machine learning.
Machine learning is by definition all about allowing a machine to learn, for itself, the relationship between a set of features X and a set of targets Y. Which is exactly what we're looking for! Now typically machine learning is looked at as something quite difficult so given we want to make creating transformations easy we've got to ask ourselves what makes machine learning difficult?
Given all of the research that's already been done on machine learning, today (in my opinion) machine learning remains difficult largely because of the vast array of options out there for how to tackle a problem. You must:
Choose a good representation of the transformation you want
Have enough data to teach the machine with
Choose good features
Choose good normalization and data cleaning patterns
Choose a good model
Choose the right parameters for that model
Choose the right evaluation strategy for your model
Let's take a look at each of these in turn and see what we can do to make life easier for ourselves and our stewards.
Tackling Difficulties
Getting Data
Machines need training data to work with - that is cases which have been labelled with ground truth. A good example of training data would be photos already labelled with what's in them. Getting training data is perhaps the most difficult part of any machine learning problem and this is especially true in our particular case. Remember the whole reason we're here is that simple rules didn't cut it - which means we need humans in the loop and humans are typically expensive - particularly if they have to be professionals. Thankfully, like with data collection, citizen science can come to our aid as a way to get cheap (if not free) crowd-sourced labels for our data. Websites like zooniverse already allow for folks to post data that needs labeling in a community of citizen labelers!
Unfortunately there's a catch. Crowdsourcing works when the task is simple enough for someone with little to no training to accurately do (like identifying if something is in an image or not and where it is) but sometimes a particular label can only be provided by an expert. For example, suppose we've got pictures of two subspecies of elephant. You might not expect the average person to know which is which consistently. In these situations we need to do everything we can to use the expert's time as efficiently as possible (not that we'd use crowdsourced time inefficiently, of course).
Thankfully there's an answer to this call in the form of Active Learning. This is a paradigm of human in the loop machine learning where the machine determines which data points (for which it doesn't have labels already) would be most useful for its learning. These model chosen examples can then be sent to humans to be labelled, the model can retrain itself with the new information, and the whole process can repeat. In theory this means that you don't have to label as much data because the machine is being specific about what it actually needs to learn.
So, with citizen science and active learning we can significantly reduce the headache to our stewards in creating the data we need to train our models.
Choosing the Right Parameters
All models come with a series of knobs called hyper-parameters. By turning these knobs we change the specifics of the model and therefore its shape and flexibility. Thankfully there are many standardized procedures in place for automatically tuning these hyper-parameters. So we can make these procedures standard and allow our stewards to get on with better things.
Choosing the Right Model
Today, for the vast majority of problems, there are already a pretty standard set of well developed models and model architectures. Put another way, if you know the problem you are dealing with, a handful of models are all you really need to know about to get most of the bang for your buck. Better yet all of these models follow more or less the same APIs so you can build, train, and test all of them with the same code. What this means for us is that your steward only needs to specify the kind of problem (computer vision, classification, regression, time series prediction) and we can choose the best model by simply building the full handful and selecting the best out of the bunch. Yet another thing automated away!
Choosing Good Features
Features are the specific representations of the data that you want the machine to use. As such they themselves are actually transformations. So if you have your raw data X, features (and indeed normalizations) really create a new vector U that is given to the machine to produce Y. Given that we've associated our labels with X and not with U, features can really just be swapped out ad hoc without disrupting the rest of our pipeline. This means the only "difficulty" that comes with features is just creating them in the first place but this is always a creative endeavor and thus totally belongs in the human realm. In other words everything but the creative part has already been automated away by associated our Y directly with X and by making model selection, training, tuning, and testing automatic.
Representation and Evaluation
Evaluation for its part is easy. Our stewards just need to select the criterion for evaluation (something a machine should never do on its own) and then the rules for that evaluation are coded up. Over the years dozens of such evaluations have been dreamed and coded up and it's highly unlikely that stewards would need something new.
As far as representation is concerned this is a place where humans must be the ones making the determination. You can't expect anyone (and especially not a machine) to know what to do if you don't tell them. What's interesting about this part of the problem is that if it's difficult to actually figure out what you want the machine to do you probably have no idea what you're doing yourself. And if you don't know that, you can't automate it. So this bit should be easy once you know what you're automating.
Summary
Okay that was a lot. Let's summarize.
Representing the problem is only hard if you can't describe what you want automated.
There's already a vast array of evaluation techniques - why not just allow our stewards to pick one?
Gathering labels for data is expensive. We can make it less so by engaging citizen scientists to crowdsource labels and using active learning to allow the machine to direct the humans to the training it needs the most.
Even if we had a model selected, we'd still need to tune its hyper-parameters. Thankfully well established, automatic procedures already exist for this.
Thanks to all the research that's been done, for any particular kind of problem there's a handful of agreed upon best model candidates. Thanks to the fact that they usually share APIs we can simply train them all and pick the best one. Model selection automated.
By associating our Y directly with our X and by automating selection, tuning, training, and testing of our models, the only manual part of creating features and normalizations is actually building them - which is an entirely creative process and thus one better suited to humans anyways.
Effectively then, all this represents is pulling together existent tools and knowledge into a framework that allows for plug and play features, built in model selection, tuning, training, and testing, built in active learning, and easy accessibility to crowdsourcing.
With this framework in place we'd have automated everything about building transformations except the bits where we absolutely need humans in the loop. Then with these automated transformations in hand we can deploy at scale and let our stewards get back to the creative, thoughtful work we, as humans, excel at.
This is project Kameleon.
Comments