In the past four years, there’s been a lot of progress in the field of machine learning, and here’s a story seen from the outskirts.
Eight years ago, for a mock start-up project, we tried to do some basic headtracking. At that time, my professor Stéphane Mallat told us that the most efficient way to do this was the Viola-Jones algorithm, which was still based on hard-coded features (integral images and Haar features) and a hard classifier (adaboost.)(I was thrilled when a few years later Amazon Firephone was embedding similar features; unfortunately, this was a complete bomb — better technologies now exist and will make a splash pretty soon.)
By then, the most advanced book on machine learning was “Information Theory, Inference, and Learning” by David McKay, a terrific book to read, and also “Pattern Recognition and Machine Learning” by Chris Bishop (which I never read past chapter 3, lack of time.)
Oh boy, how things have changed!Now, machines are incredibly accurate at face recognition, they can drive cars, they can learn how to play video games, they can beat the best players at go (even though that might not be such an impressive feat) and even reprogram themselves. The prospects I found the most interesting is ML-synthesis, with first examples such as DeepDream, generating fake speech, or even ML-denoising/upscaling (now developed by Samsung to improve their display, which have too many pixels anyways:) Here’s a list of cool ongoing projects.Somehow, machine learning is sucking up all the air; there are podcasts about Machine Learning (such as “Talking Machines“), Ali Baba’s Jack Ma is about to spent $15billions on Machine learning and derivatives, and my fellow at Berkeley Lab all want to become data scientists (through the Insight fellowship program.) Not to mention the many researcher who pepper “Machine learning” in their grant proposal to get magic money (e.g. in my field — EUV lithography, or in computational imaging, though the good researchers know where to draw the line.)The last time I saw my former classmate Sebatien Bubeck (his blog: I’m a bandit), it was in Berkeley in 2014. At that time, I was working on coded aperture, and all the rage was about Donoho and Candès‘ compressed sensing. The news just broke that Yann LeCun had joined Facebook during NIPS (Sebastien is now at Microsoft Research.) I remembering him mocking research in Deep Learning, because no one understood how things worked — and we still don’t (that’s a major problem of the “AI revolution”: it is impossible to debug; if a car crash after misinterpreting a sign or if an application is racist, there’s nothing much you can do about it.) — except maybe for… Stéphane Mallat, a pioneer in the field of wavelet transforms, possibly the last mathematical tool that I could grasp.It was a great pleasure to learn that Stéphane Mallat is now becoming a professor at College de France, the most prestigious institution for a French scientist. During that time, he had worked on ways to understand why neural networks work so well, and in doing so he found (with his student Joan Bruna) a way to compete with DNN without creating a network from scratch and thus ad hoc features (the technique is called Invariant Scattering Convolution Networks, and is basically a mix of wavelet transforms and Fourier ring coefficients)It seems that all the rage nowadays is with adversarial network — I find that to be an interesting parallel between machine learning and the theory of the mind developed by Mercier and Sperber in “Why do humans reason? Arguments for an argumentative theory.”I wonder what will happen when machines will be able to understand how they work and rewire themselves to be more efficient. We might get instant boosts in performance and live forever happy, or on the contrary rub the singularity and be burnt by it.