{"id":1803,"date":"2018-01-10T23:06:16","date_gmt":"2018-01-11T07:06:16","guid":{"rendered":"http:\/\/antoine.wojdyla.fr\/blog\/?p=1803"},"modified":"2018-01-16T15:25:54","modified_gmt":"2018-01-16T23:25:54","slug":"learning-deep","status":"publish","type":"post","link":"http:\/\/antoine.wojdyla.fr\/blog\/2018\/01\/10\/learning-deep\/","title":{"rendered":"Learning Deep"},"content":{"rendered":"<p>In the past four years, there&#8217;s been a lot of progress in the field of machine learning, and here&#8217;s a story seen from the outskirts.<\/p>Eight years ago, for a mock start-up project, we tried to do some basic headtracking. At that time, my professor St\u00e9phane Mallat told us that the most efficient way to do this was the <a href=\"https:\/\/en.wikipedia.org\/wiki\/Viola%E2%80%93Jones_object_detection_framework\" onclick=\"_gaq.push(['_trackEvent', 'outbound-article', 'https:\/\/en.wikipedia.org\/wiki\/Viola%E2%80%93Jones_object_detection_framework', 'Viola-Jones algorithm']);\" >Viola-Jones algorithm<\/a>, which was still based on hard-coded features (integral images and Haar features) and a hard classifier (adaboost.)<\/p><iframe loading=\"lazy\" title=\"Perceptive 3D with webcam headtracking\" width=\"580\" height=\"435\" src=\"https:\/\/www.youtube.com\/embed\/_WKJOXqi_0w?feature=oembed\" frameborder=\"0\" allow=\"accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share\" referrerpolicy=\"strict-origin-when-cross-origin\" allowfullscreen><\/iframe><\/p>\n<h6><em>(I was thrilled when a few years later Amazon Firephone was <a href=\"https:\/\/www.theverge.com\/2014\/6\/18\/5821294\/amazons-fire-phone-uses-four-cameras-to-offer-3d-perspective\" onclick=\"_gaq.push(['_trackEvent', 'outbound-article', 'https:\/\/www.theverge.com\/2014\/6\/18\/5821294\/amazons-fire-phone-uses-four-cameras-to-offer-3d-perspective', 'embedding similar features']);\" >embedding similar features<\/a>; unfortunately, this was a complete bomb &#8212; <a href=\"https:\/\/www.leiainc.com\/\" onclick=\"_gaq.push(['_trackEvent', 'outbound-article', 'https:\/\/www.leiainc.com\/', 'better technologies']);\" >better technologies<\/a> now exist and will make a splash pretty soon.)<\/em><\/h6><p>By then, the most advanced book on machine learning was &#8220;<a href=\"http:\/\/www.inference.org.uk\/itprnn\/book.pdf\" onclick=\"_gaq.push(['_trackEvent','download','http:\/\/www.inference.org.uk\/itprnn\/book.pdf']);\" >Information Theory, Inference, and Learning<\/a>&#8221; by David McKay, a terrific book to read, and also &#8220;<a href=\"http:\/\/www.springer.com\/us\/book\/9780387310732\" onclick=\"_gaq.push(['_trackEvent', 'outbound-article', 'http:\/\/www.springer.com\/us\/book\/9780387310732', 'Pattern Recognition and Machine Learning']);\" >Pattern Recognition and Machine Learning<\/a>&#8221; by Chris Bishop (which I never read past chapter 3, lack of time.)<\/p>Oh boy, how things have changed!<\/p><!--more--><\/p>Now, machines are incredibly accurate at face recognition, they can drive cars, they can <a href=\"http:\/\/www.nature.com\/nature\/journal\/v518\/n7540\/full\/nature14236.html\" onclick=\"_gaq.push(['_trackEvent', 'outbound-article', 'http:\/\/www.nature.com\/nature\/journal\/v518\/n7540\/full\/nature14236.html', 'learn how to play video games']);\" >learn how to play video games<\/a>, they <a href=\"https:\/\/www.nature.com\/articles\/nature24270\" onclick=\"_gaq.push(['_trackEvent', 'outbound-article', 'https:\/\/www.nature.com\/articles\/nature24270', 'can beat the best players at go']);\" >can beat the best players at go<\/a> (even though that might not be <a href=\"https:\/\/medium.com\/@josecamachocollados\/is-alphazero-really-a-scientific-breakthrough-in-ai-bf66ae1c84f2\" onclick=\"_gaq.push(['_trackEvent', 'outbound-article', 'https:\/\/medium.com\/@josecamachocollados\/is-alphazero-really-a-scientific-breakthrough-in-ai-bf66ae1c84f2', 'such an impressive feat']);\" >such an impressive feat<\/a>) and even <a href=\"http:\/\/dx.doi.org\/10.1038\/nature20101\" onclick=\"_gaq.push(['_trackEvent', 'outbound-article', 'http:\/\/dx.doi.org\/10.1038\/nature20101', 'reprogram themselves']);\" >reprogram themselves<\/a>. The prospects I found the most interesting is ML-synthesis, with first examples such as <a href=\"https:\/\/github.com\/google\/deepdream\" onclick=\"_gaq.push(['_trackEvent', 'outbound-article', 'https:\/\/github.com\/google\/deepdream', 'DeepDream,']);\" >DeepDream,<\/a> generating <a href=\"http:\/\/www.washington.edu\/news\/2017\/07\/11\/lip-syncing-obama-new-tools-turn-audio-clips-into-realistic-video\/\" onclick=\"_gaq.push(['_trackEvent', 'outbound-article', 'http:\/\/www.washington.edu\/news\/2017\/07\/11\/lip-syncing-obama-new-tools-turn-audio-clips-into-realistic-video\/', 'fake speech']);\" >fake speech<\/a>, or even <a href=\"https:\/\/dmitryulyanov.github.io\/deep_image_prior\" onclick=\"_gaq.push(['_trackEvent', 'outbound-article', 'https:\/\/dmitryulyanov.github.io\/deep_image_prior', 'ML-denoising\/upscaling']);\" >ML-denoising\/upscaling<\/a> (now developed by Samsung to <a href=\"https:\/\/www.youtube.com\/watch?v=dbx2bIweieg\" onclick=\"_gaq.push(['_trackEvent', 'outbound-article', 'https:\/\/www.youtube.com\/watch?v=dbx2bIweieg', 'improve their display']);\" >improve their display<\/a>, which have too many pixels anyways:) Here&#8217;s a list of <a href=\"https:\/\/medium.mybridge.co\/30-amazing-machine-learning-projects-for-the-past-year-v-2018-b853b8621ac7\" onclick=\"_gaq.push(['_trackEvent', 'outbound-article', 'https:\/\/medium.mybridge.co\/30-amazing-machine-learning-projects-for-the-past-year-v-2018-b853b8621ac7', 'cool ongoing projects']);\" >cool ongoing projects<\/a>.<\/p>Somehow, <strong>machine learning is sucking up all the air<\/strong>; there are podcasts about Machine Learning (such as &#8220;<a href=\"https:\/\/www.thetalkingmachines.com\/\" onclick=\"_gaq.push(['_trackEvent', 'outbound-article', 'https:\/\/www.thetalkingmachines.com\/', 'Talking Machines']);\" >Talking Machines<\/a>&#8220;), Ali Baba&#8217;s Jack Ma is about to spent <a href=\"https:\/\/qz.com\/1100583\/singapore-will-be-the-first-stop-in-jack-mas-15-billion-global-tech-research-expansion\/\" onclick=\"_gaq.push(['_trackEvent', 'outbound-article', 'https:\/\/qz.com\/1100583\/singapore-will-be-the-first-stop-in-jack-mas-15-billion-global-tech-research-expansion\/', '$15billions on Machine learning and derivatives']);\" >$15billions on Machine learning and derivatives<\/a>, and my fellow at Berkeley Lab all want to become data scientists (through the <a href=\"http:\/\/insightdatascience.com\/\" onclick=\"_gaq.push(['_trackEvent', 'outbound-article', 'http:\/\/insightdatascience.com\/', 'Insight fellowship program']);\" >Insight fellowship program<\/a>.) Not to mention the many researcher who pepper &#8220;Machine learning&#8221; in their grant proposal to get magic money (e.g. in my field &#8212; <a href=\"https:\/\/doi.org\/10.1364\/AO.56.006485\" onclick=\"_gaq.push(['_trackEvent', 'outbound-article', 'https:\/\/doi.org\/10.1364\/AO.56.006485', 'EUV lithography']);\" >EUV lithography<\/a>, or in computational imaging, though the <a href=\"http:\/\/doi.org\/10.1038\/523416a\" onclick=\"_gaq.push(['_trackEvent', 'outbound-article', 'http:\/\/doi.org\/10.1038\/523416a', 'good researchers know where to draw the line']);\" >good researchers know where to draw the line<\/a>.)<\/p>The last time I saw my former classmate Sebatien Bubeck (his blog: <a href=\"https:\/\/blogs.princeton.edu\/imabandit\" onclick=\"_gaq.push(['_trackEvent', 'outbound-article', 'https:\/\/blogs.princeton.edu\/imabandit', 'I&#8217;m a bandit']);\" >I&#8217;m a bandit<\/a>), it was in Berkeley in 2014. At that time, I was working on coded aperture, and all the rage was about Donoho and <a href=\"https:\/\/news.stanford.edu\/2017\/10\/11\/emmanuel-candes-wins-genius-award\/\" onclick=\"_gaq.push(['_trackEvent', 'outbound-article', 'https:\/\/news.stanford.edu\/2017\/10\/11\/emmanuel-candes-wins-genius-award\/', 'Cand\u00e8s']);\" >Cand\u00e8s<\/a>&#8216; compressed sensing. The news just broke that Yann LeCun had joined Facebook during NIPS (Sebastien is now at <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/people\/sebubeck\/\" onclick=\"_gaq.push(['_trackEvent', 'outbound-article', 'https:\/\/www.microsoft.com\/en-us\/research\/people\/sebubeck\/', 'Microsoft Research']);\" >Microsoft Research<\/a>.) I remembering him mocking research in Deep Learning, because no one understood how things worked &#8212; and we still don&#8217;t (that&#8217;s a major problem of the &#8220;AI revolution&#8221;: it is impossible to debug; if a <a href=\"https:\/\/edgylabs.com\/autonomous-cars-confused-decals-fake-signs\" onclick=\"_gaq.push(['_trackEvent', 'outbound-article', 'https:\/\/edgylabs.com\/autonomous-cars-confused-decals-fake-signs', 'car crash after misinterpreting a sign']);\" >car crash after misinterpreting a sign<\/a> or if an <a href=\"https:\/\/www.theguardian.com\/inequality\/2017\/aug\/08\/rise-of-the-racist-robots-how-ai-is-learning-all-our-worst-impulses\" onclick=\"_gaq.push(['_trackEvent', 'outbound-article', 'https:\/\/www.theguardian.com\/inequality\/2017\/aug\/08\/rise-of-the-racist-robots-how-ai-is-learning-all-our-worst-impulses', 'application is racist']);\" >application is racist<\/a>, there&#8217;s nothing much you can do about it.) &#8212; except maybe for&#8230; St\u00e9phane Mallat, a pioneer in the field of wavelet transforms, possibly the last mathematical tool that I could grasp.<\/p>It was a great pleasure to learn that St\u00e9phane Mallat is <a href=\"http:\/\/www.college-de-france.fr\/site\/stephane-mallat\/index.htm\" onclick=\"_gaq.push(['_trackEvent', 'outbound-article', 'http:\/\/www.college-de-france.fr\/site\/stephane-mallat\/index.htm', 'now becoming a professor at College de France']);\" >now becoming a professor at College de France<\/a>, the most prestigious institution for a French scientist. During that time, he had worked on ways to understand why neural networks work so well, and in doing so he found (with his student Joan Bruna) a <strong>way to compete with DNN without creating a network from scratch<\/strong> and thus <em>ad hoc<\/em> features (the technique is called <a href=\"https:\/\/arxiv.org\/pdf\/1203.1513.pdf\" onclick=\"_gaq.push(['_trackEvent','download','https:\/\/arxiv.org\/pdf\/1203.1513.pdf']);\" >Invariant Scattering Convolution Networks<\/a>, and is basically a mix of wavelet transforms and Fourier ring coefficients)<\/p><iframe loading=\"lazy\" title=\"Professor St\u00e9phane Mallat: &quot;High-Dimensional Learning and Deep Neural Networks&quot;\" width=\"580\" height=\"326\" src=\"https:\/\/www.youtube.com\/embed\/nHXO43BqeQw?feature=oembed\" frameborder=\"0\" allow=\"accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share\" referrerpolicy=\"strict-origin-when-cross-origin\" allowfullscreen><\/iframe><\/p>It seems that all the rage nowadays is with <strong>adversarial network<\/strong> &#8212; I find that to be an interesting parallel between machine learning and the theory of the mind developed by Mercier and Sperber in &#8220;<a href=\"https:\/\/www.dan.sperber.fr\/wp-content\/uploads\/2009\/10\/MercierSperberWhydohumansreason.pdf\" onclick=\"_gaq.push(['_trackEvent','download','https:\/\/www.dan.sperber.fr\/wp-content\/uploads\/2009\/10\/MercierSperberWhydohumansreason.pdf']);\" >Why do humans reason? Arguments for an argumentative theory<\/a>.&#8221;<\/p>I wonder what will happen <strong>when machines will be able to understand how they work and rewire themselves to be more efficient<\/strong>. We might get instant boosts in performance and live forever happy, or on the contrary rub the singularity and be burnt by it.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>In the past four years, there&#8217;s been a lot of progress in the field of machine learning, and here&#8217;s a story seen from the outskirts.Eight years ago, for a mock start-up project, we tried to do some basic headtracking. At that time, my professor St\u00e9phane Mallat told us that the most efficient way to do [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[11,13,14,16,5],"tags":[],"class_list":["post-1803","post","type-post","status-publish","format-standard","hentry","category-english","category-epistemology","category-projects","category-resources","category-science-life"],"_links":{"self":[{"href":"http:\/\/antoine.wojdyla.fr\/blog\/wp-json\/wp\/v2\/posts\/1803","targetHints":{"allow":["GET"]}}],"collection":[{"href":"http:\/\/antoine.wojdyla.fr\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/antoine.wojdyla.fr\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/antoine.wojdyla.fr\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/antoine.wojdyla.fr\/blog\/wp-json\/wp\/v2\/comments?post=1803"}],"version-history":[{"count":7,"href":"http:\/\/antoine.wojdyla.fr\/blog\/wp-json\/wp\/v2\/posts\/1803\/revisions"}],"predecessor-version":[{"id":1815,"href":"http:\/\/antoine.wojdyla.fr\/blog\/wp-json\/wp\/v2\/posts\/1803\/revisions\/1815"}],"wp:attachment":[{"href":"http:\/\/antoine.wojdyla.fr\/blog\/wp-json\/wp\/v2\/media?parent=1803"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/antoine.wojdyla.fr\/blog\/wp-json\/wp\/v2\/categories?post=1803"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/antoine.wojdyla.fr\/blog\/wp-json\/wp\/v2\/tags?post=1803"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}