D] Confession as an AI researcher; seeking advice

I have a confession to make.


I was a CS major in college and took very few advanced math or stats courses. Besides basic calculus, linear algebra, and probability 101, I took only one machine learning class. It was about very specific SVMs/decision tree/probabilistic graphical models that I rarely encounter today.


I joined a machine learning lab in college and was mentored by a senior PhD. We actually had a couple of publications together, though they were nothing but minor architecture changes. Now that I’m in grad school doing AI research full-time, I thought I could continue to get away with zero math and clever lego building. Unfortunately, I fail to produce anything creative. What’s worse, I find it increasingly hard to read some of the latest papers, which probably don’t look complicated at all to math-minded students. The gap in my math/stats knowledge is taking a hefty toll on my career.


For example, I’ve never heard of the term “Lipschitz” or “Wasserstein distance” before, so I’m unable to digest the Wasserstein GAN paper, let alone invent something like that by myself. Same with f-GAN , and SeLU (. I don’t have the slightest clue what the 100-page SeLU proof is doing. The “Normalizing Flow” ( paper even involves physics (Langevin Flow, stochastic differential equation) … each term seems to require a semester-long course to master. I don’t even know where to start wrapping my head around.


I’ve thought about potential solutions. The top-down approach is to google each unfamiliar jargon in the paper. That doesn’t work at all because the explanation of 1 unknown points to 3 more unknowns. It’s an exponential tree expansion. The alternative bottom-up approach is to read real analysis, functional analysis, probability theory textbooks. I prefer a systematic treatment, but …


reading takes a huge amount of time. I have the next conference deadline to meet, so I can’t just set aside two months without producing anything. My advisor wouldn’t be happy.


but if I don’t read, my mindless lego building will not yield anything publishable for the next conference. What a chicken-and-egg vicious cycle.


the “utility density” of reading those 1000-page textbooks is very low. A lot of pages are not relevant, but I don’t have an efficient way to sift them out. I understand that some knowledge might be useful some day, but the reward is too sparse to justify my attention budget. The vicious cycle kicks in again.


in the ideal world, I can query an oracle with “Langevin flow”. The oracle would return a list of pointers, “given your current math capability, you should first read chapter 7 of Bishop’s PRML book, and then chapter 10 of information theory, and then chapter 12 of …”. Google is not such an oracle for my purpose.


I’m willing to spend 1 - 2 hours a day to polish my math, but I need a more effective oracle. Is it just me, or does anyone else have the same frustration?


EDIT: I'd appreciate it if someone could recommend specific books or MOOC series that focus more on intuition and breadth. Google lists tons of materials on real analysis, functional analysis, information theory, stochastic process, probability and measure theory, etc. Not all of them fit my use case, since I'm not seeking to redo a rigorous math major. Thanks in advance for any recommendation!


EDIT: wow, I didn't expect so many people from different backgrounds to join the discussion. Looks like there are many who resonate with me! And thank you so much for all the great advice and recommendations. Please keep adding links, book titles, and your stories! This post might help another distraught researcher out of the Valley 




자세한 글은 밑에서 

reddit.com/r/MachineLearning/comments/73n9pm/d_confession_as_an_ai_researcher_seeking_advice/