Optimization plays an important role in many machine learning methods. However, the two areas have very different focuses. The gap has caused that, on the one hand, some machine learning tasks may not suitably use optimization techniques, and on the other hand, optimization researchers may wrongly consider irrelevant issues when applying their methodology to machine learning. In this talk I will discuss my experiences on kernel and linear classification. In particular, we discuss support vector machines (SVM), which involve some challenging optimization problems. As a machine learning researcher with an optimization background, I will show lessons learned in the past and how we eventually construct some widely used machine learning software.